Now throttling indexing for shard segment writing can t keep up – How to solve this Elasticsearch error

Opster Team

Aug-23, Version: 6.8-8.9

Briefly, this error occurs when Elasticsearch is unable to write new segments to the disk fast enough due to high indexing load. This could be due to insufficient disk I/O capacity, or the system is under heavy load. To resolve this issue, you can consider the following: 1) Increase the disk I/O capacity; 2) Optimize your indexing operations by reducing the number of shards, using bulk requests, or increasing the refresh interval; 3) Monitor and manage the system load to ensure it’s within the capacity of your Elasticsearch cluster.

To avoid receiving the log “now throttling indexing for shard: segment writing can’t keep up” in the future, we recommend you try running the Elasticsearch Error Check-Up which can resolve issues that cause many errors.

This guide will help you check for common problems that cause the log to appear. It’s important to understand the issues related to it, so to get started, read the general overview on common issues and tips related to the Elasticsearch concepts: indexing, indices, memory and shard.

Quick overview

This log means that Elasticsearch is putting back-pressure on the indexing process. It is essential to look at this log and take appropriate actions to ensure ES doesn’t crash. Note that this is not an error message. Make sure:

  1. Elasticsearch can cope with your indexing requirements.
  2. Search performs in near real-time, as delay in indexing leads to documents becoming unavailable for search and leads to poor search and user experience.

Here are some important terms to help you understand this log message:


Each Elasticsearch index is made up of one or more shards, and each shard is an Apache Lucene index. Elasticsearch internally uses Lucene to index and search.


Every shard is made of multiple segments, and these segments are immutable for better performance (they can be cached and used in a multithreading environment).

Indexing buffer

Each index request (a document that needs to be indexed) is first written in memory called indexing buffer (every shard has its indexing buffer), and from buffer it writes to segments. Writing in the segment means writing on the disk.

Log message deep-dive

Elasticsearch reserves the 10% (default) of total heap for all indexing activities, which is shared across all the indexing shards on the nodes.

indices.memory.index_buffer_size controls the above. 

It also maintains a set of throttled shards for which it can’t keep up with segment writes, and therefore it throttles the indexing.

When a particular shard takes more memory to write the segments, Elasticsearch puts that shard into the throttled set and stops the indexing for that shard, which is then logged in the same log message:

l"now throttling indexing for shard [{}]: segment writing can't keep up", largest.shard.shardId()); 

Troubleshooting steps

  1. Figure out which shard and index it’s throttling: shard ID is mentioned in the log message and using _cat/shards API you can locate the problematic index, shard and data node.
  2. Once you figure out the problematic index, based on your requirements you can fine-tune its indexing requirement. For example, if it’s bulk migration, disable refresh_interval.
  3. If there is far less search traffic and yours is a write-heavy system, then you can allocate more heap for indexing by increasing the default % of indices.memory.index_buffer_size to a bigger size.
  4. You can increase your RAM of data nodes and increase JVM of Elasticsearch to max 31GB for optimal performance.

Note: For detailed troubleshooting, you can contact Opster’s community support team, as this process can require more details and live debugging.

Log Context

Log “now throttling indexing for shard [{}]: segment writing can’t keep up” classname is
We extracted the following from Elasticsearch source code for those seeking an in-depth context :

                    totalBytesUsed -= largest.bytesUsed;
                    if (doThrottle && throttled.contains(largest.shard) == false) {
              "now throttling indexing for shard [{}]: segment writing can't keep up"; largest.shard.shardId());


How helpful was this guide?

We are sorry that this post was not useful for you!

Let us improve this post!

Tell us how we can improve this post?