Now throttling indexing for shard : segment writing cant keep up – How to solve related issues

Average Read Time

3 Mins

Now throttling indexing for shard : segment writing cant keep up – How to solve related issues

Opster Team

Feb-21, Version: 1.7-8.0

To avoid receiving the log “now throttling indexing for shard: segment writing can’t keep up” in the future, we recommend you try running the Elasticsearch Error Check-Up which can resolve issues that cause many errors.


This guide will help you check for common problems that cause the log to appear. It’s important to understand the issues related to it, so to get started, read the general overview on common issues and tips related to the Elasticsearch concepts: indexing, indices, memory and shard.

Quick overview

This log means that Elasticsearch is putting back-pressure on the indexing process. It is essential to look at this log and take appropriate actions to ensure ES doesn’t crash. Note that this is not an error message. Make sure:

  1. Elasticsearch can cope with your indexing requirements.
  2. Search performs in near real-time, as delay in indexing leads to documents becoming unavailable for search and leads to poor search and user experience.

Here are some important terms to help you understand this log message:

Shard

Each Elasticsearch index is made up of one or more shards, and each shard is an Apache Lucene index. Elasticsearch internally uses Lucene to index and search.

Segments

Every shard is made of multiple segments, and these segments are immutable for better performance (they can be cached and used in a multithreading environment).

Indexing buffer

Each index request (a document that needs to be indexed) is first written in memory called indexing buffer (every shard has its indexing buffer), and from buffer it writes to segments. Writing in the segment means writing on the disk.

Log message deep-dive

Elasticsearch reserves the 10% (default) of total heap for all indexing activities, which is shared across all the indexing shards on the nodes.

indices.memory.index_buffer_size controls the above. 

It also maintains a set of throttled shards for which it can’t keep up with segment writes, and therefore it throttles the indexing.

When a particular shard takes more memory to write the segments, Elasticsearch puts that shard into the throttled set and stops the indexing for that shard, which is then logged in the same log message:

l"now throttling indexing for shard [{}]: segment writing can't keep up", largest.shard.shardId()); 

Troubleshooting steps

  1. Figure out which shard and index it’s throttling: shard ID is mentioned in the log message and using _cat/shards API you can locate the problematic index, shard and data node.
  2. Once you figure out the problematic index, based on your requirements you can fine-tune its indexing requirement. For example, if it’s bulk migration, disable refresh_interval.
  3. If there is far less search traffic and yours is a write-heavy system, then you can allocate more heap for indexing by increasing the default % of indices.memory.index_buffer_size to a bigger size.
  4. You can increase your RAM of data nodes and increase JVM of Elasticsearch to max 31GB for optimal performance.

Note: For detailed troubleshooting, you can contact Opster’s community support team, as this process can require more details and live debugging.

Log Context

{{mpg_logstarted}}

Log “Now throttling indexing for shard [{}]: segment writing can’t keep up” classname is IndexingMemoryController.java.
We extracted the following from Elasticsearch source code for those seeking an in-depth context :

                     logger.debug("write indexing buffer to disk for shard [{}] to free up its [{}] indexing buffer";
                        largest.shard.shardId(); new ByteSizeValue(largest.bytesUsed));
                    writeIndexingBufferAsync(largest.shard);
                    totalBytesUsed -= largest.bytesUsed;
                    if (doThrottle && throttled.contains(largest.shard) == false) {
                        logger.info("now throttling indexing for shard [{}]: segment writing can't keep up"; largest.shard.shardId());
                        throttled.add(largest.shard);
                        activateThrottling(largest.shard);
                    }
                }
            }




 

Run the Check-Up to get a customized report like this:

Analyze your cluster