How To Solve Issues Related to Log – Now throttling indexing for shard : segment writing cant keep up

How To Solve Issues Related to Log – Now throttling indexing for shard : segment writing cant keep up

Updated: April-20

Elasticsearch Version: 1.7-8.0

Background

Before you begin reading this guide try our beta Elasticsearch Health Check-Up it analyses JSON’s to provide personalized recommendations that can improve your clusters performance.


To troubleshoot log “Now throttling indexing for shard : segment writing cant keep up” it’s important to understand a few problems related to Elasticsearch concepts indexing, indices, memory, shard. See bellow important tips and explanations on these concepts

Quick Overview: 

This log means that Elasticsearch is putting back-pressure on the indexing process. It is essential to look at this log and take appropriate action to ensure ES doesn’t crash. Note that this is not an error message. Make sure:
  1. Elasticsearch can cope up with your indexing requirements.
  2. Search performs in near real-time, as delay in indexing leads to documents becoming unavailable for search and leads to lousy search and user experience.
Here are some important terms to help you understand this log message:

Shard: 

Each Elasticsearch index is made up of one or more shards, and each shard is an Apache Lucene index. Elasticsearch internally uses Lucene to index and search.

Segments: 

Every shard is made of multiple segments, and these segments are immutable for better performance (they can be cached and used in a multithreading environment).

Indexing buffer: 

Each index request (a document that needs to be indexed) is first written in memory called indexing buffer (every shard has its indexing buffer), and from buffer it writes to segments. Writing in the segment means writing on the disk.

Log message deep-dive:

Elasticsearch reserves the 10% (default) of total heap for all indexing activities, which is shared across all the indexing shards on the nodes. indices.memory.index_buffer_size controls the above.  It also maintains a set of throttled shards for which it can’t keep up with segment writes, and therefore it throttles the indexing. When a particular shard takes more memory to write the segments, Elasticsearch puts that shard into the throttled set and stops the indexing for that shard, which is then logged in the same log message:
l"now throttling indexing for shard [{}]: segment writing can't keep up", largest.shard.shardId()); 
  Troubleshooting steps:
  1. Figure out which shard and index it’s throttling: shard ID is mentioned in the log message and using _cat/shards API you can locate the problematic index, shard and data node.
  2. Once you figure out the problematic index, based on your requirement you can fine-tune it’s indexing requirement. For example, if it’s bulk migration, disable refresh_interval.
  3. If there is far less search traffic and yours is a write-heavy system, then you can allocate more heap for indexing by increasing the default % of indices.memory.index_buffer_size to a bigger size.
  4. You can increase your RAM of data nodes and increase JVM of Elasticsearch to max 31GB for optimal performance.
  5. You can stop throttle indexing to problematic index.
Note :  For detailed troubleshooting, you can contact Opster’s community support team, as this process can require more details and live debugging.

Log Context

Log”Now throttling indexing for shard [{}]: segment writing can’t keep up” classname is IndexingMemoryController.java
We extracted the following from Elasticsearch source code for those seeking an in-depth context :

                     logger.debug("write indexing buffer to disk for shard [{}] to free up its [{}] indexing buffer";
                        largest.shard.shardId(); new ByteSizeValue(largest.bytesUsed));
                    writeIndexingBufferAsync(largest.shard);
                    totalBytesUsed -= largest.bytesUsed;
                    if (doThrottle && throttled.contains(largest.shard) == false) {
                        logger.info("now throttling indexing for shard [{}]: segment writing can't keep up"; largest.shard.shardId());
                        throttled.add(largest.shard);
                        activateThrottling(largest.shard);
                    }
                }
            }




Related issues to this log

We have gathered selected Q&A from the community and issues from Github, that can help fix related issues please review the following for further information :

1 Best practices for ElasticSearch initial bulk import

1.29 K 

Es5 Indexing Performance Seems Slow  

About Opster

Opster detects root causes of Elasticsearch problems, provides automated recommendations and can perform various actions to prevent issues and optimize performance

Find Configuration Errors

Analyze Now