Log Now throttling indexing for shard : segment writing cant keep up – How To Solve Related Issues



Log Now throttling indexing for shard : segment writing cant keep up – How To Solve Related Issues

Opster Team

April-20, Version: 1.7-8.0



Before you begin reading this guide, we recommend you try running the Elasticsearch Error Check-Up  which can resolve issues causing many log errors (free and no installation required)

 

This guide will help you check for common problems that cause the log “Now throttling indexing for shard : segment writing cant keep up” to appear. It’s important to understand the issues related to the log, so to get started, read the general overview on common issues and tips related to the Elasticsearch concepts: indexing, indices, memory, shard.


Advanced users might want to skip right to the common problems section in each concept or try running the Check-Up which analyses ES to discover the cause of many errors and provides suitable actionable recommendations. 

Quick Overview: 

This log means that Elasticsearch is putting back-pressure on the indexing process. It is essential to look at this log and take appropriate action to ensure ES doesn’t crash. Note that this is not an error message. Make sure:
  1. Elasticsearch can cope up with your indexing requirements.
  2. Search performs in near real-time, as delay in indexing leads to documents becoming unavailable for search and leads to lousy search and user experience.
Here are some important terms to help you understand this log message:

Shard: 

Each Elasticsearch index is made up of one or more shards, and each shard is an Apache Lucene index. Elasticsearch internally uses Lucene to index and search.

Segments: 

Every shard is made of multiple segments, and these segments are immutable for better performance (they can be cached and used in a multithreading environment).

Indexing buffer: 

Each index request (a document that needs to be indexed) is first written in memory called indexing buffer (every shard has its indexing buffer), and from buffer it writes to segments. Writing in the segment means writing on the disk.

Log message deep-dive:

Elasticsearch reserves the 10% (default) of total heap for all indexing activities, which is shared across all the indexing shards on the nodes. indices.memory.index_buffer_size controls the above.  It also maintains a set of throttled shards for which it can’t keep up with segment writes, and therefore it throttles the indexing. When a particular shard takes more memory to write the segments, Elasticsearch puts that shard into the throttled set and stops the indexing for that shard, which is then logged in the same log message:
l"now throttling indexing for shard [{}]: segment writing can't keep up", largest.shard.shardId()); 
  Troubleshooting steps:
  1. Figure out which shard and index it’s throttling: shard ID is mentioned in the log message and using _cat/shards API you can locate the problematic index, shard and data node.
  2. Once you figure out the problematic index, based on your requirement you can fine-tune it’s indexing requirement. For example, if it’s bulk migration, disable refresh_interval.
  3. If there is far less search traffic and yours is a write-heavy system, then you can allocate more heap for indexing by increasing the default % of indices.memory.index_buffer_size to a bigger size.
  4. You can increase your RAM of data nodes and increase JVM of Elasticsearch to max 31GB for optimal performance.
  5. You can stop throttle indexing to problematic index.
Note :  For detailed troubleshooting, you can contact Opster’s community support team, as this process can require more details and live debugging.

Log Context

Log”Now throttling indexing for shard [{}]: segment writing can’t keep up” classname is IndexingMemoryController.java
We extracted the following from Elasticsearch source code for those seeking an in-depth context :

                     logger.debug("write indexing buffer to disk for shard [{}] to free up its [{}] indexing buffer";
                        largest.shard.shardId(); new ByteSizeValue(largest.bytesUsed));
                    writeIndexingBufferAsync(largest.shard);
                    totalBytesUsed -= largest.bytesUsed;
                    if (doThrottle && throttled.contains(largest.shard) == false) {
                        logger.info("now throttling indexing for shard [{}]: segment writing can't keep up"; largest.shard.shardId());
                        throttled.add(largest.shard);
                        activateThrottling(largest.shard);
                    }
                }
            }




 

Related issues to this log

We have gathered selected Q&A from the community and issues from Github, that can help fix related issues please review the following for further information :

1 Best practices for ElasticSearch initial bulk import -views 1.29 K ,score –

Es5 Indexing Performance Seems Slow  

 

About Opster

Opster line of products and support services detects, prevents, optimizes and automates everything needed to manage mission-critical Elasticsearch.

Find Configuration Errors

Analyze Now