Elasticsearch Heavy Merges Were Detected



Elasticsearch Heavy Merges Were Detected

Opster Team

July 2020, Version: 1.7-8.0


Before you begin reading the explanation below, try running the free ES Health Check-Up get actionable recommendations that can improve Elasticsearch performance and prevent serious incidents. Just 2 minutes to complete and you can check your threadpools, memory, snapshots and many more

What Does it Mean?

Elasticsearch indices are stored in shards, and each shard in turn stores the data on disk in segments. Elasticsearch processes such as updates and deletion can result in many small segments being created on disk, which Elasticsearch will merge into bigger sized segments in order to optimize disk usage. The merging process uses cpu, memory and disk resources, which can slow down the cluster’s response speed.

How to Fix it

In general, the Elasticsearch merging process is controlled in the background, taking into account the other resource requirements of the cluster such as search and indexing. Therefore, it is neither necessary nor desirable to interfere with these processes.  

However, if you temporarily want to reduce or limit the merge processes, then you have two options.

The less aggressive method is to reduce the max_thread_count for the merge scheduler.

This reduces the maximum number of threads that Elasticsearch will dedicate to merge activity.  (This number cannot be reduced to zero)

PUT my_index/_settings
{"index.merge.scheduler.max_thread_count":1}

If you believe it necessary to take this step, it is highly recommended to restore the default as soon as possible to avoid other issues such as excess file descriptors on the nodes.

PUT my_index/_settings
{"index.merge.scheduler.max_thread_count":null}

Another option is to roll over the index to stop further writing to it and then directly cancel merge tasks that have been started on that index, according to the following steps.

Roll over the index

To roll over the index you will need to be using index aliasing.  

To create an index that uses aliases, you would need to either create the index with an alias

PUT /logs-000001 
{
  "aliases": {
    "my_log_alias": {}
  }
}

Or add an alias to an existing index:

POST /_aliases
{
	"actions" : [
    	{ "add" : { "index" : "logs-000001", "alias" : "my_log_alias","is_write_index":true } }
 
	]
}

Your application will then need to write to the alias and not directly to the index.

Once you have created the alias, you can force a roll over by creating the new index and changing the write index like this:

PUT logs-000002
POST /_aliases
{
	"actions" : [
    	{ "add" : { "index" : "logs-000001", "alias" : "my_log_alias","is_write_index":false } },
    	{ "add" : { "index" : "logs-000002", "alias" : "my_log_alias","is_write_index":true } }
	]
}

Cancel the merge task

Usually you should try to avoid cancelling merge tasks, since this is an optimisation operation regulated automatically by the cluster.  However as a last resort to free up resource to handle heavy search or indexing operations you can cancel the task like this:

POST _tasks/aitueURTbdu58VeiohTt8A:12345/_cancel

How To Prevent It Happening

Heavy merge activity usually occurs for indices which are being actively indexed or updated. Heavy merge activity may also be an indication of other things not being right in your cluster. In particular it could be that your shards are too large. Learn more about how that happens and how to address it here: Shards Too Large in Elasticsearch – A Complete Guide.



About Opster

Opster detects, prevents, optimizes and automates everything needed to run mission-critical Elasticsearch

Find Configuration Errors

Analyze Now