Elasticsearch High Cluster Pending Tasks

Elasticsearch High Cluster Pending Tasks

Opster Team

March 2021


In addition to reading this guide, we recommend you run the Elasticsearch Health Check-Up. It will detect issues and improve your Elasticsearch performance by analyzing your shard sizes, threadpools, memory, snapshots, disk watermarks and more.

The Elasticsearch Check-Up is free and requires no installation.

To improve your task management in Elasticsearch, we recommend you run the Elasticsearch Configuration Check-Up. The Check-Up will also help you optimize other important settings in Elasticsearch to improve performance.

What are cluster pending tasks?

Cluster pending tasks are updates to the cluster state which may have been initiated directly by a user or by the cluster itself.

Note that cluster pending tasks are specific tasks relating to the cluster state, and are not necessarily the same as the tasks from the task API (although there may be some overlap).  The task API relates to tasks created by users or the cluster but these are not necessarily related to cluster state. The reason to be particularly concerned about cluster state is because delays to these tasks can cause the cluster to lose data coherence.

How to resolve

You can list the cluster pending tasks by running:

GET _cluster/pending_tasks

It is very common for this to return an empty list, because cluster tasks are usually carried out very quickly, and so the queue of tasks to be carried out is empty.

However if there are pending tasks in the queue, you may get a response like this:

     {
         "insert_order": 109,
         "priority": "URGENT",
         "source": "create-index [logs_23], cause [api]",
         "executing" : true,
         "time_in_queue_millis": 76,
         "time_in_queue": "76ms"
      },
      {
         "insert_order": 36,
         "priority": "HIGH",
         "source": "shard-started ([logs_21][1], node[dMooQyuriet30A], [P], s[INITIALIZING]), reason [after recovery from shard_store]",
         "executing" : false,
         "time_in_queue_millis": 642,
         "time_in_queue": "642ms"
      },
      {
         "insert_order": 66,
         "priority": "HIGH",
         "source": "shard-started ([logs_2][0], node[dMooQyuriet30A], [P], s[INITIALIZING]), reason [after recovery from shard_store]",
         "executing" : false,
         "time_in_queue_millis": 651,
         "time_in_queue": "651ms"
      }
  ]
}

The insert_order indicates the order in which items were added to the queue, executing (true or false) indicates whether or not the job is being executed currently, while the source provides an explanation of why the task is required.

The Cluster Pending Tasks are usually carried out automatically and will resolve themselves with no operator intervention. Bear in mind that if you have pending tasks then this is likely to be the effect and not the cause of a problem with the cluster.

To diagnose the issue further it is useful to check the logs on the current active master node, looking for errors or issues which prevent the master node from carrying out tasks as expected.

Typical causes may be:



Run the Check-Up to get a customized report like this:

Analyze your cluster