Elasticsearch Search is Slow in nodesNames

Elasticsearch Search is Slow in nodesNames

Opster Team

July 2020, Version: 1.7-7.9

In addition to reading this guide, run the free Elasticsearch Health Check-Up. Get actionable recommendations that can improve performance and prevent incidents (does not require any installation). Among the dozens of checks included are: shards sizes, search errors, thread pools, management queue size, circuit breakers and many more. Join over 700 users who use this free tool.

What Does it Mean

Slow search might become a bottleneck and may cause a waiting queue to build.

There are a number of possible causes for slow search on particular nodes.

  • Your application is not load balancing properly across all of the data nodes.
  • Search and/or indexing operations are concentrated on specific nodes because of the way shards are allocated.
  • The queries running on certain indices (concentrated on the nodes in question) are slow and need optimization.
  • There are other processes (such as merges or snapshots) running on the nodes in question which are using resources on the slow nodes.

How to Fix it

Look at the monitoring data to determine which indices are receiving most search / indexing operations. Often you will see that certain nodes have a high CPU usage, and that there are certain “hot” index shards that exist on those nodes. For example, imagine the following scenario:

  • Index A has a high search rate  
  • Index A has  shards only on node 3 and 4
  • Nodes 3 and 4 have higher cpu usage than the other data nodes.

Index A is a strong candidate to be one of the  causes of the problem.
Check to see whether you have any of the issues described in loaded data nodes.

Activate slow logging

If you have slow logging activated, check which logs are taking the most time. In particular look for aggregations with large “size” values or nested queries which are the most common culprits of slow queries.  

To activate slow logging for a given index you can run the following command:

PUT /my-index/_settings
  "index.search.slowlog.threshold.query.warn": "10s",
  "index.search.slowlog.threshold.query.info": "5s",
   "index.search.slowlog.level": "info"

To save resources it is best to leave slow logging deactivated when you have finished using the value of -1.

PUT /my-index/_settings
  "index.search.slowlog.threshold.query.warn": "-1",
  "index.search.slowlog.threshold.query.info": "-1",
  "index.search.slowlog.level": "info"

Once you have activated slow logging you can see the queries which are taking a long time.  Look for the issues mentioned in the article 10 Important Tips to Improve Search in Elasticsearch

Further Reading on Slow Search

Elasticsearch Slow Log Search Queries – A Complete Guide.

Improve Elasticsearch Performance

Run The Analysis