In addition to reading this guide, run the Elasticsearch Health Check-Up. Detect problems and improve performance by analyzing your shard sizes, threadpools, memory, snapshots, disk watermarks and many more.
Free tool that requires no installation with +1000 users.
Sometimes you can observe that the CPU and load on some coordinating nodes (client nodes) is higher than others.
This can be caused by applications that are not load balancing correctly across the coordinating nodes, and are making all their HTTP calls to just one or some of the nodes.
A saturated coordinating node could cause an increase in search or indexing response latency, or an increase in write queue/search queue when the cluster is under load (despite there being processing capacity on data nodes). Eventually this could lead to queries timing out.
How to resolve it
You should fix this by putting a load balancer in front of your Elasticsearch nodes, or by including ALL of the nodes in the client application.
es = Elasticsearch( ['clientNode1', 'clientNode2','clientNode3'], http_auth=('user', 'secret'), scheme="https", port=443, )
The above example is how you can load balance across 3 nodes (you should include all the nodes) when using the python client without a load balancer. All of the official Elasticsearch clients use similar arrays in their construction.