Elasticsearch Elasticsearch Hotspots – Load Balancing, Data Allocation and How to Avoid Hotspots

Average Read Time

3 Mins

Elasticsearch Elasticsearch Hotspots – Load Balancing, Data Allocation and How to Avoid Hotspots

Opster Team

April-2022

Average Read Time

3 Mins

Opster Team

October 2021

Average Read Time

3 Mins


In addition to reading this guide, we recommend you run the Elasticsearch Health Check-Up. It will detect issues and improve your Elasticsearch performance by analyzing your shard sizes, threadpools, memory, snapshots, disk watermarks and more.

The Elasticsearch Check-Up is free and requires no installation.

To avoid hotspots in Elasticsearch and OpenSearch, we recommend you run the free Check-Up. The Check-Up will also help you optimize other important settings to improve performance.

Run the Elasticsearch check-up to receive recommendations like this:

checklist Run Check-Up
error

An indexing burst is affecting the performance of the following nodes

error-img

Description

The node is unable to keep up with indexing requests, and as a result indexing requests are being queued. If the write queue reaches full capacity, index requests will be rejected, which may cause data loss if...

error-img

Recommendation

In order to resolve your indexing bursts, based on your specific ES deployment, we recommend that you...

1

X-PUT curl -H "Content-Type: application/json" [customized recommendation]

This article refers both to hotspots in Elasticsearch and hotspots in OpenSearch, as currently everything explained and advised applies to both.

What are hotspots in Elasticsearch/OpenSearch?

Hotspots is a term referring to a situation in which a cluster with multiple nodes is not balanced – some nodes, for whatever reason, are handling greater load than other nodes. The nodes that are working harder are called hotspots. 

Why is it important to take care of hotspots?

Hotspots should not be ignored. When they occur, they form bottlenecks and become the weakest link in the ES/OS process, harming the performance of the entire cluster. 

How are hotspots formed?

Hotspots are formed when data is not allocated evenly across the cluster. This can happen for several reasons:

  1. Data is not allocated evenly across the cluster because sharding hasn’t been configured optimally by the user. An index either has fewer shards than it should or has many more than it should (respectively undersharding and oversharding) . For guidelines on how to choose the correct number of shards for your system and configure them correctly, see this guide
  2. For large, multi-tenant deployments, data may not be allocated evenly for a different reason. Elasticsearch can evenly spread shards and indices across nodes so long as the deployment is relatively small / single tenant. Once the system is more complex, the automatic process performs less well. In general, Elasticsearch ensures that every node has the same number of shards (regardless of their sizes!), but because the load on each shard is different, especially in multi-tenancy situations, some nodes can become more loaded than others. 
  3. Scaling up the number of data nodes can cause a hotspot to form. Balancing shards across all nodes is done automatically by Elasticsearch, but there are some scenarios in which hotspots form because the rebalancing process takes too long. This can occur when scaling up, as the balancing process will take a long time when nodes have hundreds or thousands of shards. During this time the cluster is still unbalanced, and any new index being created during that time would be allocated to the new nodes which will be carrying the majority of the indexing activity and can become hotspots.
  4. When a two-node cluster loses one node, replicas automatically get unassigned and all shards present on the remaining node get promoted to primary. When the second node gets back up, it will only host replica shards, hence all the write activity will happen on a single node. The same situation can also arise with bigger HA clusters with forced zone awareness configured.

Impact of hotspots on the cluster

Hotspots negatively impact clusters and performance in multiple ways:

Real-time indexing 

Hotspots can cause indexing operations to be delayed and suffer from high latency and lag. This in turn causes fresh data to not be visible in real-time. 

Search performance

Hotspots can also cause searches to perform less well. The overloaded node will be lacking sufficient resources to return timely results, and coordinating requests that have at least one shard on this node will be affected, even if most of the shards are not on the hotspot node. 

Disconnected nodes 

If a certain node is too loaded, it could stop responding to discovery requests, which would then lead to the master disconnecting it from the cluster. Disconnected nodes will then lose their shards which will get reassigned to other live nodes. This can cause cascading effects where the shards are constantly being unassigned/reassigned across nodes in the cluster, causing a lot of traffic (+increase inter-node data transfer costs) and degraded performance. 

How to resolve hotspots

To resolve hotspots, the first thing you need is visibility into what is causing the hotspot itself. If you don’t know the source, you won’t be able to address it and get to the correct resolution path. 

First, you have to locate which node/s are the bottleneck. Then, you need to take all indices that have shards on these nodes and determine whether:

  • The index shards are spread out evenly across the data nodes or not
  • Whether the number of primary shards is correct per the indexing load
  • Whether the shard sizes are too large

Once you’ve achieved this level of visibility and detected the nodes and unbalanced indices, you need to react with the appropriate actions on these indices. If an index doesn’t have enough primary shards, for instance, then you need to replace the index with an index with more primary shards or just roll over the index after having changed the default number of primary shards in the index template settings. You can also move shards from the loaded nodes to less loaded ones and add or remove replicas as needed. 

One way to achieve this level of visibility is to use the Shard View option in AutoOps to automatically detect and resolve hotspots with recommendations tailored to your system. Shard View is made up of heat maps displaying the load of all the index shards on a specific node.

The heat maps can be sorted by different workloads, such as indexing load, search load and storage utilization, affecting the load on the nodes.

A few examples of what you can accomplish using Shard View:

  • Detect hotspots and resolve slow indexing
  • Improve search performance and balance search across nodes
  • Spread storage utilization across nodes and fix watermark issues


Run the Check-Up to get a customized report like this:

Analyze your cluster