How To Solve Issues Related to Log – Cluster health status changed from to reason: .

Get an Elasticsearch Check-Up


Check if your ES issues are caused from misconfigured settings
(Free 2 min process)

ES Check Up

Elasticsearch Error Guide In Page Navigation :

Troubleshooting Background – start here to get the full picture       
Related Issues – selected resources on related issues  
Log Context – usefull for experts
About Opster – offering a diffrent approach to troubleshoot Elasticsearch

Check My Elasticsearch 


Troubleshooting background

To troubleshoot Elasticsearch log “Cluster health status changed from to reason: .” it’s important to know common problems related to Elasticsearch concepts: allocation, cluster, routing. See below-detailed explanations complete with common problems, examples and useful tips.

Cluster in Elasticsearch

What is it

In Elasticsearch a cluster is a collection of one or more nodes (servers / VMs). A cluster can consist of an unlimited number of nodes. The cluster provides interface for indexing and storing data and search capability across all of the data which is stored in the data nodes

Each cluster has a single master node that is elected by the master eligible nodes. In cases where the master is not available the other connected master eligible nodes elect a new master. Clusters are identified by a unique name, which defaults to “Elasticsearch”.

Routing in Elasticsearch

What it is

In Elasticsearch, routing refers to document routing. When you index a document, Elasticsearch will determine which shard will be used to index the document to. 

The shard is selected based on the following formula:

shard = hash(_routing) % number_of_primary_shards

Where the default value of _routing is _id.
It is important to know which shard the document is routed to, because Elasticsearch will need to determine where to find that document later on for document retrieval requests. 

Examples

In twitter index with 2 primary shards, the document with _id equal to “440” gets routed to the shard number:   

shard = hash( 440 ) % 2
PUT twitter/_doc/440
{
...
}

Notes
  • In order to improve search performance speed you can create custom routing. For example, you can enable custom routing that will ensure only a single shard is queried (the shard that contains your data).
  • To create custom routing in Elasticsearch, you will need to configure and define  that not all routing will be completed by default settings. ( v <= 5.0)
PUT my_index/customer/_mapping
{
   "order":{
      "_routing":{
         "required":true
      }
   }
}
  •  This will ensure that every document in the “customer” type must specify a custom routing.  For elasticsearch 6 or above you will need to update the same mapping as:
PUT my_index/_mapping
{
   "order":{
      "_routing":{
         "required":true
      }
   }
}


To help troubleshoot related issues we have gathered selected Q&A from the community and issues from Github , please review the following for further information :

Elasticsearch Error – Cluster health status changed from [YELLOW] to [RED] (reason: [shards failed
stackoverflow.com/questions/52114340/elasticsearch-error-cluster-health-status-changed-from-yellow-to-red-reas

Number of Views : 0.45 K  Score on Stackoverflow : -1

Cluster Goes Yellow A Few Times Dur
discuss.elastic.co/t/cluster-goes-yellow-a-few-times-during-the-night/108972

 


Log Context

Log ”Cluster health status changed from to reason: .” classname is AllocationService.java
We have extracted the following from Elasticsearch source code to get an in-depth context :

 
    private void logClusterHealthStateChange(ClusterStateHealth previousStateHealth; ClusterStateHealth newStateHealth; String reason) {
        ClusterHealthStatus previousHealth = previousStateHealth.getStatus();
        ClusterHealthStatus currentHealth = newStateHealth.getStatus();
        if (!previousHealth.equals(currentHealth)) {
            logger.info("Cluster health status changed from [{}] to [{}] (reason: [{}])."; previousHealth; currentHealth; reason);
        }
    }

    private boolean hasDeadNodes(RoutingAllocation allocation) {
        for (RoutingNode routingNode : allocation.routingNodes()) {






About Opster

Incorporating deep knowledge and broad history of Elasticsearch issues. Opster’s solution identifies and predicts root causes of Elasticsearch problems, provides recommendations and can automatically perform various actions to manage, troubleshoot and prevent issues

We are constantly updating our analysis of Elasticsearch logs, errors, and exceptions. Sharing best practices and providing troubleshooting guides.

Learn more: Glossary | Blog| Troubleshooting guides | Error Repository

Need help with any Elasticsearch issue ? Contact Opster

Did this page help you?