High disk watermark exceeded on ; shards will be relocated away from this node – How to solve related issues

High disk watermark exceeded on ; shards will be relocated away from this node – How to solve related issues

Opster Team

Feb-21, Version: 1.7-8.0

Crossing high disk watermarks can be avoided if detected earlier. In addition to reading this guide, we strongly recommend you run the Elasticsearch Error Check-Up which detects issues in ES that cause ES errors and specifically problems that causes disk space to run out quickly and prevent high disk watermark from exceeding.

It’s a free tool that requires no installation and takes 2 minutes to complete. You can run the Check-Up here.

Quick Summary

The cause of this error is low disk space on a data node, and as a preventive measure, Elasticsearch throws this log message and takes some preventive measures explained further.

Explanation

Elasticsearch considers the available disk space before deciding whether to allocate new shards, relocate shards away or put all indices on read mode based on a different threshold of this error. The reason is Elasticsearch indices consists of different shards which are persisted on data nodes and low disk space can cause issues.
Relevant settings related to log:
cluster.routing.allocation.disk.watermark – have three thresholds of low, high, and flood_stage and can be changed dynamically, accepting absolute values as well as percentage values.

Permanent fixes

a). Delete unused indices.
b) Merge segments to reduce the size of the shard on the affected node, more info on opster’s Elasticsearch expert’s STOF answer
c) Attach external disk or increase the disk used by the data node.

Temp hack/fixes

a) Changed settings values to a higher threshold by dynamically update settings using update cluster API:

PUT _cluster/settings

{

""transient"": {

""cluster.routing.allocation.disk.watermark.low"": ""100gb"", -->adjust according to your situations

""cluster.routing.allocation.disk.watermark.high"": ""50gb"",

""cluster.routing.allocation.disk.watermark.flood_stage"": ""10gb"",

""cluster.info.update.interval"": ""1m""

}

}

b) Disable disk check by hitting below cluster update API

{

""transient"": {

""cluster.routing.allocation.disk.threshold_enabled"" : false

}

}

C) Even After all these fixes, Elasticsearch won’t bring indices in write mode for that this API needs to be hit.

PUT _all/_settings

{

""index.blocks.read_only_allow_delete"": null

}

Log Context

Log “High disk watermark [{}] exceeded on {}; shards will be relocated away from this node” classname is DiskThresholdMonitor.java.
We extracted the following from Elasticsearch source code for those seeking an in-depth context :

entry; DiskThresholdDecider.this.rerouteInterval);
                        }
                    }
                }
                if (reroute) {
                    logger.info("high disk watermark exceeded on one or more nodes; exceeded on one or more nodes. rerouting shards");
                    // Execute an empty reroute; but don't block on the response
                    client.admin().cluster().prepareReroute().execute();
                }
            }
        }




 

Run the Check-Up to get a customized report like this:

Analyze your cluster