Elasticsearch Disk Watermark

Avoid the Next Incident

Check if your ES issues are caused from misconfigured settings

2-min process

Stop Elasticsearch settings related incidents :  Fix My Settings

Last Update: February 2020

Disk watermarks in Elasticsearch

Elasticsearch considers the available disk space before deciding whether to allocate new shards, relocate shards away or put all indices on read mode based on a different threshold of this error, Reason being Elasticsearch indices consists of different shards which are persisted on data nodes and low disk space can cause issues above.

Relevant settings related 

cluster.routing.allocation.disk.watermark and have three thresholds of low, high, and flood_stage and can be changed dynamically as well. It Accepts absolute as well as percentage values. More information available on official ES doc.

How to fix logs messages related to Disk watermarks

Permanent fixes:

a) Delete unused indices. 

b) Merge segments to reduce the size of the shard on the affected node, more info on opster’s Elasticsearch expert’s STOF answer 

c) Attach external disk or increase the disk used by the data node

Temp hack/fixes:

a) Changed these settings values to a higher threshold by dynamically update settings using below update cluster API.

PUT _cluster/settings :

{

  “transient”: {

    “cluster.routing.allocation.disk.watermark.low”: “100gb”, –>adjust according to your situations

    “cluster.routing.allocation.disk.watermark.high”: “50gb”,

    “cluster.routing.allocation.disk.watermark.flood_stage”: “10gb”,

    “cluster.info.update.interval”: “1m”

  }

}

b) Disable disk check by hitting below cluster update API

{

    “transient”: {

       “cluster.routing.allocation.disk.threshold_enabled” : false

    }

}

Even After all these fixes, elasticsearch won’t bring indices in write mode, and for that below, API needs to be hit.

PUT _all/_settings

{

     “index.blocks.read_only_allow_delete”: null

}

About Opster

Incorporating deep knowledge and broad history of Elasticsearch issues. Opster solution identifies and predicts root causes of Elasticsearch problems, provides recommendations and can automatically perform various actions to manage, troubleshoot and prevent issues

Learn more: Glossary | Blog| Troubleshooting guides

Need help with any Elasticsearch issue ? Contact Opster

Avoid the next incident use our settings check-up : Prevent Issues


Click below to learn how to fix common problems related to these concepts
« Back to Index