1 Mins
In addition to reading this guide, we recommend you run the Elasticsearch Health Check-Up. It will detect issues and improve your Elasticsearch performance by analyzing your shard sizes, threadpools, memory, snapshots, disk watermarks and more.The Elasticsearch Check-Up is free and requires no installation.
Aside from learning about Elasticsearch low disk watermarks, we recommend you run the Elasticsearch Health Check-Up. It will detect issues and improve your Elasticsearch performance by analyzing your shard sizes, threadpools, memory, snapshots, disk watermarks and more.
The Elasticsearch Check-Up is free and requires no installation.
If you’d like to learn how you can reduce the cost of your cluster by adjusting your disk watermarks, run Opster’s Cost Insight tool.
Run the Elasticsearch check-up to receive recommendations like this:
Run Check-Up
The following configuration error was detected on node 123...
Description
This error can have a severe impact on your system. It's important to understand that it was caused by...
Recommendation
In order to resolve this issue and prevent it from occurring again, we recommend that you begin by changing the configuration to...
X-PUT curl -H "Content-Type: application/json" [customized recommendation]
Overview
There are various “watermark” thresholds on your Elasticsearch cluster. As the disk fills up on a node, the first threshold to be crossed will be the “low disk watermark”. Once this threshold is crossed, the Elasticsearch cluster will stop allocating shards to that node. This means that your cluster may become yellow.
How to resolve it
Passing this threshold is a warning and you should not delay in taking action before the higher thresholds are reached. Here are possible actions you can take to resolve the issue:
- Delete old indices
- Remove documents from existing indices
- Increase disk space on the node
- Add new nodes to the cluster
You can see the settings you have applied with this command:
GET _cluster/settings
If they are not appropriate, you can modify them using a command such as below:
PUT _cluster/settings
{
"transient": {
"cluster.routing.allocation.disk.watermark.low": "85%",
"cluster.routing.allocation.disk.watermark.high": "90%",
"cluster.routing.allocation.disk.watermark.flood_stage": "95%",
"cluster.info.update.interval": "1m"
}
}How to avoid it
There are various mechanisms to automatically delete stale data.
How to automatically delete stale data:
- Apply ILM (Index Lifecycle Management)
Using ILM you can get Elasticsearch to automatically delete an index when your current index size reaches a given age.
- Use date based indices
If your application uses date based indices, then it is easy to delete old indices using a script or a tool such as Elasticsearch curator.
- Use snapshots to store data offline
It may be appropriate to store snapshotted data offline and restore it in the event that the archived data needs to be reviewed or studied.
- Automate / simplify process to add new data nodes
Use automation tools such as terraform to automate the addition of new nodes to the cluster. If this is not possible, at the very least ensure you have a clearly documented process to create new nodes, add TLS certificates and configuration and bring them into the Elasticsearch cluster in a short and predictable time frame.
