How to fine-tune how much disk and memory resources are needed in Elasticsearch
Overview
When you’d like to check if your resources are both efficient and cost-efficient, one way to do so is to evaluate the ratio of disk usage to the memory allocated.
Elasticsearch nodes require a lot of RAM memory, for both indexing and search operations. The RAM memory required to run an Elasticsearch cluster is generally proportional to the volume of data on the cluster.
Recommended disk/RAM ratio
The recommended disk/RAM ratio differs according to tier. The recommendations are as follows:
- Data/Hot/Content – 30x
- Warm – 160x
- Cold – 300x
- Frozen – 1000x
Memory to disk ratio is high
According to the best practice for ratio between memory and disk, if you have more than 1GB of memory to 20GB of disk space, this would be considered high memory to disk ratio, meaning the cluster has a lot of memory.
If the cluster’s performance is good and you’re looking to reduce costs, reducing the memory might be an opportunity to cut expenses because the ratio here is high. In this case it is improbable that you will be able to take advantage of all of the RAM resources on your cluster.
You may have high memory to disk ratios in situations such as:
- Very low data retentions (eg. 1 week)
- High volume of updates rather than new data indexing
- Search intensive applications (large number of queries or heavy aggregations against a relatively low volume of data)
If you’re interested in reducing costs, then you should consider reducing the RAM memory on the existing nodes to cut down your expenses.
Memory to disk ratio is low
According to the best practice for ratio between memory and disk, if you have less than 1 GB of memory to 80 GB of disk space, the cluster does not have enough memory resources.
In this case you will likely be unable to take advantage of all of the available disk space, or if you do, you are likely to have performance issues. You may have low memory to disk ratios in situations such as:
- Very long data retentions
- Non-search intensive applications (low client query rates, minimal aggregations)
- Warm nodes data tier
If your cluster performance is poorer than you’d like, then you may want to consider one or more of the following options:
- Increase the RAM memory of your nodes up to a heap size of 32GB
- Reduce the disk size on your nodes or add additional data nodes