Elasticsearch Elasticsearch Memory and Disk Usage Management

Average Read Time

2 Mins

Elasticsearch Elasticsearch Memory and Disk Usage Management

Opster Team

May-2022

Average Read Time

2 Mins

Opster Team

May 24, 2022

Average Read Time

2 Mins


In addition to reading this guide, we recommend you run the Elasticsearch Health Check-Up. It will detect issues and improve your Elasticsearch performance by analyzing your shard sizes, threadpools, memory, snapshots, disk watermarks and more.

The Elasticsearch Check-Up is free and requires no installation.

To avoid hotspots in Elasticsearch and OpenSearch, we recommend you run the free Check-Up. The Check-Up will also help you optimize other important settings to improve performance.

Run the Elasticsearch check-up to receive recommendations like this:

checklist Run Check-Up
error

An indexing burst is affecting the performance of the following nodes

error-img

Description

The node is unable to keep up with indexing requests, and as a result indexing requests are being queued. If the write queue reaches full capacity, index requests will be rejected, which may cause data loss if...

error-img

Recommendation

In order to resolve your indexing bursts, based on your specific ES deployment, we recommend that you...

1

X-PUT curl -H "Content-Type: application/json" [customized recommendation]

How to fine-tune how much disk and memory resources are needed in Elasticsearch

This article is related to Opster’s Cost Insight tool. Cost Insight is free, does not require any installation and helps users reduce Elasticsearch and OpenSearch hardware costs. Read more about it here, and run the tool here.

Overview

When you’d like to check if your resources are both efficient and cost-efficient, one way to do so is to evaluate the ratio of disk usage to the memory allocated.

Elasticsearch nodes require a lot of RAM memory, for both indexing and search operations. The RAM memory required to run an Elasticsearch cluster is generally proportional to the volume of data on the cluster.

Memory to disk ratio is high 

According to the best practice for ratio between memory and disk, if you have more than 1GB of memory to 20GB of disk space, this would be considered high memory to disk ratio, meaning the cluster has a lot of memory.

If the cluster’s performance is good and you’re looking to reduce costs, reducing the memory might be an opportunity to cut expenses because the ratio here is high. In this case it is improbable that you will be able to take advantage of all of the RAM resources on your cluster.

You may have high memory to disk ratios in situations such as:

  • Very low data retentions (eg. 1 week)
  • High volume of updates rather than new data indexing
  • Search intensive applications (large number of queries or heavy aggregations against a relatively low volume of data)

If you’re interested in reducing costs, then you should consider reducing the RAM memory on the existing nodes to cut down your expenses.

Memory to disk ratio is low

According to the best practice for ratio between memory and disk, if you have less than 1 GB of memory to 80 GB of disk space, the cluster does not have enough memory resources.

In this case you will likely be unable to take advantage of all of the available disk space, or if you do, you are likely to have performance issues. You may have low memory to disk ratios in situations such as:

  • Very long data retentions 
  • Non-search intensive applications (low client query rates, minimal aggregations)
  • Warm nodes data tier

If your cluster performance is poorer than you’d like, then you may want to consider one or more of the following options:

  • Increase the RAM memory of your nodes up to a heap size of 32GB
  • Reduce the disk size on your nodes or add additional data nodes


Run the Check-Up to get a customized report like this:

Analyze your cluster
Skip to content