Elasticsearch Index Lifecycle Management

Elasticsearch Index Lifecycle Management

Opster Team

March 2021


In addition to reading this guide, we recommend you run the Elasticsearch Health Check-Up. It will detect issues and improve your Elasticsearch performance by analyzing your shard sizes, threadpools, memory, snapshots, disk watermarks and more.

The Elasticsearch Check-Up is free and requires no installation.

To improve your index lifecycle management in Elasticsearch, we recommend you run the Elasticsearch Configuration Check-Up. The Check-Up will also help you optimize other important settings in Elasticsearch to improve performance.

Run the Elasticsearch check-up to receive recommendations like this:

checklist Run Check-Up
error

An indexing burst is affecting the performance of the following node

error-img

Description

The node is unable to keep up with indexing requests, and as a result indexing requests are being queued. If the write queue reaches full capacity, index requests will be rejected...

error-img

Recommendation

Based on your specific Elasticsearch deployment, we recommend that you...

1

X-PUT curl -H "Content-Type: application/json" [customized recommendation]

Why index lifecycle management is necessary

Index lifecycle management is a feature that helps automate the creation, management and deletion of an Elasticsearch index.

Being able to automate the creation of a new index when the index reaches the optimum size of 50GB per shard is very useful. Setting up a time-based index with one index per day, or one index per month, is likely to create index shards that have an optimal size. Shards that are either too small or too large can cause various issues- see this guide for more info. 

Once an index is no longer being written to, it is useful to optimize the index to save cluster resources. This can involve:

  • Force merging the index to optimize the space used by the index on disk.
  • Shrinking the index to reduce the number of shards.
  • Allocating the shard to less performant / storage optimized hardware.

As an index nears the end of its life we may want to: 

  • Ensure that a snapshot of the index has been successfully taken
  • Delete the index

How to set up index lifecycle management

Define lifecycle management policy

An index lifecycle management policy can be applied to any number of indices. The policy consists of a number of defined phases and indicates the actions that should be taken when transitioning from one phase to the next.  

All policies must start with a HOT phase. The rest of the phases are optional. The transition from one phase to the next is defined by the time passed since the creation of the index.

PUT _ilm/policy/hot_warm_delete
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "50gb",
            "max_age": "3d"
          },
          "set_priority": {
            "priority": 50
          }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "forcemerge": {
            "max_num_segments": 1
          },
          "set_priority": {
            "priority": 25
          },
          "shrink": {
            "number_of_shards": 1
          }
        }
      },
      "delete": {
        "min_age": "15d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}

The above policy defines the following:

Hot phase 

During the hot phase, a new index will be created when the size of the index reaches 50GB or when the index is 3 days old. 

Warm phase

After 7 days, the hot index will pass to the warm phase. The index will be force merged and a shrink operation will be carried out to reduce the index to a single shard.

Delete phase

After 15 days, the index will be deleted.

Note: In this case no “cold” or “frozen” phases have been configured.

Setting up a template to add the ILM Policy to the index settings

PUT _index_template/my_apache_log
{
  "index_patterns": ["my_apache_log*"],                 
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "index.lifecycle.name": "hot_warm_delete",      
      "index.lifecycle.rollover_alias": "my_apache_log"    
    }
  }
}

The above command creates an index template to apply the hot_warm_delete policy we created above to any indices with the prefix “my_apache_log”.

It also defines the alias that will be applied to the new index when the original index rolls over.

Manually create the first index

We need to manually create the first index for the my_apache_log series.

PUT my_apache_log-000001
{
  "aliases": {
    "my_apache_log": {
      "is_write_index": true
    }
  }
}

The above index creates the first index with the my_apache_log alias.

We then need to make sure the client application sends all writes to the alias “my_apache_log”  (not the full index name). Elasticsearch will write these logs to index my_apache_log-000001 until that index reaches a size of 50GB (as defined in our ILM policy) and then roll over the index, automatically creating a new index my_apache_log-000002 and so on.   

Modifying the ILM policy

If you modify the ILM policy, the modifications will be applied to all of the indices that use that policy. However the changes will only be applied for the new changes of phase. Any changes of phase that have already been applied with previous versions of the policy cannot be rolled back.

ILM actions

The most common actions are rollover, force merge, wait for snapshot and delete. However all of the following actions are possible:

  • Allocate settings to the index to move the shards to different nodes, or change the number of replicas.
  • Delete the index.
  • Force merge the index to optimize structure on disk.
  • Freeze the index to minimize its memory footprint. This is only appropriate for old indices which are very rarely searched. Note that a frozen index will take seconds or even minutes to search since certain data structures will need to be rebuilt on the fly.
  • Migrate index shards to the data tier being used by the ILM phase. This is only appropriate if you are using data tiers in your architecture. 
  • Read only to prevent writing to the index.
  • Rollover to create a new write index.
  • Searchable snapshot the index to enable the use of this mechanism to reduce memory and disk requirements for infrequently searched indices.
  • Set priority of an index so that Elasticsearch knows which indices to recover first.
  • Shrink the number of primary shards in an index (this creates a new index).
  • Unfollow a cross cluster replication index. This is only appropriate for clusters using Cross Cluster Replication.
  • Wait for snapshot before deleting index.

ILM errors

If you see that ILM is not behaving as expected, you can check the status of ILM on an index using the following command:

GET /my-index-000001/_ilm/explain

Read the error message carefully and try to fix the error. Typically the error may be corrected, either by modifying the ILM policy, or by correcting the index settings. Once you have done this, you can run this command:

POST /my-index-000001/_ilm/retry

Run the Elasticsearch check-up to receive recommendations like this:

checklist Run Check-Up
error

An indexing burst is affecting the performance of the following node

error-img

Description

The node is unable to keep up with indexing requests, and as a result indexing requests are being queued. If the write queue reaches full capacity, index requests will be rejected...

error-img

Recommendation

Based on your specific Elasticsearch deployment, we recommend that you...

1

X-PUT curl -H "Content-Type: application/json" [customized recommendation]


Related log errors to this ES concept


Top warming has been interrupted
No version match ; default to
Version and NOT supported for
Version of mappings for not found; recreating
Starting segment upgrade upgradeOnlyAncientSegments=
Adding index
Adding index ; preferred over
Dropping index due to conflict with
Failed to close node
Failed to close node on failed start
Removing index
Removing node

< Page: 1 of 147 >


Improve Elasticsearch Performance

Run The Analysis