Elasticsearch Setting up Hot-Warm architecture for ISM

Opster Expert Team - Gustavo

Oct 30, 2022 | 4 min read

Opster Team

Oct 30, 2022 | 4 min read


In addition to reading this guide, we recommend you run the Elasticsearch Health Check-Up. It will detect issues and improve your Elasticsearch performance by analyzing your shard sizes, threadpools, memory, snapshots, disk watermarks and more.

The Elasticsearch Check-Up is free and requires no installation.

To manage all aspects of your OpenSearch operation, you can use Opster’s Management Console (OMC). The OMC makes it easy to orchestrate and manage OpenSearch in any environment. Using the OMC you can deploy multiple clusters, configure node roles, scale cluster resources, manage certificates and more – all from a single interface, for free. Check it out here.

Quick links:

Introduction 

In the ISM review we learned how to create a hot-warm-delete policy. For the purposes of that article, we assumed that all the nodes were equal in terms of hardware, so OpenSearch would move the shards across nodes without being aware of the specs of each of them.

What if we want to optimize the resources by assigning different hardware polifices to different nodes?

Hot-Wam Architecture

The hot-warm-cold architecture aims for resource optimization using different hardware profiles for different phases of our data life. 

  • Hot: Used for high read/write operations. For this we need our best CPU/RAM nodes here, and we use SSD storage.
  • Warm: Lighter search, read only. We can have less powerful nodes here. Warm nodes can use very large spindle drives instead of SSD storage.

Once we have our retention policies defined, we can start thinking where to store our data on each phase, and configure the ILM to put the data in that node.

IMPORTANT: We can store as many shards as we want in a node (the recommendation is no more than 25 shards per GB of heap). However, we cannot store the primary and the replica of the same shard, in the same node.

This means we cannot have 2 nodes and have an index with 1 primary shard and 2 replicas, because  two replicas of the same shard, or the primary + 1 replica of the same shard would live in the same node.

For this example we will work with 2 nodes, and configure one to be “hot”, and the other to be “warm”. To see your current nodes you can run this command in Dev Tools:

GET _cat/nodes

Shard allocation

We will configure one node to be “hot”, and the second one to be “warm”. Initially all the index  shards will go to the hot node (“node1”) and when ISM changes to the warm phase, the data will be moved to the warm node (“node2”). 
First we need to “tag” our nodes to the “temperature” we want them to be. This is achieved by adding the following configuration to each node yml file: 

node.attr.temp: hot #node1
node.attr.temp: warm #node2

Note: attribute name “temp” and values “hot, warm” are arbitrary, we can use any names we want as long as they are consistent across configurations.

To confirm the settings were applied, you can run the following command in Dev Tools. Remember to restart your nodes after applying the configuration.

GET _cat/nodeattrs?v&h=node,attr,value

You will see:

node           attr    value
opensearch-node1 temp  hot
opensearch-node2 temp  warm

Index pattern

Now we have to tell our index pattern to allocate the shards to the hot zone on creation, and to then move them through ISM.

Using our data streams example, we’ll run the following:

PUT _index_template/logs-template
{
  "index_patterns": [
    "datalogs-*"
  ],
  "data_stream": {
    "timestamp_field": {
      "name": "@timestamp"
    }
  },
  "template": {
    "settings": {
      "index": {
        "routing": {
          "allocation": {
            "require": {
              "temp": "hot"
            }
          }
        }
      }
    }
  }
}

Now all the indices created under the datalogs-* template will be allocated to the hot zone. Let’s try:

POST datalogs-hot/_doc
{
  "@timestamp": "2022-07-07",
  "name": "1"
}

Let’s see if it picked our ISM policy created in the previous article:

We can see it picked the policy and it’s enabled.

Let’s see now if it allocated correctly:

GET _cat/shards/datalogs-hot?v

As you can see, it only assigned the primary shard (p on prirep) to the node1, which is perfect. It didn’t assign the replica shard (r on prirep) because as we said, you cannot assign the primary and the replica to the same node, and we only have 2 nodes. Node2 is a warm node, so it will not be allocated there because of the limitation we set in the index pattern. 

To test our configuration, let’s create the following ISM policy: 

  • Starting phase will be hot and shards will live in hot node
  • After 1 minute do rollover (create a new underlying index)
  • After 5 minutes from rollover move to warm
  • Move shards to warm node
  • Remove replicas

*Times will be longer because ISM takes some minutes between checking status and applying settings. It may take about 10 minutes to start.

Let’s create an ISM policy for this purpose. We will link it to the datalogs-hotwarm pattern. Note there is no wildcard so the name of the index must be exactly datalogs-hotwarm.

PUT _plugins/_ism/policies/hot_warm_test
{
  "policy": {
    "description": "Shards moving across nodes test",
    "schema_version": 1,
    "error_notification": null,
    "default_state": "hot",
    "states": [
      {
        "name": "hot",
        "actions": [
          {
            "retry": {
              "count": 3,
              "backoff": "exponential",
              "delay": "1m"
            },
            "rollover": {
              "min_index_age": "1m"
            }
          }
        ],
        "transitions": [
          {
            "state_name": "warm",
            "conditions": {
              "min_rollover_age": "5m"
            }
          }
        ]
      },
      {
        "name": "warm",
        "actions": [
          {
            "retry": {
              "count": 3,
              "backoff": "exponential",
              "delay": "1m"
            },
            "replica_count": {
              "number_of_replicas": 0
            }
          },
          {
            "retry": {
              "count": 3,
              "backoff": "exponential",
              "delay": "1m"
            },
            "allocation": {
              "require": {
                "temp": "warm"
              },
              "include": {},
              "exclude": {},
              "wait_for": false
            }
          }
        ],
        "transitions": []
      }
    ],
    "ism_template": [
      {
        "index_patterns": [
          "datalogs-hotwarm"
        ],
        "priority": 999
      }
    ]
  }
}

Now let’s create one document and check:

POST datalogs-hotwarm/_doc
{
  "@timestamp": "2022-07-07",
  "name": "1"
}

The output of:

GET _opendistro/_ism/explain/datalogs-hotwarm
GET _cat/shards/datalogs-hotwarm?

Looks just like the beginning.

We can monitor the current ILM state through Index management -> Managed Indices.

The ISM policy takes some time to activate. If you click the policy name you will see the following message: 

If we wait a couple of minutes we will see the ISM policy in action: 


The rollover is finished, so now we have two indices:

Now after 5 minutes the rolled over index will move to warm phase, allocate shards to warm node, and then remove replicas:

Let’s confirm there is only 1 shard allocated to node2, and no UNASSIGNED ones:

GET _cat/shards/.ds-datalogs-hotwarm-000001?v

Note that we are now querying the index directly, and not the data stream, because we only want to see where the shards belonging to the rollovered index are being allocated: 

Perfect! Only one shard, and allocated to our warm node (node2). From now we can go one step further and create a deletion phase if we want to remove our indices after a period of time.

Conclusion

OpenSearch offers an easy way to configure hot/warm architectures under very specific conditions, visually or using JSON files. By tagging your nodes, you can decide where your data should live depending on hardware specifications and data freshnes.



Find and fix issues Elasticsearch issues, try AutoOps

Watch product tour

Try For Free
Skip to content