Elasticsearch OpenSearch Data Streams

Average Read Time

3 Mins

Elasticsearch OpenSearch Data Streams

Opster Expert Team - Gustavo

Sep-2022

Average Read Time

3 Mins

Opster Team

Sep 11, 2022

Average Read Time

3 Mins


In addition to reading this guide, we recommend you run the Elasticsearch Health Check-Up. It will detect issues and improve your Elasticsearch performance by analyzing your shard sizes, threadpools, memory, snapshots, disk watermarks and more.

The Elasticsearch Check-Up is free and requires no installation.

To manage all aspects of your OpenSearch operation, you can use Opster’s Management Console (OMC). The OMC makes it easy to orchestrate and manage OpenSearch in any environment. Using the OMC you can deploy multiple clusters, configure node roles, scale cluster resources, manage certificates and more – all from a single interface, for free. Check it out here.

Quick Links

Introduction

There are two main types of data sources when we’re talking about search operations: 

  1. Time-based data (for example, access logs)
  2. Non-time-based data (for example, product inventory) 

Time-based logs have a set of common features: 

  • @timestamp field is mandatory and most important.
  • Documents are not updated.
  • Documents are split into indices based on time.
  • Old documents are less likely to be searched, or may not searched at all, but still need to be stored.

To move data around efficiently we can use ISM (Index State Management) policies. Index State Management allows you to define custom policies to automate routine tasks like moving data across nodes, or delete under certain conditions. In this way, we can split our data based on age and define its move from the fastest hardware, to slower hardware, and eventually to the trash can. 

This is when data streams come into play. We need a way to tell the ISM policy how to split the data into indices. In the past we would use rollover aliases, but now data streams implement this behavior out-of-the-box. Data streams also leverage index templates, to make the generation even simpler.

How to create data streams in OpenSearch

First, we need to create the index template that will configure all the indices under the name we define as Data Streams:

PUT _index_template/logs-template
{
  "index_patterns": [
    "datalogs-*"
  ],
  "data_stream": {}
}

As we know, ISM allows you to define custom policies to automate routine tasks like moving data across nodes, or delete under certain conditions. Without ISM, it does not make sense to use a data stream index. 

From here, all the indices we create starting with “datalogs-” will be data streams. By default, you need to include a @timestamp field in your documents (this field can be changed).

Now we have 2 options: 

1. We can call the data stream API directly:

PUT _data_stream/datalogs-example

2. We can just ingest a document to that index name:

POST datalogs-test/_doc
{
  "@timestamp": "2022-06-20T16:24:03.417+00:00",
  "message": "this is an example log"
}

Finally, just run this last command and pay attention to the response:

{
  "_index" : ".ds-datalogs-test-000001",
  "_id" : "trjdqIEB7jRCWfUHatx-",
  "_version" : 1,
  "result" : "created",
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "failed" : 0
  },
  "_seq_no" : 0,
  "_primary_term" : 1
}

What happened here? We pushed to the data stream, but Opensearch created an underlying index that will hold the data. The data stream name is just an alias. The good thing is we don’t have to worry about which index we are pushing. We just have to push to the data stream, configure the ISM policy, and everything will just work.

If we set a policy to perform a rollover after one day, in one day we will see a .ds-datalogs-test-000002 index, and so on. Notice the index starts with a dot, this means the index is hidden. We are not supposed to interact with the index directly.

We can get the details about our data stream with this command:

GET _data_stream/datalogs-test

And even more insights:

GET _data_stream/datalogs-test/_stats

How to search against a data stream

Searching against a data stream is like searching against a regular index. All the underlying indices will be queried.

GET datalogs-test/_search

How to delete a data stream

Deleting a data stream will remove all the underlying indices:

DELETE _data_stream/datalogs-test/

Data stream limitations

Data streams are optimized for time-based series.

  • They are designed primarily for append-only data.
  • @timestamp field is required.
  • Search requests are mapped to all indices, write requests to the latest write index.

Data streams can be leveraged using ISM policies, making the process of moving our data across nodes and managing the retention policies even simpler.

Conclusion

Data streams enforce a setup that works perfectly with time-based data, making the ISM policies much easier to configure.

If you want to ingest logs or any document that is timestamp centered, you can use data streams and forget about configuring rolling aliases. Just go ahead and start creating ISM policies to handle your data retention.



Run the Check-Up to get a customized report like this:

Analyze your cluster
Skip to content