Before you dig into reading this guide, have you tried asking OpsGPT what this log means? You’ll receive a customized analysis of your log.
Try OpsGPT now for step-by-step guidance and tailored insights into your OpenSearch operation.
Briefly, this error occurs when OpenSearch modifies a repository that contains snapshots older than a certain version, and it needs to pause (sleep) to ensure consistency. This is a built-in mechanism to prevent potential data loss or corruption. To resolve this, you can either wait for the sleep period to end, or if the older snapshots are not needed, you can delete them. However, be cautious with deletion as it might lead to data loss. If the issue persists, consider upgrading your snapshots to the current version.
For a complete solution to your to your search operation, try for free AutoOps for Elasticsearch & OpenSearch . With AutoOps and Opster’s proactive support, you don’t have to worry about your search operation – we take charge of it. Get improved performance & stability with less hardware.
This guide will help you check for common problems that cause the log ” Sleeping for [{}] after modifying repository [{}] because it contains snapshots older than version [{}] ” to appear. To understand the issues related to this log, read the explanation below about the following OpenSearch concepts: repository-s3, repository, version, repositories, plugins.
Quick links
Overview
An OpenSearch snapshot provides a backup mechanism that takes the current state and data in the cluster and saves it to a repository (read snapshot for more information). The backup process requires a repository to be created first. The repository needs to be registered using the _snapshot endpoint, and multiple repositories can be created per cluster. The following repository types are supported:
Repository types
| Repository type | Configuration type |
|---|---|
| Shared file system | Type: “fs” |
| S3 | Type : “s3” |
| HDFS | Type :“hdfs” |
| Azure | Type: “azure” |
| Google Cloud Storage | Type : “gcs” |
Examples
To register an “fs” repository:
PUT _snapshot/my_repo_01
{
"type": "fs",
"settings": {
"location": "/mnt/my_repo_dir"
}
}Notes and good things to know
- S3, HDFS, Azure and Google Cloud require a relevant plugin to be installed before it can be used for a snapshot.
- The setting, path.repo: /mnt/my_repo_dir needs to be added to opensearch.yml on all the nodes if you are planning to use the repo type of file system. Otherwise, it will fail.
- When using remote repositories, the network bandwidth and repository storage throughput should be high enough to complete the snapshot operations normally, otherwise you will end up with partial snapshots.
Overview
A version corresponds to the OpenSearch built-in tracking system that tracks the changes in each document’s update. When a document is indexed for the first time, it is assigned a version 1 using _version key. When the same document gets a subsequent update, the _version is incremented by 1 with every index, update or delete API call.
What it is used for
A version is used to handle the concurrency issues in OpenSearch which come into play during simultaneous accessing of an index by multiple users. OpenSearch handles this issue with an optimistic locking concept using the _version parameter to avoid letting multiple users edit the same document at the same time and protects users from generating incorrect data.
Notes
You cannot see the history of the document using _version. That means OpenSearch does not use _version to keep track of original changes that had been performed on the document. For example, if a document has been updated 10 times, it’s _version would be marked by OpenSearch as 11, but you cannot go back and see what version 5 of the document looked like. This has to be implemented independently.
Common problems
If optimistic locking is not implemented while making updates to a document, OpenSearch may return a conflict error with the 409 status code, which means that multiple users are trying to update the same version of the document at the same time.
POST /ratings/123?version=50
{
"name": "Joker",
"rating": 50
}Log Context
Log “Sleeping for [{}] after modifying repository [{}] because it contains snapshots older than version [{}]” classname is S3Repository.java.
We extracted the following from OpenSearch source code for those seeking an in-depth context :
}
};
}
private void logCooldownInfo() {
logger.info("Sleeping for [{}] after modifying repository [{}] because it contains snapshots older than version [{}]" +
" and therefore is using a backwards compatible metadata format that requires this cooldown period to avoid " +
"repository corruption. To get rid of this message and move to the new repository metadata format; either remove " +
"all snapshots older than version [{}] from the repository or create a new repository at an empty location.";
coolDown; metadata.name(); SnapshotsService.SHARD_GEN_IN_REPO_DATA_VERSION;
SnapshotsService.SHARD_GEN_IN_REPO_DATA_VERSION);
