Before you begin reading this guide, we recommend you try running the Elasticsearch Error Check-Up which can resolve issues causing many log errors (free and no installation required)
This guide will help you check for common problems that cause the log “Recovery failed” to appear. It’s important to understand the issues related to the log, so to get started, read the general overview on common issues and tips related to the Elasticsearch concepts: recovery.
Advanced users might want to skip right to the common problems section in each concept or try running the Check-Up which analyses ES to discover the cause of many errors and provides suitable actionable recommendations.
Recovery in Elasticsearch
What it is
In Elasticsearch, recovery refers to the process of recovering an index/shard when something goes wrong. You can recover an index/shards in many ways such as by re-indexing the data from a backup/failover cluster to the current one or by restoring from an Elasticsearch snapshot. Alternatively, Elasticsearch may be performing recoveries automatically in some cases, such as when a node restarts or when a node disconnects and connects again. There is an API to check the updated status of index/shard recoveries.
GET /<index>/_recoveryGET /_recovery
In summary, recovery can happen in the following situations:
- Node startup or failure ( local store recovery )
- Replication of Primary shards to replica shards
- Relocation of a shard to a different node in the same cluster
- Restoring a Snapshot
Examples:
Getting recovery information about several indices:
GET my_index1,my_index2/_recovery
Common Problems Related to Recovery Settings
- When a node is disconnected from the cluster, all of its shards go to an unassigned state. After a certain time, the shards will be allocated somewhere else on other nodes. This setting determines the number of concurrent shards per node that will be recovered.
PUT _cluster/settings{ "transient" : { "cluster.routing.allocation.node_concurrent_recoveries" : 3 }}
- You can also control when to start recovery after a node disconnects. ( This is useful if the node just restarts, for example, because you may not want to initiate any recovery for such transient events )
PUT _all/_settings{ "settings": { "index.unassigned.node_left.delayed_timeout": "6m" }}
- Elasticsearch limits the speed that is allocated to recovery in order to avoid overloading the cluster. This setting can be updated to make the recovery faster or slower, depending on your requirements.
PUT _cluster/settings{ "transient" : { "indices.recovery.max_bytes_per_sec" : "100mb"}}
Log Context
Log”Recovery failed” classname is GatewayService.java
We extracted the following from Elasticsearch source code for those seeking an in-depth context :
} else {
if (recovered.compareAndSet(false; true)) {
threadPool.generic().execute(new AbstractRunnable() {
Override
public void onFailure(Exception e) {
logger.warn("Recovery failed"; e);
// we reset `recovered` in the listener don't reset it here otherwise there might be a race
// that resets it to false while a new recover is already running?
recoveryListener.onFailure("state recovery failed: " + e.getMessage());
}
Related issues to this log
We have gathered selected Q&A from the community and issues from Github, that can help fix related issues please review the following for further information :
1 Failing to start shard in ElasticSearch IndexShardGatewayRecoveryException “sending failed” -views 13.09 K ,score 9
About Opster
Opster line of products and support services detects, prevents, optimizes and automates everything needed to manage mission-critical Elasticsearch.
