Get an Elasticsearch Check-Up
Check if your ES issues are caused from misconfigured settings
(Free 2 min process)
Last update: Jan-20
Elasticsearch Error Guide In Page Navigation (click to jump) :
Troubleshooting Background – start here to get the full picture
Related Issues – selected resources on related issues
Log Context – usefull for experts
About Opster – offering a diffrent approach to troubleshoot Elasticsearch
Check Your Elasticsearch Settings for Painfull Mistakes
Troubleshooting background
To troubleshoot Elasticsearch log “Recovery failed” it’s important to know common problems related to Elasticsearch concepts: recovery. See below-detailed explanations complete with common problems, examples and useful tips.
Recovery in Elasticsearch
What it is
In Elasticsearch, recovery refers to the process of recovering an index/shard when something goes wrong. You can recover an index/shards in many ways such as by re-indexing the data from a backup/failover cluster to the current one or by restoring from an Elasticsearch snapshot. Alternatively, Elasticsearch may be performing recoveries automatically in some cases, such as when a node restarts or when a node disconnects and connects again. There is an API to check the updated status of index/shard recoveries.
GET /<index>/_recoveryGET /_recovery
In summary, recovery can happen in the following situations:
- Node startup or failure ( local store recovery )
- Replication of Primary shards to replica shards
- Relocation of a shard to a different node in the same cluster
- Restoring a Snapshot
Examples:
Getting recovery information about several indices:
GET my_index1,my_index2/_recovery
Common Problems Related to Recovery Settings
- When a node is disconnected from the cluster, all of its shards go to an unassigned state. After a certain time, the shards will be allocated somewhere else on other nodes. This setting determines the number of concurrent shards per node that will be recovered.
PUT _cluster/settings{ "transient" : { "cluster.routing.allocation.node_concurrent_recoveries" : 3 }}
- You can also control when to start recovery after a node disconnects. ( This is useful if the node just restarts, for example, because you may not want to initiate any recovery for such transient events )
PUT _all/_settings{ "settings": { "index.unassigned.node_left.delayed_timeout": "6m" }}
- Elasticsearch limits the speed that is allocated to recovery in order to avoid overloading the cluster. This setting can be updated to make the recovery faster or slower, depending on your requirements.
PUT _cluster/settings{ "transient" : { "indices.recovery.max_bytes_per_sec" : "100mb"}}
Related issues to this log
To help troubleshoot related issues we have gathered selected Q&A from the community and issues from Github , please review the following for further information :
1. Failing to start shard in ElasticSearch IndexShardGatewayRecoveryException “sending failed” – Stats : ♥ 13.09 K Ι √ 9
Log Context
Log ”Recovery failed” classname is GatewayService.java
We have extracted the following from Elasticsearch source code to get an in-depth context :
} else {
if (recovered.compareAndSet(false; true)) {
threadPool.generic().execute(new AbstractRunnable() {
Override
public void onFailure(Exception e) {
logger.warn("Recovery failed"; e);
// we reset `recovered` in the listener don't reset it here otherwise there might be a race
// that resets it to false while a new recover is already running?
recoveryListener.onFailure("state recovery failed: " + e.getMessage());
}
About Opster
Incorporating deep knowledge and broad history of Elasticsearch issues. Opster’s solution identifies and predicts root causes of Elasticsearch problems, provides recommendations and can automatically perform various actions to manage, troubleshoot and prevent issues
Learn more: Glossary | Blog| Troubleshooting guides | Error Repository
Need help with any Elasticsearch issue ? Contact Opster