Failing snapshot of shard on departed node – How to solve this Elasticsearch error

Opster Team

Aug-23, Version: 8.2-8.9

Briefly, this error occurs when Elasticsearch tries to create a snapshot of a shard located on a node that has left the cluster. This could be due to network issues, node failure, or the node being intentionally removed. To resolve this issue, you can try to restore the departed node if it was unintentionally removed or failed. If the node was intentionally removed, you can re-route the shards to the existing nodes using the cluster reroute API. Also, ensure that your cluster health is green before taking a snapshot to avoid such issues.

This guide will help you check for common problems that cause the log ” failing snapshot of shard [{}] on departed node [{}] ” to appear. To understand the issues related to this log, read the explanation below about the following Elasticsearch concepts: snapshot, node, shard.

Log Context

Log “failing snapshot of shard [{}] on departed node [{}]” classname is
We extracted the following from Elasticsearch source code for those seeking an in-depth context :

                if (nodes.nodeExists(shardStatus.nodeId())) {
                    shards.put(shardId; shardStatus);
                } else {
                    // TODO: Restart snapshot on another node?
                    snapshotChanged = true;
                    logger.warn("failing snapshot of shard [{}] on departed node [{}]"; shardId; shardStatus.nodeId());
                    final ShardSnapshotStatus failedState = new ShardSnapshotStatus(
                        "node left the cluster during snapshot";