How To Solve Issues Related to Log – Cant read metadata from store; will not reuse local files during restore

Improve Your Elasticsearch Configuration

No Installation Required
No Security Risk
3 Free Minute Process !

No Installation

Updated: Jan-20

In-Page Navigation (click to jump) :

Opster Offer’s World-Class Elasticsearch Expertise In One Powerful Product
Try Our Free Elasticsearch Check-Up   Fix Painful Mistakes !

Troubleshooting background

To troubleshoot Elasticsearch log “Cant read metadata from store; will not reuse local files during restore” it’s important to understand common problems related to Elasticsearch concepts: blobstore, discovery-file, repositories, restore. See detailed explanations below complete with common problems, examples and useful tips.

Repository in Elasticsearch

What it is

An Elasticsearch snapshot provides a backup mechanism that takes the current state and data in the cluster and saves it to a repository (read the Glossary term Snapshot for more information). The backup process requires a repository to be created first. The repository needs to be registered using the _snapshot endpoint, and multiple repositories per cluster can be created. The following repository types are supported. 

Repository Types:
Repository TypeConfiguration Type
Shared file systemType: “fs”
S3Type : “s3”
HDFSType :“hdfs”
AzureType: “azure”
Google Cloud StorageType : “gcs”
Examples

To register a repository of type fs:

PUT _snapshot/my_repo_01
{
"type": "fs",
"settings": {
"location": "/mnt/my_repo_dir"
  }
}
Notes and common problems
  • S3, HDFS , Azure and Google Cloud requires a relevant plugin to be installed before it can be used for a snapshot.
  • The setting, path.repo: /mnt/my_repo_dir needs to be added to elasticsearch.yml on all the nodes in case you are planning to use repo type of file system otherwise it will fail
  • In case of using remote repositories , the network bandwidth and repository storage throughput should be high enough to complete the snapshot operations normally , otherwise you will end up in partial snapshots.


To help troubleshoot related issues we have gathered selected Q&A from the community and issues from Github , please review the following for further information :

1 At Does Stale File Handle In Linux  

2Github Issue Number 37960  

Github Issue Number 36816


Log Context

Log ”[{}] [{}] Can’t read metadata from store; will not reuse local files during restore” classname is FileRestoreContext.java
We have extracted the following from Elasticsearch source code to get an in-depth context :

<pre class="wp-block-syntaxhighlighter-code">             } catch (org.apache.lucene.index.IndexNotFoundException e) {
                // happens when restore to an empty shard; not a big deal
                logger.trace("[{}] [{}] restoring from to an empty shard"; shardId; snapshotId);
                recoveryTargetMetadata = Store.MetadataSnapshot.EMPTY;
            } catch (IOException e) {
                logger.warn(new ParameterizedMessage("[{}] [{}] Can't read metadata from store; will not reuse local files during restore";
                    shardId; snapshotId); e);
                recoveryTargetMetadata = Store.MetadataSnapshot.EMPTY;
            }

            final List<BlobStoreIndexShardSnapshot.FileInfo> filesToRecover = new ArrayList<>();




</pre>


About Opster

Opster identifies and predicts root causes of Elasticsearch problems, provides recommendations and can automatically perform various actions to prevent issues, optimize performance and save resources.

Learn more: Glossary | Blog| Troubleshooting guides | Error Repository

Need help with any Elasticsearch issue ? Contact Opster