How To Solve Issues Related to Log – Failed to list directory some of files might not be deleted

Prevent Your Next ELK Incident

Try our free Check Up to test if your ES issues are caused from misconfigured settings

Prevent Issue

Updated: Jan-20

In-Page Navigation (click to jump) :

Opster Offer’s World-Class Elasticsearch Expertise In One Powerful Product
Try Our Free ES Check-Up   Prevent Incident

Troubleshooting background

To troubleshoot Elasticsearch log “Failed to list directory some of files might not be deleted” it’s important to understand common problems related to Elasticsearch concepts: blobstore, discovery-file, repositories, restore. See detailed explanations below complete with common problems, examples and useful tips.

Repository in Elasticsearch

What it is

An Elasticsearch snapshot provides a backup mechanism that takes the current state and data in the cluster and saves it to a repository (read the Glossary term Snapshot for more information). The backup process requires a repository to be created first. The repository needs to be registered using the _snapshot endpoint, and multiple repositories per cluster can be created. The following repository types are supported. 

Repository Types:
Repository TypeConfiguration Type
Shared file systemType: “fs”
S3Type : “s3”
HDFSType :“hdfs”
AzureType: “azure”
Google Cloud StorageType : “gcs”
Examples

To register a repository of type fs:

PUT _snapshot/my_repo_01
{
"type": "fs",
"settings": {
"location": "/mnt/my_repo_dir"
  }
}
Notes and common problems
  • S3, HDFS , Azure and Google Cloud requires a relevant plugin to be installed before it can be used for a snapshot.
  • The setting, path.repo: /mnt/my_repo_dir needs to be added to elasticsearch.yml on all the nodes in case you are planning to use repo type of file system otherwise it will fail
  • In case of using remote repositories , the network bandwidth and repository storage throughput should be high enough to complete the snapshot operations normally , otherwise you will end up in partial snapshots.

DELETE Elasticsearch API

What is it

DELETE  is an Elasticsearch API which removes a document from a specific index. This API requires an index name and _id document to delete the document. 

Delete a document

DELETE /my_index/_doc/1
Notes
  • A delete request throws 404 error code if the document does not already exist in the index.
  • If you want to delete a set of documents that  matches a query, you need to use delete by query API.

To help troubleshoot related issues we have gathered selected Q&A from the community and issues from Github , please review the following for further information :

1 Failed to start Sonar Webservice 5.51 K 9

2 Elasticsearch, Failed to obtain node lock, is the following location writable 38.79 K  50


Log Context

Log ”[{}] [{}] failed to list directory – some of files might not be deleted” classname is FileRestoreContext.java
We have extracted the following from Elasticsearch source code to get an in-depth context :

                     } catch (IOException e) {
                        logger.warn("[{}] [{}] failed to delete file [{}] during snapshot cleanup"; shardId; snapshotId; storeFile);
                    }
                }
            } catch (IOException e) {
                logger.warn("[{}] [{}] failed to list directory - some of files might not be deleted"; shardId; snapshotId);
            }
        } finally {
            store.decRef();
        }
    }






About Opster

Opster identifies and predicts root causes of Elasticsearch problems, provides recommendations and can automatically perform various actions to prevent issues, optimize performance and save resources.

Learn more: Glossary | Blog| Troubleshooting guides | Error Repository

Need help with any Elasticsearch issue ? Contact Opster