Briefly, this error occurs when Elasticsearch cannot access the security index, which is crucial for user authentication and authorization. This could be due to issues like insufficient permissions, network problems, or the index being deleted or corrupted. To resolve this, you can check the index’s status and restore it if necessary, ensure the user has appropriate permissions, or troubleshoot network connectivity. If the index is corrupted, you may need to recreate it. Always ensure to have regular backups to prevent data loss.
This log could be avoided if detected earlier. Before you read this guide, we recommend you run the Elasticsearch Error Check-Up which detects issues in ES that cause log errors.The Check-Up includes checks that would help you prevent log “Security index is unavailable. short circuiting retrieval of user”. It’s a free tool that requires no installation and takes 2 minutes to complete. You can run the Check-Up here.
What the security index is
From Elasticsearch version 6.8 and onwards, the Security feature is available for free. This means you can secure your cluster by creating multiple users and roles, and all of this information is stored in a unique index called .security-<es-major-version>.
Please note the dot ‘.’ at the beginning of the index name.
What this error means
Elasticsearch index can have several states, and sometimes due to several factors, it can become unavailable, for instance because of missing primary shards, an Elasticsearch cluster running out of disk space and so on. When Elasticsearch needs to read the user information for a request, several steps occur internally.
Note that `_security` is the endpoint name used for the security API call. This is an API that would require Elasticsearch to find the information stored in the security index. The following things happen internally to figure out the user information (its ID, role, permission etc):
- Elasticsearch checks if the security index is available or not.
- Elasticsearch freezes the security index, so others can’t update the security index when it’s reading the sensitive (security) information.
- If the index isn’t available, then there is no point of querying the security index and short circuiting the query part, and it logs this as an error message as below:
security index is unavailable. short circuiting retrieval of user.
Quick troubleshooting steps
- Check if the `.security` index exists or not, by using below _cat/indices?v API and if the index exists, the output of this API would look like:
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .security-7 9blPln4uSKScEzWtMfJXNA 1 0 7 0 24.3kb 24.3kb
- Check if the security index is available, because there is no direct API that can show this. Still, as mentioned earlier, cluster RED state or disk space can cause an index to become unavailable, and checking and fixing these issues will help make the index available.
This Opster Guide can help identify and fix issues caused by low disk space.
Overview
In Elasticsearch, an index (plural: indices) contains a schema and can have one or more shards and replicas. An Elasticsearch index is divided into shards and each shard is an instance of a Lucene index.
Indices are used to store the documents in dedicated data structures corresponding to the data type of fields. For example, text fields are stored inside an inverted index whereas numeric and geo fields are stored inside BKD trees.
Examples
Create index
The following example is based on Elasticsearch version 5.x onwards. An index with two shards, each having one replica will be created with the name test_index1
PUT /test_index1?pretty
{
"settings" : {
"number_of_shards" : 2,
"number_of_replicas" : 1
},
"mappings" : {
"properties" : {
"tags" : { "type" : "keyword" },
"updated_at" : { "type" : "date" }
}
}
}List indices
All the index names and their basic information can be retrieved using the following command:
GET _cat/indices?v
Index a document
Let’s add a document in the index with the command below:
PUT test_index1/_doc/1
{
"tags": [
"opster",
"elasticsearch"
],
"date": "01-01-2020"
}Query an index
GET test_index1/_search
{
"query": {
"match_all": {}
}
}Query multiple indices
It is possible to search multiple indices with a single request. If it is a raw HTTP request, index names should be sent in comma-separated format, as shown in the example below, and in the case of a query via a programming language client such as python or Java, index names are to be sent in a list format.
GET test_index1,test_index2/_search
Delete indices
DELETE test_index1
Common problems
- It is good practice to define the settings and mapping of an Index wherever possible because if this is not done, Elasticsearch tries to automatically guess the data type of fields at the time of indexing. This automatic process may have disadvantages, such as mapping conflicts, duplicate data and incorrect data types being set in the index. If the fields are not known in advance, it’s better to use dynamic index templates.
- Elasticsearch supports wildcard patterns in Index names, which sometimes aids with querying multiple indices, but can also be very destructive too. For example, It is possible to delete all the indices in a single command using the following commands:
DELETE /*
To disable this, you can add the following lines in the elasticsearch.yml:
action.destructive_requires_name: true
Log Context
Log “security index is unavailable. short circuiting retrieval of user [{}]” classname is NativeUsersStore.java.
We extracted the following from Elasticsearch source code for those seeking an in-depth context :
final SecurityIndexManager frozenSecurityIndex = securityIndex.freeze();
if (frozenSecurityIndex.isAvailable() == false) {
if (frozenSecurityIndex.indexExists() == false) {
logger.trace("could not retrieve user [{}] because security index does not exist"; user);
} else {
logger.error("security index is unavailable. short circuiting retrieval of user [{}]"; user);
}
listener.onResponse(null);
} else {
securityIndex.checkIndexVersionThenExecute(
listener::onFailure;
