Briefly, this error occurs when the Elasticsearch Hadoop plugin is configured to use SIMPLE authentication, but a Kerberos principal is provided. This mismatch in configuration and actual authentication method leads to the error. To resolve this, you can either change the authentication method to Kerberos in the Elasticsearch Hadoop settings or remove the Kerberos principal if it’s not required. Alternatively, ensure that the correct authentication method is set in the Hadoop configuration files.
This guide will help you check for common problems that cause the log ” Hadoop authentication method is set to [SIMPLE]; but a Kerberos principal is ” to appear. To understand the issues related to this log, read the explanation below about the following Elasticsearch concepts: plugins, repositories.
Overview
An Elasticsearch snapshot provides a backup mechanism that takes the current state and data in the cluster and saves it to a repository (read snapshot for more information). The backup process requires a repository to be created first. The repository needs to be registered using the _snapshot endpoint, and multiple repositories can be created per cluster. The following repository types are supported:
Repository types
| Repository type | Configuration type |
|---|---|
| Shared file system | Type: “fs” |
| S3 | Type : “s3” |
| HDFS | Type :“hdfs” |
| Azure | Type: “azure” |
| Google Cloud Storage | Type : “gcs” |
Examples
To register an “fs” repository:
PUT _snapshot/my_repo_01
{
"type": "fs",
"settings": {
"location": "/mnt/my_repo_dir"
}
}Notes and good things to know
- S3, HDFS, Azure and Google Cloud require a relevant plugin to be installed before it can be used for a snapshot.
- The setting, path.repo: /mnt/my_repo_dir needs to be added to elasticsearch.yml on all the nodes if you are planning to use the repo type of file system. Otherwise, it will fail.
- When using remote repositories, the network bandwidth and repository storage throughput should be high enough to complete the snapshot operations normally, otherwise you will end up with partial snapshots.
Log Context
Log “Hadoop authentication method is set to [SIMPLE]; but a Kerberos principal is ” classname is HdfsRepository.java.
We extracted the following from Elasticsearch source code for those seeking an in-depth context :
// Check if the user added a principal to use; and that there is a keytab file provided
String kerberosPrincipal = repositorySettings.get(CONF_SECURITY_PRINCIPAL);
// Check to see if the authentication method is compatible
if (kerberosPrincipal != null && authMethod.equals(AuthenticationMethod.SIMPLE)) {
logger.warn("Hadoop authentication method is set to [SIMPLE]; but a Kerberos principal is " +
"specified. Continuing with [KERBEROS] authentication.");
SecurityUtil.setAuthenticationMethod(AuthenticationMethod.KERBEROS; hadoopConfiguration);
} else if (kerberosPrincipal == null && authMethod.equals(AuthenticationMethod.KERBEROS)) {
throw new RuntimeException("HDFS Repository does not support [KERBEROS] authentication without " +
"a valid Kerberos principal and keytab. Please specify a principal in the repository settings with [" +
[ratemypost]