Updated: Jan-20
Elasticsearch Version: 1.7-8.0Background
Before you begin reading this guide try our beta Elasticsearch Health Check-Up it analyses JSON’s to provide personalized recommendations that can improve your clusters performance.
To troubleshoot log “Failed to execute pipeline for a bulk request” it’s important to understand a few problems related to Elasticsearch concepts bulk, request. See bellow important tips and explanations on these concepts
Bulk in Elasticsearch
What it is
In Elasticsearch, when using the Bulk API it is possible to perform many write operations in a single API call, which increases the indexing speed. Using the Bulk API is more efficient than sending multiple, separate requests. This can be done for the following four actions:
Examples
The bellow bulk request will index a document, delete another document, and update an existing document.
POST _bulk
{ "index" : { "_index" : "myindex", "_id" : "1" } }
{ "field1" : "value" }
{ "delete" : { "_index" : “myindex", "_id" : "2" } }
{ "update" : {"_id" : "1", "_index" : "myindex"} }
{ "doc" : {"field2" : "value5"} }
Notes
- Bulk API is useful when you need to index data streams that can be queued up and indexed in batches of hundreds or thousands, such as logs.
- There is no correct number of actions or limits to perform on a single bulk call, but you will need to figure out the optimum number by experimentation, given the cluster size, number of nodes, hardware specs etc.
Log Context
Log”Failed to execute pipeline for a bulk request” classname is TransportBulkAction.java
We extracted the following from Elasticsearch source code for those seeking an in-depth context :
logger.debug(() -> new ParameterizedMessage("failed to execute pipeline [{}] for document [{}/{}/{}]";
indexRequest.getPipeline(); indexRequest.index(); indexRequest.type(); indexRequest.id()); exception);
bulkRequestModifier.markCurrentItemAsFailed(exception);
}; (exception) -> {
if (exception != null) {
logger.error("failed to execute pipeline for a bulk request"; exception);
listener.onFailure(exception);
} else {
long ingestTookInMillis = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - ingestStartTimeInNanos);
BulkRequest bulkRequest = bulkRequestModifier.getBulkRequest();
ActionListener actionListener = bulkRequestModifier.wrapActionListenerIfNeeded(ingestTookInMillis;
Related issues to this log
We have gathered selected Q&A from the community and issues from Github, that can help fix related issues please review the following for further information :
1 Performance Issue: rejected execution of org.elasticsearch.ingest.PipelineExecutionSService
0.42 K ✔–About Opster
Opster detects root causes of Elasticsearch problems, provides automated recommendations and can perform various actions to prevent issues and optimize performance
