How to Improve your Elasticsearch Indexing Rate

How to Improve your Elasticsearch Indexing Rate

Last Updated : May 2020

Learn how to improve your Elasticsearch Indexing speed for better ES performance by following these 11 useful tips:

  1. Tune Refresh Interval

    Tune refresh_interval (default 1 sec) according to your system requirements.

  2. Disable Replicas

    You can follow this official guide to disable replicas and set according to your requirements.

  3. Automatic ID Field

    Do not set the “_id” field of the document.  If not necessary, it is better to allow Elasticsearch to set the “_id” automatically.

  4. Use Multiple Workers/Threads

    Use multiple workers / threads to index.

  5. Use Official Clients

    Use “official” Elasticsearch clients since they have been designed to optimize connection pooling and keep alives.

  6. Avoid Frequent Updates

    Avoid frequent updates (to the same document), as every update creates a new document in Elasticsearch and marks the old document as deleted. This can cause there to be several deleted documents and altogether larger segment sizes, which don’t always merge during the segment merging process. To work around this, you may be able to collect all of these updates in the application used to call the index API (such as the search service written in Java or Python) in order to 1) remove unnecessary updates (like multiple updates to counter fields), and 2) send only a few updates to Elasticsearch.

  7. Design Index Mapping Carefully

    Be careful while designing your index mapping. Don’t index fields if they are not used for search (default is true), as this will reduce the inverted index size of Elasticsearch and save the analysis cost on the field. The index option controls it.

  8. Use Analyzers Carefully

    Use analyzers carefully on your fields; some analyzers (ngram, etc.) take significant resources and can slow down the indexing speed and significantly increase the index size of large text fields.

  9. Use Wait_For Param

    If some of your requirements are to search the indexed document immediately, then instead of an explicit refresh, use wait_for param while indexing.

  10. Use Bulk API

    Use bulk API to index multiple documents instead of individually indexing numerous documents. Bulk API performance depends on the size, not on the number of documents in the request.

  11. Use SSD

    Use SSD instead of magnetic disks for a faster segment merge process.

To get personalized detailed recommendations that can improve indexing and search performance run our Elasticsearch Check-Up (no installation required)!