How to use the AutoOps Operator

Quick links:

• Overview
• Index management API
  • Delete index
  • Delete empty index
  • Rollover index
    • Rollover single index
    • Rollover index scan
  • Path: POST operator/rollover/scan
  • Reindex
  • Shrink index
    • Single run
    • Shrink index scan
  • Optimize indices
  • Split index
• Job management API
  • Get list of jobs
  • Get a single job
  • Delete job
  • Deletes a particular job (by its job ID)
• Operator Management API
  • Get configuration
  • Get health

Overview

Every job is triggered by posting a REST request defining the job and its configuration as the body. For every successful request, a response with the taskId will be returned to the client.

Example response for delete empty indices task:

delete-empty-index-6d377563-4243-4b6f-b26f-222222

Index management API

Delete index

Delete index job deletes existing indices that match the index prefix. When the job finds candidate indices that are older then the retention, it deletes them. The job can be triggered for several clusters on a single REST request.

Path: POST operator/delete-index Query params: scheduled (Optional. Default is false), interval (Optional. Default is 10 sec)

Example request body:

{
   "indexPrefix": "acc-*",
   "retention": "14d",
   "timeField": "timestamp",
   "backends": [1, 2]
}

Delete empty index

The delete empty index job deletes indices (using the alias) that are empty, meaning that the number of documents on these indices is zero.

The job currently supports 2 different strategies:

1. Delete indices matching the index pattern with creation date older than today, meaning created 24H ago and older.
2. Delete indices matching the index pattern that are not the current write index for the alias defined for them.

Path: POST operator/delete-empty-index Query params: scheduled (Optional. Default is false), interval (Optional. Default is 10)

Example for by date delete request body:

{
 "indexPatterns": ["*some_string*"],
 "indexPatternStrategy" : "BY_DATE",
 "datePatternGroup": 1,
 "backends": [1],
 "deletionStrategy": "BY_CREATION_DATE",
 "datePattern": ".*(\\d{4}.\\d{2}.\\d{2})",
 "dateFormat": "yyyy.MM.dd"
}

Example for by attribute delete request body:

{
 "indexPatterns": ["*index-example*", "*000001*"],
 "backends": [
   1
 ],
 "deletionStrategy": "BY_WRITE_ATTRIBUTE"
}

Rollover index

The rollover job performs rollovers for existing indices that match certain conditions. The conditions are size, max documents and index age. This job can be triggered to rollover 1 index (by alias) or as a scan job for all the indices on the cluster.

Rollover single index

Path: POST operator/rollover Query params: scheduled (Optional. Default is false) interval (Optional. Default is 10)

Example request body:

{
   "indexAlias": ["alias-example333"],
   "sizeThreshold": 10240,
   "maxDocsThreshold":10000,
   "indexRetentionInHours":24,
   "indexCreationStrategies": "TAKE_FROM_CONFIG",
   "shardsConfig": {
       "primaryShards": 2,
       "replicaShards": 1,
       "refreshInterval":1
   },
   "shardCreateExpressions": {
       "primary": "data_node / 2",
       "replica": "1",
       "refreshInterval": "3"
   },
   "desireDigitsCount": 4,
   "postRolloverActions": {
       "force-merge": {
           "maxSegments": 4
       },
       "time-range-filter": {}
   },
   "timeField": "timestamp",
   "clusterDestinationId": 1,
   "shouldCopyMapping": "true"
}

Rollover index scan

The job will scan the cluster/s and create a list of all indices to rollover, then generate and execute jobs for every index.

Path: POST operator/rollover/scan

QueryParams: scheduled (Optional. Default is false), interval (Optional. Default is 10)

Example request body:

{
   "ignoreAliasPatterns": {
       "1": [
           "sample-index"
       ],
       "100": [
           ".kibana",
           ".kibana_task_manager",
           ".slm-history-1",
           ".opster-operator"
       ],
       "200": [
           ".kibana",
           ".kibana_task_manager",
           ".slm-history-1",
           ".opster-operator"
       ]
   },
   "rolloverRequest": {
       "shouldCopyMapping": false,
       "sizeThreshold": 10474825485,
       "indexCreationStrategies": "BY_EXPRESSION",
       "shardCreateExpressions": {
           "primary": "(data_node / 2)",
           "replica": "1"
       },
       "postRolloverActions": {
           "time-range-filter": {}
       },
       "timeField": "timestamp"
   },
   "rolloverIndexScheduled": false,
   "rolloverIndexIntervalInSeconds": 10
}

Reindex

Reindex is a process that reads data from a certain index and writes it to a new index. There are various reasons why a reindex would be required/recommended, but a few possible reasons would be: changes in templates, a move to a new index with more primary shards, and more.

The job uses multiple sub processes to parallelize the work as much as possible in order to speed up the process.

Path: POST operator/reindex Example request body:

{
   "indexNamesList": [
       "acc-44-00001",
       "acc-42-00001"
   ],
   "targetIndexName": "acct-1",
   "numberOfConcurrentProcessedIndices": 2,
   "threadsPerIndex": 10,
   "continueOnFailure": true,
   "retriesOnFailedTasks": 10,
   "timeFrameInMinutes": 10,
   "dateField": "timestamp",
   "clusterDestinationId": 1,
   "useNewIndexOnly": false,
   "indexCreationStrategy": “TAKE_FROM_CONFIG”,
   "shardsConfig": {
       "primaryShards": 2,
       "replicaShards": 1,
       "refreshInterval":1
   },
   "shardCreateExpressions": {
       "primary": "data_node / 2",
       "replica": "1",
       "refreshInterval": "3"
   }
}

Shrink index

The shrink job allows you to shrink an existing index into a new index with fewer primary shards. This is usually a good solution when the shards of such an index are small in size (for example, an index with 8 primary shards with 1GB of size per shard). The shrink job can shrink this index to 1 primary shard with 8GB. This operation will reduce the management overhead on the cluster.

Note: This operation can add load on the cluster so it’s recommended to run it when the cluster is less loaded. Also, make sure you have enough disk space before running the Operation (At least the side of the source index)

Single run

Path: POST operator/shrink Example request body:

{  
   "indexPattern": "acc-13-00002",
   "maxShardSizeInGbBeforeShrink": 1,
   "maxShardSizeInGbAfterShrink": 20,
   "desiredNumberOfShardsAfterShrink": 1,
   "clusterDestinationId" : 1,
   "sleepBetweenShrinksInSec": 0,
   "datePattern": ".*(\\d{4}.\\d{2}.\\d{2})",
   "dateFormat": "yyyy.MM.dd",
   "datePatternGroup": 1,
   "timeField": "timestamp",
   "noWritesToIndexThresholdInMinutes": 5,
   "indexDesiredDigitsCount": 5,
   "shrinkTodayIndex": false
}

Shrink index scan

The job will scan the cluster/s and create a list of all indices to rollover, then generate and execute jobs for every index.

Path: POST operator/shrink/scan QueryParams: scheduled (Optional. Default is false) , interval (Optional. Default is 10 second) Example request body:

{  
"indexPattern": {"1": ["acc-1-*"]},
"excludeIndices": {"1": ["acc-1001"]},
"maxTaskIsParallel": {"1": 1},
"shrinkRequest": {
   "maxShardSizeInGbBeforeShrink": 1,
   "maxShardSizeInGbAfterShrink": 20,
   "desiredNumberOfShardsAfterShrink": 1,
   "sleepBetweenShrinksInSec": 0,
   "datePattern": ".*(\\d{4}.\\d{2}.\\d{2})",
   "dateFormat": "yyyy.MM.dd",
   "datePatternGroup": 1,
   "timeField": "timestamp",
   "noWritesToIndexThresholdInMinutes": 5,
   "indexDesiredDigitsCount": 5
}
}

Optimize indices

One important task for an Elasticsearch master is to manage the cluster state which holds all the information about the indices, including their mapping, size, data location and more. As the number of indices increases, the cluster state becomes bigger and the overhead on the cluster can harm the cluster integrity and performance. When there are a lot of small indices, the job will reindex them to one index according to their size and reduce the overhead on the cluster.

Path: POST operator/shards/optimizer

Example request body:

{
   "indexPattern": "*acc*",
   "indexReplicas": 0,
   "reindexMaxSize": 30720,
   "maxWritesPerIndex": 5,
   "maxWritesPerIndexIntervalInSec": 10,
   "reindexWaitForCompletion": false,
   "retainExistingAliases": false,
   "sleepBetweenMergesInSec": 15,
   "noOptimizePattern": true,
   "indexDesiredDigitsCount" : 6,
   "clusterDestinationId": 1,
   "optimizeTodayIndex": true,
   "preTaskFilters": [{"type": "EXISTS", "modal": "must_exists", "fields": ["tenant"]}],
   "postReIndexActions": [{"type": "create_alias", "filters": [{"FilterType": "field", "field":"tenant"}]}]
}

Split index

Splits an existing index into a new index with more primary shards. This job is used when an index has a number of primaries that are big in size (more than 50GB). The management for such an index in Elasticsearch can cause overhead on the cluster state and can be problematic when the shard needs to move to another node as part of the regular Elasticsearch operations. The split process improves indexing performance by creating a new index with more primary shards.

Note: This operation can add load on the cluster so it’s recommended to run it when the cluster is less loaded. Also, make sure you have enough disk space before running the Operation (At least the side of the source index)

Path: POST operator/split QueryParams: scheduled (Optional. Default is false)

Example request body:

{
   "sourceIndex" : "src-index-00001",
   "targetIndex": "src-index-split-00001",
   "sourceIndexIsReadOnly": true,
   "clusterDestinationId" : 1,
   "sleepBetweenSplitsInSec": 10,
   "splitTodayIndex": false,
   "desiredNumberOfShardsAfterSplit": 4,
   "shouldDeleteSourceIndex": false,
   "retryIntervalOnSplitFailedAttemptString": "15s"
}

Job management API

Get list of jobs

Gets the list of jobs currently configured for the Operator to execute.

Path: GET /operator/jobs Query params (all are optional): status – can be one of: (SCHEDULED,IN_PROGRESS,RUNNING,CANCELED,DONE,ERROR,NOT_FOUND) strategy – can be one of: (EXECUTE_ONCE,SCHEDULED) IncludeCancelled – boolean. Default is false. Example request body: None Response:

[
  {
    "jobId": "<job type>-<UUID>",
    "status": status,
    "createdOn": datetime,
    "lastExecutionTime": datetime,
    "interval": 0,
    "strategy": strategy,
    "jobConfig": escaped JSON string
  },
  ...
]

Get a single job

Gets a particular job (by its ID) currently configured for the Operator to execute. Path: GET /operator/jobs/<job ID> Example request body: None Response:

 {
    "jobId": "<job type>-<UUID>",
    "status": status,
    "createdOn": datetime,
    "lastExecutionTime": datetime,
    "interval": 0,
    "strategy": strategy,
    "jobConfig": escaped JSON string
  }

Delete job

Deletes a particular job (by its job ID)

Path: DELETE /operator/jobs/{jobId} Example request body: None Response:

{
  "jobId": jobId,
  "status": "CANCELED",
  "interval": 0
}

Operator Management API

Get configuration

Path: GET /operator/config Example request body: NoneResponse: JSON representation of full configuration, example:

{
    "restServer": {
        "port": 5530
    },
    "shardOptimization": {
        "sleepBetweenMergesInSec": 0,
        "retryIntervalOnFailedIndexOptimizationAttemptInMinutes": 60,
        "retryIntervalOnFailedIndexOptimizationAttemptAfterThreeTriesInHours": 11,
        "indexPattern": null,
        "maxWritesPerIndex": 10,
        "targetIndexShardsCount": 2,
        "reindexMaxSize": 0,
        "reindexRequestPerSecond": -1.0,
        "reindexWaitForCompletion": false,
        "retainExistingAliases": true,
        "reindexTimeout": "5m",
        "indexReplicas": 2,
        "noOptimizePattern": true,
        "optimizeTodayIndex": false,
        "maxWritesPerIndexIntervalInSec": 15,
        "numOfConcurrentIndicesToOptimize": 1,
        "indexDesiredDigitsCount": 4,
        "retryTimesForOperation": 3
    },
    "jobScheduler": {
        "jobExecutionIntervalInSec": 10,
        "poolSize": 1,
        "timeFromLastSchedulerExecutionTimeThresholdInSec": 60,
        "jobToConcurrentExecutions": {
            "com.opster.operator.optimizers.shrink.SplitIndexTask": 1,
            "com.opster.operator.optimizers.shard.jobs.ShardOptimizerTask": 1,
            "com.opster.operator.optimizers.shrink.ShrinkIndexTask": 1,
            "com.opster.operator.reindex.SplitIndexTask": 1
        }
    },
    "jobStore": {
        "url": "$ES_URL",
        "indexName": ".opster-operator",
        "logsTemplateName": "operator-template-logs",
        "templateName": "operator",
        "cancelableJobsIndex": ".cancelable-tasks",
        "connectRetries": 5,
        "connectRetriesIntervalInMilli": 1000,
        "authInfo": {
            "type": "NONE",
            "credentials": {}
        }
    },
    "jobExecutor": {
        "poolSize": 5
    },
    "esClusters": [
        {
            "url": "$ES_URL",
            "id": 1,
            "authInfo": {
                "type": "NONE",
                "credentials": {}
            }
        },
        {
            "url": "http://localhost:9200",
            "id": 2,
            "authInfo": {
                "type": "BASIC",
                "credentials": {
                    "user": "user1",
                    "password": "userPass1"
                }
            }
        },
        {
            "url": "http://localhost:9201",
            "id": 3,
            "authInfo": {
                "type": "BASIC",
                "credentials": {
                    "user": "user2",
                    "password": "userPass2"
                }
            }
        }
    ],
    "esClient": {
        "readTimeout": 12000,
        "connTimeout": 30000,
        "maxConnections": 30
    },
    "rollover": {
        "indexAliases": null,
        "searchAliases": null,
        "desireDigitsCount": 4,
        "indexCreationStrategies": null,
        "shardCreateExpressions": null,
        "shardsConfig": {
            "primaryShards": 1,
            "replicaShards": 1,
            "refreshInterval": null,
            "indexName": null
        },
        "sizeThreshold": -1,
        "postRolloverActions": null,
        "timeField": null,
        "retryTimesAllowed": 0,
        "forceMergeMaxNumSegments": 30,
        "clusterDestinationId": -1,
        "shouldCopyMapping": false,
        "maxDocsThreshold": -1,
        "indexRetentionInHours": -1
    },
    "shrink": {
        "indexPattern": null,
        "operatorIndexPattern": null,
        "maxShardSizeInGbBeforeShrink": 0,
        "maxShardSizeInGbAfterShrink": 0,
        "desiredNumberOfShardsAfterShrink": 0,
        "desiredNumberOfReplicasAfterShrink": 0,
        "minimumIndexAgeInMinutesBeforeShrink": 5,
        "dryRun": false,
        "clusterDestinationId": 0,
        "retryTimesForOperation": 3,
        "datePattern": null,
        "timeField": null,
        "noWritesToIndexThresholdInMinutes": 5,
        "indexDesiredDigitsCount": 3,
        "maxRetries": 0,
        "waitForPrerequisites": [],
        "dateFormat": null,
        "datePatternGroup": 0,
        "shrinkTodayIndex": false,
        "scanTask": false
    },
    "split": {
        "datePattern": null,
        "dateFormat": null,
        "datePatternGroup": 0,
        "sourceIndex": null,
        "targetIndex": null,
        "targetIndexSuffix": null,
        "dryRun": false,
        "clusterDestinationId": 0,
        "splitTodayIndex": false,
        "desiredNumberOfShardsAfterSplit": 0,
        "desiredShardSizeAfterSplitInGB": 10,
        "shouldDeleteSourceIndex": false,
        "retryIntervalOnSplitFailedAttemptString": "10h",
        "forceMergeMaxNumSegments": 0,
        "performForceMerge": false,
        "scanTask": false
    },
    "clusterIdIndexStoreConfiguration": {
        "expireAfterWriteInSeconds": 600
    }
}

Get health

Path: GET /health, /operator/health Example request body: None Response: OK <!– /wp:paragraph →