Elasticsearch How to Migrate From ECK to OpenSearch Using the Kubernetes Operator

To manage all aspects of your OpenSearch operation, you can use Opster’s Management Console (OMC). The OMC makes it easy to orchestrate and manage OpenSearch in any environment. Using the OMC you can deploy multiple clusters, configure node roles, scale cluster resources, manage certificates and more – all from a single interface, for free. Check it out here.

How to Upgrade Version with the OMC

You can also start a free trial of Opster’s full platform, AutoOps, to improve performance, ensure stability and reduce your hardware costs 💪

Quick links

• Introduction
• How to Migrate From ECK to OpenSearch Using the Kubernetes Operator
  • Hardware specifications
  • Storage classes
  • Configuring the OpenSearch Operator
  • Exposing the OpenSearch service and dashboards
• Conclusion

Introduction

This guide will cover how to use the OpenSearch Kubernetes Operator to migrate from an Elasticsearch cluster running on Kubernetes with ECK, to an OpenSearch cluster running on Kubernetes. 

The focus of this guide will be on the hardware and cluster architecture, including storage configurations, and more. Please note that this guide does not cover data migration, only how to set up a cluster with similar resources, so that users can easily migrate data between two clusters. Opster is currently developing tools for this next step as well, so stay tuned.

Opster’s OSS OpenSearch Kubernetes Operator, licensed Apache V2, simplifies spinning up and scaling clusters, optimizing configuration, version upgrades, security, and more. Whether you’re managing your own K8s cluster or going the hosted route, you can deploy the OpenSearch Operator and benefit from its capabilities on all cloud environments.

You can also use Opster’s Management Console to complete the same actions with an easy-to-use, free interface. 

How to Migrate From ECK to OpenSearch Using the Kubernetes Operator 

When migrating from ECK to OpenSearch Operator, users need to consider the hardware specifications and the YAML files for both tools. Keep reading to learn about the differences and the steps necessary to carry out migrations.

Hardware specifications

Firstly, let’s take a look at the hardware specifications for the initial ECK cluster:

3 master nodes: 1GB RAM, 16GB disk, and 500m CPU
3 data nodes: 2GB RAM, 64GB disk, and 1000m CPU
3 coordinating nodes: 1GB RAM, 64GB disk, and 500m CPU
1 Kibana node: 1GB RAM, and 500m CPU
S3-repository plugin

The YML file looks like this: 

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
 name: eck-cluster
spec:
 version: 8.6.1
 nodeSets:
   - name: master
     count: 3
     config:
       node.roles: ["master"]
       node.store.allow_mmap: false
     podTemplate:
       spec:
         containers:
           - name: elasticsearch
             imagePullPolicy: IfNotPresent
             resources:
               limits:
                 memory: 1Gi
                 cpu: 500m
             env:
               - name: ES_JAVA_OPTS
                 value: "-Xms500m -Xmx500m"
     volumeClaimTemplates:
       - metadata:
           name: elasticsearch-data # Do not change this name unless you set up a volume mount for the data path.
         spec:
           accessModes:
             - ReadWriteOnce
           resources:
             requests:
               storage: 16Gi
           storageClassName: hostpath # Use your own storageClassName
   - name: data
     count: 3
     config:
       node.roles: ["data"]
       node.store.allow_mmap: false
     podTemplate:
       spec:
         containers:
           - name: elasticsearch
             imagePullPolicy: IfNotPresent
             resources:
               limits:
                 memory: 4Gi
                 cpu: 1000m
             env:
               - name: ES_JAVA_OPTS
                 value: "-Xms2g -Xmx2g"
     volumeClaimTemplates:
       - metadata:
           name: elasticsearch-data # Do not change this name unless you set up a volume mount for the data path.
         spec:
           accessModes:
             - ReadWriteOnce
           resources:
             requests:
               storage: 64Gi
           storageClassName: hostpath # Use your own storageClassName
   - name: coordinating
     count: 3
     config:
       node.roles: []
       node.store.allow_mmap: false
     podTemplate:
       spec:
         containers:
           - name: elasticsearch
             imagePullPolicy: IfNotPresent
             resources:
               limits:
                 memory: 2Gi
                 cpu: 500m
             env:
               - name: ES_JAVA_OPTS
                 value: "-Xms1g -Xmx1g"
     volumeClaimTemplates:
       - metadata:
           name: elasticsearch-data # Do not change this name unless you set up a volume mount for the data path.
         spec:
           accessModes:
             - ReadWriteOnce
           resources:
             requests:
               storage: 64Gi
           storageClassName: hostpath # Use your own storageClassName
---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
 name: es-to-os
spec:
 version: 8.6.1
 count: 1
 podTemplate:
   spec:
     containers:
       - name: kibana
         imagePullPolicy: IfNotPresent
         resources:
           limits:
             memory: 1Gi
             cpu: 500m

Storage classes

Now, let’s focus on the ECK YAML file. The ECK configuration by default allocates a persistent volume of 1GiB for storage, using the default storage class defined for the Kubernetes cluster, but users can also set a specific storage class, under volumeClaimTemplates, such as “hostpath.” However, the ECK configuration doesn’t create storage classes, and users have to check for their default storage class by running “kubectl get sc.”

Configuring the OpenSearch Kubernetes Operator

The OpenSearch Operator will create storage classes for users. However, if clusters are reinstalled, users need to manually delete the storage classes, or the bootstrapping will fail the next time the same cluster name is deployed in the same namespace. Another key difference is that the S3-repository is installed by default in Elasticsearch, while users must install it themselves on OpenSearch. Additionally, the Kubernetes Operator can install the S3 plugin automatically (as well as any other plugin), using a rolling restart. All users need to do is list the required plugins in the cluster CRD, as can be seen in the “pluginsList” row.

apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
 name: os-cluster
spec:
 security:
   config:
   tls:
     http:
       generate: true
     transport:
       generate: true
       perNode: true
 general:
   httpPort: 9200
   serviceName: os-cluster
   version: 2.3.0
   pluginsList: ["repository-s3"]
   drainDataNodes: true
 dashboards:
   tls:
     enable: true
     generate: true
   version: 2.3.0
   enable: true
   replicas: 0
   resources:
     requests:
       memory: "1Gi"
       cpu: "500m"
     limits:
       memory: "1Gi"
       cpu: "500m"
 confMgmt:
   smartScaler: true
 nodePools:
   - component: master
     replicas: 2
     diskSize: "16Gi"
     jvm: -Xmx512M -Xms512M
     nodeSelector:
     resources:
       requests:
         memory: "1Gi"
         cpu: "500m"
       limits:
         memory: "1Gi"
         cpu: "500m"
     roles:
       - "master"
   - component: data
     replicas: 2
     diskSize: "64Gi"
     jvm: -Xmx2g -Xms2g
     nodeSelector:
     resources:
       requests:
         memory: "4Gi"
         cpu: "1000m"
       limits:
         memory: "4Gi"
         cpu: "1000m"
     roles:
       - "data"
   - component: coordinators
     replicas: 2
     diskSize: "64Gi"
     jvm: -Xmx1g -Xms1g
     nodeSelector:
     resources:
       requests:
         memory: "2Gi"
         cpu: "500m"
       limits:
         memory: "2Gi"
         cpu: "500m"
     roles:
       - "ingest"

Moving on to the OpenSearch Operator YAML file, instead of using volumeClaimTemplates, users can define diskSize for persistent volumes. Additionally, the S3-repository must be added to the pluginsList and users must set “replicas” instead of node counts. 

It’s important to note that the coordinating node is represented as an ingest role only.

Exposing OpenSearch Service and Dashboards

To expose the OpenSearch service, use the following command:

kubectl port-forward service/os-cluster 9200:9200

To expose the OpenSearch dashboards service, use the following:

kubectl port-forward service/os-cluster-dashboards 5601:5601

Conclusion

Overall, migrating from ECK to OpenSearch Operator can be a smooth process with proper planning and attention to detail. By analyzing the differences between the two tools and modifying the YAML files, users can successfully complete the migration.

After successfully spinning up your OpenSearch cluster, remember to:

1. Transfer the configurations – this will include translating some configs, such as ILM and snapshot management.
2. Transfer data.

Opster is currently building migration tools for data and configuration transfers, so stay tuned!

Read more about the Kubernetes Operator here.