Before you dig into the details of this technical guide, have you tried asking OpsGPT?
You'll receive concise answers that will help streamline your Elasticsearch/OpenSearch operations.
Try OpsGPT now for step-by-step guidance and tailored insights into your Elasticsearch/ OpenSearch operation.
Before you dig into the details of this guide, have you tried asking OpsGPT? You’ll receive concise answers that will help streamline your OpenSearch/Elasticsearch operation.
Try OpsGPT now for step-by-step guidance and tailored insights into your search operation.
To manage all aspects of your OpenSearch operation, you can use Opster’s Management Console (OMC). The OMC makes it easy to orchestrate and manage OpenSearch in any environment. Using the OMC you can deploy multiple clusters, configure node roles, scale cluster resources, manage certificates and more – all from a single interface, for free. Check it out here.
You can also start a free trial of Opster’s full platform, AutoOps, to improve performance, ensure stability and reduce your hardware costs 💪
Quick links
Introduction
This guide will cover how to use the OpenSearch Kubernetes Operator to migrate from an Elasticsearch cluster running on Kubernetes with ECK, to an OpenSearch cluster running on Kubernetes.
The focus of this guide will be on the hardware and cluster architecture, including storage configurations, and more. Please note that this guide does not cover data migration, only how to set up a cluster with similar resources, so that users can easily migrate data between two clusters. Opster is currently developing tools for this next step as well, so stay tuned.
Opster’s OSS OpenSearch Kubernetes Operator, licensed Apache V2, simplifies spinning up and scaling clusters, optimizing configuration, version upgrades, security, and more. Whether you’re managing your own K8s cluster or going the hosted route, you can deploy the OpenSearch Operator and benefit from its capabilities on all cloud environments.
You can also use Opster’s Management Console to complete the same actions with an easy-to-use, free interface.
How to Migrate From ECK to OpenSearch Using the Kubernetes Operator
When migrating from ECK to OpenSearch Operator, users need to consider the hardware specifications and the YAML files for both tools. Keep reading to learn about the differences and the steps necessary to carry out migrations.
Hardware specifications
Firstly, let’s take a look at the hardware specifications for the initial ECK cluster:
3 master nodes: 1GB RAM, 16GB disk, and 500m CPU
3 data nodes: 2GB RAM, 64GB disk, and 1000m CPU
3 coordinating nodes: 1GB RAM, 64GB disk, and 500m CPU
1 Kibana node: 1GB RAM, and 500m CPU
S3-repository plugin
The YML file looks like this:
apiVersion: elasticsearch.k8s.elastic.co/v1 kind: Elasticsearch metadata: name: eck-cluster spec: version: 8.6.1 nodeSets: - name: master count: 3 config: node.roles: ["master"] node.store.allow_mmap: false podTemplate: spec: containers: - name: elasticsearch imagePullPolicy: IfNotPresent resources: limits: memory: 1Gi cpu: 500m env: - name: ES_JAVA_OPTS value: "-Xms500m -Xmx500m" volumeClaimTemplates: - metadata: name: elasticsearch-data # Do not change this name unless you set up a volume mount for the data path. spec: accessModes: - ReadWriteOnce resources: requests: storage: 16Gi storageClassName: hostpath # Use your own storageClassName - name: data count: 3 config: node.roles: ["data"] node.store.allow_mmap: false podTemplate: spec: containers: - name: elasticsearch imagePullPolicy: IfNotPresent resources: limits: memory: 4Gi cpu: 1000m env: - name: ES_JAVA_OPTS value: "-Xms2g -Xmx2g" volumeClaimTemplates: - metadata: name: elasticsearch-data # Do not change this name unless you set up a volume mount for the data path. spec: accessModes: - ReadWriteOnce resources: requests: storage: 64Gi storageClassName: hostpath # Use your own storageClassName - name: coordinating count: 3 config: node.roles: [] node.store.allow_mmap: false podTemplate: spec: containers: - name: elasticsearch imagePullPolicy: IfNotPresent resources: limits: memory: 2Gi cpu: 500m env: - name: ES_JAVA_OPTS value: "-Xms1g -Xmx1g" volumeClaimTemplates: - metadata: name: elasticsearch-data # Do not change this name unless you set up a volume mount for the data path. spec: accessModes: - ReadWriteOnce resources: requests: storage: 64Gi storageClassName: hostpath # Use your own storageClassName --- apiVersion: kibana.k8s.elastic.co/v1 kind: Kibana metadata: name: es-to-os spec: version: 8.6.1 count: 1 podTemplate: spec: containers: - name: kibana imagePullPolicy: IfNotPresent resources: limits: memory: 1Gi cpu: 500m
Storage classes
Now, let’s focus on the ECK YAML file. The ECK configuration by default allocates a persistent volume of 1GiB for storage, using the default storage class defined for the Kubernetes cluster, but users can also set a specific storage class, under volumeClaimTemplates, such as “hostpath.” However, the ECK configuration doesn’t create storage classes, and users have to check for their default storage class by running “kubectl get sc.”
Configuring the OpenSearch Kubernetes Operator
The OpenSearch Operator will create storage classes for users. However, if clusters are reinstalled, users need to manually delete the storage classes, or the bootstrapping will fail the next time the same cluster name is deployed in the same namespace. Another key difference is that the S3-repository is installed by default in Elasticsearch, while users must install it themselves on OpenSearch. Additionally, the Kubernetes Operator can install the S3 plugin automatically (as well as any other plugin), using a rolling restart. All users need to do is list the required plugins in the cluster CRD, as can be seen in the “pluginsList” row.
apiVersion: opensearch.opster.io/v1 kind: OpenSearchCluster metadata: name: os-cluster spec: security: config: tls: http: generate: true transport: generate: true perNode: true general: httpPort: 9200 serviceName: os-cluster version: 2.3.0 pluginsList: ["repository-s3"] drainDataNodes: true dashboards: tls: enable: true generate: true version: 2.3.0 enable: true replicas: 0 resources: requests: memory: "1Gi" cpu: "500m" limits: memory: "1Gi" cpu: "500m" confMgmt: smartScaler: true nodePools: - component: master replicas: 2 diskSize: "16Gi" jvm: -Xmx512M -Xms512M nodeSelector: resources: requests: memory: "1Gi" cpu: "500m" limits: memory: "1Gi" cpu: "500m" roles: - "master" - component: data replicas: 2 diskSize: "64Gi" jvm: -Xmx2g -Xms2g nodeSelector: resources: requests: memory: "4Gi" cpu: "1000m" limits: memory: "4Gi" cpu: "1000m" roles: - "data" - component: coordinators replicas: 2 diskSize: "64Gi" jvm: -Xmx1g -Xms1g nodeSelector: resources: requests: memory: "2Gi" cpu: "500m" limits: memory: "2Gi" cpu: "500m" roles: - "ingest"
Moving on to the OpenSearch Operator YAML file, instead of using volumeClaimTemplates, users can define diskSize for persistent volumes. Additionally, the S3-repository must be added to the pluginsList and users must set “replicas” instead of node counts.
It’s important to note that the coordinating node is represented as an ingest role only.
Exposing OpenSearch Service and Dashboards
To expose the OpenSearch service, use the following command:
kubectl port-forward service/os-cluster 9200:9200
To expose the OpenSearch dashboards service, use the following:
kubectl port-forward service/os-cluster-dashboards 5601:5601
Conclusion
Overall, migrating from ECK to OpenSearch Operator can be a smooth process with proper planning and attention to detail. By analyzing the differences between the two tools and modifying the YAML files, users can successfully complete the migration.
After successfully spinning up your OpenSearch cluster, remember to:
- Transfer the configurations – this will include translating some configs, such as ILM and snapshot management.
- Transfer data.
Opster is currently building migration tools for data and configuration transfers, so stay tuned!
Read more about the Kubernetes Operator here.