Managing Elasticsearch & OpenSearch – Expectations VS Reality

Managing Elasticsearch & OpenSearch – Expectations VS Reality

A Comparison of Elastic, AWS, Consulting Services & Opster

Quick links:

Overview

Many times people ask:

  • What do managed services cover?
  • I don’t want to deal with Elasticsearch at all – what’s my best option?
  • How can I reduce my costs and keep performance high?

Running search operations can be extremely difficult and many companies struggle to maintain their deployments. To achieve the best results at the lowest possible cost, companies need to efficiently manage both the infrastructure and data layers of operations. As operation scales, this becomes increasingly more difficult and expensive.

The infrastructure layer of operation includes: Elasticsearch/OpenSearch orchestration, deploying clusters, scaling resources, configuring security, provisioning and more.

The data layer includes: taking care of ingesting data, configuring and maintaining data structure, optimizing shards, avoiding latency, preventing incidents, improving performance and more.

Companies can either remain self-hosted, and hire internal experts to manage both layers of the search deployment, or turn to managed services. Managed services provide tools and support for managing the infrastructure layer, but not the data layer. There are also various consulting firms that offer assistance for internal teams based on manual processes.

As far as managed services go, the two main options in the field are, of course, Elastic (for Elasticsearch) and AWS (for OpenSearch).

In this blog we’ll go over what Elastic and AWS say they cover (and what they actually do), as well as what you can expect from consulting firms. 

Then we’ll dive into what Opster offers. 

In this blog we will not be covering considerations regarding choosing between Elasticsearch and OpenSearch because there is no material difference between them at this point. If you’re wondering whether to run self-hosted or look for a different solution, you can read about different hosting considerations in this blog

Opster VS. Managed Services VS. Consulting Services VS. Self-Managed

Self Managed
Consulting Services
Managed Services
Opster
Operational Burden
High

Every operation aspect is your sole responsibility
High

Though the consultant can cover operation aspects, every process is manual & time consuming
High

Though initial deployment is easy, everything not related to infrastructure is not covered
Low

AutoOps along with Opster’s support team keeps your burden to a bare minimum
Cost
Medium

Though you avoid paying external providers, funds are wasted on inefficient hardware and production crises
High

Internal teams must continue to actively manage the cluster and develop & maintain internal tools
High

Up to 3x the cost of underlying infrastructure and does not include internal teams’ wasted time and performance degradation
Low

Opster reduces hardware costs by optimizing resource utilization, ensuring stability and improving performance
Performance & Stability
Low

Internal tooling needed & requires developing internal Elasticsearch/ OpenSearch expertise
Low-Medium

Depends on the quality or the consultant’s expertise and tools
Low-Medium

You are responsible for optimizing performance and dependent on ticket-based support for the resolution of production & latency issues
High

Opster’s technology prevents and resolves issues and ensures peak performance and stability
Support
Internal
Not real-time

Based on manual processes
Limited

Ticket-based, does not cover the data layer
Real-time & preventive

Covers all layers of operation
Self Managed
Operational Burden
High

Every operation aspect is your sole responsibility
Cost
Medium

Though you avoid paying external providers, funds are wasted on inefficient hardware and production crises
Performance & Stability
Low

Internal tooling needed & requires developing internal Elasticsearch/ OpenSearch expertise
Support
Internal
Consulting Services
Operational Burden
High

Though the consultant can cover operation aspects, every process is manual & time consuming
Cost
High

Internal teams must continue to actively manage the cluster and develop & maintain internal tools
Performance & Stability
Low-Medium

Depends on the quality or the consultant’s expertise and tools
Support
Not real-time

Based on manual processes
Managed Services
Operational Burden
High

Though initial deployment is easy, everything not related to infrastructure is not covered
Cost
High

Up to 3x the cost of underlying infrastructure and does not include internal teams’ wasted time and performance degradation
Performance & Stability
Low-Medium

You are responsible for optimizing performance and dependent on ticket-based support for the resolution of production & latency issues
Support
Limited

Ticket-based, does not cover the data layer

What do managed services cover

Expectations VS. Reality

Users often start out self-managed before moving to managed services. The reality of managing ES/OS is usually not what they expected. 

Managed services structure their business in such a way that they do not take care of the application layer. They don’t provide proactive support, and they take no responsibility for the way you configure your data. This last part makes perfect sense, seeing as it would be out of scope for a managed service to be responsible for the data you want to put in your system. Their agenda is not to intervene with the way you decide to structure, configure and manage your data.

Self-Managed – Expectations
Self-Managed – Reality
Ensure that mission-critical applications are running at peak performance and stable
Generic monitoring tools are not enough to operate successfully
Keep costs low – by avoiding payments to outside providers for support and services
Skilled DevOps engineers are hard to find. They then need to develop & maintain internal expertise tooling for multiple technologies. Hardware & employee costs pile up
Ensure that mission-critical applications are running at peak performance and stable
Keep costs low – by avoiding payments to outside providers for support and services
Ensure that mission-critical applications are running at peak performance and stable
Keep costs low – by avoiding payments to outside providers for support and services

Then again, users often have unrealistic expectations of what they’ll get from managed services as well.

Managed Services – Expectations
Managed Services – Reality
Easy to start with & operate
Stability & performance are the responsibility of internal teams
Move ownership to the managed service and enjoy peak performance for the application
Support is limited to infrastructure and doesn’t cover application needs
Cost efficiency
Costs continually increase when trying to solve performance by adding hardware
Features added by the managed service will add great value
Most features go unused, as they are irrelevant to many use casesMost features go unused, as they are irrelevant to many use cases
Easy to start with & operate
Move ownership to the managed service and enjoy peak performance for the application
Cost efficiency
Features added by the managed service will add great value
Easy to start with & operate
Move ownership to the managed service and enjoy peak performance for the application
Cost efficiency
Features added by the managed service will add great value

The majority of managed services work with a ticket-based support system, which often entails long wait times and an inability to keep up with the dynamic demands of the system, not to mention being reliant on them when there are production crises. 

What does Elastic cover?

Elastic ECK or Elastic Cloud offer a convenient way to manage your Elasticsearch infrastructure. In addition, you get access to features on top of the managed service that can be used for specific use cases.

Many companies who turn to Elastic Cloud to manage their Elasticsearch deployment have high expectations of the service, hoping to transfer complete ownership of their deployment over to Elastic. However, Elastic themselves have a clear breakdown on their website regarding which elements of operations they cover, and which they don’t. They call this “shared responsibility”, meaning that some responsibilities remain solely on the customer.

Image from elastic.co

Elastic Cloud and ECK cover the infrastructure, orchestration and platform & networking aspects of deployments. Everything else is up to the customer: running operations, managing data and configuring security controls. Each company needs to make their own decisions regarding configuration and ascertain that their deployment can successfully meet their business needs.

Among the responsibilities that Elastic says they leave to the customer are some complicated and often confusing aspects of operations. Many companies end up struggling to successfully manage these elements on their own, especially as operations scale.

What does AWS OpenSearch Service cover?

AWS offers an OpenSearch managed service to easily deploy and manage OpenSearch infrastructure.

Similarly to Elastic, AWS also makes it clear that they do not operate or configure applications for their customers – the focus is on managing AWS infrastructure more efficiently and securely. While AWS can assist security and some capacity optimization, they do not support the data layer.  

Image from aws.amazon.com
Image from aws.amazon.com

What do consulting firms offer?

There are various consulting firms in the field who offer support for Elasticsearch/OpenSearch deployments, lending their expertise on both the infrastructure and data layers of operations. However, getting support from them is often a labor-intensive and time consuming process which relies on generic monitoring tools and manual actions. Support is often disconnected and can’t fully occur in real-time. 

What does Opster offer and when is it not a good fit?

What Opster offers

Opster’s products and support services cover both the infrastructure layer and the data layer of managing Elasticsearch & OpenSearch, with a focus on the data layer that users have always been left alone to handle. 

  • Automatic incident resolution and prevention.
  • Support powered by the AutoOps platform.
  • Optimized and improved performance.
  • Resource utilization and reduction of infrastructure costs.

Opster’s AutoOps detects issues, provides automatic resolution paths and optimizes resource utilization. The technology and tools perform automatic optimizations with the AutoOps operator, an on-prem service, which also enables Opster’s support team to interact with the system in real-time. 

When is Opster not a good fit?

If you’re running a very small, non-mission-critical installation, composed of 1-2 nodes, then the price of Opster’s subscription will probably not be up your alley. If your installation is mission-critical, you’ll find that working with Opster can actually reduce the cost of your Elasticsearch/OpenSearch deployment by reducing your hardware costs and ensuring peak performance.

Summary

No matter how you’re hosting your Elasticsearch or OpenSearch, you can benefit from Opster’s complete solutions. With an Opster AutoOps subscription, your database administration will be taken care of from start to finish. You’ll benefit from complete resolution of issues in your infrastructure & data layers, end-to-end support and constant optimization of your clusters. To learn more about Opster AutoOps, click here.

Skip to content