Couldn t schedule ML memory update – node might be shutting down – How to solve this Elasticsearch error

Opster Team

Aug-23, Version: 6.8-8.9

Before you dig into reading this guide, have you tried asking OpsGPT what this log means? You’ll receive a customized analysis of your log.

Try OpsGPT now for step-by-step guidance and tailored insights into your Elasticsearch operation.

Briefly, this error occurs when Elasticsearch tries to schedule a Machine Learning (ML) memory update, but the node might be in the process of shutting down. This could be due to insufficient resources, network issues, or a planned shutdown. To resolve this, ensure the node has enough resources (CPU, memory, disk space), check the network connectivity between nodes, and verify if the node is in the process of a planned shutdown. If it’s a recurring issue, consider increasing the node’s resources or adjusting the ML job’s memory requirements.

For a complete solution to your to your search operation, try for free AutoOps for Elasticsearch & OpenSearch . With AutoOps and Opster’s proactive support, you don’t have to worry about your search operation – we take charge of it. Get improved performance & stability with less hardware.

This guide will help you check for common problems that cause the log ” Couldn’t schedule ML memory update – node might be shutting down ” to appear. To understand the issues related to this log, read the explanation below about the following Elasticsearch concepts: plugin, memory, node.

Log Context

Log “Couldn’t schedule ML memory update – node might be shutting down” classname is MlMemoryTracker.java.
We extracted the following from Elasticsearch source code for those seeking an in-depth context :

                );
                threadPool.executor(MachineLearning.UTILITY_THREAD_POOL_NAME)
                    .execute(() -> refresh(clusterService.state().getMetadata().custom(PersistentTasksCustomMetadata.TYPE); listener));
                return true;
            } catch (EsRejectedExecutionException e) {
                logger.warn("Couldn't schedule ML memory update - node might be shutting down"; e);
            }
        }

        return false;
    }

 

How helpful was this guide?

We are sorry that this post was not useful for you!

Let us improve this post!

Tell us how we can improve this post?