Troubleshoot “Elasticsearch log queue is full” and “Elasticsearch log storing failed” in Managed Cluster

[mutaz]

Warning messages “Elasticsearch log queue is full” and “Elasticsearch log storing failed” occur when the attempted ingestion of logs exceeds the limits from the managed cluster side and when the Elasticsearch mount on a given node is out of disk space, this limit is not static, and is influenced by the number of nodes, storage, and CPU cores provided to each node.

From CMC events,  you can observe events like:

Elasticsearch log queue is full
Your Elasticsearch deployment requires scaling

Elasticsearch log storing failed
Check your Elasticsearch deployment state

As far as the details behind those event messages, this occurs when the queue for event ingestion is full. At that point, Elasticsearch is no longer capable of event ingestion. Unless lowering the ingest limit is an option, we generally recommend scaling the nodes by either increasing the CPU cores available to each node or adding extra nodes to the cluster. This approach should give a greater capacity for processing incoming log events:

• Adding extra nodes to the cluster
• Increasing the number of processors in each node.
• Optional: Lower the maximum ingest of log events per minute from CMC.

Alternatively, to reduce the processing of incoming log events and contribute to less saturation of the log queue within Elasticsearch as event ingestion is attempted.  You can reconfigure the base log filter to only ingest relevant logs in the log ingest rules instead of all logs.

Go to Settings --> Log Monitoring --> Log ingest rules

Note: You can check DDU consumption of log monitoring in your environment, which provides an approximate measure of the log sources sending the most logs, to help you control log ingesting rules and reduce them if possible.

Consumption -->  Davis data units --> Log