Elasticsearch shard per node is 80% of maximum limit

[shalinikumar]

Hi Team ,

We can see warning event for our Dynatrace cluster node.

The warning message is :

The number of Elasticsearch shard per Node is at 80% of maximum limit.

Elasticsearch no longer assign shards to nodes or create new indices . To precent this scale your cluster or reduce number of shards

We have had a look in /opt/dynatrace-managed/elasticsearch/indices 

We could see that for today's date the bytes is 20480 and index is not created. But previous dates the byte size is in 2 digit only.

What could we check here ?

Note - We have only 1 cluster node available in this environment  and no new configuration changes done recently.

Thanks,

Shalini.

[Mohamed_Hamdy]

Hello @shalinikumar ,

As this environment is running on a single cluster node, the shard capacity is limited and may lead to such warnings once the threshold is approached.

I would recommend to check the below on the cluster node:
curl -s "http://127.0.0.1:9200/_health_report/shards_capacity?pretty"
curl -s "http://127.0.0.1:9200/_cat/shards?v"
curl -s "http://127.0.0.1:9200/_cat/indices?v&s=index"
curl -s "http://127.0.0.1:9200/_cluster/settings?include_defaults=true&pretty"

These will help confirm the current shard usage versus the maximum limit and identify the contributing indices.

Recommendation:
- The most effective long-term solution is to scale the cluster by adding additional nodes.
- Alternatively, reducing data retention (if applicable) can help limit shard growth (check with Dynatrace support team).
- Increasing the shard limit parameter (cluster.max_shards_per_node) is possible but should only be considered as a temporary workaround (check with Dynatrace support team).

[rastislav_danis]

this is strange, our lab managed shows same message, but when i run 1st command:

 curl -s "http://127.0.0.1:9200/_health_report/shards_capacity?pretty"
{
"cluster_name" : "21cbc799-f7f0-4801-ac78-6cb045b48cd2",
"indicators" : {
"shards_capacity" : {
"status" : "green",
"symptom" : "The cluster has enough room to add new shards.",
"details" : {
"data" : {
"max_shards_in_cluster" : 3000
},
"frozen" : {
"max_shards_in_cluster" : 3000
}
}
}
}
}