Suppressing "Host gracefully shutdown" alerts during cluster autoscaling

[skhamitkar]

Dear community, 

We are currently receiving "Host gracefully shutdown" alerts that we consider false positives, as they are triggered by our standard cluster autoscaling process.

Since the "Alert on graceful host shutdown" setting is global, we cannot exclude specific cluster nodes. While we considered using Maintenance Windows, but MW is mainly targeted for planned activity and cluster uograde will be automatic so don't think it will be the good solution,  I wanted to check if there is a more efficient or automated best practice for handling these alerts during autoscaling.

Could you please advise on the best way to suppress these specific notifications?

[AntonPineiro]

Hi,

What do you want to do? You can switch off alerts in host level, if you want.

Question is, when do you want to be alerted? 

Best regards

[parthsoni512]

Have you tried creating custom alerting profile and attaching it at host group level? you can configure alerting differently on host group level which takes precedence above cluster level.

[skhamitkar]

So we are getting alerts when our cluster autoscalar triggers. As you might know during autoscaling process the node can be replaced by the new one, due to resource constraints. This is making panic situation in our operation center.  Since autoscaling is pretty normal process we dont want alerts to be triggers durint autoscaling. 

[AntonPineiro]

Hi,

Looks like you can create an alert in process group instead of hosts, and alert only when 0 process are running for example.

It means, alert in process group instead of host individually.

Best regards

[ChadTurner]

This is a common issue for entities being targeted for spin up/spin down - Much like Park my Cloud. The issue is that Dynatrace has no idea that the shutdown is desired even though it is graceful. yes you can turn off graceful shutdown alerts, but that could open up a risk in missed alerts. The best option is for the system to mark the host in Dynatrace with a tag. "Marked for Shutdown:Yes" then your alert profiles could incorporate the negate to alert on everything except entities tagged with "Marked for Shutdown:Yes"

Some other creative ideas is to set Maintenance windows for entities that power down in the off hours and define Business hours to prevent alerts on entities that power down outside Business Hours. 

[Sam25]

Hi. 'Host gracefully shutdown' at global level will not work as expected and you will keep recving false positives in an env with auto scaling. you will have to come up with a customized solution utilizing to filter by tags, APs and host groups. you can also consider using 'missing data' feature in metric events/anomaly detection, which is useful for servers which go in hung state or where OA gets uninstalled. bottom line is, in your scenario, these alerts will need to be verified before forwarding to mass emails.