topic Re: Suppressing "Host gracefully shutdown" alerts during cluster autoscaling in Alerting

Suppressing "Host gracefully shutdown" alerts during cluster autoscaling

skhamitkar — Mon, 11 May 2026 11:52:52 GMT

Dear community,

We are currently receiving "Host gracefully shutdown" alerts that we consider false positives, as they are triggered by our standard cluster autoscaling process.

Since the "Alert on graceful host shutdown" setting is global, we cannot exclude specific cluster nodes. While we considered using Maintenance Windows, but MW is mainly targeted for planned activity and cluster uograde will be automatic so don't think it will be the good solution, I wanted to check if there is a more efficient or automated best practice for handling these alerts during autoscaling.

Could you please advise on the best way to suppress these specific notifications?

Re: Suppressing "Host gracefully shutdown" alerts during cluster autoscaling

AntonPineiro — Thu, 07 May 2026 08:49:17 GMT

Hi,

What do you want to do? You can switch off alerts in host level, if you want.

Question is, when do you want to be alerted?

Best regards

Re: Suppressing "Host gracefully shutdown" alerts during cluster autoscaling

parthsoni512 — Thu, 07 May 2026 14:52:06 GMT

Have you tried creating custom alerting profile and attaching it at host group level? you can configure alerting differently on host group level which takes precedence above cluster level.

Re: Suppressing "Host gracefully shutdown" alerts during cluster autoscaling

skhamitkar — Mon, 11 May 2026 11:15:20 GMT

So we are getting alerts when our cluster autoscalar triggers. As you might know during autoscaling process the node can be replaced by the new one, due to resource constraints. This is making panic situation in our operation center. Since autoscaling is pretty normal process we dont want alerts to be triggers durint autoscaling.

Re: Suppressing "Host gracefully shutdown" alerts during cluster autoscaling

AntonPineiro — Mon, 11 May 2026 13:05:56 GMT

Hi,

Looks like you can create an alert in process group instead of hosts, and alert only when 0 process are running for example.

It means, alert in process group instead of host individually.

Best regards

Re: Suppressing "Host gracefully shutdown" alerts during cluster autoscaling

ChadTurner — Mon, 11 May 2026 19:24:56 GMT

This is a common issue for entities being targeted for spin up/spin down - Much like Park my Cloud. The issue is that Dynatrace has no idea that the shutdown is desired even though it is graceful. yes you can turn off graceful shutdown alerts, but that could open up a risk in missed alerts. The best option is for the system to mark the host in Dynatrace with a tag. "Marked for Shutdown:Yes" then your alert profiles could incorporate the negate to alert on everything except entities tagged with "Marked for Shutdown:Yes"

Some other creative ideas is to set Maintenance windows for entities that power down in the off hours and define Business hours to prevent alerts on entities that power down outside Business Hours.

Re: Suppressing "Host gracefully shutdown" alerts during cluster autoscaling

Sam25 — Mon, 11 May 2026 19:56:10 GMT

Hi. 'Host gracefully shutdown' at global level will not work as expected and you will keep recving false positives in an env with auto scaling. you will have to come up with a customized solution utilizing to filter by tags, APs and host groups. you can also consider using 'missing data' feature in metric events/anomaly detection, which is useful for servers which go in hung state or where OA gets uninstalled. bottom line is, in your scenario, these alerts will need to be verified before forwarding to mass emails.

Re: Suppressing "Host gracefully shutdown" alerts during cluster autoscaling

skhamitkar — Tue, 12 May 2026 12:03:46 GMT

We currently use an autoscaler based on resource consumption. When demand increases, additional nodes are provisioned; when demand drops, those nodes are scaled down and removed. I’m curious how Kubernetes handles active workloads during these scale-down events (OneAgent process shut down gracefully or just killed). You must have dealt with this scenario before?

Re: Suppressing "Host gracefully shutdown" alerts during cluster autoscaling

skhamitkar — Tue, 12 May 2026 12:12:01 GMT