topic Re: "No Pod ready" alerts during maintenance window in Alerting

"No Pod ready" alerts during maintenance window

gurugogi — Wed, 16 Oct 2024 07:13:49 GMT

Hi All,

I have configured maintenance window(Detect problems but don't alert) for k8s clusters using MZ, cluster name and tag for workloads, during maintenance window it triggers notifications(mail, slack) particularly for "No Pod ready" for some workloads, the workloads for which notifications are triggered are part of MZ only.

please let me know whether anyone faced this issue and any fixes you have applied.

Re: "No Pod ready" alerts during maintenance window

Peter_Youssef — Mon, 14 Oct 2024 18:11:22 GMT

Hello @gurugogi

Lets start from scratch

Under anomaly detection > Workload.

As per the automatic detection

In case Custom thresholds are configured to receive alerts as an example

So, to receive alert on a specific case of these highlighted cases the alerting condition should be met.
You have to check the K8s MZ rules properly to ensure all K8s related entities are listed and included in the intended MZ.
Check the alerting profiles settings and tags rules as well > problem notifications.
Final check will be the Maintenance window settings.

KR,

peter.

Re: "No Pod ready" alerts during maintenance window

gurugogi — Tue, 15 Oct 2024 11:07:12 GMT

@Peter_Youssef thanks for you explanation.

Already we have enabled these settings at workload level and we are getting the alerts but actual issue is problem notifications (mail and slack alerts) are getting triggered during maintenance window.

Above I shared snapshot of our maintenance window setup.

Particularly we receive few "No Pod ready" alerts on few workloads, though these workloads are part of

Re: "No Pod ready" alerts during maintenance window

Stephen1_Lee — Thu, 17 Oct 2024 20:15:13 GMT

I was also looking at using Maintenance Windows to block events from being send out to an external system but enabling the Kubernetes builtin alert thresholds.

The goal is to "see" Problems but no events as it could cause an event storm for our external systems.

Re: "No Pod ready" alerts during maintenance window

gurugogi — Mon, 28 Oct 2024 06:54:08 GMT

With the above setup it is working, its not triggering slack or mail notifications now.
{Added tag for workloads and included it in Maintenance windows
type(CLOUD_APPLICATION),toRelationships.isClusterOfCa(type(KUBERNETES_CLUSTER),entityName.in("k8s cluster name")) }