"No Pod ready" alerts during maintenance window

[gurugogi]

Hi All,

I have configured  maintenance window(Detect problems but don't alert) for k8s clusters using MZ, cluster name and tag for workloads, during maintenance window it triggers notifications(mail, slack) particularly for "No Pod ready" for some workloads, the workloads for which notifications are triggered are part of MZ only.

please let me know whether anyone faced this issue and any fixes you have applied.

[Peter_Youssef]

Hello @gurugogi 

Lets start from scratch

Under anomaly detection > Workload.

As per the automatic detection 

In case Custom thresholds are configured to receive alerts as an example 

• So, to receive alert on a specific case of these highlighted cases the alerting condition should be met.
• You have to check the K8s MZ rules properly to ensure all K8s related entities are listed and included in the intended MZ.
• Check the alerting profiles settings and tags rules as well > problem notifications.
• Final check will be the Maintenance window settings.

KR,

peter.

[gurugogi]

@Peter_Youssef  thanks for you explanation.

Already we have enabled these settings at workload level and we are getting the alerts but actual issue is problem notifications (mail and slack alerts) are getting triggered during maintenance window.

Above I shared snapshot of our maintenance window setup.

Particularly we receive few "No Pod ready" alerts on few workloads, though these workloads are part of 

[Stephen1_Lee]

Hi

I was also looking at using Maintenance Windows to block events from being send out to an external system but enabling the Kubernetes builtin alert thresholds.

The goal is to "see" Problems but no events as it could cause an event storm for our external systems.

[gurugogi · Accepted solution]

With the above setup it is working, its not triggering slack or mail notifications now.
{Added tag for  workloads and included it in Maintenance windows 
type(CLOUD_APPLICATION),toRelationships.isClusterOfCa(type(KUBERNETES_CLUSTER),entityName.in("k8s cluster name")) }