cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

"No Pod ready" alerts during maintenance window

gurugogi
Guide

Hi All,

I have configured  maintenance window(Detect problems but don't alert) for k8s clusters using MZ, cluster name and tag for workloads, during maintenance window it triggers notifications(mail, slack) particularly for "No Pod ready" for some workloads, the workloads for which notifications are triggered are part of MZ only.

please let me know whether anyone faced this issue and any fixes you have applied.

gurugogi_0-1728915379187.png

 

4 REPLIES 4

Peter_Youssef
Champion

Hello @gurugogi 

Lets start from scratch

Under anomaly detection > Workload.

2024-10-14_21h56_17.png

As per the automatic detection 

2024-10-14_21h57_58.png

In case Custom thresholds are configured to receive alerts as an example 

2024-10-14_22h02_41.png

2024-10-14_22h04_07.png

2024-10-14_22h05_10.png

2024-10-14_22h06_14.png

  • So, to receive alert on a specific case of these highlighted cases the alerting condition should be met.
  • You have to check the K8s MZ rules properly to ensure all K8s related entities are listed and included in the intended MZ.
  • Check the alerting profiles settings and tags rules as well > problem notifications.
  • Final check will be the Maintenance window settings.

KR,

peter.

 

 

gurugogi
Guide

@Peter_Youssef  thanks for you explanation.

Already we have enabled these settings at workload level and we are getting the alerts but actual issue is problem notifications (mail and slack alerts) are getting triggered during maintenance window.

Above I shared snapshot of our maintenance window setup.

Particularly we receive few "No Pod ready" alerts on few workloads, though these workloads are part of 

Hi

I was also looking at using Maintenance Windows to block events from being send out to an external system but enabling the Kubernetes builtin alert thresholds.

The goal is to "see" Problems but no events as it could cause an event storm for our external systems.

gurugogi
Guide

With the above setup it is working, its not triggering slack or mail notifications now.
{Added tag for  workloads and included it in Maintenance windows 
type(CLOUD_APPLICATION),toRelationships.isClusterOfCa(type(KUBERNETES_CLUSTER),entityName.in("k8s cluster name")) }

gurugogi_0-1730098294271.png

 

Featured Posts