topic Re: Show Container name on Kubernetes container restart alerts/problem cards in Alerting

Show Container name on Kubernetes container restart alerts/problem cards

gurugogi — Wed, 25 Sep 2024 07:26:42 GMT

Hi All,

I have enabled alerting for container restart at anomaly detection-> Workload, it is creating alerts/Problem card when any container gets restarted, it is printing workload name in alerts but I want the particular container name to be printed in alerts (container name which got restarted.).

I tried with custom alerts(metric events), auto-tags etc looks like its not working, please guide if anyone has implemented this by modifying message body in mail alerts/slack alerts etc.

Thanks,
Guru Gogi

Re: Show Container name on Kubernetes container restart alerts/problem cards

gurugogi — Thu, 03 Oct 2024 10:45:05 GMT

anyone tried ?

Re: Show Container name on Kubernetes container restart alerts/problem cards

Mizső — Thu, 03 Oct 2024 11:01:19 GMT

Hi @gurugogi,

Have you tride this metric experession in the metric event?

Then you can use the container dimension in the metric event:

I hope it helps.

Best regards,

Mizső

Re: Show Container name on Kubernetes container restart alerts/problem cards

gurugogi — Thu, 03 Oct 2024 11:34:04 GMT

@Mizső thanks for reply.
I tried below expression and its triggering alerts for container restarts.
builtin:kubernetes.container.restarts
:splitBy("k8s.container.name","k8s.pod.name","k8s.cluster.name","k8s.namespace.name")
:count:sort(value(avg,descending))

I have not sleeted any MZ for this metric event so it is creating Problems in default alerting profile but I want the Problems to be grouped under specific alerting profiles tagged to specific

Re: Show Container name on Kubernetes container restart alerts/problem cards

gurugogi — Thu, 03 Oct 2024 11:36:29 GMT

I want to use this single "Metric event" for all k8s clusters alerting, particularly we use slack notifications, so I want some particular container restart should be grouped under that particular alerting profile.

Re: Show Container name on Kubernetes container restart alerts/problem cards

gurugogi — Thu, 03 Oct 2024 11:45:02 GMT

Only option is to create "Metric event"/"Custom alert" for each K8S cluster and MZ filter ?

Re: Show Container name on Kubernetes container restart alerts/problem cards

Peter_Youssef — Tue, 08 Oct 2024 12:55:19 GMT

Hello @gurugogi

let's agree that to get notified > problem alerting profile should be Created and hence > Management zone rules should be configured and customized to include all related entities.
The point here is to ensure the K8s configured management zone contains all K8s related entities and then you can easily create metrics and receive alerts upon the specified conditions.

For the K8s related anomaly detections, feel free to customize it as per the K8s admin inputs

BR,

Peter

Re: Show Container name on Kubernetes container restart alerts/problem cards

GosiaMurawska — Thu, 19 Dec 2024 09:57:06 GMT

Hi @gurugogi! Did you manage to solve this one? 🤔

Re: Show Container name on Kubernetes container restart alerts/problem cards

gurugogi — Thu, 19 Dec 2024 10:49:33 GMT

Hi Gosia,

Yes I solved it through metric expression

Expression:
builtin:kubernetes.container.restarts
:filter(and(eq("k8s.cluster.name","cluster-name"),eq("k8s.namespace.name","ns")))
:splitBy("k8s.cluster.name","k8s.pod.name","k8s.container.name","k8s.namespace.name")
:sort(value(auto,descending))

Used above expression in Metric events(custom alert)
Descrption:
container Name:{dims:k8s.pod.name}-{dims:k8s.container.name}
Cluster Name:{dims:k8s.cluster.name}
Namespace:{dims:k8s.namespace.name}