topic Re: Kubernetes Alerting in Alerting

Kubernetes Alerting

Vikas_g1997 — Mon, 14 Apr 2025 06:51:12 GMT

Hi Team,

We have a customer requirement to set up alerting for specific Kubernetes lifecycle events. Specifically, we need alerts for:

Lifecycle events such as pending, running, or successful state changes.
Scaling events or pod restarts/failures with the ability to isolate or find logs just before these events.

I've reviewed the documentation but couldn’t find relevant metrics for these lifecycle states in Dynatrace. Could anyone please advise if there’s a way to achieve this, or suggest any alternative approach?

Re: Kubernetes Alerting

Peter_Youssef — Thu, 14 Nov 2024 09:54:24 GMT

Hello @Vikas_g1997

Regarding the raised points:

Lifecycle events such as pending, running, or successful state changes.
Scaling events or pod restarts/failures with the ability to isolate or find logs just before these events.

Suggestion:

Setup Issue-tracking for releases under cloud automation and connect the target K8s endpoint.
Proceed with metric events either through the metric key or selector.
Alerting related concerns customize the anomaly detection for cluster, namespace, workload.

Any event will be triggered will be displayed through the monitored entity UI and you create the required alerting based on the event details / Logs

Hoping it helps.

BR,

Peter

Re: Kubernetes Alerting

ChadTurner — Thu, 14 Nov 2024 12:48:03 GMT

You can also create a custom alert via the Settings>Anomaly Detection>Metric Events and toss in your metric selector for the Data Explorer. Granted my example is looking at all phases but you can take that and adjust it as you see fit to be as specific or as generic as desired for alerting.

Rule of thumb is if you can make it in the Data Explorer, you can make it alert by way of metric events. There is also the Davis Anomaly Detector that you can leverage if you have shifted to the new Dynatrace Layout.

Re: Kubernetes Alerting

Vikas_g1997 — Sun, 17 Nov 2024 07:40:03 GMT

Hi Team,

Thank you for your valuable input.

As per the customer's requirements, they are looking for alerts related to Pod restarts and the series of events that triggered these restarts. However, upon checking, I couldn't find a specific metric directly related to Pod restarts. Could you please suggest how we can achieve this monitoring effectively?

Re: Kubernetes Alerting

p_devulapalli — Sun, 17 Nov 2024 10:48:45 GMT

@Vikas_g1997 - You can make use of the builtin:kubernetes.container.restarts as well, below should give you the container restart count

builtin:kubernetes.container.restarts:splitBy(k8s.namespace.name,k8s.workload.kind,k8s.workload.name):sum:default(0.0)

Take a look at the below doc

https://docs.dynatrace.com/docs/platform-modules/infrastructure-monitoring/container-platform-monitoring/kubernetes-monitoring/alert-on-kubernetes-issues#workload-metric-dql-expressions

Re: Kubernetes Alerting

Vikas_g1997 — Sun, 17 Nov 2024 11:11:49 GMT

Hi @p_devulapalli ,

Thank you so much for the suggestion; I really appreciate it. However, I have a doubt—are Pod restarts and container restarts the same? Also, is there a way to identify the series of events that triggered these restarts?

Re: Kubernetes Alerting

p_devulapalli — Sun, 17 Nov 2024 23:25:58 GMT

@Vikas_g1997 , though there is some relationship between pod restarts and container restarts , they are not the same (one pod might have multiple containers) , but there is no direct metric to alert on pod restart that I know of. One way of explaining it is when we say a pod restarts in Kubernetes the existing pod’s containers are restarted according to the pod’s restart policy, so there is a bit of correlation between pod and container restarts

Pod events can be viewed from under the even events tab if you are using the kubernetes app . The events written here are similar to what you would see in events when you do a kubectl describe pods as an example .

Re: Kubernetes Alerting

ve00685603 — Thu, 10 Apr 2025 23:45:15 GMT

Hello Phani, thanks for the details notes. Wondering if there's an opportunity to setup an alert based on the pod not switching from "initialized" state to "Ready" state. So that, we will get a count of how many number of time pods not switched here. Pls advise. Thanks

Re: Kubernetes Alerting

p_devulapalli — Fri, 11 Apr 2025 02:04:29 GMT

@ve00685603 You might want to take a look at the "currentCondition" field which seems to include some information you are after and see if it helps when setting up the alerts

fetch dt.entity.cloud_application_instance | fieldsAdd lifetime,nodeName,desiredContainersCount,entity.type,runningContainersCount,entity.detected_name,id,namespaceName,internalIpAddresses,containerRestartCount,currentCondition,workloadName