Kubernetes Alerting

[Vikas_g1997]

Hi Team,

We have a customer requirement to set up alerting for specific Kubernetes lifecycle events. Specifically, we need alerts for:

• Lifecycle events such as pending, running, or successful state changes.
• Scaling events or pod restarts/failures with the ability to isolate or find logs just before these events.

I've reviewed the documentation but couldn’t find relevant metrics for these lifecycle states in Dynatrace. Could anyone please advise if there’s a way to achieve this, or suggest any alternative approach?

[Peter_Youssef]

Hello @Vikas_g1997 

Regarding the raised points:

1. Lifecycle events such as pending, running, or successful state changes.
2. Scaling events or pod restarts/failures with the ability to isolate or find logs just before these events.

Suggestion:

• Setup Issue-tracking for releases under cloud automation and connect the target K8s endpoint. 
• Proceed with metric events either through the metric key or selector.
• Alerting related concerns customize the anomaly detection for cluster, namespace, workload.

Any event will be triggered will be displayed through the monitored entity UI and you create the required alerting based on the event details / Logs 

Hoping it helps.

BR,

Peter

[ChadTurner · Accepted solution]

You can also create a custom alert via the Settings>Anomaly Detection>Metric Events and toss in your metric selector for the Data Explorer. Granted my example is looking at all phases but you can take that and adjust it as you see fit to be as specific or as generic as desired for alerting. 

Rule of thumb is if you can make it in the Data Explorer, you can make it alert by way of metric events. There is also the Davis Anomaly Detector that you can leverage if you have shifted to the new Dynatrace Layout. 

[Vikas_g1997]

Hi Team,

Thank you for your valuable input.

As per the customer's requirements, they are looking for alerts related to Pod restarts and the series of events that triggered these restarts. However, upon checking, I couldn't find a specific metric directly related to Pod restarts. Could you please suggest how we can achieve this monitoring effectively?

[p_devulapalli]

@Vikas_g1997 - You can make use of the builtin:kubernetes.container.restarts as well, below should give you the container restart count

builtin:kubernetes.container.restarts:splitBy(k8s.namespace.name,k8s.workload.kind,k8s.workload.name):sum:default(0.0)

Take a look at the below doc

https://docs.dynatrace.com/docs/platform-modules/infrastructure-monitoring/container-platform-monito...

[Vikas_g1997]

Hi @p_devulapalli ,

Thank you so much for the suggestion; I really appreciate it. However, I have a doubt—are Pod restarts and container restarts the same? Also, is there a way to identify the series of events that triggered these restarts?

[p_devulapalli]

@Vikas_g1997 , though there is some relationship between pod restarts and container restarts , they are not the same (one pod might have multiple containers) , but there is no direct metric to alert on pod restart that I know of. One way of explaining it is when we say a pod restarts in Kubernetes the existing pod’s containers are restarted according to the pod’s restart policy, so there is a bit of correlation between pod and container restarts 

Pod events can be viewed from under the even events tab if you are using the kubernetes app . The events written here are similar to what you would see in events when you do a kubectl describe pods as an example .