cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Openshift: Getting the delta value of builtin:cloud.kubernetes.pod.containerRestarts

travis_ottelien
Contributor

Hi Dynatrace Community. I'm wondering if anybody has figured out how to get a delta value for the builtin:cloud.kubernetes.pod.containerRestarts metric. A few of us are looking to alert on this value but since it's always pulling in the absolute value we find it difficult to set up alerting here. We can use the 'Code' section if there's a way to calculate this delta value ourselves else we're wondering if this can be added to a roadmap.

 

When looking at Container Restarts within Dynatrace, we see a huge incremental line (count of 600+) so we know there's an issue with this specific event but having a "container restart" value doesn't always mean there's an issue currently. If a container restarts 10 times in 1 hour due to a connection issue and then re-establishes it's connection, it'll run healthy but the the container restart value for that pod will still be 10 until the pod itself is recycled. If the pod continues to run for another 24 hours after that and restarts five more times, the value will show up as 15.

 

What we'd like to get instead is the delta value. If we look at the time period when the pod was restarting we should see it going from 1 to 10 but when it stabilizes we'd like to see that pod restart value go zero so when it starts restarting again we can see it go from 1 to 5 instead of it going from 10 to 15 (basically by taking the current pod restart count and subtracting it from the previous value).

 

By having this, we hope to add custom alerting to say if the restart value is greater than 0 for a 10 minute period to create an alert. Right now we'd have to set it as a ceiling (over 10 restarts, for example) and then to ensure we don't get false alerts after a resolution we'd need to manually destroy the pod so a new one comes up in order to "reset" the container restart value adding extra toil.  

 

Thanks!

Travis Ottelien
1 REPLY 1

travis_ottelien
Contributor

I was thinking about this more and we'd never see it go from '1 to 10' or '1 to 5' when doing delta (unless somebody is running more than one container. If running one container it would always fluxuate between 0 and 1. But everything else still stands.

Travis Ottelien