Monitor time pods spent in a phase

[marceloss1]

Is there a way to monitor the time a pod spent in a specific phase, say time spent in Pending phase? A team in our company recently got an issue where a pod was stuck in the pending phase. They want to see if there is a built-in metric to know this and create an alert. I've browsed this forum and other sites but can't find something that serves this purpose. Please advise on whether this is possible.

Thanks.

[PedroSantos · Accepted solution]

Hey @marceloss1 ,
You can get the pods with pending state as detailed here. 

builtin:kubernetes.pods:filter(and(or(eq(pod_status,Pending)))):splitBy("k8s.pod.name"):sort(value(auto,descending)):limit(20)

If this doesn't work for you, you can just search with 'builtin:kubernetes.pods' in Data Explorer and then split by whatever suits you from those results. Also there's a filter there for "Pending", could be you have none pending, try Running instead and see if it works.

Filtering for pod status (In this case Pending) will allow you to track and potentially set up alerts for pods that remain in the Pending state. Although I don't think (someone correct me if I'm wrong) that Dynatrace has a direct metric for time spent in a phase, you could always explore the option of creating custom alerts based on how long a pod remains Pending by combining this metric with a custom alert logic that triggers if the pod stays in this state beyond a certain duration.

Let me know if this helps you 🙂

[marceloss1]

Hi,

Thanks for the suggestion, right now we are testing something like you mentioned and we are positive it'll work for the other teams. Thanks for getting me on the right path.