topic Why pod restart doesn't alert dynatrace in Alerting

Why pod restart doesn't alert dynatrace

gilles_djimi-ex — Thu, 01 Sep 2022 09:34:27 GMT

We running Dynatrace on a gke cluster and when pod restarts it is not shown in Dynatrace, even worst, it shows the old pod as unmonitored.

Hence here is my question, how can I make an alerte on Dynatrace on every pod restart. Just for information I'am running Dynatrace on managed version.

Thanks

Re: Why pod restart doesn't alert dynatrace

skrystosik — Mon, 30 Sep 2019 09:29:15 GMT

It's because as you don't restart pod but destroy instance and create new one (new ID assigned to OneAgent). As I understand you are using not full stack monitoring but application only monitoring?

Sebastian

Re: Why pod restart doesn't alert dynatrace

gilles_djimi-ex — Thu, 03 Oct 2019 14:55:26 GMT

No I'am using the fullstack monitoring and we are not destroying instances, it just that pod restart doesn't show on dynatrace properly it display a message that a pod is not monitored or that the signal was lost.

If I take an example: Let's say we have 3 runing process, and all the 3 are restarted, then all the new pod will be display in dynatrace while the old one are marked as unmonitored or "signal lost" tagged.

For example in the above image we can see that the old pod are seen as unmonitored or have a "lost signal" tag.

My question is how can we properly configure Dynatrace to properly show the pod information. Or is this normal and only used to see old pod metrics

Re: Why pod restart doesn't alert dynatrace

skrystosik — Thu, 03 Oct 2019 19:05:15 GMT

This is regular behaviour.

Sebastian

Re: Why pod restart doesn't alert dynatrace

Dant3 — Thu, 03 Oct 2019 22:08:10 GMT

We had the same inquery. We notice that the pod would die before a new one showup.

We configure a minimun of running process for that PG and a minimun of pods in OCP. So in the case of a new pod we get an alert that the PG is lower than it should be.

Re: Why pod restart doesn't alert dynatrace

gilles_djimi-ex — Fri, 04 Oct 2019 10:22:56 GMT

Can we change the normal behaviour so that old pod doesn't show up as unmonitored or tagged with "lost signal" ? Additionnaly is that possible to make Dynatrace makes an alert to any pod restart.