topic Re: Alerting on Recurring process restart in Alerting

Alerting on Recurring process restart

shifu — Thu, 14 Jul 2022 09:38:16 GMT

Hi,

We got a recurring process that restarts every hour and is executed for 30 min, is it possible to create an alert when this process hasn't restarted for 1 hour?

Thanks

Re: Alerting on Recurring process restart

Mizső — Thu, 14 Jul 2022 09:39:28 GMT

Hi Shifu,

Maybe you should use this feature: Process group availability monitoring and alerting | Dynatrace Docs

It is immediately opens a problem. If you would like to receive for example an e-mail notification after one hour about this issue you should:

1. Create a management zone for this single process (in the settings/preferences),

2. Create an alerting profile (in the settings / alerting) - set the availability alert after 60 mins and the others severities based on your requirements,

3. Create an e-mail problem notification (in the settings / integrarion) - use the created alerting profile above.

I hope this is your request.

Br, Mizső

Re: Alerting on Recurring process restart

Mizső — Wed, 13 Jul 2022 14:10:36 GMT

Hi Shifu,

I have just realized that my previous comment does not provide solution for your problem...sorry.

Br, Mizső

Re: Alerting on Recurring process restart

Mizső — Wed, 13 Jul 2022 15:08:56 GMT

Hi Shifu,

I have another idea, but it is not a full solution. You can use the events api the get information about events. You can count the process restart of the specific process in a spercific time frame. Please check the environment api v2 events api. You can find a get / events endpoint. On the DT GUI you can test it with a events.read api token.

I have made an exapmle. This curl can be use to fetch the a sepecific process restarts events count for the specific process group in the last 12 hours.

curl -X GET https://yourserver/e/yourenvironmentid/api/v2/events?from=now-12h&eventSelector=eventType%28%22PROCESS_RESTART%22%29&entitySelector=entityId%28%22PROCESS_GROUP_INSTANCE-yourprocessgroupid%22%29 -H "accept: application/json; charset=utf-8" -H "Authorization: Api-Token yourreadeventv2apitoken"

With the above curl you can create a script which one could run every hours (change the from now-1h).

The response first line is the events count see an example below (for smtpd process), rest of the lines are unnecessary for you:

response:
{
"totalCount": 21,
"pageSize": 100,
"events": [
{
"eventId": "-3274538997293409360_1657721040000",
"startTime": 1657721040000,
"endTime": 1657721050000,
"eventType": "PROCESS_RESTART",
"title": "Process restart",
"entityId": {
"entityId": {
"id": "PROCESS_GROUP_INSTANCE-yourprocessid",
"type": "PROCESS_GROUP_INSTANCE"
},
"name": "smtpd"
},
"properties": [
{
"key": "dt.event.group_label",
"value": "Process restart"
},
{
"key": "dt.event.impact_level",
"value": "Infrastructure"
}
],
"status": "CLOSED",
"entityTags": [
{
"context": "CONTEXTLESS",
"key": "X hosts",
"stringRepresentation": "X hosts"
},
{
"context": "CONTEXTLESS",
"key": "ProcessGroupName",
"value": "smtpd",
"stringRepresentation": "ProcessGroupName:smtpd"
}
],
"managementZones": [],
"underMaintenance": false,
"suppressAlert": false,
"suppressProblem": false,
"frequentEvent": false
},

It is not a full solution beasue you do not have notification about it. Maybe you can ingest back this metric to DT via the Metrics API, but I do not have experience with it.

I hope it is uesful for you.

Br, Mizső

Re: Alerting on Recurring process restart

shifu — Wed, 13 Jul 2022 15:17:14 GMT

Thank you Mizsö for you replies, i'll try some of your propositions, maybe one will fetch well.

Shifu

Re: Alerting on Recurring process restart

Yosi_Neuman — Thu, 14 Jul 2022 07:35:37 GMT

Hi @Mizső

Yep, your suggestion to ingest the information back to dynatrace with metric/ingest is working.

Attached a small AG extension (v1) that pulls the restart events according to supplied tag and it push the information back to dynatrace.

Here is how the UI is look like

And this is the ingest metric in dynatrace

HTH

Yos

Re: Alerting on Recurring process restart

Mizső — Mon, 18 Jul 2022 07:19:19 GMT

Yos,

Thank you very much!!!

Br, Mizső