cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Alerting on Recurring process restart

shifu
Newcomer

Hi, 

 

We got a recurring process that restarts every hour and is executed for 30 min, is it possible to create an alert when this process hasn't restarted for 1 hour?

 

shifu_0-1657702319676.png

Thanks

6 REPLIES 6

Mizső
DynaMight Leader
DynaMight Leader

Hi Shifu,

 

Maybe you should use this feature:  Process group availability monitoring and alerting | Dynatrace Docs

It is immediately opens a problem. If you would like to receive for example an e-mail notification after one hour about this issue you should:

1. Create a management zone for this single process (in the settings/preferences),

2. Create an alerting profile (in the settings / alerting) - set the availability alert after 60 mins and the others severities based on your requirements,

3. Create an e-mail problem notification (in the settings / integrarion) - use the created alerting profile above.

 

I hope this is your request.

processgroup_availability.png

 

Br, Mizső

Certified Dynatrace Associate

Mizső
DynaMight Leader
DynaMight Leader

Hi Shifu,

 

I have just realized that my previous comment does not provide solution for your problem...sorry.

 

Br, Mizső

Certified Dynatrace Associate

Mizső
DynaMight Leader
DynaMight Leader

Hi Shifu,

 

I have another idea, but it is not a full solution. You can use the events api the get information about events. You can count the process restart of the specific process in a spercific time frame. Please check the environment api v2 events api. You can find a get / events endpoint. On the DT GUI you can test it with a events.read api token.

 

I have made an exapmle. This curl can be use to fetch the a sepecific process restarts events count for the specific process group in the last 12 hours.

curl -X GET https://yourserver/e/yourenvironmentid/api/v2/events?from=now-12h&eventSelector=eventType%28%22PROCESS_RESTART%22%29&entitySelector=entityId%28%22PROCESS_GROUP_INSTANCE-yourprocessgroupid%22%29 -H "accept: application/json; charset=utf-8" -H "Authorization: Api-Token yourreadeventv2apitoken"

 

With the above curl you can create a script which one could run every hours (change the from now-1h).

 

The response first line is the events count see an example below (for smtpd process), rest of the lines are unnecessary for you:

response:
{
"totalCount": 21,
"pageSize": 100,
"events": [
{
"eventId": "-3274538997293409360_1657721040000",
"startTime": 1657721040000,
"endTime": 1657721050000,
"eventType": "PROCESS_RESTART",
"title": "Process restart",
"entityId": {
"entityId": {
"id": "PROCESS_GROUP_INSTANCE-yourprocessid",
"type": "PROCESS_GROUP_INSTANCE"
},
"name": "smtpd"
},
"properties": [
{
"key": "dt.event.group_label",
"value": "Process restart"
},
{
"key": "dt.event.impact_level",
"value": "Infrastructure"
}
],
"status": "CLOSED",
"entityTags": [
{
"context": "CONTEXTLESS",
"key": "X hosts",
"stringRepresentation": "X hosts"
},
{
"context": "CONTEXTLESS",
"key": "ProcessGroupName",
"value": "smtpd",
"stringRepresentation": "ProcessGroupName:smtpd"
}
],
"managementZones": [],
"underMaintenance": false,
"suppressAlert": false,
"suppressProblem": false,
"frequentEvent": false
},

 

It is not a full solution beasue you do not have notification about it. Maybe you can ingest back this metric to DT via the Metrics API, but I do not have experience with it.

 

I hope it is uesful for you.

 

Br, Mizső

Certified Dynatrace Associate

Hi @Mizső 

Yep, your suggestion to ingest the information back to dynatrace with metric/ingest is working.

Attached a small AG extension (v1) that pulls the restart events according to supplied tag and it push the information back to dynatrace.

Here is how the UI is look like

Yosi_Neuman_0-1657783602115.png

And this is the ingest metric in dynatrace 

Yosi_Neuman_1-1657783952253.png

HTH

Yos 

 

 

dynatrace certificated professional - dynatrace master partner - Matrix Soft Ware Division - Israel

Yos,

 

Thank you very much!!! 

 

Br, Mizső

Certified Dynatrace Associate

shifu
Newcomer

Thank you Mizsö for you replies, i'll try some of your propositions, maybe one will fetch well.

 

Shifu