topic Re: Need opinion in choosing type of custom events to publish in a specific scenario in Alerting

Need opinion in choosing type of custom events to publish in a specific scenario

DineshBhat — Tue, 07 Sep 2021 14:00:58 GMT

Hello Team,

We do run Chaos engineering tests on the applications. The applications are currently publishing the metrics to Dynatrace. We want to generate custom events each time when Chaos experiments are run on the target application / host. So that we know Chaos experiments were run on the specific target and hence we know the reason for variation in the specific application metrics (which were impacted through chaos tests).

The Document: https://www.dynatrace.com/support/help/dynatrace-api/environment-api/events/post-event/#expand-the-element-can-hold-these-values-1309 Describes type of events supported by Dynatrace.

We want to differentiate the chaos experiment events at Dynatrace using specific event type name (example chaos-experiment).

I need your suggestion on selecting the type of events to choose in the above mentioned scenario. My opinion is that, "CUSTOM_ANNOTATION" and "CUSTOM_INFO" events might suite well in our case.
Also, I see "GET" and "POST" APIs supported for Events. But, in case of running Chaos experiments we may plan the test completion in advance but, we may have to remediate the chaos experiment earlier than planned end time as well. Hence, on sending Chaos experiment as an Event in Dynatrace while starting the Chaos experiment, we can set approximate endtime of experiment as event endtime. But, how to update the existing event in case if the Chaos experiment was to be remediated before the earlier estimated end time? We need to update the existing event.

Thanks,

Dinesh

Re: Need opinion in choosing type of custom events to publish in a specific scenario

awahab — Fri, 27 Aug 2021 20:33:39 GMT

You cannot retract or update the event for the event types you mentioned above. For problem events it can be closed or extended. But info events cannot be updated later.

The documentation also states that "The time of event closure is already known and the event IDs are returned instantly."

Re: Need opinion in choosing type of custom events to publish in a specific scenario

ChadTurner — Fri, 27 Aug 2021 20:35:53 GMT

Thats a bummer, I would recommend putting in a RFE and maybe the functionality will be added in to allow you to edit the events later on. Worse case they will say the RFE is not planned.

Re: Need opinion in choosing type of custom events to publish in a specific scenario

DineshBhat — Mon, 30 Aug 2021 11:28:30 GMT

Yes @ChadTurner . I will need to request for Feature.

@awahabrunning Chaos engineering tests in SaaS product pretty common now. What's your recommendation/suggestion on letting people know that the variation in the metric values were expected while running the Chaos engineering tests?

Re: Need opinion in choosing type of custom events to publish in a specific scenario

awahab — Wed, 01 Sep 2021 15:29:14 GMT

That would be a info event on top of the first event. So the latest event have the new timing.