topic Re: Auto-adaptive threshold/Seasonal baseline Metric event with threshold in Alerting

Auto-adaptive threshold/Seasonal baseline Metric event with threshold

kobkob — Wed, 25 Sep 2024 13:48:11 GMT

Hi,

Is there any way to create Auto-adaptive threshold/Seasonal baseline Metric events that will be triggered only if the value is greater than X?

Thanks

Re: Auto-adaptive threshold/Seasonal baseline Metric event with threshold

PacoPorro — Wed, 25 Sep 2024 14:22:55 GMT

Do you mean a mix of thresholds and baselines?
If the baseline event does not reach threshold X?

Re: Auto-adaptive threshold/Seasonal baseline Metric event with threshold

kobkob — Wed, 25 Sep 2024 14:44:58 GMT

@PacoPorro exactly

Re: Auto-adaptive threshold/Seasonal baseline Metric event with threshold

AntonPineiro — Thu, 26 Sep 2024 09:27:45 GMT

Hi,

Create both? Auto-adaptive + static threshold?

Best regards

Re: Auto-adaptive threshold/Seasonal baseline Metric event with threshold

PacoPorro — Thu, 26 Sep 2024 11:21:12 GMT

AFAIK this is not possible, probably you can check if you can implement something using slos

Re: Auto-adaptive threshold/Seasonal baseline Metric event with threshold

kobkob — Sun, 29 Sep 2024 06:39:46 GMT

@AntonPineiro Its not an option since the static threshold for one service is not relevant for another, and I have thousands of microservices.

Re: Auto-adaptive threshold/Seasonal baseline Metric event with threshold

kobkob — Tue, 01 Oct 2024 06:22:13 GMT

I saw this in the metric event settings:

Anyone know what exactly it affects?

Re: Auto-adaptive threshold/Seasonal baseline Metric event with threshold

PacoPorro — Tue, 01 Oct 2024 11:08:58 GMT

The threshold, whether static or baseline, used for monitoring the metric.

Re: Auto-adaptive threshold/Seasonal baseline Metric event with threshold

kobkob — Tue, 01 Oct 2024 11:27:09 GMT

@PacoPorro thanks for your answer. Can you please elaborate?

Re: Auto-adaptive threshold/Seasonal baseline Metric event with threshold

Peter_Youssef — Tue, 01 Oct 2024 11:49:12 GMT

Hello @kobkob

Let's agree that:

Auto-adaptive baselining can be configured to trigger alert as per the automatic baselining and reference periods,...etc.
Static-threshold which is decided and provided by the app owner to receive alerts upon specific value violation to take an action.
Seasonal thresholds can be configured as well to pickup the violations which haven't been detected either through automatic baselining and static-based.
Common case for seasonal-baselining: marketing campaigns where excessive number of purchase, payment, delivery ... etc requests are processed and executed by specific services, so you can create a metric events and make as info if you need to be notified, or if you need to have an alert created keep the event type as "custom alert"

As a conclusion

You can create multiple metric events and choose where to receive alerts or not based on the actual use case.
You can create multiple metric events, but ensure there is no conflict in the configurations to receive the accurate alerts.
You can name the metric events if required ex: Exceeding processed requests during Black Friday.
If you have a request attribute configured for example to fetch purchased amounts or other parameter of interest > create multidimensional analysis view > create a metric for example if the total purchased amounts exceeds 1M so you will be notified during the seasonal baselining.
To create metric event: Manage > Settings > Anomaly Detection > Metric events, and simply configure it in meaningful way to be notified and take prober actions.

Feel free to review Seasonal Baselining

KR,

Peter.

Re: Auto-adaptive threshold/Seasonal baseline Metric event with threshold

ChadTurner — Tue, 01 Oct 2024 11:50:21 GMT

I recommend using the Davis Anomaly Detector in the new layout as it will not incur any limits etc on your allotment of metric events. Assuming your metric does not begin with ext:

Re: Auto-adaptive threshold/Seasonal baseline Metric event with threshold

rastislav_danis — Tue, 01 Oct 2024 14:11:48 GMT

maybe you can try partition in metric selector to apply static threshold and then use baseline, f.e. for cpu util >6% (i added default to fill the gaps with data):

builtin:host.cpu.usage:splitBy():auto:partition("cpuhigh",value("gt6",gt(6))):default(6)

Re: Auto-adaptive threshold/Seasonal baseline Metric event with threshold

kobkob — Sun, 06 Oct 2024 12:16:29 GMT

@rastislav_danis I think this is what I was looking for! Ill defently try it and update if it worked. Thanks a lot!