Anomaly Detector - Disk monitoring AND condition

[AntonPineiro]

Hi,

I would like to create an Anomaly Detector rule to create a disk problem when two conditions being matched. It is important AND condition is done, not OR:

Idea would be checking 2 timeseries data to match those 2 conditions:

• Disk % higher than 80%.
• Disk space lower than X bytes.

We are using this DQL:

timeseries t_disk_percent = avg(dt.host.disk.used.percent), t_disk_used = avg(dt.host.disk.avail),
  matchesValue(entityAttr(dt.entity.disk, "entity.name"), "/opt")
}, by:{dt.entity.host, dt.entity.disk}, interval: 1m
| fieldsAdd `disk_used_percent` = arrayAvg(t_disk_percent)
| fieldsAdd `disk_used` = arrayAvg(t_disk_used)
| filter `disk_used_percent` > 80 AND `disk_used` < X

 Anomaly detector complains saying:

The query results in multiple metrics, but custom alerts only support single metric.

Do you have some idea in mind? Idea can be generating a new metric only when both conditions are applied. Something as "1" when both are matched, otherwise "0".

We cannot use Disk Edge because only OR conditions are allowed and we would like to have AND condition.

Do you have some recommendation? Are you using Anomaly Detector for that?

Best regards

[MaximilianoML]

Hello @AntonPineiro, how is going?

You can achieve this with the Anomaly Detector, but the key here is that the rule must evaluate a single resulting timeseries. That's why your current query is rejected: even though you apply an AND in the filter, the query still starts from two input metrics, and custom alerts expect one final signal. Try something like this bellow, this example I'm using 20GiB:

timeseries {
  disk_used_percent = avg(dt.host.disk.used.percent),
  disk_avail_bytes  = avg(dt.host.disk.avail)
},
filter: {
  matchesValue(entityAttr(dt.entity.disk, "entity.name"), "/opt")
},
by: { dt.entity.host, dt.entity.disk },
interval: 1m
| fieldsAdd alert_signal = if(
    disk_used_percent[] > 80 AND disk_avail_bytes[] < 21474836480,
    1,
    else: 0
  )
| fieldsKeep dt.entity.host, dt.entity.disk, alert_signal

Also, one small correction: dt.host.disk.avail is available space (No?), so I would rename that field to something like disk_avail_bytes to avoid confusion.

I hope it helps 😃

[AntonPineiro]

Hi,

Thank you for your fast feedback.  "disk_used_percent[] > 80" is not working fine. It means, no matter if you select "<" or ">", it is always considered as "true".

And it is not allowed in Anomaly detector:

Best regards

[sia_h]

It's failing because the timeframe field is no longer there.

Instead of line 15, do | fieldsRemove disk_used_percent, disk_avail_bytes to keep everything and only 1 array (alert_signal).