topic Alert on low disk space by using both disk space % and disk space bytes in Alerting

Alert on low disk space by using both disk space % and disk space bytes

RishiSuthar — Thu, 18 Jul 2024 19:27:38 GMT

Hi,

We are looking to monitor low disk space with the following setup:
alert if space <= 97% AND space <300 gig

Is this something that can be achieved within the alerting query?

Re: Alert on low disk space by using both disk space % and disk space bytes

Maheedhar_T — Fri, 19 Jul 2024 06:19:39 GMT

You can use new "Davis Anomaly Detector" app for this.
What you need is a query which checks both of the conditions.
In data explorer it looks something like this

and when you click on open with and open that with a notebook it gets converted to DQL and looks something like the below.

timeseries { free = avg(dt.host.disk.free), avail = avg(dt.host.disk.avail) }, by: { dt.entity.host }, filter: { in(dt.entity.host, classicEntitySelector("type(host),fromRelationship.isInstanceOf(type(HOST_GROUP),entityName.equals(\"XXXXXX\"))")) } | sort arrayAvg(free) desc | fieldsAdd entityName(dt.entity.host) | limit 20

Then again when you click open with you find something like this.

click on Davis anomaly detector and you will be redirected to the Anomaly detector Application where you can create the event.
Then you can see your query in the query section of the config and in the Customise parameters section this is what you have to give.

Here the query gives disk available % first and disk available (bytes) as second output hence we configure something like this.
3,322,122,547,200
Don't get confused with so many "," s.
Split it as
(3),(322,122,547,200) the first part is disk available is 3% and the second part is disk available is 322,122,547,200 bytes.
and we have set the alert to be triggered if the metric is below which means alert would be triggered if both of the conditions you mentioned above are met.

Hope this answers your query.

Regards,
@Maheedhar_T

Re: Alert on low disk space by using both disk space % and disk space bytes

Maheedhar_T — Fri, 19 Jul 2024 06:22:22 GMT

One thing I forgot to mention is that you have to convert your 300 gig to bytes to add it as threshold as the metric we are using above gives the disk available in bytes.

Also,
This is the only way I see ahead as I was not able to replicate the same using classic metric events.

Hope you're on Grail....
Happy querying @RishiSuthar

Re: Alert on low disk space by using both disk space % and disk space bytes

dlee1 — Wed, 25 Jun 2025 19:56:28 GMT

Hi,
I tried following your solution as I am interested in a similar setup, however when I get to setting up the anomaly detector, I get the following error with my query: "The query results in multiple metrics, but anomaly detectors only support single metric."

*I replaced the server name to "XXXXXX" in my screenshot, but when I put in an actual server name I get the error*

Does this functionality no longer work, or am I missing something?

Thanks.

Re: Alert on low disk space by using both disk space % and disk space bytes

rkdy — Wed, 05 Nov 2025 14:52:32 GMT

You could try something like this:

timeseries by:{dt.entity.disk, dt.entity.host}, {used_percent = avg(dt.host.disk.used.percent), used = avg(dt.host.disk.used), avail = avg(dt.host.disk.avail)}, filter:{matchesValue(entityAttr(type:"dt.entity.host", dt.entity.host, "hostGroupName"), "xxxxx")} | fieldsAdd entityName(dt.entity.disk) | fieldsAdd entityName(dt.entity.host) | fieldsAdd free_percent = 100 - used_percent[], free_space_gb = avail[] / (1024*1024*1024) | filter iAny(free_space_gb[] < 300) | fieldsAdd alert_level = if(isNull(free_percent[]), null, else:if(free_percent[] < 5, 2, else:if(free_percent[] < 10, 1, else:0))) | filter iAny(alert_level[] > 0) | fields dt.entity.disk, dt.entity.host, dt.entity.disk.name, dt.entity.host.name, alert_level, timeframe, interval, used_percent, used , avail, last_alert_level = arrayLast(alert_level), last_used_percent_display = round(arrayLast(used_percent), decimals:2), last_free_space_gb = round(arrayLast(free_space_gb), decimals:2) | fieldsAdd alert_severity = if(last_alert_level == 2, "CRITICAL", else:if(last_alert_level == 1, "WARNING", else:"OK")) | dedup dt.entity.disk