topic Re: Monitoring stopped on a shutdown host, disappears from dashboard in DQL

Monitoring stopped on a shutdown host, disappears from dashboard

m_laurie_az — Wed, 02 Apr 2025 07:12:36 GMT

There was a host entity that shutdown due to power issues and the agent stopped reporting data back from the host. Instead of it's availability state changing, it drops off the dashboard completely as if the query isn't looking for it anymore due to it not reporting any state data. Is the query wrong or something misconfigured?

Query:
timeseries availability = sum(dt.host.availability), by: { dt.entity.host, availability.state }, filter: { in(dt.entity.host, classicEntitySelector("type(host),tag(\"KeyA:ValueA\")")) OR in(dt.entity.host, classicEntitySelector("type(host),tag(\"KeyA:ValueB\")")) }
| fieldsAdd availability = arraySum(availability)
| sort availability asc
| fieldsAdd Host = arrayLast(splitString(entityName(dt.entity.host), "- "))
| limit 300

Re: Monitoring stopped on a shutdown host, disappears from dashboard

krzysztof_hoja — Tue, 20 May 2025 15:28:23 GMT

Query is correct, but this is nature of timeseries that when no data was reported we cannot expect any result. If you want to have a stable list of hosts regardless of data availability I suggest to start with fetch.dt.entites and join result with timeseries.

Full query would look like this:

fetch dt.entity.host | filter in(id, classicEntitySelector("type(host),tag(\"KeyA:ValueA\")")) OR in(id, classicEntitySelector("type(host),tag(\"KeyA:ValueB\")")) | fieldsAdd Host = arrayLast(splitString(entity.name, "- ")) | join [ timeseries availability = sum(dt.host.availability), by: { dt.entity.host, availability.state } | fieldsAdd availability = arraySum(availability) ], on:{left[id]==right[dt.entity.host]}, fields:{availability.state, availability}, executionOrder:leftFirst, kind:leftOuter | sort availability desc | limit 300

kind:leftOuter option is important here. If missing host will not be present at all.

Re: Monitoring stopped on a shutdown host, disappears from dashboard

Fin_Ubels — Wed, 21 May 2025 05:09:12 GMT

With the above, would you not still run into the same issue?

If you run fetch dt.entity.host over a period where the host hasn't been seen it won't return a result.

For example, if I query during the time my host was last see, on the 9th of May, it shows up:

If I query for hosts on the 10th of May, it doesn't show up because there weren't any:

Re: Monitoring stopped on a shutdown host, disappears from dashboard

krzysztof_hoja — Wed, 21 May 2025 08:51:29 GMT

1. For short recent timeframes my query can be used as is, because it take hours to retire host.

2. For longer timeframes, you can introduce own control on time for both queries:

fetch dt.entity.host, from:-1M | filter in(id, classicEntitySelector("type(host),tag(\"KeyA:ValueA\")")) OR in(id, classicEntitySelector("type(host),tag(\"KeyA:ValueB\")")) | fieldsAdd Host = arrayLast(splitString(entity.name, "- ")) | join [ timeseries availability = sum(dt.host.availability), by: { dt.entity.host, availability.state }, from:-24h | fieldsAdd availability = arraySum(availability) ], on:{left[id]==right[dt.entity.host]}, fields:{availability.state, availability}, executionOrder:leftFirst, kind:leftOuter | sort availability desc | limit 300

query is looking at all entities for past month and data only last 24h