PRO TIP: Alert on OneAgent running outdated version

[mosharref_hossn]

EDIT: 

Davis Anomaly Detected rule ended up generating too many alerts since it checks often. 

We moved this query to Workflow and run it once a week to send a notification via Send Email action. 

*****

Our team needed a way to alert when the OneAgent fails to upgrade to the latest available version and running an outdated version.

To address this, we created the following DQL query to be used with Davis Anomaly Detection.

Notifications can be configured using a Management Zone in combination with an Alert Profile.

Alternatively, tags can also be used to help route alerts to the appropriate team.

We also needed to exclude hosts that could not be upgraded due to unsupported OS version (lookup sub-query). 

You can exclude the subquery if that is not required. 

// Fetch hosts with installed agent versions and sort asc by Installer Version
// Sort order will help filter out any hosts that are not running the latest version in next sub-query
fetch dt.entity.host
| filter isNotNull(installerVersion) 
| fields id, entity.name, installerVersion, lifetime
// Coverting lifetime[end] value to "timestamp" enables the user to create a timeseries metric at the end to run in Davis Anomaly Detection
| fieldsAdd timestamp=toTimestamp(lifetime[end])
| fieldsRemove lifetime
// NOTE: sorted in ascending order to get oldest agent versions
| sort installerVersion asc
| limit 30
| summarize {installerVersion=max(installerVersion)}, by:{entity.name, id, timestamp}
 
// Fetch hosts with installed agent version, sort desc by Installer Version and return matching host IDs
| join [
fetch dt.entity.host
| filter isNotNull(installerVersion)
| fields  installerVersion, id
// NOTE: sorted in descending order to get latest agent versions
| sort installerVersion desc
| limit 30
], on:{id}, kind: leftOuter, fields:{latestAgentVersion=installerVersion}
// Filter out hosts matching the latest Agent version
| filter isNull(latestAgentVersion)
| fieldsRemove latestAgentVersion
 
// Fetch logs & lookup hosts with failed installation due to unsupported platform (dynamically)
// And exclude matched hosts.
| lookup [
    fetch logs, from:now()-48h
    | filter matchesValue(log.source, "Windows Application Log") and loglevel == "ERROR"
    | filter contains(content, "unsupported platform") and contains(content, "Dynatrace OneAgent")
    | dedup dt.entity.host
    | sort timestamp desc
| fields dt.entity.host
], sourceField: id, lookupField: dt.entity.host
 
// Exclude agents running in unsupported OS version
| filter isNull(lookup.dt.entity.host)
 
// Remove unnecessary fields
| fieldsRemove lookup.dt.entity.host

// Create timeseries metric
| maketimeseries Count=count(default:0), by:{dt.entity.host=entity.name, installerVersion}, interval:1m

Hope this helps the community. 

[AntonPineiro]

Thanks! 

[joyce-regions]

Can you help me with the output of this query to send email? Can we have the output table in the body message when it send to email?

[mosharref_hossn]

We ended up add these additional lines of query after the timeseries. This creates a link to the Host OneAgent settings page and add the process name via lookup. 

Use the Send Email action in Workflow and add the email body markdown table to format all fields into tables. 

Query update:

// Look up One Agent System monitoring process group instance which can be used as dt.source_entity, if needed for management zone. 
| lookup [
    fetch dt.entity.process_group_instance
    | fields id, entity.name, belongs_to
    | fieldsAdd process.belongs_to.dt.entity.host=belongs_to[dt.entity.host]
    | filter contains(entity.name, "OneAgent system monitoring")
], sourceField: dt.entity.host, lookupField: process.belongs_to.dt.entity.host, fields:{dt.entity.process_group_instance=id} 
| fieldsAdd process.name = entityName(dt.entity.process_group_instance)
// Create a link to host OneAgent settings
| fieldsAdd OneAgentSettings = concat("https://{environmentid}.apps.dynatrace.com/ui/apps/dynatrace.classic.hosts/ui/settings/", dt.entity.host, "/builtin:deployment.oneagent.updates?gtf=-2h&gf=all")
| fields host.name, installerVersion, process.name, OneAgentSettings

 Email body markdown table:

|      host.name       |      process.name                   |     installerVersion              |    OneAgentSettings    |
|----------------------------|----------------------------------|-----------------------------------|----------------------------------------|
{% for e in result("get_outdated_oneagent")["records"] %}
|  {{e["host.name"][:35]}}     | {{e["process.name"][:35]}}  | {{e["installerVersion"][:30]}}       | {{e["OneAgentSettings"][:160]}} |
{% endfor %}