Abstract
Recently a classic Dashboard preset: "Tagging worker Dashboard" was released which provides insights into one of our main engines in regard to environment segregation and filtering. This article will explain how to understand the information available in the dashboard and the effects that long rule processing times can have on the environment.
Dashboard content
In the dashboard the following set of metrics is available:
-
dsfm:server.monitored_entities.tagging.tagging_worker_runtimeProvides runtime for rule processing split by worker type and feature -
dsfm:server.monitored_entities.tagging.tagging_worker_top_slowest_rules
Provides runtime for the top 100 slowest rules executed in the environment -
dsfm:server.management_zones.active_configs
Provides count of configured Management Zones -
dsfm:server.management_zones.queries_counter
Provides information about the count of queries that poll Management Zones in the environment. -
dsfm:server.automatically_applied_tags.active_configs
Provides count of configured tags
Health assessment
Processing time
Starting at the top of the dashboard we have first a section using the runtime metric in different visual representations:
Those charts represent the total amount of time it takes our worker to process all the Management Zone/Tagging/Naming rules in your environment.
- The worker is running in back to back mode and starts as soon as the previous run finished.
- Metric is published as soon as the processing time is known. The larger the processing time the larger the gap between datapoints.
- Nested rules are not applied recursively with every run - if your Management Zone depends on a tag that depends on another tag, it can take multiple runs to apply it to your entity
Splitting by dt.tagging_runtime.feature allows to find out how long each of the Management Zone/Tagging/Naming takes to process.
Splitting by dt.tagging_runtime.type allows to find out how long each of our default/fastlane workers takes to process the rules.
Slowest rules
Next is our section about top slowest rule execution times. This section helps to understand potential outliers of your configuration.
The provided charts present both the slowest rules on average in your selected timeframe and per execution assigned to specific point in time.
Splitting by dt.tagging_rule.name allows to find out which rule is responsible for processing in given time.
Splitting by dt.tagging_rule.feature allows to find out what kind of configuration is responsible(Naming, Management Zones, Tagging)
For example here we can see that the slowest rules are:
- [Kubernetes]namespace Tagging rule with index 0 (first on the list) that is tagging Process groups
- Cloud: AWS Management Zone rule with index 9 that is matching Services
Config usage
The last section of the dashboard is dedicated to understanding the amount of configured rules for Management Zones and Tagging with addition of a chart providing information on how many times given Management Zone was queried.
Currently Tag query usage is not available.
Management zone query count can be useful to determine which configurations are not used by users and can be deleted to reduce processing time.
What is the impact?
- The longer the worker runtimes are the longer an entity will potentially have to wait until Tag, Management Zone or Name is attached to it.
- Long processing times can lead to intermediate states where entity is expected in Management Zone according to rules but does not belong there yet at given point in time
- Searches that use entity metadata for filtering will not return expected entities when in this intermediate state - if it does not have a Tag then searching for a Tag will not find it
- Events/Problems will not inherit metadata that does not exist on impacted entities
- Alerting profiles might not match in this intermediate state and as a result notifications might not be sent to the relevant teams
- Maintenance windows might not attach based on non-existing metadata
Resolution
As mentioned previously:
The worker is running in back to back mode and starts as soon as the previous run finished.
Dynatrace is already doing all it can to make this process as fast as we possibly can. If you notice long processing runtimes in your environment please consult this best practices guide: https://docs.dynatrace.com/docs/shortlink/tagging-at-scale
We also recommend to review your Management Zone usage with dsfm:server.management_zones.queries_counter all and delete all unnecessary/not-used configurations.
What's next
It is a Preset dashboard and should be available in every environment that has a supported Dynatrace version.
Here is a link to our public demo if someone is interested:
I don't have access to https://demo.live.dynatrace.com/ui/apps/dynatrace.classic.dashboards/#dashboard;gtf=-2h;gf=all;id=78...
can you add me?
