Use Case
Dynatrace users, especially those who rely on generic entity types from integrations/extensions, can sometimes run into the default limit for the number of entities per entity type. When this limit is hit, no new entities are created for that type until the monitored entity periodic junk cleanup worker clears out old entities, allowing for new entities to be created.
While `dsfm:server.monitored_entities.entity_count_per_entity_type` metric exists, it can be inconsistent or miss some generic entities in its count.
To workaround this, I've created a Platform Workflow that uses the Dynatrace API to:
- Get all possible types of entity from the Environment (Such as "HOST" or "OS:SERVICE").
- Get the count of entities, per type, filtering out all entities that have a 0 count.
- Submit the entity types + count of entities into the entity.types.count.gauge metric key.
This allows us to track the per-entity-type count of entities, and optionally setup metric events and/or Davis Anomaly Detectors which can alert when a given entity type is close to reaching the limit.
Set it up
Download the Workflow Template from Github > Import it into the Platform Workflows App.
Create a new Access Token with the following scopes: metrics.ingest, metrics.write, entities.read
Set this Access Token up within Credential Vault. Copy the Credential ID once created.
- Credential Type: Token
- Credential Name: Enter a friendly name
- Credential scope: Check the "AppEngine" box
- Dynatrace apps with access: Enable the "Allow access without app context" slider
- Token: Enter the Access Token which was created previously
Edit the following lines within each JS Code Tile on the Workflow to point to the created Credential Vault Entry - Replace CREDENTIALS_VAULT-XXXX with your Credential ID:
- count_entity_types: Line 55
- post_metric_data: Line 14
Optionally, change the trigger be to a schedule, or leave it as an on-demand trigger. I would suggest executing this workflow at most once every hour, so as to not overload the Dynatrace API Endpoints with numerous GET requests when the numbers likely won't change super frequently on an hourly basis.
Once we've had a successful workflow execution, we should then be able to setup Metric Events/Davis Anomaly Detectors which will alert when a given entity is close to breaching the limit. We can also use Data Explorer/Dashboards to chart the entity counts per type over time.
