How to Investigate an Exposed Dynatrace API Token

[ZackCompora]

• Summary
• Problem
• Prerequisite
• Required Permissions
• Troubleshooting Steps
• Step 1: Reviewing the Token in the UI
• Step 2: Reviewing Audit Logs via API
• Step 3: Investigate with DQL
• Audit Logs
• Audit Events
• Resolution
• What's next

Summary

The purpose of this community post is to provide guidance for investigating an exposed Dynatrace API token via a public Git repository.

Accidental exposure of secrets in Git repositories is more common than many teams may expect. A configuration file may be committed without noticing an embedded API token, or credentials may be pushed alongside a rapid code change. An internal repository can also become publicly accessible due to migration, misconfiguration, or administrative error. Even if secrets are later removed, they may still remain in the repository's commit history.

Whether you are contacted by Dynatrace Security, or discover credential exposure independently, it is important to understand that exposure does not automatically mean compromise. The steps provided in this article are designed to help you validate whether the token(s) was misused before it was rotated or invalidated.

Problem

Dynatrace may notify you when GitHub Secret Scanning detects a public Dynatrace token exposure. This guide shows how to Validate the exposure, review audit logs for potential misuse, and remediate.

Prerequisite

These steps assume audit logging was enabled prior to the exposure event; if it was not enabled, historical audit visibility may be limited.

Before beginning your investigation:

• Upload our prebuilt Dynatrace notebook that contains ready-to-go queries to your environment

• Ensure that audit logging is enabled in the affected environment

  • To enable audit logging

    • Go to Settings -> Preferences -> Log audit events.

    • Turn on Log all audit-related system events.

    • You can also enable audit logs via Data privacy API.

Retention Note: Dynatrace retains audit logs for 30 days and automatically deletes them afterwards. However, Grail audit events are stored for 1 year.

Required Permissions

You will need:

• auditLogs.read permission (for the Audit Logs API)

• logs.read permission (for Dynatrace DQL log searches)

• Access to the environment where the token was issued

• For IAM Policy related permissions please review here

  • storage:buckets:read

  • storage:logs:read

You can verify permissions under:

Account Management → User Management → Permissions

Troubleshooting Steps

Step 1: Reviewing the Token in the UI

Navigate to the Access Tokens application in your tenant UI and identify the token that has been exposed. An example for Personal Access Tokens is also provided.

New Dynatrace UI:

https://{your-environment-id}.live.dynatrace.com/ui/apps/dynatrace.classic.tokens/ui/access-tokens

https://{your-environment-id}.live.dynatrace.com/ui/apps/dynatrace.classic.personal.access.tokens/ui/personal-access-tokens

Dynatrace Classic UI:

https://{your-environment-id}.live.dynatrace.com/ui/access-tokens

https://{your-environment-id}.live.dynatrace.com/ui/personal-access-tokens

Here you can check:

• Creation date

• Last used timestamp

• Last used IP address

• Assigned scopes

  • Token scopes can help estimate the impact of the event at a glance, for example the scope to ingest logs could impact integrity but not confidentiality, because a potential attacker would not have been able to extract data.

• Token status (enabled/disabled)

If the token was publicly exposed, it is best to assume it may have been accessible to third parties.

Our recommendation:
Immediately disable or delete the token and verify whether the “Last used” timestamp aligns with expected activity.

Step 2: Reviewing Audit Logs via API

Audit Logs API Documentation

You can retrieve audit events related to token lifecycle activity using the audit logs.

Example API Call:

curl -X 'GET' \ "https://{your-environment-id}.live.dynatrace.com/api/v2/auditlogs?filter=category(\"TOKEN\"),entityId(\"{token-id}\")&sort=-timestamp" \ -H "accept: application/json; charset=utf-8" \ -H "Authorization: Api-Token {your-investigation-token}"

Important:

• Use a separate investigation token, not the exposed one
  • Permissions/scope for the token would be:
    • auditLogs.read
• Replace {token-id} with the token identifier, not the token secret itself
  • In a Dynatrace token dt0c01.TEST12345678901234567890.TEST123456789012345678901234567890123456789012345678901234567890 the identifier, or public part, in this case would be dt0c01.TEST12345678901234567890

This example filters for:

• Object category = TOKEN
• Specific token entityId
• Most recent events first

If the response includes a nextPageKey, request the next page by using only nextPageKey (do not include filter/sort/pageSize on the next request).

Example with Pagination:

curl -X 'GET' \
"https://{your-environment-id}.live.dynatrace.com/api/v2/auditlogs?nextPageKey={PASTE_VALUE_FROM_RESPONSE}" \
-H 'accept: application/json; charset=utf-8' \
-H 'Authorization: Api-Token {your-investigation-token}"

Other events to review:

• Token Creation
• Token Updates
• Token Deletion
• Config changes that occurred post exposure

Look for:

• Unusual timestamps
• Unknown IP addresses
  • If you observe unfamiliar IPs, compare them against your organization's known IP ranges and consider using your preferred threat intelligence source for further evaluation of suspicious IPs (for example AbuseIPDB or VirusTotal)
• New token creations
• Scope modifications

Step 3: Investigate with DQL

Audit Logs

If the exposed token was used for ingesting logs, you can search for ingestion activity using:

fetch logs // Filter for a specific token | filter dt.auth.origin == "<public part of your token>" // Fields to be displayed | fields timestamp, dt.auth.origin, content // Sort by timestamp | sort timestamp desc

dt.auth.origin is populated for ingestion-related tokens and identifies the token origin used to ingest the log record.

Note: dt.auth.origin only appears for ingestion related tokens

Audit Events

You can also search audit events stored in Grail:

fetch dt.system.events // Narrow scope to audit events only | filter event.kind == "AUDIT_EVENT" // Filter down to view only the desired token | filter authentication.token == "<public part of your token>" // Sort by timestamp | sort timestamp desc

From these results, you may be able to identify:

• When a token was last used
• The IP address associated with authentication
• Failed authentication attempts

Resolution

 The same steps are recommended for all public token exposures. Perform containment before investigation to ensure that, even if no breach has occurred yet, the exposed token cannot be abused going forward:

• Always revoke all exposed tokens as soon as possible
  • As long as an exposed token is valid, it can be abused
• Rotate credentials as necessary
  • Create replacement tokens with least privilege, update integrations, and validate everything still works
• Delete the token from code/config and re-write commit history if it was committed. Consider making the repository private if appropriate, and ensure forks/branches do not still contain the secret

If no suspicious activity is found, revoking the exposed token(s), rotating replacements, and removing the secret from the repo/history is typically sufficient for resolution.

If suspicious activity is detected, follow your organization's incident response process. The troubleshooting steps in this guide can help identify actions performed using the token.

• If the exposed token had permission to create/modify tokens, or to change permissions, revoke any newly created tokens and remove any unauthorized access
• In call cases, revoking the exposed token(s) stops further use of those credentials

What's next

If this article did not help, please open a support ticket, mention that this article was used and provide the following in the ticket:

• An authorized contact on the ticket: Add a user who can view audit logs and any required sensitive/security data in the tenant
• Link to the affected tenant
• Approximate date/time the secret may have been exposed and the timeframe you reviewed in the audit logs
• What you have already done, whether the token was revoked, replacements created/updated, and whether the commit history was rewritten/purged
• Where you are blocked, the specific step/section of this article.

Security note: Do not paste tokens into the ticket or support chat. Please obfuscate or omit any secrets from screenshots/logs.