Drift Detection for Dynatrace Workflows with GitHub Actions

Georgi_V · ‎03 Sep 2026

Drift Detection for Dynatrace Workflows with GitHub Actions

How we keep a Git repo as the source of truth for dozens of Dynatrace Automation Workflows — automatically

The Problem: Workflows Drift

Dynatrace Automation Workflows are easy to edit in the UI — which is exactly why they drift out of sync with source control. Someone tweaks a condition, adds a task, or fixes a typo directly in the browser, and the change never makes it back into Git. A few months later, nobody can tell you what's actually deployed versus what's in the repo, and a "rollback" from Git silently reverts someone's fix.

We wanted a Git repo to be the actual source of truth for our ~35 Dynatrace workflows, with automatic detection whenever live and local state diverge — in either direction.

The result is a small GitHub Actions pipeline built around dtctl (the Dynatrace CLI) and a Python script that reconciles drift automatically, plus a companion script for manual, read-only checks.

Architecture

[ Dynatrace (live workflows) ] --dtctl get workflows--> [ GitHub Actions (scheduled run) ]
                                                                |
                                                                v
                                                compare live vs. local YAML
                                                                |
                                        +---------------------+---------------------+
                                        |                                           |
                                        v                                           v
                              no drift -> exit clean               drift found -> write files,
                                                                    open branch + PR for review

Two scripts, two purposes:

sync_workflows.py — the pipeline's workhorse, and the only script actually wired into GitHub Actions. It diffs live Dynatrace state against the local YAML, and when it finds drift it writes the live state back to the repo, stages the affected files, and commits on a dedicated branch — which the GitHub Actions job then pushes and opens as a PR.
compare_workflows.py — a read-only companion script for manual, on-demand checks (CHANGED, NEW IN DYNATRACE, LOCAL-ONLY). It's not part of the CI pipeline — I run it locally when I want a quick drift report without touching any files or opening a PR.

Nothing is ever auto-merged. Every drift correction lands as a pull request that a human reviews before it hits master.

Step 1 — Exporting Workflows with dtctl

dtctl can dump every workflow in an environment as YAML in one call:

dtctl get workflows -o yaml

This returns a YAML stream of every workflow object, including its id, title, trigger, and tasks. That id field is the join key between "live" and "local" — we key everything off it rather than filename or title, since titles change but IDs don't.

Step 2 — Comparing Live vs. Local

The comparison logic — whether in the CI script or the manual one — is deliberately simple and read-only:

docs = list(yaml.safe_load_all(result.stdout))
live_by_id = {wf["id"]: wf for wf in docs if wf.get("id")}

local_files = {}
for fname in sorted(os.listdir(WF_DIR)):
    ...
    if "id" in data:
        local_files[data["id"]] = (fname, data)

for wf_id, (fname, local_data) in local_files.items():
    if wf_id not in live_by_id:
        only_local.append(fname)          # deleted in DT, still in git
        continue
    if yaml.dump(local_data, sort_keys=True) != yaml.dump(live_by_id[wf_id], sort_keys=True):
        changed.append((fname, wf_id, diff))   # drifted

It normalizes both sides to canonical YAML (sort_keys=True) before comparing, so key ordering never produces a false positive. Three buckets fall out of this:

CHANGED — the workflow exists in both places but the content differs
NEW IN DYNATRACE — someone created a workflow in the UI that was never committed
LOCAL-ONLY — a file exists in Git but Dynatrace no longer has that ID (likely deleted in the UI)

One early lesson: not every field is meaningful in version control. Dynatrace stamps a nextExecution timestamp onto schedule-triggered workflows that it auto-advances on every run. Left in the diff, it made every scheduled workflow show as "changed" on every sync. We strip it (and anything else purely runtime) before comparing or writing:

RUNTIME_FIELDS = {"nextExecution"}

def strip_runtime(data):
    if isinstance(data, dict):
        return {k: strip_runtime(v) for k, v in data.items() if k not in RUNTIME_FIELDS}
    if isinstance(data, list):
        return [strip_runtime(i) for i in data]
    return data

If you hit persistent noisy diffs in your own drift detection, look for auto-generated/runtime fields like this before assuming something is actually wrong.

Step 3 — Reconciling Drift and Opening a PR

sync_workflows.py — the script that actually runs in CI — does the same comparison, then acts on it: it writes the live Dynatrace state to the local YAML files, git adds only the files that changed, and commits with a message summarizing what changed and what's new — never touching files that didn't drift.

The GitHub Actions job wraps this in a disposable branch so the change is always proposed, not applied directly:

- name: Create sync branch
  id: branch
  run: |
    BRANCH="sync/dynatrace-workflows-$(date +%Y-%m-%d)"
    git checkout -b "$BRANCH"
    echo "name=$BRANCH" >> "$GITHUB_OUTPUT"

- name: Run workflow sync
  run: python3 scripts/sync_workflows.py

- name: Push branch and open PR
  env:
    GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
  run: |
    if git diff --quiet HEAD origin/master 2>/dev/null; then
      echo "Nothing to sync - skipping PR."
    else
      git push -u origin "${{ steps.branch.outputs.name }}"
      gh pr create --base master --head "${{ steps.branch.outputs.name }}" \
        --title "sync: update Dynatrace workflows $(date +%Y-%m-%d)" \
        --body "Automated daily sync of Dynatrace workflows via sync_workflows.py."
    fi

If nothing drifted, the job exits cleanly with no PR — no noise in the repo on a quiet day.

Step 4 — The Full Pipeline

Putting it together, the complete GitHub Actions workflow runs on a weekday schedule and can also be triggered manually:

name: Sync Dynatrace Workflows

on:
  schedule:
    - cron: '0 13 * * 1-5'   # 9 AM ET (UTC-4), Mon-Fri
  workflow_dispatch:

jobs:
  sync:
    runs-on: ubuntu-latest
    permissions:
      contents: write
      pull-requests: write

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: '3.x'

      - name: Install dependencies
        run: pip install pyyaml

      - name: Install dtctl
        run: |
          curl -fsSL https://github.com/dynatrace-oss/dtctl/releases/download/v0.28.1/dtctl_0.28.1_linux_amd64.tar.gz | tar -xz -C /usr/local/bin
          chmod +x /usr/local/bin/dtctl

      - name: Configure dtctl
        env:
          DT_PLATFORM_TOKEN: ${{ secrets.DT_PLATFORM_TOKEN }}
        run: |
          mkdir -p ~/.config/dtctl
          printf 'apiVersion: v1\nkind: Config\ncurrent-context: prod\ncontexts:\n- name: prod\n  context:\n    environment: YOUR_DT_ENVIRONMENT_URL\n    token-ref: ci-token\n    safety-level: readwrite-all\ntokens:\n- name: ci-token\n  token: "%s"\npreferences:\n  output: yaml\n' "$DT_PLATFORM_TOKEN" > ~/.config/dtctl/config

      - name: Create sync branch
        id: branch
        run: |
          BRANCH="sync/dynatrace-workflows-$(date +%Y-%m-%d)"
          git checkout -b "$BRANCH"
          echo "name=$BRANCH" >> "$GITHUB_OUTPUT"

      - name: Configure git identity
        run: |
          git config user.name  "github-actions[bot]"
          git config user.email "github-actions[bot]@users.noreply.github.com"

      - name: Run workflow sync
        env:
          GIT_ROOT: ${{ github.workspace }}
          WF_DIR:   ${{ github.workspace }}/dynatrace-workflows
          DTCTL_PATH: /usr/local/bin/dtctl
        run: python3 scripts/sync_workflows.py

      - name: Push branch and open PR
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          if git diff --quiet HEAD origin/master 2>/dev/null; then
            echo "Nothing to sync - skipping PR."
          else
            git push -u origin "${{ steps.branch.outputs.name }}"
            gh pr create \
              --base master \
              --head "${{ steps.branch.outputs.name }}" \
              --title "sync: update Dynatrace workflows $(date +%Y-%m-%d)" \
              --body "Automated daily sync of Dynatrace workflows via sync_workflows.py."
          fi

A few implementation notes worth calling out

token-ref over inline tokens: dtctl's config supports referencing a token by name from a separate tokens: list rather than inlining it directly under the context. YOUR_DT_ENVIRONMENT_URL above is a stand-in for your tenant's environment URL. We generate this config with printf (not a heredoc) — heredocs interact badly with GitHub Actions' run: | block when the payload contains a secret-derived variable, and printf sidesteps that entirely.
safety-level: readwrite-all: required for dtctl to actually write workflow updates, not just read them — set deliberately and scoped to a CI-only platform token.
Pin the dtctl version: fetching latest at every run is a supply-chain risk (an unreviewed binary changes silently under you) and a reproducibility risk (behavior can shift between CI runs). We pin the exact release tag and upgrade deliberately.
permissions: contents: write, pull-requests: write: scoped to only what the job needs — push a branch and open a PR — nothing broader.

Why a PR Instead of Auto-Apply

We considered having the pipeline just overwrite the repo and commit straight to master. We didn't, for two reasons:

Branch protection — master requires PR review + CI checks in our repo, so a direct commit would fail anyway.
Human-in-the-loop for drift: an unreviewed sync means unreviewed production changes silently land in version control as "expected." A PR forces someone to look at what changed and ask whether it should have changed — sometimes UI edits are legitimate hotfixes worth keeping, sometimes they're mistakes worth reverting live instead of accepting.

Results

Runs Monday–Friday at 9 AM ET, plus on-demand via workflow_dispatch
On a quiet day: no diff, no PR, silent success
On a drift day: a single PR lists exactly which workflows changed and why, with a full YAML diff in the PR body
Zero manual dtctl get exports needed anymore — the repo is provably a faithful mirror of production, or the pipeline tells you exactly where it isn't

If you're managing more than a handful of Dynatrace Automation Workflows, this pattern generalizes well beyond workflows — the same dtctl get ... -o yaml + diff + PR shape works for IAM policies, notification settings, or any other Dynatrace config object that has a CLI export path.

Companion post: Managing Dynatrace Automation Workflows as Code with dtctl and GitHub Copilot covers authoring and deploying individual workflows with Copilot — this post covers keeping the whole fleet in sync afterward.