Need feedback on my SDLC Root Cause Enrichment Agent

[andreas_grabner]

Hi

I have been promoting sending SDLC events to Dynatrace from GitHub, GitLab, Argo, Azure DevOps, Jenkins ... over many years. Its also well documented on how to get those events into Dynatrace using e.g: GitHub Webhooks or Argo Notifications -> e.g: Observe ArgoCD 

While this gives us an option to report and analyze ArgoCD activity, e.g: how many syncs, how long they take, how many fail - it doesnt give us automated root cause analysis when Dynatrace detects a problem.

Lifecycle Workflow and its limitation

One option we have shown over the past years is to use a Workflow that triggers on such an SDLC event, then looks up the deployed entity in Dynatrace and then creates an additonal CUSTOM_DEPLOYMENT event attached to that entity.  On the playground you can see this as the Lifecycle Event Workflow.

While this works - it has some limitations especially when you have hundreds or thousands of SDLC events coming in every day. Because- this means that this Workflow gets executed for every SDLC event - does a lookup - and then creates a new event just to make root cause analysis easier

Proposing: SDLC Root Cause Enrichment Agent

To better understand whether a deployment change is the root cause of a problem I thought about a different approach and implemented the SDLC Root Cause Enrichment Agent. You can also see it on the Playground.

This one triggers when Dynatrace detects a problem (by default filter on problems that also have a k8s.namespace attached). That workflow then queries SDLC events (they dont have to be associated with an entity but have to have a namespace which most events by default do). If found it will create an impact and root cause analysis by using those SDLC events and general ki8s events. It then posts this back to the problem as a comment / annotation. See the result here

Feedback / want to try it out?

I am looking for people that have already or can easily ingest SDLC events from their GitHub, GitLab, Argo, Azure DevOps ... and would like to give this workflow a try. I want to get feedback if this works - refine it - and then do a detailed blog post about it

Thanks in advance for your help