cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Find the Root Cause Entity in case of Multiple "Application/Service/Infra" problems.

Maheedhar_T
Mentor

Heya,
Just a tip here, we in our environment have integrated Dynatrace <-> ServiceNow and whenever a problem is generated it flows to ServiceNow, and the concerned team is alerted. However, we found an issue where if the problem has multiple impacted entities, It does not pin-point the exact root cause entity. The payload that we use in the integration is giving only "Impacted Entities" which would be multiple in this case but not the exact root cause entity.
Hence when the incidents are flowing to ServiceNow we do not have an exact rule of which queue this incident is to be mapped to, and it ends up being unnoticed or sent to the wrong queue.

Also, there is one more catch here, root_cause_entity is discovered after a problem is analysed for certain amount of time in Dynatrace but the Custom Integration using which we send alerts to ServiceNow only has two states which call the API, they are "When the problem is OPEN", "When the problem is CLOSED " but not when the problem is updated.

To overcome these two limitations and following @theharithsa's Idea we came up with a workaround that might help you.

We used workflows so if you're on older version or Dynatrace Managed this might not apply to you.


Step-1: Identify the workflow trigger.
Here, we have to trigger the workflow whenever a problem is created, updated and closed. So Go to wokflows, create a new workflow and in the trigger, select Davis Problem trigger.
Event State: Active/Closed (As we need updates whenever the event is active until it is closed)
Event Category: Select Everything
Affected entities: All entities 
Filter Query(This is where we control what kind of Dynatrace problems trigger this workflow. We do not need the workflow to run on every problem. So, we select the kind only when a problem with "Multiple application/service/infrastructure problem" title comes up.
Also, on the further steps, we use root cause entity as a field, so we wait till that is discovered. Hence the query looks like.

 

 

(matchesPhrase(event.name,"Multiple service problems") or matchesPhrase(event.name,"Multiple infrastructure problems") or matchesPhrase(event.name,"Multiple application problems") or matchesPhrase(event.name,"Multiple environment problems")) and isNotNull(root_cause_entity_id)

 

 


Next steps would be extracting the required data from the Problem.
For this we use JS and extract the needed fields.

Here is the code for the same:

 

 

// optional import of sdk modules
import { execution } from '@dynatrace-sdk/automation-utils';

export default async function ({ execution_id }) {
  // your code goes here
  // e.g. get the current execution
  const ex = await execution(execution_id);
  let payload;
  if(ex.params.event.hasOwnProperty("root_cause_entity_id")){
    //const tags = ex.params.event.entity_tags;
    const root_cause_entity_id = ex.params.event.root_cause_entity_id;
    const root_cause_entity_name = ex.params.event.root_cause_entity_name;
    const display_id = ex.params.event.display_id;
    const pid = ex.params.event['event.id'];
    const event_state = ex.params.event['event.status'];
    const event_transition_state = ex.params.event['event.status_transition'];
    const entity_type = "dt.entity."+ex.params.event.root_cause_entity_id.split('-')[0].toLowerCase();
    const event_category = ex.params.event['event.category']
    const event_name = ex.params.event['event.name']
    payload =
    {
        "root_cause_entity_id": root_cause_entity_id,
        "root_cause_entity_name": root_cause_entity_name,
        "display_id": display_id,
        "pid": pid,
        "event_state": event_state,
        "entity_type": entity_type,
        "event_transaction_state": event_transition_state,
        "event_category": event_category,
        "event_name": event_name
    }
    }

  return payload;
}

 

 


The next steps will be customized based on your requirements. In our case, I identified the root cause entity, replaced the impacted entity with the root cause entity, and posted the payload to ServiceNow. This allowed ServiceNow to accurately identify the entity causing the problem, enabling us to assign the incident to the appropriate team’s queue for action.

Hope this helps.

Regards,
@Maheedhar_T 

Maheedhar
1 REPLY 1

theharithsa
Dynatrace Champion
Dynatrace Champion

It is a great tip, Maheedhar. I am sure it will definitely help a lot of customers who are using Dynatrace. 

Love more, hate less; Technology for all, together we grow.

Featured Posts