cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Confirming or manipulating incidents via REST.

JamesKitson
Dynatrace Leader
Dynatrace Leader

Hello,

Our end goal is to have some way of automating the changing of incident states so that we can utilize smart alerting without accidentally missing an incident and thus missing an alert.

I have been looking into the REST interface where we can manipulate incidents but as I am relatively new to actually implementing something like this I had some questions.

  1. Is it possible to change the state of multiple incidents at a time (i.e confirm all incidents over 7 days old)?
  2. To get this to run as a task on a schedule would it be required to write a plugin that utilizes the REST interface or is there some other way to automatically modify incidents?
  3. Does anyone have any better ideas on how to accomplish the end goal?

My other idea if this proves difficult is to just have a scheduled report emailed out that shows if there are any unconfirmed incidents. The downside to this would be it is still possible to miss something if all actions are manual and so this is more of a fall back.

Thanks!

James

4 REPLIES 4

graeme_william1
Inactive

James,

You can't change the state of multiple incidents at the same time. The two calls that change the state of incidents take a specific incident id as a mandatory parameter.

Yes, using a plugin running on a schedule is a reasonable way of proceeding. I like using the Generic Execution Plugin and writing whatever I want in shell / PowerShell. That way, I can debug the script outside of Dynatrace and then just drop the script into the plugin. Inside the script, you can call the REST interface via curl. One obvious caveat about this technique: the plugin runs in the Collector with whatever privileges the Collector process has.

Well, someone else might have a better idea ...

Two more caveats about the doc page at

https://community.dynatrace.com/community/pages/viewpage.action?pageId=196642731:

  • The parameters and examples of each REST command are hidden behind those little triangles, so you need to expand them to see the content;
  • And you need to include authentication information in the REST calls.

If you go down this path and you end up running curl on Windows, reach out to me at firstname.lastname@dynatrace.com and I'll send you what I have. I'm sure you're better at this than me, but getting the authentication and escaping right in curl on Windows had me pulling my hair out for a day or so.

-- Graeme

Thanks Graeme, that helps a lot. Now I'm thinking about using the REST incident interface or just querying a dashlet to get a list of incident ids where the incident is unconfirmed (and maybe old if I can figure it out) and then looping through that and making REST calls to change the status to confirmed probably at a system profile level at first.

Hi James,

Did you end up writing a shell script or something to do what you described above? I'm looking for ways to periodically confirm closed incidents as well and think this is probably the best approach, but I wanted to know if you had any success with it.

Kind Regards,

Jake

Hey Jake - we basically decided against implementing it focusing on the design of incidents to limit the amount of spam (making them less sensitive as an example) but we did have an early prototype working where we actually used jmeter to pull the list of open incidents and cycle through them setting them as confirmed. Once installed on a collector if I remember correctly we were able to schedule this though we never moved forward with it. I don't believe I have much if anything from that but I can say that it is feasible, and actually a plugin would probably be the best option for that.