Well I spent some time yesterday on this in terms of attempting to get this into Splunk. I have been trying to take the approach to make it as easy and vendor supported as possible. I have not had much luck yet after trying a few things. In order to save people some time, here is what I have tried to date.
Splunkbase
I found 2 different apps that have been designed to work with Dynatrace.
- Dynatrace App for Splunk
- Dynatrace Add-on for Splunk
Both work with one another and after reading the documentation on these, it did not appear that it would work for this, but I gave it a try anyway and had no luck. They seemed to be focused on time series metrics for the most part. Also, the last version was Oct. 30, 2018 so I am not expecting to see an update anytime soon for the added feature of the audit log.
GitHub
I also did some searching on GitHub. Within that search I was able to locate both of the above Splunk apps and thought perhaps I could get my hands on the master as I believe both were built using the Splunk Add-On Builder. I did find one repository with it, but when you attempt to import it into Add-On Builder (v2.2.0), it fails. There is a new version of Add-On Builder which is 3.0.1, but I have not been able to test that yet as we first need to do a Splunk upgrade.
Splunk Add-On Builder
After going through all the above with no success, I decided to try using the Splunk Add-On Builder to just create my own. Off to a good start, but far from it working.
Next steps
I spent just about the entire day trying different things around this. In doing so, I found that it comes down to 4 options that I can think of at this time.
Option 1: License the 3rd party Splunkbase app "REST API Modular Input" app
I won't go into details here, but I found many people currently using this app for exactly this type of need. This Splunk app provides a straight forward, time savings method for polling data from RESTful endpoints. While you could do this yourself with something like the Splunk Add-On Builder which of course is free, it's also a lot of work. The developer of this app has done the hard work for you, at least from what I can tell so far. I plan to test this out to see just how good it is and will provide an update later. You can find the app on Splunkbase here.
Option 2: Start from scratch with Splunk Add-On Builder
As much as I do love to learn new things and really get into the gears of how things work, I also don't have the time right now. I have started to build an app using this method. Setting up the connection and all the initial steps are straight forward and I was able to pull the JSON of the audit long. Thats where things get difficult and you will end up spending most of your time if you go with this option. As before, I am not going to go into details here around the configuration of Splunk, but between all the mapping and other configurations that would need to be done, it makes option 1 above look much more inviting for myself anyway.
You can find the Splunk Add-On Builder here.
Note: Pay attention to the version you download. There are currently 2 version. If your instance of Splunk is below version 8.0, the Splunk Add-On Builder v3.0.1 will not work correctly. Unfortunately Splunk allows you to install apps that may not be compatible, meaning there is no check in their install engine. This is a pet peeve of mine with Splunk, but thats another story.
Option 3: Setup the Splunk HTTP Event Collector
The HTTP Event Collector is an endpoint that lets you send application events directly to the Splunk platform via HTTP or HTTPS using a token-based authentication model. You can use any standard HTTP Client that lets you send data in JavaScript Object Notation (JSON) format.
My thought here is to create a script that can be execute by either cron or some other scheduler which would pull the audit log from Dynatrace on a time interval of my choosing and would then send that log to Splunk via the HTTP Event Collector.
You can find the documentation on the Splunk HTTP Event Collector here.
Option 4: Custom script to pull down the audit log and use a Splunk Forwarder
Same thing as option 3 for the most part with the only difference being that your custom script would write the Dynatrace audit data to a local log where a configured Splunk Forwarder would pick it up and index it in Splunk like any other log. You would most likely want to ensure you have setup this custom script for rolling logs which the Splunk Forwarder knows how to work with.
Final thoughts...
Out of all these options, I think option 1 is the one I am going to try to work with first. While I have not tested this yet, it does appear that it would make everything straight forward and much easier to setup and have working. It's also a licensed app with support and not expensive. Plus, I could think of many other things I could use that app for beyond just Dynatrace providing even more value. Again, I have not tested this yet so it is only theory right now.
Recommendation to Dynatrace
Like with all vendors there are pros and cons. Splunk is a great tool, but when it comes to polling for data, this has never been an area they shine in. I would love to see Dynatrace add the ability to stream this data. That option currently exists today for user sessions and I think providing the same option around the audit log would be a huge value. You could easily then as a customer stay in a 100% vendor support solution because you could configure the Splunk HTTP Collector with a listener and then point the Dynatrace Audit Log Stream right to that Splunk HTTP Collector. In fact, it could really be considered an out of the box integration.
I will update when I do more testing. Hope this all helps!
