How to filter out security scans from OneAgent data

[kalle_lahtinen]

Hi,

I'm facing the exact same challenge as this old case regarding AppMon:

https://community.dynatrace.com/questions/97646/view.html

"We scan our site with several security tool such as Whitehat. Since these tool are sending bad request it causes our error rate to spike. I would like to either segment this traffic (based on IP address) or filter it out completely. What's the best approach to acomplish this segmentation/filtering?"

The response was for AppMon & Business Transactions, but I wonder what would be the best approach for OneAgent? What I've done so far is create a request attribute based on the UserAgent string and save it if it ends with WhiteHat Security. However this just allows me to easily filter the "bad" data, but not really get rid of it. Is there anything more I could do with this - I suppose we can't just drop requests with a certain UserAgent and/or request attribute value?

To handle the false positive alerts, I've muted those malformed requests, which appears to initially help. Of course it's a chore to do manually, and it's also not a future-proof solution because it looks like the request sets are changing in time. So it's basically a game of cat and mouse, trying to keep muting those security scans. The best option would be if I can just drop that data altogether.

[Babar_Qayyum]

Hello @Kalle L.

As the scan data does not have any value for you, then you can simply exclude the IP/or range of IPs, if it is not changing every time.

Regards,

Babar

[kalle_lahtinen]

Can you do that for PurePath data? I thought the option to exclude IPs is only available for Application/RUM data.

[Babar_Qayyum]

Hello @Kalle L.

We have an option to override the process configuratoin from the deep monitoring > troubleshooting.

Regards,

Babar

[kalle_lahtinen]

I know about that setting, but doesn't it then drop all PurePaths for the particular process group? I just want to drop the ones where UserAgent ends with WhiteHat Security.

[Babar_Qayyum]

Hello @Kalle L.

Did you try to exclude specific incoming web request URLs from the deep monitoring as well?

Regards,

Babar

[kalle_lahtinen]

No, that wouldn't help us here. The rule cannot be based on the URL, because they are seemingly random and there are even hundreds of them altogether.

[Babar_Qayyum]

Hello @Kalle L.

It means then we left only with the IP exclusion if someone else does not share any other option with us.

Regards,

Babar

[kalle_lahtinen]

And that brings us back to my earlier question: "I thought the option to exclude IPs is only available for Application/RUM data."? I'm not aware of the option to drop PurePath capturing based on Client IP...

[JamesKitson · Accepted solution]

You could do a combination of the two things you've tried. If you're not very interested in this traffic I would try making a request naming ruling using your request attribute for the scan user-agent as the filter. This would have the effect of splitting all of this traffic into one big bucket of requests all with the same name applied (e.g. Security-Scan-Requests).

Then with only one request you could more easily mute it without needing to be concerned with the paths of those requests or IP of the scans changing over time.

[kalle_lahtinen]

Great idea! Thanks James, I'll for sure implement it this way.