Log monitoring v2 - Event log server side log entry filtering

[AlanK]

I am using Log Monitoring v2 in a Managed cluster. I need to monitor for specific events in the Windows Security Event Log on our application servers. 

The Windows Security Log generates a lot of events, and if I enable monitoring this on all my application servers I am going to reach the maximum # of log events per minute limitation on our cluster.  

How can I configure server side log entry filtering in the ruxitagentloganalytics.conf so that we are only capturing the event IDs that we need?

I have read this doc https://www.dynatrace.com/support/help/how-to-use-dynatrace/log-monitoring-v1/log-analytics-configur... and looked at the comments in the ruxitagentloganalytics.conf.  The bolded line below appears to show how to only capture 'INFO' level logs into Dynatrace, but it is unclear how to filter by Event ID.  

#Server side log entry filtering
#EntryFilter=Process Group Id, log path, LAQL (https://www.dynatrace.com/support/help/infrastructure/log-analytics/dynatrace-search-query-language)
#EntryFilter=0x0,Windows Application Log,INFO=======
#EntryFilter=0x201744FC09941B85,c:\ProgramData\CrashPlan\log\service.log.#,not INFO=======

Any help would be appreciated.  Thanks

[Radoslaw_Szulgo]

I think you look at the wrong doc - it's related to Log version 1 and you claim you use Log v2. What you need I believe is a log processing rules to FILTER OUT some events. Take a look here:

https://www.dynatrace.com/support/help/shortlink/lm-log-processing-commands

[rastislav_danis]

But log processing happens on server, so problem with "maximum # of log events per minute limitation on our cluster" will not be solved by that way. Agent will still send all eventlog events (Log Processing does not affect DDU consumption of log ingest).

[Babar_Qayyum]

Hello @AlanK 

I might not understand correctly but now there is a sophisticated way to drop the log events. You can go through with the below link already shared by @Radoslaw_Szulgo 

https://www.dynatrace.com/support/help/shortlink/lm-log-processing-commands

@rastislav_danis 

To overcome the maximum log events limit, we used the same methodology to drop/filter out the events not required. In that way, we are receiving only the required events plus the random ingested log data termination is not taking out the important log events.

Regards,

Babar

[Radoslaw_Szulgo]

Then what I do is I use a log forwarder. For instance fluentd (https://www.dynatrace.com/support/help/how-to-use-dynatrace/log-monitoring/acquire-log-data/stream-l...)

And I filter in fluentd: https://docs.fluentd.org/filter

[bart_butenaers]

@Radoslaw_Szulgo 
We had previously also registered another product idea about this, but still didn't find a proper solution.  As a result we still haven't migrated yet from our ElasticSearch to Dynatrace Log Monitoring V2.

The EntryFilter solution seems to be V1 related, so we cannot use that.  The FilterOut solution is processed at the server side, so we cannot use that either (because we have a massive amount of useless log entries that we don't want to send to Dynatrace across the network).

Do I understand correctly that we need to write a custom Log Forwarder somehow, to allow the OneAgent to filter our log files (before sending them to Dynatrace managed servers or Saas)?  We would appreciate to get some tips about that!

Thanks!
Bart