topic Re: Log processing rule in DQL

Log processing rule

tjay — Wed, 31 Jan 2024 04:43:24 GMT

Hi,

I'm trying to parse an Apache log file, but "Test the rule" complains with "The matcher query doesn't match the provided log sample". Can someone tell why?

Process definition

PARSE(content, "IPADDR:'net.peer.ip'
SPACE
IPADDR:'net.proxy.ip'
' - '
LD:'enduser.id'
' [' TIMESTAMP('dd/MMM/yyyy:HH:mm:ss Z'):timestamp ']'
SPACE
'\"'
LD:'http.method'
SPACE
LD:'http.target'
SPACE
LD:'http.flavor'
'\"'
SPACE
INT:'http.status_code'
SPACE
LD:'http.response_content_length'
SPACE
'\"-\" \"' LD:'http.user_agent'
'; ' LD:'http.request.header.referrer' '\"'
(SPACE LD)? (EOL | EOF)")

Log sample

{
"content":"999.999.999.999 999.99.99.999 - - [31/Jan/2024:11:08:11 +0000] \"POST /xxx.php?doing_xxxxx=1706666891.5527100563049316406250 HTTP/1.1\" 200 - \"-\" \"MyApp/6.0; https://uat.xxxxx.com\""
}

Re: Log processing rule

tjay — Wed, 31 Jan 2024 05:06:09 GMT

Even a simple parse is failing.

Process definition

PARSE(content, "IPADDR:'net.client.ip'
(SPACE LD)? (EOL | EOF)")

Log sample

{
"content":"99.999.999.999"
}

Re: Log processing rule

PacoPorro — Wed, 31 Jan 2024 07:15:33 GMT

Are you Log classic (v2) or Grail?
If grail, you should use

fetch logs
| parse content, "IPADDR:'net.client.ip' (SPACE LD)? (EOL | EOF)"

Re: Log processing rule

tjay — Wed, 31 Jan 2024 07:50:04 GMT

Thank you for your reply.

I'm doing it in the log processing and other built-in rules don't use fetch logs, etc.

I'm using the new one (I think it suggested to start using DQL from my memory).

Re: Log processing rule

PacoPorro — Wed, 31 Jan 2024 09:19:44 GMT

Try the processing with the fetch log (DQL) , should work.

Re: Log processing rule

tjay — Thu, 01 Feb 2024 00:11:31 GMT

Process Definition does not seem to like to have "fetch log".

Re: Log processing rule

tjay — Fri, 02 Feb 2024 07:56:59 GMT

It turns out that I had to include other data in the sample section, ie, not only 'content'. The easiest way is to create a new rule from the logs and events screen by clicking "Create processing rule", which brings the necessary sample data automatically.