topic Re: Conditional Parsing in DQL

Conditional Parsing

mario_rwwa — Fri, 24 May 2024 08:37:12 GMT

Currently I'm dealing with a logfile which outputs different data depending on the action which has occurred, one parse statement cannot handle all the options so I have multiple parse statements.

I need to combine it all into one table at the end, but the only option I've found so far is via multiple fetch & appends, which feels very inefficient and clunky. Is there are way to streamline this sort of query?

As an example:

fetch logs, from: -3d | filter dt.host_group.id == "myTag" | filter matchesPhrase(content, "myFirstTextToMatch") | parse content, "DATA blah blah blah parse out fields here" | append [fetch logs, from: -3d | filter dt.host_group.id == "myTag" | filter matchesPhrase(content, "mySecondTextToMatch") | parse content, "DATA blah blah blah parse out second pattern fields here" | append [fetch logs, from: -3d | filter dt.host_group.id == "myTag" | filter matchesPhrase(content, "myThirdTextToMatch") | parse content, "DATA blah blah blah parse out third pattern fields here" ... and so on

Is there a way to use a conditional operator here maybe?

ie, IF matches FirstText then parse using First pattern ELSE IF matches SecondText parse using Second pattern...

Re: Conditional Parsing

ChadTurner — Wed, 09 Jul 2025 12:42:46 GMT

@mario_rwwa - were you able to get a solution for this?

Re: Conditional Parsing

marco_irmer — Wed, 09 Jul 2025 22:02:18 GMT

Rather than the conditional parsing, you might consider parsing at ingest time instead so that the parsing can be removed from the query entirely. This can be advantageous, because you can make the parsing rules conditional to each incoming pattern. Then when you go to query, you just query the pre-parsed fields.

If that's not feasible, the other approach is to use the fieldsAdd command, combined with the if and parseAll functions to accomplish additional parsing. You'll have to do some array flattening afterwards as well, since the parseAll function will output its result into a single field as an array.

Re: Conditional Parsing

Emm4nuel — Sat, 12 Jul 2025 20:03:25 GMT

Hi Mario,

Option 1: using the parse function with if statement

data record(content = "This is a string with IP: --"), 
record(content = "This is a string with UUID: --")
 | fieldsAdd res = if(stringLength(content) == 37, parse(content, "LD IP:out"), else: parse(content, "LD UUIDSTRING:out"))

Option 2: using parsingPrerequisite


data record(content = "This is a string with IP: --"), 
record(content = "This is a string with UUID: --") 
| parse content, "LD IP:out", parsingPrerequisite: stringLength(content) == 37 
| parse content, "LD UUIDSTRING:out", parsingPrerequisite: stringLength(content) == 64

Option 3 (worst): just hope that at least one pattern works:

data record(content = "This is a string with IP: --"), 
record(content = "This is a string with UUID: --") 
| parse content, "LD IP:out", preserveFieldsOnFailure: true 
| parse content, "LD UUIDSTRING:out", preserveFieldsOnFailure: true

As of now there is no else if option available.

Credits to: Maxim Pokrovskiy