topic DQL, DPL, Processing Pipleline of Logs, Bizevents...OH MY!! in DQL

DQL, DPL, Processing Pipleline of Logs, Bizevents...OH MY!!

r_weber — Wed, 29 Nov 2023 15:14:04 GMT

Hello,

I'm really struggling in getting all the different "D"-Languages and their usage in different scenarios right. Is it just me who thinks this is extremely confusing?

Example:
I can "easily" (ha!) create a DQL statement DPL Archtiect app that extracts values from a BizEvent (payload in XML format) that was captured before by this nice query:

fetch bizevents | filter event.type == "MHS" //| parse(payload, | parse payload, """DATA"<s0:processOrder" LD"<totalSum>"FLOAT:totalsum"</totalSum>" LD"<currency>"UPPER:currency"</currency>" LD"<amount>"FLOAT:payment_amount"</amount><code>"LOWER:payment_code"</code>" LD"<providerReferenceNumber>"LONG:providerReferenceNumber"</providerReferenceNumber>" "<paymentStatus>"UPPER:payment_status"</paymentStatus>" "<paymentGateway>"LOWER:payment_gateway"</paymentGateway>" LD"<country>"UPPER:country"</country>""""

Now if I want to use that in a BizEvent Ingest Processing rule I have no idea what to post there:

I sometimes have time to play, but honestly none of my customers will ever get this and also the differences betweeen the similarities of all the "D"-test input fields.

Re: DQL, DPL, Processing Pipleline of Logs, Bizevents...OH MY!!

sinisa_zubic — Wed, 29 Nov 2023 18:35:27 GMT

Hi @r_weber

Unfortunately copy & pasting the pattern is not 100% compatible. the biggest difference is with the double quotes. For the processing configuration you have to change the double quotes to single quotes. Following processor definition does not show any errors

PARSE(content,"DATA'<s0:processOrder' LD'<totalSum>'FLOAT:totalsum'</totalSum>' LD'<currency>'UPPER:currency'</currency>' LD'<amount>'FLOAT:payment_amount'</amount><code>'LOWER:payment_code'</code>' LD'<providerReferenceNumber>'LONG:providerReferenceNumber'</providerReferenceNumber>' '<paymentStatus>'UPPER:payment_status'</paymentStatus>' '<paymentGateway>'LOWER:payment_gateway'</paymentGateway>' LD'<country>'UPPER:country'</country>'")

And improvements are in the works, so in future you will be able to copy and paste the pattern between the DPL architect and processor definition without such adjustments.

Best,
Sini

Re: DQL, DPL, Processing Pipleline of Logs, Bizevents...OH MY!!

r_weber — Thu, 30 Nov 2023 14:00:59 GMT

Thanks!

Btw. I ended up with slightly different approach, parsing the right data out of an XML document (where there could be no fixed pattern found for all of these attributes):

PARSE(payload, "DATA'<SOAP-ENV:Body>'DATA:body") | PARSE(body, "DATA'<s0:'WORD:function SPACE'xmlns'") | PARSE(body, "DATA'<payment>'DATA:payment'</payment>'") | PARSE(body, "DATA'<totalSum>'FLOAT:totalsum'</totalSum>'") | PARSE(body, "DATA'<orderType>'UPPER:order_type'</orderType>'") | PARSE(payment, "DATA{0,4096}'<currency>'UPPER:currency'</currency>'") | PARSE(payment, "DATA'<amount>'FLOAT:payment_amount'</amount>'") | PARSE(payment, "DATA'<code>'WORD:payment_code'</code>'") | PARSE(payment, "DATA'<paymentStatus>'UPPER:payment_status'</paymentStatus>'") | PARSE(payment, "DATA'<paymentGateway>'LOWER:payment_gateway'</paymentGateway>'") | PARSE(body, "DATA'<country>'UPPER:country'</country>'") | PARSE(body, "DATA'<orderId>'ALNUM:orderid'</orderId>'") | PARSE(body, "DATA'<schiene>'ALNUM:rail'</schiene>'") | PARSE(body, "DATA'<mandant>'INT:mandant'</mandant>'") | FIELDS_REMOVE(payload,body,payment)

What I noticed is, that the "test" field will not work properly for XML documents, it always expects JSON and embedding a sample XML document into a JSON in this test field simply didn't work.

It might be a good idea to refine that test a bit.

Re: DQL, DPL, Processing Pipleline of Logs, Bizevents...OH MY!!

Keith_Harding — Wed, 21 Feb 2024 02:10:22 GMT

Great question - I have been struggling with this myself. Its seems madness that these two areas are using different languages that are very similar but so different at the same time.

Re: DQL, DPL, Processing Pipleline of Logs, Bizevents...OH MY!!

Keith_Harding — Wed, 21 Feb 2024 02:13:11 GMT

Hi Sini,

Thanks for the message above - at least I know I am not just going mad any more.
Do you know:

1) if there is any document that summarizes the difference and explains how to translate from one to the other.

2) if we are any closer to these languages coming into alignment.

Thanks