topic Help! Best Practices for Handling fields in Business Event Processing in DQL

Help! Best Practices for Handling fields in Business Event Processing

tracegazer — Fri, 13 Dec 2024 08:01:54 GMT

Hello Community! I’m trying to create processing rules for fields corresponding to already configured business events. Specifically, I am extracting a field with the structure "[{},{},{},{},{},{},{},{},{}]", which I can’t process to get the last array/object. In the file, you can see the structure that is obtained daily. The data is collected minute by minute, with a new object being added, ending in 319 with a daily cycle, but this doesn’t affect the result.

So far, I have implemented a rather rudimentary solution where I remove the brackets and read it as JSON. But the result is not as expected, although it's close.

In the previous query, I get the first value from all the objects. How can I process the field to extract only the last array/object?

I sincerely appreciate any help you can provide!

Re: Help! Best Practices for Handling fields in Business Event Processing

Tiit_Hallas — Sun, 15 Dec 2024 21:32:22 GMT

The object in the text file is an JSON array. There is a JSON_ARRAY matcher for that in DPL that can be used without additional processing. If you want to extract ONLY the last element, you can use the following DQL query:

fetch bizevents | filter event.provider == "VORTEX Nginx" | parse content, """JSON_ARRAY:parsed_json_elements""" | fields last_element = arrayLast(parsed_json_elements)

If you would like to use the JSON elements as separate fields, you can add the fieldsFlatten command to the end of the query, as follows:

fetch bizevents | filter event.provider == "VORTEX Nginx" | parse content, """JSON_ARRAY:parsed_json_elements""" | fields last_element = arrayLast(parsed_json_elements) | fieldsFlatten last_element

I created a demo for this in the Security Investigator at our Playgorund: https://wkf10640.apps.dynatrace.com/ui/apps/dynatrace.security.investigator/share/3262363c-d7a6-429a-b55d-a25dc5e1a9f0

Re: Help! Best Practices for Handling fields in Business Event Processing

tracegazer — Wed, 18 Dec 2024 16:17:38 GMT

Thanks to your comment, I was able to obtain the required information. I wasn’t aware of the use of JSON_ARRAY, but I’ve saved it for future cases.

That said, is it possible to transform the DQL query into a processing definition?

PARSE(res.body.result, "JSON_ARRAY:jsonElement")
| FIELDS_ADD(lastElement:ARRAY_LAST(jsonElement))