Help! Best Practices for Handling fields in Business Event Processing

[afuentes]

Hello Community! I’m trying to create processing rules for fields corresponding to already configured business events. Specifically, I am extracting a field with the structure "[{},{},{},{},{},{},{},{},{}]", which I can’t process to get the last array/object. In the file, you can see the structure that is obtained daily. The data is collected minute by minute, with a new object being added, ending in 319 with a daily cycle, but this doesn’t affect the result.

So far, I have implemented a rather rudimentary solution where I remove the brackets and read it as JSON. But the result is not as expected, although it's close.

fetch bizevents
| filter event.provider == "VORTEX Nginx"
| fieldsAdd Array1=replaceString(res.body.result, "[", "")
| fieldsAdd result=replaceString(Array1, "]", "")
| parse result, "JSON:json"
| fieldsFlatten json
| fields json.time

In the previous query, I get the first value from all the objects. How can I process the field to extract only the last array/object?

I sincerely appreciate any help you can provide!

[Tiit_Hallas · Accepted solution]

The object in the text file is an JSON array. There is a JSON_ARRAY matcher for that in DPL that can be used without additional processing. If you want to extract ONLY the last element, you can use the following DQL query:

fetch bizevents
| filter event.provider == "VORTEX Nginx"
| parse content, """JSON_ARRAY:parsed_json_elements"""
| fields last_element = arrayLast(parsed_json_elements)

If you would like to use the JSON elements as separate fields, you can add the fieldsFlatten command to the end of the query, as follows:

fetch bizevents
| filter event.provider == "VORTEX Nginx"
| parse content, """JSON_ARRAY:parsed_json_elements"""
| fields last_element = arrayLast(parsed_json_elements)
| fieldsFlatten last_element

I created a demo for this in the Security Investigator at our Playgorund: https://wkf10640.apps.dynatrace.com/ui/apps/dynatrace.security.investigator/share/3262363c-d7a6-429a...

[afuentes]

Thanks to your comment, I was able to obtain the required information. I wasn’t aware of the use of JSON_ARRAY, but I’ve saved it for future cases.

That said, is it possible to transform the DQL query into a processing definition?

PARSE(res.body.result, "JSON_ARRAY:jsonElement")
| FIELDS_ADD(lastElement:ARRAY_LAST(jsonElement))