How to properly parse BizEvents JSON data with OpenPipeline

[r_weber]

I'm trying to process and extract a BizEvent payload with OpenPipeline, but it seems that OpenPipeline only supports a limited amount of DQL?

Imagine the following data content of a BizEvent:

{
"report": [
  {
    "indexName": "categories-cs_cz-search",
    "count": "896"
  },
  {
    "indexName": "categories-de_ch-search",
    "count": "957"
  },
  {
    "indexName": "categories-hr_hr-search",
    "count": "491"
  }
]
}

With DQL on a Notebook I could parse this and simply convert it to individual records (which I'd like to extract metrics from with OpenPipeline)

| parse report, "JSON:json"
| fieldsAdd searchCount = json[searchCount]
| expand searchCount
| fieldsAdd indexName=searchCount[indexName], count=searchCount[count]
| fieldsKeep timestamp,indexName, count, "event.*"

However the expand command is not supported in OpenPipeline:

This seems to contradict the intention of OpenPipeline, since BizEvents should be a first class citizen for processing?

I intended to extract multiple records with a 'count' and a 'indexName' and use the latter one as dimension on a metric, but this seems to be not possible at all?
Has someone an idea how to achieve this from a single BizEvent that carries multiple metrics? 

[Peter_Youssef]

Hello @r_weber 

I hope the attached URLs add value to current situation:

• Ingest business events via API
• business-events-metric-extraction 
• https://docs.dynatrace.com/docs/shortlink/ba-events-processing-matcher 

BR,

Peter

[r_weber]

That is unfortunately not what helps for the approach with OpenPipeline.

[Maheedhar_T]

Hey @r_weber ,
expand is just like a visualisation command that Expands an array into separate records. We should be able to parse data without expand as well. Can you try this in DQL processor definition in your Openpipeline and let me know if it worked for you ?

parse report, "JSON:json" | fieldsFlatten json | fieldsAdd searchCount = json[searchCount] | fieldsAdd indexName=searchCount[indexName], count=searchCount[count]

Regards,
@Maheedhar_T 

[r_weber]

That gives me the same result as I got before, unfortunately OpenPipeline can't work further on with this content.
What is strange also is that the query in openpipeline processor is escaping the JSON value, while on notebooks it isn't. IMO the result should be a proper JSON and not an array of json-like strings...

[Maheedhar_T]

Ah.! Now I get it. The searchCount here is an array of arrays so if you want to access indexName you have to do something like indexName=searchCount[0][indexName].

Also even though it is escaping the values with \ each of that is a JSON not a string.

you can test it out by using 

parse report, "JSON:json" 
| fieldsFlatten json 
| fieldsAdd searchCount = json[searchCount]
| fieldsAdd indexName = searchCount[0][indexName]

[r_weber]

Yes, of course I can access every position in the array with an index.
But this would still not solve the initial problem of having one business event with multiple values (and the indexName as dimension) processed in OpenPipeline.

Maybe that just doesn't work at all? And OP just has a 1:1 matching (1 event = one metric point)?

[Nick-Montana · Accepted solution]

Hey,

I'm assuming the openpipeline source is the OneAgent? This data is being captured from the business capture section within the settings tab? I think what's happening is you are capturing the request body but not configuring the individual bizevent table properties (key/values)

Can you please reply to this comment with your capture rules that you set in the settings tab? Have you looked through the docs? you can dot walk the processor from the capture rules

https://docs.dynatrace.com/docs/platform-modules/business-analytics/ba-events-capturing