Workflow start when data is received in GRAIL, in a bucket

[y_buccellato]

I’d like to start a workflow whenever I receive a batch of data (that I’m expecting to receive every 2 minute).

Is there any way to implement such logic: ‘if you have received the batch of data in this bucket then start to do your thing’?

Regards

[Fin_Ubels]

If there is a clear start or end to this batch ingestion that could be extracted with DQL, you could create an openpipeline ingestion rule to extract a bizevent or davis event and use that as the trigger. Alternatively, if whatever does the batch ingestion can be easily customized, you could have that send an event at the end and use that as a trigger. I personally would explore just ingesting the custom event over setting up the openpipeline rules.

Docs for openpipeline processing rules: https://docs.dynatrace.com/docs/shortlink/openpipeline-processing 

Docs for ingesting custom events: https://docs.dynatrace.com/docs/shortlink/openpipeline-api-ingestion-reference#events 

[y_buccellato]

Good<variable> Fin 🙂

I'm using the most recent PowerConnect Integration (so I have a degree of flexibility but not full control over the data sent). Basically this is what is happening.

Every 2 min this integration sends a lot of log lines. The timing for the batch of log lines received can be identified by a single field sent by this integration: "current_timestamp" which will be the time in which the extraction of all the data was done and shipped to DT. And basically I'd like to trigger the workflow upon receiving this set of data with unique "current_timestamp".

But from other attempts I know that Pipeline is reasoning line by line, not group of log lines. So even if I manage to do an event, in reality it would be an event per line, which brings me back to square #1 (or 0 depending if you are a true IT guy). So from what I experienced so far I need to tackle the problem from another perspective.

For example: is there a pre.built event that says: "I'm a bucket and I have received data just now!"?
And can I do some meta configuration that help me achieve this goal of having an event every time, let's say, a bucket receive a big chunk of data?

Regards

PS
All of this is happening because my original goal is to constantly aggregate multiple line into a single line and send it as singles biz event. And basically this can only be achieved via WorkFlow (so far)

[y_buccellato]

I think I'm on something... can I use this as an event to trigger the WorkFlow?

[y_buccellato]

This one is a billing event which does not correspond to "Ei I have received data"

[ChristopherHejl]

Typically using a scheduled workflow (eg every minute) and the Run DQL Query Action could solve thise. So instead of trying to trigger based on the event (there is no bucket recieved data event, also this would trigger on every record again and not for a batch) - you to check if new data has arrived and then you do your thing, otherwise skip.

[y_buccellato]

Yes this is what I'm doing so far but the DQL query is not always successful as they way data is sent is:

theoretically the job start extracting data every 2 minute: but data might be collected for a range that is wider than two minute.

this means that if I put a logic to aggregate (summarize) log lines, each 2 minutes (trigger), for the past 2 minutes this will not summarize all the data that is being received on GRAIL.

Also the sending part (Job on SAP) is not synch with WorkFlow start time which means they will be gaps in data.

So I need to find approach that triggers on event ("I have received data and now you can start do the DQL")

Regards

[ChristopherHejl]

You mention you have a common attribute on a batch

---

log lines received can be identified by a single field sent by this integration: "current_timestamp"

---

so then you could query those together with this as a filter.
Then you grab everything and ingest as a biz event. Assuming you also add the same value on the bizevent, you could query the most recent bizevent by timestamp, use that timestamp as a filter for querying the log lines (filter current_timestamp > last bizevent timestamp) and then do your processing.
In order to not have "half" batches, you could do it a bit delays, and eg add a filter that the timestamp needs to < then now()-1min. That should be enough in practice, if this small delay is fine in your usecase.

Even better would be an event or API trigger directly on the workflow from the log line emmitter in the first place, but I beleive this should do the trick as well.

[Maheedhar_T]

Hey @y_buccellato ,
Didn't completely get your use case here but can't you use two workflows here?
The first workflow runs every minute and the filters the conditions that you need. If the conditions are met, we can trigger the second workflow using the API trigger. Basically you create the workflow with manual trigger and use the trigger with API option.

This is what I mean.

Regards,
Maheedhar.