topic Are BizEvents triggering workflows too early, before the BizEvent is persisted? in Automations

Are BizEvents triggering workflows too early, before the BizEvent is persisted?

r_weber — Tue, 29 Oct 2024 11:36:58 GMT

I made an interesting observation today, which seems like a race condition when triggering an SRG from a workflow.

I have this situation:

A workflow is triggered upon a bizevent. The bizevent contains some information in a JSON payload
the workflow then triggers a SRG, with objectives that query the exact same bizevent to perform analysis on the payload.
I have noticed that sometimes when the SRG executes the query for the bizevent the data seems to be not there yet? e.g. the DQL in the obejctive would get the last Bizevent of that type. Querying that bizevent in the objective with:

| sort timestamp desc | limit 1

should give us the exact last injected BizEvent. But it sometimes isn't...it's the previous one.

I have tested this by adding a time delay on the workflow, to have it wait 30s between the trigger and the SRG execution. Then this works.
So could it really be that the WF triggers are executed directly upon ingest before the bizevent is actually persisted so it can be queried by DQL?

Re: Are BizEvents triggering workflows too early, before the BizEvent is persisted?

PacoPorro — Tue, 29 Oct 2024 12:32:51 GMT

Do you mean the workflow is fired before the Bizevent is available in the Bizevents table?
Did you confirm the event firing the Workflow was present after some time?

Re: Are BizEvents triggering workflows too early, before the BizEvent is persisted?

ChristopherHejl — Tue, 29 Oct 2024 13:57:40 GMT

You are correct. The workflow is triggered as soon as the event is coming in, but Grail might take 10-15 seconds until the new data point is stored and is returned by queries.
We are looking into delaying event triggers internally to get around this, but not confirmed yet.

Re: Are BizEvents triggering workflows too early, before the BizEvent is persisted?

r_weber — Tue, 29 Oct 2024 14:34:32 GMT

@PacoPorro yes that is correct and confirmed by @ChristopherHejl .
This is important to know, because with this behavior it means that a new BizEvent can trigger a workflow which then works on old data 🙂
Simple workaround: you need to add a delay on the srg execution task.

Re: Are BizEvents triggering workflows too early, before the BizEvent is persisted?

r_weber — Tue, 29 Oct 2024 14:34:52 GMT

Thanks for confirming this!