topic Re: Traces on Grail: getting the "processing" time in Open Q&A

Traces on Grail: getting the "processing" time

r_weber — Tue, 16 Apr 2024 14:55:42 GMT

I was just asking myself how to get the overall processing time of a trace. In Grail spans and traces all have a duration field but there is no knowledge of what runs asynchronously or sequentially.

Imagine this trace in the UI:

With the async stuff going on the "response time" is 30.7ms. One can see this on the root span of the trace as well. But then there are a lot of async calls and the actual time I'd be interested is the "Processing time" which is ~1s.

If I want to know how much time is spent on a individual service int total I can't just sum up the "duration" of these service calls, neither can I add up all span durations for the overall duration of the trace?

Any ideas how to solve this? How could one get the duration/processing time of this trace using a query?

Re: Traces on Grail: getting the "processing" time

imrj99 — Tue, 10 Dec 2024 19:54:43 GMT

anyone figured this out?

Re: Traces on Grail: getting the "processing" time

Andreas_Gruber — Fri, 24 Jan 2025 13:28:18 GMT

@r_weber the processing time as you describe it would be the difference of the earliest start and latest end time of the spans of a trace when I got your question right. If you want to have this also for the spans of a the services within a trace you need to make a multi step summarize query.

I tried this in DQL and would get this result.

fetch spans
| summarize {request.start_time = min(start_time), 
request.end_time = max(end_time)}, by: {trace.id, dt.entity.service}
| summarize {trace.start_time = min(request.start_time), 
trace.end_time = max(request.end_time), 
request.processing_times = collectArray(record(dt.entity.service, request.start_time, request.end_time, request.processing_time = request.end_time-request.start_time))}, by: {trace.id}
| fieldsAdd trace.processing_time = trace.end_time - trace.start_time

Does this help? For a single trace you could add a trace.id filter right after the fetch.