topic Re: Delay in sending data through EF2 extension in Extensions

Delay in sending data through EF2 extension

AntonioSousa — Mon, 12 May 2025 19:41:57 GMT

When sending data through the EF2 framework report_metric() function, I experience what I believe are delays in data being sent to Dynatrace. This is viewable in Data Explorer, where the count function shows two data points for some minutes:

But when looking at the extension logs, that shouldn't occur. I have removed in the log below the initial part, and changed sensitive info:

[out]2025-05-12 12:03:06,999 [INFO] my_ext (ThreadPoolExecutor-0_0): App Debug finishing [out]2025-05-12 12:03:36,064 [INFO] api (ThreadPoolExecutor-1_2): Sent 1 metric lines to EEC: [MintResponse(lines_ok=1, lines_invalid=0, error=None, warnings=None)] [out]2025-05-12 12:04:07,006 [INFO] my_ext (ThreadPoolExecutor-0_0): App Debug finishing [out]2025-05-12 12:04:36,041 [INFO] api (ThreadPoolExecutor-1_0): Sent 1 metric lines to EEC: [MintResponse(lines_ok=1, lines_invalid=0, error=None, warnings=None)] [out]2025-05-12 12:05:07,002 [INFO] my_ext (ThreadPoolExecutor-0_0): App Debug finishing [out]2025-05-12 12:05:36,055 [INFO] api (ThreadPoolExecutor-1_3): Sent 1 metric lines to EEC: [MintResponse(lines_ok=1, lines_invalid=0, error=None, warnings=None)] [out]2025-05-12 12:06:07,000 [INFO] my_ext (ThreadPoolExecutor-0_0): App Debug finishing [out]2025-05-12 12:06:36,049 [INFO] api (ThreadPoolExecutor-1_2): Sent 1 metric lines to EEC: [MintResponse(lines_ok=1, lines_invalid=0, error=None, warnings=None)] [out]2025-05-12 12:07:07,000 [INFO] my_ext (ThreadPoolExecutor-0_0): App Debug finishing [out]2025-05-12 12:07:36,049 [INFO] api (ThreadPoolExecutor-1_3): Sent 1 metric lines to EEC: [MintResponse(lines_ok=1, lines_invalid=0, error=None, warnings=None)] [out]2025-05-12 12:08:06,999 [INFO] my_ext (ThreadPoolExecutor-0_0): App Debug finishing [out]2025-05-12 12:08:36,054 [INFO] api (ThreadPoolExecutor-1_2): Sent 1 metric lines to EEC: [MintResponse(lines_ok=1, lines_invalid=0, error=None, warnings=None)] [out]2025-05-12 12:09:07,003 [INFO] my_ext (ThreadPoolExecutor-0_0): App Debug finishing [out]2025-05-12 12:09:36,045 [INFO] api (ThreadPoolExecutor-1_1): Sent 1 metric lines to EEC: [MintResponse(lines_ok=1, lines_invalid=0, error=None, warnings=None)] [out]2025-05-12 12:10:06,999 [INFO] my_ext (ThreadPoolExecutor-0_0): App Debug finishing [out]2025-05-12 12:10:36,057 [INFO] api (ThreadPoolExecutor-1_3): Sent 1 metric lines to EEC: [MintResponse(lines_ok=1, lines_invalid=0, error=None, warnings=None)]

And, it seems that data is only being sent out after 29 seconds...

I normally use to send data through the line protocol directly to the AG, but using report_metric() should be better...
How can I guarantee that data that is sent by the extension is immediately sent to Dynatrace?

Re: Delay in sending data through EF2 extension

g_kat — Tue, 13 May 2025 10:42:54 GMT

Hi,

I have faced scheduling issue in the past directly sending the metrics so I had to schedule them on the minute on the second to avoid it.

Maybe you could schedule it yourself to run every minute on the second using the schedule() method?

https://dynatrace-extensions.github.io/dt-extensions-python-sdk/api/extension.html#dynatrace_extension.Extension.schedule

George

Re: Delay in sending data through EF2 extension

AntonioSousa — Tue, 13 May 2025 13:46:52 GMT

@g_kat,

In my cases, I really like the fact that the processing is distributed across the seconds of a minute. What I won't want is that data is only sent after 30 seconds, or even much more, as is happening in my case.

Re: Delay in sending data through EF2 extension

g_kat — Wed, 14 May 2025 08:41:56 GMT

Do you add the timestamp on the metric line as well? report_metric() says it defaults to the current time so I guess it gets the time of the invocation, but by specifying the timestamp explicitly Dynatrace should be able to place the metric to where the timestamp is regardless on when it was processed(at least that's what support suggested in a similar case).

George

Re: Delay in sending data through EF2 extension

AntonioSousa — Wed, 14 May 2025 20:19:50 GMT

@g_kat,

Thanks for the details. In fact, looking at the definition
(via https://dynatrace-extensions.github.io/dt-extensions-python-sdk/api/extension.html#dynatrace_extension.Extension.report_metric)

the timestamp shouldn't be necessary. And given the logs above, it doesn't explain why data isn't being sent each minute.

Also, the delay in my case of more or less 29 seconds is also something I don't understand why it's happening.

Re: Delay in sending data through EF2 extension

g_kat — Wed, 14 May 2025 23:33:04 GMT

@AntonioSousa

True, I can't explain either that delay, although it doesn't seem random as based on the log it's every minute. Maybe there is something in the internal scheduling of the EEC about when it sends to the tenant.

But, even with the delay of 29-30 seconds every time, you should still be seeing the data points in Dynatarce every minute, as 1 data point would fall inside a minute. In the log, it says it sends every minute yet we see a gap at that specific timeframe.

So, maybe we have the following scenario: you get the ok response but that doesn't mean the metric is accepted in the server. It needs to be processed, and if there is not a defined timestamp it will put the timestamp of the processed time in the data point. That's why I proposed to add the timestamp in the report_metric() method. To avoid any server-side latency that may be introduced. By the way, are the data gaps happening all the time or is it a rare occurence?

George