# The execution time of extension 2.0 created with Python longer than 60 seconds

**URL:** <https://community.dynatrace.com/t/the-execution-time-of-extension-2-0-created-with-python-longer-than-60-seconds/46196>\
**Category:** Extensions\
**Tags:** python, extensions\
**Created:** [June 18, 2024, 11:59pm UTC](https://community.dynatrace.com/t/the-execution-time-of-extension-2-0-created-with-python-longer-than-60-seconds/46196 "2024-06-18T23:59:43Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![tasuzuki](https://avatars.discourse-cdn.com/v4/letter/t/f9ae1b/32.png) [@tasuzuki](https://community.dynatrace.com/u/tasuzuki)\
**Post date:** [June 18, 2024, 11:59pm UTC](https://community.dynatrace.com/t/the-execution-time-of-extension-2-0-created-with-python-longer-than-60-seconds/46196/1 "2024-06-18T23:59:43Z")

</div>

I developed an extension that runs locally using dt-extensions-sdk.  
I have it set up so that a START log is issued at the beginning of the query function.  
After running the extension for several days in a row, I noticed that sometimes a minute’s worth of data was missing.  
When I looked at the START log in python3.log, I found that the time was as follows:

2024-06-16 22:28:59,931  
2024-06-16 22:29:59,952  
2024-06-16 22:30:59,953  
2024-06-16 22:31:59,983  
2024-06-16 22:33:00,012  
2024-06-16 22:34:00,032  
2024-06-16 22:35:00,031

There is no log that started at 22:32.  
Looking at other start logs, I found that the execution interval was between 60.00 and 60.05 seconds.  
It looks like 0.05 seconds piled up to a minute where the plugin was not executed. I think it’s strange that only the metrics from the extension are missing, even though the items that OneAgent retrieves as standard (ex: builtin:host.cpu.usage) are not missing.

I don’t know if there’s a problem with the implementation, or with OneAgent or the SDK, so please help me.

Thank you.  
Takayuki.

---

<div class="post-metadata">

**Author:** ![david\_lopes](https://dub1.discourse-cdn.com/dynatrace/user_avatar/community.dynatrace.com/david_lopes/32/27273_2.png) [@david\_lopes](https://community.dynatrace.com/u/david_lopes)\
**Post date:** [June 19, 2024, 10:45pm UTC](https://community.dynatrace.com/t/the-execution-time-of-extension-2-0-created-with-python-longer-than-60-seconds/46196/2 "2024-06-19T22:45:07Z")

</div>

Can you please share the code?

the sdk has several mechanisms where this doesn’t happen, otherwise every single dynatrace created extension would produce gaps

- methods are never scheduled to run exactly on the 00:00.00 of a minute to avoid this scenario
- even if they happen to be (very unlikely), metrics reported on scheduled methods have their timestamp recalculated to compensate for any delays

without checking what your code is doing it will not be possible to help

---

<div class="post-metadata">

**Author:** ![tasuzuki](https://avatars.discourse-cdn.com/v4/letter/t/f9ae1b/32.png) [@tasuzuki](https://community.dynatrace.com/u/tasuzuki)\
**Post date:** [June 20, 2024, 7:16am UTC](https://community.dynatrace.com/t/the-execution-time-of-extension-2-0-created-with-python-longer-than-60-seconds/46196/3 "2024-06-20T07:16:41Z")

</div>

I created a simple extension to verify the operation.

This extension mainly does the following two things.

- Outputs the START log at the beginning of the query function.  
- Registers the current time in seconds in the metric.

I check the time of the START log output every minute and determine that the plugin call time is delayed by a few milliseconds.

I check the log in /var/log/dynatrace/oneagent/extensions/datasources/custom\_{plugin name}/python3.log.

In 0.0.1, when query is called, it first registers the metric and then ends the process. The timing of the start of execution at that time was delayed by about 4 milliseconds per minute.

In 0.0.2, time.sleep(2) was added before registering the metric and time.sleep(1) was added after registering the metric.

The timing of the start of execution at that time was delayed by about 8 milliseconds per minute.

If it is delayed by about 8 milliseconds per minute, data loss may occur approximately once every 5 days and 4 hours.

I have attached the source code and execution log.  
[mysample\_extension.zip](https://community.dynatrace.com/uploads/short-url/bQdytezgb3tSqynN4acGMQXkEcd.zip) (4.14 KB)  
[mysample\_python3.log](https://community.dynatrace.com/uploads/short-url/20zR86Yl9DPSojg6CnopjZPEm42.log) (157 KB)

---

<div class="post-metadata">

**Author:** ![florent\_duchate](https://dub1.discourse-cdn.com/dynatrace/user_avatar/community.dynatrace.com/florent_duchate/32/12258_2.png) [@florent\_duchate](https://community.dynatrace.com/u/florent_duchate)\
**Post date:** [June 20, 2024, 8:30am UTC](https://community.dynatrace.com/t/the-execution-time-of-extension-2-0-created-with-python-longer-than-60-seconds/46196/4 "2024-06-20T08:30:02Z")

</div>

Hi

metric resolution within Dynatrace is 1 minute minimum so the value will be a snapshot anyway when it is taken (once per minute) so the concept of data loss makes no sense here. The time will wobble around the second but won’t creep forward slowly.

What you are pointing out is a problem when ingesting 1 minute worth of data from a log for instance. If you rely on the execution time of the query method, you could indeed lose data. This is why you want to handle the start and end time in your code and not rely on the execution time. Typically, we read older data to ensure it is all there. So if you execute at 13:45:43.05, you need to read the data from 13:44:00 and 13:45:00 for instance this way you won’t miss anything. The next minute, you might execute at 13:46:43.01 but you will read 13:45:00 to 13:46:00 etc…

---

<div class="post-metadata">

**Author:** ![tasuzuki](https://avatars.discourse-cdn.com/v4/letter/t/f9ae1b/32.png) [@tasuzuki](https://community.dynatrace.com/u/tasuzuki)\
**Post date:** [June 20, 2024, 9:16am UTC](https://community.dynatrace.com/t/the-execution-time-of-extension-2-0-created-with-python-longer-than-60-seconds/46196/5 "2024-06-20T09:16:00Z")

</div>

Yes, the minimum resolution of the metric is 1 minute. But because the query method is called every 60.0x seconds, there is a minute where data cannot be collected.

To make it easier to understand, if the query method is called every 60.1 seconds, it will look like this.

Time when the query method was called

- 2024-06-17 08:25:59.876

- 2024-06-17 08:26:59.976

- 2024-06-17 08:28:00.076

- 2024-06-17 08:29:00.176

Looking at python3.log, it looks like this, and the data for 08:27 was actually missing.

 ![tasuzuki_0-1718874830669.png](https://global.discourse-cdn.com/dynatrace/original/3X/9/d/9d1a8500379052884f10d8baf8fc3789e3862bbf.png)

Is it wrong to retrieve monitoring items from the OS in the query method and register metrics?

---

<div class="post-metadata">

**Author:** ![david\_lopes](https://dub1.discourse-cdn.com/dynatrace/user_avatar/community.dynatrace.com/david_lopes/32/27273_2.png) [@david\_lopes](https://community.dynatrace.com/u/david_lopes)\
**Post date:** [June 20, 2024, 1:25pm UTC](https://community.dynatrace.com/t/the-execution-time-of-extension-2-0-created-with-python-longer-than-60-seconds/46196/6 "2024-06-20T13:25:24Z")

</div>

Your extension is manually reporting the timestamp for the metric, and that is where your issue comes from, but lets assume folks always wanted to manually report the timestamp for their metrics (not the case), instead of letting the SDK calculate it.

This scenario you describe never happened in the logs you shared, like I mentioned before extensions are not scheduled to run at 00:00.00 of the minute.

The real logs look like:

```markup
2024-06-20 15:14:30,168 [INFO] dynatrace_extension.sdk.extension (ThreadPoolExecutor-0_0): query START mysample:0.0.2.
2024-06-20 15:15:30,173 [INFO] dynatrace_extension.sdk.extension (ThreadPoolExecutor-0_0): query START mysample:0.0.2.
2024-06-20 15:16:30,184 [INFO] dynatrace_extension.sdk.extension (ThreadPoolExecutor-0_0): query START mysample:0.0.2.

```

So there is a “cushion” of several seconds before that scenario could ever happen, 30 seconds in this case.

For that scenario to happen at 4ms per minute the extension would need to run for 7500 cycles, so because you are manually setting the timestamp of your metric, **you are responsible for reporting it every minute** , you denied the SDK the option to adjust the metric timestamp.

Even in that worst case scenario, you will lose 1 datapoint every 7500 minutes, which is acceptable because you took the risk of setting your own timestamps.

If you didn’t set the timestamp manually, the SDK calculates the metric timestamp based on the first time the method was scheduled, which guarantees that no datapoint will be lost, you can check the code [here](https://github.com/dynatrace-extensions/dt-extensions-python-sdk/blob/785c9475cd4bcd16c726779974c687fc8710c8e2/dynatrace_extension/sdk/extension.py#L990)&nbsp;and [here](https://github.com/dynatrace-extensions/dt-extensions-python-sdk/blob/785c9475cd4bcd16c726779974c687fc8710c8e2/dynatrace_extension/sdk/callback.py#L107)

In summary, remove the timestamp argument from your report\_metric method and you will never lose a datapoint

The code comments even describe the same issue you have.

 ![david_lopes_0-1718890392409.png](https://global.discourse-cdn.com/dynatrace/original/2X/9/9adc038a199af79bd1e72a22e47a44b2af7b01b6.png)

---

<div class="post-metadata">

**Author:** ![tasuzuki](https://avatars.discourse-cdn.com/v4/letter/t/f9ae1b/32.png) [@tasuzuki](https://community.dynatrace.com/u/tasuzuki)\
**Post date:** [June 21, 2024, 12:47am UTC](https://community.dynatrace.com/t/the-execution-time-of-extension-2-0-created-with-python-longer-than-60-seconds/46196/7 "2024-06-21T00:47:48Z")

</div>

Thank you. I didn’t know this mechanism.

\> In summary, remove the timestamp argument from your report\_metric method and you will never lose a datapoint

Following this advice, I modified the code as follows and ran it for about 7 hours.

```python
def query(self):
        # ### Some parts omitted ###
        now = datetime.datetime.now()
        self.report_metric(
            key=metric_key,
            value=now.second,
            dimensions=dimensions,
            # timestamp=now, # this line was commented out
        )

```

The graph for this metric now looks like this:

 ![tasuzuki_0-1718929982497.png](https://global.discourse-cdn.com/dynatrace/original/2X/0/0ecfb33f6a4e334165bf452be0da938daba4ff25.png)

However, if as you say, there is no data loss with this implementation, I will need to wait a few more hours until the start second changes from 59 to 00.

I am attaching the result of grepping this version of python3.log with START. If there are any other files that would be useful in investigating this, please let me know.  
[python3\_grep\_START.log](https://community.dynatrace.com/uploads/short-url/wOKiGKfRJdvx0POymroxKeRwuSv.log) (102 KB)

---

<div class="post-metadata">

**Author:** ![tasuzuki](https://avatars.discourse-cdn.com/v4/letter/t/f9ae1b/32.png) [@tasuzuki](https://community.dynatrace.com/u/tasuzuki)\
**Post date:** [June 21, 2024, 9:42am UTC](https://community.dynatrace.com/t/the-execution-time-of-extension-2-0-created-with-python-longer-than-60-seconds/46196/8 "2024-06-21T09:42:58Z")

</div>

Hmm, I’m in trouble.

As David\_lopes said, I removed the timestamp from the report\_metirc argument and checked to see if any metrics were missing.

```python
def query(self):
        # ### Some parts omitted ###
        now = datetime.datetime.now()
        self.report_metric(
            key=metric_key,
            value=now.second,
            dimensions=dimensions,
            # timestamp=now, # this line was commented out
        )

```

Finally, the moment when the millisecond changes from 9xx to 0xx has arrived, so take a look.

```markup
2024-06-21 17:50:59,902 [INFO] dynatra ... utor-0_0): query START mysmp2:0.0.6.
2024-06-21 17:51:59,926 [INFO] dynatra ... utor-0_0): query START mysmp2:0.0.6.
2024-06-21 17:52:59,956 [INFO] dynatra ... utor-0_0): query START mysmp2:0.0.6.
2024-06-21 17:53:59,985 [INFO] dynatra ... utor-0_0): query START mysmp2:0.0.6.
2024-06-21 17:55:00,015 [INFO] dynatra ... utor-0_0): query START mysmp2:0.0.6.
2024-06-21 17:56:00,023 [INFO] dynatra ... utor-0_0): query START mysmp2:0.0.6.
2024-06-21 17:57:00,053 [INFO] dynatra ... utor-0_0): query START mysmp2:0.0.6.
2024-06-21 17:58:00,081 [INFO] dynatra ... utor-0_0): query START mysmp2:0.0.6.
2024-06-21 17:59:00,111 [INFO] dynatra ... utor-0_0): query START mysmp2:0.0.6.
2024-06-21 18:00:00,121 [INFO] dynatra ... utor-0_0): query START mysmp2:0.0.6.

```

As you can see from the logs, the plugin did not start at 17:54.

And when I checked DataExplorer, the data at 17:55 was missing.

 ![tasuzuki_0-1718962338129.png](https://global.discourse-cdn.com/dynatrace/original/3X/e/a/eacb0394442f3669c2c8b3f472ad3cccc21d8f3e.png)

> If you didn’t set the timestamp manually, the SDK calculates the metric timestamp based on the first time the method was scheduled, which guarantees that no datapoint will be lost.

Indeed, the metric timestamp appears to be calculated automatically, but ultimately one point of data was missing.

Is it inevitable that extensions developed using the SDK will have occasional defects?&nbsp;Or is there something wrong with the way I implement it?

---

<div class="post-metadata">

**Author:** ![david\_lopes](https://dub1.discourse-cdn.com/dynatrace/user_avatar/community.dynatrace.com/david_lopes/32/27273_2.png) [@david\_lopes](https://community.dynatrace.com/u/david_lopes)\
**Post date:** [June 21, 2024, 2:31pm UTC](https://community.dynatrace.com/t/the-execution-time-of-extension-2-0-created-with-python-longer-than-60-seconds/46196/9 "2024-06-21T14:31:48Z")

</div>

Are you using the latest version of the SDK? 1.2.2 ? If so please open a github issue and we will investigate.

---

<div class="post-metadata">

**Author:** ![tasuzuki](https://avatars.discourse-cdn.com/v4/letter/t/f9ae1b/32.png) [@tasuzuki](https://community.dynatrace.com/u/tasuzuki)\
**Post date:** [June 21, 2024, 5:06pm UTC](https://community.dynatrace.com/t/the-execution-time-of-extension-2-0-created-with-python-longer-than-60-seconds/46196/10 "2024-06-21T17:06:57Z")

</div>

Yes, I tried it with SDK 1.2.2 and the result was the same.  
I’ll create an issue on GitHub.

> <https://github.com/dynatrace-extensions/dt-extensions-python-sdk/issues/69>
>
> \*\*Describe the bug\*\*
> The default execution interval is 60 seconds, but in reali…ty it is around 60.0x seconds.
> Therefore, if data collection starts at 00:00:59.995, the next data collection start time will be 00:02:00.012, and data for 00:01 will not be collected, resulting in missing data when checking metrics in Data Explorer.
> Please check this URL as it is the question I asked in the community.
> https://community.dynatrace.com/t5/Extensions/The-execution-time-of-Extension-2-0-created-with-Python-is-a/m-p/248619
> 
> \*\*To Reproduce\*\*
> Steps to reproduce the behavior:
> 1. Create an extension for OneAgent using the SDK.
> 2. Monitor any host with the extension.
> 3. If you wait a few hours or days, you will miss 1 minute of metrics.
> 
> \*\*Expected behavior\*\*
> 1-minute metrics should not be missing no matter how many days it is run in a row.
> The time when the query function is called should not be delayed no matter how many days have passed.
> I don't think it's necessary for the difference from the previous time to be 0.000 seconds, but I don't think it's acceptable for it to be delayed by 1 to 30 milliseconds each time, as is the case now.
> 
> \*\*Screenshots\*\*
> When I looked at the log, there was no start log at 17:54, and when I looked at DataExplorer, the data at 17:55 was missing.
> \`\`\`text
> 2024-06-21 17:50:59,902 \[INFO\] dynatra ... utor-0\_0): query START mysmp2:0.0.6.
> 2024-06-21 17:51:59,926 \[INFO\] dynatra ... utor-0\_0): query START mysmp2:0.0.6.
> 2024-06-21 17:52:59,956 \[INFO\] dynatra ... utor-0\_0): query START mysmp2:0.0.6.
> 2024-06-21 17:53:59,985 \[INFO\] dynatra ... utor-0\_0): query START mysmp2:0.0.6.
> 2024-06-21 17:55:00,015 \[INFO\] dynatra ... utor-0\_0): query START mysmp2:0.0.6.
> 2024-06-21 17:56:00,023 \[INFO\] dynatra ... utor-0\_0): query START mysmp2:0.0.6.
> 2024-06-21 17:57:00,053 \[INFO\] dynatra ... utor-0\_0): query START mysmp2:0.0.6.
> 2024-06-21 17:58:00,081 \[INFO\] dynatra ... utor-0\_0): query START mysmp2:0.0.6.
> 2024-06-21 17:59:00,111 \[INFO\] dynatra ... utor-0\_0): query START mysmp2:0.0.6.
> 2024-06-21 18:00:00,121 \[INFO\] dynatra ... utor-0\_0): query START mysmp2:0.0.6.
> \`\`\`
> !\[image\](https://github.com/dynatrace-extensions/dt-extensions-python-sdk/assets/8281548/90168452-6150-4331-bc7d-c3e1f8379646)
> 
> \*\*Additional context\*\*
> - OS: Amazon Linux 2 / Amazon Linux 2023 (I haven't tried it on other OSes)
> - OneAgent: 1.291
> - SDK: 1.2.2
> 
> Looking at the source code, it seemed that the cause of this problem was the use of scheduler.enter.
> For example, I thought that using scheduler.enterabs would prevent millisecond errors from accumulating.

---

<div class="post-metadata">

**Author:** ![david\_lopes](https://dub1.discourse-cdn.com/dynatrace/user_avatar/community.dynatrace.com/david_lopes/32/27273_2.png) [@david\_lopes](https://community.dynatrace.com/u/david_lopes)\
**Post date:** [June 22, 2024, 11:13pm UTC](https://community.dynatrace.com/t/the-execution-time-of-extension-2-0-created-with-python-longer-than-60-seconds/46196/11 "2024-06-22T23:13:00Z")

</div>

Thanks, we’ve published 1.2.3 which solves the accumulating timedrift
