topic Re: EF2 extension running slightly more than 60 seconds: way to not trigger log alert? in Extensions

EF2 extension running slightly more than 60 seconds: way to not trigger log alert?

AntonioSousa — Tue, 22 Oct 2024 21:05:17 GMT

I have an extension, which for a particular monitoring configuration, and once per hour, needs to run slightly more than 60 seconds. A log entry is created for each of the invocations, and looks like this:

Method=query: Callback Method=query took 64.1117 seconds to execute, which is longer than the interval of 60.0s

Is this something that can be configured, eventually at the monitoring configuration level?

Re: EF2 extension running slightly more than 60 seconds: way to not trigger log alert?

DavidMass — Tue, 22 Oct 2024 21:21:40 GMT

Hi @AntonioSousa,
I'm guessing you are using the query method that runs once per minute and then calculating the once per hour to run via a counter or some other method?

You can switch to the setting the main method of your extension to use be called via the schedule (https://dynatrace-extensions.github.io/dt-extensions-python-sdk/api/extension.html#dynatrace_extension.Extension.schedule) method that takes in an interval which you can either hard code or pass in via the monitoring config.

For example, you can pull the frequency from the config and use it to give yourself more minutes before the callback error is thrown.

from datetime import timedelta
...

def initialize(self): 
    frequency = self.activation_config.get("frequency", 1)
    self.schedule(callback=self.main_method, interval=timedelta(minutes=frequency), args=())

def main_method(self):
    # Do your work here

Re: EF2 extension running slightly more than 60 seconds: way to not trigger log alert?

AntonioSousa — Tue, 22 Oct 2024 21:58:10 GMT

@DavidMass,

Yes, using the strategy that was available in EF1 🙂
Going to try it out, but have questions:

The query method will always be executed right?
If yes, can we use both main_method and query
If we would like to run it every hour, is there a way to define which minute it would run?

Re: EF2 extension running slightly more than 60 seconds: way to not trigger log alert?

DavidMass — Tue, 22 Oct 2024 22:10:31 GMT

I understand 👍
With the EF2 SDK, you don't need to have a query method. It is now still a special method but it's just scheduled (using a similar method to the self.schedule way) internally within the SDK at a hard coded interval (1 minute). If it's not defined the SDK will still work.

Yes, you can have both your main_method and a query method where query would run once a minute (automatically) and then you would need to "manually" schedule your main_method. You can also schedule more than 1 method each with it's own interval.

Using that set up, no, it would be scheduled to run whenever the AG receives the configuration so there is no way to guarantee it to run at a specific minute in the hour.

Off the top of my head though, you could have a delay where in the initialize method you "sleep" until the minute you want comes up and then schedule it.

Re: EF2 extension running slightly more than 60 seconds: way to not trigger log alert?

MostafaHussein — Tue, 22 Oct 2024 23:04:46 GMT

Hello @AntonioSousa ,

when checking the extension source in github I've found that there's constants were defined in the file extension.py as the following figure

HEARTBEAT_INTERVAL = timedelta(seconds=50) METRIC_SENDING_INTERVAL = timedelta(seconds=30) SFM_METRIC_SENDING_INTERVAL = timedelta(seconds=60) TIME_DIFF_INTERVAL = timedelta(seconds=60)

after searching for `SFM_METRIC_SENDING_INTERVAL` I've found that it's used in class `Extension` as the following figure

so you can try simply redefine them with your own values within your own extension file __main__.py, I hope you get it works.

BR,

Mostafa Hussein.

Re: EF2 extension running slightly more than 60 seconds: way to not trigger log alert?

RTOWNSEND — Thu, 21 Nov 2024 22:43:55 GMT

Hello Antonio,

I created a custom XenServer extension, and I am encountering the same error message. Error shown below,
Method=query: Callback Method=query took 62.5491 seconds to execute, which is longer than the interval of 60.0s

Did you have successful results applying the fix David Mass recommended?

Re: EF2 extension running slightly more than 60 seconds: way to not trigger log alert?

AntonioSousa — Fri, 22 Nov 2024 09:46:51 GMT

@RTOWNSEND,

I have not yet done any changes from my side...

Re: EF2 extension running slightly more than 60 seconds: way to not trigger log alert?

RTOWNSEND — Mon, 25 Nov 2024 18:07:29 GMT

Hi @DavidMass ,

I applied the recommended code shown below and I am still receiving the same error Antonio reported. When manually running __,main.py__ , I can see the frequency set to 20min( Please reference the second output shown below). Any guidance you can provide would be greatly appreciated.

Thank You!

Error Message

Method=query: Callback Method=query took 85.6351 seconds to execute, which is longer than the interval of 60.0s

Code

def initialize(self):

self.extension_name = 'xensrvapi'

self.logger.info(f"{self.extension_name} initialized.")

frequency = int(self.activation_config.get("frequency", 20))

interval = timedelta(seconds=1200)

self.schedule(callback=self.query, interval=interval)

self.logger.info(f"scheduling query with an interval of {interval} (frequency: {frequency} minutes.")

Output from manual execution via VsCode

2024-11-25 11:53:06,966 [INFO] api (ThreadPoolExecutor-1_0): send_status: '{"status": "OK", "message": ""}'
2024-11-25 11:53:06,968 [INFO] xensrvapi (MainThread): xensrvapi initialized.
2024-11-25 11:53:06,968 [INFO] xensrvapi (MainThread): scheduling query with an interval of 0:20:00 (frequency: 20 minutes.
2024-11-25 11:53:06,968 [INFO] api (MainThread): -----------------------------------------------------
2024-11-25 11:53:06,969 [INFO] api (MainThread): Starting ExtensionImpl(name=xensrvapi, version=0.0.1)
2024-11-25 11:53:06,969 [INFO] api (MainThread): -----------------------------------------------------
2024-11-25 11:53:06,969 [INFO] api (ThreadPoolExecutor-1_0): send_sfm_metric: dsfm:datasource.python.threads,dt.extension.config.id="development_config_id" count,delta=3
2024-11-25 11:53:06,970 [INFO] xensrvapi (ThreadPoolExecutor-0_0): Query method started for xensrvapi.

Re: EF2 extension running slightly more than 60 seconds: way to not trigger log alert?

david_lopes — Wed, 27 Nov 2024 18:46:04 GMT

Please do NOT do this, these are internal values of the periods in which we communicate to the EEC, it has nothing to do with the original issue.

To fix the original issue just schedule methods to run for more than 60 seconds, do NOT attempt to schedule the query method itself, as that one is a special method that is always scheduled to run every 60 seconds.