EF2: High CPU usage detected for python datasource

[AntonioSousa]

I have got an extension that sometimes shows entries like the following in Logs:

Now, looking at the extension logs, the last line that is logged is 5 seconds before, after a reasonable number of metric lines have been submitted through the localhost AG API and the execution finishes:

[e03e9985-d95b-3de3-8b21-7ab185efb3c4][-8561665671057482893][409790][out]2024-10-22 23:09:44,275 [INFO] xxxxxxxxxxx (ThreadPoolExecutor-0_0): API HTTP response=202 Body Response={"linesOk":330,"linesInvalid":0,"error":null,"warnings":null}
[e03e9985-d95b-3de3-8b21-7ab185efb3c4][-8561665671057482893][409790][out]2024-10-22 23:09:44,276 [WARNING] api (ThreadPoolExecutor-0_0): Callback Method=query took 74.2712 seconds to execute, which is longer than the interval of 60.0s

Looking further in the logs, in python_statuses.log, I see the same entry, but even 2 seconds more in the future:

"processStatuses":[{"status":"HIGH_CPU","message":"High CPU usage detected for python datasource running configuration e03e9985-d95b-3de3-8b21-7ab185efb3c4","timestamp":"2024-10-22 22:09:51.964 UTC"}]}

So, how can I detect if the CPU alert is related to the execution of the python code in my extension, or something that occurs afterwards?

And, what are the consequences of these "High CPU usage detected for python datasource"?

[Caponcet · Accepted solution]

Hello Antonio,

There are 2 subjects here:

- SOFT/HARD limits for extension with context (AG or OA): on the AG you can set a "dedicated mode" to set higher limits. 

- schedule the frequency of collection to avoid issues with requests longer than 60 seconds: this is pure code

---

When you reach a HARD/SOFT limit, your monitoring configuration state will turn to red and logs will look like: 

2025-02-20 03:36:03.865ERROR

High CPU usage detected for python datasource running configuration 7e5f460c-4de4-3417-a42c-50aa79564683

ERROR extension.status dsfm SFM activegate-1 python custom:com.COMPANY.extension.solace 7e5f460c-4de4-3417-a42c-50aa79564683 HIGH_CPU 0x7399282e AG-group2 ["system.events:default"]

=> this will result in stopped ingestion. I noticed a security mecanism that delays from 10 to 60 minutes the next check, so you'll get 1 to 3 points per hours before reaching again the same limits. If you already set the AG to dedicated mode and you still face this issue, your machine is not strong enough to run it.

In the default extension template, we have a query() method using a fixed 1min time schedule. So a lot of extension first set a "frequency" variable in the activationschema

---

class ExtensionImpl(Extension😞

    def initialize(self😞

        self.extension_name = "beeware"

        DEFAULT_SCHEDULE = 1  # min

        self.extension_schedule = int(self.activation_config.get('run_schedule', DEFAULT_SCHEDULE))

        self.schedule(self.beeware_query, timedelta(minutes=self.extension_schedule))

    def beeware_query(self😞

        """

        The query method is automatically scheduled to run every minute

        """

Regards

Charles