PRO TIP: Solve "High memory usage detected" in AG extensions

[AntonioSousa]

For some reason, you might see in AG/extension logs the "High memory usage detected..." entry.

These are raised when an EF2 extension uses more memory than it is supposed to consume. The limits are documented in https://docs.dynatrace.com/docs/ingest-from/extensions20/extensions-concepts#resource-consumption

If you absolutely need to, like I did with the VMware extension, the per configuration limits can be raised with the following entry in extensionsuser.conf

dslimit_python_mem=2000

In this case, each monitoring configuration might use 2000 MB

Please beware that you have to make sure that you have sufficient memory in the AG to run it smoothly...

[Mike_L]

Depending on the extension you might want to switch out the middle part of the flag to, for example, wmi or SNMP.

The easiest way to figure out which data source it is, is either by looking at the extension.yaml file or the steps when creating the monitoring configuration.

[darshil]

Thank you for directing me to this post @AntonioSousa 

Hi @Mike_L 

 We are encountering the same error with our Salesforce extension configurations. I added the entry as suggested, but I am still seeing the error mentioned below.

Error: Method=process_all_logs: Callback Method=process_all_logs took 1691.0987 seconds to execute, which is longer than the interval of 600.0s

Our extensionsuser.conf looks like below currently:-

#STATSD configuration
statsdenabled=false
StatsdPort=18126

#Syslog configuration
syslogenabled=false

dslimit_python_mem=2000

Is the above error/issue totally different from "High memory usage detected for python datasource running configuration 58163fca-142f-39dc-8854-fd6d96571f71. Restarting" ? 

Could you please let me know what flag needs to be changed in regards to the Salesforce extension. I could not locate extension.yaml file

[Mike_L]

The two are probably not related. Your issue is that the call we do to Salesforce takes almost 30 minutes to return. Please go with a support case so we can look into if we could maybe split the call into smaller chunks, or something else.

[jeffrey_cather]

There are other configuration options that can be changed to allow more threads or change the timing interval for the VMware Extension.  Also there is an option list the dslimit_python_mem for cpu.

[Babar_Qayyum]

Hello @jeffrey_cather 

Where can I find the option to modify the interval in the VMware extension?

Regards,

Babar Qayyum

[Kenny_Gillette]

Don't forget about new dashboard for monitoring AGs - ActiveGate diagnostic overview

[Babar_Qayyum]

Hello @AntonioSousa & @mike 

We are utilizing the performance profile alongside a group of 4 ActiveGates; however, we continue to encounter the following issue (VMware Remote Monitoring). Also, there is no resources issue on any one of the host.

High memory usage detected for python datasource running configuration 723ecea2-82f0-35e5-bd20-a242cd5b0829. Restarting 

Regards,

Babar Qayyum

[AntonioSousa]

The VMware extension consumes a lot of memory. How many ESXs/VMs?

[Babar_Qayyum]

Hello @AntonioSousa 

We currently operate 133 ESXi hosts and nearly 3500 Virtual Machines. It is important to note that we do not face any resource limitations regarding polling servers.

Regards,

Babar Qayyum

[AntonioSousa]

@Babar_Qayyum ,

I an environment like that, I would say you need at least an ActiveGate with 24/32 GB. Check it out with:

• func:sfm_extension_memory_consumption:filter(and(or(eq("host.name",xxxxxx))):splitBy("dt.extension.name","dt.entity.extensions:configuration"):sort(value(auto,descending)):limit(100)

Beware that the amount of data coming in with the default extension will be very big. Of course it depends on items like storage, just be careful  🤣

[Babar_Qayyum]

Hello @AntonioSousa 

We have a group of four nodes, configured with a performance profile that includes dedicated limits of 16GB of memory for each node. Below is the average memory utilization.

Regards,

Babar Qayyum

[AntonioSousa]

@Babar_Qayyum ,

For one vCenter, the extension will only run in an AG at any given time. Try to split by AG and Monitoring Configuration in the above graph, and check if it changes substantially?
Please note that the extension running will not be distributed for a certain monitoring configuration, which is what you should have for that environment. Reducing the feature sets will reduce memory consumption. If you need to go beyond that, you will have to change the extension.

[Babar_Qayyum]

Hello @AntonioSousa 

The result presented below originates from the host, where the extension is functioning, and the second screenshot illustrates the overall health of the host.

Regards,

Babar Qayyum