ActiveGate Proxy & Plugins: Webserver Pool is exhausted

[gilles_tabary]

Hello.

I have oversized my host running my only one Environment ActiveGate Proxy & Plugins (10 GB RAM, 4cpu) to serve about ~400 hosts yet, I still get some WARNING like  'Webserver Pool' is exhausted :

2023-07-05 04:03:50 UTC WARNING [<41d6c1a6-a709-4874-9c13-c7bf69655d91>] [<collector.core>, CollectorSelfMonitoringUpdater] 'Webserver Pool' is exhausted (active threads = 200, max threads = 200), queue size = 187 [Suppressing further messages for 5 minutes]
2023-07-05 10:04:30 UTC WARNING [<41d6c1a6-a709-4874-9c13-c7bf69655d91>] [<collector.core>, CollectorSelfMonitoringUpdater] 'Webserver Pool' is exhausted (active threads = 200, max threads = 200), queue size = 110 [Suppressing further messages for 5 minutes]

I already have a other AG (Cluster) in failover which can be contacted by OneAgent's. Which BTW also get some Webserver Pool is exhausted.

The thing is if both AGs gets Webserver Pool exhausted, I risk getting a full Environment monitoring interruption : severe !

Also, I am planning to have a second Environment AG Proxy & Plugins as a second node. But if not any one node can hadle singlehandedly the load, I will still risk an "Environment monitoring interruption" when AG or underlying OS'es gets upgraded and restarted. Gosh... where does that end !? 🙂

Any one experiencing this ? Any solutions ?

Regards.

For the record : https://one.dynatrace.com/hc/en-us/requests/187443

[Lukasz_Halman]

I think it's best to investigate your issue through the ticket as it requires deeper analysis why this is happening. Generally the message you quoted is a sign of ActiveGate webserver reaching 100% of the thread limit. It does not mean there are requests rejected and your environment is unmonitored. It's often seen early after ActiveGate process restart and is related to spike of ssl handshakes.

When it comes to sizing, while 400 hosts doesn't sound like a lot, please keep in mind that it is not the number of hosts themselves, but the number of monitored applications & their activity that is a factor here. Recommendation is based on typical values we see in production, your milage may vary here.

Agents will not balance between Environment and Cluster ActiveGates. Environment AGs are preferred if reachable and agents will only switch over in case of connection issue.