One of our collectors normally runs around 60% heap usage. When multiple JVMs restart at once the heap spikes up to 100% and I see the following alerts:
Severe Incident occurred: Low Collector Memory.
The instrumentation of Agent 'XXX' has been refused.
The collector has around 63 Java agents reporting to it. The most recent example of when this happens was when 38 of the 63 JVMs on the collector where restarted over a 30 minute time frame during OS patching.
The collector is version 6.5 and is currently set to 4 GB heap. At one point I thought the max recommended heap for a collector was 4 GB, has this changed? Are there any issues with increasing the collector heap past 4 GB?
You can definitely add more heap, or add another collector to the group, or stagger the application restarts to help avoid this issue.
Please see this page for details:
I know of no "max", but have not seen any recommended configuration above 8 GB. I would lean towards spinning up another collector instance rather than going over 8 GB on a single instance because I would think you would get better overall throughput that way, plus you would have more failover/load balancing.
Are the best practice heap sizes for collectors the only ones supported? Can we have a Large collector with a 6 GB heap or a setup with a heap larger than 8 GB that is supported?