I'm in the process of upgrading our application servers from wildfly12 to wildfly18.0.1, and upgrading from java8 to java11 at the same time. On all 3 tiers (int/qa/prod) we have Dynatrace AppMon agents instrumenting the application servers.
The servers start without incident without the AppMon agent enabled. Once the AppMon agent is added to the startup config, the server fails to load some critical logger components and won't start. The first sign of trouble is the warning below:
2019-12-27 16:11:50.914 UTC [0000270c] info [java ] [agent ] Detected Java version : Java Eclipse OpenJ9 11.0.5
2019-12-27 16:11:50.916 UTC [0000270c] info [java ] [uem ] new JS agent versions retrieved: 7000200171032. Clearing cache.
WARNING: Failed to load the specified log manager class org.jboss.logmanager.LogManager
2019-12-27 16:11:51.807 UTC [0000270c] info [java ] [reflector ] Reflector caching mode: on
2019-12-27 16:11:51.809 UTC [0000270c] info [java ] [reflector ] Reflector access mode: memberIterating
2019-12-27 16:11:51.810 UTC [0000270c] info [java ] [agent ] Detected WildFly version : WildFly 18.0.1.Final
Dec 27, 2019 10:11:51 AM org.jboss.msc.service.ServiceContainerImpl <clinit>
INFO: JBoss MSC version 1.4.11.Final
Later on this leads to:
ERROR: WFLYCTL0013: Operation ("parallel-extension-add") failed - address: ()
java.lang.RuntimeException: WFLYCTL0079: Failed initializing module org.jboss.as.logging
and the server terminates before it ever gets started. If I remove the -agentpath:<dt_agent_path>... from my startup config, the server runs fine and the applications behave as expected.
I'm attaching a full log of the startup. Please let me know what other support artifacts I can provide to assist in resolving this. Thanks for any assistance you can provide.
I can confirm this occurs when using the eclipse j9 VM but not the hotspot VM. Tested with the latest LTS release of Java 11 from adoptOpenJdk.net . Server starts fine with Dynatrace on the hotspot VM, but throws the error above when using the J9 VM. Using the hotspot VM is a suitable workaround for the time being but long-term we wanted to migrate to the J9 VM.
You could try excluding the log class from any instrumentation. But this may be a symptom, not the root cause. Worth a try. Here's some samples of using the exclude parameter on the -agentpath command line string.
-agentpath:....,exclude="starts:com/dynatrace/diagnostics/foobar;contains:Proxy" would exclude classes that start with com.dynatrace.diagnostics.foobar and classes that contain the string "Proxy" in their name.
Quotes are optional (the following is also possible):
I may also suggest that you open a support case to provide better guidance in diagnosing this conflict.
It would certainly be useful to support to know whether using Exclude resolves the issue.
Hi Joseph, Thanks for the suggestion. I tried several versions of the excludes including differing packages and class loaders where I thought the issue was bubbling up from. Ultimately that didn't fix the issue, although it did change the way the error manifested and/or made it occur sooner or later in the startup process.
Ultimately, after checking some other resources, I came upon this solution which has the server starting with the J9 VM and the dynatrace agent. in standalone.conf.bat, add this line:
set "JAVA_OPTS=%JAVA_OPTS% -Djboss.modules.system.pkgs=org.jboss.logmanager -Xbootclasspath/a:%JBOSS_HOME%/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-2.1.14.Final.jar;%JBOSS_HOME%/modules/system/layers/base/org/wildfly/common/main/wildfly-common-1.5.1.Final.jar -Dsun.util.logging.disableCallerCheck=true -Djava.util.logging.manager=org.jboss.logmanager.LogManager"
With those config changes in place, the server will start with the dynatrace agent installed. I believe the root cause of this is that when instrumenting, the dynatrace agent is causing the modules classloader to see the logging manager prior to the system classloader, which isn't what wildfly expects. Might actually be a bug on their end based on some issues i found in their bug tracker (https://issues.redhat.com/browse/MODULES-393 and https://issues.redhat.com/projects/LOGMGR/issues/LOGMGR-254)
There is still one error being displayed at startup but it appears to be harmless
14:09:41,726 INFO [org.jboss.modules] (main) JBoss Modules version 1.9.1.Final
Dec 30, 2019 2:09:41 PM java.lang.System$LoggerFinder lambda$getLoggerFinder$0
WARNING: Failed to instantiate LoggerFinder provider; Using default.
I did try adding an exclude=contains:LoggerFinder snippet to my dyantrace agent config, that didnt' impact the warning.