It seems that instrumentation interferes and breaks the java 8 lambda deserialization process.
We get numerous errors like the following when trying to deserialize a lambda when dynatrace instrumentation is turned on (does not occur when instrumentation is disabled):
Reading the javadocs of SerializedLambda:
We find that the lambda is looking for a method called "$deserializeLambda$" in it's capturing class. Without Dynatrace, this capturing class would normally be (using the example above) DefaultOrganizationImpl. However, it appears dynatrace instrumentation is creating a new class DefaultOrganizationImpl$$dtt and that is what the lambda identifies as its capturing class. This dtt class apparently does not implement a $deserializeLambda$ method (the initial compiled code would have placed the method in DefaultOrganizationImpl).
The only way I've found to fix the problem so far is to write excludes at the agent level that include both the initial capturing class and all its lambdas.
However, we have numerous classes that use lambdas and it's impractical to exclude all of them. Is this deserialization issue a known defect? Are there any work arounds (including a better exclude value to use in the agentpath)?
Can you clarify which version of Dynatrace you are using? Lambda expressions are not supported by dynatrace instrumentation, but this should be properly handled by the agent automatically in v6.2+.
Looks like we're running into the same issue with lambdas and playing whack-a-lambda-mole at the moment. I'm running the recently-released 6.5.12. Was there a resolution? Didn't see anything in the Knowledge Base if there's some type of workaround.
It's been a year since I worked with it, but there was a property called com.dynatrace.diagnostics.core.instrumentation.enableJava8Transformations
on the collector config that I set to false and it seemed to fix the issue but I didn't test it thoroughly.
Ran into the same issue, here is what Support recommended.
This is an issue related to Java 8. We suggest to disable Java 8 instrumentation as follows:
Please add above line to dtcollector.ini and restart the collector, then restart the Java application (agent).
The change suggested by Erik and Srikar worked for me, which is fantastic.
However, the follow-up question is: what blind spots do I now have by virtue of turning this off? I was unable to find documentation on what the flag does. For example, might I not see exceptions, or method execution metrics resulting from lambda or stream usage?
Hi Mike, I had the same question and this was the response from support:
The flag disables instrumentation specific to
Java 8. This is only a minor part of instrumentation, not essential to get
visibility into any JVM. In this case, Java 8 instrumentation has to be
disabled due to compatibility issues. The flag should be kept permanently. It is the suggested workaround.
IMO, that response from Support is pretty vague and doesn't satisfy, but I'm going to roll with it. Too much else to do.
Thank you (and Erik) for your time and effort with the follow-ups. Much appreciated - got me out of a jam.
FYI: In our case the debug flag introduced a major issue with HTTP request correlation over multiple JVM's (cross-process PP's are splitted). We have therefore removed it from our default collector group and proceeded with isolating all problematic Java 8 applications to use a separate collector group.
Support is aware of the issues and we are hopeful that the workarounds are only temporary.
This is with AppMon 18.104.22.1689 btw.
Mike, To extend what Srikar said: This switch disables instrumentation of classes that implement java8 specific functionality, such as Lambda functions. You'll likely still see exceptions as those exception classes are not java8 unique. Since Java8 specific classes will not have explicit instrumentation, the timing metrics for those classes will be based on the less accurate autoSensor technology. However these Java8 classes/methods will still show up in the purepath due to autosensor technology which does not use explicit instrumentation to get basic metrics. If (for example) your lambda functions are the root cause of performance problems in your application, I am confident you will still be able to determine that and thus focus your teams efforts appropriately.