cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Instrumented Java 8 Lambdas cannot be deserialized

triboue
Newcomer

It seems that instrumentation interferes and breaks the java 8 lambda deserialization process.

We get numerous errors like the following when trying to deserialize a lambda when dynatrace instrumentation is turned on (does not occur when instrumentation is disabled):

java.lang.NoSuchMethodException: policy.DefaultOrganizationImpl$$dtt.$deserializeLambda$(java.lang.invoke.SerializedLambda)
atjava.lang.Class.getDeclaredMethod(Class.java:2130) [rt.jar:1.8.0_66]
atjava.lang.invoke.SerializedLambda$1.run(SerializedLambda.java:224) [rt.jar:1.8.0_66]
atjava.lang.invoke.SerializedLambda$1.run(SerializedLambda.java:221) [rt.jar:1.8.0_66]
atjava.security.AccessController.doPrivileged(NativeMethod) [rt.jar:1.8.0_66]
atjava.lang.invoke.SerializedLambda.readResolve(SerializedLambda.java:221) [rt.jar:1.8.0_66]
atsun.reflect.GeneratedMethodAccessor2233.invoke(UnknownSource) [:1.8.0_66]
atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_66]
atjava.lang.reflect.Method.invoke(Method.java:497) [rt.jar:1.8.0_66]
atjava.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1148) [rt.jar:1.8.0_66]

Reading the javadocs of SerializedLambda:

https://docs.oracle.com/javase/8/docs/api/java/lan...

We find that the lambda is looking for a method called "$deserializeLambda$" in it's capturing class. Without Dynatrace, this capturing class would normally be (using the example above) DefaultOrganizationImpl. However, it appears dynatrace instrumentation is creating a new class DefaultOrganizationImpl$$dtt and that is what the lambda identifies as its capturing class. This dtt class apparently does not implement a $deserializeLambda$ method (the initial compiled code would have placed the method in DefaultOrganizationImpl).

The only way I've found to fix the problem so far is to write excludes at the agent level that include both the initial capturing class and all its lambdas.

-agentpath:...SETUPINFO...,exclude=starts:policy/DefaultOrganizationImpl

However, we have numerous classes that use lambdas and it's impractical to exclude all of them. Is this deserialization issue a known defect? Are there any work arounds (including a better exclude value to use in the agentpath)?

12 REPLIES 12

Joe_Hoffman
Dynatrace Champion
Dynatrace Champion

Erik,

Can you clarify which version of Dynatrace you are using? Lambda expressions are not supported by dynatrace instrumentation, but this should be properly handled by the agent automatically in v6.2+.

triboue
Newcomer

I'm using dynatrace 6.3 (problem is also reproducible under 6,2)

kurt_aigner
Dynatrace Pro
Dynatrace Pro

Hi Erik,

My I ask you to open a support ticket for this issue?

https://support.dynatrace.com/supportportal/secure...

That would be the best way to handle this.

Thanks,
Kurt

mike_gardner
Guide

Looks like we're running into the same issue with lambdas and playing whack-a-lambda-mole at the moment. I'm running the recently-released 6.5.12. Was there a resolution? Didn't see anything in the Knowledge Base if there's some type of workaround.

Thanks, Mike

triboue
Newcomer

It's been a year since I worked with it, but there was a property called com.dynatrace.diagnostics.core.instrumentation.enableJava8Transformations

on the collector config that I set to false and it seemed to fix the issue but I didn't test it thoroughly.

Thanks for the response, Erik. I will give it a shot.

Srikar_Mohan2
Helper

Ran into the same issue, here is what Support recommended.

This is an issue related to Java 8. We suggest to disable Java 8 instrumentation as follows:

-Dcom.dynatrace.diagnostics.core.instrumentation.enableJava8Transformations=false

Please add above line to dtcollector.ini and restart the collector, then restart the Java application (agent).

mike_gardner
Guide

The change suggested by Erik and Srikar worked for me, which is fantastic.

However, the follow-up question is: what blind spots do I now have by virtue of turning this off? I was unable to find documentation on what the flag does. For example, might I not see exceptions, or method execution metrics resulting from lambda or stream usage?

Hi Mike, I had the same question and this was the response from support:

The flag disables instrumentation specific to
Java 8. This is only a minor part of instrumentation, not essential to get
visibility into any JVM. In this case, Java 8 instrumentation has to be
disabled due to compatibility issues. The flag should be kept permanently. It is the suggested workaround.

IMO, that response from Support is pretty vague and doesn't satisfy, but I'm going to roll with it. Too much else to do.

Thank you (and Erik) for your time and effort with the follow-ups. Much appreciated - got me out of a jam.

Mike

FYI: In our case the debug flag introduced a major issue with HTTP request correlation over multiple JVM's (cross-process PP's are splitted). We have therefore removed it from our default collector group and proceeded with isolating all problematic Java 8 applications to use a separate collector group.

Support is aware of the issues and we are hopeful that the workarounds are only temporary.

This is with AppMon 6.5.15.1019 btw.

Joe_Hoffman
Dynatrace Champion
Dynatrace Champion

Mike, To extend what Srikar said: This switch disables instrumentation of classes that implement java8 specific functionality, such as Lambda functions. You'll likely still see exceptions as those exception classes are not java8 unique. Since Java8 specific classes will not have explicit instrumentation, the timing metrics for those classes will be based on the less accurate autoSensor technology. However these Java8 classes/methods will still show up in the purepath due to autosensor technology which does not use explicit instrumentation to get basic metrics. If (for example) your lambda functions are the root cause of performance problems in your application, I am confident you will still be able to determine that and thus focus your teams efforts appropriately.