We have a set of AWS lambda function written in Java which are not able to post data to & do not appear in Dynatrace. However at the same time many others are able to appear on & post data to Dynatrace.
All the function are running on memory recommended which 1.5 GB (1500 MB). For brevity, I am keeping this post limited to one function which has even additional memory: 3000 MB (3 GB).
I compared cloudwatch logs of the both the function, passing & failing the Dynatrace communication. The Passing function has the logs below.
The logs of failing function have the <start-section> but do not have the <End section> of logs which executes flush and send operation. I think the process of lambda function shuts down before OneAgent can write the data to Dynatrace.
Please assists to identify the potential factors responsible, solutions or steps for further investigation on this issue.
<Function invoked: start section>
[Dynatrace] <date..> UTC [00000001] [Logger] ========= Begin logging of 8 buffered messages =========
[Dynatrace] <date..> UTC [00000001] [Global] Expanded "log-stdout"="true" from "logging/destination"="stdout"
[Dynatrace] <date..> UTC [00000001] [Global] Expanded "log-Transformer"="true" from "logging/java/flags"="log-Transformer=true,log-EmbeddedJarClassLoader=true,log-OpenTelemetryUtils=true,log-AsyncClassRetransformer=true,log-ClassValue=true"
[Dynatrace] <date..> UTC [00000001] [Global] Expanded "log-EmbeddedJarClassLoader"="true" from "logging/java/flags"="log-Transformer=true,log-EmbeddedJarClassLoader=true,log-OpenTelemetryUtils=true,log-AsyncClassRetransformer=true,log-ClassValue=true"
...
<mid section omitted for brevity>
...
<End section>
[Dynatrace] <date...> [00000001] [OpenTelemetryUtils] Force flushing spans ...
[Dynatrace] <date...> [0000000b] [OpenTelemetryUtils] Starting last chance resource callback ...
[Dynatrace] <date...> [0000000b] [OpenTelemetryUtils] + Finished last chance resource callback, adding: Resource{schemaUrl=null, attributes={}}
[Dynatrace] <date...> [0000000b] [Transformer] Opened 'module java.base' package 'sun.net.www.http' to blackbox module.
[Dynatrace] <date...> [0000000b] [Transformer] Opened 'module java.base' package 'sun.security.ssl' to blackbox module.
[Dynatrace] <date...> [0000000b] [com.dynatrace.opentelemetry.DynatraceHttpConnection] WARNING span upload: HTTPS://{environmentid}.live.dynatrace.com
DT_CONNECTION_AUTH_TOKEN: !Ref DtConnectionAuthToken
DT_LOGGING_DESTINATION: stdout
DT_LOGGING_JAVA_FLAGS: log-Transformer=true,log-EmbeddedJarClassLoader=true,log-OpenTelemetryUtils=true,log-AsyncClassRetransformer=true,log-ClassValue=true
DT_ENABLE_LAMBDA_EXTENSION_REGISTRATION: false
DT_OPEN_TELEMETRY_ENABLE_INTEGRATION: true
AWS details:
Memory: 3072MB
Ephemeral storage: 512MB
Edit:
I noticed formatting issue after the edit @aciejNeumann. The last section starting from {environment}.live.dyantrace.com is not the part of the actual log but is the set of environment variables supplied to lambda. However I am not able to fix it because the code section won't let me edit itself
