One Agent and OpenTelemetry: how do they handle tracing headers in Java applications?

[luis_mpa]

Hi there!

After reading all Dynatrace documentation about the combination Opentelemetry (OTEL) & Dynatrace, I've a question about the One Agent behavior.

When you have a Java application, including OTEL, and it is being monitored by One Agent (in full stack mode): what is the behavior of One Agent? Specifically, I need to know if One Agent detects that there is an OTEL code, if it takes the W3C headers generated by OTEL. In this way, One Agent would not be generating headers for the application, only propagating what OpenTel creates.

This behavior is very important, especially for third-party applications where you are not able to modify their code or disable some part of their functionalities. Think that even these applications can be using the OpenT specific instrumentation, and disabling it could generate internal failures in the application.

Thank you in advance!

Luis Miguel.

[techean]

tracers components are replaced like jagger. 

[Julius_Loman]

@luis_mpa I assume you are talking about inter-process calls. Did you check the Span context propagation rules in Dynatrace? It allows you control the inter-process context propagation.

[luis_mpa]

Hi @Julius_Loman , nice to read you 🙂

Thank you for your help, Julius. I've not read this page before. So, by configuring the Span context propagation, the One Agent would be only propagating the headers that OTEL is managing, right? In this way, we would be delegating into OTEL the managing of that spans and in the One Agent the propagation, right?

This configuration is at tenant level? MZ level? Or can you choose a specific process to set?

Finally, then the following limitation would be solved by applying the configuration that you have indicated, right?

Thank you very much for your support!

[Julius_Loman · Accepted solution]

The answer is - it depends. Based on my experience, if you have OneAgent on both sides, Dynatrace will propagate the context using its headers, but it will also use W3C headers if available. If you don't have OneAgent on both sides (one side covered by OpenTelemetry only), then you will need to have W3C tracing headers enabled (enabled now by default) and ingest the traces into Dynatrace. Then Dynatrace will connect the traces when it has ingested them.

Context propagation rules are handy in the case when you for example have a custom protocol and you want Dynatrace to propagate the headers. Because Dynatrace overrides the tracers when OneAgent is in place, you need to specify when to propagate. It will propagate both W3C and Dynatrace headers.

Regarding the settings I mentioned, the configuration is done on the tenant level as far as I know. Those limitations highlighted are for in-process spans. The example below:

• OneAgent is deployed
• OpenTelemetry Java Agent is deployed too
• only internal spans (no context propagation to external services)

You can see you have a duplicate node - the QuoteController handleQuote method. This should be however configurable by span settings.

[luis_mpa]

Ok, understood. Thanks a lot for your detailed explanation.

This excerpt from your answer is specially interesting

---

@Julius_Loman wrote:

If you don't have OneAgent on both sides (one side covered by OpenTelemetry only), then you will need to have W3C tracing headers enabled (enabled now by default) and ingest the traces into Dynatrace. Then Dynatrace will connect the traces when it has ingested them.

It is very interesting because of the message-driven communications (e.g., through Kafka queues). Currently, Dynatrace is not applying W3C Trace Context to trace event-driven communications (Dynatrace stills using x-dynatrace headers). So, what happens if I have a producer built in Python -with OTEL-  and a Java consumer with OTEL too that is being monitored by the One Agent? Will Dynatrace be able to continue the tracing by applying the before configuration?

Related to that, should I disable the check "Spring for Apache Kafka" for that service to avoid unexpected effects? In this case, is something that I can set only for specific services? Think about this: this flag enables an automatic tracing of event-driven communications for Spring Boot services, but may that not all the Spring Boot services have include OTEL. So, I would need disable this feature only for Spring Boot services with OTEL. In this way, the rest of services will be being automatically traced.