I am creating an example project to play with the OTEL tracing possibilities in Dynatrace with OneAgent installed. But supporting the use case of linking the traces together when an unsupported protocol is used.
At the calling side I write the current tracing context to a text file.
The callee side reads this text file and sets the info (traceparent, tracestate) as parent for it's span.
For the sake of simplicity I made it two separate console apps. I configured custom services on both apps as entry-point.
It's complex to explain but when I use ActivityKind.Internal for the span at the caller side, the parentcontext that is written to the textfile is the same as the current activityId. When I use ActivityKind.Client (which is what I want) the parentcontext is not the same as the current activityId and it also contains an X-dynaTrace baggage and tracestate.
I configured the ActivityKind.Client type in Dynatrace as to be Propagated via "Span context propagation" in settings. So, that's probably the reason.
Any way, both with Internal as with Client, the distributed traces are not connected.
Any ideas or tips?
Solved! Go to Solution.
kindclient.png is the output of the caller when I use ActivityKind.Client. As you can see the traceparent which is written via the TextMapPropagator is not the same as the Id of the current Activity. Which is strange.
kindinternal.png is the output of the caller when I use ActivityKind.Internal. Here it is the same
But no X-dynaTrace context ...
What does Dynatrace require for the contexts to get linked
The reason for this is that our .NET / OpenTelemetry integration currently has a limitation: Spans cannot act as PurePath entrypoints. That means spans can only be embedded into existing PurePaths. Custom context propagation on CLIENT spans via propagators also works (if properly configured). But picking up a SERVER span, without pre-existing PurePath does not work at the moment.
The good news is that this limitation is scheduled to be fixed with OneAgent 1.241. There will be a new .NET feature called "Improved .NET In-Process Context Propagation" in the OneAgent Features section. When enabled, span as PurePath entrypoints will work. I've picked the example code here and it worked as expected (with that feature enabled). OneAgent 1.241 is scheduled to be rolled out ~end of May.
In the meantime, using the OneAgent SDK for .NET would be the alternative.
Hello @Bert_VanderHeyd ,
For external calls you should use Span Kind Client, then pass the context. In the first example, you have a different trace id in the traceparent header. I believe you are starting a new trace instead of a span.
I'm not familiar with .NET OTEL API, but have you checked this example? I believe it's just the situation you need.
I am actually using that example as a basis indeed. It used Kind.Producer in the example at the caller side. But Client or Producer is quite the same effect as I configured them in Dynatrace to Propagate context. See here
In .NET it works with Activities which are Spans in fact. Could it be that Dynatrace starts a new trace when this PropagationContext passing option is set. I only start a new Activity.
I understand. What is your outut if you run the code outside oneagent with OpenTelemetry instrumentation only? (you don't have to ingest traces, just compare the trace id)
The output without oneagent is the same as in the kindinternal.png. So then then the traceId's match. Conclusion: it's the contextpropagation setting in Dynatrace that messes it up. Off course, with the contextpropagation enabled I see the X-dynaTrace header which seen as OTEL baggage. Maybe I need to do something with that on the callee side.
@Bert_VanderHeyd context propagation is required here. You will see both W3C headers and X-dynatrace header in the baggage. That's normal.
I played with a similar scenario in Java and it works. Context propagation was required. Otherwise, I got no context to propagate.
Maybe @arlindo or @sonja can explain why this is happening.