Propagate trace context to child threads

[CW-Logic32]

Sorry if this has been asked before but this is a hard topic to search for.  Most of the results I got were about applications doing "background tasks" that don't need traced.  That is not the case for me.

We have a LOT of apps that do a divide-and-conquer approach to request processing.  Example:

1. A HTTP request comes in.
2. The request is broken down into some sub-components.
3. Multiple Threads are started, 1 per sub-component.
4. We wait for all the sub-requests to return.
5. Assemble a final response from all the sub-requests and return it.

We'd like to have distributed tracing properly group all the sub-requests into a single trace so Davis can properly root-cause issues with the sub-requests.  Right now, we're in a state where the parent request can experience a "problem" and that is all the information we get.  The actual issue is in one of the sub-requests but we can't drill down into it.

With OpenTracing/Zipkin, this is a super easy problem to solve.  We can just propagate the trace context to the sub-thread and all is well.  With Dynatrace, I can't seem to find how to do that propagation with the trace context automatically created by Dynatrace.

So, my ask is: how can I manually propagate the automatically created trace context to a child thread without re-inventing tracing using OpenTracing?

[Enrico_F]

Usually this should happen by default.

It looks like you are either using a framework that cannot be instrumented automatically by the injected OneAgent code module or you haven't enabled the appropriate (opt-in) OneAgent feature for it (check "Settings -> Preferences -> OneAgent features").

If it's the former case and assuming this is for a technology like Java, NodeJS, Go, PHP or .NET and you have some internal knowledge about the source code (either first-hand or via analysis of stack-traces/thread-dumps): One solution you can try is to create a custom service detection config on a (common) method/interface that gets called early in the sub-component which will run in a separate thread of the same process.

This can be done via "Settings -> Service Detection -> Custom service detection" and might require a restart of the instrumented process/service depending on the technology and/or your settings for option "Settings -> Server-side service monitoring -> Deep monitoring -> Real-time updates to Java and PHP services".

HTH

[CW-Logic32]

This is for Java and I do have full access to the source (having instrumented it for Zipkin before).  Is there a code-based solution for this?  I don't seem to have the proper level of access to see "Service Detection" and would rather empower Devs to be able to address this kind of situation on their own, as-needed.

[Eric_Yu · Accepted solution]

If you're looking to propagate the context in your code and have more control over your traces, I'd recommend trying either:

• The OneAgent SDK

or

• Use OpenTelemetry to instrument your application and send those traces to your environment. You can do this through the API or if you have the OneAgent installed in that host, you can ingest it directly through there.

Additional doc: https://docs.dynatrace.com/docs/shortlink/otel-getstarted-otlpexport#export-to-oneagent-traces-only

[CW-Logic32]

The OneAgent SDK looks like it'll suit my needs perfect.  I don't know why I didn't stumble across that on Google.  Thank you!

[Eric_Yu]

Glad, it was helpful.

You might not have as much freedom propagating the context with the OneAgent SDK. So, if you ever decide to use OpenTelemetry instead, just note that OTel is an open standard backed by the CNCF and Dynatrace is one of the top contributors to the project, so it's also a very valid alternative for this case.

Here's a blog post about this: https://www.dynatrace.com/news/blog/opentelemetry-observability-and-dynatrace-for-answers-at-scale/

[CW-Logic32]

I'll have to give that blog post a read.  I love OTel but was under the impression that OTel traces and Dynatrace-agent traces couldn't interop?  Meaning, I couldn't add a span to an existing Dynatrace trace using OTel.  The only option for custom spans like that is the OneAgent SDK.  Additionally, it incurred extra cost to report OTel traces to Dynatrace?