Dynatrace & Opentelemetry - context seaming?

[danielbr]

Hi,

I'm instrumenting an app,  using Otel' SDK. Since Oneagent is not able to process spans with custom Start and End times, I am forced to export the spans to DT via the API ingest.

The result is that I have in DT a Span tree that is context-less,  that is to say - is completely disconnected from the DT Services.

Does anyone know how can I set a service context for the root span so I can see the span tree connected to the rest of the PP ?

[techean]

I believe we can set context using purepath4 ? which version are you using?

[danielbr]

Hi,

Am using SAAS so latest version:   1.247.
I am missing a way of connecting the spans to an existing PP..

Daniel

[Julius_Loman]

Hi @danielbr,
have a look at the Span entry points - these are rules how Dynatrace decides if it creates a service from a span or not.

[danielbr]

Hi Julius,

Using the Span entry points might allow me to split the spans to different services (which is great!)

But how do I get the spans to backtrace to a DT OA generated service?
I need the spans to continue an already existing PP...

Daniel

[Julius_Loman]

@danielbr backtrace / service flow are not yet available for OTEL spans. Maybe @arlindo can tell you more on this topic. I'm almost sure it is on the roadmap already.

If your distributed trace (PurePath) is started by OneAgent and continues somewhere to opentelemetry instrumented code and you are missing the link, then you likely need to setup context propagation rules. But it's difficult to advise without knowing your architecture.

[danielbr]

Hi,

My code is actually running an a OA instrumented host.
I am reporting the spans only because OA does not yet support custom start/end times of spans.  (-hence I am forced to ingest the spans via the ingest API 😞  )

I have no issue using the OA SDK if needed.   I am just unsure how am I suppose the seam the PP with the O'tel spans I am generating.   

I believe this all would have been seamless if I could simply have the OA collect the spans.... (but as mentioned-  I can't).

Daniel

[Julius_Loman]

Not sure why you need custom start/end times. I believe you have issues with the time difference and that's why Dynatrace does not merge it into single purepath. I know there are quite tight constraints when it comes to timestamps.

Can you elaborate on your use case more? Especially why you need custom timestamps and how does it relate to the services detected by oneagent. 

It does not have much 

[danielbr]

It's a bit complicated story, but i'll try to relay it as best I can...  🙂

The case is that we have an ESB product that we want to monitor.
It is a 3rd party product that is based on C++,  so there is not possible instrumentation.
The product records all its internal processes and timings, and is able to generate a log file.   The product is running on a host with IIS, which is monitored by oneagent.
So....
1) In order to create some sort of monitoring,  we have configured the tool to run an external .net app.    The product delivers all its recorded internal data, business logic and timings to that .net app.    The activation of the app is done periodically, every few minutes.

2) We wish to create some sort of implementation within that .net app, in order to construct a PP that represent what had actually occurred in the ESB product. 
So, you may think of it as some sort of a post-mortem  Purepath constructor.

3) Since DT SDK does not allow custom timings,  we have turned to Otel SDK.

Since oneagent is not capable of collecting spans with custom timings, we have ingested the spans via API.     
-This part works fine (though we may do more refinement using separation of services)

4)  Most importantly,  we would really love to have the Otel spans backtrace into the IIS requests.   The ESB product can forward the app any header (such as the x-dynatrace) in order to make the correlation.  So, it only comes to... how do we seam the spans to the PP ?

What do you think?

Daniel

[arlindo]

Hi @danielbr,

You mentioned:

---

the ESB product can forward the app any header (such as the x-dynatrace) in order to make the correlation

---

In the case of OpenTelemetry, the header that is used is the W3C header, so that would be the header to forward, not the x-dynatrace. The OneAgent also understands and propagates the W3C header. It is one of the prerequisites to use the trace ingest API, so I'm sure you already have it enabled in your environment.

Nonetheless, as you mentioned that your scenario is complicated, @danielbr, how about getting help from our product specialists, by using the in-product chat or opening a Support ticket? They can have a closer look at your environment and better help you troubleshoot the situation. You'll get to a resolution faster.

[danielbr]

Hi @arlindo ,

The instrumented IIS is first to receive the request and then only is the ESB processing it.  So it's the  spans that should backtrace to the already created PP.

So I imagine I need to seam the x-dynatrace header to the spans somehow.
The way you suggested it seems to me like the other way around?

Since I am struggling with this implementation I was hoping to find someone who might had similar experience.   I suppose I should try the ticketing method either 🙂

Daniel

[arlindo]

The trace will have both x-dynatrace and W3C headers (the OneAgent creates and propagates both). So if you make sure the ESB is propagating the W3C header that OpenTelemetry understands (OTel doesn't understand the x-dynatrace header), in theory, the OTel spans you're ingesting should be connected to the bigger trace that started with the OneAgent...

Nonetheless, via a ticket you'll have more guidance.

[Julius_Loman]

@danielbr if I got your situation, then you are just missing context propagation. I'm pretty sure you have the W3C headers enabled in Dynatrace environment - but doublecheck this first.

Then those W3C headers should be part of request from IIS to the ESB. Then you must ensure this header gets written in the log file from which you construct your opentelemetry data. In your .net app you must set the context (with the w3c headers) so Dynatrace can then properly stitch the otel spans from your .net app to the OneAgent trace (from IIS). 

So if you can ensure the w3c headers will be written into the log file (from which you now get the timings and other span data), then this is doable.

Also, be sure to add the metadata as described here unless you are already doing it.

[danielbr]

Hi @Julius_Loman ,

Yes, you got it right.

" In your .net app you must set the context (with the w3c headers)...."
How do I do that? 
I have already these metadata code lines,  but frankly i have no idea what they do or what they are good for....  😥

List<KeyValuePair<string, object>> dt_metadata = new List<KeyValuePair<string, object>>();
foreach (string name in new string[] {"dt_metadata_e617c525669e072eebe3d0f08212e8f2.properties", "/var/lib/dynatrace/enrichment/dt_metadata.properties"}) {
    try {
        foreach (string line in System.IO.File.ReadAllLines(name.StartsWith("/var") ? name : System.IO.File.ReadAllText(name))) {
            var keyvalue = line.Split("=");
            dt_metadata.Add( new KeyValuePair<string, object>(keyvalue[0], keyvalue[1]));
        }
    }
    catch { }
}

[Julius_Loman]

@danielbr  the context - not sure how exactly this is done with .NET but this is example in Java, in .Net you have some example of the context propagation usage here. Not sure where this is documented.


The metadata code is used for the enriching the Otel data with the context of process and host entity IDs which generated the otel traces. This is especially handy for licensing - because otherwise the trace ingest will cost you DDUs (Dynatrace does not know you have it already covered by OneAgent in fullstack - Dynatrace won't charge you for OTEL traces if they are coming from hosts with OneAgent in fullstack mode).