Istio/Envoy Tracing with B3 (zipkin) headers

[r_weber]

Hello,

I'm wondering if at some point Dynatrace has changed its support for Istio/Envoys based tracing and it's ability to trace across services.

I'm working with an environment where I'd expect to see service level traces across istio/envoy proxied services. I know that envoy creates and uses the zipkin headers (x-b3-*) and not the W3C OpenTracing headers. This seems to be the default and istio doesn't support others.

According to this (dated) blog post by Thomas one should still be able to see traces across istio/envoy services.

But for me I only see process connections but nothing on service level (no incoming or outgoing service connections):

The process-level connections are ok:

I'm aware that the application itself has to take care of propagating the x-b3-* headers and the headers are propagated properly (verified by capturing them via request attributes).

So did the described behavior of the blog post change and DT only supports W3C standard or did istio change it's headers?

Thanks!

[Julius_Loman]

Hello @r_weber ,

do you have istio support enabled in settings > monitoring > monitored technologies? AFAIK it's not enabled by default.

[r_weber]

Hi @Julius_Loman ,

the section in monitored tehnologies is called "Envoy" (Istio's sidecar proxies). That is enabled.

In the screenshots above we can see the envoy proxies (and the services) but no tracing connection to other services where there should be some.

Reinhard 

[Julius_Loman]

Hi @r_weber ,

I somehow overlooked the first screenshot and the service itself is detected. I think OneAgent uses the W3C headers and does not care about the b3 headers at all. Do you have W3C headers enabled in the environment?

Julius

[r_weber]

Hi @Julius_Loman ,

W3C headers are active in this environment, but the Istio service mesh that is in place is set up to use zipkin headers which are injected by envoy proxies. As far as I understood the application services itself are responsible to forward any tracing headers.

I can confirm that the b3 headers are forwarded as well as the W3C traceparent header (if it exists). But if noone takes care of injecting W3C headers that would explain why there are no purepaths or services.

Question is if Dynatrace on envoy proxies would also inject W3C headers at all?

So if you have:

envoy --> static go application --> envoy --> java application 

Then to get a complete path you would need istio/envoy to create W3C headers instead or in addition to B3 headers for a complete service trace, right?

[sean_mcconville]

Hi, Did you ever get an answer / figure this one out?

[r_weber]

@sean_mcconville unfortunately not. I do have a standing case open with Dynatrace support for some time now....they are looking into it. Doesn't seem to be too trivial given the time it takes 🙂

Fun fact:

In the very same environment the tracing worked for approx. 2 months nicely, and then stopped suddenly at 2 am in the morning across the whole environment.
Now trying to figure out if it was an auto DT update or someone changed something else...

[sean_mcconville]

Thanks for the info!
That sounds like a fun investigation! I've had a few similar ones over the years where it is hard to understand if a tenant update has caused issues.

[kristopher_sarp]

On Oct 21st we lost our Istio/Envoy tracing after upgrading our oneagents to 1.227 (released Oct 12).  Dynatrace engineers just confirmed that there was a change in 1.227 that caused this.  We are going to downgrade to 1.225 in hopes of restoring our tracing but at the moment Dynatrace engineers do not appear to have a solution for how to restore envoy tracing permanently.  The mechanism that was being used is deprecated.
Oneagent 1.227 supports static routes only but all envoy routes in Istio are dynamic so tracing effectively does not work in Istio at all (we just see healthchecks and prometheus scraping traces).

[alexander_sagme]

is there an update on this?

[kristopher_sarp]

Appears to be fixed in the latest release of Oneagent.

• Restored the ability to monitor dynamically added routes in certain cases such as typical Istio configurations. However, this remains unsupported (there is no guarantee this will continue to work). (ONE-67298)

https://www.dynatrace.com/support/help/whats-new/release-notes/oneagent/sprint-233#resolved-issues