cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Istio/Envoy Tracing with B3 (zipkin) headers

r_weber
DynaMight Champion
DynaMight Champion

Hello,

 

I'm wondering if at some point Dynatrace has changed its support for Istio/Envoys based tracing and it's ability to trace across services.

I'm working with an environment where I'd expect to see service level traces across istio/envoy proxied services. I know that envoy creates and uses the zipkin headers (x-b3-*) and not the W3C OpenTracing headers. This seems to be the default and istio doesn't support others.

According to this (dated) blog post by Thomas one should still be able to see traces across istio/envoy services.

But for me I only see process connections but nothing on service level (no incoming or outgoing service connections):

r_weber_0-1633632863492.png

The process-level connections are ok:

r_weber_1-1633632918963.png

 

I'm aware that the application itself has to take care of propagating the x-b3-* headers and the headers are propagated properly (verified by capturing them via request attributes).

So did the described behavior of the blog post change and DT only supports W3C standard or did istio change it's headers?

 

Thanks!

 

Certified Dynatrace Master, Dynatrace Partner - 360Performance.net
10 REPLIES 10

Julius_Loman
DynaMight Legend
DynaMight Legend

Hello @r_weber ,

do you have istio support enabled in settings > monitoring > monitored technologies? AFAIK it's not enabled by default.

Certified Dynatrace Master | Alanata a.s., Slovakia, Dynatrace Master Partner

Hi @Julius_Loman ,

the section in monitored tehnologies is called "Envoy" (Istio's sidecar proxies). That is enabled.

In the screenshots above we can see the envoy proxies (and the services) but no tracing connection to other services where there should be some.

 

Reinhard 

Certified Dynatrace Master, Dynatrace Partner - 360Performance.net

Hi @r_weber ,


I somehow overlooked the first screenshot and the service itself is detected. I think OneAgent uses the W3C headers and does not care about the b3 headers at all. Do you have W3C headers enabled in the environment?

Julius

Certified Dynatrace Master | Alanata a.s., Slovakia, Dynatrace Master Partner

Hi @Julius_Loman ,

 

W3C headers are active in this environment, but the Istio service mesh that is in place is set up to use zipkin headers which are injected by envoy proxies. As far as I understood the application services itself are responsible to forward any tracing headers.

I can confirm that the b3 headers are forwarded as well as the W3C traceparent header (if it exists). But if noone takes care of injecting W3C headers that would explain why there are no purepaths or services.

Question is if Dynatrace on envoy proxies would also inject W3C headers at all?

 

So if you have:

 

envoy --> static go application --> envoy --> java application 

 

Then to get a complete path you would need istio/envoy to create W3C headers instead or in addition to B3 headers for a complete service trace, right?

 

Certified Dynatrace Master, Dynatrace Partner - 360Performance.net

Hi, Did you ever get an answer / figure this one out?

@sean_mcconville unfortunately not. I do have a standing case open with Dynatrace support for some time now....they are looking into it. Doesn't seem to be too trivial given the time it takes 🙂

Fun fact:

In the very same environment the tracing worked for approx. 2 months nicely, and then stopped suddenly at 2 am in the morning across the whole environment.
Now trying to figure out if it was an auto DT update or someone changed something else...

Certified Dynatrace Master, Dynatrace Partner - 360Performance.net

Thanks for the info!
That sounds like a fun investigation! I've had a few similar ones over the years where it is hard to understand if a tenant update has caused issues.

kristopher_sarp
Observer

On Oct 21st we lost our Istio/Envoy tracing after upgrading our oneagents to 1.227 (released Oct 12).  Dynatrace engineers just confirmed that there was a change in 1.227 that caused this.  We are going to downgrade to 1.225 in hopes of restoring our tracing but at the moment Dynatrace engineers do not appear to have a solution for how to restore envoy tracing permanently.  The mechanism that was being used is deprecated.
Oneagent 1.227 supports static routes only but all envoy routes in Istio are dynamic so tracing effectively does not work in Istio at all (we just see healthchecks and prometheus scraping traces).

alexander_sagme
Observer

is there an update on this?

Appears to be fixed in the latest release of Oneagent.

  • Restored the ability to monitor dynamically added routes in certain cases such as typical Istio configurations. However, this remains unsupported (there is no guarantee this will continue to work). (ONE-67298)

https://www.dynatrace.com/support/help/whats-new/release-notes/oneagent/sprint-233#resolved-issues 

Featured Posts