📢 Heads-Up – Upcoming changes in Istio/Envoy observability (Envoy >=1.30, Istio >= 1.22)

[stefan_penner]

TL; DR
Envoy deprecated and will removed OpenTracing. This affects the Dynatrace Envoy code module starting with Envoy version 1.30 (expected released in April 2024) and Istio version 1.22 (released in May 2024). Dynatrace offers a new solution based on OpenTelemetry for Istio/Envoy observability: https://docs.dynatrace.com/docs/shortlink/otel-integrations

Please make sure to run at minimum OneAgent version 1.283.123.20240201-075622 in case you're updating to Envoy >= 1.29.

Note: Initially, the Envoy community planned to deprecate OpenTracing already with 1.29. As this has been shifted to Envoy 1.30, this post got updated accordingly. 
 

Read more:

1. OpenTracing deprecation in Envoy
2. What about Istio?
3. Future Istio/Envoy observability with Dynatrace
4. FAQ

---

1. OpenTracing deprecation in Envoy

Envoy announced the deprecation of OpenTracing & Opencensus alongside the latest Envoy release (Envoy 1.28) in favor of OpenTelemetry:

• tracing: OpenTracing is deprecated and will be removed at version 1.30, since the upstream project has been archived.
• tracing: Opencensus is deprecated and will be removed at version 1.30, since the upstream project has been archived.

The deprecation and removal of OpenTracing in Envoy directly affects Dynatrace’s Envoy code module, as this code module is based on the OpenTracing API. The deprecation and breaking change policy in Envoy follow a 3 step approach:

• Envoy 1.28 (released on October 18, 2023):
  • Envoy will log warning messages for OpenTracing
  • Dynatrace Envoy code module / OpenTracing is fully functional and works as expected
• Envoy 1.29 (release expected in January 2024): -> Postponed to Envoy 1.30
  • Envoy per default will cause a failure when loading OpenTracing configurations
  • This marks the end-of-life for OpenTracing and the Dynatrace Envoy code module
  • Note: As of writing this community post, the community hasn’t confirmed whether this hard failure for OpenTracing will really be merged into Envoy 1.29. There are chances that this slips to Envoy 1.30. This slipped into Envoy 1.30.
• Envoy 1.30 (released on April 16, 2024):
  • OpenTracing libraries are removed from Envoy

Update: Following the initial Envoy deprecation policy, Dynatrace OneAgent 1.281 won't inject into Envoy containers with Envoy version >= 1.29 (this can overruled by support - see FAQ). Starting with OneAgent 1.283.123.20240201-075622 (or later), this has been adapted as follows:
* Injection into Envoy 1.29 should work as expected
* Injection into Envoy 1.30+ is prohibited in order to avoid any configuration failures raised by Envoy.

We recommend to update to OneAgent 1.283.123.20240201-075622 before updating to Envoy 1.29+. 

---

2. What about Istio?

Istio and other service meshes (e.g. Kong Mesh, Hashicorp Consul, AWS App Mesh, OpenServiceMesh, etc..) leverage Envoy proxies as data plane. Consequently, any change in Envoy directly affects any service mesh using upcoming Envoy versions.

Istio provides a mapping between Istio and Envoy versions in the Istio documentation:

• Envoy 1.25.x -> Istio 1.17.x
• Envoy 1.26.x -> Istio 1.18.x
• Envoy 1.27.x -> Istio 1.19.x
• Envoy 1.28.x -> Istio 1.20.x
  
• Envoy 1.29.x -> Istio 1.21.x

The next Istio version, 1.22.x, is expected to leverage uses Envoy 1.30.

---

3. Future Istio/Envoy observability with Dynatrace

Dynatrace already planned the transition from OpenTracing to OpenTelemetry ahead of time and worked on an improved Istio/envoy observability. Based on the feedback / product ideas we got from you, we’ve identified & analyzed the most important requirements:

• Support for dynamically configured routes (the Envoy code module is limited to static routes) and don’t automatically overwrite Envoy configurations in an all-or-nothing approach.
• Support (Istio-native tracing) configurations alongside the vertical Kubernetes Topology (Cluster, Namespace, Workload/Service level)

For this purpose, we heavily contributed OpenTelemetry functionalities to Envoy in the last releases (i.e. http exporter, support for resource detectors, sampling).

On top of Envoy, we’re currently contributing additional configurations to Istio in order to unlock the new Envoy Open Telemetry capabilities. We will update our documentation with detailed instructions on how-to leverage the new OTEL based Envoy/Istio observability, soon. You can find the instructions on how to configure Istio/Envoy with OpenTelemetry for Dynatrace in our documentation: https://docs.dynatrace.com/docs/shortlink/otel-integrations

New unified service detection

The new Istio observability based on OpenTelemetry can already be based on Unified Services (Dynatrace version 1.274+).

Outlook

We will continue our community contributions to Envoy/Istio and plan to add additional possibilities for intelligent sampling (Update 2024-04: Dynatrace sampling added to Envoy 1.30 / Istio 1.22). More details around the new OpenTelemetry based Envoy/Istio observability will be shared in an upcoming blog post and product documentation. 

---

4. FAQ

Can I change the Envoy configuration to still allow OpenTracing in Envoy 1.29?

Yes, according to the Envoy breaking change policy this is possible. In this case, envoy.features.enable_all_deprecated_features needs to be enabled within Envoy. Moreover, please reach out to Dynatrace support to re-enable the Dynatrace code-module injection for Envoy 1.29.

What do I need to consider when updating to Envoy 1.29?

Please make sure to run at minimum OneAgent 1.283.123.20240201-075622 in your environments before updating to Envoy 1.29. For OneAgent 1.281, deep monitoring for Envoy needs to be explicitly enabled in your environment by Dynatrace support. 

Is there any (immediate) action needed for older Envoy versions (up until Envoy 1.28)?

No action is needed. However, you can configure the OpenTelemetry tracer with http-export already in Envoy 1.28.  

Note: We’ll update this post once we have additional information/insights regarding the upcoming Istio version (Istio 1.21) and provide links to the new documentation.

[ChadTurner]

@stefan_penner Thank you for sharing this