I am grappling with an intriguing issue related to the x-dynatrace header while working with an IIS hosted web application and a custom OpenTelemetry Collector service, particularly during communications via GRPC protocol. The expanding header eventually leads to the inability to send metrics to the OT Collector as it starts rejecting requests after the header size exceeds a particular threshold (~20KB).
Environment Details:
- Web application running on IIS written in C#
- Custom Open Telemetry Collector service utilizing GRPC protocol
- .NET Framework 6.0
- Application and Collector hosted on AWS
- Dynatrace version : Dynatrace-OneAgent-Windows-1.251.218
- OpenTelemetry Libraries:
- OpenTelemetry 1.6.0
- OpenTelemetry.Exporter.OpenTelemetryProtocol 1.6.0
Issue In-depth:
- Post approximately 1 hour and 30 minutes of operation, the x-dynatrace header balloons to around 28KB in size.
- A recycling of the app pool mitigates the header size temporarily, but the augmentation cycle repeats with each subsequent request.
- Suspending the Dynatrace Agent and recycling the app pool facilitated stable communication between our application and the OT Collector.
Seeking Solutions
- Has anyone been confronted with a similar Dynatrace header expansion issue in a comparable environment?
- Are there established solutions or workarounds to restrain the header from ballooning or to handle it effectively
- Are there efficient ways to intercept and alter headers in GRPC calls, particularly in a .NET 6.0 environment, that won’t be too resource-exhaustive?
- Does a way exist to handle this issue programmatically? Specifically, I'm looking for guidance on the most effective way to manage the "x-dynatrace" header's size dynamically within our codebase.
Your insights, suggestions, or shared experiences with similar scenarios would be greatly valuable. Thank you in advance for your expertise and assistance!
