NGINX ingress segfault issue (AKS App Routing Add-on)

[bhaveshbjoshi]

Summary

We’ve observed an segfault issue impacting customers using the following ingress controller image:mcr.microsoft.com/oss/v2/ingress-nginx/controller:v1.13.9

 

Problem

• Intermittent request failures
• Frequent crashes of nginx ingress worker processes
• Kernel logs showing repeated segfaults:

kernel: nginx[4143235]: segfault at 7 ip 00007feb5b80845b sp 00007ffde8362200 error 4 in liboneagentnginx.so

Root cause (likely)

This image is part of AKS Application Routing add-on (managed NGINX ingress by Microsoft)

• It uses a patched/custom nginx binary. This image is part of AKS Application Routing add-on (managed NGINX ingress by Microsoft)
• Dynatrace NGINX code module (liboneagentnginx.so) relies on fixed assumptions about nginx internals.
• Due to this binary mismatch, instrumentation can lead to segfaults in nginx processes.
• Microsoft:  https://learn.microsoft.com/en-us/azure/aks/app-routing

 

Important notes 

• AKS App Routing is fully managed, so:
• Pod/deployment-level changes are restricted
• Forced runtime instrumentation is NOT possible

 

Resolution

Disable OneAgent injection for ingress pods/namespace

For cloudNativeFullStack / applicationMonitoring:

• Use annotations mentioned in Docs: https://docs.dynatrace.com/docs/shortlink/annotate

For classicFullStack

• Use container monitoring rules to exclude the process - https://docs.dynatrace.com/docs/shortlink/container-monitoring-rules

Disabling injection successfully stops crashes and restores stability.

What's next

• Link to Azure Github Issue - https://github.com/Azure/AKS/issues/5796
• This is currently the only supported workaround.
• In case of any other patched NGINX, please follow document
• Please open a support ticket incase of any issues/concerns along OneAgent Support archive.

[artigas]

Microsoft on the issue https://github.com/Azure/AKS/issues/5796, is claiming that is outside the AKS support scope to provided a fix for AKS App Routing, even be full managed (Pod/deployment-level changes are restricted) by AKS. @MaciejNeumann and @bhaveshbjoshi Is there any evidence, recommendation or guidance that we can provide to our customer to escalate this to AKS support? 

[bhaveshbjoshi]

Hi @artigas it seems the monitoring was not working earlier. With the changes introduced by MS it seems that OA is injecting into the process. Usually, OA doesn't inject into the patched nginx. You can verify the binaries loaded in earlier patch and impacted patch by using following command.

ldd