Summary: This post describes why Python services on Kubernetes may show only partial instrumentation. It walks through pod logs, OneAgent behavior, supported frameworks, and configuration checks needed to restore full visibility.
Hi,
I’m experiencing issues with Dynatrace instrumentation in my Python-based Kubernetes service
I have implemented tracing using the autodynatrace SDK in my FastAPI application, which runs on Uvicorn.
However, I see a lot of “no response” errors for my Kubernetes service endpoints in Dynatrace.
As verified through application logs, these errors do not correspond to actual failures.
Additionally, I’m receiving the following diagnostic error in Dynatrace:
“Some data could not be collected or transmitted. This is most likely due to resource congestion on the network, host, or process level in your monitored environment
(Diagnostic codes: C1, A5)"
Here are some details of my Kubernetes Python service:
- Framework: FastAPI.
- Server: Uvicorn.
- Autodynatrace Setup:
- Installed autodynatrace in the Dockerfile.
- Set the AUTOWRAPT_BOOTSTRAP=autodynatrace environment variable
1. What could be causing these “no response” errors, given that the endpoints function correctly?
2.Could the diagnostic error codes (C1, A5) be related to this behavior?
3. Are there additional configurations or adjustments recommended for Python FastAPI services in Kubernetes to optimize tracing and prevent these issues?
Here are OneAgent analyses:
[C/Python SDK] end() order wrong or unsupported async/gevent usage (736)
Nov 17 2024, 16:10:43 - Yesterday, 15:40:38Classification: Warning
The end() calls on the SDK tracers are in a wrong order. Can result from usage of async/await or greenlets (gevent, green threads) e.g. with Python gunicorn. Please see OneAgent-SDK-for-Python/README.md at master · Dynatrace/OneAgent-SDK-for-Python · GitHub Can also be simply forgetting to call start (tracers are not started at creation) or end, or skipping them (e.g., by exception)
Recommendation: Check that you are calling the end in the reverse order of the start and in the correct (OS) thread. In Python, prefer with blocks to explicit start/end calls.
Especially if using async/await, green thread,s or other threading abstractions, check OneAgent-SDK-for-Python/README.md at master · Dynatrace/OneAgent-SDK-for-Python · GitHub.
If using a server like gunicorn, you might be able to configure it to not use green threads (normal threads are fine) but be aware that this will have a performance impact.