12 Apr 2022 05:14 PM - last edited on 21 Apr 2022 07:08 AM by Karolina_Linda
Hi Team,
I have enabled W3C trace context headers in settings -> server side monitoring -> Distributed tracing -> Send W3C trace context HTTP headers.
We have service invocation flow as following in Kubernetes cluster. The cluster is fully monitored by Dynatrace:
Service-A -> Service-B -> Service-C
With this setup, the Dynatrace pure path for Service-A has trace ID and Service-B, Service-C have trace ID and the corresponding Parent span ID.
Sample screen shot for pure-path of Service-A that shows Service-C:
From tracing perspective this looks perfect, but I now need to access Trace ID(for all 3 services) and Parent span ID(for Service-B and Service-C) in the application code, so that I can enhance the log messages with this metadata of trace Id and parent span Id for correlation. I have a separate central logging service to which we ship log data. But, unfortunately these values are not accessible in the request headers in the application. Is there any Dynatrace configuration to let Dynatrace send these values in the request header to application code.
If this is not possible I will end up generating this manually in the application code 😞
Thanks,
Santosh
Solved! Go to Solution.
13 Apr 2022 07:41 AM
Hi @santosh_nv,
log messages can be enhanced automatically with trace id and span id. Ff it fits your technology stack, just by enabling feature flag and configuring your log framework (if required). See more here:
Connecting log data to traces.
If it does not fit your use case for any reason, then I think you will be able to get the trace ID and span ID in your code by using opentelemetry API.
In addition I've been told the OneAgent SDK will be also enhanced to provide trace and span IDs shortly.
14 Apr 2022 05:29 PM
Hi @Julius_Loman ,
Thank you for the clear answer. The approach mentioned in the Connecting log data to traces
worked.
But, what surprises me is the impressive magic behind the scenes. I thought this might work only if logs are in Dynatrace environment. But, this also works when our application logs are shipped to a different backend logging service. The logging framework we use formats log messages into JSON and enriches with some metadata which doesn't by default have fields for trace id or span id, but after the change log messages now have 'trace.id' and 'span.id' fields. I'm curious how does this actually work?
14 Apr 2022 09:22 PM
@santosh_nv Actually enriching the log files by Dynatrace OneAgent (putting trace ID and span ID) does not require you to use Dynatrace as log ingest tool. Enriching the files and collecting log entries are two independent things.
You can enrich the log files without shipping logs to Dynatrace. You will just miss the power of Dynatrace to see all data in context in a single view - for example viewing log messages for a particular purepath directly, or viewing messages from failed requests. The power of having it in a single tool is enormous.
Under the hood - Dynatrace instruments the common logging libraries and adds the placeholders for trace and span ids. Also for structured logs, it enables those fields directly. For unstructured (plain text) logs, you mostly need to configure your logging format string, so enabling it won't break your current parsers.
14 Apr 2022 06:26 PM - edited 14 Apr 2022 06:29 PM
I am working in an environment where (frustratingly) we must send our log data to a different platform (i.e. Splunk). Therefore the Dynatrace SaaS cluster I am working with does not have "Log Monitoring v2" enabled. I am interested in including the Dynatrace's trace ID as part of our application log messages (in Java).
If the OneAgent SDK can't currently be used to get trace ID or span ID information (seemingly contrary to this documentation), is the OpenTelemetry SDK the route to take in this scenario? Is there an example of how I would do this in Java anywhere?
In the Connecting log data to traces documentation you provided, there is an Example of manually enriching Log4j log data (using the Log4j PatternFormatter). Is "Log Monitoring v2" a requirement for log enrichment?
14 Apr 2022 09:29 PM
@swin12 as I'm mentioning in my previous post - the log monitoring v2 is not required for enriching log files with trace/span ids. Those are two different things. You will only need to enable the feature globally or for desired process groups in the new oneagent features (settings > server side service monitoring > deep monitoring) and configure logging in your apps to include the fields where required. You can then ingest the logs in any other tool. You will just miss the power of having the logs in a single tool with all the other observability data which is pretty awesome. But I understand your frustration about your policy to use another tool. 😐
The OneAgent SDK does not provide you with any means to get the trace/span ID at the moment, but it should be available in a future release of OneAgent SDK. In the meantime, you will need the OpenTelemetry approach to get the trace ID in your code if required.
13 Apr 2022 07:48 AM
Sorry 😞 . You have to write in code at this moment.
14 Apr 2022 06:31 PM
Hi @Santosh I have very similar requirements, so if you have managed to solve this I would be interested in understanding what method(s) you used.
15 Apr 2022 06:17 AM
@swin12 , I already have a specific logging library(based on java logback) developed within organisation that formats messages in standard internal JSON format. So for my use case, enabling the log enrichment globally for Java apps as mentioned in the section automatic log enrichment in Connecting log data to traces offered the solution. The log data is then sent to central backend logging service via fluent bit components in our K8S cluster
11 May 2022 09:38 AM - edited 11 May 2022 10:06 AM
Hi @santosh_nv and @swin12 , I have the same concern. Indeed, I raised this idea (Improving the W3C tracecontext for log correlation) a year and a half ago (the idea contains even a picture that illustrates the lack that we perceive).
It is right that W3C Trace Context does not specify that the third field necessary for correlation -the real parent_span_id- should be introduced in the tracecontext or tracestate tracing headers. But solutions like Opentelemetry (aka. OTEL) include this field because it is necessary for correlate logs. So, how is Dynatrace correlating the spans and logs? I don't know, but I can imagine, but probably through some kind of internal pointer/reference to a some similar field. I think that Dynatrace should show the real parent_span_id at least in the PurePath, like Dynatrace does with OTEL ingested traces (see the picture picked from this Dynatrace's blog entry😞
Indeed, if you inspect the header that the One Agent inserts in message events (to trace event-driven communications), you will see that Dynatrace only inserts a kind of pointer/object, that probably contains information about the trace. This is a really big lack, because it forces us to use another complementary standard (like B3) or SDK to be able to correlate logs, because we can't extract the trace-id and span-id associated to that event-driven communication (on the contrary to HTTP communication, where you can get the trace-id and span-id at least), like I explain in this other idea (RFE - Implement W3C Trace Context to trace in Event Drive Architectures (EDAs) - Apache Kafka, Confl...)
To sum up, I'm afraid that the solution is to use OTEL SDK and delegate to it the creation, propagation and management of these headers, to be able to correlate logs out of Dynatrace. In this way, you'll be able to access to the trace_id, span_id and parent_span_id from your code (e.g., Java application).
Regards,
Luis.