topic Re: "dt.trace_sampled: , dt.trace_id: , dt.span_id:" are getting appended in each property in log in Log Analytics

"dt.trace_sampled: , dt.trace_id: , dt.span_id:" are getting appended in each property in log

RajeshPr — Fri, 02 Dec 2022 09:27:48 GMT

I have installed one agent in my local and after that i am seeing "dt.trace_sampled: , dt.trace_id: , dt.span_id:" are getting appended in logs.

I am using logstash for log format

ex log:

{"@timestamp":"2022-12-01T17:30:19.403Z","level":"INFO dt.trace_sampled: , dt.trace_id: , dt.span_id: ","service":"Logback-demo dt.trace_sampled: , dt.trace_id: , dt.span_id: ","traceId":" dt.trace_sampled: , dt.trace_id: , dt.span_id: ","spanId":" dt.trace_sampled: , dt.trace_id: , dt.span_id: ","parentSpanId":" dt.trace_sampled: , dt.trace_id: , dt.span_id: ","pid":"22608 dt.trace_sampled: , dt.trace_id: , dt.span_id: ","thread":"main dt.trace_sampled: , dt.trace_id: , dt.span_id: ","class":"c.r.logbackdemo.LogbackDemoApplication dt.trace_sampled: , dt.trace_id: , dt.span_id: ","message":"dt.trace_sampled: , dt.trace_id: , dt.span_id: Starting LogbackDemoApplication using Java 11.0.15 on USHYDRAJESHPR5 with PID 22608 (C:\\Users\\prrajesh\\Downloads\\logback-demo\\logback-demo\\build\\classes\\java\\main started by prrajesh in C:\\Users\\prrajesh\\Documents\\Code Repo\\New folder\\eai-codegen\\eai-codegen)"}

logback xml configuration :

<springProperty scope="context" name="appName" source="spring.application.name"/>
<appender name="JSON_CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
    <encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
      <!--<jsonGeneratorDecorator class="net.logstash.logback.decorate.PrettyPrintingJsonGeneratorDecorator"/>-->
      <providers>
              <timestamp>
                  <timeZone>UTC</timeZone>
              </timestamp>
              <mdc>
                  <excludeMdcKeyName>dt.span_id</excludeMdcKeyName>
                  <excludeMdcKeyName>dt.trace_id</excludeMdcKeyName>
                  <excludeMdcKeyName>dt.entity.host</excludeMdcKeyName>
              </mdc>
              <pattern>
                  <pattern>
                      {
                      "level": "%level",
                      "service": "${appName:-}",
                      "traceId": "%X{traceId:-}",
                      "spanId": "%X{spanId:-}",
                      "parentSpanId": "%X{parentSpanId:-}",
                      "pid": "${PID:-}",
                      "thread": "%thread",
                      "class": "%logger{40}",
                      "message": "%message"
                      }
                  </pattern>
              </pattern>
          <stackTrace>

          </stackTrace>
          </providers>
    </encoder>
  </appender>

I dont want these new properties to be appended in my logs
Is there any way to disable this log enrichment from logstash xml instead from dynatrace console

Re: "dt.trace_sampled: , dt.trace_id: , dt.span_id:" are getting appended in each property in log

stefan_hauser — Fri, 02 Dec 2022 09:31:49 GMT

We have encountered issues with logback's logstash extension in the past whenever there is a custom pattern (as in this case) defined.

In order to resolve your issue, please disable unstructured log enrichment for this process (via "Settings" - "Preferences" - "OneAgent features" - "Java - Trace/span context enrichment for unstructured logs").

In order to have your logs enriched with DT spanId and traceId nevertheless, you can adapt your XML configuration file like this:

...
<mdc>
    <includeMdcKeyName>dt.span_id</includeMdcKeyName>
    <includeMdcKeyName>dt.trace_id</includeMdcKeyName>
    <includeMdcKeyName>dt.entity.host</includeMdcKeyName>
</mdc>
...

Our documentation regarding Logback and logstash will be updated soon.

Re: "dt.trace_sampled: , dt.trace_id: , dt.span_id:" are getting appended in each property in log

RajeshPr — Fri, 02 Dec 2022 10:12:54 GMT

is it not supported with custom pattern currently?

Re: "dt.trace_sampled: , dt.trace_id: , dt.span_id:" are getting appended in each property in log

stefan_hauser — Fri, 02 Dec 2022 10:50:26 GMT

Unstructured log enrichment in general is currently unsupported when logstash is used. The updated documentation will point that out.

Re: "dt.trace_sampled: , dt.trace_id: , dt.span_id:" are getting appended in each property in log

harald_berger — Thu, 18 Jan 2024 07:57:51 GMT

Please note that with the version 7.4 of logstash this workaround is not necessary anymore.