Exclude specific thread processing from Dynatrace management / tagging

[vd]

We are trying to trace a request across multiple servers and one of the server calls is asynchronous. We make a rest call and in the rest service we store the request to DB and then background thread pull that request from the DB and processing it.

We could see the same tag / trace id until the request being processed by Rest service but when it pulls the request from the DB by thread it started new tag / trace id. So, we are trying to tag the thread process to the original trace and started using OneAgentSDK java API ("IncomingWebRequestTracer.traceIncomingWebRequest") to accomplish that action.

When we do that getting the following

WARN  [org.arbfile.dint.client.tracer.DynatraceLoggingCallback] (pool-19-thread-1) Dynatrace warning - IncomingWebRequestTracer with uplink received on already started path. tagging from uplink FW4;1557990956;5;1795331316;0;1;1776860673;742;abde;2h01;3h6b0294f4;4h00;5h01 is being lost. The highlighted text represent the tag id.

So question is can we exclude the thread that processing those asynchronous requests being automatically managed by Dynatrace?? I am assuming this will allow our programmatic tagging works as expected.

Any thoughts or alternative approaches are appreciated.

Thank you,

Venkat 

[Romanenkov_Al3x]

Hello.

I see no other answers, I will try to answer. Others will correct me if I'm wrong, that's just my thoughts.

I understand that your goal is to have full chain in one 'PurePath'. 

But is it really necessary to answer the question - do all systems work correctly?

 If you have 3 unrelated services - for working with this transaction, including the asynchronous part - if all 3 services work, receive traffic, there are no events/warnings/changes from the baseline - we can say that the system works correctly.

SDK/Custom service can helps to define and get "Service level" of "background thread work" and achive all the great features of this object available out of the box. 

If your goal to have 1  trace instead several - maybe it can be reached more easly with trace ingestion.  For example - for every build step CI/CD calls tracepusher and we have "response time" of pipeline execution. 

Workaround to have it in 1 trace: for example as last step of successful execution call to execute push trace with values on each step, for example with tracepusher. 

As summary you will have separated services in Dynatrace and "full trace" in external service.

Regards,

Alex Romanenkov

[vd]

Thank you, Alex, for your thoughts.

[Julius_Loman · Accepted solution]

@vd this is tricky. Since in your case the trace already started on a background thread, so there is a trace ID present. One of the options is to turn off the sensor for java threading in the process group, which stops tracing background threads and you can start the trace with the IncomingWebRequestTracer.

Anyway, since the request you mention is asynchronous, I'd use messaging API of OneAgent SDK, but you need to do that on both caller and calling sides.

There is an option to use OpenTelemetry instead, however, there will be interferences if you have both OneAgent and OpenTelemetry in your case as only one trace can "win" 😀

[vd]

Thank you, Julius for your response and will explore option to turn off sensor for Java Thread. This is not a pure asynchronous process implemented with queue / topic and we have a background thread read data from DB. We have other instances where we need to stitch JMS message processing too. I am simply trying to implement stitching the trace context in the code and do not have exposure on administration. I will reach out to our Dynatrace admin experts to change sensor configuration as you suggested. Please share if you have any document reference to update this sensor configuration. 

[Julius_Loman]

@vd  I believe it's the Java Threading OneAgent feature - which can be modified for a process group, but by disabling you may break some existing traces for you.

The (ideal) solution for that would be likely the span linking OpenTelemetry feature. Since in your case you are trying to merge two traces. The second one starts with spinning up the thread, but you want to start it when the thread picks up the record from a database. You cannot tell OneAgent not to start a trace from your code. 

[vd]

Ok. Thank you @Julius_Loman