Netty - full PurePath visibility. (incoming request + outgoing requests/DB query in one PurePath)

[Romanenkov_Alex]

Hello community.

I have Netty process that service incoming requests. (micro-services in docker containers)

When request come, one thread service this request, another one make outgoing http connection to another service (another netty services).There are 5-10 such outgoing requests.

I have 3 services for my main Netty process. (they are: requests executed in background threads - where i can see outgoing connections via okhttp3, service (fireChannelRead).

On back Netty services i also can see service maked by default rules (fireChannelRead method)

So i have such PurePaths (example)

1) Incoming request "/purchase" on the first Netty process on service (fireChannelRead)

2) Request executed in background threads on the first Netty process

*

-> /purchase_status on the back Netty (for example status.jar)

-> /purchase_create on the back Netty (for example create.jar)

3) Request executed in background threads on the back Netty "create.jar"

* -> outgoung request to another server or DB queries.

query...

There are 3 different PurePaths - we can see “pieces”( fragments) of PurePath, but can`t get it in normal view like:

"/purchase"

-> /purchase_status

-> /purchase_create

-> request to unmonitored hosts

->DB query

I have tried to add 10-20 classess/methods to get more visibility, but there is no result.Some classess i can`t instrument - they have "for security reasons these can not be instumented.

Maybe anyone have some experience with Netty or another Java service with multi-thread execution, please answer to me, what i have done wrong?

Maybe it is case to use SDK, but i have no idea will it help to me or not. (requests are executed in different threads)

Also maybe anyone try to install Dynatrace OneAgent on server with Spring Cloud Gateway (OpenSource).

I can see only incoming requests (it is simple gateway/proxy) I can`t see outgoing requests.

Regards,

Alexander

[skrystosik]

If requests are executed in separate threads, without direct option to pass DT tag between them it may be hard to connect them into one. But maybe somebody will have some extra idea here.

Sebastian

[Arnaud_GERMAIN]

Hello Alexander,

According this link, OneAgent 1.175 support Reactor-Netty.

https://www.dynatrace.com/support/help/whats-new/release-notes/oneagent/sprint-175/

Did you try with this version if there is full PurePath visibility (incoming request + outgoing requests/DB query in one PurePath) ?

Arnaud

[bselvaraj · Accepted solution]

Hi Alex,

Need help, We are also using Spring Cloud Gateway and that uses the Netty framework and we are using agent version 1.81 still no luck to see the outgoing requests, Even with Vanilla sprint cloud gateway app also not able to see the HTTP proxy outgoing requests.

Any idea, we placed the custom services which calls the downstream services method still no luck to see the outgoing request, its only showing within API gateway asyn cals.

Please share if you have any idea and how you able to fix your environment. Thanks!

Baskaran

[Romanenkov_Alex]

Hello Baskaran.

We had support ticket about my question.

The main reason was that we can`t see request in other threads, because netty is multi thread application, with different methods of transfer execution to other threads.

Support recommended to us ask Customer change application:

• Use "thenApply" instead of "thenApplyAsync". The code will still run asynchronously in one of many "OkHttp Dispatcher" threads.
• rewrite the code and use retrofit2.Call::enqueue instead of CompletableFuture

Another way it was to use Java SDK.

Additionally R&D team added in 1.173 version "CompletableFuture" support out-of-the-box

I think it is better for you to create support ticket.

Regards,

Alexander

[bselvaraj]

Hi Alex,

Thanks for the response, We couldn't change the application level also they suggested solution is more on spring internal changes, so we don't have an option until support provide the solutions. thanks!

Regards,

Baskaran

[harald_berger · Accepted solution]

Hi,

Actually only the server side of reactor netty is supported. Spring cloud gateway uses the reactory netty http client for outgoing calls which is not detected up to now. We are working to include support for this scenario.

regards
Harry

[Arnaud_GERMAIN]

Thanks Harry for this information.

Hopefully it will be available as soon as possible.

Arnaud

[tarjei]

Checking in if there is any progress on this?

[harald_berger]

Hi, we are still working on this. There are many different scenarios to cover because of the reactive nature of the netty http client.

regards
Harry

[afguerra]

Hi Harry, I would like to know if there is any update of this case.

Regards

Andrés

[harald_berger]

Hi,

With agent 209 there is an early adopter feature called 'Reactor Netty Http Client'. Please give it a try.

Regards

Harry

[juancgalvis]

Hi,

Is this feature compatible with Istio? I have a clúster with istio enabled and it does not show the service flow. When I disable istio injection in the namespace, the service flow is shown as expected.

[harald_berger]

Hi, 

As long as Istio does propagate all additional HTTP headers (x-dynaTrace) there should not be any side effects.

Best

Harry

[bselvaraj]

Quick update, Support has the solution for spring cloud gateway to capture the outgoing requests, they enabled the Netty proxy flags on the Netty applications process group level. After the app restart, it starts detecting the downstream services calls. They told this fix will be released on the upcoming agent releases.

So if you have any issue in spring cloud gateway, Ask Support to apply the Netty proxy flag on the process group or node level, so it will detect the outgoing services calls.

I hope it will be helpful until the agent fix releases.

Regards,

Baskaran

[Stanislau_Shumi]

Hello. We had exactly the same issue.

We use Spring CLoud Gateway (based on Netty). PurePaths were broken in dynatrace. In OneAgent software support has enabled special flag 'optionEnableNettyProxyModeJava=true' which has fixed the issues.

Thanks

Stas

[afguerra]

Hello Stanislau Shumi;

Do you have an example of what you were saying?

[tarjei]

Checking in on the status here. Do you know if there has been any updates? And also do you know if the micronaut client apis are supported?