Outbound calls in Services app : no grouping of URLs ?

[DavidFT]

Hello,

I tried to figure out how the "Outbound calls" tab in the Services app works. Specifically, I don't understand why there are so many lines whereas the endpoint on the called side is well configured.

Example :

On the called side, here is the endpoint :

Moreover, on the caller side the request rate is 5120/min for each line, which is of course wrong since it's 1 or 2 in total for each line.

Hence the following questions :

1. Why isn't there a grouping in Outbound calls similar to the one in Endpoint section of the called service ?
2. Where does the request rate come from ?
3. Why is the column "Called service" often (but not always !) empty ?

[sujit_k_singh]

Hi @DavidFT 

1. Why isn't there a grouping in Outbound calls similar to the one in Endpoint section of the called service ? 

   The Outbound calls tab and the Endpoints view use different aggregation logic.

   Inbound endpoints are grouped using stable route patterns.

   Outbound calls, on the other hand, are grouped based on the URL observed by the caller. If the target URL contains dynamic path segments or other variations, each distinct URL may appear as a separate row. As a result, many outbound call entries can map to the same endpoint on the receiving service, creating the appearance of higher cardinality on the caller side.

2. Where does the request rate come from ? 

   The request rate shown in Outbound calls is a service-level aggregate across all process group instances of the calling service.

   For example, if 2,560 instances of a service each generate 2 outbound calls per minute to the same target, Dynatrace will display a request rate of 5,120 requests/minute. This value represents the total traffic across all instances rather than the rate produced by a single process or pod.

3. Why is the column "Called service" often (but not always !) empty ? 

   The Called service field is populated only when Dynatrace can correlate the outbound request to a monitored service entity.

   This typically requires:

   1. Monitoring on the target side (OneAgent or OpenTelemetry instrumentation).
   2. Successful trace-context propagation between caller and callee.
   3. A matching server-side span that Dynatrace can associate with a service.

   If any of these conditions are not met, Dynatrace can still display the outbound call but may not be able to identify the corresponding service, leaving the Called service column empty.

Thanks,

Sujit

[DavidFT]

Hi @sujit_k_singh,

Thank you very much for your quick reply.

If we step into a final user's shoes, this behaviour is hard to understand : 

1. I have to deal with different behaviours depending on which side I am, caller or called.
2. The request rate is unreliable.

About your answer on the third point, it feels very strange : each call is made with the exact same manner. So if the called service is displayed once, it should be displayed every time, for each line.

[sujit_k_singh]

Hi @DavidFT 

I did some additional trace analysis and found an interesting difference

From trace analysis, Dynatrace appears to populate the Called service column either from the supportability.external.dt.entity.service attribute (when present on the client span) or by correlating client and server spans belonging to the same trace.

Span Type	supportability.external.dt.entity.service	Called Service likely visible
HTTP GET	Present	Yes
CONNECT TradeManagement	Missing	No

In the examples above, HTTP client spans contain supportability.external.dt.entity.service, while the CONNECT TradeManagement span does not. This appears to explain why some outbound calls resolve a Called Service and others do not

In the examples above, the HTTP client span contains supportability.external.dt.entity.service, while the CONNECT TradeManagement span does not. This appears to explain why some outbound calls resolve a Called Service and others do not.

Looking at the traces, it seems that OneAgent writes supportability.external.dt.entity.service onto certain client spans when it can resolve the target to a monitored service entity. For standard HTTP calls between monitored services, this resolution is typically available. In contrast, CONNECT spans represent connection establishment at a lower level and may occur before Dynatrace can associate the call with a service entity, which could explain why the attribute is absent and the Called service column remains empty.

This would also explain why the behavior is not always consistent even when the target appears to be the same from an application perspective.

Thanks,

Sujit