Is the possible to merge service created by OpenTelemetry and OneAgent as one?

[xuzhao2024]

Hi guys,

We have set up a OneAgent process running in a Kubernetes (K8S) pod, which automatically collects trace and log data for the containerized application.

At the same time, we have implemented OpenTelemetry within the application to send trace and log data to the Dynatrace SaaS endpoint.

As a result, two services have been created on the Dynatrace platform: one through custom service detection and the other through OpenTelemetry.

 

Would it make sense to combine these two services into one for easier monitoring and maintenance? If so, how should we approach this?

[Eric_Yu · Accepted solution]

Are you ingesting the traces through the API? In this case, since you already have the agent, you might want to try and send those trace through the OneAgent.

OpenTelemetry traces with OneAgent - Dynatrace Docs

This should merge any span you created into a single trace:

[xuzhao2024]

Hi, 

I tried to follow the steps as mentioned in this page https://docs.dynatrace.com/docs/shortlink/otel-getstarted-otlpexport#export-to-oneagent-traces-only. But it did not work on the K8S port. 

I got the open telemetry logs as below

2024-09-13T03:48:40.0018813Z:Activity started. Name = '{0}', Id = '{1}'.{PublishMessage}{00-0a5b9b74e081d09479bf7322d5b2d97e-ec47d561a2c29e90-01}
2024-09-13T03:48:40.1295165Z:Activity stopped. Name = '{0}', Id = '{1}'.{PublishMessage}{00-0a5b9b74e081d09479bf7322d5b2d97e-ec47d561a2c29e90-01}
2024-09-13T03:48:40.1314323Z:Activity started. Name = '{0}', Id = '{1}'.{PublishMessage}{00-706a3ce4b0c6c457196904a867a3e0cc-52fb616820110fba-01}
2024-09-13T03:48:40.2209046Z:Activity started. Name = '{0}', Id = '{1}'.{ConsumeMessage}{00-0a5b9b74e081d09479bf7322d5b2d97e-593d0d535d9d97d4-01}
2024-09-13T03:48:40.2738128Z:Activity stopped. Name = '{0}', Id = '{1}'.{PublishMessage}{00-706a3ce4b0c6c457196904a867a3e0cc-52fb616820110fba-01}
2024-09-13T03:48:40.3615783Z:Activity stopped. Name = '{0}', Id = '{1}'.{ConsumeMessage}{00-0a5b9b74e081d09479bf7322d5b2d97e-593d0d535d9d97d4-01}
2024-09-13T03:48:40.4938106Z:Activity started. Name = '{0}', Id = '{1}'.{ConsumeMessage}{00-706a3ce4b0c6c457196904a867a3e0cc-d4f8600f3b097a2a-01}
2024-09-13T03:48:40.7048522Z:Activity stopped. Name = '{0}', Id = '{1}'.{ConsumeMessage}{00-706a3ce4b0c6c457196904a867a3e0cc-d4f8600f3b097a2a-01}
2024-09-13T03:48:42.2621686Z:Exporter failed send data to collector to {0} endpoint. Data will not be sent. Exception: {1}{http://localhost:14499/otlp/v1/traces}{System.Net.Http.HttpRequestException: Connection refused (localhost:14499)
 ---> System.Net.Sockets.SocketException (111): Connection refused
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|285_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(QueueItem queueItem)
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellation(CancellationToken cancellationToken)
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.HttpMessageHandlerStage.Send(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpMessageHandlerStage.Send(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.SocketsHttpHandler.Send(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.Send(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
   at OpenTelemetry.Exporter.OpenTelemetryProtocol.Implementation.ExportClient.BaseOtlpHttpExportClient`1.SendExportRequest(TRequest request, DateTime deadlineUtc, CancellationToken cancellationToken)}

It looks like the default 14499 port for one agent to capture trace logs from OPTL is not listened on the localhost.  Any idea about this part? Is that anything I need to do to enable it on the k8s pod?
 

[Julius_Loman]

@xuzhao2024 in Kubernetes, the EEC controller runs on the K8s node, but it's not accessible from pods. You need to send OTLP data to ActiveGate or Cluster. See https://docs.dynatrace.com/docs/shortlink/api-otlp-traces 
Depending on your DynaKube configuration in your k8s cluster, you might be able to send it through the ActiveGate running in kubernes - if it's deployed in your setup, and you have dataingest enabled. By default the embedded activegate in k8s is reachable under https://<dynakubename>.dynatrace:443 , but by default it has the self-signed certificate, so you have to configure it properly either on the ActiveGate (in DynaKube, see this and this) or on the exporter.

Anyway - your issue is from double instrumentation and I believe you will end up with two services anyway. Services of different types can't be merged. You need to decide if you want to do it using Dynatrace instrumentation (custom services in your case) or OpenTelemetry.

[Peter_Youssef]

Hi @xuzhao2024 

• If the services are of the main type: create a service detection rule.
• In previous Dynatrace versions, there was a service merging rules, currently not available.
• If these services aren't of the same type: Can't be merged.
• Depending on the actual deployment of K8s, you should send the Opentelemetry data to the same end point where pods and AG are connected.
• You can apply a simple change in the Opentelemetry configuration, make sure it's sending the data to either cluster or ActiveGates.

You can unify the communication configurations between K8s and Opentelemetry to ensure the ingested data is correlated without more configurations.

Regards,