OpenTelemetry - NodeJs - Custom Service

[Bert_VanderHeyd]

We have a nodejs application, the notification service, which is typically called at night.
In fact a web-call "/v1/notifications/sync" is the initiator but it starts a separate thread which sends out notifications. Acts like a kind of batch process and could potentially take a long time.

We have implemented OTEL to  have more visibility of what's really going on. This works, but since it starts in a OneAgent captured trace everything is linked to that single trace.

Since this process can take a long time I fear that we will hit the purepath/trace limitations on production. So, we actually need the OTEL spans as separate service with separate entries.

How could this be achieved?

- Using custom service detection does not work on nodejs. Only Kafka messaging is allowed.

[dannemca]

What if you try to instrument your OTEL using the "without oneagent" pointing to your tenant endpoint?

[Julius_Loman]

Exactly as @dannemca mentions. Just adding another exporter (export that using OTLP to Dynatrace trace ingest API) should do the trick I think. 

[Ana_Kuzmenchuk]

Hey @Bert_VanderHeyd, did Danne's comment help you? Hope it did! 

[Bert_VanderHeyd]

It's not that easy as it looks because we require an extra component (OTLP collector). And the fact that we have a OneAgent on it should be a plus, not a minus.

[Julius_Loman]

@Bert_VanderHeyd you don't need OTLP collector, you can pass directly to the local ingest endpoint on the OneAgent - http://localhost:14499/otlp/v1/traces (just not sure if it already accepts gzip compression).

[alexander_dt]

Hi Bert,

Four questions 🙂

1) For starters I'd like to clarify this sentence

---

Since this process can take a long time I fear that we will hit the purepath/trace limitations on production

Which limitations are you concerned by? If this is something on the backend side, it may not matter all that much how you ingest it (via OneAgent, directly, via exporter), as that would be a backend limitation.

Otherwise, if it is a OneAgent limit, skipping automatic ingestion might just fix that.

2) How do you instrument your application? Is this automatic instrumention or did you do it manually?

3) I presume you currently let OneAgent import the traces automatically, right?

4) Last question, and just to clarify, you essentially want to split the automatic OneAgent spans and traces from your own ones, did I understand this correctly?

[Bert_VanderHeyd]

1) the goal is indeed to get a good service flow of the entire process. Currently it all lands in one single trace which has limitations in terms of time and nodes. If this limitations were not there, we would still have one gigantic trace. It's the automatic instrumentation that currently starts the trace since it's a web call. I already tried to exclude it but then nothing comes in, not even the OTEL spans. It looks like they only enrich existing traces but don't start new ones.

2) Yes, automatic. Which we wanted to extend via OTEL code.

3) Yes, we don't use an exporter (yet) - Tip of Julius is however very usable.

4) See point 1. It's about creating a readable and workable service flow. I would like a trace for every record which is processed in that batch. 

[alexander_dt]

---

I would like a trace for every record which is processed in that batch

---

So the issue is not so much how to ingest the data, but rather to have separate traces instead of one global one, correct?

That should be possible via individual traces obtained from getTracer()

https://www.dynatrace.com/support/help/extend-dynatrace/opentelemetry/opentelemetry-traces/opentelem...