OpenTelemetry Data is never sent

[paulao]

Hi,

Hopefully someone can help me out as I have tried everything and cannot get any span data into dynatrace at all.

I am trying to instrument a node js lambda function which calls RDS, I have tried both examples from the following code to no avail.

I am following the guide here: https://www.dynatrace.com/support/help/setup-and-configuration/setup-on-cloud-platforms/amazon-web-s...

Whatever I try it just does not work, there are no errors but there is nothing under distributed traces and I am unable to work out why.

Here is an extract from the lambda logs:

```
[Dynatrace] 2022-03-04 09:51:01.594 UTC [9-00000000] debug [LambdaSensor] Response attributes
[Dynatrace] 2022-03-04 09:51:01.604 UTC [9-00000000] debug [LambdaSensor] Span end
[Dynatrace] 2022-03-04 09:51:01.605 UTC [9-00000000] debug [DtSpanProcessor] forceFlush spanCnt: 1, exportingSpanCnt: 0, exportIterator:false, isExporting:false, forceFlushTimer:false
[Dynatrace] 2022-03-04 09:51:01.605 UTC [9-00000000] debug [DtSpanProcessor] _export spanCnt: 1, exportingSpanCnt: 0, exportIterator:false, isExporting:false, forceFlushTimer:false
[Dynatrace] 2022-03-04 09:51:01.610 UTC [9-00000000] debug [DtSpanProcessor] export spanCnt: 1, exportingSpanCnt: 1, exportIterator:false, isExporting:false, forceFlushTimer:false
[Dynatrace] 2022-03-04 09:51:01.612 UTC [9-00000000] info [Exporter] Resource elements: 12, size: 459
[Dynatrace] 2022-03-04 09:51:01.612 UTC [9-00000000] debug [Exporter] Resource values: telemetry.sdk.name=odin, telemetry.sdk.language=nodejs, telemetry.sdk.version=1.231.3.20211109-101449, dt.os.type=LINUX, process.runtime.name=NodeJS, process.runtime.version=v14.18.3, faas.name=nsp-38-smec-tpi-service, cloud.provider=aws, cloud.region=eu-west-1, faas.id=arnlambda:eu-west-1:745463184050:function:nsp-38-smec-tpi-service, cloud.account.id=745463184050, faas.max_memory=512
[Dynatrace] 2022-03-04 09:51:01.614 UTC [9-00000000] debug [Exporter] export: spanCnt: 1, size: 713, type: ForceFlush, retries: 0
[Dynatrace] 2022-03-04 09:51:01.625 UTC [9-00000000] debug [Exporter] request#socket: connecting: true
[Dynatrace] 2022-03-04 09:51:01.627 UTC [9-00000000] debug [Exporter] socket#connect
[Dynatrace] 2022-03-04 09:51:01.660 UTC [9-00000000] debug [Exporter] request#onResponse: statusCode: 202
[Dynatrace] 2022-03-04 09:51:01.664 UTC [9-00000000] debug [DtSpanProcessor] exportDone type: ForceFlush, result: SUCCESS, spanCnt: 1, exportingSpanCnt: 1, exportIterator:false, isExporting:true, forceFlushTimer:false
```

[dannemca]

A tip for troubleshooting. Make sure to leave your MZ filter as all, since ingested traces are not linked to any MZ, yet.

[uros_djukic1]

Hi @paulao , 
You are trying to get appropriate way to ingest traces , however did you create a wrapper function with the same signature as the original Lambda function ? For instance, call AWSLambdaWrapper.Trace() API and pass TracerProvider, original Lambda function and its inputs as parameters.

What is the version of Node.JS ? 

Thanks

[paulao]

Hi do you have an example? The documentation does not suggest that creating a wrapper is required?

I have not done this by the way. I am using node 14.

[uros_djukic1]

here please find this doc : OpenTelemetry interoperability on AWS Lambda | Dynatrace Docs, we find hyperlink , and now we have to select "opentelemetry AWS lambda for Node JS" :
open-telemetry/opentelemetry-js-contrib: OpenTelemetry instrumentation for JavaScript modules (githu...
Can you confirm me exact approach what you have adopted ? 

[paulao]

I have tried both examples in the documentation, as that is the same page I linked in my original post but neither seems to create a wrapper function, neither example works at the moment and I get no trace data in dynatrace.

I have also tried doing this in a trimmed down function and interestingly if I remove the lambda layer (dynatrace layer) then it looks like the instrumentation works, as in it captures the stored procedure call etc, but then there is nowhere to export the data to.

[uros_djukic1]

Ok just to clarify, what kind of scenario are you ? 

case 1 ) picking up Opentelemetry spans auto via Oneagent ? 
case 2) where Oneagent isn't avaiblable ? (I guess this one , right? )

For case 2 : Send OpenTelemetry trace data to Dynatrace | Dynatrace Docs

In other words, are you trying to instrument AWS LAMbda functions running on NodeJS which use the "environment variables" setting method to instrument?
maybe if you try to pass via ApiGateway , there will be some proxy limitations.

Did you take a look into metrics and let me know if you get this one : dsfm:server.spans.persisted ?

[paulao]

Hi again,

My preference would be to use the layer and environment variables and have the auto instrumentation work but this doesnt and I cannot work out why this is. So, one agent is available and I followed the docs but this doesnt seem to work.

I do not have any metrics for "dsfm:server.spans.persisted".

The requests are going through API gateway but I am trying to trigger them using the lambda console with a mock event as well and do not see any trace data for these.

[gurugogi]

Hi All,

I also have similar issue, I used environment variables option, we have firewall allowed, function is running every 5mins, still we can see it in DT services section.
In logs I can see "Not authorized to send spans", dont see any other issues like timeouts.

please help me to identify the issue.

Its nodeJS 16.x (v16.20.2 runtime), we dont have Env AG.

AWS_LAMBDA_EXEC_WRAPPER = /opt/dynatrace
DT_TENANT = ABCABC-ABCABC
DT_CLUSTER_ID = 123456
DT_CONNECTION_BASE_URL = https://xyz.com
DT_CONNECTION_AUTH_TOKEN = dt0a01.4ba67xyxyxy.f32160xyxyxy
DT_OPEN_TELEMETRY_ENABLE_INTEGRATION = true
DT_LOGGING_DESTINATION: stdout
DT_LOGGING_NODEJS_FLAGS: Exporter=true,LambdaSensor=true
DT_LOG_COLLECTION_LOG_LEVEL = debug 

Thanks