Custom events for alerting from OpenTelemetry

[sp]

Hello, 

I would like to know if there is a possibility to use from  OpenTelemetry traces>ConfigurationData.find>Timing section to create Custom events for alerting.  

[dannemca]

Hi, sp

Try to use the builtin:span_responsetime metric, and split the dimension by span.kind, filtering by the resulting span service from your ingestion. 

Check if this fit for your need.

[sp]

Hi,  in code for metric, you have spin.kind. And in the first screen health controller index. This means that you set span kind for health controller index? I cannot relate both screens together. However very insightful for me at first place. 

[dannemca]

HealthController#index is the trace name, indeed you will not be able to use it as a filter for the metric monitoring, but you can monitor the span service where this trace is associated with, and then, the monitoring will be done on all the traces.

[sp]

I wrote the following but got no response:

builtin:span_responsetime:filter(and(in("dt.entity.span:service",entitySelector("type(span:service),entityId(~"SERVICE-A0248DA9AF8D0000~")")))):splitBy("span.kind"):avg:auto:sort(value(avg,descending)):limit(100)

I assume from my span: kind is by me  "client". And Operation is "ConfigurationData.find", (where this appears by me as "HealthController#index" in your case) .

And by my app, only trace from OpenTelemetry is integrated and nothing more. If other steps should be done to get back entries from this query? 

Plus: pure "builtin:span_responsetime" does not work, as well, any idea, what could be the reason?

 

[dannemca]

How are you instrumenting the OpenTelemetry to get the ingested traces?

If you are using the OT instrumentation, you should see a service span type on your service list... but if you look closer, they are not services, but Custom devices.

This was added to the SaaS version 1.238, https://www.dynatrace.com/support/help/shortlink/release-notes-saas-sprint-238#services-for-opentele...

not sure about Managed (found nothing on release notes)

So, if you are seeing those span services on your env, I suggest to try the Explore Data on Build mode, instead code, to make things easy.

Or do this, to make even easier:

[sp]

https://opentelemetry.io/docs/instrumentation/go/getting-started/#trace-instrumentation

applied this approach here. Instrumenting trace. and under services don't have span service in UI..

[dannemca]

What is the version of your Dynatrace tenant? Is it SaaS or Managed?

[sp]

1.236.119.20220321-160129 and is managed 

[dannemca · Accepted solution]

The span services were introduced on version 1.238 (SaaS), not sure about Managed.

You can try to update the version to get it automatically or define your own manually.

https://www.dynatrace.com/support/help/shortlink/custom-topology#define-new-entity-type

Here are the rules for the span services that is created since 1.238 version: