Gravitee.io - Tracing support with OpenTelemetry

[Julius_Loman]

The Gravitee.io API gateway is not a directly supported technology by OneAgent. Without support you are unable to see any processing within the API gateway and traces stop at the API gateway. Luckily, there is a trace plugin for  OpenTelemetry at https://www.gravitee.io/plugins/gravitee-tracer-opentelemetry .

Steps to get this working:

• If your version is 4.3 or lower, please add opentelemetry-api-1.39.0.jar and opentelemetry-context-1.39.0.jar libraries to lib/ext/ directory. You can download them from maven (links above).
• If your version is 4.3 or lower, please add gravitee-tracer-opentelemetry-1.0.1.zip (or newer) to plugins directory unless you are already using a distribution with the plugin.
• Create Api Token in your Dynatrace environment with scope trace ingest scope.
• Deploy OpenTelemetry collector to the host where the Gravitee API gateway is running or where it fits your case. You can also reuse existing collectors. I recommend using Dynatrace distribution directly, see https://docs.dynatrace.com/docs/shortlink/otel-collector-deploy for deployment
• Configure your OpenTelemetry collector, sample config is below and it includes OpenTelemetry resource attributes to map the ingested traces to existing entities in Dynatrace. In the same change:
  <HOST> for your HOST entity ID  where your API gateway is running,
  <PROCESS_GROUP_INSTANCE> for your process group instance of the API gateway process on the host
  <ENVIRONMENT> for your Dynatrace environment URL e.g. https://abc12345.live.dynatrace.com
  <APITOKEN> for your API token created in the previous step

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  resource:
    attributes:
      - key: dt.entity.host
        value: <HOST>
        action: upsert
      - key: dt.entity.process_group_instance
        value: <PROCESS_GROUP_INSTANCE>
        action: upsert

exporters:
  otlphttp:
    endpoint: <ENVIRONMENT>/api/v2/otlp
    headers:
      Authorization: "Api-Token <APITOKEN>"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [resource]
      exporters: [otlphttp]
    metrics:
      receivers: [otlp]
      processors: [resource]
      exporters: [otlphttp]
    logs:
      receivers: [otlp]
      processors: [resource]
      exporters: [otlphttp]
​

• Start or restart the OpenTelemetry collector.

• Configure Gravitee API gateway in gravitee.yml to include OpenTelemetry tracer. In case the OpenTelemetry collector is not running on the same host, change the URL accordingly.

  tracing:                 
    enabled: true 
    type: otel                                                   
    otel:                                                                         
      url: grpc://localhost:4317​

• Restart the API gateway process, you should see the enablement of the otel tracing in the API gateway logs:

13:58:09.943 [graviteeio-node] [] INFO  i.g.node.tracing.TracingService - Tracing support is enabled with tracer: name[otel]​

• Traces now should be ingested and a new Unified service should be created on the API Gateway process:
  
• You should now see full traces including the API Gateway processing.  In the image below - the red part is from OneAgent, the blue part is from the OpenTelemetry tracing plugin, and the green part is from another service monitored by OneAgent.
  

Just a few notes:

• OneAgent deployment is not required but I recommend doing it for licensing and better request monitoring
• Although the OpenTelemetry plugin for Gravitee includes OTLP http output, it supports only the JSON payload which is on the other side not supported by Dynatrace. Dynatrace supports protobuf only for OTLP. That's why OpenTelemetry collector is required.

[agonzalez]

Hello Julius, thanks for this tutorial! Especially for the note on OTLP HTTP output. The OneAgent is deployed on the gateway. We thought we would be able to capture the OpenTelemetry traces directly, but it was not possible without the OTel Collector.

[Julius_Loman]

If you have OneAgent on the gravitee host, normally OneAgent will collect the OTEL traces and the collector is not required. Unfortunately, Gravitee configuration for OpenTelemetry plugin requires a valid configuration and output, otherwise, it will produce a lot of errors about the plugin being unable to export traces.

So a valid destination in Gravitee is needed. It's not possible to include and enable the plugin without it.

[AntonioSousa]

@Julius_Loman,

Thanks for the detailed configuration. While not a Gravitee.io user, it's great to know how certain services are implementing OpenTelemetry and how they integrate with Dynatrace.

[agonzalez]

It seems that my traces aren't complete, I think Gravitee remove headers, did it happened to you ?

Can you confirm the header that I need to be forwarded by Gravitee, I supposed the following:

- x-dynatrace

- traceparent

- tracestate

Is there anything else I should be carefull about ?

[Julius_Loman]

@agonzalez it did not happen in my case. Maybe this is specific to your gravitee.io setup (a policy removing specific headers?) or something else is removing headers. I recommend setting up request attributes on the services to capture the values. You need just w3c headers (traceparent, tracestate) for this to work.

Basically, the tricky part was just the OTLP HTTP setup which produces JSON only and this is not mentioned in the docs.

One important caveat is the enrichment of traces with Dynatrace resource attributes, so your trace ingest is covered by a full-stack oneagent license - but that's already in my setup above.