Gravitee.io - Tracing support with OpenTelemetry

[Julius_Loman]

The Gravitee.io API gateway is not a directly supported technology by OneAgent. Without support, you are unable to see any processing within the API gateway, and traces stop at the API gateway.  Luckily, there is OpenTelemetry support for Gravitee 4.x helping to overcome this.

Steps to get this working for Gravitee 4.6 or higher:

Prerequisites

If you run OneAgent on the Gravitee Gateway host, either turn off deep monitoring for the Gateway process or turn off the following OneAgent features for the Gravitee Gateway process groups, otherwise, you will have duplicate spans and/or tracing is broken.

• Java Netty
• Java OpenTelemetry

Option 1 - send traces directly to Dynatrace (SaaS or Managed)

1. Create an Api Token in your Dynatrace environment with scope trace ingest scope.
2. Configure tracing in Gravitee based on https://documentation.gravitee.io/apim/analyze-and-monitor-apis/opentelemetry#enable-opentelemetry-f..., typically in gravitee.yml, where
   <DT_ENVIRONMENT> is URL for the Dynatrace environment like https://abc12345.live.dynatrace.com for SaaS or https://host/e/<environmentid for Managed

services:
  opentelemetry:
    enabled: true
    verbose: false    
    exporter:
      endpoint: <DT_ENVIRONMENT>/api/v2/otlp
      protocol: http/protobuf
      compression: gzip
      headers:
        - Authorization: api-token <API_TOKEN>​

Adjust other parameters, if necessary, such as SSL (typically needed in Managed environments only) or proxy.

Option 2 - Send traces using locally running fullstack OneAgent

If you want to send traces using locally running full-stack OneAgent, you can simplify the configuration a little bit, and spans are also included in the OneAgent full-stack license compared to Option 1.

1. Ensure EEC is enabled on the OneAgent hosts or globally in Dynatrace, see https://docs.dynatrace.com/docs/shortlink/extension-concepts 
2.  Configure Gravitee Gateway in gravitee.yml (or accordingly if other options are used). 

services:
  opentelemetry:
    enabled: true
    verbose: false    
    exporter:
      endpoint: http://localhost:14499/otlp
      protocol: http/protobuf
      compression: none

Option 3 - Send traces if Gravitee runs in Kubernetes using Dynatrace Operator Telemetry ingest

This requires Dynatrace Operator 1.6 or newer on the cluster running Gravitee gateway.

1. Enable and configure the Telemetry ingest with Dynatrace Operator by following  Telemetry ingest support.
2. Configure Gravitee Gateway in gravitee.yml (or accordingly if other options are used), adjust the hostname accordingly to your dynakube configuration:

services:
  opentelemetry:
    enabled: true
    verbose: false    
    exporter:
      endpoint: http://dynakube-telemetry-ingest.dynatrace.svc:4317
      protocol: grpc

Additional considerations

If you want a relation between the Unified service and process group instance, you need to add the resource attributes dt.entity.process_group_instance and dt.entity.host (added automatically if you send traces using locally installed OneAgent (option two). Although adding resource attributes can be accomplished by adding extraAttributes parameter in Gravitee, it does not work for keys with an underscore character, so an easier solution is to use OpenTelemetry Collector or Dynatrace collector to add the values. See an example below using the locally installed OneAgent as final ingest endpoint:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  resource:
    attributes:
      - key: dt.entity.host
        value: <HOST>
        action: upsert
      - key: dt.entity.process_group_instance
        value: <PROCESS_GROUP_INSTANCE>
        action: upsert

exporters:
  otlphttp:
    endpoint: http://localhost:14499/otlp

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [resource]
      exporters: [otlphttp]
    metrics:
      receivers: [otlp]
      processors: [resource]
      exporters: [otlphttp]
    logs:
      receivers: [otlp]
      processors: [resource]
      exporters: [otlphttp]

Steps to get this working for Gravitee 4.5 or lower:

• If your version is 4.3 or lower, please add opentelemetry-api-1.39.0.jar and opentelemetry-context-1.39.0.jar libraries to lib/ext/ directory. You can download them from maven (links above).
• If your version is 4.3 or lowser, please add gravitee-tracer-opentelemetry-1.0.1.zip (or newer) to plugins directory unless you are already using a distribution with the plugin.
• Create an Api Token in your Dynatrace environment with scope trace ingest scope.
• Deploy OpenTelemetry collector to the host where the Gravitee API gateway is running or where it fits your case. You can also reuse existing collectors. I recommend using Dynatrace distribution directly, see https://docs.dynatrace.com/docs/shortlink/otel-collector-deploy for deployment
• Configure your OpenTelemetry collector, sample config is below and it includes OpenTelemetry resource attributes to map the ingested traces to existing entities in Dynatrace. In the same change:
  <HOST> for your HOST entity ID  where your API gateway is running,
  <PROCESS_GROUP_INSTANCE> for your process group instance of the API gateway process on the host
  <ENVIRONMENT> for your Dynatrace environment URL e.g. https://abc12345.live.dynatrace.com
  <APITOKEN> for your API token created in the previous step

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  resource:
    attributes:
      - key: dt.entity.host
        value: <HOST>
        action: upsert
      - key: dt.entity.process_group_instance
        value: <PROCESS_GROUP_INSTANCE>
        action: upsert

exporters:
  otlphttp:
    endpoint: <ENVIRONMENT>/api/v2/otlp
    headers:
      Authorization: "Api-Token <APITOKEN>"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [resource]
      exporters: [otlphttp]
    metrics:
      receivers: [otlp]
      processors: [resource]
      exporters: [otlphttp]
    logs:
      receivers: [otlp]
      processors: [resource]
      exporters: [otlphttp]
​

• Start or restart the OpenTelemetry collector.

• Configure Gravitee API gateway in gravitee.yml to include OpenTelemetry tracer. In case the OpenTelemetry collector is not running on the same host, change the URL accordingly.

  tracing:                 
    enabled: true 
    type: otel                                                   
    otel:                                                                         
      url: grpc://localhost:4317​

• Restart the API gateway process, you should see the enablement of the otel tracing in the API gateway logs:

13:58:09.943 [graviteeio-node] [] INFO  i.g.node.tracing.TracingService - Tracing support is enabled with tracer: name[otel]​

• Traces now should be ingested and a new Unified service should be created on the API Gateway process:
  
• You should now see full traces including the API Gateway processing.  In the image below - the red part is from OneAgent, the blue part is from the OpenTelemetry tracing plugin, and the green part is from another service monitored by OneAgent.
  

Just a few notes:

• OneAgent deployment is not required but I recommend doing it for licensing and better request monitoring
• Although the OpenTelemetry plugin for Gravitee includes OTLP http output, it supports only the JSON payload which is on the other side not supported by Dynatrace. Dynatrace supports protobuf only for OTLP. That's why OpenTelemetry collector is required.

[agonzalez]

Hello Julius, thanks for this tutorial! Especially for the note on OTLP HTTP output. The OneAgent is deployed on the gateway. We thought we would be able to capture the OpenTelemetry traces directly, but it was not possible without the OTel Collector.

[Julius_Loman]

If you have OneAgent on the gravitee host, normally OneAgent will collect the OTEL traces and the collector is not required. Unfortunately, Gravitee configuration for OpenTelemetry plugin requires a valid configuration and output, otherwise, it will produce a lot of errors about the plugin being unable to export traces.

So a valid destination in Gravitee is needed. It's not possible to include and enable the plugin without it.

[AntonioSousa]

@Julius_Loman,

Thanks for the detailed configuration. While not a Gravitee.io user, it's great to know how certain services are implementing OpenTelemetry and how they integrate with Dynatrace.

[agonzalez]

It seems that my traces aren't complete, I think Gravitee remove headers, did it happened to you ?

Can you confirm the header that I need to be forwarded by Gravitee, I supposed the following:

- x-dynatrace

- traceparent

- tracestate

Is there anything else I should be carefull about ?

[Julius_Loman]

@agonzalez it did not happen in my case. Maybe this is specific to your gravitee.io setup (a policy removing specific headers?) or something else is removing headers. I recommend setting up request attributes on the services to capture the values. You need just w3c headers (traceparent, tracestate) for this to work.

Basically, the tricky part was just the OTLP HTTP setup which produces JSON only and this is not mentioned in the docs.

One important caveat is the enrichment of traces with Dynatrace resource attributes, so your trace ingest is covered by a full-stack oneagent license - but that's already in my setup above.