OpenTelemetry and new metric dimension

[mkaszub]

Hi,

After some time, I decided to add a new dimension (tag) to my metric. However, it appears that Dynatrace is not recognizing it. I can't see the new dimension when attempting to split the metric or in the metric details where all dimensions are listed. For testing purposes, I created an identical metric with a different name, and the additional dimension is visible there. This suggests there might be a caching issue or that the metric definition hasn't been updated.

I haven't found a way to remove the OpenTelemetry metric. What are my options for adding the new dimension to the metric without changing its name?

Regards,
Marcin

[Mizső]

Hi @mkaszub 

I am not an OTEL expert but, have you checked these domcumentaion topics:

Configure OpenTelemetry metrics ingest — Dynatrace Docs

OpenTelemetry metrics limitations — Dynatrace Docs

I hope it helps.

Best regards,

János

[RohitBisht · Accepted solution]

Hi @mkaszub ,
Here are few things you can check.
1. Protocol and syntax for dimension: Check if it is following this: https://docs.dynatrace.com/docs/shortlink/metric-ingestion-protocol 
2. Test using API and send dummy values onto the metric using that dimension. https://docs.dynatrace.com/docs/shortlink/api-metrics-v2 

If point 2 works, maybe there is something wrong with the otel code sending data.

[gopher]

Hi mkaszub,

I haven't had any issues adding / removing - example of our processors below. 
On changing them (adding too or deleting), The only way I could replicate what you are seeing is by looking within the same time window.   after the time window has passed for the last metric captured it won't be shown. 

  processors:
    attributes:
      actions:
      - key: k8s.cluster.name
        value: 'my cluster name'
        action: insert          
    cumulativetodelta: {}      
    filter:
      metrics:
        exclude:
          match_type: expr
          expressions:
          - MetricType == "Summary"
    memory_limiter:
      check_interval: 5s
      limit_percentage: 80
      spike_limit_percentage: 25
    batch/traces:
      send_batch_size: 5000
      send_batch_max_size: 5000
      timeout: 60s
    batch/metrics:
      send_batch_size: 3000
      send_batch_max_size: 3000
      timeout: 60s
    batch/logs:
      send_batch_size: 1800
      send_batch_max_size: 2000
      timeout: 60s      
    k8sattributes:
      auth_type: serviceAccount
      passthrough: false      
      extract:
        metadata:
        - k8s.pod.name
        - k8s.pod.uid
        - k8s.deployment.name
        - k8s.statefulset.name
        - k8s.daemonset.name
        - k8s.cronjob.name
        - k8s.namespace.name
        - k8s.node.name
        - k8s.cluster.uid
      pod_association:
      - sources:
        - from: resource_attribute
          name: k8s.pod.name
        - from: resource_attribute
          name: k8s.namespace.name
      - sources:
        - from: resource_attribute
          name: k8s.pod.ip
      - sources:
        - from: resource_attribute
          name: k8s.pod.uid
      - sources:
        - from: connection
    transform:
      error_mode: ignore
      trace_statements:
        - context: resource
          statements:
              - set(attributes["dt.kubernetes.workload.kind"], "statefulset") where IsString(attributes["k8s.statefulset.name"])
              - set(attributes["dt.kubernetes.workload.name"], attributes["k8s.statefulset.name"]) where IsString(attributes["k8s.statefulset.name"])
              - set(attributes["dt.kubernetes.workload.kind"], "deployment") where IsString(attributes["k8s.deployment.name"])
              - set(attributes["dt.kubernetes.workload.name"], attributes["k8s.deployment.name"]) where IsString(attributes["k8s.deployment.name"])
              - set(attributes["dt.kubernetes.workload.kind"], "daemonset") where IsString(attributes["k8s.daemonset.name"])
              - set(attributes["dt.kubernetes.workload.name"], attributes["k8s.daemonset.name"]) where IsString(attributes["k8s.daemonset.name"])
              - set(attributes["dt.kubernetes.cluster.id"], attributes["k8s.cluster.uid"]) where IsString(attributes["k8s.cluster.uid"])
      log_statements:
        - context: resource
          statements:
              - set(attributes["dt.kubernetes.workload.kind"], "statefulset") where IsString(attributes["k8s.statefulset.name"])
              - set(attributes["dt.kubernetes.workload.name"], attributes["k8s.statefulset.name"]) where IsString(attributes["k8s.statefulset.name"])
              - set(attributes["dt.kubernetes.workload.kind"], "deployment") where IsString(attributes["k8s.deployment.name"])
              - set(attributes["dt.kubernetes.workload.name"], attributes["k8s.deployment.name"]) where IsString(attributes["k8s.deployment.name"])
              - set(attributes["dt.kubernetes.workload.kind"], "daemonset") where IsString(attributes["k8s.daemonset.name"])
              - set(attributes["dt.kubernetes.workload.name"], attributes["k8s.daemonset.name"]) where IsString(attributes["k8s.daemonset.name"])
              - set(attributes["dt.kubernetes.cluster.id"], attributes["k8s.cluster.uid"]) where IsString(attributes["k8s.cluster.uid"])