Is it possible to pull in String Value with JMX Extensions?

[sivart_89]

I've created a JMX extension for some Kafka metrics but now want to add the connector status metric. According to the confluent link below the connector status can be running, paused, or stopped. Per the JMX Dynatrace documentation, the attributes available in the list are only the numeric ones.

To see other attributes, for example, attributes of the String type, select Include non-numeric attributes. Note that selected attributes must be parsable to double, or boolean type. Otherwise, Dynatrace OneAgent will not collect any data from such a metric. I'm led to believe that the status metric cannot be captured and have seen the same when adding that metric to the JMX extension. Has anyone been able to pull in strings of text such as this status metric?

An alternative way we are thinking of is hitting an API to get the value and then ingesting that into Dynatrace as a metric. However, just want to be sure I'm not missing something with the JMX extension route.

https://docs.confluent.io/kafka-connectors/self-managed/monitoring.html#use-jmx-to-monitor-kconnect

Dynatrace JMX doc
https://www.dynatrace.com/support/help/extend-dynatrace/extend-metrics/ingestion-methods/jmx-extensi...

[Mike_L · Accepted solution]

The new JMX extensions in Extension Framework 2.0 will be able to handle this use case. It's currently wrapping up development and will be out in 2 or so months if all goes well. One of the first extensions we are bringing out together with the data source is actually an improved version of the Kafka extension. @victor_balbuena, is connector status one of the new metrics you added?

[victor_balbuena · Accepted solution]

The connector status is not one of the metrics defined to be captured by the first version of the Kafka extension in EF2.

[sivart_89]

Thank you @victor_balbuena  and @Mike_L. We hope the status is eventually incorporated into the kafka 2.0 extension but in the meantime I can utilize the new jmx framework once available.

[guruprasadv]

Do we know when this kafka connector status metrics will be available ?

[victor_balbuena]

The expected date for release of the new EF2 Kafka extension is on the 9th of May.

[sivart_89]

Thanks for the info. Will this be rolled into a cluster upgrade and be mentioned in release notes? Just wondering how we will be aware of this being available.

It's my understanding that the newer version of the kafka extension will not have the connector status we are looking for BUT we can still create a custom jmx extension to pull out the metric string whereas before EF2 you could not, is that accurate?

[Mike_L]

It is now available for download from the Dynatrace hub.

you can indeed also create your own JMX 2.0 extensions starting with 1.265.

[Stephen1_Lee]

Hi Mike_L

Could you link in the new documentation for JMX 2.0 for the Dynatrace hub?

[Mike_L]

I recommend that you grab one of the new extensions, such as ActiveMQ, which has it in the extension.yaml: https://www.dynatrace.com/hub/detail/activemq/?filter=all&type=extension#versions

[guruprasadv]

Hi,

We have updated to latest dynatrace version and still kafka connector status does not get populated in dynatrace dataexplorer.

[sivart_89]

To add onto @guruprasadv comment, we have updated the OneAgents to the required version for JMX 2.0 but I can't seem to get the string value to be pulled out for the JMX extension I have created. I looked at (with a dynatrace resource we have) at the ActiveMQ 2.0 extension linked above by @Mike_L and we did not see where that extension is pulling out a string value.

I'm hoping someone has a working example of pulling out a string value using 2.0 JMX. Specifically we are looking for the connector status located at the link below.

https://docs.confluent.io/platform/current/connect/monitoring.html#connector-metrics

[Mike_L]

Here is an example:

query: "org.apache.activemq:type=Broker"
queryFilters:
  - field: brokerName
    filter: var:broker_name-filter
  - field: destinationName
    filter: var:queue_name-filter
dimensions:
  - key: broker_name
    value: property:brokerName
  - key: queue_name
    value: property:destinationName
  - key: queue_type
    value: property:destinationType
  - key: product_name
    value: const:ActiveMQ
  - key: driver_version
    value: attribute:DriverVersion
  - key: foo_name
    value:
      attribute: Foo
      accessor: getName()
metrics:
  - key: com.dynatrace.extension.jmx.activemq.queue.const
    type: gauge
    value: const:1

[victor_balbuena]

As mentioned in my previous comment on this thread, the connector status is not one of the metrics captured by the default Kafka extension. You will have to create an extension of your own. When creating it, take into account that metrics in Dynatrace cannot have a value of a string, which means that I see two possible scenarios you might want to accomplish:

1. Ingest the value of the MBean as a dimension, while keeping the value of the metric constant as it is irrelevant. In this case, you can do something like:

- subgroup: kafka connector status
  query: kafka.connect:type=connector-metrics,connector=*
  dimensions:
    - key: connector
      value: property:connector
    - key: status
      value: attribute:status
  metrics: 
    - key: com.dynatrace.extension.kafka.status
      type: gauge
      value: const:1

Where the dimension is not a property of the MBean, but an attribute instead.

1. Transform the captured string into a number depending on its value. For this, you can just do booleans, i.e. check if the captured string is equal to a value or not. It could look something like this (in this case, paused would also report a 0):

- subgroup: kafka connector status
  query: kafka.connect:type=connector-metrics,connector=*
  dimensions:
    - key: connector
      value: property:connector
  metrics:
    - key: com.dynatrace.extension.kafka.status
      type: gauge
      value:
        attribute:status
        accessor: equals("running")

If you want to do something like running=2, paused=1 and stopped=0, that cannot be done at the moment. As mentioned aboved, it is restricted to a boolean so just 0 and 1.

---

I personally don't see anything else you could do with a string MBean, considering that Dynatrace does not support strings being ingested as the value of the metric.

PD: the dimension for the connector is not necessary, I just added it for more clarity on the example.

---

EDIT: This solution works only for immutable attributes as the value is not updated, just read once.

[sivart_89]

I went with option 1 here. Seems to be working so far. I've contacted our app side who work with Kafka and are looking for this metric, to confirm it is showing what they need.