How to troubleshoot missing Topic metrics from Apache Kafka extension

[Lefteris]

• Summary
• Problem
• Troubleshooting steps
• JMX MBeans
• JConsole
• Resolution
• What's next

Summary

The Apache Kafka extension promises deep and wide monitoring of all Kafka components. And yet, even if the configuration is in line with documentation, metrics might still be missing. How can we troubleshoot that?

Problem

Often, topics set up to be monitored with the Apache Kafka extension show no or limited metrics, even if they appear as detected entities.

The answer as to why this happens almost certainly lies outside the context of the extension itself. 

Troubleshooting steps

There’s a simple, general truth that guides the function of all Dynatrace extensions: the extension will only show what it can find. When it comes to Apache Kafka, the source of the metric data points is JMX MBeans.

JMX MBeans

In short, JMX (Java Management Extensions) is a set of built-in management tools enabled out-of-the-box for JVMs. MBeans are Java objects that are designed for the JMX specification and are used to expose attributes and operations of various types of resources.

Topic Metrics

Topic metrics are gathered from three distinct JMX domains: kafka.server, kafka.consumer, and kafka.producer. That’s because, unlike Kafka Brokers, Producers, and Consumers, Topics are not individual processes with their own specific metrics. They are logical channels that receive messages from Producers, work with Brokers to manage storage, and expose those messages for reading to Consumers.

So in order to be complete, Topic monitoring requires data from all other Apache Kafka entities:

• Broker metrics (kafka.server) provide information about data storage, partition count, and message throughput.
• Producer metrics (kafka.producer) offer data on message production rates, request latency, and error rates.
• Consumer metrics (kafka.consumer) include data on message consumption rates, lag, and processing errors.

MBeans specific to Kafka Consumers and Producers are not present on any processes that are part of the Kafka Broker instances. They are only present in the processes running the consumer and producer clients. It follows that, if you don't see kafka.consumer metrics, it's likely because the consumer processes themselves are not running in your deployment or are not configured to expose JMX metrics.

 Features to enable and configure

The steps below apply to Consumer metrics, but they can be applied equally to Producer metrics.

• No running client means no MBeans. So, first, make sure that you have running Kafka Consumer applications.
• Once the above is taken care of, the Consumer JVMs must be configured to expose JMX metrics. You can do that with environment variables or with Java system properties. Going by the former option, you need to use the following:
  • JMX_PORT: The port that JMX listens on for incoming connections.
  • JMX_HOSTNAME: The IP that JMX clients will use to connect to the JMX server to access MBeans.
  • KAFKA_JMX_OPTS: JMX options. Use this variable to override the default JMX options, such as whether authentication is enabled. The default options are set as follows:

    -Djava.rmi.server.hostname=127.0.0.1
    -Dcom.sun.management.jmxremote=true
    -Dcom.sun.management.jmxremote.authenticate=false
    -Dcom.sun.management.jmxremote.ssl=false

    Here's what each of them refers to:

  • java.rmi.server.hostname: IP address that the RMI server uses to bind MBeans. The default is localhost. This doesn’t have to be the same as JMX_HOSTNAME. For example, in a Docker environment, you might set java.rmi.server.hostname to the container's internal IP address and JMX_HOSTNAME to the host machine's IP address to ensure proper connectivity from outside the container.

  • jmxremote=true: If that’s set to false, no remote client can connect to the JMX server.

  • jmxremote.authenticate=false: Indicates that authentication is off by default.

  • jmxremote.ssl=false: Indicates that TLS/SSL is off by default.

    So, for example, to start Kafka and specify a JMX port, you can use the following command:

    JMX_PORT=2020 ./bin/kafka-server-start.sh ./etc/kafka/kraft/server.properties

 

JConsole

Now that the above are enabled, you should be able to see consumer MBeans in JConsole, which comes with Java out-of-the-box. To quote the official docs:

The JConsole graphical user interface is a monitoring tool that complies to the Java Management Extensions (JMX) specification. JConsole uses the extensive instrumentation of the Java Virtual Machine (Java VM) to provide information about the performance and resource consumption of applications running on the Java platform.

You can start it by simply typing the jconsole command. When the “New Connection” window comes up, if Kafka is running locally, you will see it as a selectable Local Process. If, instead, you run Kafka remotely, you need to select Remote Process, then enter the hostname and port you specified in your JMX configuration. Once that’s done, you can then Connect.

Screenshot taken from Confluent docs.

If you’ve set authentication to true, you will have to provide a valid username and password. Otherwise, you may be prompted to make an insecure connection.

Resolution

Now that the connection is established, you should be able to select the MBeans tab and expand the folders to see the JMX events and attributes for those events.

Screenshot taken from Confluent docs.

Notice that you should be looking for MBeans queried with:
kafka.consumer:type=consumer-metrics

kafka.consumer:type=consumer-fetch-manager-metrics

kafka.producer:type=producer-topic-metrics

Also, keep in mind that those metrics have client-id and topic dimensions – you might want to look at the values in those too.

If you do not see an MBean that you expected for a feature or component, you should then make sure that the feature or component that generates it is successfully enabled. Also, that you are viewing the correct process.

What's next

If you're still stuck, your next step should be to open a support ticket. Make sure you include:

• Links and screenshots that show which entities are missing which metrics.
• Full description of the outcome from the above steps. Make sure the exposed mbeans are visible. You can also use a cli tool to dump them (e.g. jmxterm) and share them in the ticket as a file.