Trace error "Error getting JMX attribute 'records-lag-avg'" from Apache Kafka Hub Extension

[erh_inetum]

Hi,

We added to our environment Apache Kafka extension from Hub. The version configured is 1.1.2.

We configured two endpoints each pointing to a different machine. The configurations are the same.

The extension is capturing information sucessfully.

But the following trace has started to appear in one of the machines in spike of the configuration is the same for the two machines:

2024-07-15T00:00:24,232 WARN  [oneagentperiodiceventsmanaged] [org.apache.kafka.common.metrics.JmxReporter] Error getting JMX attribute 'records-lag-avg'
javax.management.AttributeNotFoundException: Could not find attribute records-lag-avg
        at org.apache.kafka.common.metrics.JmxReporter$KafkaMbean.getAttribute(JmxReporter.java:255) ~[kafka-clients-2.8.1.jar:?]
        at org.apache.kafka.common.metrics.JmxReporter$KafkaMbean.getAttributes(JmxReporter.java:263) [kafka-clients-2.8.1.jar:?]
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttributes(DefaultMBeanServerInterceptor.java:701) [?:?]
        at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttributes(JmxMBeanServer.java:705) [?:?]
        at com.dynatrace.agent.measures.jmx.MBeanServerProxy$MBeanGetAttributesExecutor.execute(MBeanServerProxy.java:182) [?:1.287.149.20240416-100607]
        at com.dynatrace.agent.measures.jmx.MBeanServerProxy$MBeanExecutionStrategy.execute(MBeanServerProxy.java:232) [?:1.287.149.20240416-100607]
        at com.dynatrace.agent.measures.jmx.MBeanServerProxy.executeMBeanMethod(MBeanServerProxy.java:92) [?:1.287.149.20240416-100607]
        at com.dynatrace.agent.measures.jmx.MBeanServerProxy.getAttributeList(MBeanServerProxy.java:73) [?:1.287.149.20240416-100607]
        at com.dynatrace.agent.measures.jmx.MBeanReference.getAttributes(MBeanReference.java:58) [?:1.287.149.20240416-100607]
        at com.dynatrace.agent.measures.jmx.MBeanAttributeReader.read(MBeanAttributeReader.java:95) [?:1.287.149.20240416-100607]
        at com.dynatrace.agent.measures.jmx.MBeanSubscriptionManager.performMeasurements(MBeanSubscriptionManager.java:222) [?:1.287.149.20240416-100607]
        at com.dynatrace.agent.measures.jmx.MBeanTracker.performMeasurements(MBeanTracker.java:79) [?:1.287.149.20240416-100607]
        at com.dynatrace.agent.measures.AbstractJmxMetricsProvider.capture(AbstractJmxMetricsProvider.java:82) [?:1.287.149.20240416-100607]
        at com.dynatrace.agent.measures.JmxMetricsProviderV2.getMintLines(JmxMetricsProviderV2.java:103) [?:1.287.149.20240416-100607]
        at com.dynatrace.agent.measures.MetricsManaged.getMintLines(MetricsManaged.java:161) [?:1.287.149.20240416-100607]

Does anyone know why it's started to appear and how to fix it?

Thanks in advance.

Regards,

Elena.

[TomásSeroteRoos]

From the error it looks like your MBean is missing the records-lag-avg attribute, and looking at the yaml for that extension, that particular attribute seems to be collected from this JMX query:

kafka.consumer:type=consumer-fetch-manager-metrics,client-id=*,partition=*,topic=*

I would go through the MBeans of the host where you are getting the error, look for something that matches that query and check whether or not the attribute is there.

[alain_vonallmen]

we see something similar and we opened a ticket with dynatrace for that. right now we do not know where this is coming from exactly. this is what we see:

2024-08-09 09:28:29.877  WARN 7 --- [iceventsmanaged] o.a.k.c.m.JmxReporter  : Error getting JMX attribute 'records-lag-avg' javax.management.AttributeNotFoundException: Could not find attribute records-lag-avg

• Besides the "records-lag-avg", we also have warnings for "records-lag" and records-lag-max
• we have several kafka consumers with this warnings.
• the consumers are all configured correctly exposing the beans via jmx
• the warnings are not continuous, they can disappear for a week, then appear again for a week under production load.
• we use dynatrace managed with kafka extension on openshift operator

[BrendanSchembri]

We are having the exact same issue - any chance we get a solution from Dynatrace please? @TomásSeroteRoos 

[BrendanSchembri · Accepted solution]

Update @erh_inetum @alain_vonallmen ,

I've managed to find a way to disable these 🙂

1. Go to Dynatrace
2. Dynatrace Hub
3. Apache Kafka Extension
4. Mentoring Configuration and Select Edit:
   
5. From Monitoring Source, click Next Step.
6. From JMX, click Next Step.
7. On the Finalize Configuration Screen, Search for the Metric in the 'Search Metrics' section and 'untick' it.

The above fixed it for us.

[erh_inetum]

Hi @BrendanSchembri ,

Thank you so much for the update.

I think this is not valid as final solution because disabling the consumer-metric family (because it's not possible to disable only specific metrics, you know) in which is avg-lag metric, we lose the whole consumers monitoring.

We 've also opened a support ticket. If we have any further information, I'll share it in this link.

Thanks again. Regards,
Elena.

[erh_inetum]

Hi everyone specially @BrendanSchembri , @alain_vonallmen  and @TomásSeroteRoos ,

Support have answered us.

They said that on the exposed bean Kafka-consumer there isn't an attribute called record-lag. So each time the extension query the JMX metrics its module can't find this attribute and the warning is reported in the log.
A solution to this will be having a custom extension that does the same but querying for this attribute, https://docs.dynatrace.com/docs/shortlink/extensions-jmx-customize#metrics

Hope it helps.

Regards,

Elena.

[benzhonghai008]

Hi, @erh_inetum 

  I run into the same problem now, so locky to get your message.
  Do you know how to use the custom extension?

[diosonne_jh]

Were you able to figure out how to develop/use a customized extension for this particular metric?

[harald_berger]

Please note that the record-lags attribute are not exposed every time in Kafka. See also known behavior / bug https://issues.apache.org/jira/browse/KAFKA-17652

[sivart_89]

I am seeing the same thing here, a lot of exceptions being logged for 3 attributes not being found, records-lag being one of them. It seems maybe this is something on the kafka side and nothing with the dynatrace agent. However, I do see metrics being collected for these attributes even through we are seeing the exceptions. I'm not real proficient with the inner working of java or kafka but is this normal based off how the agent works + the notes logged in the jira card you linked? I just find it a little strange we are seeing the data even with the exceptions, we are using extension version 2.0 for the kafka version, not 1.1.2 that was noted in the original post

https://issues.apache.org/jira/browse/KAFKA-17652