Solved: Trace error "Error getting JMX attribute 'records-lag-avg'" from Apache Kafka Hub Extension

erh_inetum · ‎17 Jul 2024

Hi,

We added to our environment Apache Kafka extension from Hub. The version configured is 1.1.2.

We configured two endpoints each pointing to a different machine. The configurations are the same.

The extension is capturing information sucessfully.

But the following trace has started to appear in one of the machines in spike of the configuration is the same for the two machines:

2024-07-15T00:00:24,232 WARN [oneagentperiodiceventsmanaged] [org.apache.kafka.common.metrics.JmxReporter] Error getting JMX attribute 'records-lag-avg'
javax.management.AttributeNotFoundException: Could not find attribute records-lag-avg
        at org.apache.kafka.common.metrics.JmxReporter$KafkaMbean.getAttribute(JmxReporter.java:255) ~[kafka-clients-2.8.1.jar:?]
        at org.apache.kafka.common.metrics.JmxReporter$KafkaMbean.getAttributes(JmxReporter.java:263) [kafka-clients-2.8.1.jar:?]
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttributes(DefaultMBeanServerInterceptor.java:701) [?:?]
        at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttributes(JmxMBeanServer.java:705) [?:?]
        at com.dynatrace.agent.measures.jmx.MBeanServerProxy$MBeanGetAttributesExecutor.execute(MBeanServerProxy.java:182) [?:1.287.149.20240416-100607]
        at com.dynatrace.agent.measures.jmx.MBeanServerProxy$MBeanExecutionStrategy.execute(MBeanServerProxy.java:232) [?:1.287.149.20240416-100607]
        at com.dynatrace.agent.measures.jmx.MBeanServerProxy.executeMBeanMethod(MBeanServerProxy.java:92) [?:1.287.149.20240416-100607]
        at com.dynatrace.agent.measures.jmx.MBeanServerProxy.getAttributeList(MBeanServerProxy.java:73) [?:1.287.149.20240416-100607]
        at com.dynatrace.agent.measures.jmx.MBeanReference.getAttributes(MBeanReference.java:58) [?:1.287.149.20240416-100607]
        at com.dynatrace.agent.measures.jmx.MBeanAttributeReader.read(MBeanAttributeReader.java:95) [?:1.287.149.20240416-100607]
        at com.dynatrace.agent.measures.jmx.MBeanSubscriptionManager.performMeasurements(MBeanSubscriptionManager.java:222) [?:1.287.149.20240416-100607]
        at com.dynatrace.agent.measures.jmx.MBeanTracker.performMeasurements(MBeanTracker.java:79) [?:1.287.149.20240416-100607]
        at com.dynatrace.agent.measures.AbstractJmxMetricsProvider.capture(AbstractJmxMetricsProvider.java:82) [?:1.287.149.20240416-100607]
        at com.dynatrace.agent.measures.JmxMetricsProviderV2.getMintLines(JmxMetricsProviderV2.java:103) [?:1.287.149.20240416-100607]
        at com.dynatrace.agent.measures.MetricsManaged.getMintLines(MetricsManaged.java:161) [?:1.287.149.20240416-100607]

Does anyone know why it's started to appear and how to fix it?

Thanks in advance.

Regards,

Elena.

TomásSeroteRoos · ‎17 Jul 2024

From the error it looks like your MBean is missing the records-lag-avg attribute, and looking at the yaml for that extension, that particular attribute seems to be collected from this JMX query:

kafka.consumer:type=consumer-fetch-manager-metrics,client-id=*,partition=*,topic=*

I would go through the MBeans of the host where you are getting the error, look for something that matches that query and check whether or not the attribute is there.

alain_vonallmen · ‎09 Aug 2024

we see something similar and we opened a ticket with dynatrace for that. right now we do not know where this is coming from exactly. this is what we see:

2024-08-09 09:28:29.877 [33m WARN[m [35m7[m --- [iceventsmanaged] [36mo.a.k.c.m.JmxReporter [m : Error getting JMX attribute 'records-lag-avg' javax.management.AttributeNotFoundException: Could not find attribute records-lag-avg

Besides the "records-lag-avg", we also have warnings for "records-lag" and records-lag-max
we have several kafka consumers with this warnings.
the consumers are all configured correctly exposing the beans via jmx
the warnings are not continuous, they can disappear for a week, then appear again for a week under production load.
we use dynatrace managed with kafka extension on openshift operator

BrendanSchembri · ‎30 Aug 2024

We are having the exact same issue - any chance we get a solution from Dynatrace please? @TomásSeroteRoos

BrendanSchembri · ‎10 Sep 2024

Update @erh_inetum @alain_vonallmen ,

I've managed to find a way to disable these 🙂

Go to Dynatrace
Dynatrace Hub
Apache Kafka Extension
Mentoring Configuration and Select Edit:
From Monitoring Source, click Next Step.
From JMX, click Next Step.
On the Finalize Configuration Screen, Search for the Metric in the 'Search Metrics' section and 'untick' it.

The above fixed it for us.

erh_inetum · ‎11 Sep 2024

Hi @BrendanSchembri ,

Thank you so much for the update.

I think this is not valid as final solution because disabling the consumer-metric family (because it's not possible to disable only specific metrics, you know) in which is avg-lag metric, we lose the whole consumers monitoring.

We 've also opened a support ticket. If we have any further information, I'll share it in this link.

Thanks again. Regards,
Elena.

erh_inetum · ‎12 Sep 2024

Hi everyone specially @BrendanSchembri , @alain_vonallmen and @TomásSeroteRoos ,

Support have answered us.

They said that on the exposed bean Kafka-consumer there isn't an attribute called record-lag. So each time the extension query the JMX metrics its module can't find this attribute and the warning is reported in the log.
A solution to this will be having a custom extension that does the same but querying for this attribute, https://docs.dynatrace.com/docs/shortlink/extensions-jmx-customize#metrics

Hope it helps.

Regards,

Elena.