17 Jul 2024 11:30 AM
Hi,
We added to our environment Apache Kafka extension from Hub. The version configured is 1.1.2.
We configured two endpoints each pointing to a different machine. The configurations are the same.
The extension is capturing information sucessfully.
But the following trace has started to appear in one of the machines in spike of the configuration is the same for the two machines:
2024-07-15T00:00:24,232 WARN [oneagentperiodiceventsmanaged] [org.apache.kafka.common.metrics.JmxReporter] Error getting JMX attribute 'records-lag-avg'
javax.management.AttributeNotFoundException: Could not find attribute records-lag-avg
at org.apache.kafka.common.metrics.JmxReporter$KafkaMbean.getAttribute(JmxReporter.java:255) ~[kafka-clients-2.8.1.jar:?]
at org.apache.kafka.common.metrics.JmxReporter$KafkaMbean.getAttributes(JmxReporter.java:263) [kafka-clients-2.8.1.jar:?]
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttributes(DefaultMBeanServerInterceptor.java:701) [?:?]
at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttributes(JmxMBeanServer.java:705) [?:?]
at com.dynatrace.agent.measures.jmx.MBeanServerProxy$MBeanGetAttributesExecutor.execute(MBeanServerProxy.java:182) [?:1.287.149.20240416-100607]
at com.dynatrace.agent.measures.jmx.MBeanServerProxy$MBeanExecutionStrategy.execute(MBeanServerProxy.java:232) [?:1.287.149.20240416-100607]
at com.dynatrace.agent.measures.jmx.MBeanServerProxy.executeMBeanMethod(MBeanServerProxy.java:92) [?:1.287.149.20240416-100607]
at com.dynatrace.agent.measures.jmx.MBeanServerProxy.getAttributeList(MBeanServerProxy.java:73) [?:1.287.149.20240416-100607]
at com.dynatrace.agent.measures.jmx.MBeanReference.getAttributes(MBeanReference.java:58) [?:1.287.149.20240416-100607]
at com.dynatrace.agent.measures.jmx.MBeanAttributeReader.read(MBeanAttributeReader.java:95) [?:1.287.149.20240416-100607]
at com.dynatrace.agent.measures.jmx.MBeanSubscriptionManager.performMeasurements(MBeanSubscriptionManager.java:222) [?:1.287.149.20240416-100607]
at com.dynatrace.agent.measures.jmx.MBeanTracker.performMeasurements(MBeanTracker.java:79) [?:1.287.149.20240416-100607]
at com.dynatrace.agent.measures.AbstractJmxMetricsProvider.capture(AbstractJmxMetricsProvider.java:82) [?:1.287.149.20240416-100607]
at com.dynatrace.agent.measures.JmxMetricsProviderV2.getMintLines(JmxMetricsProviderV2.java:103) [?:1.287.149.20240416-100607]
at com.dynatrace.agent.measures.MetricsManaged.getMintLines(MetricsManaged.java:161) [?:1.287.149.20240416-100607]
Does anyone know why it's started to appear and how to fix it?
Thanks in advance.
Regards,
Elena.
Solved! Go to Solution.
17 Jul 2024 03:49 PM
From the error it looks like your MBean is missing the records-lag-avg attribute, and looking at the yaml for that extension, that particular attribute seems to be collected from this JMX query:
kafka.consumer:type=consumer-fetch-manager-metrics,client-id=*,partition=*,topic=*
I would go through the MBeans of the host where you are getting the error, look for something that matches that query and check whether or not the attribute is there.
09 Aug 2024 12:27 PM
we see something similar and we opened a ticket with dynatrace for that. right now we do not know where this is coming from exactly. this is what we see:
2024-08-09 09:28:29.877 [33m WARN[m [35m7[m --- [iceventsmanaged] [36mo.a.k.c.m.JmxReporter [m : Error getting JMX attribute 'records-lag-avg' javax.management.AttributeNotFoundException: Could not find attribute records-lag-avg
30 Aug 2024 07:34 AM
We are having the exact same issue - any chance we get a solution from Dynatrace please? @TomásSeroteRoos
10 Sep 2024 02:24 PM
Update @erh_inetum @alain_vonallmen ,
I've managed to find a way to disable these 🙂
The above fixed it for us.
11 Sep 2024 10:45 AM
Hi @BrendanSchembri ,
Thank you so much for the update.
I think this is not valid as final solution because disabling the consumer-metric family (because it's not possible to disable only specific metrics, you know) in which is avg-lag metric, we lose the whole consumers monitoring.
We 've also opened a support ticket. If we have any further information, I'll share it in this link.
Thanks again. Regards,
Elena.
12 Sep 2024 01:35 PM
Hi everyone specially @BrendanSchembri , @alain_vonallmen and @TomásSeroteRoos ,
Support have answered us.
They said that on the exposed bean Kafka-consumer there isn't an attribute called record-lag. So each time the extension query the JMX metrics its module can't find this attribute and the warning is reported in the log.
A solution to this will be having a custom extension that does the same but querying for this attribute, https://docs.dynatrace.com/docs/shortlink/extensions-jmx-customize#metrics
Hope it helps.
Regards,
Elena.
13 Nov 2024 09:30 AM
Hi, @erh_inetum
I run into the same problem now, so locky to get your message.
Do you know how to use the custom extension?
04 Dec 2024 03:42 PM
Please note that the record-lags attribute are not exposed every time in Kafka. See also known behavior / bug https://issues.apache.org/jira/browse/KAFKA-17652