cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Trace error "Error getting JMX attribute 'records-lag-avg'" from Apache Kafka Hub Extension

Hi,

We added to our environment Apache Kafka extension from Hub. The version configured is 1.1.2.

We configured two endpoints each pointing to a different machine. The configurations are the same.

The extension is capturing information sucessfully.

But the following trace has started to appear in one of the machines in spike of the configuration is the same for the two machines:

2024-07-15T00:00:24,232 WARN  [oneagentperiodiceventsmanaged] [org.apache.kafka.common.metrics.JmxReporter] Error getting JMX attribute 'records-lag-avg'
javax.management.AttributeNotFoundException: Could not find attribute records-lag-avg

        at org.apache.kafka.common.metrics.JmxReporter$KafkaMbean.getAttribute(JmxReporter.java:255) ~[kafka-clients-2.8.1.jar:?]
        at org.apache.kafka.common.metrics.JmxReporter$KafkaMbean.getAttributes(JmxReporter.java:263) [kafka-clients-2.8.1.jar:?]
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttributes(DefaultMBeanServerInterceptor.java:701) [?:?]
        at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttributes(JmxMBeanServer.java:705) [?:?]
        at com.dynatrace.agent.measures.jmx.MBeanServerProxy$MBeanGetAttributesExecutor.execute(MBeanServerProxy.java:182) [?:1.287.149.20240416-100607]
        at com.dynatrace.agent.measures.jmx.MBeanServerProxy$MBeanExecutionStrategy.execute(MBeanServerProxy.java:232) [?:1.287.149.20240416-100607]
        at com.dynatrace.agent.measures.jmx.MBeanServerProxy.executeMBeanMethod(MBeanServerProxy.java:92) [?:1.287.149.20240416-100607]
        at com.dynatrace.agent.measures.jmx.MBeanServerProxy.getAttributeList(MBeanServerProxy.java:73) [?:1.287.149.20240416-100607]
        at com.dynatrace.agent.measures.jmx.MBeanReference.getAttributes(MBeanReference.java:58) [?:1.287.149.20240416-100607]
        at com.dynatrace.agent.measures.jmx.MBeanAttributeReader.read(MBeanAttributeReader.java:95) [?:1.287.149.20240416-100607]
        at com.dynatrace.agent.measures.jmx.MBeanSubscriptionManager.performMeasurements(MBeanSubscriptionManager.java:222) [?:1.287.149.20240416-100607]
        at com.dynatrace.agent.measures.jmx.MBeanTracker.performMeasurements(MBeanTracker.java:79) [?:1.287.149.20240416-100607]
        at com.dynatrace.agent.measures.AbstractJmxMetricsProvider.capture(AbstractJmxMetricsProvider.java:82) [?:1.287.149.20240416-100607]
        at com.dynatrace.agent.measures.JmxMetricsProviderV2.getMintLines(JmxMetricsProviderV2.java:103) [?:1.287.149.20240416-100607]
        at com.dynatrace.agent.measures.MetricsManaged.getMintLines(MetricsManaged.java:161) [?:1.287.149.20240416-100607]

 

Does anyone know why it's started to appear and how to fix it?

Thanks in advance.

Regards,

Elena.

 

8 REPLIES 8

TomásSeroteRoos
Dynatrace Helper
Dynatrace Helper

From the error it looks like your MBean is missing the records-lag-avg attribute, and looking at the yaml for that extension, that particular attribute seems to be collected from this JMX query:

kafka.consumer:type=consumer-fetch-manager-metrics,client-id=*,partition=*,topic=*

I would go through the MBeans of the host where you are getting the error, look for something that matches that query and check whether or not the attribute is there.

alain_vonallmen
Frequent Guest

we see something similar and we opened a ticket with dynatrace for that. right now we do not know where this is coming from exactly. this is what we see:

 

2024-08-09 09:28:29.877  WARN 7 --- [iceventsmanaged] o.a.k.c.m.JmxReporter  : Error getting JMX attribute 'records-lag-avg' javax.management.AttributeNotFoundException: Could not find attribute records-lag-avg
  • Besides the "records-lag-avg", we also have warnings for "records-lag" and records-lag-max
  • we have several kafka consumers with this warnings.
  • the consumers are all configured correctly exposing the beans via jmx
  • the warnings are not continuous, they can disappear for a week, then appear again for a week under production load.
  • we use dynatrace managed with kafka extension on openshift operator

 

 

BrendanSchembri
Newcomer

We are having the exact same issue - any chance we get a solution from Dynatrace please? @TomásSeroteRoos 

BrendanSchembri
Newcomer

Update @erh_inetum @alain_vonallmen ,

 

I've managed to find a way to disable these 🙂

  1. Go to Dynatrace
  2. Dynatrace Hub
  3. Apache Kafka Extension
  4. Mentoring Configuration and Select Edit:
    BrendanSchembri_0-1725974613845.png

     

  5. From Monitoring Source, click Next Step.
  6. From JMX, click Next Step.
  7. On the Finalize Configuration Screen, Search for the Metric in the 'Search Metrics' section and 'untick' it.

 

The above fixed it for us.

 

Hi @BrendanSchembri ,

Thank you so much for the update.

I think this is not valid as final solution because disabling the consumer-metric family (because it's not possible to disable only specific metrics, you know) in which is avg-lag metric, we lose the whole consumers monitoring.

We 've also opened a support ticket. If we have any further information, I'll share it in this link.

 

Thanks again. Regards,
Elena.

Hi everyone specially @BrendanSchembri , @alain_vonallmen  and @TomásSeroteRoos ,

Support have answered us.

They said that on the exposed bean Kafka-consumer there isn't an attribute called record-lag. So each time the extension query the JMX metrics its module can't find this attribute and the warning is reported in the log.
A solution to this will be having a custom extension that does the same but querying for this attribute, https://docs.dynatrace.com/docs/shortlink/extensions-jmx-customize#metrics

Hope it helps.

Regards,

Elena.

Hi, @erh_inetum 

  I run into the same problem now, so locky to get your message.
  Do you know how to use the custom extension?

benzhonghai008_0-1731490069885.png

 

harald_berger
Dynatrace Champion
Dynatrace Champion

Please note that the record-lags attribute are not exposed every time in Kafka. See also known behavior / bug https://issues.apache.org/jira/browse/KAFKA-17652

Featured Posts