cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Metrics with Kafka and Spark

clederson_bispo
Newcomer

Hey guys, how are you?


My squad have a project that we need to use Dynatrace as our main monitoring tool.

But we need to know if Dynatrace can track some metrics for Kafka and Spark.


Could you help us?


Down below there is a list of metrics we need to monitoring.


Kafka

Health Alert

consumed messages (Topics)

storage Kafka % --> Alert for 80% of storage


Spark

Health Alert

Total of Jobs

Number of bytes recorded

Number of bytes processed

Number of active's tasks

Number of failing Tasks alert

Total of Input Bytes (per second)

Total of Output Bytes (per second)

total of executors

waiting time ---> alert

storage ----> Alert 80%

% memory consume

% queue process

execution time ----> Alert (per Second) *

Average time of messages processed time ----> Alert*


3 REPLIES 3

jonathan_schull
Dynatrace Advisor
Dynatrace Advisor

Hello Clederson,


Here are some links to our documentation on both of these technologies, and the metrics we can provide.

Kafka: https://www.dynatrace.com/support/help/technology-support/application-software/other-technologies/su...

Spark: https://www.dynatrace.com/support/help/technology-support/application-software/other-technologies/su...


Cheers,

Jonathan


clederson_bispo
Newcomer

@ Jonathan S. ,


Obrigado pelo seu tempo.

Na verdade, eu consegui essas páginas para identificar métricas, mas não encontrei nenhuma informação sobre as métricas que preciso.

Essas páginas têm todas as métricas que o Dynatrace pode suportar para kafka e nifi?

Cumprimentos,


Leandro R.


Remember that if some of metrics are missing out of the box but are available as jmx mbean, you can always create custom JMX plugin using built in JMX editor to expose those metrics. But you have to have license for custom metrics to do that.

https://www.dynatrace.com/news/blog/jmx-plugins-editor-now-available-eap/

Sebastian