Hey guys, how are you?
My squad have a project that we need to use Dynatrace as our main monitoring tool.
But we need to know if Dynatrace can track some metrics for Kafka and Spark.
Could you help us?
Down below there is a list of metrics we need to monitoring.
consumed messages (Topics)
storage Kafka % --> Alert for 80% of storage
Total of Jobs
Number of bytes recorded
Number of bytes processed
Number of active's tasks
Number of failing Tasks alert
Total of Input Bytes (per second)
Total of Output Bytes (per second)
total of executors
waiting time ---> alert
storage ----> Alert 80%
% memory consume
% queue process
execution time ----> Alert (per Second) *
Average time of messages processed time ----> Alert*
Here are some links to our documentation on both of these technologies, and the metrics we can provide.
Remember that if some of metrics are missing out of the box but are available as jmx mbean, you can always create custom JMX plugin using built in JMX editor to expose those metrics. But you have to have license for custom metrics to do that.