Looking for Kafka producer, consumer and broker metrics such as response time, wait time and result wait time

Platform Automations
,
1

we are doing performance testing for an AI Application, where Kafka is acting as messaging system to handle the message requests from client server.

I would like to get the below metrics from the dynatrace,

Kafka Idle time

kafka Broker message processing time

Kafka invoking API time

API’s response time

Kafka result wait time to write it in DB

these are metrics which im looking to get to know how my Kafka is performing when actual load applied in my application.

I heard that if dev is not exposing any data from the kafka, then we couldn’t get track the above mentioned metrics from kafka. Kindly help me if you have ever been into that issue!

Thanks,

Prasanth S

2

About dev - Dynatrace automatically recognizes Kafka processes and instantly gathers Kafka metrics on the process and cluster levels.

See documentation https://www.dynatrace.com/support/help/technology-support/dynatrace-extensions/supported-out-of-the-box/kafka you will find there exhaustive information about out of the box kafka metrics. You can use calculatade metrics for built custom metrics based on them.

3

Hi Prasanth,
To answer you quickly, you can check this link, if you did not, according to it you have possibility to catch more finely the required metrics Custom messaging services | Dynatrace Docs . 
To achieve this you need to enable the deep monitoring for Apache Kafka and to have purepaths you need to define a custom service (with some requisites). For info, available since Oneagent 1.225 version.

Moreover, you have those metrics : Confluent Cloud (Kafka) extension — Dynatrace Docs

For more out of the scope, we use extension via our Hub :  Apache Kafka | Dynatrace Hub
Thanks,
Uros