11 Oct 202302:05 PM - last edited on 11 Oct 202302:10 PM by Ana_Kuzmenchuk
We are looking into tracking our AWS MSK (Kafka) metrics on a more granular level, being able to pinpoint message throughput from topic to topic and finally to the database, similar to what you could achieve with AWS x-ray.
A demo setup could be: raw.topic -> transformer -> processed.topic -> database -> API.
We'd like to be able to monitor how long from a message being published in the raw.topic it takes to be served from the API. Also the duration of different steps, such as how long it takes from transformer reading the message from the raw topic and saving it in the processed topic would be interesting or the time it takes from raw.topic to database.
Is this sort of monitoring possible? I have been looking at the AWS MSK integration and also written a couple of custom JMX extensions to monitor things such as kafka connector and kafka streams, but this seems something not supported out of the box.