AWS MSK throughput similar to purepath/x-ray/spans

[snurmivaara]

Greetings,

We are looking into tracking our AWS MSK (Kafka) metrics on a more granular level, being able to pinpoint message throughput from topic to topic and finally to the database, similar to what you could achieve with AWS x-ray.

A demo setup could be: raw.topic -> transformer -> processed.topic -> database -> API.

We'd like to be able to monitor how long from a message being published in the raw.topic it takes to be served from the API. Also the duration of different steps, such as how long it takes from transformer reading the message from the raw topic and saving it in the processed topic would be interesting or the time it takes from raw.topic to database.

Is this sort of monitoring possible? I have been looking at the AWS MSK integration and also written a couple of custom JMX extensions to monitor things such as kafka connector and kafka streams, but this seems something not supported out of the box.

[snurmivaara]

A similar setup such as this: Tracing End-to-End Performance of Messaging Applications Built on Apache Kafka Using AWS X-Ray | AWS... . Using X-ray as an example just because AWS is familiar for me, the underlying concept is the important part, not the technology used.