Showing results for 
Show  only  | Search instead for 
Did you mean: 

AWS MSK throughput similar to purepath/x-ray/spans

Frequent Guest


We are looking into tracking our AWS MSK (Kafka) metrics on a more granular level, being able to pinpoint message throughput from topic to topic and finally to the database, similar to what you could achieve with AWS x-ray.

A demo setup could be: raw.topic -> transformer -> processed.topic -> database -> API.

We'd like to be able to monitor how long from a message being published in the raw.topic it takes to be served from the API. Also the duration of different steps, such as how long it takes from transformer reading the message from the raw topic and saving it in the processed topic would be interesting or the time it takes from raw.topic to database.

Is this sort of monitoring possible? I have been looking at the AWS MSK integration and also written a couple of custom JMX extensions to monitor things such as kafka connector and kafka streams, but this seems something not supported out of the box.


Frequent Guest

A similar setup such as this: Tracing End-to-End Performance of Messaging Applications Built on Apache Kafka Using AWS X-Ray | AWS... . Using X-ray as an example just because AWS is familiar for me, the underlying concept is the important part, not the technology used.

Featured Posts