How is the messages per second production number calculated for your Apache Kafka® topics? The calculations are based on polling the end offsets of all partitions in the topic. The simple example below shows how the calculations are performed in gradient fox for one topic. At cluster level, the same logic is simply applied to all topics in the cluster and the topic numbers are added up to get to the cluster-level number.
Imagine a topic with three partitions P1-3 as shown below, where messages are shown as rectangles and the offset of each message as a number inside the rectangle. The first polling is done at 22:52:09 and the following end offsets are found
The next polling is done at 22:52:11 and the following end offsets are found
The messages added after the first poll are shown in green. By subtracting the first end offsets from the second set of end offsets, we get the total number of added messages:
(11 - 6) + (6 - 3) + (14 - 8) = 14
As can be seen, this is also the number of green rectangles in the picture. Now to get the messages / second number we simply divide the number of added messages by the number of seconds between the polls:
14 messages / 2 seconds = 7 messages/second
Now, in reality the polling interval is much higher than two seconds. In many cases your production number is also a lot higher than 7 messages per second, but this simple example gives an idea of how the calculations are performed behind the scenes in gradient fox.