Kafka consumer lag

Kafka consumer lag is the number of messages a consumer group has not yet processed in a partition. It is the gap between the newest message written to the partition and the last offset the group has committed. Lag tells you whether consumers keep up with producers, and how far behind real time your data is.

How to calculate it

Lag per partition = log end offset minus committed offset. Total lag for a group is the sum across its partitions. For example, if a partition's log end offset is 1,250,000 and the group has committed offset 1,200,000, lag is 50,000 messages. If the consumer processes 1,000 messages per second and producers stop, it needs about 50 seconds to catch up. kafka-consumer-groups.sh --describe --group <name> prints CURRENT-OFFSET, LOG-END-OFFSET and LAG for each partition.

Why it matters in production

Rising lag is often the first sign of a slow consumer, a hot partition, a rebalance loop or a downstream dependency that has slowed down. If lag grows past the topic's retention period, Kafka deletes messages before they are read, and that data is lost to the consumer. Alert on lag trend and on time behind, not on a single raw number.