Services

Resources

Company

/

Apache Kafka

Apache Kafka

Apache Kafka is an open-source event streaming platform. Producers append events to named topics, Kafka stores them on a cluster of brokers, and consumers read them at their own pace. Unlike a classic queue, reading an event does not delete it. Events stay for a retention period (7 days by default), so many independent consumers can read the same stream and a consumer can replay it from an earlier point.

How it works

A topic is split into partitions, and each partition is an append-only log replicated across brokers. Every event in a partition gets an offset, its position in that log. Producers pick the partition from a hash of the event key, so all events for one key, such as one order ID, land in the same partition and stay in order. Kafka only guarantees order within a partition, not across a whole topic.

Consumers that share a group.id form a consumer group. Kafka gives each partition to exactly one consumer in the group and the group commits its offset per partition, so a restarted consumer resumes where the group left off. Each separate group gets every event.

For example, a topic orders has 6 partitions and producers write 2,400 events a second. The billing group runs 3 consumers that each handle 500 a second, so it reads 1,500 a second and falls 900 events further behind every second. That gap is consumer lag. Scale to 6 consumers, one per partition, and the group reads 3,000 a second and catches up at 600 a second. A 7th consumer would sit idle, because the partition count caps how many consumers in a group can work.

Offsets also explain duplicates: if a consumer processes offsets 5,100 to 5,149 on partition 2 and crashes before committing, its replacement starts again at 5,100 and those 50 events are processed twice. That is at-least-once delivery, so consumers need idempotent handling or exactly-once semantics.

Why it matters in production

Running Kafka gets hard as it grows. In One2N's work on one large setup there were 78 clusters across 17 environments, with 10,000+ topics and nearly 40,000 partitions, on versions from 0.8.2 to 4.x. The fix was to manage topics and access control lists (ACLs) as code with GitOps.

Plan partition counts early: you can add partitions but not remove them, and adding them moves keys to new partitions, which breaks per-key ordering for a while. Watch consumer lag, disk use against retention, and under-replicated partitions. Kafka is also the usual transport for change data capture with Debezium and for the outbox pattern.

See also

On the One2N blog

References

  1. Apache Kafka documentation

  2. Apache Kafka: Introduction

Diagram of Apache Kafka: producers writing 2,400 events a second fan out across the six partitions, p0 to p5, of the topic orders, each drawn as an append-only log. A billing consumer group of three consumers reads two partitions per consumer at 500 a second each. Underneath, 2,400 in − 3 x 500 read = 900 a second of lag, with the lag shown in red.