Apache Kafka
Apache Kafka is an open-source event streaming platform. Producers append events to named topics, Kafka stores them on a cluster of brokers, and consumers read them at their own pace. Unlike a classic queue, reading an event does not delete it. Events stay for a retention period (7 days by default), so many independent consumers can read the same stream and a consumer can replay it from an earlier point.
How it works
A topic is split into partitions, and each partition is an append-only log replicated across brokers. Every event in a partition gets an offset, its position in that log. Producers pick the partition from a hash of the event key, so all events for one key, such as one order ID, land in the same partition and stay in order. Kafka only guarantees order within a partition, not across a whole topic.
Consumers that share a group.id form a consumer group. Kafka gives each partition to exactly one consumer in the group and the group commits its offset per partition, so a restarted consumer resumes where the group left off. Each separate group gets every event.
For example, a topic orders has 6 partitions and producers write 2,400 events a second. The billing group runs 3 consumers that each handle 500 a second, so it reads 1,500 a second and falls 900 events further behind every second. That gap is consumer lag. Scale to 6 consumers, one per partition, and the group reads 3,000 a second and catches up at 600 a second. A 7th consumer would sit idle, because the partition count caps how many consumers in a group can work.
Offsets also explain duplicates: if a consumer processes offsets 5,100 to 5,149 on partition 2 and crashes before committing, its replacement starts again at 5,100 and those 50 events are processed twice. That is at-least-once delivery, so consumers need idempotent handling or exactly-once semantics.
Why it matters in production
Running Kafka gets hard as it grows. In One2N's work on one large setup there were 78 clusters across 17 environments, with 10,000+ topics and nearly 40,000 partitions, on versions from 0.8.2 to 4.x. The fix was to manage topics and access control lists (ACLs) as code with GitOps.
Plan partition counts early: you can add partitions but not remove them, and adding them moves keys to new partitions, which breaks per-key ordering for a while. Watch consumer lag, disk use against retention, and under-replicated partitions. Kafka is also the usual transport for change data capture with Debezium and for the outbox pattern.
See also
On the One2N blog
Fixing dual writes with the transactional outbox pattern and Debezium
Temporal versioning to stop config drift in stream processing
References
