Optimizing Kafka Performance with Automated Benchmarks
Lead Software Engineer @One2N
Curious about getting the best performance out of your Kafka setup? In this talk, Kalpesh from One2N walks us through his experience of benchmarking Kafka producers using the kafka-producer-perf-test.sh tool. By experimenting with settings like compression.type, linger.ms, and batch.size, Kalpesh’s team managed to boost throughput by 245%, cut down latency by 90%, and save 89% on storage all by fine-tuning some key configurations.
What you’ll learn:
How tuning Kafka producer settings impacts real throughput, latency, and storage.
The power of scripting to automate and simplify performance benchmarking.
Techniques to interpret and apply benchmarking results for your own Kafka deployment.
Kalpesh’s 16+ years in backend, cloud, and distributed engineering shine through as he delivers actionable insights for engineers seeking to fine-tune their data pipelines
