Apache Kafka® Meetup Pune

💡 Speaker:
Kalpesh S. Deo, Lead Software Engineer at One2N

Talk:
Benchmarking Kafka Performance with kafka-producer-perf-test

Abstract:
We will explore how to benchmark Kafka performance using the kafka-producer-perf-test.sh script with various parameter combinations. By writing a script that automates the testing of different configurations, such as compression.type, linger.ms, and batch.size, we were able to optimize Kafka producer throughput by 245%, latency by 90% and storage by 89% in one of our use cases. This approach simplifies the process of identifying the optimal configuration for your specific workload, making it easier to fine-tune Kafka for maximum efficiency.

What the audience will learn:
- Configuring Kafka: Learn how different Kafka configurations, such as compression.type, linger.ms, and batch.size, affect producer performance.
- Automated Benchmarking: See how writing a script to automate the testing of various parameter combinations can simplify performance benchmarking.
- Interpreting Benchmark Results: Insights into how to interpret benchmarking results and use them to make better decisions about Kafka setup for your specific workload.

Bio:
Kalpesh is a tech lead with 16+ years of experience who excels at getting things done. He has extensive expertise in designing and developing API-based backend systems in cloud environments, as well as crafting fast and responsive Mobile and Web UIs. As a Lead Software Engineer at One2N, Kalpesh drives backend engineering in Golang and Java, helping companies successfully launch products.

-----
💡 Speaker:
Suyog Chadawar, Senior Software Engineer, Confluent

Talk:
The Architectural Evolution of Data Platform: From Batch Processing to Real-Time AI

Abstract:
We will dive into the architectural evolution of a data platform for modern recommendation systems, charting a strategic course from foundational batch processing to a real-time, AI-driven system. It begins with a cost-effective architecture using Spark and Hive for daily recommendations and progresses through the introduction of Kafka for real-time ingestion, Flink for in-session personalization, and Apache Iceberg for creating a reliable data lakehouse. The journey continues by simplifying the streaming pipeline with declarative tools like Tableflow, operationalizing insights with Reverse ETL.

Bio:
Suyog Chadawar is a Senior Software Engineer at Confluent, where he specializes in backend engineering and distributed systems. With over 6 years of hands-on experience, Suyog has consistently demonstrated his expertise in building scalable, reliable, and high-performance backend architectures. Driven by a deep passion for modern software architecture, he thrives on solving complex system design challenges and crafting robust solutions that align with the evolving needs of cloud-native platforms.