Services

Resources

Company

Our Work

Blog

Data Engineering

Back to Tags

Blogs

ClickHouse schema optimization for sparse data sets

Amit Sadafule

Lead Software Engineer

Optimize ClickHouse schema design for efficiently storing and querying sparse data sets. Explore strategies like sparse matrix, separate tables, and datatype-specific columns to improve compression, query performance, and reduce RAM usage.

September 19, 2024 | 10 min read

ClickHouse schema optimization for sparse data sets

Amit Sadafule

Lead Software Engineer

Optimize ClickHouse schema design for efficiently storing and querying sparse data sets. Explore strategies like sparse matrix, separate tables, and datatype-specific columns to improve compression, query performance, and reduce RAM usage.

September 19, 2024 | 10 min read

ClickHouse schema optimization for sparse data sets

Amit Sadafule

Lead Software Engineer

Optimize ClickHouse schema design for efficiently storing and querying sparse data sets. Explore strategies like sparse matrix, separate tables, and datatype-specific columns to improve compression, query performance, and reduce RAM usage.

September 19, 2024 | 10 min read

ClickHouse schema optimization for sparse data sets

Amit Sadafule

Lead Software Engineer

Optimize ClickHouse schema design for efficiently storing and querying sparse data sets. Explore strategies like sparse matrix, separate tables, and datatype-specific columns to improve compression, query performance, and reduce RAM usage.

September 19, 2024 | 10 min read

ClickHouse schema optimization for sparse data sets

Amit Sadafule

Lead Software Engineer

Optimize ClickHouse schema design for efficiently storing and querying sparse data sets. Explore strategies like sparse matrix, separate tables, and datatype-specific columns to improve compression, query performance, and reduce RAM usage.

September 19, 2024 | 10 min read

Database schema migrations: a practical guide for mastering change

Spandan Ghosh

Content @One2N

Database schema changes are inevitable in software development. This guide teaches you how to automate schema migrations, reducing deployment headaches and increasing agility.

April 25, 2024 | 5 min read

Database schema migrations: a practical guide for mastering change

Spandan Ghosh

Content @One2N

Database schema changes are inevitable in software development. This guide teaches you how to automate schema migrations, reducing deployment headaches and increasing agility.

April 25, 2024 | 5 min read

Database schema migrations: a practical guide for mastering change

Spandan Ghosh

Content @One2N

Database schema changes are inevitable in software development. This guide teaches you how to automate schema migrations, reducing deployment headaches and increasing agility.

April 25, 2024 | 5 min read

Database schema migrations: a practical guide for mastering change

Spandan Ghosh

Content @One2N

Database schema changes are inevitable in software development. This guide teaches you how to automate schema migrations, reducing deployment headaches and increasing agility.

April 25, 2024 | 5 min read

Database schema migrations: a practical guide for mastering change

Spandan Ghosh

Content @One2N

Database schema changes are inevitable in software development. This guide teaches you how to automate schema migrations, reducing deployment headaches and increasing agility.

April 25, 2024 | 5 min read

Implementing a real-time analytics engine using big data principles

Jaideep Khandelwal

Founder, CTO @One2N

Learn how to implement a real-time analytics engine using big data principles. The challenges, approach, and tools used to build a scalable, high-performance data analytics system. Improve your data engineering skills with this in-depth technical blog.

July 14, 2022 | 8 min read

Implementing a real-time analytics engine using big data principles

Jaideep Khandelwal

Founder, CTO @One2N

Learn how to implement a real-time analytics engine using big data principles. The challenges, approach, and tools used to build a scalable, high-performance data analytics system. Improve your data engineering skills with this in-depth technical blog.

July 14, 2022 | 8 min read

Implementing a real-time analytics engine using big data principles

Jaideep Khandelwal

Founder, CTO @One2N

Learn how to implement a real-time analytics engine using big data principles. The challenges, approach, and tools used to build a scalable, high-performance data analytics system. Improve your data engineering skills with this in-depth technical blog.

July 14, 2022 | 8 min read

Implementing a real-time analytics engine using big data principles

Jaideep Khandelwal

Founder, CTO @One2N

Learn how to implement a real-time analytics engine using big data principles. The challenges, approach, and tools used to build a scalable, high-performance data analytics system. Improve your data engineering skills with this in-depth technical blog.

July 14, 2022 | 8 min read

Implementing a real-time analytics engine using big data principles

Jaideep Khandelwal

Founder, CTO @One2N

Learn how to implement a real-time analytics engine using big data principles. The challenges, approach, and tools used to build a scalable, high-performance data analytics system. Improve your data engineering skills with this in-depth technical blog.

July 14, 2022 | 8 min read

Talks

Optimizing Kafka Performance with Automated Benchmarks

Kalpesh Deo

Lead Software Engineer @One2N

Curious about getting the best performance out of your Kafka setup? In this talk, Kalpesh from One2N walks us through his experience of benchmarking Kafka producers using the kafka-producer-perf-test.sh tool. By experimenting with settings like compression.type, linger.ms, and batch.size, Kalpesh’s team managed to boost throughput by 245%, cut down latency by 90%, and save 89% on storage all by fine-tuning some key configurations. What you’ll learn: How tuning Kafka producer settings impacts real throughput, latency, and storage. The power of scripting to automate and simplify performance benchmarking. Techniques to interpret and apply benchmarking results for your own Kafka deployment. Kalpesh’s 16+ years in backend, cloud, and distributed engineering shine through as he delivers actionable insights for engineers seeking to fine-tune their data pipelines

June 7, 2025 | 27 min

Optimizing Kafka Performance with Automated Benchmarks

Kalpesh Deo

Lead Software Engineer @One2N

Curious about getting the best performance out of your Kafka setup? In this talk, Kalpesh from One2N walks us through his experience of benchmarking Kafka producers using the kafka-producer-perf-test.sh tool. By experimenting with settings like compression.type, linger.ms, and batch.size, Kalpesh’s team managed to boost throughput by 245%, cut down latency by 90%, and save 89% on storage all by fine-tuning some key configurations. What you’ll learn: How tuning Kafka producer settings impacts real throughput, latency, and storage. The power of scripting to automate and simplify performance benchmarking. Techniques to interpret and apply benchmarking results for your own Kafka deployment. Kalpesh’s 16+ years in backend, cloud, and distributed engineering shine through as he delivers actionable insights for engineers seeking to fine-tune their data pipelines

June 7, 2025 | 27 min

Optimizing Kafka Performance with Automated Benchmarks

Kalpesh Deo

Lead Software Engineer @One2N

Curious about getting the best performance out of your Kafka setup? In this talk, Kalpesh from One2N walks us through his experience of benchmarking Kafka producers using the kafka-producer-perf-test.sh tool. By experimenting with settings like compression.type, linger.ms, and batch.size, Kalpesh’s team managed to boost throughput by 245%, cut down latency by 90%, and save 89% on storage all by fine-tuning some key configurations. What you’ll learn: How tuning Kafka producer settings impacts real throughput, latency, and storage. The power of scripting to automate and simplify performance benchmarking. Techniques to interpret and apply benchmarking results for your own Kafka deployment. Kalpesh’s 16+ years in backend, cloud, and distributed engineering shine through as he delivers actionable insights for engineers seeking to fine-tune their data pipelines

June 7, 2025 | 27 min

Optimizing Kafka Performance with Automated Benchmarks

Kalpesh Deo

Lead Software Engineer @One2N

Curious about getting the best performance out of your Kafka setup? In this talk, Kalpesh from One2N walks us through his experience of benchmarking Kafka producers using the kafka-producer-perf-test.sh tool. By experimenting with settings like compression.type, linger.ms, and batch.size, Kalpesh’s team managed to boost throughput by 245%, cut down latency by 90%, and save 89% on storage all by fine-tuning some key configurations. What you’ll learn: How tuning Kafka producer settings impacts real throughput, latency, and storage. The power of scripting to automate and simplify performance benchmarking. Techniques to interpret and apply benchmarking results for your own Kafka deployment. Kalpesh’s 16+ years in backend, cloud, and distributed engineering shine through as he delivers actionable insights for engineers seeking to fine-tune their data pipelines

June 7, 2025 | 27 min

Optimizing Kafka Performance with Automated Benchmarks

Kalpesh Deo

Lead Software Engineer @One2N

Curious about getting the best performance out of your Kafka setup? In this talk, Kalpesh from One2N walks us through his experience of benchmarking Kafka producers using the kafka-producer-perf-test.sh tool. By experimenting with settings like compression.type, linger.ms, and batch.size, Kalpesh’s team managed to boost throughput by 245%, cut down latency by 90%, and save 89% on storage all by fine-tuning some key configurations. What you’ll learn: How tuning Kafka producer settings impacts real throughput, latency, and storage. The power of scripting to automate and simplify performance benchmarking. Techniques to interpret and apply benchmarking results for your own Kafka deployment. Kalpesh’s 16+ years in backend, cloud, and distributed engineering shine through as he delivers actionable insights for engineers seeking to fine-tune their data pipelines

June 7, 2025 | 27 min

What does It take to build a Software Defined Data Center at scale

Srivatsa RV

SRE @One2N

Join Srivatsa RV, SRE at One2N, as he breaks down the complexities of scaling a Software-Defined Data Center. Learn how modern infrastructure uses virtualization, hyperconvergence, and distributed systems to deliver resilient, API-driven solutions. 🔍 Key Topics Covered - Virtualization & Hypervisors: Type-1 vs. Type-2, KVM, and Nutanix AHV - Hyperconverged Infrastructure (HCI): Scalability, fault tolerance, and real-world deployments (e.g., 15,000+ VMs across 800 servers) - Failure Modes: Disk, node, and cluster-level resilience strategies Cool Tech Demos: - VM Live Migrations: Mitigating hardware failures without downtime - Data Center-Level Failovers: Ensuring business continuity - Capacity Planning: Metrics-driven approaches for CPU, memory, and storage

March 1, 2025 | 85 min

What does It take to build a Software Defined Data Center at scale

Srivatsa RV

SRE @One2N

Join Srivatsa RV, SRE at One2N, as he breaks down the complexities of scaling a Software-Defined Data Center. Learn how modern infrastructure uses virtualization, hyperconvergence, and distributed systems to deliver resilient, API-driven solutions. 🔍 Key Topics Covered - Virtualization & Hypervisors: Type-1 vs. Type-2, KVM, and Nutanix AHV - Hyperconverged Infrastructure (HCI): Scalability, fault tolerance, and real-world deployments (e.g., 15,000+ VMs across 800 servers) - Failure Modes: Disk, node, and cluster-level resilience strategies Cool Tech Demos: - VM Live Migrations: Mitigating hardware failures without downtime - Data Center-Level Failovers: Ensuring business continuity - Capacity Planning: Metrics-driven approaches for CPU, memory, and storage

March 1, 2025 | 85 min

What does It take to build a Software Defined Data Center at scale

Srivatsa RV

SRE @One2N

Join Srivatsa RV, SRE at One2N, as he breaks down the complexities of scaling a Software-Defined Data Center. Learn how modern infrastructure uses virtualization, hyperconvergence, and distributed systems to deliver resilient, API-driven solutions. 🔍 Key Topics Covered - Virtualization & Hypervisors: Type-1 vs. Type-2, KVM, and Nutanix AHV - Hyperconverged Infrastructure (HCI): Scalability, fault tolerance, and real-world deployments (e.g., 15,000+ VMs across 800 servers) - Failure Modes: Disk, node, and cluster-level resilience strategies Cool Tech Demos: - VM Live Migrations: Mitigating hardware failures without downtime - Data Center-Level Failovers: Ensuring business continuity - Capacity Planning: Metrics-driven approaches for CPU, memory, and storage

March 1, 2025 | 85 min

What does It take to build a Software Defined Data Center at scale

Srivatsa RV

SRE @One2N

Join Srivatsa RV, SRE at One2N, as he breaks down the complexities of scaling a Software-Defined Data Center. Learn how modern infrastructure uses virtualization, hyperconvergence, and distributed systems to deliver resilient, API-driven solutions. 🔍 Key Topics Covered - Virtualization & Hypervisors: Type-1 vs. Type-2, KVM, and Nutanix AHV - Hyperconverged Infrastructure (HCI): Scalability, fault tolerance, and real-world deployments (e.g., 15,000+ VMs across 800 servers) - Failure Modes: Disk, node, and cluster-level resilience strategies Cool Tech Demos: - VM Live Migrations: Mitigating hardware failures without downtime - Data Center-Level Failovers: Ensuring business continuity - Capacity Planning: Metrics-driven approaches for CPU, memory, and storage

March 1, 2025 | 85 min

What does It take to build a Software Defined Data Center at scale

Srivatsa RV

SRE @One2N

Join Srivatsa RV, SRE at One2N, as he breaks down the complexities of scaling a Software-Defined Data Center. Learn how modern infrastructure uses virtualization, hyperconvergence, and distributed systems to deliver resilient, API-driven solutions. 🔍 Key Topics Covered - Virtualization & Hypervisors: Type-1 vs. Type-2, KVM, and Nutanix AHV - Hyperconverged Infrastructure (HCI): Scalability, fault tolerance, and real-world deployments (e.g., 15,000+ VMs across 800 servers) - Failure Modes: Disk, node, and cluster-level resilience strategies Cool Tech Demos: - VM Live Migrations: Mitigating hardware failures without downtime - Data Center-Level Failovers: Ensuring business continuity - Capacity Planning: Metrics-driven approaches for CPU, memory, and storage

March 1, 2025 | 85 min

Mohit Kumar - Billion Rows Walk Into a Go Bar

Mohit Kumar

Software Engineer

In this talk from GopherCon India 2024, we delve into the quest for extreme performance optimization by tackling the 1BRC (1 Billion Row Challenge). Discover how a program initially taking several minutes to complete was transformed to run in just a few seconds. Along the way, we’ll explore how to move beyond abstractions and deal directly with data at massive scale. We’ll examine practical techniques, such as using pprof for pinpointing memory allocations, garbage collection overhead, and performance bottlenecks. Learn about the subtle code changes—like one that cut memory allocations from 15+GB down to a mere 2MB—and gain insights that you can immediately apply to your own Go projects. By watching this talk, you’ll: Develop “mechanical sympathy” when handling large-scale data. Understand how pprof can identify the hidden bottlenecks lurking in your code. Learn to write more efficient and performant Go programs in your everyday development work.

December 1, 2024 | 24 min

Mohit Kumar - Billion Rows Walk Into a Go Bar

Mohit Kumar

Software Engineer

In this talk from GopherCon India 2024, we delve into the quest for extreme performance optimization by tackling the 1BRC (1 Billion Row Challenge). Discover how a program initially taking several minutes to complete was transformed to run in just a few seconds. Along the way, we’ll explore how to move beyond abstractions and deal directly with data at massive scale. We’ll examine practical techniques, such as using pprof for pinpointing memory allocations, garbage collection overhead, and performance bottlenecks. Learn about the subtle code changes—like one that cut memory allocations from 15+GB down to a mere 2MB—and gain insights that you can immediately apply to your own Go projects. By watching this talk, you’ll: Develop “mechanical sympathy” when handling large-scale data. Understand how pprof can identify the hidden bottlenecks lurking in your code. Learn to write more efficient and performant Go programs in your everyday development work.

December 1, 2024 | 24 min

Mohit Kumar - Billion Rows Walk Into a Go Bar

Mohit Kumar

Software Engineer

In this talk from GopherCon India 2024, we delve into the quest for extreme performance optimization by tackling the 1BRC (1 Billion Row Challenge). Discover how a program initially taking several minutes to complete was transformed to run in just a few seconds. Along the way, we’ll explore how to move beyond abstractions and deal directly with data at massive scale. We’ll examine practical techniques, such as using pprof for pinpointing memory allocations, garbage collection overhead, and performance bottlenecks. Learn about the subtle code changes—like one that cut memory allocations from 15+GB down to a mere 2MB—and gain insights that you can immediately apply to your own Go projects. By watching this talk, you’ll: Develop “mechanical sympathy” when handling large-scale data. Understand how pprof can identify the hidden bottlenecks lurking in your code. Learn to write more efficient and performant Go programs in your everyday development work.

December 1, 2024 | 24 min

Mohit Kumar - Billion Rows Walk Into a Go Bar

Mohit Kumar

Software Engineer

In this talk from GopherCon India 2024, we delve into the quest for extreme performance optimization by tackling the 1BRC (1 Billion Row Challenge). Discover how a program initially taking several minutes to complete was transformed to run in just a few seconds. Along the way, we’ll explore how to move beyond abstractions and deal directly with data at massive scale. We’ll examine practical techniques, such as using pprof for pinpointing memory allocations, garbage collection overhead, and performance bottlenecks. Learn about the subtle code changes—like one that cut memory allocations from 15+GB down to a mere 2MB—and gain insights that you can immediately apply to your own Go projects. By watching this talk, you’ll: Develop “mechanical sympathy” when handling large-scale data. Understand how pprof can identify the hidden bottlenecks lurking in your code. Learn to write more efficient and performant Go programs in your everyday development work.

December 1, 2024 | 24 min

Mohit Kumar - Billion Rows Walk Into a Go Bar

Mohit Kumar

Software Engineer

In this talk from GopherCon India 2024, we delve into the quest for extreme performance optimization by tackling the 1BRC (1 Billion Row Challenge). Discover how a program initially taking several minutes to complete was transformed to run in just a few seconds. Along the way, we’ll explore how to move beyond abstractions and deal directly with data at massive scale. We’ll examine practical techniques, such as using pprof for pinpointing memory allocations, garbage collection overhead, and performance bottlenecks. Learn about the subtle code changes—like one that cut memory allocations from 15+GB down to a mere 2MB—and gain insights that you can immediately apply to your own Go projects. By watching this talk, you’ll: Develop “mechanical sympathy” when handling large-scale data. Understand how pprof can identify the hidden bottlenecks lurking in your code. Learn to write more efficient and performant Go programs in your everyday development work.

December 1, 2024 | 24 min

Monitoring the Future: Insights from VictoriaMetrics with Aliaksandr Valialkin

Chinmay Naik

Founder, CEO @One2N

In this inaugural episode, Chinmay Naik welcomes Aliaksandr Valialkin, Co-founder and CTO of VictoriaMetrics, to discuss the latest trends and innovations in monitoring and observability. They explore the challenges of scaling monitoring systems, the role of open-source solutions, and the future of monitoring in the cloud-native era.

June 28, 2024 | 93 min

Monitoring the Future: Insights from VictoriaMetrics with Aliaksandr Valialkin

Chinmay Naik

Founder, CEO @One2N

In this inaugural episode, Chinmay Naik welcomes Aliaksandr Valialkin, Co-founder and CTO of VictoriaMetrics, to discuss the latest trends and innovations in monitoring and observability. They explore the challenges of scaling monitoring systems, the role of open-source solutions, and the future of monitoring in the cloud-native era.

June 28, 2024 | 93 min

Monitoring the Future: Insights from VictoriaMetrics with Aliaksandr Valialkin

Chinmay Naik

Founder, CEO @One2N

In this inaugural episode, Chinmay Naik welcomes Aliaksandr Valialkin, Co-founder and CTO of VictoriaMetrics, to discuss the latest trends and innovations in monitoring and observability. They explore the challenges of scaling monitoring systems, the role of open-source solutions, and the future of monitoring in the cloud-native era.

June 28, 2024 | 93 min

Monitoring the Future: Insights from VictoriaMetrics with Aliaksandr Valialkin

Chinmay Naik

Founder, CEO @One2N

In this inaugural episode, Chinmay Naik welcomes Aliaksandr Valialkin, Co-founder and CTO of VictoriaMetrics, to discuss the latest trends and innovations in monitoring and observability. They explore the challenges of scaling monitoring systems, the role of open-source solutions, and the future of monitoring in the cloud-native era.

June 28, 2024 | 93 min

Monitoring the Future: Insights from VictoriaMetrics with Aliaksandr Valialkin

Chinmay Naik

Founder, CEO @One2N

In this inaugural episode, Chinmay Naik welcomes Aliaksandr Valialkin, Co-founder and CTO of VictoriaMetrics, to discuss the latest trends and innovations in monitoring and observability. They explore the challenges of scaling monitoring systems, the role of open-source solutions, and the future of monitoring in the cloud-native era.

June 28, 2024 | 93 min

Just enough queuing theory to be dangerous

Chinmay Naik

Founder, CEO @One2N

Queues are everywhere around us - both in real life and in software systems. In this talk, you'll learn just enough queuing theory to make you better at building asynchronous and event-driven systems. You’ll understand how to monitor and scale asynchronous systems and how to do capacity planning. Although the title of the talk has the word “theory” in it, you'll understand the practical view and learn to use the principles of queueing theory in practice to solve a real-world use case at the scale of millions of requests per day.

December 4, 2023 | 70 min

Just enough queuing theory to be dangerous

Chinmay Naik

Founder, CEO @One2N

Queues are everywhere around us - both in real life and in software systems. In this talk, you'll learn just enough queuing theory to make you better at building asynchronous and event-driven systems. You’ll understand how to monitor and scale asynchronous systems and how to do capacity planning. Although the title of the talk has the word “theory” in it, you'll understand the practical view and learn to use the principles of queueing theory in practice to solve a real-world use case at the scale of millions of requests per day.

December 4, 2023 | 70 min

Just enough queuing theory to be dangerous

Chinmay Naik

Founder, CEO @One2N

Queues are everywhere around us - both in real life and in software systems. In this talk, you'll learn just enough queuing theory to make you better at building asynchronous and event-driven systems. You’ll understand how to monitor and scale asynchronous systems and how to do capacity planning. Although the title of the talk has the word “theory” in it, you'll understand the practical view and learn to use the principles of queueing theory in practice to solve a real-world use case at the scale of millions of requests per day.

December 4, 2023 | 70 min

Just enough queuing theory to be dangerous

Chinmay Naik

Founder, CEO @One2N

Queues are everywhere around us - both in real life and in software systems. In this talk, you'll learn just enough queuing theory to make you better at building asynchronous and event-driven systems. You’ll understand how to monitor and scale asynchronous systems and how to do capacity planning. Although the title of the talk has the word “theory” in it, you'll understand the practical view and learn to use the principles of queueing theory in practice to solve a real-world use case at the scale of millions of requests per day.

December 4, 2023 | 70 min

Just enough queuing theory to be dangerous

Chinmay Naik

Founder, CEO @One2N

Queues are everywhere around us - both in real life and in software systems. In this talk, you'll learn just enough queuing theory to make you better at building asynchronous and event-driven systems. You’ll understand how to monitor and scale asynchronous systems and how to do capacity planning. Although the title of the talk has the word “theory” in it, you'll understand the practical view and learn to use the principles of queueing theory in practice to solve a real-world use case at the scale of millions of requests per day.

December 4, 2023 | 70 min

‹ Cost Optimization

Data Migration ›