Services

Resources

Company

Our Work

Blog

#Data-Engineering

Back to Tags

Blogs

September 19, 2024 | 10 min read

ClickHouse Schema Optimization for Sparse Data Sets

Amit Sadafule

Lead Software Engineer @One2N

Optimize ClickHouse schema design for efficiently storing and querying sparse data sets. Explore strategies like sparse matrix, separate tables, and datatype-specific columns to improve compression, query performance, and reduce RAM usage.

September 19, 2024 | 10 min read

ClickHouse Schema Optimization for Sparse Data Sets

Amit Sadafule

Lead Software Engineer @One2N

Optimize ClickHouse schema design for efficiently storing and querying sparse data sets. Explore strategies like sparse matrix, separate tables, and datatype-specific columns to improve compression, query performance, and reduce RAM usage.

September 19, 2024 | 10 min read

ClickHouse Schema Optimization for Sparse Data Sets

Amit Sadafule

Lead Software Engineer @One2N

Optimize ClickHouse schema design for efficiently storing and querying sparse data sets. Explore strategies like sparse matrix, separate tables, and datatype-specific columns to improve compression, query performance, and reduce RAM usage.

September 19, 2024 | 10 min read

ClickHouse Schema Optimization for Sparse Data Sets

Amit Sadafule

Lead Software Engineer @One2N

Optimize ClickHouse schema design for efficiently storing and querying sparse data sets. Explore strategies like sparse matrix, separate tables, and datatype-specific columns to improve compression, query performance, and reduce RAM usage.

September 19, 2024 | 10 min read

ClickHouse Schema Optimization for Sparse Data Sets

Amit Sadafule

Lead Software Engineer @One2N

Optimize ClickHouse schema design for efficiently storing and querying sparse data sets. Explore strategies like sparse matrix, separate tables, and datatype-specific columns to improve compression, query performance, and reduce RAM usage.

April 25, 2024 | 5 min read

Database Schema Migrations: A Practical Guide for Mastering Change

Spandan Ghosh

Content @One2N

Database schema changes are inevitable in software development. This guide teaches you how to automate schema migrations, reducing deployment headaches and increasing agility.

April 25, 2024 | 5 min read

Database Schema Migrations: A Practical Guide for Mastering Change

Spandan Ghosh

Content @One2N

Database schema changes are inevitable in software development. This guide teaches you how to automate schema migrations, reducing deployment headaches and increasing agility.

April 25, 2024 | 5 min read

Database Schema Migrations: A Practical Guide for Mastering Change

Spandan Ghosh

Content @One2N

Database schema changes are inevitable in software development. This guide teaches you how to automate schema migrations, reducing deployment headaches and increasing agility.

April 25, 2024 | 5 min read

Database Schema Migrations: A Practical Guide for Mastering Change

Spandan Ghosh

Content @One2N

Database schema changes are inevitable in software development. This guide teaches you how to automate schema migrations, reducing deployment headaches and increasing agility.

April 25, 2024 | 5 min read

Database Schema Migrations: A Practical Guide for Mastering Change

Spandan Ghosh

Content @One2N

Database schema changes are inevitable in software development. This guide teaches you how to automate schema migrations, reducing deployment headaches and increasing agility.

July 14, 2022 | 8 min read

Implementing a Real-Time Analytics Engine Using Big Data Principles

Jaideep Khandelwal

Founder, CTO @One2N

Learn how to implement a real-time analytics engine using big data principles. The challenges, approach, and tools used to build a scalable, high-performance data analytics system. Improve your data engineering skills with this in-depth technical blog.

July 14, 2022 | 8 min read

Implementing a Real-Time Analytics Engine Using Big Data Principles

Jaideep Khandelwal

Founder, CTO @One2N

Learn how to implement a real-time analytics engine using big data principles. The challenges, approach, and tools used to build a scalable, high-performance data analytics system. Improve your data engineering skills with this in-depth technical blog.

July 14, 2022 | 8 min read

Implementing a Real-Time Analytics Engine Using Big Data Principles

Jaideep Khandelwal

Founder, CTO @One2N

Learn how to implement a real-time analytics engine using big data principles. The challenges, approach, and tools used to build a scalable, high-performance data analytics system. Improve your data engineering skills with this in-depth technical blog.

July 14, 2022 | 8 min read

Implementing a Real-Time Analytics Engine Using Big Data Principles

Jaideep Khandelwal

Founder, CTO @One2N

Learn how to implement a real-time analytics engine using big data principles. The challenges, approach, and tools used to build a scalable, high-performance data analytics system. Improve your data engineering skills with this in-depth technical blog.

July 14, 2022 | 8 min read

Implementing a Real-Time Analytics Engine Using Big Data Principles

Jaideep Khandelwal

Founder, CTO @One2N

Learn how to implement a real-time analytics engine using big data principles. The challenges, approach, and tools used to build a scalable, high-performance data analytics system. Improve your data engineering skills with this in-depth technical blog.

Case Studies

Zero downtime MySQL schema migrations for 400M row table

Problem Statement. DB schema migrations on large tables (400+ million rows or 150+ GB in size for a single table) caused replication lag and impacted latencies. Adding indexes on large tables also resulted in replication lag and degraded query performance. Developers had to wait months to roll out their features which needed schema changes.

400 M

Biggest table migrated

7

migrations per month

0

Downtime due to schema migrations

Learn More

Zero downtime MySQL schema migrations for 400M row table

Problem Statement. DB schema migrations on large tables (400+ million rows or 150+ GB in size for a single table) caused replication lag and impacted latencies. Adding indexes on large tables also resulted in replication lag and degraded query performance. Developers had to wait months to roll out their features which needed schema changes.

400 M

Biggest table migrated

7

migrations per month

0

Downtime due to schema migrations

Learn More

Zero downtime MySQL schema migrations for 400M row table

Problem Statement. DB schema migrations on large tables (400+ million rows or 150+ GB in size for a single table) caused replication lag and impacted latencies. Adding indexes on large tables also resulted in replication lag and degraded query performance. Developers had to wait months to roll out their features which needed schema changes.

400 M

Biggest table migrated

7

migrations per month

0

Downtime due to schema migrations

Learn More

Zero downtime MySQL schema migrations for 400M row table

Problem Statement. DB schema migrations on large tables (400+ million rows or 150+ GB in size for a single table) caused replication lag and impacted latencies. Adding indexes on large tables also resulted in replication lag and degraded query performance. Developers had to wait months to roll out their features which needed schema changes.

400 M

Biggest table migrated

7

migrations per month

0

Downtime due to schema migrations

Learn More

Zero downtime MySQL schema migrations for 400M row table

Problem Statement. DB schema migrations on large tables (400+ million rows or 150+ GB in size for a single table) caused replication lag and impacted latencies. Adding indexes on large tables also resulted in replication lag and degraded query performance. Developers had to wait months to roll out their features which needed schema changes.

400 M

Biggest table migrated

7

migrations per month

0

Downtime due to schema migrations

Learn More

Migrating 1.3TB of Primary Data from Self-Hosted MySQL to GCP CloudSQL

Problem Statement. Build an automated backup and recovery process for the SIEM data Provide administrative controls for customizing the backup and restore options Implement tenant-wise customizations for data retention, data locality, and other parameters

Learn More

Migrating 1.3TB of Primary Data from Self-Hosted MySQL to GCP CloudSQL

Problem Statement. Build an automated backup and recovery process for the SIEM data Provide administrative controls for customizing the backup and restore options Implement tenant-wise customizations for data retention, data locality, and other parameters

Learn More

Migrating 1.3TB of Primary Data from Self-Hosted MySQL to GCP CloudSQL

Problem Statement. Build an automated backup and recovery process for the SIEM data Provide administrative controls for customizing the backup and restore options Implement tenant-wise customizations for data retention, data locality, and other parameters

Learn More

Migrating 1.3TB of Primary Data from Self-Hosted MySQL to GCP CloudSQL

Problem Statement. Build an automated backup and recovery process for the SIEM data Provide administrative controls for customizing the backup and restore options Implement tenant-wise customizations for data retention, data locality, and other parameters

Learn More

Backup and recovery solution for SIEM data at Terabyte scale

Problem Statement. Build an automated backup and recovery process for the SIEM data Provide administrative controls for customizing the backup and restore options Implement tenant-wise customizations for data retention, data locality, and other parameters

1.5TB

Dataset handeled daily

100+

tenents using backup restore solution

$200

Running cost per month

Learn More

Backup and recovery solution for SIEM data at Terabyte scale

Problem Statement. Build an automated backup and recovery process for the SIEM data Provide administrative controls for customizing the backup and restore options Implement tenant-wise customizations for data retention, data locality, and other parameters

1.5TB

Dataset handeled daily

100+

tenents using backup restore solution

$200

Running cost per month

Learn More

Backup and recovery solution for SIEM data at Terabyte scale

Problem Statement. Build an automated backup and recovery process for the SIEM data Provide administrative controls for customizing the backup and restore options Implement tenant-wise customizations for data retention, data locality, and other parameters

1.5TB

Dataset handeled daily

100+

tenents using backup restore solution

$200

Running cost per month

Learn More

Backup and recovery solution for SIEM data at Terabyte scale

Problem Statement. Build an automated backup and recovery process for the SIEM data Provide administrative controls for customizing the backup and restore options Implement tenant-wise customizations for data retention, data locality, and other parameters

1.5TB

Dataset handeled daily

100+

tenents using backup restore solution

$200

Running cost per month

Learn More

Backup and recovery solution for SIEM data at Terabyte scale

Problem Statement. Build an automated backup and recovery process for the SIEM data Provide administrative controls for customizing the backup and restore options Implement tenant-wise customizations for data retention, data locality, and other parameters

1.5TB

Dataset handeled daily

100+

tenents using backup restore solution

$200

Running cost per month

Learn More

Talks

March 1, 2025 | 85 min

What does It take to build a Software Defined Data Center at scale

Srivatsa RV

SRE @One2N

Join Srivatsa RV, SRE at One2N, as he breaks down the complexities of scaling a Software-Defined Data Center. Learn how modern infrastructure uses virtualization, hyperconvergence, and distributed systems to deliver resilient, API-driven solutions. 🔍 Key Topics Covered - Virtualization & Hypervisors: Type-1 vs. Type-2, KVM, and Nutanix AHV - Hyperconverged Infrastructure (HCI): Scalability, fault tolerance, and real-world deployments (e.g., 15,000+ VMs across 800 servers) - Failure Modes: Disk, node, and cluster-level resilience strategies Cool Tech Demos: - VM Live Migrations: Mitigating hardware failures without downtime - Data Center-Level Failovers: Ensuring business continuity - Capacity Planning: Metrics-driven approaches for CPU, memory, and storage

March 1, 2025 | 85 min

What does It take to build a Software Defined Data Center at scale

Srivatsa RV

SRE @One2N

Join Srivatsa RV, SRE at One2N, as he breaks down the complexities of scaling a Software-Defined Data Center. Learn how modern infrastructure uses virtualization, hyperconvergence, and distributed systems to deliver resilient, API-driven solutions. 🔍 Key Topics Covered - Virtualization & Hypervisors: Type-1 vs. Type-2, KVM, and Nutanix AHV - Hyperconverged Infrastructure (HCI): Scalability, fault tolerance, and real-world deployments (e.g., 15,000+ VMs across 800 servers) - Failure Modes: Disk, node, and cluster-level resilience strategies Cool Tech Demos: - VM Live Migrations: Mitigating hardware failures without downtime - Data Center-Level Failovers: Ensuring business continuity - Capacity Planning: Metrics-driven approaches for CPU, memory, and storage

March 1, 2025 | 85 min

What does It take to build a Software Defined Data Center at scale

Srivatsa RV

SRE @One2N

Join Srivatsa RV, SRE at One2N, as he breaks down the complexities of scaling a Software-Defined Data Center. Learn how modern infrastructure uses virtualization, hyperconvergence, and distributed systems to deliver resilient, API-driven solutions. 🔍 Key Topics Covered - Virtualization & Hypervisors: Type-1 vs. Type-2, KVM, and Nutanix AHV - Hyperconverged Infrastructure (HCI): Scalability, fault tolerance, and real-world deployments (e.g., 15,000+ VMs across 800 servers) - Failure Modes: Disk, node, and cluster-level resilience strategies Cool Tech Demos: - VM Live Migrations: Mitigating hardware failures without downtime - Data Center-Level Failovers: Ensuring business continuity - Capacity Planning: Metrics-driven approaches for CPU, memory, and storage

March 1, 2025 | 85 min

What does It take to build a Software Defined Data Center at scale

Srivatsa RV

SRE @One2N

Join Srivatsa RV, SRE at One2N, as he breaks down the complexities of scaling a Software-Defined Data Center. Learn how modern infrastructure uses virtualization, hyperconvergence, and distributed systems to deliver resilient, API-driven solutions. 🔍 Key Topics Covered - Virtualization & Hypervisors: Type-1 vs. Type-2, KVM, and Nutanix AHV - Hyperconverged Infrastructure (HCI): Scalability, fault tolerance, and real-world deployments (e.g., 15,000+ VMs across 800 servers) - Failure Modes: Disk, node, and cluster-level resilience strategies Cool Tech Demos: - VM Live Migrations: Mitigating hardware failures without downtime - Data Center-Level Failovers: Ensuring business continuity - Capacity Planning: Metrics-driven approaches for CPU, memory, and storage

March 1, 2025 | 85 min

What does It take to build a Software Defined Data Center at scale

Srivatsa RV

SRE @One2N

Join Srivatsa RV, SRE at One2N, as he breaks down the complexities of scaling a Software-Defined Data Center. Learn how modern infrastructure uses virtualization, hyperconvergence, and distributed systems to deliver resilient, API-driven solutions. 🔍 Key Topics Covered - Virtualization & Hypervisors: Type-1 vs. Type-2, KVM, and Nutanix AHV - Hyperconverged Infrastructure (HCI): Scalability, fault tolerance, and real-world deployments (e.g., 15,000+ VMs across 800 servers) - Failure Modes: Disk, node, and cluster-level resilience strategies Cool Tech Demos: - VM Live Migrations: Mitigating hardware failures without downtime - Data Center-Level Failovers: Ensuring business continuity - Capacity Planning: Metrics-driven approaches for CPU, memory, and storage

December 1, 2024 | 24 min

Mohit Kumar - Billion Rows Walk Into a Go Bar

Mohit Kumar

Software Engineer @One2N

In this talk from GopherCon India 2024, we delve into the quest for extreme performance optimization by tackling the 1BRC (1 Billion Row Challenge). Discover how a program initially taking several minutes to complete was transformed to run in just a few seconds. Along the way, we’ll explore how to move beyond abstractions and deal directly with data at massive scale. We’ll examine practical techniques, such as using pprof for pinpointing memory allocations, garbage collection overhead, and performance bottlenecks. Learn about the subtle code changes—like one that cut memory allocations from 15+GB down to a mere 2MB—and gain insights that you can immediately apply to your own Go projects. By watching this talk, you’ll: Develop “mechanical sympathy” when handling large-scale data. Understand how pprof can identify the hidden bottlenecks lurking in your code. Learn to write more efficient and performant Go programs in your everyday development work.

December 1, 2024 | 24 min

Mohit Kumar - Billion Rows Walk Into a Go Bar

Mohit Kumar

Software Engineer @One2N

In this talk from GopherCon India 2024, we delve into the quest for extreme performance optimization by tackling the 1BRC (1 Billion Row Challenge). Discover how a program initially taking several minutes to complete was transformed to run in just a few seconds. Along the way, we’ll explore how to move beyond abstractions and deal directly with data at massive scale. We’ll examine practical techniques, such as using pprof for pinpointing memory allocations, garbage collection overhead, and performance bottlenecks. Learn about the subtle code changes—like one that cut memory allocations from 15+GB down to a mere 2MB—and gain insights that you can immediately apply to your own Go projects. By watching this talk, you’ll: Develop “mechanical sympathy” when handling large-scale data. Understand how pprof can identify the hidden bottlenecks lurking in your code. Learn to write more efficient and performant Go programs in your everyday development work.

December 1, 2024 | 24 min

Mohit Kumar - Billion Rows Walk Into a Go Bar

Mohit Kumar

Software Engineer @One2N

In this talk from GopherCon India 2024, we delve into the quest for extreme performance optimization by tackling the 1BRC (1 Billion Row Challenge). Discover how a program initially taking several minutes to complete was transformed to run in just a few seconds. Along the way, we’ll explore how to move beyond abstractions and deal directly with data at massive scale. We’ll examine practical techniques, such as using pprof for pinpointing memory allocations, garbage collection overhead, and performance bottlenecks. Learn about the subtle code changes—like one that cut memory allocations from 15+GB down to a mere 2MB—and gain insights that you can immediately apply to your own Go projects. By watching this talk, you’ll: Develop “mechanical sympathy” when handling large-scale data. Understand how pprof can identify the hidden bottlenecks lurking in your code. Learn to write more efficient and performant Go programs in your everyday development work.

December 1, 2024 | 24 min

Mohit Kumar - Billion Rows Walk Into a Go Bar

Mohit Kumar

Software Engineer @One2N

In this talk from GopherCon India 2024, we delve into the quest for extreme performance optimization by tackling the 1BRC (1 Billion Row Challenge). Discover how a program initially taking several minutes to complete was transformed to run in just a few seconds. Along the way, we’ll explore how to move beyond abstractions and deal directly with data at massive scale. We’ll examine practical techniques, such as using pprof for pinpointing memory allocations, garbage collection overhead, and performance bottlenecks. Learn about the subtle code changes—like one that cut memory allocations from 15+GB down to a mere 2MB—and gain insights that you can immediately apply to your own Go projects. By watching this talk, you’ll: Develop “mechanical sympathy” when handling large-scale data. Understand how pprof can identify the hidden bottlenecks lurking in your code. Learn to write more efficient and performant Go programs in your everyday development work.

December 1, 2024 | 24 min

Mohit Kumar - Billion Rows Walk Into a Go Bar

Mohit Kumar

Software Engineer @One2N

In this talk from GopherCon India 2024, we delve into the quest for extreme performance optimization by tackling the 1BRC (1 Billion Row Challenge). Discover how a program initially taking several minutes to complete was transformed to run in just a few seconds. Along the way, we’ll explore how to move beyond abstractions and deal directly with data at massive scale. We’ll examine practical techniques, such as using pprof for pinpointing memory allocations, garbage collection overhead, and performance bottlenecks. Learn about the subtle code changes—like one that cut memory allocations from 15+GB down to a mere 2MB—and gain insights that you can immediately apply to your own Go projects. By watching this talk, you’ll: Develop “mechanical sympathy” when handling large-scale data. Understand how pprof can identify the hidden bottlenecks lurking in your code. Learn to write more efficient and performant Go programs in your everyday development work.

June 28, 2024 | 93 min

Monitoring the Future: Insights from VictoriaMetrics with Aliaksandr Valialkin

Chinmay Naik

Founder, CEO @One2N

In this inaugural episode, Chinmay Naik welcomes Aliaksandr Valialkin, Co-founder and CTO of VictoriaMetrics, to discuss the latest trends and innovations in monitoring and observability. They explore the challenges of scaling monitoring systems, the role of open-source solutions, and the future of monitoring in the cloud-native era.

June 28, 2024 | 93 min

Monitoring the Future: Insights from VictoriaMetrics with Aliaksandr Valialkin

Chinmay Naik

Founder, CEO @One2N

In this inaugural episode, Chinmay Naik welcomes Aliaksandr Valialkin, Co-founder and CTO of VictoriaMetrics, to discuss the latest trends and innovations in monitoring and observability. They explore the challenges of scaling monitoring systems, the role of open-source solutions, and the future of monitoring in the cloud-native era.

June 28, 2024 | 93 min

Monitoring the Future: Insights from VictoriaMetrics with Aliaksandr Valialkin

Chinmay Naik

Founder, CEO @One2N

In this inaugural episode, Chinmay Naik welcomes Aliaksandr Valialkin, Co-founder and CTO of VictoriaMetrics, to discuss the latest trends and innovations in monitoring and observability. They explore the challenges of scaling monitoring systems, the role of open-source solutions, and the future of monitoring in the cloud-native era.

June 28, 2024 | 93 min

Monitoring the Future: Insights from VictoriaMetrics with Aliaksandr Valialkin

Chinmay Naik

Founder, CEO @One2N

In this inaugural episode, Chinmay Naik welcomes Aliaksandr Valialkin, Co-founder and CTO of VictoriaMetrics, to discuss the latest trends and innovations in monitoring and observability. They explore the challenges of scaling monitoring systems, the role of open-source solutions, and the future of monitoring in the cloud-native era.

June 28, 2024 | 93 min

Monitoring the Future: Insights from VictoriaMetrics with Aliaksandr Valialkin

Chinmay Naik

Founder, CEO @One2N

In this inaugural episode, Chinmay Naik welcomes Aliaksandr Valialkin, Co-founder and CTO of VictoriaMetrics, to discuss the latest trends and innovations in monitoring and observability. They explore the challenges of scaling monitoring systems, the role of open-source solutions, and the future of monitoring in the cloud-native era.

December 4, 2023 | 70 min

Just enough queuing theory to be dangerous

Chinmay Naik

Founder, CEO @One2N

Queues are everywhere around us - both in real life and in software systems. In this talk, you'll learn just enough queuing theory to make you better at building asynchronous and event-driven systems. You’ll understand how to monitor and scale asynchronous systems and how to do capacity planning. Although the title of the talk has the word “theory” in it, you'll understand the practical view and learn to use the principles of queueing theory in practice to solve a real-world use case at the scale of millions of requests per day.

December 4, 2023 | 70 min

Just enough queuing theory to be dangerous

Chinmay Naik

Founder, CEO @One2N

Queues are everywhere around us - both in real life and in software systems. In this talk, you'll learn just enough queuing theory to make you better at building asynchronous and event-driven systems. You’ll understand how to monitor and scale asynchronous systems and how to do capacity planning. Although the title of the talk has the word “theory” in it, you'll understand the practical view and learn to use the principles of queueing theory in practice to solve a real-world use case at the scale of millions of requests per day.

December 4, 2023 | 70 min

Just enough queuing theory to be dangerous

Chinmay Naik

Founder, CEO @One2N

Queues are everywhere around us - both in real life and in software systems. In this talk, you'll learn just enough queuing theory to make you better at building asynchronous and event-driven systems. You’ll understand how to monitor and scale asynchronous systems and how to do capacity planning. Although the title of the talk has the word “theory” in it, you'll understand the practical view and learn to use the principles of queueing theory in practice to solve a real-world use case at the scale of millions of requests per day.

December 4, 2023 | 70 min

Just enough queuing theory to be dangerous

Chinmay Naik

Founder, CEO @One2N

Queues are everywhere around us - both in real life and in software systems. In this talk, you'll learn just enough queuing theory to make you better at building asynchronous and event-driven systems. You’ll understand how to monitor and scale asynchronous systems and how to do capacity planning. Although the title of the talk has the word “theory” in it, you'll understand the practical view and learn to use the principles of queueing theory in practice to solve a real-world use case at the scale of millions of requests per day.

December 4, 2023 | 70 min

Just enough queuing theory to be dangerous

Chinmay Naik

Founder, CEO @One2N

Queues are everywhere around us - both in real life and in software systems. In this talk, you'll learn just enough queuing theory to make you better at building asynchronous and event-driven systems. You’ll understand how to monitor and scale asynchronous systems and how to do capacity planning. Although the title of the talk has the word “theory” in it, you'll understand the practical view and learn to use the principles of queueing theory in practice to solve a real-world use case at the scale of millions of requests per day.

‹ Cost-Optimization

Data-Migration ›