Context.
Our client operates an AI-powered fleet management platform used by commercial vehicle operators across India. Their IoT devices, installed at the OEM level in trucks and buses, continuously stream telemetry data, including vehicle health, speed, fuel, and engine diagnostics, back to their backend systems. Fleet owners rely on this data to predict maintenance, prevent breakdowns, and optimize operational efficiency.
With a rapidly growing customer base and millions of data points flowing in daily, their existing architecture was struggling to keep up. One2N was brought in to rearchitect their data engineering infrastructure from the ground up.
Problem Statement.
Outcome/Impact.
Solution.
One2N redesigned the data engineering architecture across several parallel workstreams.
1. Decoupling ingestion from processing
We introduced RabbitMQ as a persistent buffer between MQTT ingestion and backend processing. MQTT servers now only accept packets and acknowledge immediately, while separate background workers handle processing asynchronously through RabbitMQ and Kafka. This eliminated backpressure and allowed ingestion throughput to scale independently of processing load.
2. Stateless MQTT connection management
We moved TLS and connection termination from backend servers to an NGINX or HAProxy layer at the L4 proxy. Backend MQTT servers became stateless, so restarts no longer broke device connections. This eliminated the data loss that had been causing revenue impacting downtime.
3. Migration from MongoDB to TimescaleDB
We migrated time-series IoT data from MongoDB to TimescaleDB with hypertables, which enabled:
Delta plus LZ compression, reducing data volume from 30GB per day to 3GB per day, a 90 percent reduction.
Columnar storage that is optimized for IoT data patterns.
Separate read and write optimized storage paths.
Removal of the 16TB ceiling, which made the platform capable of scaling to millions of vehicles.
4. Data lifecycle management
We implemented hot and cold data tiering with S3 as the backend for archival storage. Old data moves to S3 but remains queryable. Proper retention policies were set up, which significantly reduced the primary instance disk footprint.
5. Dedicated ML data mart
We separated the analytics data store from the transactional store, which gave the ML team direct and independent access to query data in any format, at any time, without depending on nightly exports or other teams.
6. Infrastructure observability and cost attribution
We set up monitoring and observability with proper service tagging and boundary definitions, which gave the team clear visibility into infrastructure cost per vehicle for the first time.
Additional work included CI/CD setup, Kubernetes cluster upgrades, and auto scaling configuration for databases.
