Services

Resources

Company

Running Debezium with MongoDB in production: four incidents and what we learnt

Running Debezium with MongoDB in production: four incidents and what we learnt

Running Debezium with MongoDB in production: four incidents and what we learnt

Running Debezium with MongoDB in production: four incidents and what we learnt

We were building a multi-tenant compliance platform that processed 20 million messages per day. The platform identified messages that violated a tenant’s compliance policies. In the previous blog, we covered how we used Debezium for change data capture (CDC), along with the transactional outbox pattern. In this blog, we cover how we made Debezium setup production-ready, the issues we faced along the way, and how we solved those.

Before I talk about the production setup, here’s a high-level architecture diagram of our setup.

Data flow: application services write outbox collections in MongoDB, Debezium outbox connectors capture the changes and publish them to Kafka event topics for consumer services; a second path carries legacy config collections through config connectors to config topics

Data flow in our compliance platform

Our architecture consisted of several Java services that stored business documents and outbox events in MongoDB. Debezium used change data capture (CDC) to publish the outbox events to Kafka, which distributed them to downstream services. We also used this pipeline to sync configuration changes from the legacy system to the newer services.

1. Increase in Debezium connector lag during peak traffic

During peak traffic, our Debezium connectors started processing MongoDB change events 40–50 minutes late. This happened around the same time every day.

The delay affected downstream services that depended on these events. If the lag increased further, we risked breaching the platform’s SLOs. Although the connectors eventually caught up on their own, we needed to understand what was causing this delay.

We detected the issue through our existing alerts and dashboards for Debezium’s JMX metrics. One of these metrics was MilliSecondsBehindSource, which measures the time between MongoDB generating a change event and Debezium processing it.

We started by checking where the delay was happening. Were records piling up inside Debezium while waiting to be published to Kafka, or was Debezium taking longer to fetch events from MongoDB?

To investigate, we checked QueueRemainingCapacity. This metric measures how much space is available in Debezium’s internal buffer queue. It stayed near its maximum, which meant the queue was mostly empty. Records weren’t piling up while waiting to be published to Kafka. The connector’s CPU and memory usage were also normal.

This suggested that the delay was happening before events entered Debezium’s queue.

We then checked the MongoDB dashboard and found that CPU usage spiked around the same time every day.

Debezium reads MongoDB changes through a change stream cursor. It repeatedly requests the next batch of events from MongoDB. When MongoDB is busy, it can take longer to return those batches, delaying the events reaching Debezium.

This also explains why MilliSecondsBehindSource was increasing even though Debezium’s internal queue was mostly empty. The metric uses the change event’s oplog timestamp, so it includes the time spent waiting for MongoDB to return the event.

Increasing Debezium’s queue size wouldn’t have helped because the records weren’t waiting inside Debezium. The delay was happening before they reached the queue.

We made two fixes.

Increasing MongoDB CPU capacity

We investigated the slow queries behind the CPU spikes and found that workloads from other teams’ services on the same cluster were driving them. We identified those services and notified the teams that owned them.

The cluster also couldn’t keep up with the combined traffic, so we increased its CPU capacity.

Filtering change events earlier

Each Debezium connector opens a MongoDB change stream with an aggregation pipeline. The pipeline filters events down to the collections that the connector monitors.

In our Debezium 2.3 setup, each connector opened a change stream for the entire MongoDB deployment rather than only for the collections it monitored. The connector therefore saw changes from unrelated databases and collections before filtering them out.

For example, a connector interested only in database_a.messages also saw changes from database_b.messages and database_c.messages.

Debezium’s default aggregation pipeline first used $replaceRoot to restructure each change event and build a namespace field from the database and collection names. It then applied a regex filter to retain events from the configured collections.

This meant MongoDB had to restructure every change event, including those that the next stage would discard.

Before: the change stream sends events from every database through $replaceRoot, then $match keeps only database_a. After: a $match on ns.db and ns.coll runs first (cursor.pipeline.order=user_first), so $replaceRoot only processes database_a events

Change stream pipeline before and after the fix

We changed the pipeline to filter events before restructuring them. We added a $match stage on ns.db and ns.coll so MongoDB could filter events by database and collection before running Debezium’s default pipeline. We also set cursor.pipeline.order=user_first to make our stage run first.

{
  "cursor.pipeline.order": "user_first",
  "cursor.pipeline": "[{\"$match\":{\"ns.db\":\"database_a\",\"ns.coll\":\"messages\"}}]"

connector config pipeline properties after aggregation pipeline fix

This reduced the number of events processed by the rest of the aggregation pipeline. It didn’t remove the competing database load, so we still needed the additional CPU capacity. After both changes, MilliSecondsBehindSource dropped from 40–50 minutes during peak periods to under 10 seconds.

2. Debezium failure in processing a valid MongoDB document

One of our Debezium connectors stopped processing events because it couldn’t handle a valid MongoDB document containing an empty array nested inside another array. The connector task failed and stopped publishing new events to Kafka.

This blocked downstream services that depended on these events. The longer the connector remained stopped, the more events it would need to catch up on after restarting.

We checked the connector logs and found this error:

Only Struct objects supported for [Read Outbox Event], found

We started by inspecting the document that had caused the failure. It was valid BSON, and its structure didn’t look unusual. On closer inspection, we found an empty array (tags) inside an object that was part of another array (items).

{
  "payload": {
    "items": [
      {
        "name": "item-1",
        "tags"

The affected connector had transforms.outbox.collection.expand.json.payload set to true. With this setting, Debezium converts the JSON payload into a structured Kafka Connect value. To do this, it first infers a schema from the document’s fields. Downstream consumers then receive a structured value instead of a JSON string.

The problem wasn’t with empty arrays; Debezium could handle an empty array on its own. The bug occurred when the empty array was a field of an object inside another array, as tags is in this example.

While inferring the schema, Debezium identified the object inside items but included only the name field in its schema. It missed tags, even though the field was present in the document.

When Debezium tried to convert the document into a Kafka Connect Struct, the schema and the actual document structure no longer matched. The conversion failed.

We traced this to how Debezium 2.3 inferred schemas in two passes.

In the first pass, Debezium skipped empty arrays and empty documents because they didn’t contain enough information to determine their types. In the second pass, it revisited these fields only if the object had no other fields.

In our case, the object already had a name field. Debezium therefore didn’t revisit tags, leaving it out of the inferred schema.

How we fixed it

We considered three options:

  1. Skip failing events. We could set errors.tolerance=all to allow the connector to continue processing other events. However, Kafka Connect supports dead-letter queues (DLQs) only for sink connectors, not source connectors. Without a DLQ to capture skipped events, we risked silently losing compliance-related data. We ruled this out.

  2. Upgrade Debezium. Debezium 2.7 and later fixed the problem, but those versions were incompatible with the MongoDB 5.0 version we were running. We ruled this out as well.

  3. Disable JSON payload expansion. We could set transforms.outbox.collection.expand.json.payload=false to bypass schema inference. Debezium would then publish the payload as a JSON string instead of a structured Kafka Connect value. This meant downstream consumers would need to change how they parsed the payload.

We chose the third option and updated the connector configuration:

"transforms.outbox.collection.expand.json.payload" : false

This bypassed schema inference and allowed the connector to process the document without failing.

However, disabling payload expansion introduced another problem, which we discovered when the connector processed a large document. I have covered this issue below.

3. Debezium connector failure due to Kafka message size limit

In the previous issue, we disabled JSON payload expansion to bypass schema inference for the documents. This allowed the connector to process documents containing nested empty arrays. But this change introduced another problem.

One of our connectors failed when processing a 2.5 MB MongoDB document. After Debezium converted the document into a Kafka record, its size increased to 5.3 MB. This exceeded the 4 MB request-size limit configured for Kafka Connect’s producer.

As a result, the connector task stopped publishing events to Kafka, blocking downstream processing until we increased the limit and restarted it.

The connector logs showed the following exception:

org.apache.kafka.common.errors.RecordTooLargeException: The message is 5323164 bytes when serialized,
which is larger than 4194304

We wanted to understand why a 2.5 MB document had grown large enough to exceed Kafka’s 4 MB limit.

To reproduce the issue, we serialized the same logical payload as compact JSON and published it with a plain Kafka producer; it remained within the existing 4 MB limit. But when Debezium processed the same payload through its outbox transform, the serialized record grew to 5.3 MB.

In the previous issue, we used expand.json.payload=false as the connector configuration. This allowed Debezium to bypass the issue with inferring the schema for documents, but it also changed how Debezium serialized the payload.

We traced the serialization code and found that Debezium used MongoDB’s JSON writer with pretty-printing enabled:

JsonWriterSettings jsonWriterSettings = JsonWriterSettings.builder()
        .outputMode(JsonMode.EXTENDED)
        .indent(true)
        .newLineCharacters("\n")
        .build();

Source: Debezium MongoEventRouter.java

Two settings increased the payload size:

  • indent(true) added whitespace and line breaks throughout the document.

  • JsonMode.EXTENDED preserved BSON-specific types in the JSON representation. With this, values such as dates and 64-bit integers were represented using wrapper objects like {"$date": ...} and {"$numberLong": ...} . These representations take up more space than their compact JSON equivalents.

To confirm this, we serialized the document locally using the same settings and got 5,323,164 bytes, the exact size reported in the connector’s error.

These writer settings were hard-coded in the Debezium 2.3 we used. There was no connector setting to disable the indentation.

Here’s how we solved this issue.

Increasing Kafka’s message size limits

Since we couldn’t change Debezium’s JSON serialization settings, we needed Kafka to accept the larger records.

We first checked which Kafka limit was rejecting the record. The exception pointed to max.request.size, which limits the size of requests sent by a Kafka producer. We increased producer.override.max.request.size in the Kafka Connect configuration to allow the producer to send larger requests.

Kafka also has broker-level (message.max.bytes) and topic-level (max.message.bytes) limits. Increasing these alone wouldn’t have fixed our issue because the producer rejected the record before sending it to Kafka. We analysed our existing data and the largest payloads we expected from upstream services. Based on that analysis, we increased the producer and topic limits with some headroom.

After increasing the limits, the connector could publish the record again. But the change only allowed Kafka to accept larger records. It didn’t address the increase in size caused by Debezium’s JSON serialization.

For large payloads, we decided to store the document separately (in S3) and publish only a reference through Kafka. Consumers can then retrieve the document when they need it, rather than having the entire document pass through the event pipeline.

4. Kafka Connect converted timestamps to dates at midnight

Another issue we ran into was related to timestamps. Some messages published to Kafka had incomplete timestamps. Downstream consumers expected a full date and time, but some messages contained only the date. This caused those messages to fail during downstream processing.

For example, consumers expected: 2026-01-01T00:00:00.000Z But received: 2026-01-01 .

This issue affected only messages with timestamps at exactly midnight UTC. Messages with other timestamp values continued to work normally. Unlike the previous issues, this didn’t stop the connector or block the entire pipeline, but the affected messages failed during downstream processing.

We first checked the original MongoDB documents and confirmed that they contained complete timestamps. We also verified that the consumers weren’t modifying the headers. This pointed us to how Debezium was creating the Kafka headers.

Our outbox connectors used transforms.outbox.collection.fields.additional.placement to copy timestamp fields from the outbox document into Kafka headers. While creating these headers, Kafka Connect used its Values utility to convert timestamps into strings. We traced the issue to the dateFormatFor method, which selected the date format using this condition:

public static DateFormat dateFormatFor(java.util.Date value) {
    if (value.getTime() < MILLIS_PER_DAY) {
        return new SimpleDateFormat(ISO_8601_TIME_FORMAT_PATTERN);
    }
    if (value.getTime() % MILLIS_PER_DAY == 0) {
        return new SimpleDateFormat(ISO_8601_DATE_FORMAT_PATTERN);
    }
    return new SimpleDateFormat(ISO_8601_TIMESTAMP_FORMAT_PATTERN);
}

Source: Apache Kafka

The method checks whether the timestamp’s epoch value is divisible by the number of milliseconds in a day.

A day contains 24 * 60 * 60 * 1000 = 86,400,000 milliseconds. The epoch value for a timestamp at exactly midnight UTC is therefore an exact multiple of that number. For 2026-01-01T00:00:00Z:

1767225600000 % 86400000 == 0

Because the condition evaluated to true, Kafka Connect treated the value as a date and formatted it as 2026-01-01. It inferred the format from the numeric value instead of using the field’s logical type to distinguish a Timestamp from a Date.

We reported the bug on the Kafka users mailing list. A Kafka maintainer confirmed it, opened KAFKA-20752, and raised a pull request. At the time of writing, the pull request is still under review.

How we worked around it

As a workaround, we stored the timestamp fields in our outbox documents as ISO-8601 strings instead of BSON dates. This meant Kafka Connect received the timestamp as a string and no longer tried to determine whether it should be formatted as a Date or a Timestamp. The workaround fixed the issue for new events. However, downstream consumers still needed to handle date-only values from events published before we applied the change.

Conclusion

Across these incidents, the symptom often appeared in Debezium even when the failure started elsewhere. Database load delayed events before they reached the connector. Debezium transforms and Kafka Connect converters changed how the data was represented before it reached Kafka. Looking only at the connector metrics would have led us towards the wrong fixes.

The second and third incidents also showed how a workaround can affect a later stage of the pipeline. Emitting the payload as a string bypassed the schema-inference bug, but Debezium’s string serialization more than doubled the size of one record. We now test workarounds against the largest and most unusual data the system can receive, not only the input that originally triggered the problem.

We were building a multi-tenant compliance platform that processed 20 million messages per day. The platform identified messages that violated a tenant’s compliance policies. In the previous blog, we covered how we used Debezium for change data capture (CDC), along with the transactional outbox pattern. In this blog, we cover how we made Debezium setup production-ready, the issues we faced along the way, and how we solved those.

Before I talk about the production setup, here’s a high-level architecture diagram of our setup.

Data flow: application services write outbox collections in MongoDB, Debezium outbox connectors capture the changes and publish them to Kafka event topics for consumer services; a second path carries legacy config collections through config connectors to config topics

Data flow in our compliance platform

Our architecture consisted of several Java services that stored business documents and outbox events in MongoDB. Debezium used change data capture (CDC) to publish the outbox events to Kafka, which distributed them to downstream services. We also used this pipeline to sync configuration changes from the legacy system to the newer services.

1. Increase in Debezium connector lag during peak traffic

During peak traffic, our Debezium connectors started processing MongoDB change events 40–50 minutes late. This happened around the same time every day.

The delay affected downstream services that depended on these events. If the lag increased further, we risked breaching the platform’s SLOs. Although the connectors eventually caught up on their own, we needed to understand what was causing this delay.

We detected the issue through our existing alerts and dashboards for Debezium’s JMX metrics. One of these metrics was MilliSecondsBehindSource, which measures the time between MongoDB generating a change event and Debezium processing it.

We started by checking where the delay was happening. Were records piling up inside Debezium while waiting to be published to Kafka, or was Debezium taking longer to fetch events from MongoDB?

To investigate, we checked QueueRemainingCapacity. This metric measures how much space is available in Debezium’s internal buffer queue. It stayed near its maximum, which meant the queue was mostly empty. Records weren’t piling up while waiting to be published to Kafka. The connector’s CPU and memory usage were also normal.

This suggested that the delay was happening before events entered Debezium’s queue.

We then checked the MongoDB dashboard and found that CPU usage spiked around the same time every day.

Debezium reads MongoDB changes through a change stream cursor. It repeatedly requests the next batch of events from MongoDB. When MongoDB is busy, it can take longer to return those batches, delaying the events reaching Debezium.

This also explains why MilliSecondsBehindSource was increasing even though Debezium’s internal queue was mostly empty. The metric uses the change event’s oplog timestamp, so it includes the time spent waiting for MongoDB to return the event.

Increasing Debezium’s queue size wouldn’t have helped because the records weren’t waiting inside Debezium. The delay was happening before they reached the queue.

We made two fixes.

Increasing MongoDB CPU capacity

We investigated the slow queries behind the CPU spikes and found that workloads from other teams’ services on the same cluster were driving them. We identified those services and notified the teams that owned them.

The cluster also couldn’t keep up with the combined traffic, so we increased its CPU capacity.

Filtering change events earlier

Each Debezium connector opens a MongoDB change stream with an aggregation pipeline. The pipeline filters events down to the collections that the connector monitors.

In our Debezium 2.3 setup, each connector opened a change stream for the entire MongoDB deployment rather than only for the collections it monitored. The connector therefore saw changes from unrelated databases and collections before filtering them out.

For example, a connector interested only in database_a.messages also saw changes from database_b.messages and database_c.messages.

Debezium’s default aggregation pipeline first used $replaceRoot to restructure each change event and build a namespace field from the database and collection names. It then applied a regex filter to retain events from the configured collections.

This meant MongoDB had to restructure every change event, including those that the next stage would discard.

Before: the change stream sends events from every database through $replaceRoot, then $match keeps only database_a. After: a $match on ns.db and ns.coll runs first (cursor.pipeline.order=user_first), so $replaceRoot only processes database_a events

Change stream pipeline before and after the fix

We changed the pipeline to filter events before restructuring them. We added a $match stage on ns.db and ns.coll so MongoDB could filter events by database and collection before running Debezium’s default pipeline. We also set cursor.pipeline.order=user_first to make our stage run first.

{
  "cursor.pipeline.order": "user_first",
  "cursor.pipeline": "[{\"$match\":{\"ns.db\":\"database_a\",\"ns.coll\":\"messages\"}}]"

connector config pipeline properties after aggregation pipeline fix

This reduced the number of events processed by the rest of the aggregation pipeline. It didn’t remove the competing database load, so we still needed the additional CPU capacity. After both changes, MilliSecondsBehindSource dropped from 40–50 minutes during peak periods to under 10 seconds.

2. Debezium failure in processing a valid MongoDB document

One of our Debezium connectors stopped processing events because it couldn’t handle a valid MongoDB document containing an empty array nested inside another array. The connector task failed and stopped publishing new events to Kafka.

This blocked downstream services that depended on these events. The longer the connector remained stopped, the more events it would need to catch up on after restarting.

We checked the connector logs and found this error:

Only Struct objects supported for [Read Outbox Event], found

We started by inspecting the document that had caused the failure. It was valid BSON, and its structure didn’t look unusual. On closer inspection, we found an empty array (tags) inside an object that was part of another array (items).

{
  "payload": {
    "items": [
      {
        "name": "item-1",
        "tags"

The affected connector had transforms.outbox.collection.expand.json.payload set to true. With this setting, Debezium converts the JSON payload into a structured Kafka Connect value. To do this, it first infers a schema from the document’s fields. Downstream consumers then receive a structured value instead of a JSON string.

The problem wasn’t with empty arrays; Debezium could handle an empty array on its own. The bug occurred when the empty array was a field of an object inside another array, as tags is in this example.

While inferring the schema, Debezium identified the object inside items but included only the name field in its schema. It missed tags, even though the field was present in the document.

When Debezium tried to convert the document into a Kafka Connect Struct, the schema and the actual document structure no longer matched. The conversion failed.

We traced this to how Debezium 2.3 inferred schemas in two passes.

In the first pass, Debezium skipped empty arrays and empty documents because they didn’t contain enough information to determine their types. In the second pass, it revisited these fields only if the object had no other fields.

In our case, the object already had a name field. Debezium therefore didn’t revisit tags, leaving it out of the inferred schema.

How we fixed it

We considered three options:

  1. Skip failing events. We could set errors.tolerance=all to allow the connector to continue processing other events. However, Kafka Connect supports dead-letter queues (DLQs) only for sink connectors, not source connectors. Without a DLQ to capture skipped events, we risked silently losing compliance-related data. We ruled this out.

  2. Upgrade Debezium. Debezium 2.7 and later fixed the problem, but those versions were incompatible with the MongoDB 5.0 version we were running. We ruled this out as well.

  3. Disable JSON payload expansion. We could set transforms.outbox.collection.expand.json.payload=false to bypass schema inference. Debezium would then publish the payload as a JSON string instead of a structured Kafka Connect value. This meant downstream consumers would need to change how they parsed the payload.

We chose the third option and updated the connector configuration:

"transforms.outbox.collection.expand.json.payload" : false

This bypassed schema inference and allowed the connector to process the document without failing.

However, disabling payload expansion introduced another problem, which we discovered when the connector processed a large document. I have covered this issue below.

3. Debezium connector failure due to Kafka message size limit

In the previous issue, we disabled JSON payload expansion to bypass schema inference for the documents. This allowed the connector to process documents containing nested empty arrays. But this change introduced another problem.

One of our connectors failed when processing a 2.5 MB MongoDB document. After Debezium converted the document into a Kafka record, its size increased to 5.3 MB. This exceeded the 4 MB request-size limit configured for Kafka Connect’s producer.

As a result, the connector task stopped publishing events to Kafka, blocking downstream processing until we increased the limit and restarted it.

The connector logs showed the following exception:

org.apache.kafka.common.errors.RecordTooLargeException: The message is 5323164 bytes when serialized,
which is larger than 4194304

We wanted to understand why a 2.5 MB document had grown large enough to exceed Kafka’s 4 MB limit.

To reproduce the issue, we serialized the same logical payload as compact JSON and published it with a plain Kafka producer; it remained within the existing 4 MB limit. But when Debezium processed the same payload through its outbox transform, the serialized record grew to 5.3 MB.

In the previous issue, we used expand.json.payload=false as the connector configuration. This allowed Debezium to bypass the issue with inferring the schema for documents, but it also changed how Debezium serialized the payload.

We traced the serialization code and found that Debezium used MongoDB’s JSON writer with pretty-printing enabled:

JsonWriterSettings jsonWriterSettings = JsonWriterSettings.builder()
        .outputMode(JsonMode.EXTENDED)
        .indent(true)
        .newLineCharacters("\n")
        .build();

Source: Debezium MongoEventRouter.java

Two settings increased the payload size:

  • indent(true) added whitespace and line breaks throughout the document.

  • JsonMode.EXTENDED preserved BSON-specific types in the JSON representation. With this, values such as dates and 64-bit integers were represented using wrapper objects like {"$date": ...} and {"$numberLong": ...} . These representations take up more space than their compact JSON equivalents.

To confirm this, we serialized the document locally using the same settings and got 5,323,164 bytes, the exact size reported in the connector’s error.

These writer settings were hard-coded in the Debezium 2.3 we used. There was no connector setting to disable the indentation.

Here’s how we solved this issue.

Increasing Kafka’s message size limits

Since we couldn’t change Debezium’s JSON serialization settings, we needed Kafka to accept the larger records.

We first checked which Kafka limit was rejecting the record. The exception pointed to max.request.size, which limits the size of requests sent by a Kafka producer. We increased producer.override.max.request.size in the Kafka Connect configuration to allow the producer to send larger requests.

Kafka also has broker-level (message.max.bytes) and topic-level (max.message.bytes) limits. Increasing these alone wouldn’t have fixed our issue because the producer rejected the record before sending it to Kafka. We analysed our existing data and the largest payloads we expected from upstream services. Based on that analysis, we increased the producer and topic limits with some headroom.

After increasing the limits, the connector could publish the record again. But the change only allowed Kafka to accept larger records. It didn’t address the increase in size caused by Debezium’s JSON serialization.

For large payloads, we decided to store the document separately (in S3) and publish only a reference through Kafka. Consumers can then retrieve the document when they need it, rather than having the entire document pass through the event pipeline.

4. Kafka Connect converted timestamps to dates at midnight

Another issue we ran into was related to timestamps. Some messages published to Kafka had incomplete timestamps. Downstream consumers expected a full date and time, but some messages contained only the date. This caused those messages to fail during downstream processing.

For example, consumers expected: 2026-01-01T00:00:00.000Z But received: 2026-01-01 .

This issue affected only messages with timestamps at exactly midnight UTC. Messages with other timestamp values continued to work normally. Unlike the previous issues, this didn’t stop the connector or block the entire pipeline, but the affected messages failed during downstream processing.

We first checked the original MongoDB documents and confirmed that they contained complete timestamps. We also verified that the consumers weren’t modifying the headers. This pointed us to how Debezium was creating the Kafka headers.

Our outbox connectors used transforms.outbox.collection.fields.additional.placement to copy timestamp fields from the outbox document into Kafka headers. While creating these headers, Kafka Connect used its Values utility to convert timestamps into strings. We traced the issue to the dateFormatFor method, which selected the date format using this condition:

public static DateFormat dateFormatFor(java.util.Date value) {
    if (value.getTime() < MILLIS_PER_DAY) {
        return new SimpleDateFormat(ISO_8601_TIME_FORMAT_PATTERN);
    }
    if (value.getTime() % MILLIS_PER_DAY == 0) {
        return new SimpleDateFormat(ISO_8601_DATE_FORMAT_PATTERN);
    }
    return new SimpleDateFormat(ISO_8601_TIMESTAMP_FORMAT_PATTERN);
}

Source: Apache Kafka

The method checks whether the timestamp’s epoch value is divisible by the number of milliseconds in a day.

A day contains 24 * 60 * 60 * 1000 = 86,400,000 milliseconds. The epoch value for a timestamp at exactly midnight UTC is therefore an exact multiple of that number. For 2026-01-01T00:00:00Z:

1767225600000 % 86400000 == 0

Because the condition evaluated to true, Kafka Connect treated the value as a date and formatted it as 2026-01-01. It inferred the format from the numeric value instead of using the field’s logical type to distinguish a Timestamp from a Date.

We reported the bug on the Kafka users mailing list. A Kafka maintainer confirmed it, opened KAFKA-20752, and raised a pull request. At the time of writing, the pull request is still under review.

How we worked around it

As a workaround, we stored the timestamp fields in our outbox documents as ISO-8601 strings instead of BSON dates. This meant Kafka Connect received the timestamp as a string and no longer tried to determine whether it should be formatted as a Date or a Timestamp. The workaround fixed the issue for new events. However, downstream consumers still needed to handle date-only values from events published before we applied the change.

Conclusion

Across these incidents, the symptom often appeared in Debezium even when the failure started elsewhere. Database load delayed events before they reached the connector. Debezium transforms and Kafka Connect converters changed how the data was represented before it reached Kafka. Looking only at the connector metrics would have led us towards the wrong fixes.

The second and third incidents also showed how a workaround can affect a later stage of the pipeline. Emitting the payload as a string bypassed the schema-inference bug, but Debezium’s string serialization more than doubled the size of one record. We now test workarounds against the largest and most unusual data the system can receive, not only the input that originally triggered the problem.

We were building a multi-tenant compliance platform that processed 20 million messages per day. The platform identified messages that violated a tenant’s compliance policies. In the previous blog, we covered how we used Debezium for change data capture (CDC), along with the transactional outbox pattern. In this blog, we cover how we made Debezium setup production-ready, the issues we faced along the way, and how we solved those.

Before I talk about the production setup, here’s a high-level architecture diagram of our setup.

Data flow: application services write outbox collections in MongoDB, Debezium outbox connectors capture the changes and publish them to Kafka event topics for consumer services; a second path carries legacy config collections through config connectors to config topics

Data flow in our compliance platform

Our architecture consisted of several Java services that stored business documents and outbox events in MongoDB. Debezium used change data capture (CDC) to publish the outbox events to Kafka, which distributed them to downstream services. We also used this pipeline to sync configuration changes from the legacy system to the newer services.

1. Increase in Debezium connector lag during peak traffic

During peak traffic, our Debezium connectors started processing MongoDB change events 40–50 minutes late. This happened around the same time every day.

The delay affected downstream services that depended on these events. If the lag increased further, we risked breaching the platform’s SLOs. Although the connectors eventually caught up on their own, we needed to understand what was causing this delay.

We detected the issue through our existing alerts and dashboards for Debezium’s JMX metrics. One of these metrics was MilliSecondsBehindSource, which measures the time between MongoDB generating a change event and Debezium processing it.

We started by checking where the delay was happening. Were records piling up inside Debezium while waiting to be published to Kafka, or was Debezium taking longer to fetch events from MongoDB?

To investigate, we checked QueueRemainingCapacity. This metric measures how much space is available in Debezium’s internal buffer queue. It stayed near its maximum, which meant the queue was mostly empty. Records weren’t piling up while waiting to be published to Kafka. The connector’s CPU and memory usage were also normal.

This suggested that the delay was happening before events entered Debezium’s queue.

We then checked the MongoDB dashboard and found that CPU usage spiked around the same time every day.

Debezium reads MongoDB changes through a change stream cursor. It repeatedly requests the next batch of events from MongoDB. When MongoDB is busy, it can take longer to return those batches, delaying the events reaching Debezium.

This also explains why MilliSecondsBehindSource was increasing even though Debezium’s internal queue was mostly empty. The metric uses the change event’s oplog timestamp, so it includes the time spent waiting for MongoDB to return the event.

Increasing Debezium’s queue size wouldn’t have helped because the records weren’t waiting inside Debezium. The delay was happening before they reached the queue.

We made two fixes.

Increasing MongoDB CPU capacity

We investigated the slow queries behind the CPU spikes and found that workloads from other teams’ services on the same cluster were driving them. We identified those services and notified the teams that owned them.

The cluster also couldn’t keep up with the combined traffic, so we increased its CPU capacity.

Filtering change events earlier

Each Debezium connector opens a MongoDB change stream with an aggregation pipeline. The pipeline filters events down to the collections that the connector monitors.

In our Debezium 2.3 setup, each connector opened a change stream for the entire MongoDB deployment rather than only for the collections it monitored. The connector therefore saw changes from unrelated databases and collections before filtering them out.

For example, a connector interested only in database_a.messages also saw changes from database_b.messages and database_c.messages.

Debezium’s default aggregation pipeline first used $replaceRoot to restructure each change event and build a namespace field from the database and collection names. It then applied a regex filter to retain events from the configured collections.

This meant MongoDB had to restructure every change event, including those that the next stage would discard.

Before: the change stream sends events from every database through $replaceRoot, then $match keeps only database_a. After: a $match on ns.db and ns.coll runs first (cursor.pipeline.order=user_first), so $replaceRoot only processes database_a events

Change stream pipeline before and after the fix

We changed the pipeline to filter events before restructuring them. We added a $match stage on ns.db and ns.coll so MongoDB could filter events by database and collection before running Debezium’s default pipeline. We also set cursor.pipeline.order=user_first to make our stage run first.

{
  "cursor.pipeline.order": "user_first",
  "cursor.pipeline": "[{\"$match\":{\"ns.db\":\"database_a\",\"ns.coll\":\"messages\"}}]"

connector config pipeline properties after aggregation pipeline fix

This reduced the number of events processed by the rest of the aggregation pipeline. It didn’t remove the competing database load, so we still needed the additional CPU capacity. After both changes, MilliSecondsBehindSource dropped from 40–50 minutes during peak periods to under 10 seconds.

2. Debezium failure in processing a valid MongoDB document

One of our Debezium connectors stopped processing events because it couldn’t handle a valid MongoDB document containing an empty array nested inside another array. The connector task failed and stopped publishing new events to Kafka.

This blocked downstream services that depended on these events. The longer the connector remained stopped, the more events it would need to catch up on after restarting.

We checked the connector logs and found this error:

Only Struct objects supported for [Read Outbox Event], found

We started by inspecting the document that had caused the failure. It was valid BSON, and its structure didn’t look unusual. On closer inspection, we found an empty array (tags) inside an object that was part of another array (items).

{
  "payload": {
    "items": [
      {
        "name": "item-1",
        "tags"

The affected connector had transforms.outbox.collection.expand.json.payload set to true. With this setting, Debezium converts the JSON payload into a structured Kafka Connect value. To do this, it first infers a schema from the document’s fields. Downstream consumers then receive a structured value instead of a JSON string.

The problem wasn’t with empty arrays; Debezium could handle an empty array on its own. The bug occurred when the empty array was a field of an object inside another array, as tags is in this example.

While inferring the schema, Debezium identified the object inside items but included only the name field in its schema. It missed tags, even though the field was present in the document.

When Debezium tried to convert the document into a Kafka Connect Struct, the schema and the actual document structure no longer matched. The conversion failed.

We traced this to how Debezium 2.3 inferred schemas in two passes.

In the first pass, Debezium skipped empty arrays and empty documents because they didn’t contain enough information to determine their types. In the second pass, it revisited these fields only if the object had no other fields.

In our case, the object already had a name field. Debezium therefore didn’t revisit tags, leaving it out of the inferred schema.

How we fixed it

We considered three options:

  1. Skip failing events. We could set errors.tolerance=all to allow the connector to continue processing other events. However, Kafka Connect supports dead-letter queues (DLQs) only for sink connectors, not source connectors. Without a DLQ to capture skipped events, we risked silently losing compliance-related data. We ruled this out.

  2. Upgrade Debezium. Debezium 2.7 and later fixed the problem, but those versions were incompatible with the MongoDB 5.0 version we were running. We ruled this out as well.

  3. Disable JSON payload expansion. We could set transforms.outbox.collection.expand.json.payload=false to bypass schema inference. Debezium would then publish the payload as a JSON string instead of a structured Kafka Connect value. This meant downstream consumers would need to change how they parsed the payload.

We chose the third option and updated the connector configuration:

"transforms.outbox.collection.expand.json.payload" : false

This bypassed schema inference and allowed the connector to process the document without failing.

However, disabling payload expansion introduced another problem, which we discovered when the connector processed a large document. I have covered this issue below.

3. Debezium connector failure due to Kafka message size limit

In the previous issue, we disabled JSON payload expansion to bypass schema inference for the documents. This allowed the connector to process documents containing nested empty arrays. But this change introduced another problem.

One of our connectors failed when processing a 2.5 MB MongoDB document. After Debezium converted the document into a Kafka record, its size increased to 5.3 MB. This exceeded the 4 MB request-size limit configured for Kafka Connect’s producer.

As a result, the connector task stopped publishing events to Kafka, blocking downstream processing until we increased the limit and restarted it.

The connector logs showed the following exception:

org.apache.kafka.common.errors.RecordTooLargeException: The message is 5323164 bytes when serialized,
which is larger than 4194304

We wanted to understand why a 2.5 MB document had grown large enough to exceed Kafka’s 4 MB limit.

To reproduce the issue, we serialized the same logical payload as compact JSON and published it with a plain Kafka producer; it remained within the existing 4 MB limit. But when Debezium processed the same payload through its outbox transform, the serialized record grew to 5.3 MB.

In the previous issue, we used expand.json.payload=false as the connector configuration. This allowed Debezium to bypass the issue with inferring the schema for documents, but it also changed how Debezium serialized the payload.

We traced the serialization code and found that Debezium used MongoDB’s JSON writer with pretty-printing enabled:

JsonWriterSettings jsonWriterSettings = JsonWriterSettings.builder()
        .outputMode(JsonMode.EXTENDED)
        .indent(true)
        .newLineCharacters("\n")
        .build();

Source: Debezium MongoEventRouter.java

Two settings increased the payload size:

  • indent(true) added whitespace and line breaks throughout the document.

  • JsonMode.EXTENDED preserved BSON-specific types in the JSON representation. With this, values such as dates and 64-bit integers were represented using wrapper objects like {"$date": ...} and {"$numberLong": ...} . These representations take up more space than their compact JSON equivalents.

To confirm this, we serialized the document locally using the same settings and got 5,323,164 bytes, the exact size reported in the connector’s error.

These writer settings were hard-coded in the Debezium 2.3 we used. There was no connector setting to disable the indentation.

Here’s how we solved this issue.

Increasing Kafka’s message size limits

Since we couldn’t change Debezium’s JSON serialization settings, we needed Kafka to accept the larger records.

We first checked which Kafka limit was rejecting the record. The exception pointed to max.request.size, which limits the size of requests sent by a Kafka producer. We increased producer.override.max.request.size in the Kafka Connect configuration to allow the producer to send larger requests.

Kafka also has broker-level (message.max.bytes) and topic-level (max.message.bytes) limits. Increasing these alone wouldn’t have fixed our issue because the producer rejected the record before sending it to Kafka. We analysed our existing data and the largest payloads we expected from upstream services. Based on that analysis, we increased the producer and topic limits with some headroom.

After increasing the limits, the connector could publish the record again. But the change only allowed Kafka to accept larger records. It didn’t address the increase in size caused by Debezium’s JSON serialization.

For large payloads, we decided to store the document separately (in S3) and publish only a reference through Kafka. Consumers can then retrieve the document when they need it, rather than having the entire document pass through the event pipeline.

4. Kafka Connect converted timestamps to dates at midnight

Another issue we ran into was related to timestamps. Some messages published to Kafka had incomplete timestamps. Downstream consumers expected a full date and time, but some messages contained only the date. This caused those messages to fail during downstream processing.

For example, consumers expected: 2026-01-01T00:00:00.000Z But received: 2026-01-01 .

This issue affected only messages with timestamps at exactly midnight UTC. Messages with other timestamp values continued to work normally. Unlike the previous issues, this didn’t stop the connector or block the entire pipeline, but the affected messages failed during downstream processing.

We first checked the original MongoDB documents and confirmed that they contained complete timestamps. We also verified that the consumers weren’t modifying the headers. This pointed us to how Debezium was creating the Kafka headers.

Our outbox connectors used transforms.outbox.collection.fields.additional.placement to copy timestamp fields from the outbox document into Kafka headers. While creating these headers, Kafka Connect used its Values utility to convert timestamps into strings. We traced the issue to the dateFormatFor method, which selected the date format using this condition:

public static DateFormat dateFormatFor(java.util.Date value) {
    if (value.getTime() < MILLIS_PER_DAY) {
        return new SimpleDateFormat(ISO_8601_TIME_FORMAT_PATTERN);
    }
    if (value.getTime() % MILLIS_PER_DAY == 0) {
        return new SimpleDateFormat(ISO_8601_DATE_FORMAT_PATTERN);
    }
    return new SimpleDateFormat(ISO_8601_TIMESTAMP_FORMAT_PATTERN);
}

Source: Apache Kafka

The method checks whether the timestamp’s epoch value is divisible by the number of milliseconds in a day.

A day contains 24 * 60 * 60 * 1000 = 86,400,000 milliseconds. The epoch value for a timestamp at exactly midnight UTC is therefore an exact multiple of that number. For 2026-01-01T00:00:00Z:

1767225600000 % 86400000 == 0

Because the condition evaluated to true, Kafka Connect treated the value as a date and formatted it as 2026-01-01. It inferred the format from the numeric value instead of using the field’s logical type to distinguish a Timestamp from a Date.

We reported the bug on the Kafka users mailing list. A Kafka maintainer confirmed it, opened KAFKA-20752, and raised a pull request. At the time of writing, the pull request is still under review.

How we worked around it

As a workaround, we stored the timestamp fields in our outbox documents as ISO-8601 strings instead of BSON dates. This meant Kafka Connect received the timestamp as a string and no longer tried to determine whether it should be formatted as a Date or a Timestamp. The workaround fixed the issue for new events. However, downstream consumers still needed to handle date-only values from events published before we applied the change.

Conclusion

Across these incidents, the symptom often appeared in Debezium even when the failure started elsewhere. Database load delayed events before they reached the connector. Debezium transforms and Kafka Connect converters changed how the data was represented before it reached Kafka. Looking only at the connector metrics would have led us towards the wrong fixes.

The second and third incidents also showed how a workaround can affect a later stage of the pipeline. Emitting the payload as a string bypassed the schema-inference bug, but Debezium’s string serialization more than doubled the size of one record. We now test workarounds against the largest and most unusual data the system can receive, not only the input that originally triggered the problem.

We were building a multi-tenant compliance platform that processed 20 million messages per day. The platform identified messages that violated a tenant’s compliance policies. In the previous blog, we covered how we used Debezium for change data capture (CDC), along with the transactional outbox pattern. In this blog, we cover how we made Debezium setup production-ready, the issues we faced along the way, and how we solved those.

Before I talk about the production setup, here’s a high-level architecture diagram of our setup.

Data flow: application services write outbox collections in MongoDB, Debezium outbox connectors capture the changes and publish them to Kafka event topics for consumer services; a second path carries legacy config collections through config connectors to config topics

Data flow in our compliance platform

Our architecture consisted of several Java services that stored business documents and outbox events in MongoDB. Debezium used change data capture (CDC) to publish the outbox events to Kafka, which distributed them to downstream services. We also used this pipeline to sync configuration changes from the legacy system to the newer services.

1. Increase in Debezium connector lag during peak traffic

During peak traffic, our Debezium connectors started processing MongoDB change events 40–50 minutes late. This happened around the same time every day.

The delay affected downstream services that depended on these events. If the lag increased further, we risked breaching the platform’s SLOs. Although the connectors eventually caught up on their own, we needed to understand what was causing this delay.

We detected the issue through our existing alerts and dashboards for Debezium’s JMX metrics. One of these metrics was MilliSecondsBehindSource, which measures the time between MongoDB generating a change event and Debezium processing it.

We started by checking where the delay was happening. Were records piling up inside Debezium while waiting to be published to Kafka, or was Debezium taking longer to fetch events from MongoDB?

To investigate, we checked QueueRemainingCapacity. This metric measures how much space is available in Debezium’s internal buffer queue. It stayed near its maximum, which meant the queue was mostly empty. Records weren’t piling up while waiting to be published to Kafka. The connector’s CPU and memory usage were also normal.

This suggested that the delay was happening before events entered Debezium’s queue.

We then checked the MongoDB dashboard and found that CPU usage spiked around the same time every day.

Debezium reads MongoDB changes through a change stream cursor. It repeatedly requests the next batch of events from MongoDB. When MongoDB is busy, it can take longer to return those batches, delaying the events reaching Debezium.

This also explains why MilliSecondsBehindSource was increasing even though Debezium’s internal queue was mostly empty. The metric uses the change event’s oplog timestamp, so it includes the time spent waiting for MongoDB to return the event.

Increasing Debezium’s queue size wouldn’t have helped because the records weren’t waiting inside Debezium. The delay was happening before they reached the queue.

We made two fixes.

Increasing MongoDB CPU capacity

We investigated the slow queries behind the CPU spikes and found that workloads from other teams’ services on the same cluster were driving them. We identified those services and notified the teams that owned them.

The cluster also couldn’t keep up with the combined traffic, so we increased its CPU capacity.

Filtering change events earlier

Each Debezium connector opens a MongoDB change stream with an aggregation pipeline. The pipeline filters events down to the collections that the connector monitors.

In our Debezium 2.3 setup, each connector opened a change stream for the entire MongoDB deployment rather than only for the collections it monitored. The connector therefore saw changes from unrelated databases and collections before filtering them out.

For example, a connector interested only in database_a.messages also saw changes from database_b.messages and database_c.messages.

Debezium’s default aggregation pipeline first used $replaceRoot to restructure each change event and build a namespace field from the database and collection names. It then applied a regex filter to retain events from the configured collections.

This meant MongoDB had to restructure every change event, including those that the next stage would discard.

Before: the change stream sends events from every database through $replaceRoot, then $match keeps only database_a. After: a $match on ns.db and ns.coll runs first (cursor.pipeline.order=user_first), so $replaceRoot only processes database_a events

Change stream pipeline before and after the fix

We changed the pipeline to filter events before restructuring them. We added a $match stage on ns.db and ns.coll so MongoDB could filter events by database and collection before running Debezium’s default pipeline. We also set cursor.pipeline.order=user_first to make our stage run first.

{
  "cursor.pipeline.order": "user_first",
  "cursor.pipeline": "[{\"$match\":{\"ns.db\":\"database_a\",\"ns.coll\":\"messages\"}}]"

connector config pipeline properties after aggregation pipeline fix

This reduced the number of events processed by the rest of the aggregation pipeline. It didn’t remove the competing database load, so we still needed the additional CPU capacity. After both changes, MilliSecondsBehindSource dropped from 40–50 minutes during peak periods to under 10 seconds.

2. Debezium failure in processing a valid MongoDB document

One of our Debezium connectors stopped processing events because it couldn’t handle a valid MongoDB document containing an empty array nested inside another array. The connector task failed and stopped publishing new events to Kafka.

This blocked downstream services that depended on these events. The longer the connector remained stopped, the more events it would need to catch up on after restarting.

We checked the connector logs and found this error:

Only Struct objects supported for [Read Outbox Event], found

We started by inspecting the document that had caused the failure. It was valid BSON, and its structure didn’t look unusual. On closer inspection, we found an empty array (tags) inside an object that was part of another array (items).

{
  "payload": {
    "items": [
      {
        "name": "item-1",
        "tags"

The affected connector had transforms.outbox.collection.expand.json.payload set to true. With this setting, Debezium converts the JSON payload into a structured Kafka Connect value. To do this, it first infers a schema from the document’s fields. Downstream consumers then receive a structured value instead of a JSON string.

The problem wasn’t with empty arrays; Debezium could handle an empty array on its own. The bug occurred when the empty array was a field of an object inside another array, as tags is in this example.

While inferring the schema, Debezium identified the object inside items but included only the name field in its schema. It missed tags, even though the field was present in the document.

When Debezium tried to convert the document into a Kafka Connect Struct, the schema and the actual document structure no longer matched. The conversion failed.

We traced this to how Debezium 2.3 inferred schemas in two passes.

In the first pass, Debezium skipped empty arrays and empty documents because they didn’t contain enough information to determine their types. In the second pass, it revisited these fields only if the object had no other fields.

In our case, the object already had a name field. Debezium therefore didn’t revisit tags, leaving it out of the inferred schema.

How we fixed it

We considered three options:

  1. Skip failing events. We could set errors.tolerance=all to allow the connector to continue processing other events. However, Kafka Connect supports dead-letter queues (DLQs) only for sink connectors, not source connectors. Without a DLQ to capture skipped events, we risked silently losing compliance-related data. We ruled this out.

  2. Upgrade Debezium. Debezium 2.7 and later fixed the problem, but those versions were incompatible with the MongoDB 5.0 version we were running. We ruled this out as well.

  3. Disable JSON payload expansion. We could set transforms.outbox.collection.expand.json.payload=false to bypass schema inference. Debezium would then publish the payload as a JSON string instead of a structured Kafka Connect value. This meant downstream consumers would need to change how they parsed the payload.

We chose the third option and updated the connector configuration:

"transforms.outbox.collection.expand.json.payload" : false

This bypassed schema inference and allowed the connector to process the document without failing.

However, disabling payload expansion introduced another problem, which we discovered when the connector processed a large document. I have covered this issue below.

3. Debezium connector failure due to Kafka message size limit

In the previous issue, we disabled JSON payload expansion to bypass schema inference for the documents. This allowed the connector to process documents containing nested empty arrays. But this change introduced another problem.

One of our connectors failed when processing a 2.5 MB MongoDB document. After Debezium converted the document into a Kafka record, its size increased to 5.3 MB. This exceeded the 4 MB request-size limit configured for Kafka Connect’s producer.

As a result, the connector task stopped publishing events to Kafka, blocking downstream processing until we increased the limit and restarted it.

The connector logs showed the following exception:

org.apache.kafka.common.errors.RecordTooLargeException: The message is 5323164 bytes when serialized,
which is larger than 4194304

We wanted to understand why a 2.5 MB document had grown large enough to exceed Kafka’s 4 MB limit.

To reproduce the issue, we serialized the same logical payload as compact JSON and published it with a plain Kafka producer; it remained within the existing 4 MB limit. But when Debezium processed the same payload through its outbox transform, the serialized record grew to 5.3 MB.

In the previous issue, we used expand.json.payload=false as the connector configuration. This allowed Debezium to bypass the issue with inferring the schema for documents, but it also changed how Debezium serialized the payload.

We traced the serialization code and found that Debezium used MongoDB’s JSON writer with pretty-printing enabled:

JsonWriterSettings jsonWriterSettings = JsonWriterSettings.builder()
        .outputMode(JsonMode.EXTENDED)
        .indent(true)
        .newLineCharacters("\n")
        .build();

Source: Debezium MongoEventRouter.java

Two settings increased the payload size:

  • indent(true) added whitespace and line breaks throughout the document.

  • JsonMode.EXTENDED preserved BSON-specific types in the JSON representation. With this, values such as dates and 64-bit integers were represented using wrapper objects like {"$date": ...} and {"$numberLong": ...} . These representations take up more space than their compact JSON equivalents.

To confirm this, we serialized the document locally using the same settings and got 5,323,164 bytes, the exact size reported in the connector’s error.

These writer settings were hard-coded in the Debezium 2.3 we used. There was no connector setting to disable the indentation.

Here’s how we solved this issue.

Increasing Kafka’s message size limits

Since we couldn’t change Debezium’s JSON serialization settings, we needed Kafka to accept the larger records.

We first checked which Kafka limit was rejecting the record. The exception pointed to max.request.size, which limits the size of requests sent by a Kafka producer. We increased producer.override.max.request.size in the Kafka Connect configuration to allow the producer to send larger requests.

Kafka also has broker-level (message.max.bytes) and topic-level (max.message.bytes) limits. Increasing these alone wouldn’t have fixed our issue because the producer rejected the record before sending it to Kafka. We analysed our existing data and the largest payloads we expected from upstream services. Based on that analysis, we increased the producer and topic limits with some headroom.

After increasing the limits, the connector could publish the record again. But the change only allowed Kafka to accept larger records. It didn’t address the increase in size caused by Debezium’s JSON serialization.

For large payloads, we decided to store the document separately (in S3) and publish only a reference through Kafka. Consumers can then retrieve the document when they need it, rather than having the entire document pass through the event pipeline.

4. Kafka Connect converted timestamps to dates at midnight

Another issue we ran into was related to timestamps. Some messages published to Kafka had incomplete timestamps. Downstream consumers expected a full date and time, but some messages contained only the date. This caused those messages to fail during downstream processing.

For example, consumers expected: 2026-01-01T00:00:00.000Z But received: 2026-01-01 .

This issue affected only messages with timestamps at exactly midnight UTC. Messages with other timestamp values continued to work normally. Unlike the previous issues, this didn’t stop the connector or block the entire pipeline, but the affected messages failed during downstream processing.

We first checked the original MongoDB documents and confirmed that they contained complete timestamps. We also verified that the consumers weren’t modifying the headers. This pointed us to how Debezium was creating the Kafka headers.

Our outbox connectors used transforms.outbox.collection.fields.additional.placement to copy timestamp fields from the outbox document into Kafka headers. While creating these headers, Kafka Connect used its Values utility to convert timestamps into strings. We traced the issue to the dateFormatFor method, which selected the date format using this condition:

public static DateFormat dateFormatFor(java.util.Date value) {
    if (value.getTime() < MILLIS_PER_DAY) {
        return new SimpleDateFormat(ISO_8601_TIME_FORMAT_PATTERN);
    }
    if (value.getTime() % MILLIS_PER_DAY == 0) {
        return new SimpleDateFormat(ISO_8601_DATE_FORMAT_PATTERN);
    }
    return new SimpleDateFormat(ISO_8601_TIMESTAMP_FORMAT_PATTERN);
}

Source: Apache Kafka

The method checks whether the timestamp’s epoch value is divisible by the number of milliseconds in a day.

A day contains 24 * 60 * 60 * 1000 = 86,400,000 milliseconds. The epoch value for a timestamp at exactly midnight UTC is therefore an exact multiple of that number. For 2026-01-01T00:00:00Z:

1767225600000 % 86400000 == 0

Because the condition evaluated to true, Kafka Connect treated the value as a date and formatted it as 2026-01-01. It inferred the format from the numeric value instead of using the field’s logical type to distinguish a Timestamp from a Date.

We reported the bug on the Kafka users mailing list. A Kafka maintainer confirmed it, opened KAFKA-20752, and raised a pull request. At the time of writing, the pull request is still under review.

How we worked around it

As a workaround, we stored the timestamp fields in our outbox documents as ISO-8601 strings instead of BSON dates. This meant Kafka Connect received the timestamp as a string and no longer tried to determine whether it should be formatted as a Date or a Timestamp. The workaround fixed the issue for new events. However, downstream consumers still needed to handle date-only values from events published before we applied the change.

Conclusion

Across these incidents, the symptom often appeared in Debezium even when the failure started elsewhere. Database load delayed events before they reached the connector. Debezium transforms and Kafka Connect converters changed how the data was represented before it reached Kafka. Looking only at the connector metrics would have led us towards the wrong fixes.

The second and third incidents also showed how a workaround can affect a later stage of the pipeline. Emitting the payload as a string bypassed the schema-inference bug, but Debezium’s string serialization more than doubled the size of one record. We now test workarounds against the largest and most unusual data the system can receive, not only the input that originally triggered the problem.

We were building a multi-tenant compliance platform that processed 20 million messages per day. The platform identified messages that violated a tenant’s compliance policies. In the previous blog, we covered how we used Debezium for change data capture (CDC), along with the transactional outbox pattern. In this blog, we cover how we made Debezium setup production-ready, the issues we faced along the way, and how we solved those.

Before I talk about the production setup, here’s a high-level architecture diagram of our setup.

Data flow: application services write outbox collections in MongoDB, Debezium outbox connectors capture the changes and publish them to Kafka event topics for consumer services; a second path carries legacy config collections through config connectors to config topics

Data flow in our compliance platform

Our architecture consisted of several Java services that stored business documents and outbox events in MongoDB. Debezium used change data capture (CDC) to publish the outbox events to Kafka, which distributed them to downstream services. We also used this pipeline to sync configuration changes from the legacy system to the newer services.

1. Increase in Debezium connector lag during peak traffic

During peak traffic, our Debezium connectors started processing MongoDB change events 40–50 minutes late. This happened around the same time every day.

The delay affected downstream services that depended on these events. If the lag increased further, we risked breaching the platform’s SLOs. Although the connectors eventually caught up on their own, we needed to understand what was causing this delay.

We detected the issue through our existing alerts and dashboards for Debezium’s JMX metrics. One of these metrics was MilliSecondsBehindSource, which measures the time between MongoDB generating a change event and Debezium processing it.

We started by checking where the delay was happening. Were records piling up inside Debezium while waiting to be published to Kafka, or was Debezium taking longer to fetch events from MongoDB?

To investigate, we checked QueueRemainingCapacity. This metric measures how much space is available in Debezium’s internal buffer queue. It stayed near its maximum, which meant the queue was mostly empty. Records weren’t piling up while waiting to be published to Kafka. The connector’s CPU and memory usage were also normal.

This suggested that the delay was happening before events entered Debezium’s queue.

We then checked the MongoDB dashboard and found that CPU usage spiked around the same time every day.

Debezium reads MongoDB changes through a change stream cursor. It repeatedly requests the next batch of events from MongoDB. When MongoDB is busy, it can take longer to return those batches, delaying the events reaching Debezium.

This also explains why MilliSecondsBehindSource was increasing even though Debezium’s internal queue was mostly empty. The metric uses the change event’s oplog timestamp, so it includes the time spent waiting for MongoDB to return the event.

Increasing Debezium’s queue size wouldn’t have helped because the records weren’t waiting inside Debezium. The delay was happening before they reached the queue.

We made two fixes.

Increasing MongoDB CPU capacity

We investigated the slow queries behind the CPU spikes and found that workloads from other teams’ services on the same cluster were driving them. We identified those services and notified the teams that owned them.

The cluster also couldn’t keep up with the combined traffic, so we increased its CPU capacity.

Filtering change events earlier

Each Debezium connector opens a MongoDB change stream with an aggregation pipeline. The pipeline filters events down to the collections that the connector monitors.

In our Debezium 2.3 setup, each connector opened a change stream for the entire MongoDB deployment rather than only for the collections it monitored. The connector therefore saw changes from unrelated databases and collections before filtering them out.

For example, a connector interested only in database_a.messages also saw changes from database_b.messages and database_c.messages.

Debezium’s default aggregation pipeline first used $replaceRoot to restructure each change event and build a namespace field from the database and collection names. It then applied a regex filter to retain events from the configured collections.

This meant MongoDB had to restructure every change event, including those that the next stage would discard.

Before: the change stream sends events from every database through $replaceRoot, then $match keeps only database_a. After: a $match on ns.db and ns.coll runs first (cursor.pipeline.order=user_first), so $replaceRoot only processes database_a events

Change stream pipeline before and after the fix

We changed the pipeline to filter events before restructuring them. We added a $match stage on ns.db and ns.coll so MongoDB could filter events by database and collection before running Debezium’s default pipeline. We also set cursor.pipeline.order=user_first to make our stage run first.

{
  "cursor.pipeline.order": "user_first",
  "cursor.pipeline": "[{\"$match\":{\"ns.db\":\"database_a\",\"ns.coll\":\"messages\"}}]"

connector config pipeline properties after aggregation pipeline fix

This reduced the number of events processed by the rest of the aggregation pipeline. It didn’t remove the competing database load, so we still needed the additional CPU capacity. After both changes, MilliSecondsBehindSource dropped from 40–50 minutes during peak periods to under 10 seconds.

2. Debezium failure in processing a valid MongoDB document

One of our Debezium connectors stopped processing events because it couldn’t handle a valid MongoDB document containing an empty array nested inside another array. The connector task failed and stopped publishing new events to Kafka.

This blocked downstream services that depended on these events. The longer the connector remained stopped, the more events it would need to catch up on after restarting.

We checked the connector logs and found this error:

Only Struct objects supported for [Read Outbox Event], found

We started by inspecting the document that had caused the failure. It was valid BSON, and its structure didn’t look unusual. On closer inspection, we found an empty array (tags) inside an object that was part of another array (items).

{
  "payload": {
    "items": [
      {
        "name": "item-1",
        "tags"

The affected connector had transforms.outbox.collection.expand.json.payload set to true. With this setting, Debezium converts the JSON payload into a structured Kafka Connect value. To do this, it first infers a schema from the document’s fields. Downstream consumers then receive a structured value instead of a JSON string.

The problem wasn’t with empty arrays; Debezium could handle an empty array on its own. The bug occurred when the empty array was a field of an object inside another array, as tags is in this example.

While inferring the schema, Debezium identified the object inside items but included only the name field in its schema. It missed tags, even though the field was present in the document.

When Debezium tried to convert the document into a Kafka Connect Struct, the schema and the actual document structure no longer matched. The conversion failed.

We traced this to how Debezium 2.3 inferred schemas in two passes.

In the first pass, Debezium skipped empty arrays and empty documents because they didn’t contain enough information to determine their types. In the second pass, it revisited these fields only if the object had no other fields.

In our case, the object already had a name field. Debezium therefore didn’t revisit tags, leaving it out of the inferred schema.

How we fixed it

We considered three options:

  1. Skip failing events. We could set errors.tolerance=all to allow the connector to continue processing other events. However, Kafka Connect supports dead-letter queues (DLQs) only for sink connectors, not source connectors. Without a DLQ to capture skipped events, we risked silently losing compliance-related data. We ruled this out.

  2. Upgrade Debezium. Debezium 2.7 and later fixed the problem, but those versions were incompatible with the MongoDB 5.0 version we were running. We ruled this out as well.

  3. Disable JSON payload expansion. We could set transforms.outbox.collection.expand.json.payload=false to bypass schema inference. Debezium would then publish the payload as a JSON string instead of a structured Kafka Connect value. This meant downstream consumers would need to change how they parsed the payload.

We chose the third option and updated the connector configuration:

"transforms.outbox.collection.expand.json.payload" : false

This bypassed schema inference and allowed the connector to process the document without failing.

However, disabling payload expansion introduced another problem, which we discovered when the connector processed a large document. I have covered this issue below.

3. Debezium connector failure due to Kafka message size limit

In the previous issue, we disabled JSON payload expansion to bypass schema inference for the documents. This allowed the connector to process documents containing nested empty arrays. But this change introduced another problem.

One of our connectors failed when processing a 2.5 MB MongoDB document. After Debezium converted the document into a Kafka record, its size increased to 5.3 MB. This exceeded the 4 MB request-size limit configured for Kafka Connect’s producer.

As a result, the connector task stopped publishing events to Kafka, blocking downstream processing until we increased the limit and restarted it.

The connector logs showed the following exception:

org.apache.kafka.common.errors.RecordTooLargeException: The message is 5323164 bytes when serialized,
which is larger than 4194304

We wanted to understand why a 2.5 MB document had grown large enough to exceed Kafka’s 4 MB limit.

To reproduce the issue, we serialized the same logical payload as compact JSON and published it with a plain Kafka producer; it remained within the existing 4 MB limit. But when Debezium processed the same payload through its outbox transform, the serialized record grew to 5.3 MB.

In the previous issue, we used expand.json.payload=false as the connector configuration. This allowed Debezium to bypass the issue with inferring the schema for documents, but it also changed how Debezium serialized the payload.

We traced the serialization code and found that Debezium used MongoDB’s JSON writer with pretty-printing enabled:

JsonWriterSettings jsonWriterSettings = JsonWriterSettings.builder()
        .outputMode(JsonMode.EXTENDED)
        .indent(true)
        .newLineCharacters("\n")
        .build();

Source: Debezium MongoEventRouter.java

Two settings increased the payload size:

  • indent(true) added whitespace and line breaks throughout the document.

  • JsonMode.EXTENDED preserved BSON-specific types in the JSON representation. With this, values such as dates and 64-bit integers were represented using wrapper objects like {"$date": ...} and {"$numberLong": ...} . These representations take up more space than their compact JSON equivalents.

To confirm this, we serialized the document locally using the same settings and got 5,323,164 bytes, the exact size reported in the connector’s error.

These writer settings were hard-coded in the Debezium 2.3 we used. There was no connector setting to disable the indentation.

Here’s how we solved this issue.

Increasing Kafka’s message size limits

Since we couldn’t change Debezium’s JSON serialization settings, we needed Kafka to accept the larger records.

We first checked which Kafka limit was rejecting the record. The exception pointed to max.request.size, which limits the size of requests sent by a Kafka producer. We increased producer.override.max.request.size in the Kafka Connect configuration to allow the producer to send larger requests.

Kafka also has broker-level (message.max.bytes) and topic-level (max.message.bytes) limits. Increasing these alone wouldn’t have fixed our issue because the producer rejected the record before sending it to Kafka. We analysed our existing data and the largest payloads we expected from upstream services. Based on that analysis, we increased the producer and topic limits with some headroom.

After increasing the limits, the connector could publish the record again. But the change only allowed Kafka to accept larger records. It didn’t address the increase in size caused by Debezium’s JSON serialization.

For large payloads, we decided to store the document separately (in S3) and publish only a reference through Kafka. Consumers can then retrieve the document when they need it, rather than having the entire document pass through the event pipeline.

4. Kafka Connect converted timestamps to dates at midnight

Another issue we ran into was related to timestamps. Some messages published to Kafka had incomplete timestamps. Downstream consumers expected a full date and time, but some messages contained only the date. This caused those messages to fail during downstream processing.

For example, consumers expected: 2026-01-01T00:00:00.000Z But received: 2026-01-01 .

This issue affected only messages with timestamps at exactly midnight UTC. Messages with other timestamp values continued to work normally. Unlike the previous issues, this didn’t stop the connector or block the entire pipeline, but the affected messages failed during downstream processing.

We first checked the original MongoDB documents and confirmed that they contained complete timestamps. We also verified that the consumers weren’t modifying the headers. This pointed us to how Debezium was creating the Kafka headers.

Our outbox connectors used transforms.outbox.collection.fields.additional.placement to copy timestamp fields from the outbox document into Kafka headers. While creating these headers, Kafka Connect used its Values utility to convert timestamps into strings. We traced the issue to the dateFormatFor method, which selected the date format using this condition:

public static DateFormat dateFormatFor(java.util.Date value) {
    if (value.getTime() < MILLIS_PER_DAY) {
        return new SimpleDateFormat(ISO_8601_TIME_FORMAT_PATTERN);
    }
    if (value.getTime() % MILLIS_PER_DAY == 0) {
        return new SimpleDateFormat(ISO_8601_DATE_FORMAT_PATTERN);
    }
    return new SimpleDateFormat(ISO_8601_TIMESTAMP_FORMAT_PATTERN);
}

Source: Apache Kafka

The method checks whether the timestamp’s epoch value is divisible by the number of milliseconds in a day.

A day contains 24 * 60 * 60 * 1000 = 86,400,000 milliseconds. The epoch value for a timestamp at exactly midnight UTC is therefore an exact multiple of that number. For 2026-01-01T00:00:00Z:

1767225600000 % 86400000 == 0

Because the condition evaluated to true, Kafka Connect treated the value as a date and formatted it as 2026-01-01. It inferred the format from the numeric value instead of using the field’s logical type to distinguish a Timestamp from a Date.

We reported the bug on the Kafka users mailing list. A Kafka maintainer confirmed it, opened KAFKA-20752, and raised a pull request. At the time of writing, the pull request is still under review.

How we worked around it

As a workaround, we stored the timestamp fields in our outbox documents as ISO-8601 strings instead of BSON dates. This meant Kafka Connect received the timestamp as a string and no longer tried to determine whether it should be formatted as a Date or a Timestamp. The workaround fixed the issue for new events. However, downstream consumers still needed to handle date-only values from events published before we applied the change.

Conclusion

Across these incidents, the symptom often appeared in Debezium even when the failure started elsewhere. Database load delayed events before they reached the connector. Debezium transforms and Kafka Connect converters changed how the data was represented before it reached Kafka. Looking only at the connector metrics would have led us towards the wrong fixes.

The second and third incidents also showed how a workaround can affect a later stage of the pipeline. Emitting the payload as a string bypassed the schema-inference bug, but Debezium’s string serialization more than doubled the size of one record. We now test workarounds against the largest and most unusual data the system can receive, not only the input that originally triggered the problem.

Share
Share
On this page

Continue reading.

Subscribe for more such content

Get the latest in software engineering best practices straight to your inbox. Subscribe now!

Subscribe for more such content

Get the latest in software engineering best practices straight to your inbox. Subscribe now!

Subscribe for more such content

Get the latest in software engineering best practices straight to your inbox. Subscribe now!