Services

Resources

Company

Fixing dual writes with the transactional outbox pattern and Debezium

Fixing dual writes with the transactional outbox pattern and Debezium

Fixing dual writes with the transactional outbox pattern and Debezium

Fixing dual writes with the transactional outbox pattern and Debezium

We were building a multi-tenant compliance platform, which processed 20 million messages per day for a single tenant. The aim of the platform was to identify messages that violated the tenant’s compliance policies. The platform consisted of multiple Spring Boot-based microservices. These services communicated with each other via Kafka and used MongoDB as a database.

Each microservice was responsible for performing specific actions and publishing events over Kafka for downstream services to process. Some services also needed to write their state to MongoDB and publish events over Kafka for processing a single message.

This introduced the dual write problem. The MongoDB write could succeed, but Kafka publishing could fail. This could introduce inconsistency across the platform, as the downstream service will never know of a failed event. We needed a way to keep database writes and Kafka publishing consistent, and this led us to the transactional outbox pattern and Debezium.

The dual write problem

Consider a service that needs to store its processing result in MongoDB and also needs to publish an event to Kafka. In this scenario, what things can go wrong? Let’s assume the order of operations is as follows:

1. Write state to MongoDB

2. Publish event to Kafka

Let’s say the first operation, writing state to MongoDB, succeeds, but for some reason our Kafka broker is down, and publishing the event fails. Now one can argue that we can add retries and retry the failed operation; that seems fair. If the Kafka broker comes up within retries, then we are good, but what happens if all retries are exhausted? We have the same problem again.

Another issue is what if the server running our service crashes after the service writes to MongoDB but before publishing to Kafka. When the server recovers and the service comes up, it will not retry the publish, because nothing recorded that the last event still needed to be sent.

What if we reverse the order of the above operations?

1. Publish event to Kafka

2. Write state to MongoDB

Again, we can observe that if we have an issue connecting to MongoDB or there is a network issue, even though the service has published the event for downstream services, it fails to write its state to MongoDB.

How about we use MongoDB transactions or Kafka transactions for doing these two operations? A MongoDB transaction cannot solve this because it only controls operations performed within MongoDB. Similarly, a Kafka transaction can make a group of Kafka operations atomic, but it cannot include a MongoDB write. Neither system can atomically commit the database update and event publication together.

If the pod stops after the MongoDB write, the state exists without its event. If it stops after the Kafka publish, the event exists without its state.

Fig 1: A crash between the MongoDB write and the Kafka publish leaves either the event or the stored state missing.

Solving the first half of the dual write problem

Basically, we want both operations to succeed, but they run in two different systems. We separated the responsibility of persisting the event from publishing it. As we are storing the state in MongoDB, we can also store the event that we want to publish in MongoDB. With MongoDB transactions, we have a guarantee that this will be an atomic operation. It will look like this:

function save(state):
    session = db.startTransaction()

    db.state.save(state, session)
    db.outbox.save(buildEvent(state), session)

    session.commit()

MongoDB either commits both writes or rolls back both. If the state was stored successfully, we knew that a corresponding event had also been recorded.

If we are storing our state in the collection ‘A’ in MongoDB, then for events we can have a different collection; let’s call it the outbox collection. It was called an outbox because every event written to it eventually needed to be published to Kafka. This is the first step of using the transactional outbox pattern.

Publishing events from the outbox

We still needed to move the recorded events from MongoDB to Kafka. Our first approach was a small Spring Boot application with a scheduled job. It periodically queried the outbox collection and published the results to Kafka.

The proof of concept exposed another set of problems. The poller app needed a durable checkpoint so that it could resume after a crash. It also needed to handle the failure window between publishing an event and saving its checkpoint. If it published the event and crashed before advancing the checkpoint, it would publish the same event again after restarting. If it advanced the checkpoint before publishing, then the event could be lost due to a crash.

Polling also placed a continuous load on MongoDB. The application queried the collection at fixed intervals even when there were no new events. Increasing the interval reduced that load but also increased the delay before an event reached Kafka.

We could have built the necessary checkpointing, retries, duplicate handling, monitoring, and recovery ourselves. But this was infrastructure code rather than part of our business domain, and it would have required a lot of effort to build this from scratch.

We therefore looked for an existing change data capture solution and chose Debezium. Instead of periodically querying the outbox collection, Debezium captures committed MongoDB changes through its change-stream mechanism, maintains its position in the change log, and publishes those changes to Kafka.

Debezium as the publisher

Debezium is an open-source change data capture platform that supports several databases. For our use case, we used its MongoDB outbox connector with Kafka Connect.

The MongoDB outbox connector reads committed changes from MongoDB change streams and publishes corresponding records to Kafka. Instead of repeatedly querying collections, it follows the database change stream and stores its progress through Kafka Connect. After a restart, it resumes from its last recorded position.

Debezium provides an at-least-once guarantee. During failure recovery, Debezium can deliver the same change more than once. We therefore created a unique identifier for each event and designed downstream consumers to be idempotent.

For the transactional outbox, we configured the MongoDB Outbox Event Router transformation. It converts an inserted outbox document into a Kafka record by selecting the message key, determining the destination topic, and extracting the event payload.

Here is what one of our outbox documents looks like. We kept our own camelCase field names:

{
  "_id": "outbox-1",
  "eventId": "event-1",
  "aggregateId": "alert-1",
  "aggregateType": "alert",
  "eventName": "AlertCreated",
  "payload": {
    "alertId": "alert-1",
    "status": "OPEN"

The fields have distinct responsibilities:

  • eventId uniquely identifies the event. We copy it into the Kafka record headers so consumers can deduplicate.

  • aggregateId becomes the Kafka message key. Events for the same business entity use the same value, preserving order within a Kafka partition.

  • aggregateType decides the destination topic.

  • eventName is also copied into the headers, so consumers can route events without parsing the payload.

  • payload becomes the value of the Kafka record.

We mapped these names in the Outbox Event Router configuration with three settings:

"transforms.outbox.collection.field.event.key": "aggregateId",
"transforms.outbox.route.by.field": "aggregateType",
"transforms.outbox.collection.fields.additional.placement": "eventId:header,eventName:header"
Transactional outbox pattern with Debezium: the service writes the state collection and the outbox collection in one MongoDB transaction, then Debezium publishes outbox events to a Kafka topic

Fig 2: The service records state and event together in one MongoDB transaction. Debezium publishes the event to Kafka later, at least once.

Using Debezium for general change data capture

Once we had Debezium running for the outbox, we found another use for the same change data capture infrastructure. An older system remained the source of truth for configuration. When a user changed a configuration there, each of our services needed an up-to-date copy in its own MongoDB collection.

We configured another Debezium MongoDB connector to capture changes from the legacy configuration collections. The connector published an event whenever a source document was inserted, updated, or deleted. Our services consumed those events and updated their local configuration collections.

Debezium's MongoDB connector streams changes from the legacy configuration collection through Kafka to each service's own configuration collection

Fig 3: Syncing legacy configuration with Debezium’s MongoDB connector.

For updates, Debezium can include the document’s state after the change. It can also include the state before the change when MongoDB pre-images are enabled, and the connector is configured to capture them.

This use case differed from the outbox flow. The Outbox Event Router expected documents with fields such as aggregateid, aggregatetype, type, and payload and transformed each document into a business event. The general MongoDB connector imposed no outbox schema on the source collections. It emitted change-event envelopes containing the operation, source metadata, and available document state.

Design decisions and limitations

The transactional outbox removed the immediate dual write inconsistency from our request path. The business state and the event were now recorded in one MongoDB transaction, without introducing a distributed transaction or two-phase commit. Downstream services remained eventually consistent, and the design introduced several trade-offs that we needed to manage.

Consumers must handle duplicate delivery. If the connector crashes, Debezium resumes from the last offset persisted by Kafka Connect. It can therefore emit some changes again during recovery. Every event carries a unique eventId, and consumers either record the IDs they have processed or make their state changes idempotent.

Downstream state is eventually consistent. There is a delay between committing the MongoDB transaction and a consumer applying the resulting event. MongoDB load, connector lag, Kafka availability, and consumer processing time can all increase this delay.

Resume history is finite. MongoDB change streams depend on the oplog, whose retained history moves forward as new writes arrive. If the connector’s saved resume point falls outside that history, it cannot resume normally. Recovery then requires a separate plan. An initial database snapshot can reconstruct the current state, but it cannot recreate every historical event that has disappeared from the MongoDB oplog.

The delivery pipeline requires its own operations. Connector offsets, lag, failures, configuration, and upgrades now belong to infrastructure outside the application. We needed monitoring and recovery procedures for that infrastructure.

Summary

We began with a service that had to persist state and publish an event to the rest of the platform. The outbox separated those responsibilities without relying on application memory to remember what still needed to be published.

The service commits the business change and its event in the same MongoDB transaction. Debezium captures the outbox record and publishes it to Kafka; consumers handle possible duplicates.

The outbox did not turn MongoDB and Kafka into one transaction. It removed the need for them to be one.

If you liked this post, you might find these interesting too

We were building a multi-tenant compliance platform, which processed 20 million messages per day for a single tenant. The aim of the platform was to identify messages that violated the tenant’s compliance policies. The platform consisted of multiple Spring Boot-based microservices. These services communicated with each other via Kafka and used MongoDB as a database.

Each microservice was responsible for performing specific actions and publishing events over Kafka for downstream services to process. Some services also needed to write their state to MongoDB and publish events over Kafka for processing a single message.

This introduced the dual write problem. The MongoDB write could succeed, but Kafka publishing could fail. This could introduce inconsistency across the platform, as the downstream service will never know of a failed event. We needed a way to keep database writes and Kafka publishing consistent, and this led us to the transactional outbox pattern and Debezium.

The dual write problem

Consider a service that needs to store its processing result in MongoDB and also needs to publish an event to Kafka. In this scenario, what things can go wrong? Let’s assume the order of operations is as follows:

1. Write state to MongoDB

2. Publish event to Kafka

Let’s say the first operation, writing state to MongoDB, succeeds, but for some reason our Kafka broker is down, and publishing the event fails. Now one can argue that we can add retries and retry the failed operation; that seems fair. If the Kafka broker comes up within retries, then we are good, but what happens if all retries are exhausted? We have the same problem again.

Another issue is what if the server running our service crashes after the service writes to MongoDB but before publishing to Kafka. When the server recovers and the service comes up, it will not retry the publish, because nothing recorded that the last event still needed to be sent.

What if we reverse the order of the above operations?

1. Publish event to Kafka

2. Write state to MongoDB

Again, we can observe that if we have an issue connecting to MongoDB or there is a network issue, even though the service has published the event for downstream services, it fails to write its state to MongoDB.

How about we use MongoDB transactions or Kafka transactions for doing these two operations? A MongoDB transaction cannot solve this because it only controls operations performed within MongoDB. Similarly, a Kafka transaction can make a group of Kafka operations atomic, but it cannot include a MongoDB write. Neither system can atomically commit the database update and event publication together.

If the pod stops after the MongoDB write, the state exists without its event. If it stops after the Kafka publish, the event exists without its state.

Fig 1: A crash between the MongoDB write and the Kafka publish leaves either the event or the stored state missing.

Solving the first half of the dual write problem

Basically, we want both operations to succeed, but they run in two different systems. We separated the responsibility of persisting the event from publishing it. As we are storing the state in MongoDB, we can also store the event that we want to publish in MongoDB. With MongoDB transactions, we have a guarantee that this will be an atomic operation. It will look like this:

function save(state):
    session = db.startTransaction()

    db.state.save(state, session)
    db.outbox.save(buildEvent(state), session)

    session.commit()

MongoDB either commits both writes or rolls back both. If the state was stored successfully, we knew that a corresponding event had also been recorded.

If we are storing our state in the collection ‘A’ in MongoDB, then for events we can have a different collection; let’s call it the outbox collection. It was called an outbox because every event written to it eventually needed to be published to Kafka. This is the first step of using the transactional outbox pattern.

Publishing events from the outbox

We still needed to move the recorded events from MongoDB to Kafka. Our first approach was a small Spring Boot application with a scheduled job. It periodically queried the outbox collection and published the results to Kafka.

The proof of concept exposed another set of problems. The poller app needed a durable checkpoint so that it could resume after a crash. It also needed to handle the failure window between publishing an event and saving its checkpoint. If it published the event and crashed before advancing the checkpoint, it would publish the same event again after restarting. If it advanced the checkpoint before publishing, then the event could be lost due to a crash.

Polling also placed a continuous load on MongoDB. The application queried the collection at fixed intervals even when there were no new events. Increasing the interval reduced that load but also increased the delay before an event reached Kafka.

We could have built the necessary checkpointing, retries, duplicate handling, monitoring, and recovery ourselves. But this was infrastructure code rather than part of our business domain, and it would have required a lot of effort to build this from scratch.

We therefore looked for an existing change data capture solution and chose Debezium. Instead of periodically querying the outbox collection, Debezium captures committed MongoDB changes through its change-stream mechanism, maintains its position in the change log, and publishes those changes to Kafka.

Debezium as the publisher

Debezium is an open-source change data capture platform that supports several databases. For our use case, we used its MongoDB outbox connector with Kafka Connect.

The MongoDB outbox connector reads committed changes from MongoDB change streams and publishes corresponding records to Kafka. Instead of repeatedly querying collections, it follows the database change stream and stores its progress through Kafka Connect. After a restart, it resumes from its last recorded position.

Debezium provides an at-least-once guarantee. During failure recovery, Debezium can deliver the same change more than once. We therefore created a unique identifier for each event and designed downstream consumers to be idempotent.

For the transactional outbox, we configured the MongoDB Outbox Event Router transformation. It converts an inserted outbox document into a Kafka record by selecting the message key, determining the destination topic, and extracting the event payload.

Here is what one of our outbox documents looks like. We kept our own camelCase field names:

{
  "_id": "outbox-1",
  "eventId": "event-1",
  "aggregateId": "alert-1",
  "aggregateType": "alert",
  "eventName": "AlertCreated",
  "payload": {
    "alertId": "alert-1",
    "status": "OPEN"

The fields have distinct responsibilities:

  • eventId uniquely identifies the event. We copy it into the Kafka record headers so consumers can deduplicate.

  • aggregateId becomes the Kafka message key. Events for the same business entity use the same value, preserving order within a Kafka partition.

  • aggregateType decides the destination topic.

  • eventName is also copied into the headers, so consumers can route events without parsing the payload.

  • payload becomes the value of the Kafka record.

We mapped these names in the Outbox Event Router configuration with three settings:

"transforms.outbox.collection.field.event.key": "aggregateId",
"transforms.outbox.route.by.field": "aggregateType",
"transforms.outbox.collection.fields.additional.placement": "eventId:header,eventName:header"
Transactional outbox pattern with Debezium: the service writes the state collection and the outbox collection in one MongoDB transaction, then Debezium publishes outbox events to a Kafka topic

Fig 2: The service records state and event together in one MongoDB transaction. Debezium publishes the event to Kafka later, at least once.

Using Debezium for general change data capture

Once we had Debezium running for the outbox, we found another use for the same change data capture infrastructure. An older system remained the source of truth for configuration. When a user changed a configuration there, each of our services needed an up-to-date copy in its own MongoDB collection.

We configured another Debezium MongoDB connector to capture changes from the legacy configuration collections. The connector published an event whenever a source document was inserted, updated, or deleted. Our services consumed those events and updated their local configuration collections.

Debezium's MongoDB connector streams changes from the legacy configuration collection through Kafka to each service's own configuration collection

Fig 3: Syncing legacy configuration with Debezium’s MongoDB connector.

For updates, Debezium can include the document’s state after the change. It can also include the state before the change when MongoDB pre-images are enabled, and the connector is configured to capture them.

This use case differed from the outbox flow. The Outbox Event Router expected documents with fields such as aggregateid, aggregatetype, type, and payload and transformed each document into a business event. The general MongoDB connector imposed no outbox schema on the source collections. It emitted change-event envelopes containing the operation, source metadata, and available document state.

Design decisions and limitations

The transactional outbox removed the immediate dual write inconsistency from our request path. The business state and the event were now recorded in one MongoDB transaction, without introducing a distributed transaction or two-phase commit. Downstream services remained eventually consistent, and the design introduced several trade-offs that we needed to manage.

Consumers must handle duplicate delivery. If the connector crashes, Debezium resumes from the last offset persisted by Kafka Connect. It can therefore emit some changes again during recovery. Every event carries a unique eventId, and consumers either record the IDs they have processed or make their state changes idempotent.

Downstream state is eventually consistent. There is a delay between committing the MongoDB transaction and a consumer applying the resulting event. MongoDB load, connector lag, Kafka availability, and consumer processing time can all increase this delay.

Resume history is finite. MongoDB change streams depend on the oplog, whose retained history moves forward as new writes arrive. If the connector’s saved resume point falls outside that history, it cannot resume normally. Recovery then requires a separate plan. An initial database snapshot can reconstruct the current state, but it cannot recreate every historical event that has disappeared from the MongoDB oplog.

The delivery pipeline requires its own operations. Connector offsets, lag, failures, configuration, and upgrades now belong to infrastructure outside the application. We needed monitoring and recovery procedures for that infrastructure.

Summary

We began with a service that had to persist state and publish an event to the rest of the platform. The outbox separated those responsibilities without relying on application memory to remember what still needed to be published.

The service commits the business change and its event in the same MongoDB transaction. Debezium captures the outbox record and publishes it to Kafka; consumers handle possible duplicates.

The outbox did not turn MongoDB and Kafka into one transaction. It removed the need for them to be one.

If you liked this post, you might find these interesting too

We were building a multi-tenant compliance platform, which processed 20 million messages per day for a single tenant. The aim of the platform was to identify messages that violated the tenant’s compliance policies. The platform consisted of multiple Spring Boot-based microservices. These services communicated with each other via Kafka and used MongoDB as a database.

Each microservice was responsible for performing specific actions and publishing events over Kafka for downstream services to process. Some services also needed to write their state to MongoDB and publish events over Kafka for processing a single message.

This introduced the dual write problem. The MongoDB write could succeed, but Kafka publishing could fail. This could introduce inconsistency across the platform, as the downstream service will never know of a failed event. We needed a way to keep database writes and Kafka publishing consistent, and this led us to the transactional outbox pattern and Debezium.

The dual write problem

Consider a service that needs to store its processing result in MongoDB and also needs to publish an event to Kafka. In this scenario, what things can go wrong? Let’s assume the order of operations is as follows:

1. Write state to MongoDB

2. Publish event to Kafka

Let’s say the first operation, writing state to MongoDB, succeeds, but for some reason our Kafka broker is down, and publishing the event fails. Now one can argue that we can add retries and retry the failed operation; that seems fair. If the Kafka broker comes up within retries, then we are good, but what happens if all retries are exhausted? We have the same problem again.

Another issue is what if the server running our service crashes after the service writes to MongoDB but before publishing to Kafka. When the server recovers and the service comes up, it will not retry the publish, because nothing recorded that the last event still needed to be sent.

What if we reverse the order of the above operations?

1. Publish event to Kafka

2. Write state to MongoDB

Again, we can observe that if we have an issue connecting to MongoDB or there is a network issue, even though the service has published the event for downstream services, it fails to write its state to MongoDB.

How about we use MongoDB transactions or Kafka transactions for doing these two operations? A MongoDB transaction cannot solve this because it only controls operations performed within MongoDB. Similarly, a Kafka transaction can make a group of Kafka operations atomic, but it cannot include a MongoDB write. Neither system can atomically commit the database update and event publication together.

If the pod stops after the MongoDB write, the state exists without its event. If it stops after the Kafka publish, the event exists without its state.

Fig 1: A crash between the MongoDB write and the Kafka publish leaves either the event or the stored state missing.

Solving the first half of the dual write problem

Basically, we want both operations to succeed, but they run in two different systems. We separated the responsibility of persisting the event from publishing it. As we are storing the state in MongoDB, we can also store the event that we want to publish in MongoDB. With MongoDB transactions, we have a guarantee that this will be an atomic operation. It will look like this:

function save(state):
    session = db.startTransaction()

    db.state.save(state, session)
    db.outbox.save(buildEvent(state), session)

    session.commit()

MongoDB either commits both writes or rolls back both. If the state was stored successfully, we knew that a corresponding event had also been recorded.

If we are storing our state in the collection ‘A’ in MongoDB, then for events we can have a different collection; let’s call it the outbox collection. It was called an outbox because every event written to it eventually needed to be published to Kafka. This is the first step of using the transactional outbox pattern.

Publishing events from the outbox

We still needed to move the recorded events from MongoDB to Kafka. Our first approach was a small Spring Boot application with a scheduled job. It periodically queried the outbox collection and published the results to Kafka.

The proof of concept exposed another set of problems. The poller app needed a durable checkpoint so that it could resume after a crash. It also needed to handle the failure window between publishing an event and saving its checkpoint. If it published the event and crashed before advancing the checkpoint, it would publish the same event again after restarting. If it advanced the checkpoint before publishing, then the event could be lost due to a crash.

Polling also placed a continuous load on MongoDB. The application queried the collection at fixed intervals even when there were no new events. Increasing the interval reduced that load but also increased the delay before an event reached Kafka.

We could have built the necessary checkpointing, retries, duplicate handling, monitoring, and recovery ourselves. But this was infrastructure code rather than part of our business domain, and it would have required a lot of effort to build this from scratch.

We therefore looked for an existing change data capture solution and chose Debezium. Instead of periodically querying the outbox collection, Debezium captures committed MongoDB changes through its change-stream mechanism, maintains its position in the change log, and publishes those changes to Kafka.

Debezium as the publisher

Debezium is an open-source change data capture platform that supports several databases. For our use case, we used its MongoDB outbox connector with Kafka Connect.

The MongoDB outbox connector reads committed changes from MongoDB change streams and publishes corresponding records to Kafka. Instead of repeatedly querying collections, it follows the database change stream and stores its progress through Kafka Connect. After a restart, it resumes from its last recorded position.

Debezium provides an at-least-once guarantee. During failure recovery, Debezium can deliver the same change more than once. We therefore created a unique identifier for each event and designed downstream consumers to be idempotent.

For the transactional outbox, we configured the MongoDB Outbox Event Router transformation. It converts an inserted outbox document into a Kafka record by selecting the message key, determining the destination topic, and extracting the event payload.

Here is what one of our outbox documents looks like. We kept our own camelCase field names:

{
  "_id": "outbox-1",
  "eventId": "event-1",
  "aggregateId": "alert-1",
  "aggregateType": "alert",
  "eventName": "AlertCreated",
  "payload": {
    "alertId": "alert-1",
    "status": "OPEN"

The fields have distinct responsibilities:

  • eventId uniquely identifies the event. We copy it into the Kafka record headers so consumers can deduplicate.

  • aggregateId becomes the Kafka message key. Events for the same business entity use the same value, preserving order within a Kafka partition.

  • aggregateType decides the destination topic.

  • eventName is also copied into the headers, so consumers can route events without parsing the payload.

  • payload becomes the value of the Kafka record.

We mapped these names in the Outbox Event Router configuration with three settings:

"transforms.outbox.collection.field.event.key": "aggregateId",
"transforms.outbox.route.by.field": "aggregateType",
"transforms.outbox.collection.fields.additional.placement": "eventId:header,eventName:header"
Transactional outbox pattern with Debezium: the service writes the state collection and the outbox collection in one MongoDB transaction, then Debezium publishes outbox events to a Kafka topic

Fig 2: The service records state and event together in one MongoDB transaction. Debezium publishes the event to Kafka later, at least once.

Using Debezium for general change data capture

Once we had Debezium running for the outbox, we found another use for the same change data capture infrastructure. An older system remained the source of truth for configuration. When a user changed a configuration there, each of our services needed an up-to-date copy in its own MongoDB collection.

We configured another Debezium MongoDB connector to capture changes from the legacy configuration collections. The connector published an event whenever a source document was inserted, updated, or deleted. Our services consumed those events and updated their local configuration collections.

Debezium's MongoDB connector streams changes from the legacy configuration collection through Kafka to each service's own configuration collection

Fig 3: Syncing legacy configuration with Debezium’s MongoDB connector.

For updates, Debezium can include the document’s state after the change. It can also include the state before the change when MongoDB pre-images are enabled, and the connector is configured to capture them.

This use case differed from the outbox flow. The Outbox Event Router expected documents with fields such as aggregateid, aggregatetype, type, and payload and transformed each document into a business event. The general MongoDB connector imposed no outbox schema on the source collections. It emitted change-event envelopes containing the operation, source metadata, and available document state.

Design decisions and limitations

The transactional outbox removed the immediate dual write inconsistency from our request path. The business state and the event were now recorded in one MongoDB transaction, without introducing a distributed transaction or two-phase commit. Downstream services remained eventually consistent, and the design introduced several trade-offs that we needed to manage.

Consumers must handle duplicate delivery. If the connector crashes, Debezium resumes from the last offset persisted by Kafka Connect. It can therefore emit some changes again during recovery. Every event carries a unique eventId, and consumers either record the IDs they have processed or make their state changes idempotent.

Downstream state is eventually consistent. There is a delay between committing the MongoDB transaction and a consumer applying the resulting event. MongoDB load, connector lag, Kafka availability, and consumer processing time can all increase this delay.

Resume history is finite. MongoDB change streams depend on the oplog, whose retained history moves forward as new writes arrive. If the connector’s saved resume point falls outside that history, it cannot resume normally. Recovery then requires a separate plan. An initial database snapshot can reconstruct the current state, but it cannot recreate every historical event that has disappeared from the MongoDB oplog.

The delivery pipeline requires its own operations. Connector offsets, lag, failures, configuration, and upgrades now belong to infrastructure outside the application. We needed monitoring and recovery procedures for that infrastructure.

Summary

We began with a service that had to persist state and publish an event to the rest of the platform. The outbox separated those responsibilities without relying on application memory to remember what still needed to be published.

The service commits the business change and its event in the same MongoDB transaction. Debezium captures the outbox record and publishes it to Kafka; consumers handle possible duplicates.

The outbox did not turn MongoDB and Kafka into one transaction. It removed the need for them to be one.

If you liked this post, you might find these interesting too

We were building a multi-tenant compliance platform, which processed 20 million messages per day for a single tenant. The aim of the platform was to identify messages that violated the tenant’s compliance policies. The platform consisted of multiple Spring Boot-based microservices. These services communicated with each other via Kafka and used MongoDB as a database.

Each microservice was responsible for performing specific actions and publishing events over Kafka for downstream services to process. Some services also needed to write their state to MongoDB and publish events over Kafka for processing a single message.

This introduced the dual write problem. The MongoDB write could succeed, but Kafka publishing could fail. This could introduce inconsistency across the platform, as the downstream service will never know of a failed event. We needed a way to keep database writes and Kafka publishing consistent, and this led us to the transactional outbox pattern and Debezium.

The dual write problem

Consider a service that needs to store its processing result in MongoDB and also needs to publish an event to Kafka. In this scenario, what things can go wrong? Let’s assume the order of operations is as follows:

1. Write state to MongoDB

2. Publish event to Kafka

Let’s say the first operation, writing state to MongoDB, succeeds, but for some reason our Kafka broker is down, and publishing the event fails. Now one can argue that we can add retries and retry the failed operation; that seems fair. If the Kafka broker comes up within retries, then we are good, but what happens if all retries are exhausted? We have the same problem again.

Another issue is what if the server running our service crashes after the service writes to MongoDB but before publishing to Kafka. When the server recovers and the service comes up, it will not retry the publish, because nothing recorded that the last event still needed to be sent.

What if we reverse the order of the above operations?

1. Publish event to Kafka

2. Write state to MongoDB

Again, we can observe that if we have an issue connecting to MongoDB or there is a network issue, even though the service has published the event for downstream services, it fails to write its state to MongoDB.

How about we use MongoDB transactions or Kafka transactions for doing these two operations? A MongoDB transaction cannot solve this because it only controls operations performed within MongoDB. Similarly, a Kafka transaction can make a group of Kafka operations atomic, but it cannot include a MongoDB write. Neither system can atomically commit the database update and event publication together.

If the pod stops after the MongoDB write, the state exists without its event. If it stops after the Kafka publish, the event exists without its state.

Fig 1: A crash between the MongoDB write and the Kafka publish leaves either the event or the stored state missing.

Solving the first half of the dual write problem

Basically, we want both operations to succeed, but they run in two different systems. We separated the responsibility of persisting the event from publishing it. As we are storing the state in MongoDB, we can also store the event that we want to publish in MongoDB. With MongoDB transactions, we have a guarantee that this will be an atomic operation. It will look like this:

function save(state):
    session = db.startTransaction()

    db.state.save(state, session)
    db.outbox.save(buildEvent(state), session)

    session.commit()

MongoDB either commits both writes or rolls back both. If the state was stored successfully, we knew that a corresponding event had also been recorded.

If we are storing our state in the collection ‘A’ in MongoDB, then for events we can have a different collection; let’s call it the outbox collection. It was called an outbox because every event written to it eventually needed to be published to Kafka. This is the first step of using the transactional outbox pattern.

Publishing events from the outbox

We still needed to move the recorded events from MongoDB to Kafka. Our first approach was a small Spring Boot application with a scheduled job. It periodically queried the outbox collection and published the results to Kafka.

The proof of concept exposed another set of problems. The poller app needed a durable checkpoint so that it could resume after a crash. It also needed to handle the failure window between publishing an event and saving its checkpoint. If it published the event and crashed before advancing the checkpoint, it would publish the same event again after restarting. If it advanced the checkpoint before publishing, then the event could be lost due to a crash.

Polling also placed a continuous load on MongoDB. The application queried the collection at fixed intervals even when there were no new events. Increasing the interval reduced that load but also increased the delay before an event reached Kafka.

We could have built the necessary checkpointing, retries, duplicate handling, monitoring, and recovery ourselves. But this was infrastructure code rather than part of our business domain, and it would have required a lot of effort to build this from scratch.

We therefore looked for an existing change data capture solution and chose Debezium. Instead of periodically querying the outbox collection, Debezium captures committed MongoDB changes through its change-stream mechanism, maintains its position in the change log, and publishes those changes to Kafka.

Debezium as the publisher

Debezium is an open-source change data capture platform that supports several databases. For our use case, we used its MongoDB outbox connector with Kafka Connect.

The MongoDB outbox connector reads committed changes from MongoDB change streams and publishes corresponding records to Kafka. Instead of repeatedly querying collections, it follows the database change stream and stores its progress through Kafka Connect. After a restart, it resumes from its last recorded position.

Debezium provides an at-least-once guarantee. During failure recovery, Debezium can deliver the same change more than once. We therefore created a unique identifier for each event and designed downstream consumers to be idempotent.

For the transactional outbox, we configured the MongoDB Outbox Event Router transformation. It converts an inserted outbox document into a Kafka record by selecting the message key, determining the destination topic, and extracting the event payload.

Here is what one of our outbox documents looks like. We kept our own camelCase field names:

{
  "_id": "outbox-1",
  "eventId": "event-1",
  "aggregateId": "alert-1",
  "aggregateType": "alert",
  "eventName": "AlertCreated",
  "payload": {
    "alertId": "alert-1",
    "status": "OPEN"

The fields have distinct responsibilities:

  • eventId uniquely identifies the event. We copy it into the Kafka record headers so consumers can deduplicate.

  • aggregateId becomes the Kafka message key. Events for the same business entity use the same value, preserving order within a Kafka partition.

  • aggregateType decides the destination topic.

  • eventName is also copied into the headers, so consumers can route events without parsing the payload.

  • payload becomes the value of the Kafka record.

We mapped these names in the Outbox Event Router configuration with three settings:

"transforms.outbox.collection.field.event.key": "aggregateId",
"transforms.outbox.route.by.field": "aggregateType",
"transforms.outbox.collection.fields.additional.placement": "eventId:header,eventName:header"
Transactional outbox pattern with Debezium: the service writes the state collection and the outbox collection in one MongoDB transaction, then Debezium publishes outbox events to a Kafka topic

Fig 2: The service records state and event together in one MongoDB transaction. Debezium publishes the event to Kafka later, at least once.

Using Debezium for general change data capture

Once we had Debezium running for the outbox, we found another use for the same change data capture infrastructure. An older system remained the source of truth for configuration. When a user changed a configuration there, each of our services needed an up-to-date copy in its own MongoDB collection.

We configured another Debezium MongoDB connector to capture changes from the legacy configuration collections. The connector published an event whenever a source document was inserted, updated, or deleted. Our services consumed those events and updated their local configuration collections.

Debezium's MongoDB connector streams changes from the legacy configuration collection through Kafka to each service's own configuration collection

Fig 3: Syncing legacy configuration with Debezium’s MongoDB connector.

For updates, Debezium can include the document’s state after the change. It can also include the state before the change when MongoDB pre-images are enabled, and the connector is configured to capture them.

This use case differed from the outbox flow. The Outbox Event Router expected documents with fields such as aggregateid, aggregatetype, type, and payload and transformed each document into a business event. The general MongoDB connector imposed no outbox schema on the source collections. It emitted change-event envelopes containing the operation, source metadata, and available document state.

Design decisions and limitations

The transactional outbox removed the immediate dual write inconsistency from our request path. The business state and the event were now recorded in one MongoDB transaction, without introducing a distributed transaction or two-phase commit. Downstream services remained eventually consistent, and the design introduced several trade-offs that we needed to manage.

Consumers must handle duplicate delivery. If the connector crashes, Debezium resumes from the last offset persisted by Kafka Connect. It can therefore emit some changes again during recovery. Every event carries a unique eventId, and consumers either record the IDs they have processed or make their state changes idempotent.

Downstream state is eventually consistent. There is a delay between committing the MongoDB transaction and a consumer applying the resulting event. MongoDB load, connector lag, Kafka availability, and consumer processing time can all increase this delay.

Resume history is finite. MongoDB change streams depend on the oplog, whose retained history moves forward as new writes arrive. If the connector’s saved resume point falls outside that history, it cannot resume normally. Recovery then requires a separate plan. An initial database snapshot can reconstruct the current state, but it cannot recreate every historical event that has disappeared from the MongoDB oplog.

The delivery pipeline requires its own operations. Connector offsets, lag, failures, configuration, and upgrades now belong to infrastructure outside the application. We needed monitoring and recovery procedures for that infrastructure.

Summary

We began with a service that had to persist state and publish an event to the rest of the platform. The outbox separated those responsibilities without relying on application memory to remember what still needed to be published.

The service commits the business change and its event in the same MongoDB transaction. Debezium captures the outbox record and publishes it to Kafka; consumers handle possible duplicates.

The outbox did not turn MongoDB and Kafka into one transaction. It removed the need for them to be one.

If you liked this post, you might find these interesting too

We were building a multi-tenant compliance platform, which processed 20 million messages per day for a single tenant. The aim of the platform was to identify messages that violated the tenant’s compliance policies. The platform consisted of multiple Spring Boot-based microservices. These services communicated with each other via Kafka and used MongoDB as a database.

Each microservice was responsible for performing specific actions and publishing events over Kafka for downstream services to process. Some services also needed to write their state to MongoDB and publish events over Kafka for processing a single message.

This introduced the dual write problem. The MongoDB write could succeed, but Kafka publishing could fail. This could introduce inconsistency across the platform, as the downstream service will never know of a failed event. We needed a way to keep database writes and Kafka publishing consistent, and this led us to the transactional outbox pattern and Debezium.

The dual write problem

Consider a service that needs to store its processing result in MongoDB and also needs to publish an event to Kafka. In this scenario, what things can go wrong? Let’s assume the order of operations is as follows:

1. Write state to MongoDB

2. Publish event to Kafka

Let’s say the first operation, writing state to MongoDB, succeeds, but for some reason our Kafka broker is down, and publishing the event fails. Now one can argue that we can add retries and retry the failed operation; that seems fair. If the Kafka broker comes up within retries, then we are good, but what happens if all retries are exhausted? We have the same problem again.

Another issue is what if the server running our service crashes after the service writes to MongoDB but before publishing to Kafka. When the server recovers and the service comes up, it will not retry the publish, because nothing recorded that the last event still needed to be sent.

What if we reverse the order of the above operations?

1. Publish event to Kafka

2. Write state to MongoDB

Again, we can observe that if we have an issue connecting to MongoDB or there is a network issue, even though the service has published the event for downstream services, it fails to write its state to MongoDB.

How about we use MongoDB transactions or Kafka transactions for doing these two operations? A MongoDB transaction cannot solve this because it only controls operations performed within MongoDB. Similarly, a Kafka transaction can make a group of Kafka operations atomic, but it cannot include a MongoDB write. Neither system can atomically commit the database update and event publication together.

If the pod stops after the MongoDB write, the state exists without its event. If it stops after the Kafka publish, the event exists without its state.

Fig 1: A crash between the MongoDB write and the Kafka publish leaves either the event or the stored state missing.

Solving the first half of the dual write problem

Basically, we want both operations to succeed, but they run in two different systems. We separated the responsibility of persisting the event from publishing it. As we are storing the state in MongoDB, we can also store the event that we want to publish in MongoDB. With MongoDB transactions, we have a guarantee that this will be an atomic operation. It will look like this:

function save(state):
    session = db.startTransaction()

    db.state.save(state, session)
    db.outbox.save(buildEvent(state), session)

    session.commit()

MongoDB either commits both writes or rolls back both. If the state was stored successfully, we knew that a corresponding event had also been recorded.

If we are storing our state in the collection ‘A’ in MongoDB, then for events we can have a different collection; let’s call it the outbox collection. It was called an outbox because every event written to it eventually needed to be published to Kafka. This is the first step of using the transactional outbox pattern.

Publishing events from the outbox

We still needed to move the recorded events from MongoDB to Kafka. Our first approach was a small Spring Boot application with a scheduled job. It periodically queried the outbox collection and published the results to Kafka.

The proof of concept exposed another set of problems. The poller app needed a durable checkpoint so that it could resume after a crash. It also needed to handle the failure window between publishing an event and saving its checkpoint. If it published the event and crashed before advancing the checkpoint, it would publish the same event again after restarting. If it advanced the checkpoint before publishing, then the event could be lost due to a crash.

Polling also placed a continuous load on MongoDB. The application queried the collection at fixed intervals even when there were no new events. Increasing the interval reduced that load but also increased the delay before an event reached Kafka.

We could have built the necessary checkpointing, retries, duplicate handling, monitoring, and recovery ourselves. But this was infrastructure code rather than part of our business domain, and it would have required a lot of effort to build this from scratch.

We therefore looked for an existing change data capture solution and chose Debezium. Instead of periodically querying the outbox collection, Debezium captures committed MongoDB changes through its change-stream mechanism, maintains its position in the change log, and publishes those changes to Kafka.

Debezium as the publisher

Debezium is an open-source change data capture platform that supports several databases. For our use case, we used its MongoDB outbox connector with Kafka Connect.

The MongoDB outbox connector reads committed changes from MongoDB change streams and publishes corresponding records to Kafka. Instead of repeatedly querying collections, it follows the database change stream and stores its progress through Kafka Connect. After a restart, it resumes from its last recorded position.

Debezium provides an at-least-once guarantee. During failure recovery, Debezium can deliver the same change more than once. We therefore created a unique identifier for each event and designed downstream consumers to be idempotent.

For the transactional outbox, we configured the MongoDB Outbox Event Router transformation. It converts an inserted outbox document into a Kafka record by selecting the message key, determining the destination topic, and extracting the event payload.

Here is what one of our outbox documents looks like. We kept our own camelCase field names:

{
  "_id": "outbox-1",
  "eventId": "event-1",
  "aggregateId": "alert-1",
  "aggregateType": "alert",
  "eventName": "AlertCreated",
  "payload": {
    "alertId": "alert-1",
    "status": "OPEN"

The fields have distinct responsibilities:

  • eventId uniquely identifies the event. We copy it into the Kafka record headers so consumers can deduplicate.

  • aggregateId becomes the Kafka message key. Events for the same business entity use the same value, preserving order within a Kafka partition.

  • aggregateType decides the destination topic.

  • eventName is also copied into the headers, so consumers can route events without parsing the payload.

  • payload becomes the value of the Kafka record.

We mapped these names in the Outbox Event Router configuration with three settings:

"transforms.outbox.collection.field.event.key": "aggregateId",
"transforms.outbox.route.by.field": "aggregateType",
"transforms.outbox.collection.fields.additional.placement": "eventId:header,eventName:header"
Transactional outbox pattern with Debezium: the service writes the state collection and the outbox collection in one MongoDB transaction, then Debezium publishes outbox events to a Kafka topic

Fig 2: The service records state and event together in one MongoDB transaction. Debezium publishes the event to Kafka later, at least once.

Using Debezium for general change data capture

Once we had Debezium running for the outbox, we found another use for the same change data capture infrastructure. An older system remained the source of truth for configuration. When a user changed a configuration there, each of our services needed an up-to-date copy in its own MongoDB collection.

We configured another Debezium MongoDB connector to capture changes from the legacy configuration collections. The connector published an event whenever a source document was inserted, updated, or deleted. Our services consumed those events and updated their local configuration collections.

Debezium's MongoDB connector streams changes from the legacy configuration collection through Kafka to each service's own configuration collection

Fig 3: Syncing legacy configuration with Debezium’s MongoDB connector.

For updates, Debezium can include the document’s state after the change. It can also include the state before the change when MongoDB pre-images are enabled, and the connector is configured to capture them.

This use case differed from the outbox flow. The Outbox Event Router expected documents with fields such as aggregateid, aggregatetype, type, and payload and transformed each document into a business event. The general MongoDB connector imposed no outbox schema on the source collections. It emitted change-event envelopes containing the operation, source metadata, and available document state.

Design decisions and limitations

The transactional outbox removed the immediate dual write inconsistency from our request path. The business state and the event were now recorded in one MongoDB transaction, without introducing a distributed transaction or two-phase commit. Downstream services remained eventually consistent, and the design introduced several trade-offs that we needed to manage.

Consumers must handle duplicate delivery. If the connector crashes, Debezium resumes from the last offset persisted by Kafka Connect. It can therefore emit some changes again during recovery. Every event carries a unique eventId, and consumers either record the IDs they have processed or make their state changes idempotent.

Downstream state is eventually consistent. There is a delay between committing the MongoDB transaction and a consumer applying the resulting event. MongoDB load, connector lag, Kafka availability, and consumer processing time can all increase this delay.

Resume history is finite. MongoDB change streams depend on the oplog, whose retained history moves forward as new writes arrive. If the connector’s saved resume point falls outside that history, it cannot resume normally. Recovery then requires a separate plan. An initial database snapshot can reconstruct the current state, but it cannot recreate every historical event that has disappeared from the MongoDB oplog.

The delivery pipeline requires its own operations. Connector offsets, lag, failures, configuration, and upgrades now belong to infrastructure outside the application. We needed monitoring and recovery procedures for that infrastructure.

Summary

We began with a service that had to persist state and publish an event to the rest of the platform. The outbox separated those responsibilities without relying on application memory to remember what still needed to be published.

The service commits the business change and its event in the same MongoDB transaction. Debezium captures the outbox record and publishes it to Kafka; consumers handle possible duplicates.

The outbox did not turn MongoDB and Kafka into one transaction. It removed the need for them to be one.

If you liked this post, you might find these interesting too

Share
Share
On this page
Section
On this page

Continue reading.

Subscribe for more such content

Get the latest in software engineering best practices straight to your inbox. Subscribe now!

Subscribe for more such content

Get the latest in software engineering best practices straight to your inbox. Subscribe now!

Subscribe for more such content

Get the latest in software engineering best practices straight to your inbox. Subscribe now!