Transactional Outbox Pattern¶

Overview¶

VitalBridge uses the Transactional Outbox Pattern to guarantee reliable event delivery between microservices.

The platform relies heavily on Apache Kafka for inter-service communication. To ensure consistency between database state and published events, all services publish events through an outbox table rather than directly to Kafka.

This approach eliminates the need for distributed transactions while ensuring that business events are never lost.

The Problem¶

Consider the process of booking an appointment.

The Appointment Service must:

Save the appointment
Publish an appointment.created event

A naive implementation might look like this:

flowchart LR

    CMD["Create Appointment"]

    DB["Save Appointment"]

    KAFKA["Publish Event"]

    CMD --> DB

    DB --> KAFKA

Hold "Alt" / "Option" to enable pan & zoom

Failure Scenario¶

What happens if the database commit succeeds but Kafka is unavailable?

flowchart LR

    DB["Database Commit ✓"]

    KAFKA["Kafka Publish ✗"]

    DB --> KAFKA

Hold "Alt" / "Option" to enable pan & zoom

Result:

Appointment exists
Event never published
Downstream services never react

This creates inconsistent state across the platform.

Why Not Use Distributed Transactions?¶

Distributed transactions introduce significant complexity.

flowchart LR

    SERVICE["Service"]

    DB["Database"]

    KAFKA["Kafka"]

    SERVICE --> DB

    SERVICE --> KAFKA

Hold "Alt" / "Option" to enable pan & zoom

Distributed transaction protocols:

Increase latency
Reduce scalability
Create operational complexity
Introduce tight coupling

VitalBridge intentionally avoids distributed transactions.

The Solution¶

Instead of publishing directly to Kafka, the service writes an outbox record within the same database transaction.

flowchart LR

    CMD["Business Command"]

    TX["Database Transaction"]

    DATA["Business Record"]

    OUTBOX["Outbox Event"]

    CMD --> TX

    TX --> DATA

    TX --> OUTBOX

Hold "Alt" / "Option" to enable pan & zoom

Both records are committed atomically.

Outbox Table¶

Each service owns its own outbox table.

flowchart TB

    DB[("Service Database")]

    DATA["Business Tables"]

    OUTBOX["Outbox Table"]

    DB --> DATA

    DB --> OUTBOX

Hold "Alt" / "Option" to enable pan & zoom

The outbox table stores pending events waiting to be published.

Typical fields include:

Field	Purpose
id	Event identifier
event_type	Event name
aggregate_type	Business entity
aggregate_id	Entity identifier
payload	Event payload
status	Publication status
created_at	Creation timestamp

Publishing Events¶

A background publisher continuously scans the outbox table.

flowchart LR

    OUTBOX["Outbox Table"]

    PUBLISHER["Outbox Publisher"]

    KAFKA["Apache Kafka"]

    OUTBOX --> PUBLISHER

    PUBLISHER --> KAFKA

Hold "Alt" / "Option" to enable pan & zoom

The publisher:

Reads unpublished events
Publishes events to Kafka
Marks events as published

Complete Event Flow¶

The complete flow looks like this:

sequenceDiagram

    participant Client

    participant Service

    participant DB

    participant Outbox

    participant Publisher

    participant Kafka

    Client->>Service: Execute Command

    Service->>DB: Save Business Data

    Service->>Outbox: Create Outbox Event

    DB-->>Service: Commit Transaction

    Publisher->>Outbox: Read Pending Events

    Publisher->>Kafka: Publish Event

    Publisher->>Outbox: Mark Published

Hold "Alt" / "Option" to enable pan & zoom

This guarantees that an event cannot exist without corresponding business data and business data cannot exist without a corresponding event.

Example: Tenant Creation¶

When a tenant is created:

flowchart LR

    SA["Super Administrator"]

    TR["Tenant Registry Service"]

    DB["Tenant Registry DB"]

    OUTBOX["Outbox Event"]

    KAFKA["Apache Kafka"]

    SA --> TR

    TR --> DB

    TR --> OUTBOX

    OUTBOX --> KAFKA

Hold "Alt" / "Option" to enable pan & zoom

Event:

{
  "event_type": "tenant.created",
  "tenant_id": "123",
  "tenant_name": "Starlight Hospital"
}

The event is guaranteed to be published because it was persisted together with the tenant record.

Retry Mechanism¶

Publishing failures are expected.

flowchart LR

    OUTBOX["Outbox Event"]

    PUB["Publisher"]

    FAIL["Publish Failed"]

    RETRY["Retry"]

    KAFKA["Kafka"]

    OUTBOX --> PUB

    PUB --> FAIL

    FAIL --> RETRY

    RETRY --> KAFKA

Hold "Alt" / "Option" to enable pan & zoom

The event remains in the outbox until publication succeeds.

No events are lost.

Idempotency¶

Event publication may occur more than once.

Consumers must therefore be idempotent.

flowchart LR

    EVENT["appointment.created"]

    CONSUMER["Consumer Service"]

    EVENT --> CONSUMER

    EVENT --> CONSUMER

Hold "Alt" / "Option" to enable pan & zoom

Processing the same event multiple times must not create duplicate business effects.

Examples:

Do not create duplicate appointments
Do not create duplicate notifications
Do not create duplicate audit records

Benefits¶

The Transactional Outbox Pattern provides:

Reliable event delivery
No distributed transactions
Eventual consistency
Retry safety
Improved fault tolerance
Clear service ownership

Architectural Rules¶

VitalBridge enforces the following rules:

Required¶

Every domain event must be persisted in an outbox table
Business data and outbox events must be written in the same transaction
Events must be published asynchronously
Consumers must be idempotent

Forbidden¶

Direct Kafka publication inside business transactions
Distributed transactions
Cross-service database writes
Fire-and-forget event publication

Relationship to Event-Driven Architecture¶

The Transactional Outbox Pattern is the foundation that enables VitalBridge's event-driven architecture.

flowchart TB

    COMMAND["Business Command"]

    OUTBOX["Transactional Outbox"]

    KAFKA["Apache Kafka"]

    CONSUMER["Consumer Service"]

    COMMAND --> OUTBOX

    OUTBOX --> KAFKA

    KAFKA --> CONSUMER

Hold "Alt" / "Option" to enable pan & zoom

Without the outbox pattern, event delivery could not be guaranteed.

With the outbox pattern, VitalBridge achieves reliable, scalable, event-driven communication across all services.