Transactional Inbox Pattern for Reliable Messaging

The transactional inbox pattern ensures reliable message processing by storing incoming messages in a persistent inbox before processing them. This guarantees at-least-once processing without duplicating work during retries.

The Problem

In distributed systems, messages can be delivered multiple times. Network failures, consumer crashes, and broker retries all cause duplicate deliveries without the producer knowing. Without the inbox pattern, duplicate messages cause duplicate side effects—charging a customer twice, creating duplicate records.

How It Works

When a message arrives, the consumer first checks if it exists in the inbox by its unique message ID. If it is new, the consumer stores the message in the inbox table and then processes it. If the message ID already exists, the consumer skips processing (idempotent consumption).

The inbox is typically a database table with a unique constraint on message ID. The consumer uses a database transaction to atomically check for the message and process it. If the consumer crashes after processing but before acknowledging the message, the next delivery finds the message already in the inbox and skips processing.

Inbox Table Schema

A minimal inbox table contains: message_id (UUID, primary key), message_type, payload, status (received, processing, completed), created_at, processed_at. Additional columns can store retry count, error messages, and correlation IDs.

Relationship to Transactional Outbox

The transactional inbox and outbox are complementary. The outbox pattern ensures outgoing messages are reliably delivered. The inbox pattern ensures incoming messages are reliably processed. Together, they provide end-to-end reliability in asynchronous communication.

Implementation Considerations

Periodically clean up processed inbox records to prevent table growth. Archive after 7-30 days depending on reprocessing requirements. Monitor inbox table size and processing latency. Set up alerts for stuck messages (status=processing for too long).

Idempotency Keys

For REST API consumers, idempotency keys serve a similar purpose. The client generates a unique key for each request. The server stores processed keys to detect and reject duplicates. Stripe's idempotency key implementation is a well-known example of this pattern.