Idempotency and Exactly-Once: Making Retries Safe

Somewhere in every serious backend there is a piece of work that must not happen twice. A payment that must not double-charge. An email that must not send five times. An inventory decrement that must not oversell. And in every distributed system, the machinery underneath — networks, retries, message queues — guarantees that some operations will be attempted more than once. Idempotency is how you reconcile those two facts. It is the discipline that lets an operation be retried safely, and it is what "exactly-once" actually means in practice.

Why duplicates are inevitable

The instinct is to prevent duplicates from ever happening. In a distributed system, you cannot, and the reason is fundamental: you can never be sure a request succeeded.

Picture a client calling your payment endpoint. The server charges the card successfully, then the network drops the response on its way back. From the client's side, the request looks failed — no response arrived — so it retries, exactly as it should. Now the charge happens twice. Nothing here is a bug. The server did its job, the client did the sensible thing, and the network did what networks do. The ambiguity is unavoidable: a missing response could mean the operation failed, or it could mean it succeeded and the acknowledgment was lost. The client cannot tell the difference, so it must retry, and retries produce duplicates.

The same logic runs through at-least-once message delivery: a worker might process a message and crash before acknowledging it, so the queue redelivers it. Duplicates are not an edge case you can engineer away. They are a structural property of any system where messages can be lost. So instead of preventing duplicates, you make them harmless.

What idempotency means

An operation is idempotent if performing it multiple times has the same effect as performing it once. The word comes from mathematics, but the practical definition is simpler: running it twice leaves the system in the same state as running it once.

Some operations are naturally idempotent. Setting a value — status = 'shipped' — is idempotent, because setting it again changes nothing. Deleting a specific record is idempotent; once it is gone, deleting it again is a no-op. These you get for free.

The dangerous operations are the ones that are not naturally idempotent, and they are exactly the important ones. "Charge the card fifty dollars" is not idempotent — do it twice and you charged a hundred. "Increment the balance" is not idempotent. "Append a row" is not idempotent. For these, you have to add idempotency deliberately, because the operation itself will not save you.

The idempotency key pattern

The standard tool is an idempotency key. The client generates a unique key for the operation — a UUID it creates once and reuses across retries — and sends it with the request. The server uses that key to recognize retries and refuse to do the work twice.

The flow on the server:

1. Receive the request with its idempotency key.
2. Check whether you have already processed that key. If yes, return the stored result of the original operation without doing the work again.
3. If no, perform the operation and record the key along with its result, atomically.
4. Return the result.

The critical detail is step three: recording the key and performing the operation have to be atomic, or the pattern has a hole. If you charge the card and then separately try to record the key, a crash in between leaves you having charged without recording, and the retry charges again. The clean way is to make the key insertion part of the same database transaction as the work, or to use a unique constraint on the key so a duplicate insert fails and signals "already done." A unique index on the idempotency key turns the database itself into the deduplicator, which is about as reliable as it gets.

"Exactly-once" is really "effectively-once"

People ask for exactly-once delivery, and it is worth being clear about why they cannot have it and what they get instead. True exactly-once delivery at the transport level is impossible for the same reason duplicates are inevitable: the sender can never be certain a message arrived, so it must either risk losing it (at-most-once) or risk sending it again (at-least-once). There is no third option at the transport layer.

What you can achieve is exactly-once processing, and the recipe is always the same: at-least-once delivery plus idempotent handlers. Let the transport deliver the message one or more times — that part is easy and reliable — and make your processing idempotent so that duplicates collapse into a single effect. The message might arrive three times; the charge happens once because the handler recognizes the repeats. The result is indistinguishable from exactly-once, which is all anyone actually wanted. This is why "make your consumers idempotent" is the first advice anyone gives about queues.

The outbox pattern for the dual-write problem

Idempotency has a close cousin of a problem: the dual write. Often an operation needs to both change your database and send a message — save an order and publish an "order placed" event. If you write to the database and then publish to the queue as two separate steps, a crash in between leaves them inconsistent: the order exists but no event was sent, or vice versa. There is no way to make two separate systems commit atomically.

The outbox pattern solves this elegantly. Instead of publishing to the queue directly, you write the message into an "outbox" table in the same database transaction as your business change. Now the order and the intent to publish its event commit together, atomically — either both happen or neither does. A separate process then reads the outbox and publishes the messages to the queue, marking them sent. If it crashes mid-way, it just re-reads the unsent rows and publishes again, which is safe precisely because your consumers are idempotent. The outbox turns an impossible atomic dual-write into a reliable single-database write plus an idempotent relay.

Where this shows up

The clearest example is payments. A well-built Stripe integration leans on idempotency at every turn: Stripe accepts an idempotency key on charge requests so your retries do not double-charge, and its webhooks are delivered at-least-once, so your webhook handler must be idempotent or a redelivered event will process a payment twice. Payments are simply the domain where the cost of getting this wrong is most obvious, but the same reasoning applies anywhere a repeated operation would cause harm.

The mental shift worth keeping: in distributed systems, stop trying to guarantee something happens exactly once at the point of sending, and start guaranteeing it has the same effect no matter how many times it arrives. That reframing — from preventing duplicates to neutralizing them — is one of the most practically important ideas in the whole system design toolkit.