If you have ever shipped a distributed system that “usually works,” you already know the villain: the retry. The client times out, the queue redelivers, a worker crashes after performing the side effect but before acknowledging it, and suddenly your “create order” logic runs twice. This is exactly the class of failure that idempotent operations are meant to neutralize. Nothing about this scenario is exotic, it is the default behavior of networks and distributed fleets.
Idempotency is the discipline of making those retries boring. Plainly: idempotent operations are when running it multiple times produces the same externally visible result as running it once. The definition sounds simple. The implementation is where teams struggle, because your system has to remember what it already did, even when the original caller never saw success.
While researching this topic, a consistent theme emerged across experienced system builders: retries, timeouts, and idempotence are not edge cases. They are the foundation of reliability at scale. If you want systems that survive partial failure, you design for duplication from the start.
The synthesis is blunt: you do not “add idempotency later.” You design it the same way you design authentication or authorization, because the retry is inevitable.
Start with the failure model you actually have
Most distributed workloads quietly operate with at least once execution, even if no one planned it that way. Message queues redeliver. HTTP clients retry. Background jobs restart. Humans click “Submit” again.
So the real question is not whether duplicates will happen. It is where you will absorb them, and what state you will use to prove that the work already happened.
A quick worked example shows why this matters. Suppose you process 1,000,000 “create payment” requests per day. Assume 0.2% of requests hit a timeout or connection drop where the client cannot tell if the server succeeded. That is 2,000 ambiguous outcomes. If even 70% of those actually succeeded on the server but the response was lost, retries will cause 1,400 potential duplicate charges per day.
With an average payment of $50, that is $70,000 per day in accidental duplicates, before you account for support costs, refunds, or reputational damage. Your exact numbers will differ, but the shape of the risk does not.
Pick an idempotency strategy that matches your workload
There are two broad families of idempotency strategies.
First is natural idempotency, where the request already contains a stable identifier. Operations like “set user 123’s email to X” are inherently safe to retry, because the desired state is explicit.
Second is deduplication based idempotency, where you attach a key to the intent and ensure the side effect executes at most once per key. This is the standard pattern for “create,” “charge,” and “trigger” style operations.
A quick comparison helps clarify the tradeoffs.
| Approach | Best for | What you store | Failure to watch |
|---|---|---|---|
| Natural key in request | Set state operations | Resource row or version | Lost updates without concurrency control |
| Client idempotency key | Create or charge actions | Key to result or status | Key reuse across different intents |
| Payload hash | Internal jobs | Hash to result | Same payload, different intent |
| Conditional update | Competing writers | Version or token | Clients ignoring preconditions |
There is no universally correct choice. The right strategy depends on how explicit the intent is and how costly duplication would be.
Build the operation around a single source of truth for “already done”
Idempotency is not a header. It is a state machine.
You need one authoritative record that answers a single question: has intent X already been executed, and if so, what was the result?
The hardest part is handling the awkward middle state, where the system started processing and then crashed.
A reliable pattern looks like this:
- Write an idempotency record first, including the key, request metadata, and a status of IN_PROGRESS.
- Execute the side effect.
- Commit the final result, updating the status to COMPLETED and storing the outcome.
When another worker sees the same key:
- If the status is COMPLETED, return the stored result.
- If the status is IN_PROGRESS, either wait, return a “still processing” response, or direct the caller to a status endpoint.
This is where many teams discover, too late, that they needed an explicit “get operation status” API from day one.
How to implement idempotency in practice
Step 1: Define the intent identity and make it unambiguous
For client facing “create” operations, use a client generated idempotency key that represents the business intent, not the network attempt.
A good key is:
- Unique per business action
- Scoped to the relevant account or resource
- Long enough to avoid collisions
- Logged and traceable across services
If clients cannot reliably generate keys, generate them server side and return an operation handle that clients can poll. This shifts retries from “do it again” to “check what happened.”
Step 2: Put the dedupe check and side effect in the same atomic boundary
The cleanest implementation uses a single database transaction to write both the idempotency record and the business change.
If that is not possible, use an outbox pattern where the business state change and an event record are committed together, and downstream consumers deduplicate by event ID.
What you want to avoid is the worst case split brain: remembering that you started, without knowing whether the side effect actually happened.
Step 3: Treat “in progress” as a normal state
Retries that arrive mid flight are not exceptional. They are expected.
Responding with “accepted, still processing” and a stable operation ID is often better than pushing clients toward aggressive retries. It also prevents duplicate fan out under load.
Step 4: Decide retention and replay behavior upfront
Your idempotency store needs a retention policy. Keep records long enough to cover realistic retry windows, delayed queue delivery, and human retry behavior.
Also decide what you will replay:
- The exact original response
- Or a normalized result, such as a resource identifier
The key requirement is determinism. Clients will build assumptions on top of whatever behavior you choose.
The edge cases that usually bite teams
Many systems are technically idempotent but still incorrect because they ignore ordering.
Watch carefully for:
- Non-commutative sequences: “add $10” then “remove $10” is unsafe under reordering.
- External side effects: emails, shipments, and third party calls need their own deduplication.
- Partial failures: the side effect succeeded but state persistence failed.
- Key misuse: the same idempotency key reused for different intents.
Idempotency only works if the identity truly represents the intent.
FAQ
Is idempotent the same as exactly once?
Not really. Most real systems approximate “exactly once” by combining at least once execution with idempotent handling.
Are HTTP methods already idempotent?
Some are defined that way conceptually, but your implementation still has to enforce the behavior.
Where should idempotency live, in the gateway or the service?
At the service that commits the side effect. That is the only place that can prove the work already happened.
What about async workers and queues?
Same rule. Deduplicate on message or intent ID, and persist the “processed” marker durably.
Honest Takeaway
If there is one idea to carry forward, it is this: idempotency is state, not a retry policy. Retries just make the underlying duplication visible faster.
Once you define clear intent identities and persist “already done” decisions, distributed systems become calmer. Timeouts stop being emergencies. Redeliveries stop being incidents. And operators stop doing painful math about how many customers were charged twice overnight.
That calm is not accidental. It is designed.
Kirstie a technology news reporter at DevX. She reports on emerging technologies and startups waiting to skyrocket.
