Idempotency: Concepts, Trade-offs & Failure Modes

Idempotency

Why Engineers Care About This

Idempotency is how you make retries safe. In a distributed system, the network lies: a client may never see the success response even when the server already charged the card, reserved inventory, or wrote the row. The client retries. Without idempotency, that retry is a second charge, a second reservation, or a second email.

Timeouts, load balancers, mobile apps with flaky radios, and at-least-once queues all create the same shape of problem: at least one delivery of the same intent. Exactly-once delivery across machines is usually a myth; at-least-once plus idempotent handlers is the pattern that ships.

In interviews, when someone asks "How do you handle retries?" or "What if the payment API times out?", they're really asking: "Do you understand idempotency—keys, natural designs, races, and what breaks when the key store fails?" Most engineers say "add an Idempotency-Key header" and stop. Senior answers name who owns the key, where state lives, what concurrent duplicates do, and how long records survive.

For retry storms and backoff, see Retries & Backoff. For at-least-once consumers, see Message Queues. For a concrete double-charge design challenge, see Payment Double-Charge.

Core Intuitions You Must Build

• Idempotency is about effect, not about HTTP verbs. Performing the operation once or N times must leave the system in the same state (and usually return the same result). GET is usually safe; PUT/DELETE are often idempotent by design; POST is often not—but a carefully designed POST can be idempotent with a key. Verb folklore is not a design.

• Natural idempotency beats keys when you can get it. Prefer SET / upsert / "create if absent with a stable id" over INCREMENT / "append forever." When the write itself converges, retries are free. When the write is "charge $50" or "send email," you need an external record of the intent.

• Idempotency keys record intent, not just "we saw this header." The client (or gateway) sends a unique key for one logical attempt. The server stores that the key was accepted and what result to return. A later retry with the same key must not re-execute side effects—and should replay the same success or failure outcome when the operation finished.

• Check-then-act without a unique constraint races. Two requests with the same key can both miss the store and both charge. Production designs use a unique index on the key, a conditional INSERT, or a lock—then execute once.

• In-process maps are not an idempotency store. Behind multiple pods, each process has its own memory. Shared Redis or a database table is the consistency boundary—same lesson as local vs shared cache.

• TTL is a correctness dial. Expire too soon and a late retry double-applies. Keep forever and storage grows. Payments often need days or weeks; short-lived internal RPCs may need hours. Pair TTL with reconciliation for money paths.

• Every saga step needs its own idempotency. A shared order key is not enough if inventory, payment, and shipping each can be retried independently after a partial failure. Use step keys (orderKey:charge-payment) and idempotent compensations.

Pattern comparison at a glance

Four approaches differ mainly in where safety comes from and what fails under concurrency:

Approach	Where safety lives	Concurrent same intent	Pick when
Natural (SET / upsert)	Operation semantics	Usually safe	State can be overwritten to a target value
Idempotency key + store	Shared Redis/DB record	Needs unique constraint or lock	Charges, emails, one-shot side effects
Stable resource id	Client-supplied id as primary key	Unique PK rejects duplicates	Create-once resources (POST /orders with client id)
Outbox / dedupe table	Consumer offset or message id	Unique message id	Queue consumers under at-least-once delivery

Client                API / Worker              Side effect           Key store
  |  intent + key  -----> |                        |                     |
  |                       |-- reserve key -------->|                     |
  |                       |   (unique / NX)        |                     |
  |                       |-- execute once ------->|                     |
  |                       |-- save result -------->|                     |
  |  same key again ----->|-- return stored result (no second charge) -->|

Subtopics (Taught Through Real Scenarios)

Natural Idempotency vs Key-Based Idempotency

What people usually get wrong:

Engineers treat every write as "needs a UUID header." Or they assume PUT is always safe while a POST /charge with a key is "the only real idempotency." Natural designs (set balance to X, upsert profile by user id) remove a whole class of bugs. Key-based designs are for operations that are not naturally replay-safe.

How this breaks systems in the real world:

A billing job used INCREMENT balance on every "add credit" event. At-least-once delivery applied the same event twice—balances drifted. Switching to "apply credit event id E once" (key in applied_events) fixed money. Separately, a profile update API used PUT with the full profile body—retries were already safe without keys. But the real lesson is: pick natural idempotency when the domain allows it; add keys only for irreversible or additive side effects.

What interviewers are really listening for:

They want you to name both styles and when each fits. Junior engineers say "always use Idempotency-Key." Senior engineers say "SET and upsert first; keys for charges, sends, and increments—and the key must be stored with the outcome."

Idempotency Keys and the Client Contract

What people usually get wrong:

Teams add an Idempotency-Key header but do not define scope (per user? per endpoint?), who generates it, or what the server returns on reuse. Some clients mint a new key on every retry (defeats the purpose). Others reuse one key across different payloads (should be rejected as a conflict).

How this breaks systems in the real world:

A mobile checkout generated a new key whenever the user tapped Retry after a spinner timeout. The first request had already charged the card; the second key charged again. Support saw "I only pressed pay once" tickets. The fix? Persist the key with the checkout session before the first attempt, retry with the same key, and return the stored payment result. Document that changing the body with the same key returns 409. For HTTP API conventions, see API Design. But the real lesson is: the key is part of the product contract—generation, reuse, and conflict rules belong in the client SDK docs.

What interviewers are really listening for:

They want a clear client story: same logical attempt → same key; different attempt → new key; same key + different body → conflict. Junior engineers say "UUID in a header." Senior engineers say who creates it, how long the client must keep it, and what status codes mean on reuse.

Race Conditions on the Same Key

What people usually get wrong:

Sample code does if (!store.has(key)) { doWork(); store.set(key); } in memory. Two concurrent requests both pass the check. Unique constraints and atomic "claim key" operations exist for this reason.

How this breaks systems in the real world:

A payment API checked Redis for the key, missed, charged the PSP, then wrote Redis. Under timeout-driven retries, two workers both missed and both charged—classic double-capture. The fix was a unique row (or Redis SET key NX) before calling the PSP, a processing state for in-flight work, and only then execute. Retries that arrive while processing wait or return 409/425 depending on product rules. Practice this shape in Payment Double-Charge and Double Submit. But the real lesson is: claim the key atomically before the side effect—never after.

What interviewers are really listening for:

They want the race called out and a concrete fix (unique index, INSERT … ON CONFLICT, Redis NX + lock). Junior engineers draw a happy-path flowchart. Senior engineers talk about the in-flight window and what the second request sees.

TTL, Storage, and Late Retries

What people usually get wrong:

Teams copy a one-hour TTL from a blog post. A delayed mobile retry or a nightly reconciliation arrives after expiry and re-applies the write. Or they never expire keys and Redis becomes an unbounded ledger.

How this breaks systems in the real world:

An API kept idempotency records for one hour. A partner's batch client retried failed posts the next morning with the original keys—keys were gone, posts duplicated. Extending payment and partner TTLs to fourteen days, plus a business unique constraint on (partner_id, external_ref), stopped the duplicates. Short TTLs stayed for internal fan-out RPCs. But the real lesson is: TTL must exceed the longest realistic retry window for that operation class—and money paths need a second line of defense (business unique keys, reconciliation).

What interviewers are really listening for:

They want a reasoned TTL, not a magic number. Junior engineers say "Redis TTL one day." Senior engineers say "match retry and support windows; use domain uniqueness for anything that must never double."

When the Idempotency Store Is Unavailable

What people usually get wrong:

The key store is treated as always up. When Redis blips, the API either hangs or skips the check and processes "best effort"—which means duplicate risk exactly when the system is already unhealthy.

How this breaks systems in the real world:

During a Redis failover, a checkout service timed out on key lookup and fell through to "process without key" for availability. Duplicate charges spiked. The safer policy for payments was fail-closed: return 503 and ask the client to retry later with the same key once the store is healthy. For non-critical analytics ingest, fail-open with best-effort dedupe was acceptable. But the real lesson is: decide fail-open vs fail-closed per operation class—money usually fails closed on the idempotency dependency.

What interviewers are really listening for:

They want an explicit degrade path. Junior engineers say "Redis is HA." Senior engineers say what happens in the first thirty seconds of a store outage and whether duplicates or rejects are the lesser harm.

Multi-Service and Saga Steps

What people usually get wrong:

One order-level key is checked only at the API gateway. Downstream inventory, payment, and shipping services each retry independently without step-level dedupe. Compensations (refund, release stock) are not idempotent either—so recovery itself double-applies.

How this breaks systems in the real world:

An order saga reserved inventory, charged payment, then timed out before shipping. On resume, payment ran again because the payment service keyed only on a new request id from the orchestrator. The fix used stable step keys derived from the order idempotency key, checkpointed step state, and idempotent compensations. See also Distributed Transactions and Fault Tolerance. But the real lesson is: distributed workflows need per-step idempotency and idempotent undo—not a single header at the edge.

What interviewers are really listening for:

They want step keys, checkpoints, and compensation safety. Junior engineers say "saga with retries." Senior engineers say how each service recognizes "I already did this step" after a crash mid-flight.

---

Interview questions to practice

• A client times out on POST /payments and retries with the same body—what must be true for the user to be charged once?
• You chose an Idempotency-Key header—who generates it, and what do you return if the same key arrives with a different amount?
• Two requests with the same key hit two pods at once—where does the unique constraint live, and what does the loser do?
• Redis that stores keys is down for thirty seconds—fail open or fail closed for checkout vs for analytics ingest?
• Your key TTL is one hour; a partner retries tomorrow—what else prevents a duplicate write?
• Walk an order saga: inventory, payment, shipping—what are the step keys, and how do compensations stay idempotent?
• When is natural upsert enough, and when do you still need a key?

---

FAQs

Q: Are PUT and DELETE always idempotent?

A: Often by HTTP semantics—but only if your implementation matches. A DELETE that soft-deletes once is fine; a DELETE that appends an audit "deleted" row per call is not. Always reason about side effects, not only the verb.

Q: What's the difference between natural idempotency and an idempotency key?

A: Natural means replaying the write converges (SET, upsert). A key means you record that a specific intent already ran and skip or replay the stored result. Use natural when you can; keys when the operation is additive or externally visible once.

Q: Do I need to store the response, or just "mark key used"?

A: Store enough to return a stable outcome—full response body, resource id, or status. Clients retrying after a timeout need the same success payload, not a vague "already processed" with no id.

Q: How long should idempotency keys live?

A: Longer than the longest retry window you support for that API. Internal RPCs: hours. Partner and payment APIs: often days to weeks, plus a business unique constraint for money.

Q: Is exactly-once delivery the goal?

A: Across unreliable networks, end-to-end exactly-once is rarely free. Design for at-least-once delivery and idempotent handlers (and optionally an outbox). See Message Queues.

Q: Where should I practice this under incident pressure?

A: Payment Double-Charge and Double Submit force the timeout → retry → duplicate path. Pair with Retries & Backoff for client behavior.

---