Idempotency: Concepts, Trade-offs & Failure Modes

Idempotency ensures that performing an operation multiple times has the same effect as performing it once. Critical for handling retries and failures in distributed systems.

What is Idempotency?

An operation is idempotent if:

• Performing it once has the same effect as performing it N times
• Safe to retry without side effects
• Duplicate requests don't cause problems

Examples:

• Idempotent: GET, PUT, DELETE (usually)
• Not idempotent: POST (usually creates new resource)

Why Idempotency Matters

Network failures: Requests can be retried, causing duplicates.

Timeouts: Client doesn't know if request succeeded, retries.

Race conditions: Multiple clients send same request.

At-least-once delivery: Message queues may deliver messages multiple times.

Implementing Idempotency

Idempotency Keys

Client provides unique key, server tracks processed requests.

1class

Natural Idempotency

Design operations to be naturally idempotent.

1// Idempotent: Set value (not increment)
2async setBalance(accountId: string, balance: number): Promise<void> {
3  await db.update('accounts', { balance }, { id: accountId });
4}
5
6// Not idempotent: Increment
7async incrementBalance(accountId: string, amount: number): Promise<void> {
8  await db.query('UPDATE accounts SET balance = balance + $1 WHERE id = $2', [amount, accountId])

Examples

Payment Processing

1class PaymentService {
2  async processPayment(
3    paymentId: string,
4    amount: number,
5    idempotencyKey: string
6  ): Promise<Payment> {
7    // Check if payment already processed
8    const existing = await this.db.getPaymentByKey(idempotencyKey);
9    if (existing) {
10      return existing; // Return same payment
11    }
12
13    // Process payment
14    const payment = await this.db.createPayment({
15      id: paymentId

Distributed Lock with Idempotency

1class IdempotentLock {
2  async acquireLock(
3    resourceId: string,
4    idempotencyKey: string,
5    ttl: number
6  ): Promise<boolean> {
7    // Try to acquire lock with idempotency key
8    const acquired = await this.redis.set(
9      `lock:${resourceId}`,
10      idempotencyKey,
11      'EX', ttl,
12      'NX' // Only set if not exists
13    );
14
15    if (acquired) {

Common Pitfalls

• Not using idempotency keys: Relying on natural idempotency only works for some operations
• Short TTL: Idempotency keys expire too soon, allowing duplicates. Fix: Use appropriate TTL based on operation
• Not storing response: Must return same response for same key
• Race conditions: Multiple requests with same key processed concurrently. Fix: Use database constraints or locks
• Not cleaning up: Idempotency keys accumulate. Fix: Use TTL, periodic cleanup
• Key collision: Different operations use same key. Fix: Include operation type in key

Interview Questions

Beginner

Q: What is idempotency and why is it important?

A: Idempotency means performing an operation multiple times has the same effect as performing it once.

Why important:

• Retries: Network failures cause retries, must handle safely
• Timeouts: Client doesn't know if request succeeded, retries
• Duplicates: Message queues may deliver messages multiple times
• Race conditions: Multiple clients send same request

Example: Payment processing - charging a card twice for the same order would be bad. Idempotency ensures duplicate requests don't cause duplicate charges.

Intermediate

Q: How do you implement idempotency for a payment API? What are the challenges?

A:

Implementation:

1async processPayment(idempotencyKey: string, amount: number): Promise<Payment> {
2  // Check if already processed
3  const existing = await db.getPaymentByKey(idempotencyKey);
4  if (existing) {
5    return existing; // Return same payment (idempotent)
6  }
7
8  // Process payment
9  const payment = await this.chargeCard(amount, idempotencyKey);
10  await db.savePayment(payment);
11  return payment;
12}

Challenges:

1. Race conditions: Two requests with same key arrive simultaneously

   • Fix: Use database unique constraint on idempotency key
   • Or use distributed lock

2. Storage: Must store idempotency keys and responses

   • Fix: Use Redis with TTL, or database table

3. TTL: How long to keep idempotency keys?

   • Fix: Based on operation (payments: days/weeks, API calls: hours)

4. Key generation: Client must generate unique keys

   • Fix: Use UUIDs, or client ID + timestamp + operation

Senior

Q: Design an idempotent distributed system for processing orders. Orders involve inventory, payment, and shipping. How do you ensure idempotency across all services and handle partial failures?

A:

Architecture:

• Idempotency keys: Each order has unique idempotency key
• Distributed idempotency: Each service checks key before processing
• Saga with idempotency: Each saga step is idempotent

Design:

1class IdempotentOrderSystem {
2  async processOrder(orderId: string, idempotencyKey: string): Promise<void> {
3    // Check if order already processed
4    const existing = await this.getOrder(orderId);
5    if (existing && existing.status === 'completed') {
6      return; // Already processed
7    }
8
9    // Process with idempotent saga
10    await this.idempotentSaga.execute(orderId, idempotencyKey);
11  }
12}

Handling Partial Failures:

• Checkpoint state: Store state after each step
• Resume on retry: Check state, resume from last completed step
• Idempotent compensation: Compensations must also be idempotent

Distributed Idempotency:

• Shared storage: Use Redis or database for idempotency keys
• Consistent hashing: Route same key to same node for caching
• Replication: Replicate idempotency state for availability

• Idempotency is critical for handling retries and failures safely

• Idempotency keys: Client provides unique key, server tracks processed requests

• Natural idempotency: Design operations to be idempotent (SET not INCREMENT)

• Store responses: Must return same response for same idempotency key

• Handle race conditions: Use database constraints or locks

• TTL management: Clean up old idempotency keys appropriately

• Distributed systems: Each service must check idempotency independently

• Saga steps: Each step in a saga should be idempotent

• Distributed Transactions - Maintaining ACID properties across nodes

• Fault Tolerance - Handling failures in distributed systems

• Two-Phase Commit (2PC) - Transaction protocols that require idempotency

• Heartbeats & Health Checks - Monitoring system health

• Replication Lag - Handling eventual consistency