Idempotency

Idempotency ensures that performing an operation multiple times has the same effect as performing it once. Critical for handling retries and failures in distributed systems.

---

What is Idempotency?

An operation is idempotent if:

• Performing it once has the same effect as performing it N times
• Safe to retry without side effects
• Duplicate requests don't cause problems

Examples:

• Idempotent: GET, PUT, DELETE (usually)
• Not idempotent: POST (usually creates new resource)

---

Why Idempotency Matters

Network failures: Requests can be retried, causing duplicates.

Timeouts: Client doesn't know if request succeeded, retries.

Race conditions: Multiple clients send same request.

At-least-once delivery: Message queues may deliver messages multiple times.

---

Implementing Idempotency

Idempotency Keys

Client provides unique key, server tracks processed requests.

class IdempotentService {
  private processed: Map<string, Response> = new Map();
  private ttl: number = 3600000; // 1 hour

  async processRequest(request: Request): Promise<Response> {
    const idempotencyKey = request.headers['idempotency-key'];
    
    if (!idempotencyKey) {
      throw new Error('Idempotency key required');
    }

    // Check if already processed
    const cached = this.processed.get(idempotencyKey);
    if (cached) {
      return cached; // Return same response
    }

    // Process request
    const response = await this.executeRequest(request);
    
    // Cache response
    this.processed.set(idempotencyKey, response);
    
    // Cleanup after TTL
    setTimeout(() => {
      this.processed.delete(idempotencyKey);
    }, this.ttl);

    return response;
  }
}

Natural Idempotency

Design operations to be naturally idempotent.

// Idempotent: Set value (not increment)
async setBalance(accountId: string, balance: number): Promise<void> {
  await db.update('accounts', { balance }, { id: accountId });
}

// Not idempotent: Increment
async incrementBalance(accountId: string, amount: number): Promise<void> {
  await db.query('UPDATE accounts SET balance = balance + $1 WHERE id = $2', [amount, accountId]);
}

// Make increment idempotent with idempotency key
async incrementBalanceIdempotent(
  accountId: string, 
  amount: number,
  idempotencyKey: string
): Promise<void> {
  // Check if this increment was already applied
  const applied = await db.query(
    'SELECT * FROM applied_operations WHERE idempotency_key = $1',
    [idempotencyKey]
  );

  if (applied.length > 0) {
    return; // Already applied
  }

  // Apply increment
  await db.query('UPDATE accounts SET balance = balance + $1 WHERE id = $2', [amount, accountId]);
  
  // Record that operation was applied
  await db.query(
    'INSERT INTO applied_operations (idempotency_key, account_id, amount) VALUES ($1, $2, $3)',
    [idempotencyKey, accountId, amount]
  );
}

---

Examples

Payment Processing

class PaymentService {
  async processPayment(
    paymentId: string,
    amount: number,
    idempotencyKey: string
  ): Promise<Payment> {
    // Check if payment already processed
    const existing = await this.db.getPaymentByKey(idempotencyKey);
    if (existing) {
      return existing; // Return same payment
    }

    // Process payment
    const payment = await this.db.createPayment({
      id: paymentId,
      amount,
      idempotencyKey,
      status: 'processing'
    });

    try {
      await this.chargeCard(payment);
      payment.status = 'completed';
    } catch (error) {
      payment.status = 'failed';
    }

    await this.db.updatePayment(payment);
    return payment;
  }
}

Distributed Lock with Idempotency

class IdempotentLock {
  async acquireLock(
    resourceId: string,
    idempotencyKey: string,
    ttl: number
  ): Promise<boolean> {
    // Try to acquire lock with idempotency key
    const acquired = await this.redis.set(
      `lock:${resourceId}`,
      idempotencyKey,
      'EX', ttl,
      'NX' // Only set if not exists
    );

    if (acquired) {
      return true; // Lock acquired
    }

    // Check if we already own the lock
    const currentOwner = await this.redis.get(`lock:${resourceId}`);
    if (currentOwner === idempotencyKey) {
      return true; // We already own it (idempotent)
    }

    return false; // Locked by someone else
  }
}

---

Common Pitfalls

• Not using idempotency keys: Relying on natural idempotency only works for some operations
• Short TTL: Idempotency keys expire too soon, allowing duplicates. Fix: Use appropriate TTL based on operation
• Not storing response: Must return same response for same key
• Race conditions: Multiple requests with same key processed concurrently. Fix: Use database constraints or locks
• Not cleaning up: Idempotency keys accumulate. Fix: Use TTL, periodic cleanup
• Key collision: Different operations use same key. Fix: Include operation type in key

---

Interview Questions

Beginner

Q: What is idempotency and why is it important?

A: Idempotency means performing an operation multiple times has the same effect as performing it once.

Why important:

• Retries: Network failures cause retries, must handle safely
• Timeouts: Client doesn't know if request succeeded, retries
• Duplicates: Message queues may deliver messages multiple times
• Race conditions: Multiple clients send same request

Example: Payment processing - charging a card twice for the same order would be bad. Idempotency ensures duplicate requests don't cause duplicate charges.

---

Intermediate

Q: How do you implement idempotency for a payment API? What are the challenges?

A:

Implementation:

async processPayment(idempotencyKey: string, amount: number): Promise<Payment> {
  // Check if already processed
  const existing = await db.getPaymentByKey(idempotencyKey);
  if (existing) {
    return existing; // Return same payment (idempotent)
  }

  // Process payment
  const payment = await this.chargeCard(amount, idempotencyKey);
  await db.savePayment(payment);
  return payment;
}

Challenges:

1. Race conditions: Two requests with same key arrive simultaneously

   • Fix: Use database unique constraint on idempotency key
   • Or use distributed lock

2. Storage: Must store idempotency keys and responses

   • Fix: Use Redis with TTL, or database table

3. TTL: How long to keep idempotency keys?

   • Fix: Based on operation (payments: days/weeks, API calls: hours)

4. Key generation: Client must generate unique keys

   • Fix: Use UUIDs, or client ID + timestamp + operation

---

Senior

Q: Design an idempotent distributed system for processing orders. Orders involve inventory, payment, and shipping. How do you ensure idempotency across all services and handle partial failures?

A:

Architecture:

• Idempotency keys: Each order has unique idempotency key
• Distributed idempotency: Each service checks key before processing
• Saga with idempotency: Each saga step is idempotent

Design:

class IdempotentOrderSystem {
  async processOrder(orderId: string, idempotencyKey: string): Promise<void> {
    // Check if order already processed
    const existing = await this.getOrder(orderId);
    if (existing && existing.status === 'completed') {
      return; // Already processed
    }

    // Process with idempotent saga
    await this.idempotentSaga.execute(orderId, idempotencyKey);
  }
}

class IdempotentSaga {
  async execute(orderId: string, idempotencyKey: string): Promise<void> {
    const steps = [
      {
        name: 'reserve-inventory',
        execute: () => this.reserveInventory(orderId, idempotencyKey),
        compensate: () => this.releaseInventory(orderId, idempotencyKey)
      },
      {
        name: 'charge-payment',
        execute: () => this.chargePayment(orderId, idempotencyKey),
        compensate: () => this.refundPayment(orderId, idempotencyKey)
      },
      {
        name: 'create-shipment',
        execute: () => this.createShipment(orderId, idempotencyKey),
        compensate: () => this.cancelShipment(orderId, idempotencyKey)
      }
    ];

    for (const step of steps) {
      await this.executeStepIdempotently(step, orderId, idempotencyKey);
    }
  }

  async executeStepIdempotently(
    step: Step,
    orderId: string,
    idempotencyKey: string
  ): Promise<void> {
    // Create step-specific idempotency key
    const stepKey = `${idempotencyKey}:${step.name}`;

    // Check if step already executed
    const state = await this.getStepState(orderId, step.name);
    if (state === 'completed') {
      return; // Already done
    }

    try {
      await step.execute();
      await this.saveStepState(orderId, step.name, 'completed', stepKey);
    } catch (error) {
      await this.saveStepState(orderId, step.name, 'failed', stepKey);
      throw error;
    }
  }
}

// Each service implements idempotency
class InventoryService {
  async reserveInventory(
    orderId: string,
    idempotencyKey: string
  ): Promise<void> {
    const stepKey = `${idempotencyKey}:reserve-inventory`;

    // Check if already reserved
    const reservation = await db.getReservation(stepKey);
    if (reservation) {
      return; // Already reserved (idempotent)
    }

    // Reserve inventory
    await db.reserveInventory(orderId, stepKey);
  }
}

Handling Partial Failures:

• Checkpoint state: Store state after each step
• Resume on retry: Check state, resume from last completed step
• Idempotent compensation: Compensations must also be idempotent

Distributed Idempotency:

• Shared storage: Use Redis or database for idempotency keys
• Consistent hashing: Route same key to same node for caching
• Replication: Replicate idempotency state for availability

---

Key Takeaways

• Idempotency is critical for handling retries and failures safely
• Idempotency keys: Client provides unique key, server tracks processed requests
• Natural idempotency: Design operations to be idempotent (SET not INCREMENT)
• Store responses: Must return same response for same idempotency key
• Handle race conditions: Use database constraints or locks
• TTL management: Clean up old idempotency keys appropriately
• Distributed systems: Each service must check idempotency independently
• Saga steps: Each step in a saga should be idempotent