The Deadlock That Froze Our Payment System

This is a story about how a "harmless" refactor—splitting payment processing into two microservices—introduced a lock-order inversion that froze checkout on Black Friday. When concurrent checkouts hit the same order (retry + webhook race), Service A locked payments then waited on orders, while Service B locked orders then waited on inventory. PostgreSQL detected deadlocks, but at 5x normal traffic victims piled up faster than retries cleared: 200 transactions hung for 8 minutes, ~$40K in abandoned carts, and a pager that screamed while users stared at "Processing payment…"

The lesson: deadlock detection is a safety net, not a strategy—consistent lock ordering across every code path is the fix.

Related reading on this site: For the four deadlock conditions and prevention patterns, see Deadlock Conditions & Prevention. For mutex/semaphore intuition applied to DB row locks, read Synchronization (Mutex, Semaphore). For transaction fundamentals, see Databases. For a sibling concurrency bug without deadlocks, see The Race Condition That Cost Us.

---

Context

We ran an e-commerce checkout platform on PostgreSQL 14. A September refactor extracted Payment Service A (charge + ledger) and Payment Service B (inventory hold + order state) from a monolith. Both services updated the same orders row during checkout—nobody wrote down a global lock order.

Original Architecture:

Technology Choices:

• Database: PostgreSQL 14, READ COMMITTED, row-level locks on UPDATE
• Service A: Charges card, UPDATE payments, then UPDATE orders SET payment_status
• Service B: Validates stock, UPDATE orders SET fulfillment_status, then UPDATE inventory
• Peak load: ~120 checkouts/min normal; ~600/min on Black Friday
• Deadlock handling: Rely on PostgreSQL victim selection + app retry (max 3 attempts)

Assumptions Made:

• Deadlocks would be rare if PostgreSQL auto-resolved them
• Services touching different tables wouldn't deadlock (wrong—they shared orders)
• Retries would be enough under load
• Unit tests covered concurrency (they didn't)

---

The Incident

---

Symptoms

What We Saw:

• Deadlock rate: 0–1/day → 40/minute at peak
• Hung transactions: 200 blocked longer than 30 seconds (UI timeout was 60s)
• Checkout p95 latency: 800ms → 12s during worst 3 minutes
• Payment success rate: 99.2% → 91% for 8 minutes
• PostgreSQL: deadlock detected lines in logs; pg_stat_database.deadlocks counter +1,847 in 15 min
• Connection pools: Both services pinned at 92/100 connections

How We Detected It:

• Pager: checkout success rate SLO breach (< 95%)
• pg_stat_activity showed sessions waiting on RowExclusiveLock on orders and payments
• Grafana panel for rate(pg_deadlocks_total)—added after incident, not before

Monitoring Gaps:

• No alert on deadlock rate (only on success rate—lagging indicator)
• No distributed trace tying Service A and B spans on same order_id
• Load tests used sequential checkouts, not concurrent retries + webhooks

---

Root Cause Analysis

Primary Cause: Inconsistent lock ordering across services when updating the same order.

Mechanism chain:

1. Service A transaction: BEGIN → UPDATE payments WHERE order_id = X → UPDATE orders WHERE id = X
2. Service B transaction: BEGIN → UPDATE orders WHERE id = X → UPDATE inventory WHERE sku = Y
3. Concurrent requests on order X (user retry + inventory webhook): A holds lock on payments, waits on orders; B holds lock on orders, waits on inventory—classic circular wait.
4. PostgreSQL picks a victim and rolls back one transaction—but at 600 checkouts/min, new deadlocks form faster than retries complete.
5. Retries re-enter the same lock order, re-deadlocking; connection pool saturation adds queueing delay.

Different lock order on shared orders row → circular wait → victim kill → retry → more deadlocks → pool exhaustion

The Bug (simplified):

Service A: BEGIN; UPDATE payments; UPDATE orders; COMMIT;
Service B: BEGIN; UPDATE orders; UPDATE inventory; COMMIT;
Order X: A holds payments, wants orders. B holds orders, wants inventory.
         B may also need orders again; A waits on orders B holds → DEADLOCK

Why It Was So Bad:

• Black Friday concurrency turned a latent bug into a cascade
• Retries without backoff jitter synchronized competing requests
• Long transactions (~200ms each) widened the deadlock window
• Two services, one table—no documented global lock hierarchy

Contributing Factors:

• Refactor shipped without cross-service transaction review
• Integration tests mocked the database—no real row locks
• Webhook handler and user click path could process the same order simultaneously

---

Fix & Mitigation

Immediate Fixes (During Incident):

1. Emergency deploy: Enforced global lock order inventory → orders → payments in both services
2. Raised statement timeout temporarily to 5s (prevent infinite waits)—paired with fix, not alone
3. Paused non-critical webhooks for 10 minutes to drain backlog
4. Manual reconciliation of 14 orders stuck in payment_pending after victim rollbacks

Long-Term Improvements:

Strategy	What it does	Best when
Global lock ordering	All services acquire row locks in same table order	Multiple writers on shared rows
Short transactions	Validate outside TX; lock only for writes	Any high-concurrency OLTP
Idempotent saga / outbox	Serialize order state changes via queue	Microservices sharing order aggregate
Deadlock-aware retry	Exponential backoff + jitter on 40P01	Unavoidable rare deadlocks
Advisory locks	pg_advisory_xact_lock(order_id) for order-scoped serialization	Hot order rows
Concurrency load tests	Parallel checkout + webhook on same order_id	Before peak events

1. Documented global lock order: inventory → orders → payments → ledger in engineering handbook
2. Refactored Service B to fetch inventory first, then orders, then payments—matching A
3. Added integration test: 50 goroutines hammer same order_id; CI fails on any deadlock
4. Deadlock metrics: Alert if >5 deadlocks/minute for 2 consecutive minutes
5. Saga pattern evaluation: Long-term move to event-driven order state machine via outbox table

---

Architecture After Fix

Both services now acquire row locks in the same order; an optional advisory lock serializes hot order_id paths during peak events.

Key Changes:

• Global lock ordering across all payment code paths
• Shorter transactions—card validation moved outside BEGIN
• Advisory lock on high-contention order_id during campaigns
• CI concurrency test on shared orders
• Deadlock rate alerting—leading indicator before success rate drops

---

Key Lessons

1. Lock ordering is a cross-service contract—if two paths touch the same rows, they must lock in the same order everywhere.

2. Deadlock detection is not prevention—PostgreSQL will kill victims, but under load, victims become a retry storm.

3. Keep transactions short—validate payment methods and inventory before BEGIN; hold locks for milliseconds, not hundreds of milliseconds.

4. Test concurrency, not just correctness—unit tests won't find lock-order inversions; you need parallel integration tests.

5. Retries need jitter—synchronized retries re-deadlock the same rows simultaneously.

6. Watch deadlock metrics—success rate alerts are lagging; pg_stat_database.deadlocks is earlier.

7. Consider sagas for shared aggregates—when multiple services own one order, a serialized state machine beats ad-hoc row updates.

---

Interview Takeaways

Common Questions:

• "How do you prevent deadlocks in payment systems?"
• "What is lock ordering and why does it matter?"
• "Is retry enough when PostgreSQL detects a deadlock?"

What Interviewers Are Looking For:

• Four deadlock conditions (especially circular wait)
• Global lock hierarchy across services
• Short transactions and validation outside locks
• Load testing with concurrent access to same entities

---

What a Senior Engineer Would Do Differently

From the Start:

1. Define global lock order before splitting the monolith
2. Integration tests with concurrent updates on the same order_id
3. Deadlock metrics and alerts from day one
4. Saga or outbox for order state instead of dual writers
5. Pre-peak load test simulating retries + webhooks on hot SKUs

The Real Lesson: Concurrency bugs hide in refactors until traffic proves you wrong. Lock ordering is cheap insurance; deadlock detection is expensive triage.

---

How I'd answer in interviews

"We split payment processing into two services—one updated payments then orders, the other orders then inventory. On Black Friday, concurrent requests on the same order created circular row-lock waits; PostgreSQL killed victims, but retries at five-x load re-deadlocked until two hundred checkouts hung. I'd enforce a global lock order across every service—inventory, orders, payments—keep transactions short, add advisory locks on hot order IDs, alert on deadlock rate, and run parallel integration tests that hammer one order_id. Long term, a saga or outbox serializes order state instead of two writers on the same row."

---

Related reading on this site

• Deadlock Conditions & Prevention — mutual exclusion, hold-and-wait, circular wait, and breaking conditions.
• Synchronization (Mutex, Semaphore) — how lock discipline in code maps to database row locks.
• Databases — transactions, isolation levels, and why READ COMMITTED still deadlocks.
• Database Connection Pooling — how retry storms exhaust pools after deadlocks.
• The Race Condition That Cost Us — concurrency failure without circular wait—compare prevention tactics.

---

FAQs

Q: How do you prevent deadlocks in payment systems?

A: Enforce a global lock order on every code path that touches shared rows, keep transactions short, validate before BEGIN, and use idempotent sagas when multiple services own one aggregate. Alert on deadlock rate, not just success rate.

Q: Isn't PostgreSQL deadlock detection enough?

A: Detection picks a victim and rolls back—but under high concurrency, new deadlocks form faster than retries finish. Detection is a backstop; consistent lock ordering is prevention.

Q: What lock order should we use?

A: Pick a stable hierarchy (e.g., parent before child: inventory → orders → payments) and document it. All services must follow it—even if it feels unnatural in one service.

Q: How is a deadlock different from a race condition?

A: Deadlock is circular wait on locks—no progress until a victim is killed. Race condition is unordered access causing wrong results without necessarily waiting. Both need concurrency tests; fixes differ—lock order vs atomic operations or constraints.

Q: Should we use SELECT FOR UPDATE everywhere?

A: Only where needed—and always in the same order. FOR UPDATE acquires locks earlier, which can prevent some races but increase deadlock risk if order differs across services.

Q: What retry policy works for deadlock victims?

A: Retry on SQLSTATE 40P01 with exponential backoff and jitter, cap attempts (3–5), and surface failure to the user after exhaustion. Never tight-loop retry.

Q: When should we move to a saga instead of row locks?

A: When three or more services mutate one aggregate, or when webhook + user paths routinely overlap. A single-writer outbox or state machine removes cross-service lock coupling.