Fix the design · Incident triage

Cache Stampede: The Cache That Took Down Our API

During Amazon India’s Great Indian Festival, an accidental cache clear sent almost every request to the database, filling connection pools, maxing out database CPU, and slowing the slowest requests to 30 seconds.

You are on Amazon India’s catalog API team during the Great Indian Festival. Someone cleared the cache during a deploy. Within minutes, 95% of requests missed the cache and the database buckled.

We need your help. Identify the bottlenecks and failure modes in the current design, then redesign the path from “cache miss” → “catalog response” so a cache flush cannot flood the database, identical keys share one fetch, and slightly stale data can be served while the cache rebuilds.

Problem

Redesign Amazon India’s existing product catalog read path — it already serves product pages through a Redis cache in front of PostgreSQL.

Incident summary

  • After the cache was cleared, about 95% of requests missed the cache for roughly 8 minutes.
  • API servers ran out of database connections — 480 of 500 slots were in use.
  • Database CPU hit 100%. Some requests took 30 seconds instead of under 100 ms.
  • Even after the team limited traffic by hand, errors continued for another 40 minutes.

Assumptions made by the team

  • The database can handle a short wave of cache misses.
  • Clearing the cache during a deploy is safe at our scale.
  • Every cache key can expire at the same time — no need to stagger expiries.
  • One Redis cluster is enough for all keys.

Impacted services

  • Cache (critical) — Went empty after the accidental clear — almost nothing was cached anymore.
  • Product API (critical) — Ran out of database connections and started timing out.
  • Database (critical) — CPU maxed out handling about 8× normal read traffic.
  • Customers (critical) — About 1 in 3 requests failed at the worst point.

Triage questions

  1. What is wrong with this design when the cache goes cold and almost every request hits the database at once?
  2. How would you redesign the read path so a cache flush does not overwhelm the database?
  3. What would you add so many apps do not fetch the same missing key from the database simultaneously?
  4. Should the API serve slightly stale data while the cache rebuilds — and when is that acceptable?

← All design challenges

Loading scenario…