Fix the design · Incident triage
Flash-Sale Notifications: The System That Broke Under Traffic
Big Billion Days traffic overloaded Flipkart’s notification path, pushing the database to full CPU, delaying alerts up to 28 minutes, and sending some shoppers the same message more than once.
Flipkart’s notification path must deliver order and payment alerts over email, SMS, and push. During Big Billion Days, traffic jumped from about 5,000 to 300,000 messages per minute; alerts were 20–30 minutes late, some shoppers got duplicates, and PostgreSQL sat at 100% CPU.
We need your help. Identify the bottlenecks and failure modes in the current design, then redesign the path from “order placed” → “notification delivered” so the system can absorb flash-sale traffic, isolate failures, handle retries safely, and deliver notifications with low latency.
Problem
Redesign Flipkart’s existing notification path for shopping apps — it already sends order confirmations, payment alerts, delivery updates, and promotional messages across email, SMS, and push.
Incident summary
- When Big Billion Days traffic hit, messages arrived 20–30 minutes late.
- Some shoppers got the same notification two or three times.
- Others never received their order or payment alerts.
- The database was fully busy — every CPU core maxed out — and almost every connection slot was in use.
- The outside text-message company timed out at random, which crashed the apps that send notifications.
Assumptions made by the team
- PostgreSQL can also act as the waiting line for messages that still need to be sent.
- Checking the database every 5 seconds for new work is fine.
- One notification service is simpler and will grow on its own when traffic jumps.
- If a send fails, trying again right away with no pause is fine.
- Sending the same notification twice is rare and not a big problem.
Team assumptions
- If a send fails, trying again right away is fine.
Impacted services
- Notifications database (critical) — Fully overloaded — most of its work was repeatedly checking for messages that still needed to be sent.
- Message senders (critical) — Kept crashing and restarting when text sends to the SMS vendor started failing.
- Notification API (degraded) — Still took in new requests, but unsent messages piled up faster than they could go out.
- SMS provider (texts) (degraded) — Started responding slowly. About one in five text requests timed out during the sale.
- Email and push (degraded) — Got stuck behind the same slow send path as texts, so those alerts were late too.
- Customers (critical) — Some shoppers got the same order alert two or three times. Others never got payment or order confirmations.
Triage questions
- What is wrong with this design when traffic jumps from 5,000 to 300,000 messages per minute?
- How would you redesign the flow from order placed to customer notified? What new components would you add?
- When the SMS provider slows down or times out, how would you change the design so email and push are not blocked too?
- Why did some customers get duplicate alerts, and how would you prevent sending the same notification twice?