Fix the design · Incident triage

Duplicate Chat Sends: The System That Sent Messages Twice

On WhatsApp, a mobile network blip triggered a reconnect storm, duplicating about 12% of messages and pushing WebSocket gateway CPU to about 92%.

You are on WhatsApp’s messaging team. A mobile network blip disconnects millions of users. On reconnect, about 12% of messages are sent twice.

We need your help. Identify the bottlenecks and failure modes in the current design, then redesign the path from “client sends or retries a message” → “one copy appears in the thread” so reconnects cannot replay already-processed ids, hot threads do not pile duplicate writes, and delivery stays safe under at-least-once retries.

Problem

Redesign WhatsApp’s existing chat delivery path — it already carries messages over WebSockets from mobile clients through a gateway into a chat service and message store.

Incident summary

  • A mobile network blip disconnected millions of users at once.
  • On reconnect, clients retried every message that had not been acknowledged yet.
  • About 12% of deliveries showed duplicate messages.
  • The WebSocket gateway CPU hit about 92% during the reconnect storm.
  • The server never checked whether a client message id had already been processed.

Assumptions made by the team

  • Client-generated message ids are unique enough.
  • Retry on reconnect is the client’s problem.
  • At-least-once delivery (the same message can arrive twice) is fine for chat.
  • Ordering within a thread is best-effort only.

Team assumptions

  • Retry on reconnect is the client's problem.

Impacted services

  • WebSocket gateway (critical) — CPU hit about 92% during the reconnect storm.
  • Chat service (critical) — Accepted and stored duplicate client messages.
  • Message store (degraded) — Hot threads saw lock contention from duplicate writes.
  • Customers (critical) — About 12% of deliveries showed the same message twice.

Triage questions

  1. Why did reconnecting clients cause duplicate messages in this design?
  2. How would you redesign delivery so the same message cannot appear twice in the chat?
  3. Where should duplicate checks happen — on the gateway, the chat service, or both?
  4. How should offline clients sync their outbox without replaying messages already delivered?

← All design challenges

Loading scenario…