Notification System Design

Introduction

You request a password reset code. The app says the code was sent. Five minutes pass. You request again. Now two codes arrive, and the first one is already invalid. From the user's view, the notification system failed, even if the API returned 202 and the provider accepted the request.

A notification platform is a delivery system wrapped in product rules. It must decide what to send, which channel to use, whether the user allowed that channel, which template and language to render, how often to retry, and what to do when providers slow down. It also has to protect urgent messages from noisy ones. A receipt or OTP should not sit behind a marketing blast.

The biggest interview trap is claiming exactly-once delivery. You can make your internal processing idempotent. You can dedupe logical notifications. You can record attempts. But you cannot guarantee that a human sees exactly one alert across email inboxes, phones, carrier networks, provider retries, and app reinstalls. A strong design says the honest promise: durable enqueue, at-least-once processing with dedupe, clear suppression rules, and observable delivery attempts.

How to approach

Split the system into three phases before naming infrastructure: ingest, route, and deliver. Ingest saves the request and returns quickly. Routing applies product policy. Delivery talks to providers with retries and limits.

Clarifying Q&A

You: "Are we handling transactional messages, marketing campaigns, or both?"

Interviewer: "Both, but transactional messages such as OTP and receipts must be protected."

That answer means you need separate lanes or priority budgets. One marketing campaign must not delay password resets.

You: "Which channels are in scope: push, email, SMS, and in-app?"

Interviewer: "Use push, email, and SMS. Mention in-app if it helps reliability."

Now you should draw channel-specific queues and provider adapters, not one generic sender.

You: "Do preferences and quiet hours apply to every notification?"

Interviewer: "Marketing must respect them. Some transactional messages may bypass quiet hours."

This tells you preferences are policy, not a simple boolean. Category matters.

You: "Should POST wait until the provider delivers, or return after the job is saved?"

Interviewer: "Return after saving and enqueueing. Delivery is async."

That sets the API contract: 202 means accepted for processing, not delivered to the user's device.

You: "What should we promise about duplicates?"

Interviewer: "Avoid user-visible duplicates, but be realistic."

Now you can explain idempotency keys and dedupe without pretending the outside world is exactly once.

Minute pacing + In the room

Minutes	Focus
0-5	Clarify categories, channels, preferences, delivery promise, and duplicate tolerance.
5-12	Capacity, HLD, and queue separation.
12-24	Detailed design for ingest, routing, template, delivery, retry, and status.
24-34	Architectural dig on channel fan-out, dedupe, and provider rate limits.
34-45	Failure, observability, security, cost, bottlenecks, and interview pushes.

In the room: "I will make POST durable and fast: validate, dedupe, save, enqueue, return 202. Router workers load preferences, template, locale, and category, then create channel jobs only for allowed destinations. Channel workers send through provider adapters with per-provider rate limits and retries. I will keep transactional and marketing lanes separate, put poison messages in DLQ, and be clear that exactly-once to a human is not a promise."

Out of scope

Building the external providers. APNs, FCM, SMS carriers, and email services already exist. The design integrates with them and handles their limits.

A full marketing campaign builder. Audience selection and campaign analytics can be separate. This service receives a notification request or campaign batch and delivers safely.

Long-term business intelligence. Delivery events can stream to analytics, but the interview should focus on operational status, retries, and correctness.

Perfect human attention tracking. You can track provider status, app open, and click events, but you cannot prove the user read the message.

Capacity estimation

Notifications have bursty traffic and uneven importance. A password reset and a sale announcement may share providers, but they should not share the same failure budget.

Dimension	Example scale	What it means
Daily sends	Hundreds of millions	Queues and workers must be horizontally scalable.
Campaign burst	Millions in minutes	Batch fan-out and throttle to provider quotas.
Transactional latency	Seconds, not minutes	Isolate OTP, security, and receipt lanes.
Device tokens	Many per user, many stale	Token registry needs cleanup on provider errors.
Templates and locales	Thousands of variants	Version templates and validate payload variables.
Provider limits	Per account, region, domain, or number	Rate limit per provider and tenant, not only globally.

This pushes the design toward durable queues, channel-specific workers, and backpressure. Adding more workers is not always correct; if the provider allows only a fixed send rate, extra workers just create retries and cost.

High-level architecture

The system has an ingestion layer, a routing layer, channel queues, provider adapters, and status storage. Producers do not call Twilio or FCM directly. They call the notification API with a logical notification request.

The Notification API validates the caller, checks idempotency, stores a notification row, and enqueues a routing job. Router workers load user preferences, device registry, template version, locale, and category. They suppress disallowed messages, schedule messages for quiet hours, or create channel jobs. Channel workers send through provider adapters, enforce rate limits, record attempts, and retry transient failures. DLQ tooling holds poison jobs that should not be retried forever.

[ Product services ]
        |
        v
[ Notification API ] -> [ Notification DB ] -> [ Routing queue ]
        |                                           |
        |                                           v
        |                              [ Router: prefs + template ]
        |                                           |
        |                 +-------------------------+-------------------+
        |                 v                         v                   v
        |          [ Push queue ]             [ Email queue ]      [ SMS queue ]
        |                 |                         |                   |
        v                 v                         v                   v
[ Status API ]     FCM / APNs adapters        SES / SendGrid       Twilio / carrier

DLQ and replay tools sit beside each queue.

In the room: Say that 202 from the API means the notification was saved and queued. It does not mean a provider delivered it, and it certainly does not mean the user saw it.

Core design approaches

The most important approach is lane isolation. Transactional messages such as OTPs, password resets, security alerts, receipts, and booking confirmations need a faster and more protected path than marketing campaigns. That can be separate queues, strict priorities, or per-lane worker budgets. The key is that a campaign cannot consume all provider quota while OTPs wait.

The second approach is policy before fan-out. Do not create email, SMS, and push jobs for every event and then filter later. Load preferences, category, quiet hours, locale, and template metadata first. If the user opted out of marketing SMS, the system should write SUPPRESSED and stop before paying for an SMS job.

The third approach is at-least-once with dedupe. Internal queues usually deliver messages at least once. Workers can crash after sending but before acknowledging. Providers can timeout after accepting a message. Instead of disabling retries, use a dedupe key for the logical notification and keep attempt records. Accept that rare duplicates may still happen outside your boundary, and design the client or in-app inbox to collapse duplicates where possible.

Data model

Record	Important fields	Why it exists
notification	notification_id, producer, user, category, dedupe_key, status	Logical notification request and idempotency anchor.
template	template_id, version, locale, channel, required variables	Keeps rendering deterministic and reviewable.
preference	user, channel, category, quiet hours, version	Decides whether a message may be sent.
destination	device token, email address, phone, verified status, last_seen	Tells workers where to send and when to stop trying.
channel_job	notification id, channel, priority, scheduled_at, attempt_count	Work item for a specific channel.
delivery_attempt	provider, provider message id, status, error code, latency	Explains what happened with each provider call.
suppression	user or destination, reason, source, created_at	Prevents sends after hard bounce, opt-out, or invalid token.
dlq_message	original job, error, first_seen, last_seen, replay status	Holds poison work for investigation.

Store dedupe_key as a uniqueness rule per producer or tenant. A useful shape is producer_event_id + user_id + notification_type. If the producer retries the same logical event, the API returns the existing notification id.

Detailed design

The user-facing API should be fast because it is usually called from another product service during a user action. When checkout completes, the checkout service should not wait for email, SMS, or push providers. It should ask the notification service to send a receipt, receive 202 Accepted, and continue.

On ingest, the Notification API authenticates the producer service, validates the payload against the chosen template contract, checks the idempotency or dedupe key, and saves a notification row. If the same logical notification already exists, the API returns the existing id and current status instead of creating a duplicate. Then it enqueues a routing job and returns 202. This is the point where the producer's synchronous responsibility ends.

A router worker picks up the routing job and loads the user's preferences, available destinations, category rules, and template metadata. If the message is marketing and the user opted out of email, the router records SUPPRESSED for email. If quiet hours apply, it schedules the job for the next allowed time. If the message is transactional, it may bypass quiet hours but still respect hard legal suppressions such as unsubscribed marketing email or invalid phone numbers, depending on the rule. The router renders or prepares template variables with a specific template version so retries do not accidentally change wording halfway through.

Fan-out happens after policy. If the notification can use push and email, the router creates separate channel jobs. Push may fan out further to multiple device tokens for one user. Email may choose a verified email address. SMS may choose a phone number and country-specific provider. Each job carries priority, scheduled time, dedupe key, and enough template context for the worker to send without calling every upstream service again.

Channel workers send with provider-specific rules. Push workers talk to FCM or APNs and delete stale tokens when providers return terminal errors such as unregistered token. Email workers respect domain reputation, bounce suppressions, and template rendering rules. SMS workers respect per-number, per-country, and carrier throughput limits. All workers use exponential backoff with jitter for transient 429 and 5xx errors. They do not retry permanent template errors forever; those go to DLQ with enough detail for a person to fix and replay.

Provider failover is useful but not magic. If SendGrid is down, SES might help only if the sending domain, DKIM, template, unsubscribe header, and reputation setup are ready. If one SMS provider is throttled in a country, another provider may have different compliance or sender-id rules. Failover should be configured by channel and region, and it should preserve dedupe so the user does not receive the same OTP twice from two vendors.

Status is written throughout the journey. A notification can be ACCEPTED, SUPPRESSED, QUEUED, SENT_TO_PROVIDER, DELIVERED_BY_PROVIDER, BOUNCED, FAILED, or DLQ. These names should not overpromise. DELIVERED_BY_PROVIDER means the provider says it delivered to a device or mailbox endpoint. It does not mean the human read it.

In the room recap: "POST validates, dedupes, saves, queues, and returns 202. Router checks preferences, quiet hours, template, locale, and destinations before creating channel jobs. Channel workers enforce provider limits, retry transient errors, record attempts, remove dead tokens, and send poison messages to DLQ. Transactional and marketing lanes are isolated, and exactly-once to the human is not the promise."

Architectural dig

One logical notification across three channels

The hardest part is not making one HTTP call to a provider. It is keeping one logical notification consistent while it fans out to several unreliable channels. A weak design lets every producer call every provider directly. Then preferences are inconsistent, retries duplicate messages, and marketing can drown transactional traffic.

Bad design

Checkout service -> SendGrid directly
Auth service     -> Twilio directly
Marketing tool   -> FCM directly

Result: no shared dedupe, no preference audit, no lane control, no common retries

The better design accepts one logical notification, then fans out after policy.

Great design

POST notification(dedupe_key, user, template, category)
  -> save notification row
  -> router loads prefs + template + destinations
  -> create allowed channel jobs
  -> channel workers send with rate limits and retries
  -> attempts update one status trail

Dedupe lives at more than one level. The API dedupes producer retries using dedupe_key. The router should not create duplicate channel jobs for the same notification-channel-destination tuple. Channel workers should store provider message ids when providers return them. The mobile app or in-app inbox can collapse notifications with the same logical id if a provider retry still creates a duplicate visible alert.

Provider rate limits are also part of the design, not an ops footnote. Use token buckets per provider account, tenant, channel, and priority. If Twilio returns 429, do not let every SMS worker retry immediately. Pause or slow that provider lane with jitter, protect transactional jobs first, and let marketing backlog or shed according to product rules.

Key challenges

Challenge	What users or teams see	Strong answer
Channel fan-out	User receives too many or too few messages	Route after preferences and dedupe per channel destination.
Preferences	Opted-out users still receive marketing	Check preferences before fan-out and audit preference version.
Duplicate producer events	Same receipt or OTP sent twice	Dedupe key on logical notification and channel job.
Provider retries	Worker sends, times out, retries, and duplicates	Attempt log, provider id, dedupe, and client/inbox collapse.
Provider limits	Queue age grows and 429s explode	Per-provider token buckets and backoff with jitter.
Poison templates	One bad payload blocks a partition	DLQ with alert and replay after fix.
Exactly-once myth	Interview answer overpromises	Promise at-least-once processing with best-effort dedupe.

Scaling the system

Scale by lane and channel, not only by total worker count. Push, email, SMS, and in-app have different providers, costs, throughput, and failure modes. Transactional and marketing traffic should have separate queues or strict priority so a campaign does not starve password reset.

Partition queues by tenant or user id for fairness, but be careful with ordering. For notifications, strict ordering is usually category-specific rather than global. A password reset should not wait behind a newsletter for the same user. Use priority and scheduled time in the job, then let workers pull within provider budgets.

Campaign fan-out should be batched. A campaign job can expand recipients into routing jobs in chunks, respecting preference snapshots and provider budgets. Do not create a single enormous message that one worker must process for hours. Monitor queue age by lane because CPU can look normal while provider quotas are the real ceiling.

Failure handling

Failure	What happens	What to build
Provider 429	Sends slow down	Token bucket, backoff with jitter, protect transactional lane.
Provider 5xx	Transient failures	Retry with capped attempts and failover if configured safely.
Invalid push token	Push never reaches device	Mark token inactive on terminal provider error.
Hard email bounce	Future email should stop	Add suppression and protect sender reputation.
Template render error	Job cannot be sent correctly	DLQ and alert; fix template or payload, then replay.
Preference service down	Router cannot decide	Fail closed for marketing; use careful cached preferences for allowed transactional categories only if policy permits.
Queue backlog	OTPs arrive late	Scale workers if provider budget allows; shed or pause marketing first.

The degraded state is delayed or suppressed noncritical messages. The outage state is lost or severely delayed transactional messages, stale opt-out enforcement, or uncontrolled duplicate sends.

API design

Endpoint	Role
POST /v1/notifications	Accept a logical notification request and return 202.
GET /v1/notifications/{notification_id}	Read status and attempts.
POST /v1/users/{user_id}/preferences	Update preferences and quiet hours.
GET /v1/templates/{template_id}	Inspect template version metadata.
POST /v1/internal/dlq/{message_id}:replay	Replay a fixed dead-letter message.

POST /v1/notifications should include user_id, template_id, template_version if pinned, category, data, optional channels, optional send_after, and dedupe_key or idempotency_key. The response should include notification_id, initial status, and a status URL.

POST /notifications
  -> validate producer and template contract
  -> dedupe by key
  -> save notification
  -> enqueue routing job
  -> 202 Accepted

router -> prefs/template/destinations -> channel jobs -> providers

Important errors: 400 for malformed payload, 404 for missing template, 409 for idempotency conflict with different payload, 429 for producer quota, and 503 when the service cannot durably enqueue the request.

Observability

Signal	Why it matters
Queue age by lane and channel	Shows OTP starvation before users complain.
Provider error code rate	Separates invalid tokens, throttling, and provider outages.
Dedupe hit rate	Reveals producer retry storms or duplicate events.
Suppression rate by category and preference version	Catches stale opt-out cache and policy mistakes.
Template render failures	Finds bad deploys quickly.
DLQ depth and oldest message age	Prevents poison work from becoming invisible.
Delivery latency p50/p95/p99 by channel	Shows whether channel promises are realistic.
Marketing shed count	Proves the system protected transactional traffic during incidents.

A useful dashboard follows one notification from accepted to routed to attempted to terminal status. If all you have is "provider API returned 200," you cannot explain missing OTPs or compliance complaints.

Security and abuse

Authenticate producer services and authorize which templates and categories they may send. A random service should not be able to send password-reset messages or marketing campaigns to every user. Validate template variables so producers cannot inject unsafe HTML or links. Store secrets for provider credentials in a secure vault and rotate them.

Preferences and unsubscribe are security and compliance concerns, not only UX. Marketing opt-out must be enforced before fan-out, logged with preference version, and protected from stale cache. Phone numbers and email addresses are personal data; avoid logging full destinations in ordinary logs. Rate-limit producers so one bug or compromised service cannot send millions of messages.

Cost awareness

SMS is expensive. Email reputation is fragile. Push is cheaper but not guaranteed to be seen. A good design chooses channels by category and user preference instead of blasting every channel for every event. For example, a password reset may use email plus SMS fallback only after delay, while a marketing campaign should obey frequency caps and cheaper channels first.

Retries also cost money. Retrying a permanent template error wastes workers. Retrying provider 429s too aggressively can make throttling worse. DLQ, backoff, and channel budgets are cost controls as much as reliability controls. Lane isolation also protects revenue: spending provider quota on marketing while OTPs fail is both costly and visible.

Production scenes

Scene 1: Password reset waits behind marketing

The moment is familiar: a campaign launches, queue depth rises, and password reset codes arrive minutes late. CPU may be fine because the bottleneck is provider rate limits or a shared queue, not compute.

The trap is one queue for everything. Workers process a giant marketing batch while transactional jobs sit behind it. What to build: separate transactional and marketing lanes, queue-age alerts by lane, and a shed rule that pauses marketing when OTP latency crosses budget.

Scene 2: Opted-out users still get marketing

A user unsubscribes, then receives another promotional SMS. The provider may have delivered exactly what the system asked it to deliver. The mistake happened earlier: stale preferences or fan-out before policy.

What to build: preference checks before channel jobs, invalidation on preference update, preference version recorded on every send, and fail-closed behavior for marketing when preferences are unavailable.

Scene 3: Provider failover doubles the message

The primary SMS provider times out. The worker sends the same OTP through a backup provider. Later, the primary provider delivers the original message too. The user sees two codes and loses trust.

Provider failover must preserve logical dedupe and understand provider uncertainty. If the primary result is unknown, ask the provider status API when possible, use a stable provider reference, and decide whether the product prefers delay or possible duplicate. Do not blindly send through every provider on timeout.

Scene 4: A bad template poisons the queue

A template deploy removes a required variable. Every job for that template fails rendering. If workers retry forever, the queue fills with work that cannot succeed and hides good messages behind it.

What to build: template validation before rollout, render checks at ingest or routing, DLQ after a small number of terminal render failures, and replay tooling after the template is fixed.

In the interview: Pick one scene and connect it to a mechanism. "Marketing starved OTP" maps to lane isolation. "Opt-out ignored" maps to preference version audit. "Bad template blocked queue" maps to DLQ.

Bottlenecks and tradeoffs

Throughput vs provider quotas

You can scale workers faster than providers accept traffic. If you ignore quotas, you create 429 storms and retry cost. Use per-provider token buckets and scale based on allowed send rate, not only queue depth.

Transactional speed vs marketing reach

Marketing wants large fan-out. Transactional wants low latency. Shared queues make this a fight. Separate lanes make the tradeoff explicit and let you shed marketing first.

Preference freshness vs router latency

Preference reads can be hot. Caching helps, but stale opt-out creates compliance incidents. Use short TTLs, explicit invalidation, and stricter reads for marketing than for harmless product reminders.

Dedupe strength vs user-visible duplicates

Internal dedupe reduces duplicates, but provider and device behavior can still create them. The honest landing point is logical idempotency, attempt tracking, and client or in-app collapse, not a claim of perfect exactly once.

Interview tips

Exactly once is the wrong promise

You might say: "We guarantee exactly-once delivery to every user."

Interview Push: "What if the provider accepted the request but your worker timed out, or the user has two devices?"

Land here: Promise durable enqueue and at-least-once processing with dedupe for the logical notification. Store a dedupe key, record attempts, and collapse duplicates where you control the client or inbox. Be honest that provider networks and human attention are outside your exact control.

202 does not mean delivered

You might say: "The send API returns success after it calls FCM."

Interview Push: "Should checkout wait for FCM, APNs, Twilio, and email?"

Land here: No. POST /notifications returns 202 after validation, dedupe, save, and enqueue. Provider delivery happens later. The status API can show attempts and terminal states, but the producer should not block its user action on external notification providers.

Preferences must run before fan-out

You might say: "We create channel jobs first, then each worker checks opt-out."

Interview Push: "A user opts out of marketing SMS. Why did a job still reach the SMS queue?"

Land here: Router workers should check preferences, quiet hours, category, and destination suppressions before channel jobs are created. Store the preference version used for the decision. Marketing should fail closed if preferences are unavailable, because stale opt-out is more serious than delayed promotion.

Rate limits need budgets, not just retries

You might say: "If Twilio returns 429, we retry until it works."

Interview Push: "What stops all workers from retrying at once and making the throttle worse?"

Land here: Use per-provider token buckets, backoff with jitter, and circuit breakers. Slow or pause the provider lane when it is throttled. Protect transactional messages first, and shed marketing if the backlog threatens important messages.

DLQ is for poison, not laziness

You might say: "Failed messages go to a dead-letter queue."

Interview Push: "Which failures belong there, and what happens next?"

Land here: DLQ is for work that will not succeed by simple retry: invalid template variables, unsupported locale, malformed payload, or repeated terminal provider errors. The DLQ needs alerts, ownership, and replay after a fix. It is not a place to hide messages forever.

What should stick

1. Ingest is not delivery. 202 means saved and queued, not provider delivered or user read.
2. Route before fan-out. Preferences, category, quiet hours, template, and destinations decide which channel jobs exist.
3. Separate lanes. Transactional traffic must not wait behind marketing blasts.
4. Dedupe beats exactly-once claims. Use logical keys, attempt records, and client or inbox collapse.
5. Providers shape the system. Rate limits, bounces, invalid tokens, and failover rules are part of the design.

Tell it in the room: "Producers call POST with a dedupe key. We save and enqueue, return 202, route through preferences and templates, create channel jobs, and send through provider workers with rate limits and retries. Transactional and marketing lanes are separate. Poison jobs go to DLQ, and exactly-once to a human is not the promise."

Key Takeaways

• Separate ingest, routing, and delivery so producers do not wait on providers.
• Check preferences and quiet hours before creating channel jobs.
• Use template versions so retries do not change message content unexpectedly.
• Protect transactional lanes from marketing campaigns.
• Retry transient provider errors with backoff and jitter; stop retrying poison payloads.
• Track delivery attempts and provider error codes, not only API success.
• Explain exactly-once as a myth and offer a realistic dedupe story.

Related Topics

• Rate Limiter System Design - Provider quota and producer abuse controls.
• Booking Waitlist System Design - Offers, notification lag, and accept idempotency.
• Airbnb Reservation System Design - Confirmation messages after a booking commit.
• Job Scheduler System Design - Retries, leases, and dead-letter queues.
• Distributed Cache System Design - Preference cache and stale data tradeoffs.

Quick revision notes

Clarify categories and channels. Ask whether the system handles transactional, marketing, or both; which channels exist; and what preferences apply.

Draw three phases. Ingest saves and queues. Router applies policy and creates channel jobs. Channel workers send with retries and rate limits.

Protect important traffic. OTPs, receipts, and security alerts need a separate lane or strict priority so campaigns cannot starve them.

Do not promise exactly once. Promise idempotent logical notifications, attempt tracking, and dedupe where you control the system.

Operate provider reality. Watch queue age, provider 429, invalid tokens, bounce rates, DLQ depth, and preference-version usage.

Practice next: Open Notification System Design on Practice System Design and run a timed attempt using one OTP and one marketing blast to explain lane isolation.