Design a Notification System

Introduction

Notification platforms are distributed queues with policy. Interviewers want backpressure, per-channel failure modes, and preference evaluation before fan-out—not a diagram with three arrows to SendGrid.

Weak answers treat “delivery” as boolean. Strong answers separate enqueue (must succeed fast) from delivery (retries, quotas, 410 Gone on dead tokens).

How to approach

Split ingress (durable record + ack) from delivery (async, retriable). Define transactional vs marketing traffic and compliance (STOP, unsubscribe) early. Then one path: enqueue → prefs → template → provider.

Interview tips

• FCM/APNs: tokens go stale; sweep invalid tokens; batch sends where APIs allow.
• DLQ is not optional at scale—poison payloads should not block the whole partition.
• Email: bounces and suppression lists; SMS: opt-in evidence—one sentence each unless they drill in.
• Per-user ordering: password reset before marketing promo in the same minute—priority queues or separate pools.
• Queue lag is your user-visible SLO when latency matters—alert on oldest message age, not only depth.

Capacity estimation

Dimension	Implication
100M+ notifications/day	Partition by tenant or hash(user) for fairness
Peak 5K/s enqueue	Kafka/SQS with provisioned throughput or autoscale consumers
Provider caps	Token bucket per provider account; smooth bursts
Device registry	Hundreds of millions of rows—TTL stale tokens

Implications: you cannot call APNs once per user in a single synchronous loop—batch, async, throttle.

High-level architecture

Producer services call Notification API with { user_id, template_id, vars, channel_hint, idempotency_key }. API persists a notification row and enqueues to Kafka/SQS (or internal queue). Router workers load prefs and template, drop if opted out, route to per-channel queues. Channel workers (push, email, SMS) respect provider rate limits, call vendor APIs, record attempts and terminal status. Scheduler ticks delayed jobs into the same pipeline.

Who owns what:

• Ingress API — Auth between services; idempotency; schema validation.
• Preference / device registry — Hot reads; cache with version.
• Template service — i18n, MJML/HTML for email.
• Channel workers — Adapter code per provider; circuit breakers.

[ Producers ] --> [ Notification API ] --> [ Durable queue ]
                              |
                    router workers (prefs + template)
                              |
         +--------------------+--------------------+
         v                    v                    v
   [ Push queue ]       [ Email queue ]      [ SMS queue ]
         |                    |                    |
         v                    v                    v
   FCM / APNs           SendGrid SES           Twilio etc.

  Scheduled: [ Scheduler ] --> same enqueue path

In the room: Say at-least-once delivery and idempotency keys so retries do not become duplicate SMS.

Core design approaches

Transactional vs marketing

Transactional: OTP, receipt—often bypass marketing throttle with abuse checks only.

Marketing: stricter frequency caps, unsubscribe enforcement, separate quota.

Single vs multi-queue

Separate queues reduce head-of-line blocking—OTP never waits behind a blast.

Detailed design

Write path (enqueue)

1. Validate caller; insert notifications row PENDING.
2. Enqueue {notification_id} to stream.
3. Return 202 with id—do not wait for FCM.

Read path (delivery)

1. Worker pulls batch.
2. Load prefs—if channel disabled, mark SUPPRESSED.
3. Render template; send via provider.
4. Record attempt; on 429/5xx requeue with backoff; on 410 delete token.

Key challenges

• Quota: one noisy tenant can burn shared provider account—per-tenant caps.
• Duplicate user-visible notifications: idempotency key per logical event (e.g. order_id + type).
• Timezone for scheduled reminders—store UTC + IANA zone.
• Hot incident: everyone enqueues at once—admission control at API or queue TTL shedding for non-critical categories.

Scaling the system

• Horizontal workers per channel; partition Kafka by tenant_id or user_id.
• Isolate noisy tenants to dedicated queues or lower priority.
• Regional FCM/APNs endpoints—pin workers near provider regions if needed.

Failure handling

Scenario	Mitigation
Provider 429	Exponential backoff; jitter; circuit open → pause partition
Invalid token	410 → remove token; do not retry forever
Template render error	DLQ; alert—bug not transient
Queue backlog	Scale consumers; shed marketing first if incident

Degraded UX: delayed marketing; outage is lost OTP—transactional class needs stricter SLO.

API design

Endpoint	Role
POST /v1/notifications	Enqueue; body: user_id, template_id, data, channels?, send_after?
GET /v1/notifications/{id}	Status and attempts
POST /v1/users/{id}/preferences	Update prefs (also from product UI)

POST /v1/notifications

Field	Role
idempotency_key	Dedup logical sends
category	transactional | marketing — routing and caps
template_id	Which template version

Internal flow diagram:

POST /v1/notifications --> DB row + queue message --> 202 Accepted
       |
       v
router --> prefs OK? --> channel worker --> provider
              | no --> SUPPRESSED (still auditable)

Errors: 400 bad template; 429 tenant over quota; 503 enqueue path unhealthy—rare if queue is healthy.

Production angles

Notification platforms sit between your reliability and Apple’s, Google’s, ESPs’, and carriers’. The worst incidents are silent: queues age, marketing drowns transactional, email reputation collapses, and compliance discovers stale opt-outs in cache. You learn these patterns after owning multi-channel dispatch through peak traffic and security incidents.

Push “delivered” but the user never sees it (and email “sent” but not read)

What it looks like — Provider dashboards show delivered; the user claims they never got an OTP or receipt. MFA support tickets spike; fraud rises when people fall back to weaker channels. iOS Focus modes and Android notification channels silently swallow categories.

Why it happens — Delivery to device is not visibility in UI. Tokens go stale after reinstall; battery optimization defers FCM; email lands in Promotions or spam because SPF/DKIM posture slipped. Transactional and marketing traffic that share a domain reputation mean one blast can poison receipts.

What good teams do — In-app inbox plus SMS fallback for high-value transactional (cost-aware); separate subdomains or IPs for marketing vs receipts; bounce and complaint webhooks feeding suppression immediately. Metrics on delivery by provider error code, not just HTTP 202 from your API.

Email reputation death spiral after a “small” campaign

What it looks like — Sudden spike in spam placement; open rates collapse; Gmail Postmaster turns red. Product wants to re-send to “make up” volume—making it worse.

Why it happens — Blast without list hygiene; purchased lists; no double opt-in; ignoring hard bounces; shared IP with bad neighbors on your ESP tier. Warm-up skipped for new sending domains.

What good teams do — Suppression list as source of truth; automatic pause on bounce-rate SLO; segment transactional from marketing at the architecture level; tight feedback loops with marketing ops. SES/SendGrid reputation is borrowed, not owned.

Queue backlog: transactional starves while workers drown in marketing payload

What it looks like — Transactional p99 breaches; a marketing enqueue spike fills Redis or Kafka; queue age SLO goes red before CPU. DLQ depth climbs—often provider throttling (429 from APNs) during a viral moment.

Why it happens — Shared infrastructure without lane isolation; large payloads or attachment generation blocking workers; no global concurrency cap per tenant and channel.

What good teams do — Weighted fair queuing or separate queues for transactional vs marketing; shed lower priority first; dynamic rate limits aligned to provider quotas. Measure enqueue p99, queue age by lane, provider error rate by code, DLQ depth, and end-to-end latency from business event to device.

[ Traffic spike ] --> queue depth grows --> age SLO red
       --> shed marketing --> protect transactional lane

Stale preferences in cache — compliance incident, not performance bug

What it looks like — User opts out; marketing still arrives for minutes. A regulator letter follows.

Why it happens — Aggressive caching of preferences without invalidation on update; eventual consistency between profile service and notification router.

What good teams do — Short TTL plus explicit invalidation on change; pessimistic read for marketing category flags; audit every send decision with the version of prefs used.

How to use this in an interview — State clearly that channel delivery ≠ human attention, and that lanes and reputation are first-class. One sentence on why stale opt-out cache is worse than slow email.

Bottlenecks and tradeoffs

• Throughput vs cost: more workers and provider spend.
• Strong prefs vs latency: cache prefs—stale opt-out is a compliance incident—short TTL + invalidation.