Notification System Design

Introduction

You reset your password. The app says "code sent." Five minutes later—nothing. You request again. Now two codes arrive and neither works.

Notification platforms are distributed queues with policy. Interviewers want backpressure, per-channel failure modes, and preference evaluation before fan-out—not a diagram with three arrows to SendGrid.

Weak answers treat "delivery" as one boolean. Strong answers separate enqueue (must succeed fast) from delivery (retries, quotas, 410 Gone on dead tokens).

If you remember one thing: Enqueue fast and durable; deliver async with lanes, retries, and idempotency—provider "delivered" is not "user saw it."

How to approach

Split the problem before you name Kafka.

1. Ask scope — Marketing blast vs transactional only? Scheduling? In-app inbox?
2. Ingress — Persist row + ack in milliseconds—do not wait for FCM.
3. One send — Enqueue → prefs → template → channel worker → provider.
4. Lanes — Transactional vs marketing—separate queues or priority.
5. Ops — Queue age, 410 token sweep, stale opt-out cache.

In the room: "I'll split enqueue from delivery, walk one transactional push, then rate limits, retries, and why marketing must not starve OTP."

If you remember one thing: 202 Accepted means we saved the job—not that Apple delivered it to the user's eyes.

Interview tips

Five exchanges that come up often. Each has what you might say, what they push on, and where to land.

Guaranteeing delivery

You: "We guarantee exactly-once delivery to every device."

They ask: "APNs retries, email bounces, user uninstalls the app—what do you actually promise?"

Land here: Aim for at-least-once with idempotency_key per logical notification. Client or inbox dedupes. Be honest: end-to-end exactly-once to humans is hard.

Provider rate limits

You: "We fan out one HTTP call per user in a loop."

They ask: "You need to notify one million users—what happens to FCM quotas?"

Land here: Batch where APIs allow. Throttle per provider account with token buckets. Partition work across workers—never one synchronous loop.

User preferences

You: "We cache preferences in Redis forever for speed."

They ask: "User opted out—marketing still arrives for three minutes. Regulator calls."

Land here: Evaluate prefs before expensive fan-out. Short TTL plus invalidation on update. For marketing flags, consider pessimistic read. Audit pref_version used at send time.

Duplicate notifications

You: "Retries are disabled so users never see duplicates."

They ask: "Worker crashes after send but before ack—do you lose the OTP?"

Land here: Retries are required—use idempotency_key (e.g. order_id + type) so retries do not create duplicate user-visible alerts if client dedupes poorly.

Transactional vs marketing priority

You: "One queue for all notifications—simpler ops."

They ask: "Black Friday marketing spike—password reset waits behind it?"

Land here: Separate queues or priority classes with budgets. Transactional bypasses marketing throttle. Shed marketing first during incidents.

If you remember one thing: After each push, name one mechanism—idempotency key, separate lane, token sweep—not "we'll scale workers."

Capacity estimation

So we cannot: call APNs once per user in a single synchronous loop from the API. Batch, async, throttle.

If you remember one thing: Provider quotas are the real ceiling—not your worker CPU.

High-level architecture

What breaks if enqueue waits for delivery

API time-outs when FCM is slow. Retries without idempotency spam users. One marketing blast blocks OTP in a shared queue.

What works: durable ingress, async lanes, provider adapters

Producer services call Notification API with { user_id, template_id, vars, channel_hint, idempotency_key }. API persists a notification row and enqueues to Kafka/SQS. Router workers load prefs and template, drop if opted out, route to per-channel queues. Channel workers respect provider rate limits, call vendor APIs, record attempts and terminal status. Scheduler ticks delayed jobs into the same pipeline.

Who owns what:

• Ingress API — Auth between services; idempotency; schema validation.
• Preference / device registry — Hot reads; cache with version.
• Template service — i18n, MJML/HTML for email.
• Channel workers — Adapter code per provider; circuit breakers.

[ Producers ] --> [ Notification API ] --> [ Durable queue ]
                              |
                    router workers (prefs + template)
                              |
         +--------------------+--------------------+
         v                    v                    v
   [ Push queue ]       [ Email queue ]      [ SMS queue ]
         |                    |                    |
         v                    v                    v
   FCM / APNs           SendGrid SES           Twilio etc.

  Scheduled: [ Scheduler ] --> same enqueue path

In the room: Say at-least-once delivery and idempotency keys so retries do not become duplicate SMS.

If you remember one thing: Three layers—ingress, router, channel workers—each with different failure modes.

Core design approaches

Transactional vs marketing

Transactional: OTP, receipt—often bypass marketing throttle with abuse checks only.

Marketing: Stricter frequency caps, unsubscribe enforcement, separate quota.

Single vs multi-queue

Separate queues reduce head-of-line blocking—OTP never waits behind a blast.

If you remember one thing: Lane isolation is a product SLO decision—not an ops nice-to-have.

Detailed design

Write path (enqueue)

1. Validate caller; insert notifications row PENDING.
2. Enqueue {notification_id} to stream.
3. Return 202 with id—do not wait for FCM.

Read path (delivery)

1. Worker pulls batch.
2. Load prefs—if channel disabled, mark SUPPRESSED.
3. Render template; send via provider.
4. Record attempt; on 429/5xx requeue with backoff; on 410 delete token.

In the room: Walk POST → 202 → worker → provider → attempt row.

If you remember one thing: SUPPRESSED is a valid terminal state—auditable opt-out, not silent drop.

Key challenges

For each, say what the user sees if you get it wrong.

• Quota — One noisy tenant burns shared provider account—per-tenant caps.
• Duplicate user-visible notifications — Idempotency key per logical event (order_id + type).
• Timezone for scheduled reminders — Store UTC + IANA zone.
• Hot incident — Everyone enqueues at once—admission control or shed non-critical categories.

If you remember one thing: Lost OTP is outage; delayed marketing is degraded UX.

Scaling the system

• Horizontal workers per channel; partition Kafka by tenant_id or user_id.
• Isolate noisy tenants to dedicated queues or lower priority.
• Regional FCM/APNs endpoints—pin workers near provider regions if needed.

If you remember one thing: Scale consumers on queue age, not only CPU.

Failure handling

Degraded UX: Delayed marketing.

Outage: Lost OTP or silent total failure on transactional lane.

If you remember one thing: DLQ poison templates—do not block the whole partition forever.

API design

POST /v1/notifications

In simple terms: save the job, return 202, workers deliver later with prefs and retries.

Internal flow diagram:

POST /v1/notifications --> DB row + queue message --> 202 Accepted
       |
       v
router --> prefs OK? --> channel worker --> provider
              | no --> SUPPRESSED (still auditable)

Errors: 400 bad template; 429 tenant over quota; 503 enqueue path unhealthy—rare if queue is healthy.

In the room: Walk POST → 202 → router → channel worker. Mention SUPPRESSED when opted out.

If you remember one thing: category routes to lane and rate limit—transactional vs marketing.

Production angles

[ Traffic spike ] --> queue depth grows --> age SLO red
       --> shed marketing --> protect transactional lane

Bottlenecks and tradeoffs

Throughput vs cost

The tension — More workers and provider spend vs budget caps.

What breaks — Queue age SLO red during peak.

What teams do — Autoscale on age; throttle marketing first.

Say in the interview — Name which lane gets budget when provider throttles.

Strong prefs vs latency

The tension — Cache prefs for hot path speed.

What breaks — Stale opt-out → compliance incident.

What teams do — Short TTL + invalidation; audit pref version at send.

Say in the interview — Marketing prefs are not the same cache policy as static assets.

If you remember one thing: Protect transactional lane first; shed marketing; sweep dead tokens.

What should stick

You do not need to memorize every box. After this guide, you should be able to:

1. Enqueue vs deliver — API returns 202 fast; workers retry with backoff—different SLOs.
2. Lanes — Transactional and marketing separate queues—OTP never waits behind blast.
3. Idempotency — idempotency_key per logical event—retries without duplicate SMS.
4. Prefs before fan-out — Opt-out enforced in router; audit pref version at send.
5. Provider reality — 410 token sweep; 429 backoff; "delivered" ≠ user saw it.

Tell it in the room: "Producers POST with idempotency key—we persist and enqueue, return 202. Router checks prefs and template, routes to per-channel queues. Workers throttle to provider quotas, record attempts, retry 5xx with jitter, delete dead tokens on 410. Transactional and marketing are separate lanes—shed marketing first in incidents."