← Back to practice catalog

System design interview guide

Notification System Design

TL;DR:

Picture forgetting your password on a shopping app. You tap Forgot password, the app says a code is on its way, and you sit waiting for a text message. A few minutes earlier the store sent a huge sale message to millions of people, and your code got stuck in the same line behind all of them, so it shows up five minutes late, when it is already useless. The part of the system that takes a request like tell this person something and gets it to a phone or an inbox is called a notification system. A good answer explains why the code was late, how urgent messages stay ahead of sale messages, and why a retry must never send a second copy.

Overview diagram for Notification System Design

Problem statement

Picture waiting for a password-reset code while a shopping app is also blasting a sale to millions of people. Your code can sit in the same line behind that blast, arrive too late to use, or show up twice after you tap the button again. A notification service is the shared part of a product that other teams ask when they want to tell a person something — a reset code, a receipt, or a sale announcement. Design one that accepts those requests, fills in the right template, respects preferences, fans out to email, SMS, push, and in-app channels, retries when outside senders fail, and keeps urgent transactional messages ahead of noisy campaigns.

Introduction

The problem: one reset code, five minutes late

Picture forgetting your password on a shopping app. You tap Forgot password, and the app says “code sent.” You switch to your messages and wait for a text. Nothing arrives. After a minute you tap the button again, and the app says “code sent” again. Five minutes after your first tap, two codes show up one after the other, and the first one has already stopped working.

From your side of the screen, the system failed. From the point of view of the company's server, it may have worked perfectly twice, because each time it answered “accepted” and moved on. Between that answer and the buzz on your phone sits a long chain: the company's own servers, an outside network that carries text messages or phone alerts, and finally your phone. Any link in that chain can be slow, can fail, or can try twice. In this story the company had also sent a huge sale message to millions of people a few minutes earlier, and your single reset code was waiting in the same line behind all of them.

The part of the system that sits in the middle of this chain is called a notification system. Other parts of the product, such as the login service, the checkout service, or the marketing tool, hand it a request that says roughly “tell this person this thing,” and it takes responsibility for getting the message to a phone or an inbox. Notice the gap in the story above: the server saying “sent” is not the same as the person seeing the message. A good design is honest about that gap from the very first step.

Why teams bother with a notification system

You meet notifications constantly as a user. A text with a login code. A receipt right after you pay. A warning that someone signed in from a new device. A reminder that your table is booked for tonight. A sale announcement that you can switch off. Behind each of those there is a reason the company cares about getting it right.

Trust and security. A login code or a security alert is only useful if it arrives while you are still looking at the screen. A code that arrives five minutes late is worse than no code, because it sends people to the support desk.

Legal rules and respect. People are allowed to say “stop sending me sale messages.” In many places that choice is the law, and ignoring it can bring fines and angry customers. The company must be able to show, later, that it honored the choice.

Cost. Text messages are paid for one at a time, and email providers watch how many people mark your messages as junk. A bug that sends the same message to a million people, or sends every message on every channel, turns directly into a bill and a damaged sender reputation.

One place for the rules. If every team talks to the outside delivery networks on its own, every team has to rebuild templates, languages, opt-outs, and retries, and no two teams do it the same way. One shared system means one set of rules and one place to look when a message goes missing.

Protecting what matters. A sale announcement is nice to send. A reset code is something a person is waiting on right now. The company needs to make sure the first can never get in the way of the second.

What problem are we solving?

Those reasons only matter if we can say, in plain terms, what goes in and what comes out. The input to the system is a request from another part of the product, such as: “tell user 42 that their password-reset code is 481516; this is a password reset.” The output is the right message, in the right language, on the right channel or channels, at an appropriate time, plus an honest record of what happened to it.

Success looks like this. A person who asks for a reset code gets one code within seconds. A person who turned off sale messages never gets one, and the company can prove it. A request that gets repeated, because of a network hiccup or a bug, does not turn into two messages on the person's screen. When an outside sender is slow, broken, or refuses to take more messages, the system degrades gracefully instead of silently losing messages. And anyone on the team can ask “what happened to this notification?” and get a truthful answer.

We are not building the outside networks themselves, such as the services that carry phone alerts, text messages, or email. Those already exist. What we are designing is everything between the product asking “tell this person” and the outside network taking over. The rest of this page works through that in the order you would think of it in a real interview: first what the system must do and how well, then how big it is, and only then the design, one decision at a time.

This page is the interview deep dive in the system design interview guides hub.

How to approach

We now know what the system is for and why it matters. In an interview you do not start by drawing boxes. You first turn a short, vague prompt into a clear contract: which messages exist, which channels matter, what the person controls, and what counts as correct. Only after that do you size the problem, make the design decisions one at a time, and draw the result.

How to design a notification system in an interview

  1. Clarify the problem: which kinds of messages exist (urgent or promotional), which channels, what the person can switch off, what “delivered” means, how duplicates should be treated, and how big the system is.
  2. Write the functional requirements (what the system does) and the non-functional requirements (how fast, how big, how reliable).
  3. Estimate the size: messages per day, the size of a sale blast, how many devices each person has, and how many messages the outside senders will accept per second.
  4. Make the design decisions in order: how a request is accepted, how urgent messages stay ahead, when preferences are checked, how the message is built, how retries avoid duplicates, how outside limits are respected, and what happens to messages that can never succeed.
  5. Draw the picture those decisions produce, then walk one notification through it.
  6. Go deeper where the interviewer pushes: duplicates, failover between senders, provider limits, opt-out correctness, and one production failure.

Clarifying questions (say these in the room)

In a real interview the prompt is only a few words: “Design a notification system.” Before you design anything, ask questions until the problem is clear. Each answer below changes the design, which is why it is worth asking.

You: “Are we sending only urgent messages, such as login codes and receipts, or sale messages too? If both, does an urgent message have to be protected from the sale ones?”

Interviewer: “Both, but urgent messages such as reset codes and receipts must be protected.”

That answer tells you the two kinds of message do not share the same promise. A sale message can wait a few minutes without harm, and a reset code cannot, so whatever you design must stop a big sale blast from slowing a reset.

You: “Which channels are in scope: phone alerts (push), email, text messages (SMS), and an inbox inside the app?”

Interviewer: “Use push, email, and SMS. Mention the in-app inbox if it helps reliability.”

Each channel is carried by a different outside company with different speed limits, costs, and failure behavior, so you will not be able to treat “sending” as one single thing.

You: “Does every message have to respect the person's settings and quiet hours, or can some skip them?”

Interviewer: “Sale messages must respect them. Some urgent messages may skip quiet hours.”

So the person's settings are a policy that depends on the kind of message, and not a single yes or no switch. A security alert at 3 a.m. may be allowed, while a sale message at 3 a.m. is not.

You: “When another service asks us to send something, should it wait until the person's phone gets it, or can we answer once the request is safely saved?”

Interviewer: “Answer after saving it. Delivery happens afterwards.”

That fixes the contract with the caller. The answer to the caller means “we have it and we will try,” and does not mean “the person saw it.” It also means the checkout page never waits on a text message carrier.

You: “What should we promise about duplicates? If the same request arrives twice, or a try fails halfway, may the person see two copies?”

Interviewer: “Avoid visible duplicates, but be realistic.”

The interviewer is asking you to be honest. You will want to say what you can control, such as recognizing a repeated request, and what you cannot, such as an outside network that delivers a message twice on its own. Park the full discussion until the design decisions, where it has a natural place.

You: “Roughly how many users and how many messages a day, and how big is the biggest sale blast? Also, how quickly should a reset code go out versus a sale message?”

Interviewer: “Tens of millions of users, hundreds of millions of sends a day, and a blast can be millions of messages in a few minutes. A reset should go out within seconds. A sale message can take minutes.”

Those numbers feed the capacity estimate in the next section. The difference between “seconds” and “minutes” tells you early that the two kinds of message need different treatment.

You: “I want to keep building the outside delivery networks, a campaign builder, and long-term analytics out of scope, so we can focus on accepting, routing, and delivering. Does that match what you want?”

Interviewer: “Yes, focus on the notification service itself.”

If the interviewer later asks about vendor names, exact queue products, or a tiny per-channel detail, you do not need to answer yet. Park those until the design decisions, where each one has a reason behind it.

Functional requirements

Functional requirements describe what the system does. From the answers above, a notification system must:

  1. Accept a request and answer quickly. Another service asks “notify this person about this,” and the system saves the request and replies right away, so the caller is never held up by an outside network.
  2. Choose the right channels and destinations. For one person it picks among push, email, and SMS, and among that person's several devices, email addresses, or phone numbers, based on the kind of message and what is allowed.
  3. Build the message from a template in the right language. The caller sends a template name and a few values, and the system fills in the words for that person's language, so wording stays consistent and reviewable.
  4. Respect the person's settings. Opt-outs, quiet hours, and unsubscribes are honored, and the reason a message was skipped is written down.
  5. Keep urgent messages apart from promotional ones. Reset codes, receipts, and security alerts must not wait behind sale messages.
  6. Retry failed attempts without sending twice. Temporary failures are retried, but a retry must not create a second visible copy of the same message.
  7. Record what happened. Every notification has a status trail, such as accepted, skipped, sent to the outside network, delivered, or failed, that anyone can read.
  8. Set aside messages that can never succeed. A message with a broken template or a dead address is parked where a person can look at it, and does not block the line.

Non-functional requirements

Non-functional requirements describe how well the system must do its job. They are the numbers and promises that shape the design.

  1. Fast acceptance. Replying to the caller must take well under a second, because the caller is usually in the middle of a user action such as a checkout.
  2. Timeliness for urgent messages. A reset code is attempted within a few seconds of the request, because the person is still on that page. A sale message may take minutes.
  3. Scale. Tens of millions of users, hundreds of millions of sends a day, and a sale blast of millions of messages in a few minutes. The blast is the heavy flow.
  4. No visible duplicates, stated honestly. The system must do everything it can to avoid sending the same message twice, and must be honest that it cannot promise zero duplicates across outside networks it does not control.
  5. Respecting outside limits. Every outside sender accepts only so many messages per second, and exceeding that makes things worse for everyone.
  6. Auditable opt-outs. When a person opts out, the system must be able to show later which setting it used for every decision.
  7. Durability. Once the system has said “accepted,” the request must not be lost if a server restarts.
  8. Visibility. The team must be able to tell whether urgent messages are on time, not just whether the system is up.

Out of scope (and why you park these)

Write a short out-of-scope list on the board so the interviewer sees you can bound the problem.

Building the outside delivery networks. Apple and Google run the phone alert networks, and other companies carry text messages and email. Your design connects to them and handles their limits, but rebuilding them is a different interview and a different company.

A full campaign builder. Choosing the audience for a sale and measuring its results can be a separate system. This service receives a batch of requests and delivers them safely.

Long-term business analytics. Delivery events can be streamed to an analytics system later. Building it here would use up the round without showing you can keep the delivery path correct.

Proving the person read the message. You can know the outside network delivered a message to a device, and sometimes that the person tapped it, but you can never prove they read it, so the design does not promise that.

That is the requirements stage. If you want to practice it before reading on, try the requirements stage on Practice System Design (about 5–10 minutes), then come back for the sizing and the decisions.

Minute pacing (about 40 minutes)

MinutesFocus
0–5Clarify the problem; write functional and non-functional requirements; agree out of scope
5–12Capacity, then the design decisions and the high-level boxes with one notification story
12–25Detailed design for one notification, then the deep dive on duplicates, provider limits, and preferences
25–35Scaling and failure, with the answers said before they are asked
35–40One production war story and questions

In the room (how you open): “I'd start by agreeing what we are sending and to whom: urgent messages like reset codes, sale messages, or both, and which channels. I'd ask what the person can switch off, whether the caller waits for delivery or only for the request being saved, how we should treat duplicates, and roughly how many messages a day and how big the biggest blast is. I'll write the requirements on the board, size the problem, and then make the design decisions one at a time, so every box I draw has a reason.”

If you remember one thing: Ask first, write the requirements down, and only then design. Say out loud that “accepted” and “the person saw it” are two different promises.

Capacity estimation

Now that the requirements are clear, the next step is to find out how big the problem is, because the size decides how much we have to spread work out and how carefully we have to treat the outside senders. Notification traffic is both bursty and uneven in importance: a reset code and a sale announcement may travel through the same outside company, but they must not share the same fate when something goes wrong.

Rough numbers to anchor the whiteboard (all of these are estimates to tune in the room):

DimensionExample scaleWhat it means
Daily sendsHundreds of millions, for example about 300 million a day, which averages out to about 3,500 a secondQueues and workers must be able to grow by adding machines
Campaign burstMillions in minutes, for example 5 million in 5 minutes, which is about 17,000 a second, around five times the daily averageSpread the blast out in chunks and slow it to what the outside senders accept
Urgent latencySeconds, not minutesUrgent messages need their own protected path
Device tokensSeveral per person; for example 50 million people with two devices each is about 100 million tokens, and many go staleThe list of device addresses needs cleanup when outside networks report dead ones
Templates and languagesThousands of combinationsVersion templates and check the values before sending
Outside limitsPer account, region, domain, or phone number; for example 1,000 texts a second on one accountRespect limits per sender and per customer, not only overall
Stored recordsAbout 1 KB per notification plus attempts, so roughly 300 GB a day, or about 9 TB for 30 daysKeep recent records in fast storage and move older ones somewhere cheaper

What the numbers imply

  • The outside limit, not our servers, is usually the ceiling. If one account is allowed 1,000 texts a second, a 5-million-message text blast needs 5,000 seconds, which is about 83 minutes, no matter how many workers we add. Say this out loud and say it is an estimate.
  • A blast and a reset must not compete for that same allowance. If a blast is allowed to use every one of those 1,000 sends a second, a reset has to wait for a free slot. This is why the allowance itself has to be divided between urgent and promotional traffic.
  • More workers are not always the answer. If the outside sender accepts a fixed rate, extra workers only create more rejections, more retries, and more cost.
  • Many records, mostly short-lived. Hundreds of gigabytes a day of status records is manageable, but it means status history needs a retention rule and cannot sit in the hot database forever.
  • Stale device addresses are normal. People uninstall apps and change phones, so a share of every push blast fails by design, and the system needs to learn from that.

Checkpoint: What is the real ceiling on a sale blast, and why does that make it dangerous for urgent messages?

If you remember one thing: The outside senders set the real speed limit, so the design is about sharing that limit wisely, not about adding more machines.

Design decisions

We now know what the system must do, how fast it must be, and roughly how big it is. Now we choose. Each decision below answers one question the requirements raised, and each one rules out an easier idea that would not survive the requirements. Read them in order, because every decision builds on the one before it.

Decision 1: How is a request accepted without waiting for delivery?

The requirements said the caller must get a reply in well under a second, and that an accepted request must never be lost. The simplest idea is for the caller to talk to the outside sender itself: the checkout service calls the text message company, waits for it to answer, and then finishes the checkout. It is easy to understand, and for one message from one service it works.

It breaks the requirements in two ways. First, the checkout now runs at the speed of the slowest outside company, so a slow text provider makes every checkout slow. Second, if that company is down or the server restarts halfway, the message is simply gone, and nobody knows it was ever meant to be sent. A quick variation, firing the call in the background without saving it anywhere, has the same flaw, because a restart loses everything that was only in memory.

What the requirements ask for is to save the request first and deliver later. The notification system checks the caller, writes one row that says “this notification was requested,” puts a job on a queue (a durable waiting line of jobs that survives restarts, so workers can pick them up one at a time), and replies 202 Accepted. The status code 202 means “we received this and we will process it,” which is exactly the promise we can keep. Delivery to the phone happens afterwards, in workers, and the caller never waits for it.

Decision 2: How do urgent messages stay ahead of sale messages?

The requirements said a reset code must go out within seconds even in the middle of a huge sale blast. The simplest design is one queue where every job waits its turn, first in and first out. It is fair in a plain sense, and it is exactly what caused the late code in the opening story: five million sale jobs went in first, so the reset waited behind all of them.

A common repair is to keep one queue but mark urgent jobs with a higher priority. That helps, but it does not fully solve the problem. The workers and the outside allowance are still shared, so a blast that has already filled every worker and used every send slot can still delay an urgent job by the time it takes to finish what is in flight. A single noisy queue also makes it hard to see which kind of traffic is slow.

The choice is separate lanes. Urgent messages, such as reset codes, security alerts, and receipts, get their own queue, their own workers, and their own protected share of each outside sender's allowance. Sale messages get a different lane with a different budget. When the outside senders slow down or the backlog grows, the sale lane is the one that waits or is paused first, and the urgent lane keeps flowing. This is the single most important idea in the design, and the next decisions all assume it.

Decision 3: When do we check the person's settings?

The requirements said opt-outs and quiet hours must be honored and must be provable later. The simplest idea is to create a send job for every channel for every event, such as one for push, one for email, and one for SMS, and let each sender check whether the person wants it. The code is short, and each sender looks at the settings just before it sends.

The trouble shows up quickly. If a person turned off sale texts, a job for a text still entered the queue, waited, used a worker, and was only then dropped, so we paid for work that was never going to happen. Worse, each sender reads the settings at a slightly different time, so one channel can see the old setting and another the new one, and nobody wrote down which setting was used. That fails the audit requirement. Checking in the caller instead is no better, because then every team has to get the rules right on its own.

The choice is a step in the middle called the router. After the request is saved, a router worker loads the person's settings, the kind of message, the template, and the person's available destinations. It then decides, before creating any send job, which channels are allowed. If a channel is not allowed, the router records a status of SUPPRESSED together with the version of the settings it used, and stops there. If quiet hours apply, it schedules the job for the next allowed time. Only the allowed channels turn into jobs. The kind of message matters at this step: urgent messages may skip quiet hours, and sale messages may not.

Decision 4: How is the actual message built?

The requirements said wording should be consistent, available in many languages, and stable across retries. The simplest idea is for the caller to send the finished sentence: “Your code is 481516.” It is trivial for the notification system, because there is nothing to build.

It fails on the requirements about languages and consistency. Every team now writes its own wording, no one can review the text in one place, and the system cannot choose the person's language, because it never sees anything but a finished sentence in one language. There is also a quieter problem: if the template were edited between the first try and a retry, the person could receive two different messages for one request.

The choice is for the caller to send a template name and a few values, such as the template password_reset and the code. The router fills in the words using the person's language and a specific template version, and it pins that version on the notification. Retries then always use the same words, the wording is reviewed in one place, and a check that every required value is present can catch a broken request before it ever reaches a sender.

Decision 5: How do we stop a retry from sending the message twice?

Sending will sometimes fail, so the system has to retry. The requirements also said the person must not see two copies. The simplest idea is to try again whenever anything goes wrong, including when no answer came back. It sounds right, and it is how duplicates are born.

Here is how. A worker sends the reset code to the outside text provider. The provider accepts the message and starts delivering it, but its reply is slow and the worker gives up waiting. The worker cannot tell whether the message was sent, so it retries, and now the person gets two codes. A worker that crashes right after sending but before noting it down does the same thing, because the queue hands the job to another worker. Never retrying is not an answer either, because then every brief outage loses messages. And promising “exactly once” is not possible, for reasons the deep dive explains.

The choice is to accept that jobs may run more than once, and to make running twice harmless wherever we can. Each logical notification carries a dedupe key, a unique label built from the caller's event id, the person, and the type of message, and the system refuses to create a second notification with the same key. Each send job is unique per notification, channel, and destination. Every attempt is written down, including its outside message id when the provider returns one. Where an outside sender supports it, we pass a stable reference so the sender can recognize the same message. And in the app, the inbox can fold two copies of the same notification into one. This reduces duplicates a great deal without pretending to remove them.

Decision 6: How do we respect the outside senders' limits?

The requirements said every outside sender accepts only so many messages per second, and exceeding that makes things worse. The simplest idea is to run as many workers as possible and retry immediately whenever a message is refused. It is easy, and it feels like the way to finish a blast faster.

It does the opposite. The outside sender replies “too many requests” (the status code 429), the workers all retry immediately, the sender refuses even more, and the system is now spending most of its effort on rejected calls. It also lets a blast take the whole allowance, which breaks Decision 2.

The choice is to give each outside sender a budget. A token bucket is a counter that refills at the rate the sender allows and hands out one token per send, so workers can only send as fast as tokens appear. We keep a bucket for each sender account, channel, and priority, and we reserve part of the allowance for the urgent lane. When a sender returns 429 or an error that may pass, workers wait using exponential backoff with jitter, which means each retry waits roughly twice as long as the last and adds a small random extra delay so workers do not all retry at the same instant. If a sender keeps failing, a circuit breaker stops sending to it for a short while. Failover to a second sender is possible, but only when its sender identity, templates, and legal setup are already in place, and it must not create a second copy of the same message.

Decision 7: What happens to a message that can never succeed?

The requirements said a broken message must not block the line. The simplest idea is to retry until it works. For a temporary outage this is right. For a message that can never work, such as one whose template is missing a value, a malformed request, or a phone number that does not exist, it is a trap, because the job keeps coming back and uses workers that good messages need.

The choice is to separate temporary problems from permanent ones. Temporary ones, such as a 5xx from the sender or a timeout, get a limited number of retries with backoff. Permanent ones are not retried. A broken message goes to a dead-letter queue, which is a separate holding line where a person can look at it, with an alert, an owner, and a way to replay it after the cause is fixed. A permanent failure that tells us something about the destination, such as an unregistered device token or a hard email bounce, also writes a suppression so we stop sending to that address and protect the sender's reputation.

Decision 8: What status do we record and promise?

The requirements said anyone must be able to find out what happened to a notification, and that the system must not overpromise. The simplest idea is a single flag, “sent.” It is easy to store, and it is the source of the confusion in the opening story, because “sent” could mean saved, handed to the outside network, or seen by a person.

The choice is a status trail whose names say exactly what we know: ACCEPTED (saved and queued), SUPPRESSED (skipped because of a setting), QUEUED, SENT_TO_PROVIDER (handed to the outside network), DELIVERED_BY_PROVIDER (the outside network says it reached the device or mailbox), BOUNCED, FAILED, and DLQ. None of these says the person read it, because we can never know that. Each change is stored with a time, so one notification can be followed from request to end.

Putting the decisions together

Taken together, the eight decisions are the design. A request is checked, saved, and answered with a 202. A router applies the person's settings and the kind of message, fills in a pinned template, and creates one job per allowed channel. Urgent and promotional jobs travel in separate lanes. Workers send through each outside network within a budget, retry only temporary problems with backoff, never create a second logical notification for the same dedupe key, and set aside what can never succeed. Every step writes a truthful status.

In the room (a fuller opening you can actually say): “I'd start by saying that accepting a request and delivering it are two different promises. The API checks the caller, drops duplicate requests using a dedupe key, saves the notification, puts a job on a durable queue, and returns 202. A router then loads the person's settings, the kind of message, and the template, and only creates jobs for channels that are allowed, recording why anything was suppressed. Urgent messages like reset codes run in their own lane with their own workers and a protected share of each provider's allowance, so a sale blast cannot delay them. Channel workers send through adapters, stay inside a token bucket per provider, retry temporary errors with backoff and jitter, and send permanent failures to a dead-letter queue. I would not promise exactly once to a human. I'd promise at least once with dedupe, and honest status names.”

The next section draws these decisions as one picture and shows who owns what.

High-level architecture

The decisions above were made one at a time. This section puts them on one page as a picture: which boxes exist, who owns what, and how one notification travels through them. The deeper mechanics come later, so keep this at the level of a map.

At a glance the path has five stages: a front door that accepts the request, a router that applies settings and builds the message, waiting lines and workers per channel, small adapters that talk to each outside sender, and a status store that records what happened. Producers, meaning the other services that want to send something, do not call the phone alert networks or the text carriers directly. They call the notification API with one logical notification request, which is Decision 1 at work.

The Notification API validates the caller, checks the dedupe key, stores a notification row, and enqueues a routing job. Router workers load the person's settings, the device list, the template version, the language, and the kind of message. They suppress what is not allowed, schedule what must wait for quiet hours, and create channel jobs for what remains. Channel workers send through provider adapters, which are small pieces of code that speak each outside company's protocol, enforce the rate budgets from Decision 6, record every attempt, and retry temporary failures. Dead-letter tooling holds the messages that should not be retried forever.

[ Product services ]
        |
        v
[ Notification API ] -> [ Notification DB ] -> [ Routing queue ]
        |                                           |
        |                                           v
        |                              [ Router: prefs + template ]
        |                                           |
        |                 +-------------------------+-------------------+
        |                 v                         v                   v
        |          [ Push queue ]             [ Email queue ]      [ SMS queue ]
        |                 |                         |                   |
        v                 v                         v                   v
[ Status API ]     FCM / APNs adapters        SES / SendGrid       Twilio / carrier

Urgent and promotional traffic use separate lanes of these queues and workers.
DLQ and replay tools sit beside each queue.
BoxOwnsDoes not own
Notification APIChecking the caller, deduping requests, saving the notification, answering 202Deciding channels or talking to outside networks
RouterSettings, quiet hours, kind of message, template version, one job per allowed channelThe actual sending
Channel workersSending, rate budgets, retries, recording attemptsDeciding whether a message is allowed
Status storageThe truthful trail from accepted to a final stateJudging whether the person read the message
Dead-letter toolingSetting aside and replaying messages that cannot succeedRetrying them forever

In the room: Say that the 202 from the API means the notification was saved and queued. It does not mean an outside network delivered it, and it certainly does not mean the person saw it. Then narrate one reset code through the boxes from left to right.

If you remember one thing: Draw the save-and-queue step first, then the router, then separate lanes into channel workers, and say that status is written at every step.

Core design approaches

The boxes in the picture rest on three ideas that came out of the decisions above. Interviewers often ask for these as a short summary once the diagram is on the board, so it helps to say them in your own words before diving into internals.

The most important approach is lane isolation (Decision 2): keeping urgent and promotional work on separate waiting lines with separate workers and separate shares of each outside sender's speed limit. Transactional messages such as one-time codes, password resets, security alerts, receipts, and booking confirmations need a faster and more protected path than marketing campaigns. That can mean separate queues, strict priorities, or per-lane worker budgets. The key is that a campaign cannot consume all of an outside sender's allowance while one-time codes wait.

The second approach is policy before fan-out (Decision 3): decide whether a message may go out, and on which channels, before you create the send jobs. Do not create email, SMS, and push jobs for every event and filter later. Load preferences, kind of message, quiet hours, language, and template information first. If the person opted out of marketing SMS, the system writes SUPPRESSED and stops before paying for an SMS job.

The third approach is at-least-once with dedupe (Decision 5): accept that a job may run more than once, and make a repeat harmless wherever you can. Internal queues usually deliver a job at least once, which means that a worker can crash after sending but before acknowledging, and the job comes back. Outside providers can also time out after accepting a message. Instead of disabling retries, use a dedupe key for the logical notification and keep attempt records. Accept that rare duplicates may still happen outside your boundary, and design the app's inbox to fold duplicates together where possible.

Data model

The boxes need somewhere to remember what they have done, so the next question is what exactly gets stored. The table below lists the main records and why each one exists. Think of the first row as the one that anchors everything else: one row per logical request, which every channel job and attempt points back to.

RecordImportant fieldsWhy it exists
notificationnotification_id, producer, user, category, dedupe_key, statusLogical notification request and the anchor for dedupe
templatetemplate_id, version, locale, channel, required variablesKeeps rendering deterministic and reviewable
preferenceuser, channel, category, quiet hours, versionDecides whether a message may be sent
destinationdevice token, email address, phone, verified status, last_seenTells workers where to send and when to stop trying
channel_jobnotification id, channel, priority, scheduled_at, attempt_countWork item for one channel
delivery_attemptprovider, provider message id, status, error code, latencyExplains what happened with each call to an outside sender
suppressionuser or destination, reason, source, created_atPrevents sends after a hard bounce, opt-out, or invalid token
dlq_messageoriginal job, error, first_seen, last_seen, replay statusHolds messages that cannot succeed until someone investigates

Store dedupe_key as a uniqueness rule per producer or tenant, so the database itself refuses a second row with the same key. A useful shape is producer_event_id + user_id + notification_type. If the producer retries the same logical event, the API returns the existing notification id instead of creating a new one.

Detailed design

We now know what is stored, so we can put the pieces in motion. This section follows one notification, a reset code, from the moment the login service asks for it until a status is final. Then it shows the same thing from the caller's side: the call the producer makes, what it receives back, which errors it can get, and the small admin calls operators use. Read it once as a story. The deeper questions come in the deep dive right after it.

The caller of the notification service is usually another service in the middle of a user action. When checkout completes, the checkout service should not wait for email, SMS, or push. It should ask the notification service to send a receipt, receive 202 Accepted, and continue.

On ingest, the Notification API authenticates the producer service, checks the request against the template's contract, checks the dedupe key, and saves a notification row. If the same logical notification already exists, the API returns the existing id and current status instead of creating a duplicate. Then it enqueues a routing job and returns 202. This is the point where the producer's responsibility ends.

A router worker picks up the routing job and loads the person's preferences, available destinations, category rules, and template metadata. If the message is marketing and the person opted out of email, the router records SUPPRESSED for email. If quiet hours apply, it schedules the job for the next allowed time. If the message is transactional, it may skip quiet hours but must still respect hard legal suppressions, such as an unsubscribed marketing email or an invalid phone number, depending on the rule. The router renders or prepares the template values with a specific template version so retries do not accidentally change the wording halfway through.

Fan-out happens after policy. If the notification can use push and email, the router creates separate channel jobs. Push may fan out further to several device tokens for one person. Email may choose a verified email address. SMS may choose a phone number and a country-specific provider. Each job carries priority, scheduled time, dedupe key, and enough template context for the worker to send without calling every other service again.

Channel workers send with rules that depend on the provider. Push workers talk to FCM or APNs and delete stale tokens when the provider returns a final error such as an unregistered token. Email workers respect domain reputation, bounce suppressions, and rendering rules. SMS workers respect per-number, per-country, and carrier speed limits. All workers use exponential backoff with jitter for temporary 429 and 5xx errors. They do not retry permanent template errors forever, because those go to the dead-letter queue with enough detail for a person to fix and replay them.

Provider failover is useful but not magic. If SendGrid is down, SES might help only if the sending domain, DKIM (the email signature that proves a message really came from you), the template, the unsubscribe header, and the sender's reputation are already set up there. If one SMS provider is throttled in a country, another provider may have different compliance or sender-id rules. Failover should be configured by channel and region, and it should keep the dedupe key so the person does not receive the same code twice from two vendors.

Status is written throughout the journey. A notification can be ACCEPTED, SUPPRESSED, QUEUED, SENT_TO_PROVIDER, DELIVERED_BY_PROVIDER, BOUNCED, FAILED, or DLQ. These names should not overpromise. DELIVERED_BY_PROVIDER means the outside network says it delivered to a device or mailbox. It does not mean the person read it.

  1. Authenticate the producer, check the template contract, and check the dedupe key.
  2. Save the notification and enqueue a routing job, then answer 202.
  3. Router: apply settings, quiet hours, kind of message, and template version; create one job per allowed channel.
  4. Channel workers send within the provider's budget and record each attempt.
  5. Retry temporary errors with backoff; send permanent failures to the dead-letter queue.
  6. Keep the status trail up to date until a final state.

API design: what producers call and what they receive

The notification service is called by other services inside the company, so its API is mostly about three things: the call a producer makes, what it gets back, and a few small calls for preferences, templates, and operators.

EndpointRole
POST /v1/notificationsAccept a logical notification request and return 202
GET /v1/notifications/{notification_id}Read status and attempts
POST /v1/users/{user_id}/preferencesUpdate preferences and quiet hours
GET /v1/templates/{template_id}Inspect template version metadata
POST /v1/internal/dlq/{message_id}:replayReplay a fixed dead-letter message

The producer's call. POST /v1/notifications includes user_id, template_id, template_version if the producer wants to pin one, category (such as transactional or marketing), data (the values for the template), optional channels, optional send_after, and a dedupe_key or idempotency_key. The dedupe key is what lets the producer retry its own request safely.

What the producer receives. The response contains the notification_id, an initial status of ACCEPTED, and a status URL. It is a 202, which means “saved and queued,” and the producer must not read it as “delivered.” If the same dedupe key arrives again with the same content, the response is the original one, with the same id and the current status.

POST /notifications
  -> validate producer and template contract
  -> dedupe by key
  -> save notification
  -> enqueue routing job
  -> 202 Accepted { notification_id, status: "ACCEPTED", status_url }

router -> prefs/template/destinations -> channel jobs -> providers

Which status code means what. A 400 means the payload is malformed. A 404 means the template does not exist. A 409 means the same dedupe key arrived with a different payload, which usually points to a bug in the producer. A 429 means the producer exceeded its own quota, which protects the providers from a runaway service. A 503 means the service cannot durably enqueue the request right now, so the producer should try again later and should not assume anything was saved. A 401 or 403 means the producer is not allowed to use that template or category.

How operators work with it. GET /v1/notifications/{id} shows the status trail and each attempt, which is how support answers “where is my code?” The preferences endpoint is called by the settings page and by unsubscribe links, and it must invalidate any cached copy right away. The dead-letter replay call is internal and is used only after the cause of the failure has been fixed.

In the room recap: “When another service posts a notification, we validate the caller and the template, check the dedupe key so a repeated request does not create a second row, save the notification, put a routing job on a durable waiting line, and return 202 Accepted. A router then checks preferences, quiet hours, the kind of message, the template, the language, and the person's destinations before it creates any channel jobs. Channel workers stay inside each outside sender's speed limit, retry temporary errors with backoff, record every attempt, remove dead device tokens, and send messages that can never succeed to a dead-letter holding line. Urgent and promotional work stay on separate lanes, and exactly-once delivery to a human is not the promise.”

Deep dive: the questions an interviewer asks next

You have drawn the design and walked one notification through it. This is the point where a good interviewer starts digging, and the questions usually come in a fairly predictable order, because each answer raises the next one. Read this section as that conversation. Each question is followed by an answer you can give, plus a hint of where the interviewer is likely to go next. When you can see the follow-up coming, you can offer it before they ask.

“Why not let every product team call the outside senders directly?”

Because the hard part is not one call to an outside sender. It is keeping one logical notification consistent while it spreads across several unreliable channels. When every team calls providers directly, the preferences are applied in a different way by each team, retries create duplicates, and a marketing tool can drown out the reset codes.

Bad design

Checkout service -> SendGrid directly
Auth service     -> Twilio directly
Marketing tool   -> FCM directly

Result: no shared dedupe, no preference audit, no lane control, no common retries

The better design accepts one logical notification and fans out only after policy has been applied.

Great design

POST notification(dedupe_key, user, template, category)
  -> save notification row
  -> router loads prefs + template + destinations
  -> create allowed channel jobs
  -> channel workers send with rate limits and retries
  -> attempts update one status trail

One logical notification routes through preferences and templates, then fans out to push, email, and SMS lanes with rate limits

The shared path is where the rules live. One place checks opt-outs, one place applies the lanes, one place knows what has already been sent, and one trail shows what happened.

Where it goes next: the interviewer will probably ask how the duplicate protection actually works.

“Walk me through a duplicate. What exactly happens, and where do you stop it?”

Take one reset code and follow it with real timings.

t=0.0s   Worker 1 takes the SMS job and sends the code to the text provider
t=0.2s   The provider accepts it and starts delivering to the phone
t=5.0s   The provider's reply never reaches Worker 1; the call times out
t=5.0s   Worker 1 does not know if the text was sent
t=35s    The queue's hand-out timeout passes, and Worker 2 gets the same job
t=35.2s  Worker 2 sends again -> the person gets two codes

There is no single fix, because duplicates can enter at several levels, so the defence has several layers.

  • At the API. A repeated request with the same dedupe key returns the existing notification instead of creating a second one. This stops duplicates caused by the producer retrying.
  • At the router. One channel job exists per notification, channel, and destination, enforced by a uniqueness rule, so a router that runs twice does not create two jobs.
  • At the worker. Before sending, the worker writes an attempt record that says “attempt started.” If the job comes back and it finds a started attempt with no known result, it knows this is a second try, and it can ask the provider about the first one when the provider offers a status lookup, or pass the same stable reference so the provider can recognize the message.
  • On the device. The app's inbox can fold two copies of the same notification id into one. Some push networks also let you pass a label that tells them to replace an earlier undelivered message instead of showing both.

For a one-time code there is one more product choice: if the outcome is unknown, resending is usually better than staying silent, because a late second code that replaces the first is less harmful than no code. Say what the product rule is, and keep the old code from being accepted once a new one is issued.

Where it goes next: “So can you promise the person exactly one message?”

“So can you promise exactly-once delivery?”

No, and saying so plainly is a strong signal. The system can promise that it saves each request once, that it processes each job at least once, and that it does its best to make a repeat harmless. Beyond that, parts of the path are not under its control. A provider can accept a message and then fail to confirm. A phone can be offline when the alert is sent and receive it later along with a retry. A person can have two devices and see the same alert twice. The honest promise is at-least-once processing, deduped on the logical notification, with honest status names, and I would show the layers from the previous answer as the way I make a repeat unlikely and harmless.

A weak answer claims exactly-once up front. A strong answer tells the duplicate story first, explains the layers, and only then names what cannot be controlled.

Where it goes next: “What if one of your outside senders is down and you have a backup?”

“The main text provider times out and you have a backup. What do you do?”

This is the same uncertainty as before, with higher stakes, because a second provider means a second route for the same code. If the first provider only timed out, it may still deliver the message later, so sending through the backup can give the person two codes from two vendors.

So failover is not “on timeout, send via the backup.” A safer rule has three parts. First, when the result of the first attempt is unknown, ask the first provider for the message status if it offers one, and wait a short, bounded time. Second, decide the product rule for the unknown case: for one-time codes, a possible duplicate is usually better than no code, while for a receipt, a short delay is better than a duplicate. Third, keep the dedupe key and the attempt trail across providers, so the status shows both attempts and nothing is silently lost. Failover also only works if the backup already has the right sender identity, templates, and legal setup in each country, so it is configured per channel and region ahead of time.

Where it goes next: “And what do you do when a provider says slow down?”

“A provider starts returning 429. What do workers do?”

A 429 means “too many requests, slow down,” and the worst reaction is for every worker to retry at once. Each outside sender gets a token bucket, a counter that refills at the rate the sender allows, and workers can only send when they can take a token. With separate buckets or reserved shares per channel and priority, a blast cannot drain the urgent lane's share.

Here is a concrete picture. Suppose one account may send 1,000 texts a second. Reserve about 200 a second for the urgent lane and give promotional traffic the other 800. A 5-million-message blast then finishes in about 5,000,000 ÷ 800 ≈ 6,250 seconds, or about 104 minutes, and during all of that time the reset codes still have their own 200 a second. These figures are illustrative, so say they would come from the actual provider contract. When a 429 or a 5xx appears, workers back off exponentially with jitter, and if the errors keep coming a circuit breaker pauses that provider's lane for a short time. Urgent messages are protected first, and the promotional backlog waits or is shed according to a product rule.

Where it goes next: “Tell me how you really guarantee opt-out is honored.”

“How do you make sure an opt-out is really honored?”

An opt-out is a legal promise, so the weak point is a stale copy of the settings. Preferences are read a lot, so they are usually cached, and a cache can hold an old value after the person has opted out. The fix has four parts. Check settings in the router, before any channel job exists. Use short cache lifetimes and also invalidate the cache directly when a person changes a setting. Write the preference version used on every decision, so you can show later which setting applied. And treat marketing differently from urgent mail: if the preference service is unavailable, marketing fails closed (it waits or is dropped), because sending to someone who opted out is worse than a late sale message.

Where it goes next: “What about a message that simply cannot be sent?”

“What goes to the dead-letter queue, and what happens next?”

The dead-letter queue is for messages that will not succeed by trying again: a template with a missing value, an unsupported language, a malformed request, or a provider error that is final. Retrying those forever blocks useful work and costs money. Each entry keeps the original job, the error, and when it first and last failed, and it triggers an alert with a named owner. After someone fixes the cause, such as the template, the entry can be replayed with the original dedupe key so the replay cannot create a duplicate. It is a place for messages that need a person, and not a place to hide failures.

Say it in the interview: “I save once, process at least once, and make a repeat harmless in layers: the API's dedupe key, one job per channel and destination, a written attempt record, and the app's inbox. I do not promise exactly-once to a human, and I show the status honestly.”

Lane isolation and provider limits are covered again in Scaling the system. Queue age and dead-letter depth belong in Observability and Bottlenecks and tradeoffs.

If you remember one thing: The deep dive is a conversation. The duplicate story comes first, the layers of defence come second, and “exactly-once is not a real promise” is the honest ending, which you could have said before they asked.

Key challenges

The deep dive showed how each piece works on its own. What makes the real system hard is everything around it, and several hard parts show up again and again. Name them early in the interview, because each one is a place the interviewer is likely to push.

ChallengeWhat users or teams seeStrong answer
Channel fan-outThe person receives too many or too few messagesRoute after preferences and dedupe per channel destination
PreferencesOpted-out people still receive marketingCheck preferences before fan-out and record the preference version
Duplicate producer eventsThe same receipt or code is sent twiceDedupe key on the logical notification and on each channel job
Provider retriesA worker sends, times out, retries, and duplicatesAttempt log, provider id, dedupe, and inbox collapse
Provider limitsQueue age grows and 429s explodeToken buckets per provider and backoff with jitter
Poison templatesOne bad payload blocks a lineDead-letter queue with an alert and replay after the fix
Exactly-once mythAn interview answer overpromisesPromise at-least-once processing with best-effort dedupe

If you remember one thing: Duplicates and provider limits are design requirements from the start, and they are not surprises you meet later in production.

Scaling the system

Those hard parts get worse as traffic grows, so the next question is what happens when the number of messages multiplies. You do not have to wait for the interviewer to ask. Right after you finish the design, say the scaling story yourself, then be ready for the follow-ups.

What you can say upfront: “I scale by lane and by channel, not just by adding workers. Push, email, SMS, and the in-app inbox each have their own provider, cost, speed, and failure behavior, so each gets its own queue and workers. Urgent and promotional traffic stay in separate lanes so a campaign cannot starve a reset code. The real ceiling is usually the provider's allowed send rate, not our CPU, so I size workers from that rate and watch queue age, not CPU. Big campaigns are expanded in chunks instead of one giant job, and I order work only where it matters, such as per person and category, not globally.”

That one paragraph answers most of what follows. Here are the follow-ups, in the order they usually come.

Interviewer: “A sale blast of five million messages starts. What happens to the system?”

You: “The campaign tool hands us a batch, and we expand it into routing jobs in chunks, not as one enormous job that a single worker would grind through for hours. Each chunk goes through the router, which checks each person's settings at that moment. The promotional lane then drains at the rate the providers allow, which can take over an hour for texts. Meanwhile the urgent lane has its own workers and its own share of the allowance, so reset codes keep going out within seconds.”

Interviewer: “Queue depth is rising. Do you add workers?”

You: “Only if the provider allows more. If the bottleneck is the provider's send rate, more workers only create 429s and retries. I would look at queue age by lane and at the provider error rate first. If urgent queue age is rising, I protect that lane by pausing or slowing the promotional lane. If there is spare provider budget and the workers are really saturated, then I add workers.”

Interviewer: “How do you partition the queues, and what about ordering?”

You: “I partition by tenant or user id so one noisy producer does not hurt everyone and so work spreads evenly. I avoid promising strict global ordering, because it would slow everything down. Ordering matters only within one person and one kind of message, and a password reset should never wait behind a newsletter for the same person. So each job carries priority and a scheduled time, and workers pull within the provider's budget.”

Interviewer: “One person has five devices. Does that change anything?”

You: “It multiplies the push work. One notification can become five device jobs, so the capacity numbers use sends, not notifications. I keep the device list in a registry, remove tokens when the provider says they are no longer valid, and make sure the inbox collapses the same notification id if two devices show it.”

Interviewer: “What about more than one region?”

You: “I keep each person's notifications processed in one home region so dedupe and status stay simple. Providers are often regional too, so I configure failover by channel and region and never send the same code through two regions at once. The thing I would not do is copy opt-out settings loosely between regions, because a stale opt-out is a legal problem, so those get stricter replication.”

If you remember one thing: Scale each lane to its provider's allowed rate, and shed promotional traffic first.

Failure handling

Scaling answers “what if there is more traffic.” This section answers “what if something breaks.” Again, say it first. Failures here come from outside senders, from the settings service, from queues, and from our own workers, and what impresses an interviewer is that the answer was decided before the failure happened.

What you can say upfront: “Every failure has a decision made in advance. If a provider returns temporary errors, I retry with backoff and jitter, and a circuit breaker pauses that provider. If a message can never succeed, it goes to a dead-letter queue and does not block the line. If the settings service is down, marketing fails closed and waits, and urgent messages use a carefully cached copy only where policy allows. If the queue backs up, I protect the urgent lane and slow or pause marketing. The degraded state is late or skipped marketing. The outage state is lost or late urgent messages, stale opt-outs, or a flood of duplicates, and I design to avoid those.”

The follow-ups are about the details of those decisions.

Interviewer: “The text provider returns errors for ten minutes. What does the system do?”

You: “Workers retry temporary errors with exponential backoff and jitter and a capped number of attempts. After repeated failures a circuit breaker, which works like a fuse, pauses that provider's lane so we stop wasting effort. If a backup provider is configured for that country and channel, urgent codes can go there, with the same dedupe key and an attempt trail. Marketing simply waits. The person sees a late text, which is degraded service, and not an outage.”

Interviewer: “The settings service is down. Do you still send?”

You: “It depends on the kind of message. For marketing I fail closed, which means I do not send, because messaging someone who opted out is worse than a late sale message. For urgent messages I may use a recently cached copy of the settings if the policy allows, because a reset code cannot wait for a settings outage, and hard legal suppressions still apply. Either way I record that the decision was made from a cached or missing preference.”

Interviewer: “A worker crashes right after sending.”

You: “The queue hands the job to another worker after its hand-out timeout. That is why an attempt record is written before sending, and why the second worker checks it, asks the provider about the first message when it can, and uses the dedupe key and a stable reference. A rare duplicate can still happen, so the inbox folds copies together.”

Interviewer: “The database is unavailable when a request arrives.”

You: “We cannot honestly say accepted, because the request is not saved, so the API returns 503 and the producer retries with the same dedupe key. For critical producers I would keep a small local retry on their side. I would not accept a request in memory only, because a restart would lose it.”

Interviewer: “Someone deploys a template that is missing a value.”

You: “Every job for that template fails to render. I do not retry those forever. After a small number of final render failures they go to the dead-letter queue with an alert, and I replay them after the template is fixed. Better still, validation at ingest rejects the request with a 400 before it is queued, and templates are checked before rollout.”

FailureWhat happensWhat to build
Provider 429Sends slow downToken bucket, backoff with jitter, protect the urgent lane
Provider 5xxTemporary failuresRetry with capped attempts; failover if configured safely
Invalid push tokenPush never reaches the deviceMark the token inactive on a final provider error
Hard email bounceFuture email should stopAdd a suppression and protect sender reputation
Template render errorThe message cannot be built correctlyDead-letter queue and alert; fix, then replay
Settings service downThe router cannot decideFail closed for marketing; careful cached copy for allowed urgent categories only
Queue backlogReset codes arrive lateAdd workers if provider budget allows; shed or pause marketing first

If you remember one thing: Late marketing is degraded service. Late or lost urgent messages, stale opt-outs, and duplicate floods are outages.

Observability (what you measure)

Failure handling decides what to do when something breaks. Observability is how you notice the break before users fill the support queue, and that takes more than one number. “The provider returned 200” is not enough, because a 200 does not say the code arrived on time, and it will not explain a missing one.

SignalWhy it matters
Queue age by lane and channelShows urgent starvation before people complain
Provider error code rateSeparates invalid tokens, throttling, and provider outages
Dedupe hit rateReveals producer retry storms or duplicate events
Suppression rate by category and preference versionCatches a stale opt-out cache and policy mistakes
Template render failuresFinds bad deploys quickly
Dead-letter depth and oldest message agePrevents poison work from becoming invisible
Delivery latency p50, p95, and p99 by channel (how long the median, the slowest 1 in 20, and the slowest 1 in 100 messages take)Shows whether channel promises are realistic
Marketing shed countProves the system protected urgent traffic during incidents

A useful dashboard follows one notification from accepted to routed to attempted to a final status.

Security and abuse

A notification system can reach every one of your users, so this section covers how to keep it honest once the delivery path works. Authenticate the producer services that call you, and authorize which templates and categories each may send. A random internal service should not be able to fire password-reset messages or marketing campaigns to every user. Validate template values so producers cannot inject unsafe HTML or links into the rendered message. Store outside-sender credentials in a secure vault and rotate them on a schedule.

Preferences and unsubscribe are security and compliance concerns, not only user-interface features. Marketing opt-out must be enforced before fan-out, logged with the preference version used for the decision, and protected from a stale cache that still thinks the person wants the message. Phone numbers and email addresses are personal data, so avoid logging full destinations in ordinary application logs. Rate-limit each producer so one bug, or one compromised service account, cannot send millions of messages before anyone notices.

Cost awareness

Every choice so far also has a price, and interviewers like to hear that you noticed. SMS is expensive per send. Email looks cheap until a blast damages the sender's reputation and future mail starts landing in spam. Push is cheaper but is not guaranteed to be seen. A good design chooses channels by kind of message and by what the person prefers, instead of blasting every channel for every event. For example, a password reset may use push or email first, and SMS only after a short delay if those fail, while a marketing campaign should obey frequency caps and use cheaper channels first.

Retries also cost money. Retrying a permanent template error wastes workers that good messages need. Retrying provider 429s too aggressively can make the throttling worse and burn the rest of the day's allowance on rejected calls. Dead-letter queues, backoff, and channel budgets are cost controls as much as reliability controls. Lane isolation also protects revenue, because spending the outside sender's allowance on marketing while reset codes fail is both costly and visible to support.

Green Dashboard. Angry Users. The Production Mindset System Design Interviews Expect.

Everything so far describes how the system should work on a good day. This section is about what happens on a bad day, told as short stories from the people who carry the pager, the on-call engineers. Diagrams show boxes and arrows. On-call shows something else: a notification system can say “sent” on every dashboard while real people sit waiting, and a quiet dashboard can hide a system that has stopped doing its job.

Four nightmares that look fine on a dashboard:

What the dashboard showsWhat is actually happening
Every provider call returns 200 and CPU is lowReset codes wait in a shared queue behind a sale blast
Delivery rate is steadyOpted-out people are receiving sale messages from a stale cache
Send volume looks normalA backup provider delivered a second copy of every code
Queue depth is flatA bad template is failing silently and filling the dead-letter queue

Scene 1: Password reset waits behind marketing

The moment: A shopping app launches a flash sale at noon. Within minutes, people who forgot their passwords tap Forgot password so they can log in and buy. The codes arrive five minutes late, by which time many have given up, and the support line fills with “where is my code?” tickets.

The trap: CPU on the workers looks fine, because the real bottleneck is the outside sender's allowed send rate and a single shared waiting line. Workers are busy grinding through a giant marketing batch while the reset codes sit behind it. The metric that lied was “provider returns 200,” which only proves the company accepted the call, not that urgent work was on time.

What to build:

  • Separate urgent and promotional lanes, each with its own workers and its own share of every outside sender's allowance.
  • Alert on queue age per lane, not only on overall depth or provider success rate.
  • Pause or slow promotional sends the moment reset-code latency crosses its budget, so the urgent lane keeps flowing.

Scene 2: Opted-out people still get marketing

The moment: A customer unsubscribes from sale texts on Monday and receives another one on Wednesday. They post a screenshot and complain to the regulator, and the company cannot quickly prove which setting it used.

The trap: The outside sender delivered exactly what the system asked it to deliver. The mistake happened earlier: a stale cached copy of the settings still said the person wanted the message, or a fan-out created channel jobs before policy was checked at all.

What to build:

  • Check preferences, quiet hours, and suppressions in the router before any channel job exists.
  • Invalidate the settings cache the moment a person changes a preference, and keep cache lifetimes short.
  • Record the preference version used on every send decision so audits can show what applied.
  • Fail closed for marketing when the settings service is unavailable, because a late sale is better than messaging someone who opted out.

Scene 3: Provider failover doubles the message

The moment: The main text provider slows down during a busy evening. The system moves to a backup provider, and soon people report receiving two login codes, the second just a moment after the first, and the first one no longer works.

The trap: A timeout is not the same as a failure. The primary provider had already accepted many of those messages and delivered them late, while the backup also sent them. The person sees two codes, trusts neither, and support has no single status trail that ties both attempts together.

What to build:

  • Keep the same dedupe key and attempt trail across providers so failover cannot silently create a second logical send.
  • Ask the first provider for message status when it offers a lookup, and wait a short bounded time before deciding.
  • Follow an explicit product rule for the unknown case (for one-time codes, a possible duplicate may be better than silence). Do not blindly send through every provider on a timeout.

Scene 4: A bad template poisons the queue

The moment: A team ships a template change on a Tuesday afternoon. A required value is renamed, and every receipt that uses that template begins to fail to render.

The trap: If workers retry those jobs forever, the waiting line fills with work that can never succeed and hides good messages behind it. Overall queue depth may look busy and healthy while nothing useful is going out.

What to build:

  • Validate templates before rollout, and reject bad payloads at intake with a clear error when required values are missing.
  • After a small number of final render failures, move the job to a dead-letter holding line with an alert and a named owner.
  • Offer replay tools that reuse the original dedupe key once the template is fixed, so the fix does not create duplicates.

In the interview: Pick one scene and connect it to a mechanism. “Marketing starved the reset codes” maps to lane isolation. “An opt-out was ignored” maps to the preference version audit. “A bad template blocked the queue” maps to the dead-letter queue. You do not need all four. You need to sound like someone who knows that green does not mean done.

Bottlenecks and tradeoffs

Those stories all trace back to a small number of choices where you cannot have both things you want. Interviewers listen for whether you can name the tension, say what breaks, and pick a side on purpose. Here are the four that matter most for a notification system.

Throughput vs provider quotas

The tension — You can add workers faster than outside senders will accept traffic, so “more machines” feels like the answer until the sender starts refusing calls.

What breaks — Ignoring quotas creates 429 storms, retry cost, and a backlog that burns the day's allowance on rejected work.

What teams do — Give each outside sender a token bucket that refills at the allowed rate, reserve a share for the urgent lane, and scale workers from that rate instead of from raw queue depth.

Say in the interview — “The real ceiling is the provider's send rate, not our CPU.”

Urgent speed vs marketing reach

The tension — Marketing wants large fan-out as fast as the blast will go, while urgent messages want low latency for the person waiting on the screen.

What breaks — A shared waiting line turns this into a fight, and a sale blast can push a reset code minutes late.

What teams do — Separate lanes with separate workers and a protected share of each sender's allowance, and shed or pause marketing first when urgent latency slips.

Say in the interview — “A campaign can wait; a reset code cannot.”

Preference freshness vs router latency

The tension — Preference reads are hot, so caching makes the router fast, but a stale opt-out creates a compliance incident.

What breaks — A long-lived cache, or a fan-out that checks settings after jobs already exist, can send marketing to people who already said stop.

What teams do — Use short cache lifetimes, invalidate on every preference change, record the preference version on each decision, and read more strictly for marketing than for harmless product reminders.

Say in the interview — “Marketing fails closed when preferences are unclear.”

Dedupe strength vs user-visible duplicates

The tension — Internal dedupe can stop most repeats you control, but outside senders and multiple devices can still show the person two copies.

What breaks — Claiming perfect exactly-once delivery, then being unable to explain a timeout-and-retry story.

What teams do — Use a logical dedupe key, one job per channel and destination, attempt records, and inbox collapse where the app controls the surface.

Say in the interview — “At-least-once with dedupe, and honest about what I cannot control.”

If you remember one thing: The notification system can become a bottleneck that hurts the most important messages if the outside senders' limits and the shared queue are ignored.

Interview tips

You now have a full design and the tradeoffs behind it, so the last thing to practice is defending it when the interviewer pushes back. These six are the pushes that come up most after you finish the design:

Exactly once is the wrong promise

In the room

You might say: “We guarantee exactly-once delivery to every user.”

Interview Push: “What if the outside sender accepted the message but your worker timed out, or the person has two devices?”

Land here: Promise what you can actually keep. Save each request once, process each job at least once, and make a repeat harmless with a dedupe key on the logical notification, an attempt record for every send, and inbox collapse where the app controls the surface. Outside networks can still deliver late or twice, and a person with two phones can see the same alert on both, so say that honestly instead of claiming perfect exactly-once end to end.

202 does not mean delivered

In the room

You might say: “The send API returns success after it calls the phone alert network.”

Interview Push: “Should checkout wait for the phone network, the text provider, and email?”

Land here: Checkout should not wait on any outside sender. POST /notifications returns 202 after the system validates the caller, checks the dedupe key, saves the notification, and puts a routing job on a durable waiting line. Delivery happens later in workers. The status API can show attempts and final states for support, but the producer must finish its user action on “accepted,” not on “the phone buzzed.”

One queue for everything starves the reset code

In the room

You might say: “All notifications go through one queue and workers pull the next job.”

Interview Push: “A sale blast of five million messages is ahead of a reset code. When does the code go out?”

Land here: After the blast drains, which at a typical text-provider limit can take an hour or more. That is why urgent and promotional traffic need separate lanes, each with its own workers and a protected share of each outside sender's allowance. Alert on queue age per lane, and pause marketing first when reset-code latency slips.

Preferences must run before fan-out

In the room

You might say: “We create channel jobs first, and each worker checks the opt-out.”

Interview Push: “A person opts out of sale texts. Why did a job still reach the SMS queue?”

Land here: Because the check happened too late. Router workers should load preferences, quiet hours, kind of message, and suppressions before any channel job is created, and they should store the preference version used for the decision. If the settings service is unavailable, marketing should fail closed and wait, because messaging someone who opted out is worse than a delayed promotion.

Rate limits need budgets, not just retries

In the room

You might say: “If the text provider returns 429, we retry until it works.”

Interview Push: “What stops all workers from retrying at once and making the throttle worse?”

Land here: A budget, not hope. Give each outside sender a token bucket that refills at the allowed rate, retry temporary errors with exponential backoff and a little random jitter so workers do not all return together, and trip a circuit breaker when the errors keep coming. Slow or pause that provider's lane while it is throttled, protect urgent messages first, and shed marketing if the backlog threatens important sends.

The dead-letter queue is for poison, not laziness

In the room

You might say: “Failed messages go to a dead-letter queue.”

Interview Push: “Which failures belong there, and what happens next?”

Land here: Only work that will not succeed by simple retry belongs there: invalid template values, an unsupported language, a malformed request, or repeated final provider errors. Each entry needs an alert, a named owner, and a replay path that reuses the original dedupe key after the cause is fixed. It is a holding line for messages that need a person, not a place to hide failures forever.

Red flags to avoid without correcting yourself: “exactly once to the user,” “one queue for everything,” “check opt-out in each worker,” “retry 429s immediately,” “wait for the provider before replying.”

After each push, close with one real part of the design you would build, such as a dedupe key, a separate urgent lane, or a token bucket, not “we will scale it later.”

What should stick

After working through the requirements, the decisions, and the depth sections, you should be able to explain:

  1. Accepting is not delivering. A 202 means saved and queued, not delivered by an outside network and not read by a person.
  2. Route before fan-out. Preferences, kind of message, quiet hours, template, and destinations decide which channel jobs exist.
  3. Separate lanes. Urgent messages must not wait behind marketing blasts, and the provider's allowance has to be shared on purpose.
  4. Dedupe beats exactly-once claims. Use logical keys, attempt records, and inbox collapse, and be honest about what outside networks can still do.
  5. Providers shape the system. Rate limits, bounces, invalid tokens, and failover rules are part of the design.

Tell it in the room: “Producers call POST with a dedupe key. We save and enqueue, return 202, route through preferences and templates, create channel jobs, and send through provider workers with rate limits and retries. Urgent and marketing lanes are separate. Poison jobs go to a dead-letter queue, and exactly-once to a human is not the promise.”

Practice next: Open Design a Notification System on Practice System Design and run a timed attempt using clarifying questions and one notification story, without peeking until you have a draft on the board.

Key Takeaways

  • Accepting a request and delivering a message are different promises, so save the request, return 202, and deliver afterwards.
  • Keep urgent messages (reset codes, receipts, security alerts) in their own lane with their own workers and a protected share of each provider's allowance, so a sale blast cannot delay them.
  • Check preferences, quiet hours, and kind of message in a router before creating any channel job, and record the preference version used.
  • Use a dedupe key on the logical notification, one job per channel and destination, and an attempt record, and do not promise exactly-once to a human.
  • Respect each outside sender's limit with token buckets, backoff with jitter, and circuit breakers; fail over only when it will not create a second copy.
  • Send permanent failures to a dead-letter queue with an alert and a replay path, and remove dead tokens and hard bounces.
  • Watch queue age by lane, provider errors, suppression rate, and dead-letter depth, not just provider 200s.

Quick revision notes

The problem in your own words. A notification system takes a request such as “tell this person their reset code” from another part of the product and gets it to their phone or inbox. The hard part is that accepted does not mean delivered, urgent messages must not wait behind sale blasts, and a retry must not send a second copy.

The design in one breath. The API checks the caller, drops a repeated request using a dedupe key, saves the notification, puts a routing job on a durable waiting line, and returns 202 Accepted. A router then applies the person's settings, the kind of message, and a pinned template version, and creates one send job per allowed channel. Urgent and promotional jobs travel in separate lanes with separate workers. Channel workers send through provider adapters within a token-bucket budget, retry temporary errors with backoff and jitter, send permanent failures to a dead-letter holding line, and write an honest status at every step.

The choices you must be able to defend. Save the request first and deliver later. Keep separate lanes for urgent and promotional traffic. Check preferences before fan-out and record the preference version used. Promise at-least-once processing with dedupe in layers, not exactly-once to a human. Give each outside sender a budget, and fail over only when it cannot create a duplicate. When preferences are unavailable, marketing fails closed.

The numbers worth remembering. Roughly 300 million sends a day averages about 3,500 a second, but a blast of 5 million in 5 minutes is about 17,000 a second. At 1,000 texts a second a 5-million-message blast takes about 83 minutes, which is why part of the allowance is reserved for urgent messages. All of these are estimates you would confirm with real provider limits.

The mistakes that cost points. Promising exactly-once, using one queue for everything, waiting for the provider before replying, checking opt-out in each worker, retrying 429s immediately, failing over without a dedupe key, and retrying broken messages forever.

Frequently asked follow-ups

  • How do email, SMS, and push fan-out work?
  • How do you prevent duplicate notifications during retries?
  • How do preferences and quiet hours apply?
  • How do provider rate limits and failover work?
  • Why is exactly-once notification delivery not a real promise?

Deep-dive questions and strong answer outlines

What does POST notification return?

It should return 202 Accepted after the request is validated, deduped, persisted, and queued. It should not wait for APNs, FCM, Twilio, or an email provider to deliver the message.

How do you route to multiple channels?

A router loads preferences, template metadata, priority, and available destinations. It creates channel jobs for push, email, SMS, or in-app only when allowed, then channel workers send through provider adapters.

How do you avoid duplicate sends?

Use a producer-provided idempotency key or dedupe key such as event id plus user id plus notification type. Store a send record and make retries reuse the same logical notification. Some providers may still deliver twice, so client or inbox dedupe also helps.

How do you handle provider limits?

Use per-provider and per-tenant token buckets, separate queues by channel and priority, exponential backoff with jitter on 429 and 5xx, and provider failover only when templates, numbers, domains, and compliance allow it.

What goes to the DLQ?

Poison messages such as invalid template variables, unsupported locale, malformed payload, or repeated terminal provider errors should go to a dead-letter queue with alerts and replay tooling. Retrying them forever blocks useful work.

AI feedback on your design

After a practice session, InterviewCrafted summarizes strengths, gaps, and interviewer-style expectations—similar to a written debrief. See a static example report, then practice this problem to get feedback on your own answer.

FAQs

Q: Are we building the phone alert networks, the text message carriers, or the email services?

A: No. Apple and Google run the phone alert networks (APNs and FCM), and companies such as Twilio and SendGrid run text and email delivery. Our system talks to them through small adapters. The design work is orchestration: durable enqueue, routing, templates, preferences, retries, rate limits, failover, and delivery tracking.

Q: Can we guarantee exactly-once delivery?

A: Not end to end. Networks timeout, providers retry, phones uninstall apps, and users may have multiple devices. Promise at-least-once processing with dedupe for the logical notification and honest status tracking.

Q: Should marketing and transactional messages share one queue?

A: They should not share one unbounded lane. Password resets, receipts, and security alerts need their own priority or queue so a marketing campaign cannot delay them.

Q: Where are user preferences checked?

A: Check preferences before channel fan-out. For marketing, opt-out and quiet hours must be enforced carefully and audited. Transactional messages may bypass some marketing preferences, but that rule should be explicit.

Practice interactively

Open the practice session to use the canvas and stages, then review AI feedback.