← Back to practice catalog

System design interview guide

Design a Rate Limiter

TL;DR:

Picture paying for something on your phone with a weak signal. The app does not hear back in time, so it sends the same checkout again, and again, until it has sent it twenty times in one second. The store would like to let a couple of those through and tell the rest to wait, so they do not all land on the payment system together. The part of the system that does this is called a rate limiter, and it only works if the store keeps one shared count of how many tries this person has made. If each of the store's front-door servers keeps its own private count instead, none of them sees the real total, and all twenty tries get through. A rate limiter slows the flood down, while a separate tool called an idempotency key makes sure the repeats still become one charge, so a good answer explains both.

Overview diagram for Design a Rate Limiter

Problem statement

Apps and websites get flooded with repeats—bad networks, buggy clients, or people guessing passwords. Design a rate limiter: the part that sits in front of your APIs (the doors an app calls) and answers one question before each request reaches the real work: is this caller still under their allowance?

If yes, let the request continue. If no, stop it and tell the caller to wait (usually HTTP 429, the standard “too many requests” code). You must know who the caller is (user, API key, or network address), remember how many times they called recently, and do this across many servers—not with a separate private count on each machine.

Interviewers care that you can explain who you limit, where the shared count lives, how two servers do not both approve the last request, what you return when someone is over limit, and what you do if the counter store fails.

Introduction

The problem: one phone, twenty retries

Picture paying for something on your phone while the signal is weak. The app does not hear back quickly, so it sends the same payment request again, and then again. Within a second it may have sent it twenty times. Nobody has done anything wrong here, because this is how apps are built to behave on a bad network. But the company running the store now has a problem. Twenty requests are about to land on its payment system, and it would much rather let a couple through and politely tell the rest to wait.

The part of the system that makes that call is called a rate limiter. It sits in front of an API (the doors your app calls to talk to the server) and, for every request, decides whether the caller has already used up their allowance for now. If they have not, the request carries on. If they have, the request is stopped and the caller is told how long to wait.

A rate limiter looks like a small piece of a system, and that is exactly why interviewers like it. The simple version takes five minutes to describe, and the real version is full of choices. This page walks through those choices in the order you would actually make them: first what we are solving and why, then what the system must do, then how big it is, and only after that the design itself.

Why teams bother with rate limiting

You have probably felt rate limiting as a user long before you thought about building it. After too many wrong passwords, a login page locks you out for a while. Email services stop you from sending hundreds of messages in a minute. Companies that offer a public API often say “you may call us 1,000 times per hour.” Behind each of those limits is a reason the company cares.

Protecting the system. One buggy app, such as one that retries instantly in a loop, can send a thousand times its normal traffic. Without a limit, that traffic can overload the servers and databases that everyone else depends on.

Fairness. One very busy caller should not slow everyone else down, so each caller gets a share of the capacity.

Security. Guessing passwords or scraping a site depends on sending huge numbers of requests. A limit turns a fast attack into a slow and expensive one.

Cost. Many requests trigger paid work, such as a payment, a text message, or a call to another company’s API. A cap keeps one mistake from turning into a large bill.

The phone retrying a payment is one more case of the same idea: protect the system, and keep one noisy caller from hurting everyone else.

What problem are we solving?

Before each request reaches the real service (checkout, login, search), the limiter asks a single question:

“Has this caller already used up their allowance for this minute (or hour)?”

If the caller is under the limit, the request goes through to the real service. If the caller is over the limit, the request is stopped. The caller gets an HTTP 429 response (the standard code for “too many requests”) and a Retry-After value that says how long to wait. This is different from 503, which means “our service is broken,” not “slow down.”

To answer that question, the limiter has to remember how many calls this caller has made recently. How to keep that memory, and where, is the heart of the design. We will not guess at it yet. First we need to be clear about what the system must do, and how well, so that each design choice later has a reason behind it.

One more point about the twenty payments, so the two ideas stay separate. A rate limiter slows the flood down. A different tool, called an idempotency key (a unique id the app sends so the server can tell “this is the same payment again”), is what makes sure twenty tries still become one charge. A strong answer explains both and says which one does which job.

This page is the interview deep dive in the system design interview guides hub. If the idea of limiting requests is completely new, skim the fundamentals series first.

How to approach

In an interview you do not start by drawing boxes. You start by turning a short, vague prompt into a clear contract: who is limited, what the system must do, how well it must do it, and what is out of scope. Only then do you size the problem, make the design decisions one at a time, and draw the result. The list below is the order this page follows.

How to design a rate limiter in an interview

  1. Clarify the problem: who is limited (user, API key, IP, route), whether short bursts are allowed, how exact the count must be, how big the system is, and what happens when something fails.
  2. Write the functional requirements (what the system does) and the non-functional requirements (how fast, how big, how reliable).
  3. Estimate the size: checks per second, number of distinct callers, memory per caller, and the time budget for one check.
  4. Make the design decisions in order: where the check runs, who counts as one caller, where the count lives, how to count, how to update safely, what to do when the store is down, and what the caller sees.
  5. Draw the picture those decisions produce, then walk one request through it.
  6. Go deeper where the interviewer pushes: the exact update step, hot callers, layered limits, what you measure, and one production failure.

Clarifying questions (say these in the room)

In a real interview the prompt is only a few words: “Design a rate limiter.” Before you design anything, ask questions until the problem is clear. Each answer below changes the design, which is why it is worth asking.

You: “Are we limiting by user id, API key, IP, route, or a combination? If someone is behind a shared office network, limiting only by IP can punish many people for one noisy neighbor.”

Interviewer: “Start with authenticated user id per route; you can mention IP as a secondary abuse signal for anonymous traffic.”

That answer fixes the unit you will count. It is a logged-in user on one route, which you can write on the board as a short label such as checkout:user:123. IP stays a rough signal for visitors who are not logged in, because a whole office can share one public address (this is called NAT).

You: “If a phone retries a few times on a bad network, can they go a little over the steady limit for a short moment, or is the limit strict every minute with no extras?”

Interviewer: “Yes, short bursts for retries are OK.”

So the product allows a small spike. This will decide how we count later, because some counting styles allow bursts naturally and others do not.

You: “Do different plans, such as free and paid, get different limits, and should the limiter live in the gateway, in each service, or in the client app?”

Interviewer: “Free and paid differ. Put the check at the gateway.”

That gives you two things to remember. The limit comes from a rule that depends on the plan, not a number fixed in code, which is why changing rules becomes its own job later. And the gateway (the single front door every request already passes through) is where the check runs, so you enforce once instead of in every service. A limit enforced only in the client app can be modified or ignored, so it is a courtesy and not a control.

You: “Roughly how many different callers are there, and how many checks per second at peak?”

Interviewer: “Assume about 100 million callers and about 10 million checks a second at peak.”

Those two numbers feed the capacity estimate in the next section. They also tell you early that one machine will not be enough.

You: “At peak, do we need exact counts, or is a small over-allow acceptable if backends stay protected? Exact counting costs more work per check.”

Interviewer: “Protect backends; a small over-allow is OK if you say the bound.”

Pick a bound you can say out loud and defend. A few percent over the limit in a one-minute window is a common product choice when backends stay healthy. The error of an approximate method depends on how spiky the traffic is, and large public deployments that use this approach have reported very small error rates, so name your ceiling and say you would measure it.

You: “If the part that keeps the counts becomes unreachable, should /login and /catalog behave the same?”

Interviewer: “You choose, and defend it.”

Two words come up here. Fail closed means block requests when the counts cannot be read, which is the usual choice for login and password reset. Fail open means let requests through, which is sometimes right for catalog reads where staying available matters more. The interviewer wants to hear you choose a different policy for each route, and say why.

You: “I want to keep network DDoS scrubbing, a long-term analytics warehouse, and client-only throttling out of scope so we can focus on server-side request limiting in the gateway. Does that match what you want?”

Interviewer: “Yes, focus on server-side request limiting.”

If the interviewer later asks about exact counts, store brands, or a tiny per-machine brake, you do not need to answer yet. Park those until the design decisions, where each one has a natural place.

Functional requirements

Functional requirements describe what the system does. From the answers above, a rate limiter must:

  1. Decide allow or deny for every request. For each request it looks at who is calling, how many calls they have made recently, and the limit that applies, then answers yes or no.
  2. Tell the caller what happened. A denied caller gets 429 and a wait time, so a well-behaved client knows when to try again. An allowed caller can be shown the limit and how much is left.
  3. Support different limits for different callers and routes. Free and paid plans differ, and a login attempt does not cost the same as a catalog search.
  4. Allow short bursts where the product wants them. A phone retrying on a bad network should not be punished for a few quick tries.
  5. Allow a bypass list. Some trusted callers, such as internal tools, should skip the limit, and an operator should be able to block an abuser quickly.
  6. Let operators change rules without redeploying. Changing “free = 100 per minute” should not require restarting every gateway.

Non-functional requirements

Non-functional requirements describe how well the system must do its job. They are the numbers and promises that shape the design.

  1. Speed. The check runs on every request, so it must add very little time, ideally well under a millisecond of the gateway’s own work. If the check is slow, every API in the company becomes slow.
  2. Scale. About 100 million distinct callers and about 10 million checks a second at peak. The check is the heavy flow.
  3. Accuracy across servers. The same limit must hold no matter which front-door server answers, so a caller cannot quietly multiply the published limit by the number of servers. A small, stated over-allow is acceptable. A large one is not.
  4. Availability. The limiter must not become the reason the whole site is down. When something in it fails, each route needs a clear rule for what happens next.
  5. Fairness. One very busy caller must not slow down the checks for everyone else.
  6. Visibility. The team must be able to tell whether the limiter is healthy, not just count how many requests it blocked.

Out of scope (and why you park these)

Write a short out-of-scope list on the board so the interviewer sees you can bound the problem.

Network-layer DDoS mitigation (CDN, WAF, scrubbing). Those systems absorb huge floods before traffic reaches your gateway. Different interview, different team.

Client-only rate limiting as the only control. Apps can be modified or ignored. Server-side enforcement is what makes the limit real.

A long-term analytics warehouse for every allow or deny. Useful later; building it here burns the round without proving you can protect the request path.

Perfect globally exact counts across every region with zero latency cost. Prefer regional budgets, sticky routing, or an explicit over-allow window, and say which accuracy you are buying.

That is the requirements stage. If you want to practice it before reading on, try the requirements stage on Practice System Design (about 5–10 minutes), then come back for the sizing and the decisions.

Minute pacing (about 40 minutes)

MinutesFocus
0–5Clarify the problem; write functional and non-functional requirements; agree out of scope
5–12Capacity, then the design decisions and the high-level boxes with one request story
12–25Counting style, stored keys, and the safe update step (the deep dive)
25–35Failure, hot callers, headers
35–40One production war story and questions

In the room (how you open): “I’d start by agreeing who we limit and what the product needs: who is counted, whether short bursts are allowed, how exact the count must be, how big it is, and what should happen if the part that keeps the counts fails. I’ll write the requirements on the board, size the problem, and then make the design decisions one at a time, so every box I draw has a reason.”

If you remember one thing: Ask first, write the requirements down, and only then design. Tell one request story before you recite command names.

Capacity estimation

Once the requirements are clear, the next step is to find out how big the problem is, because the size decides what kind of shared store you can afford. Every request has to pass through the limiter, so the check must finish so fast that users never feel it (often well under a millisecond). That means a few simple lookups, not asking every server to vote on every request.

Rough numbers to anchor the whiteboard (tune in the room):

InputOrder of magnitudeDesign implication
Aggregate checks/s10M+ (illustrative scale)Prefer a handful of simple key lookups—not heavy “search the whole history” work on every request
Distinct keys10²–10⁸ (clients × policies)Memory per key × expire idle keys with the window
Hot key QPSSingle client abusive or NATOne busy client can overload one slice of the store—plan to split that client’s counters or add a local + shared budget
Check budgetUnder about 1 ms at the 99th percentile (p99: 99 of every 100 checks finish at least this fast)Most of the time budget is the network hop to the shared store; keep connections ready
Shared-store shards10M checks/s ÷ roughly 100k simple operations per second per node ≈ 100+ primariesLua scripts cost more than a plain increment, so plan for headroom (often 150–300 primaries)

What the numbers imply

  • Shard count is part of the answer. One Redis node handles on the order of a hundred thousand simple operations a second. Ten million checks a second therefore means spreading counters across a hundred or more primary nodes, each owning a slice of the key space. Say this number out loud, and say it is an estimate you would confirm with a load test.
  • Be honest about the latency budget. A round trip to a shared store in the same availability zone usually costs a few tenths of a millisecond. A round trip across zones can already exceed a millisecond. So “under 1 ms” only holds with the store in the same zone as the gateway, pooled connections, and one round trip per check. When it cannot hold, that is the reason for a small local brake in front of the shared count.
  • Bandwidth is fine. Each check sends and receives a couple of hundred bytes, so 10M checks/s is roughly 2 GB/s across the whole fleet. Split over a hundred or more shards, that is only about ten to twenty megabytes a second per shard, which a normal network link handles easily.
  • ~10 GB of counter state at 100M keys × ~100 bytes—before replicas. Memory is a first-class design input. Real Redis adds per-key overhead, so budget closer to double that.
  • Forget idle keys when their window ends, or memory grows forever as old callers pile up.
  • Do not ask every server to vote on every request—that pattern (sometimes called consensus) is too slow for a sub-millisecond budget. Use one fast shared counter instead.
  • Prefer simple key operations, batching several commands on one connection (called pipelining), and when needed a short server-side script so a multi-step check still costs one network round trip.
  • Isolate hot keys early—one abusive caller (the principal you named in the key) can overload a single store node before the cluster-wide average CPU looks bad.

Checkpoint: What’s your check budget, and why not a multi-server vote per request?

If you remember one thing: Sub-ms budget means simple key ops—not asking every server to agree on every request.

Design decisions

We now know what the system must do, how fast it must be, and roughly how big it is. Now we choose. Each decision below answers one question that the requirements raised, and each one rules out an easier idea that would not survive the requirements. Read them in order, because every decision builds on the one before it.

Decision 1: Where does the check run?

The requirements said the limit should be enforced once, for every route, and before any expensive work happens. The one place every request already passes is the API gateway, the front door that handles encryption, finds the right service, and usually checks who is calling. So the check runs there, as the step just before the request is passed on.

Two other places were considered and set aside. Putting the check inside each service means every team writes and maintains its own copy, and they drift apart. Putting it only in the client app means anyone can change the app and skip the limit. A client-side check is still a good courtesy for well-behaved apps, but it cannot be the control.

Decision 2: Who counts as one caller?

Before the limiter can count anything, it needs a name for what it is counting. From the clarifying questions, the answer is a logged-in user on one route, written as a short label such as checkout:user:123. That label becomes the key under which the count is stored.

Using only an IP address would be simpler, but a whole office can sit behind one address, so one noisy person would block their colleagues. IP is kept as a rough signal for visitors who are not logged in.

Decision 3: Where does the count live?

The limiter has to remember how many calls each caller has made recently. The first idea most people have is the simplest one: each gateway keeps the tally in its own memory. It is fast, it needs no extra system, and for a single server it is exactly right.

It breaks as soon as there is more than one gateway. Suppose the promise is 100 requests per minute per user, and traffic is spread across 50 gateways. Each gateway sees only the requests that happened to reach it, so each one will happily allow up to 100. Across 50 gateways, one user can get about 50 × 100 = 5,000 requests through in a minute, which is fifty times the promise. That directly breaks the requirement that the same limit holds no matter which server answers.

Local counters on each gateway multiply the limit by the number of gateways while one shared counter enforces the real total

Figure: Private counters turn a 100-a-minute promise into about 5,000 a minute across 50 gateways. One shared counter holds the real line.

There are three ways out of this.

  • Keep the count in one place every gateway can reach. Every gateway reads and updates the same number, so nobody can get ahead of it.
  • Send each caller to the same gateway every time (called sticky routing). Then one gateway really does see all of that caller’s requests. The cost is that counts are lost when that gateway restarts, a busy caller piles onto one machine, and adding or removing gateways reshuffles who talks to whom.
  • Let the gateways keep telling each other their counts. In theory this works, but every message adds delay, and under heavy traffic the counts are always a little stale.

The first option is the simplest to reason about, so we choose it: one shared count. In most interviews that shared place is Redis, a very fast database that keeps its data in memory and can delete old entries by itself. It fits for three reasons: a lookup takes a fraction of a millisecond, old counts can expire on their own so memory does not grow forever, and it can run a small program as one uninterrupted step, which Decision 5 will need. The brand is optional. The property that matters is “one source of truth that every gateway can reach quickly.”

An optional extra, used later, is a tiny in-memory limit on each gateway in front of the shared count. It absorbs spikes and keeps working if the shared store is briefly unreachable. It is sometimes called a hierarchical limit (local first, shared second), and it never replaces the shared count as the real limit.

Decision 4: How do we count?

Now that the count is shared, we have to decide what exactly we count, because there is more than one way to do it. The answer comes from the requirement about bursts.

A fixed window counts calls in clock minutes and starts again at zero each minute. It is the cheapest, but it has an edge problem: at 12:00:59 someone can use the last of the old minute, and at 12:01:00 a fresh minute starts, so in a few seconds they can send about twice the promised rate. A sliding window looks at the last 60 seconds from right now, which is fairer at the edges but costs more work per check. A token bucket gives each caller a bucket of tokens that refills at a steady rate, spends one token per request, and allows a short burst up to the size of the bucket.

Because the product said short bursts for retries are fine, token bucket is the primary choice. Fixed window alone is a weak primary pick when the product wants spikes, and sliding window stays in reserve when fairness at the minute boundary is the real complaint. The full comparison, with the work each style costs, is in the decision matrix below. If an approximate version is chosen to save work, say the error bound out loud, as discussed in the clarifying questions.

Decision 5: How do we update the count safely?

Choosing a shared count creates a new problem, because now two gateways can touch the same number at the same moment. Picture two ushers both selling the last concert seat because each looked at the remaining count before either one updated it.

t0  Gateway A reads remaining → 99
t0  Gateway B reads remaining → 99
t1  Both allow → 100th and 101st both go through

Both gateways read “99 used, room for one more,” and both said yes. The count must therefore be read, checked and updated in one indivisible step, so that nothing else can slip in between the reading and the writing. This is a property we need, not a tool. The tool that gives it to us in Redis is a short script that Redis runs from start to finish before it does anything else, and the later sections show exactly what it looks like. For the first design, the property is enough: one step, never read-then-write.

Decision 6: What if the shared store is down?

The shared count now sits on the critical path of every request, so we must decide in advance what the gateway does when it cannot reach it. There is no neutral answer. Blocking everyone protects the backend but can look like an outage. Letting everyone through keeps the site up but removes the protection.

So the decision is made per route. For login and password reset, the gateway fails closed: it blocks requests, because a short outage must not become a free window for password guessing. For catalog browsing, it may fail open, letting requests through with the small local limit from Decision 3 as a safety brake, so the site stays up while the shared count is out of reach. Say the choice out loud and say why, because the interviewer is looking for that reasoning, not for a single right answer.

Decision 7: What does the caller see?

Finally, the limiter has to talk to the caller. When a request is denied, the gateway returns 429 with a Retry-After header and a short readable message, and it does so before calling the real service, so a flood cannot spend the backend’s capacity. Allowed requests can carry headers that show the limit and how much is left.

The wait time matters more than it looks. A bare 429 with no guidance teaches clients to retry immediately, and a throttle turns into a flood of retries. And 429 must stay different from 503: 429 means “slow down,” 503 means “we are broken,” and clients should treat them differently.

Putting the decisions together

Taken together, the seven decisions are the design. A check runs at the gateway. It builds a key for the caller and route. It asks one shared count, using a counting style that allows bursts, in one indivisible step. It then either lets the request through or returns a 429 with a wait time, and it follows a per-route rule when the shared count cannot be reached.

In the room (a fuller opening you can actually say): “I’d start by agreeing who we limit, usually a logged-in user id plus route, with IP only as a coarse signal for anonymous abuse. I’d ask whether the product needs short bursts for retries; if yes, I’d lean on a token bucket, and keep a sliding window in reserve when fairness at minute boundaries matters. Because there are many gateways, I’d keep the count in one shared store, often Redis, and update it in one indivisible step so two gateways cannot both approve the same last request. On failure I’d fail closed for login and password reset, and I might fail open for catalog reads with a tiny local safety brake. Every deny returns 429 with Retry-After so clients back off instead of hammering harder.”

The next section draws these decisions as one picture and shows who owns what.

High-level architecture

The decisions above were made one at a time. This section puts them on one page as a picture: which boxes exist, who owns what, and how one request travels through them. The deeper mechanics come later, so keep this at the level of a map.

How to approach this on the board

Do not start by listing every box. Draw one request left to right, then add the shared counter, then (only if asked) where humans edit limits.

  1. Client → load balancer → API gateway. Say the gateway ends the encrypted connection (TLS), routes the request, and often checks who is calling.
  2. Before expensive upstream work, the gateway asks: “Is this caller still under their limit?”
  3. To answer that, it needs one shared count (not a private counter per gateway). In interviews that store is usually Redis, a very fast in-memory store every gateway can reach.
  4. Allow → call the real backend. Deny → return 429 (“too many requests”) with Retry-After and stop—no checkout or login work.
  5. Only after that path is clear, add a small side note: humans set rates elsewhere; gateways keep a cached copy of those rules.

Say this out loud while you draw: “Same shared count for every gateway; check finishes before we spend money on the backend.”

Two jobs (keep them separate)

Think of the limiter as two different jobs—not one blob.

JobIn one sentenceHow often
1. Check“May this request go through?”Millions of times per day
2. Set rules“What is the limit for free vs premium, and what do we do if Redis is down?”A few times a day (or less)

Why split them?
If you mix them, you draw one messy box and then cannot answer: “Where does a PM change 100/min?” vs “What happens on every checkout call?” Interviewers push on both. Separating the jobs keeps the hot path tiny (fast check only) and puts slow human changes off to the side.

Job 1 — Check (every request):
Build a key → read the cached rule → one shared-count update → allow (with headers) or 429. This must stay fast.

Job 2 — Set rules (humans / ops):
Someone changes rate, burst, or fail-open vs fail-closed (config service or feature flags). Gateways cache those rules. Changing a limit is not a Redis write on every request.

You can say “check path” and “policy path.” Labels like data plane / control plane are optional—only use them if the interviewer does.

Boxes you draw (who owns what)

BoxOwnsDoes not own
API gatewayFront door: TLS, routing, auth, first abuse signals; calls the limiter before upstreamThe global counter truth
Rate limiter (library or thin service)Builds the key, applies cached policy, talks to Redis, formats 429/headersLong-lived product config UI
Redis (shared store)Authoritative counters; keys spread across a cluster (many nodes). Prefer the primary for incrementsDeciding product rates
Policy / config serviceCreate/update limits, canary a new rate, emergency global throttlePer-request allow/deny math

Library vs separate limiter service: Prefer an in-process library on the gateway calling Redis—fewest hops. Use a dedicated limiter tier only when many languages share one fleet and you need one implementation.

Measure or you’re blind: track check latency (p99) and allow vs deny counts. When Redis slows down, those metrics tell you before users only feel “site is weird.”

High-level rate limiter flow: Client to LB to API Gateway, sync check through rate limiter lib to Redis Cluster, then allow to upstream or 429; async control plane policy store to gateway cache

Figure: Sync path = allow/deny against Redis. Async path = policies refresh into the gateway cache; analytics never block the check.

In the room: Narrate one request end-to-end: key string → one Redis round trip (or Lua) → 200 with headers vs 429. Then: “If Redis times out, login fail-closed; catalog may fail-open with a tiny local brake.”

If you remember one thing: Draw the check before upstream first; Redis holds the shared count; policy edits are a side path cached on the gateway.

Follow-ups the interviewer may ask

After you draw the boxes, they often poke at ownership and failure—not algorithms yet. Short answers:

Interviewer: “Why not put the counter in each gateway’s memory? It’s faster.”

You: “Fifty gateways with a private count of 100 each become a 5,000/min limit. Memory is fine as a tiny local shock absorber; the shared Redis count is still the product promise.”

Interviewer: “Library on the gateway vs a separate rate-limiter microservice?”

You: “I’d start with an in-process library calling Redis—one less hop. I’d only add a dedicated limiter tier if many languages share one fleet and we need a single implementation.”

Interviewer: “Where do product managers change ‘free = 100/min’ without redeploying every gateway?”

You: “That’s the policy path—config service or feature flags. Gateways cache the rules with a short TTL. Changing a rate is not a Redis write on every request.”

Interviewer: “Redis is slow or times out—does the whole site die?”

You: “No—not the whole site. You choose per route. For login and password reset, block when Redis is down (safer—don’t leave the door open for password guessing). For catalog browsing, you may let traffic through so the site still works, plus a tiny local limit on that one machine as a safety net. Also watch how long the check takes—if it gets slow, metrics show it before users only feel ‘the site is weird.’”

Interviewer: “Can analytics or Kafka sit on the allow/deny path?”

You: “No. The allow/deny check has to finish before we call the real backend—and it has to stay fast. If every request waits on Kafka or an analytics write, Redis being healthy won’t matter: the queue slows down, fills up, or fails, and now checkout waits on logging. Decide first (build the key, one atomic Redis check, return 200 or 429). After that decision, you can sample or ship a small event to Kafka asynchronously—logging must not sit in front of the decision.”

How will you design the rate limiting library

With the boxes on the board, interviewers often zoom into one of them: the library (a bundle of code the gateway runs inside itself, or sometimes a small separate service) that every gateway calls to ask “is this caller allowed?”. They want to hear what that call looks like, which counting style it uses, and how it stays one fast step against Redis.

Interviewer may ask

Interviewer: “How would you design the rate limiting library the gateway calls on every request?”

You: “One small API: given who is calling and which route, return allow or deny plus headers. Inside: build a key, load the cached policy, run one atomic check against the shared store, then format 200 headers or 429 + Retry-After. The gateway stays dumb—it just invokes Check before upstream work.”

Interviewer: “What does that API look like?”

You: “Something like Check(principal, route) → { allowed, remaining, limit, reset_at, retry_after_sec }. Optional weight for expensive calls. Fail mode (fail-open vs fail-closed) comes from the policy, not hardcoded in the library.”

Interviewer: “Where does the counting algorithm live—gateway, Redis, or both?”

You: “The library chooses the algorithm from the policy, but the update must run as one step in Redis (a short Lua script that increments and sets the expiry together). If the library does GET then SET in two round trips, two gateways can both approve the last request.”

Interviewer: “Which algorithm do you ship first?”

You: “I’d ask about bursts. If mobile retries need a short spike, token bucket. If they only need a coarse abuse cap, fixed window. If minute-boundary fairness causes support tickets, sliding window (exact or approximate). I pick from burst shape and work per check—not the most impressive-sounding name.”

What the library owns

ResponsibilityIn the libraryOutside the library
Build tenant:route:user (or similar) keyYes—
Load rate / burst / fail modeFrom cached policyPolicy store / control plane
Atomic allow/denyYes (calls Redis)Redis holds counters
HTTP 429 + Retry-After + limit headersFormats decisionGateway returns the response
TLS, routing, authNoAPI gateway

Prefer an in-process library on the gateway. A dedicated limiter service is the same design with an extra network hop—only justify it when many languages need one shared implementation.

Decision matrix (pick the counting style)

Token bucket vs sliding window vs fixed window—pick from burst shape and work per check.

AlgorithmBurst behaviorWork per check (shared store)*Fairness at window edgeWhen to pick
Fixed windowWorst—can allow ~2N near boundariesLowest (increment + TTL)PoorCoarse abuse caps
Sliding window (exact)SmoothHighestBestSupport-ticket magnet on fairness
Sliding approxNear-smoothMediumGood enoughHigh QPS with admitted error
Token bucketSmooth up to capacity BMedium (server-side refill)GoodAPI products that promise bursts
Leaky bucketSmooth outflow; queues or rejectsMediumSteady rateWhen you want constant drain, not burst

*In interviews that shared store is often Redis.

Interview tip: After you pick, say burst shape + work per check in one sentence—that is how the library’s first algorithm earns its place.

The others in one breath (then go deep on token bucket)

Use the matrix above to pick. Details for fixed / sliding / leaky live in Rate limiting algorithms—here is only what you say on the board:

  • Fixed window — Cheapest (one increment plus an expiry, done together in one step). Say the edge risk out loud: near a minute boundary a client can get about 2N in a few seconds.
  • Sliding window — Fairest at boundaries; exact costs more Redis work. At huge request rates, admit an approximation bound and name it (a few percent is a common choice).
  • Leaky bucket — Steady drain into a fragile downstream (flat processed rate), not product “bursts.”

Fixed window empties at 12:00:59 and resets at 12:01:00, allowing roughly twice the published rate in two seconds; token bucket or sliding window smooths the spike

Figure: Minute boundaries can double a “100/min” promise in a two-second edge burst—why fixed window alone is a weak primary pick for API products.

Token bucket

Ship this first when the product wants short bursts—for example a phone that retries a few times on a weak network without being punished for every quick try.

Imagine each caller holds a small bucket of tokens (permission chips). Tokens refill at a steady rate r, and the bucket holds at most B chips (the burst size). Each allowed request spends one token, or a weight for expensive calls. When the bucket is empty, the request is denied with HTTP 429 (“too many requests”).

The hard part is that two gateways must not refill and spend at the same time and both think a token is still free. So each check must refill for the time that passed and then try to spend as one indivisible server-side step. When interviewers ask how, name that property first, then the tool: in Redis that step is usually a short Lua script (a small program Redis runs from start to finish before anything else), or another atomic server-side step.

Store remaining tokens and last update time so the next check knows how much to refill.

  tokens ████████░░  capacity B
  refill →→→ at rate r
  each allow consumes 1 (or weight)
  tokens == 0 → deny (429)

Ship token bucket as the primary when the product wants short bursts; keep fixed or sliding as a named fallback. Deep dive: Rate limiting algorithms.

If you remember one thing: The library is a small Check API plus one atomic Redis update; pick the algorithm for burst shape and work per check—not because the name sounds impressive.

Data model (Redis keys and policies)

The library needs somewhere to keep its counts, so the next question is what exactly gets stored. Saying “use Redis” (a very fast store that keeps data in memory) is not yet a design, because you have not said what the keys look like. Think of each Redis key as a named string that points at a small value (a count, or a few fields)—like a row id for one client’s budget.

One checkout request first: User 42 hits /checkout. Key string: rl:tb:checkout:user:42. Value: remaining tokens + last update time. The who part (user:42) is the principal; the checkout rule set is the policy. The table below is just other shapes of the same idea.

Counter keys (per-request store)

Key patternExampleFields / valueTTL
Fixed windowrl:fw:{policy}:{principal}:{yyyyMMddHHmm}integer count~2× window
Token bucketrl:tb:{policy}:{principal}hash: tokens, updated_at_msidle eviction (e.g. hours)
Sliding approxrl:sw:{policy}:{principal}:{slot}counts per sub-windowwindow + 1 slot
Hot-key shardrl:tb:{policy}:{principal}:{shard}same as parent algosame

Principal = who you already put in the key (user:123, key:abc, or ip:1.2.3.4). Policy = which rule set (route and tier) so /login and /search do not share one counter by accident. TTL (time to live) is the number of seconds after which Redis deletes an idle key by itself; it is often a bit longer than the window (for example ~2×) so a slow clock edge does not delete the key mid-read.

One rule that surprises people on a Redis cluster. A cluster spreads keys across many nodes, and a script can only touch keys that live on the same node. Redis decides the node from the key text, and it lets you force related keys together by wrapping the shared part in curly braces, called a hash tag. For example, if you write the keys as rl:tb:{user:42}:checkout and rl:tb:{user:42}:search, only the part inside the braces picks the node, so both keys always land on the same node and one script can check both. Keys for a different user or for a global budget will usually land elsewhere, so a layered check across “user, IP, and global” cannot be one script across nodes. In practice you do one script round trip per node, or you keep the layers that must be checked together on the same hash tag.

Policy records (config / control plane)

Store separately from counters (config DB or feature-flag payload), then cache on each gateway:

policy_id: "checkout_user_v3"
version: 17
algorithm: token_bucket
rate_per_sec: 10
burst: 50
scope: "route:/checkout + principal:user"
fail_mode: fail_closed   # or fail_open

Gateways should include version in metrics so mixed deploys are visible. Emergency overrides (whitelist / blocklist) are short-TTL keys checked first.

Example — hierarchical budget (local + Redis)

Published limit: 1,000 req/min for user:42.

  • Local gateway brake: ~50/sec on this machine only (shock absorber; can slightly over-allow across many gateways for a few seconds).
  • Shared global key: rl:tb:checkout:user:42 enforces the real 1,000/min.
  • If one user still overloads one machine in the store: Their counter usually lives on one node. Split that user’s budget across several keys (for example …:user:42:0 … …:user:42:15) so check traffic isn’t glued to one machine. (In Redis Cluster terms, that node ownership is a hash slot—optional label.) Splitting into 16 sub-keys must still sum to the same 1,000/min product limit—not 16 × 1,000. The simple way is to give each sub-key an equal share (about 62 per minute) and let each request pick a sub-key at random. The catch is that random picks are uneven over a short time, so one sub-key can run out while another still has room, and a caller can see a few early 429s even though their total is under 1,000. Say that tradeoff out loud: you are trading a little exactness for spreading the load across many nodes.

Local alone is not the product limit; Redis stays authoritative. Prefer checking the cheapest reject first, then the global key. Where the keys sit on the same node (same hash tag), do both in one network round trip with one script; where they sit on different nodes, expect one round trip per node and keep that count small.

Detailed design

We now know what is stored and where the rules live, so we can put the pieces in motion. This section follows one real request, a user tapping Pay, from the moment it reaches the gateway until the client either gets a normal answer or a polite “wait”. Then it shows the same thing from the caller’s side: the headers and responses a client actually sees, and the small admin API operators use to change rules. Read it once as a story. The exact script and the follow-up questions come in the deep dive right after it.

The Check path, step by step

The request arrives at the gateway, and the gateway first works out who is calling. It does this before any limiting, because a limit that belongs to a user is meaningless if the caller has not been identified yet. If the caller is not logged in at all, the gateway answers 401 and never touches the limiter, so a stranger cannot spend someone else’s allowance.

Next the library builds the counter key from three facts: the plan the rule applies to, the route, and the caller. For user 42 on /checkout that is rl:tb:checkout:user:42. It then looks up the rule for that key in the gateway’s own memory. The rule says things like “10 tokens per second, burst of 50, fail closed”. This lookup is local, not a network call, because the rules change a few times a day and the gateway keeps a fresh copy. That is the reason changing a limit never costs a Redis write on every request.

Then comes the only network call on the hot path. The library sends the key, the rule numbers, the cost of this request (usually 1) and the current time to Redis as one small script. Redis runs the whole script without letting any other command in, so the refill, the check and the spend all happen as one step. The script hands back three things: whether the request is allowed, how many tokens are left, and, if it was denied, how many seconds until enough tokens exist.

If the answer is allow, the gateway adds the limit headers and forwards the request to the real service. If the answer is deny, the gateway stops right there. It returns 429 with a Retry-After header and a short readable body, and it never calls the checkout service, so a flood cannot spend payment-system capacity. After the decision, and only after it, the gateway may drop a small event into a queue so people can later see who was blocked. That logging never sits in front of the decision.

If Redis does not answer in time, the library does not guess. It reads the fail mode from the rule. For login and password reset it denies, because a short outage must not become a free window for password guessing. For catalog reads it allows, using a small in-memory limit on this one gateway as a safety brake, so the site stays up while the shared count is dark.

  1. Identify the caller; unknown callers get 401.
  2. Build the key from plan, route and caller.
  3. Read the cached rule from local memory.
  4. Run one script in Redis: refill, check, spend.
  5. Allow with headers, or deny with 429 and Retry-After, before the backend is called.
  6. If Redis is slow, apply the route’s fail mode.
GET /v1/resource
  → Gateway: authenticate → limiter.Check(key) → Redis script
       → allowed → 200 + X-RateLimit-* headers → real service
       → denied  → 429 + Retry-After (stop; the real service is never called)

API design: what callers see and what operators change

Rate limiting is usually not a public API of its own in an interview. It is behavior added to the existing APIs, so the API design is mostly about three things: the call the gateway makes inside the system, what the client receives, and how operators change rules.

The internal call. Inside the gateway, the library exposes one small function. Given who is calling, which route, and how much this request costs, it returns the decision and the numbers the gateway needs to build the response.

Check(principal, route, cost = 1)
  → { allowed, limit, remaining, reset_at, retry_after_sec }

The fail mode (fail open or fail closed) is not a parameter here. It comes from the rule, so the same code can behave strictly on login and gently on catalog reads.

What the client receives. The gateway adds a few headers to responses so well-behaved clients can slow themselves down before they are blocked.

HeaderWhat it tells the client
X-RateLimit-LimitThe maximum number of requests allowed in the window for this rule
X-RateLimit-RemainingHow many are left; it can be approximate
X-RateLimit-ResetWhen the window resets (a Unix timestamp)
Retry-AfterHow many seconds to wait; sent with a 429

Some stacks are moving toward standardized RateLimit and RateLimit-Policy headers. In the room, naming the older X-RateLimit-* pattern is still widely understood, and you can say you would match whatever the company’s public API already documents. A 429 body should carry a machine-readable code, a human message and, optionally, retry_after_ms, which helps mobile clients.

Which status code means what. A 401 comes first, before the limiter, if the caller is not logged in. A 429 means “you are being throttled, slow down”. A 503 means “our service is overloaded or broken”. Keep 429 and 503 separate so clients do not back off in the wrong way. Retry-After on a 429 is part of the contract, not optional polish. A client should sleep for that time, add a little random delay (jitter), and then retry, rather than hammering immediately.

How operators change rules. A small internal admin API lets the team change limits without redeploying the gateways. The gateways pick up changes from their cached copy within a short time.

GET    /v1/admin/policies/{id}
PUT    /v1/admin/policies/{id}      # burst, rate per second, scope
POST   /v1/admin/overrides          # temporary allow-list / block-list

In the room: “For one request the gateway identifies the caller, builds a key from plan, route and user, reads the cached rule, and runs a single Redis script that refills the bucket and spends a token in one step. Allowed requests carry limit headers and go on to the service. Denied ones get a 429 with Retry-After and never reach the backend. Operators change rules through a small admin API, and gateways cache them. If Redis times out I follow the route’s policy: closed for login, open with a small local brake for catalog.”

Deep dive: the questions an interviewer asks next

You have drawn the design and walked one request through it. This is the point where a good interviewer starts digging, and the questions usually come in a fairly predictable order, because each answer raises the next one. Read this section as that conversation. Each question is followed by an answer you can give, plus a hint of where the interviewer is likely to go next. When the next answer is already clear from what you just said, you can offer it before they ask.

“You said the count is updated in one step. What does that actually look like?”

Start with what goes wrong without it. Two gateways read the same number at the same moment, and both act on it.

t0  Gateway A: GET count → 99
t0  Gateway B: GET count → 99
t1  A: SET 100   B: SET 100
    both allowed the 100th and 101st → over-allow

Both machines read “99 used, room for one more,” both thought they could take the last slot, and both allowed. The fix is to make the read, the check and the write one indivisible step, so nothing can interleave. In Redis that step is a short Lua script (a small program Redis runs from start to finish before it runs any other command on that node), so the refill, the check and the spend are one step and one round trip.

Here is a token-bucket version you could actually sketch in the room. It reads the clock from Redis itself, so every gateway agrees on the time, and it never trusts a caller’s clock.

-- KEYS[1] = rl:tb:checkout:user:42
-- ARGV[1] = refill rate per second, ARGV[2] = burst size, ARGV[3] = cost of this call
local t = redis.call('TIME')
local now_ms = t[1] * 1000 + math.floor(t[2] / 1000)
local rate, burst, cost = tonumber(ARGV[1]), tonumber(ARGV[2]), tonumber(ARGV[3])

local data = redis.call('HMGET', KEYS[1], 'tokens', 'updated_at_ms')
local tokens = tonumber(data[1]) or burst
local last = tonumber(data[2]) or now_ms

-- refill for the time that passed, but never above the burst size
tokens = math.min(burst, tokens + (math.max(0, now_ms - last) / 1000) * rate)

local allowed, retry_after = 0, 0
if tokens >= cost then
  tokens = tokens - cost
  allowed = 1
else
  retry_after = math.ceil((cost - tokens) / rate)   -- seconds until enough tokens exist
end

redis.call('HSET', KEYS[1], 'tokens', tokens, 'updated_at_ms', now_ms)
redis.call('PEXPIRE', KEYS[1], math.ceil(burst / rate * 1000) * 2)  -- an idle bucket is full again anyway
return { allowed, math.floor(tokens), retry_after }

To see the race closed, imagine one token left. Gateway A and gateway B send their scripts to Redis in the same millisecond. Redis runs A’s script completely, so tokens go from 1 to 0 and A is allowed. Only then does it start B’s script, which reads 0 and is denied. Neither gateway could see the other’s half-finished work, because there was no half-finished work to see.

Where it goes next: the interviewer will probably ask why it has to be a script at all.

“Why not just use a lock, or retry if someone else got there first?”

Because each alternative fails against a requirement you already agreed on.

  • A distributed lock around read-then-write adds two or three extra round trips and falls over when many gateways fight for the same hot key. That breaks the speed requirement.
  • Optimistic retry loops (read, then write only if nobody else changed it, otherwise try again) work, but a busy key makes everyone retry, and the retries add load on exactly the node that is already hot.
  • Every gateway voting on each request is too slow for a budget of about a millisecond.
  • Per-gateway counters are fast and wrong, as the 50 × 100 example showed.

A single script on the node that owns the key avoids all of that. It costs one round trip and no coordination.

Where it goes next: “Show me the simple version, for a fixed window.”

“What would the simple fixed-window version look like?”

It is much shorter, and it is where the most common slip happens. Calling INCR and then, as a separate command, EXPIRE is two steps. If the gateway crashes between them, that counter has no expiry and blocks the user forever. Put both in one script and set the expiry only when the count is 1:

local n = redis.call('INCR', KEYS[1])
if n == 1 then redis.call('EXPIRE', KEYS[1], ARGV[1]) end   -- ARGV[1] = window length in seconds
return n

Plain INCR is atomic by itself. The danger is only in separating it from the expiry. If you said “INCR and EXPIRE” in the room, this is the sentence that fixes it.

Where it goes next: “Walk me through a burst with real numbers.”

“Walk me through what a burst looks like with real numbers.”

Say the rule is 10 tokens per second with a burst of 50, and the bucket is full. A phone fires 60 calls at the same instant. The first 50 each spend one token and are allowed. The 51st finds 0 tokens, so the script returns retry_after = ceil(1 / 10) = 1, and the client is told to wait one second. One second later 10 tokens have refilled, so ten more calls fit. That one pair of numbers, a burst of 50 and then 10 a second, is what “smooth with bursts” means in practice.

Where it goes next: “What if the product needs fairness at the minute boundary instead?”

“What if burst isn’t the issue and fairness at the minute edge is?”

Then a sliding window fits better. The cheap version keeps two counters per client: the previous minute and the current minute. The estimate is previous × (share of the previous minute still inside the window) + current. With a limit of 100, 80 calls last minute and 40 so far, 30 seconds into this minute, the estimate is 80 × 0.5 + 40 = 80, so 20 calls remain. It assumes last minute’s calls were spread evenly, which is where the small error comes from, and it stores two numbers instead of a long list of timestamps. The exact version keeps every timestamp in a sorted set, which is fair but costs memory and CPU on every check.

Where it goes next: “What breaks first inside Redis when this runs at full scale?”

“What breaks first inside Redis at 10 million checks a second?”

Redis runs one command or script at a time on each node. A script that loops over a long history, or one hot key hit millions of times a second, uses up that node’s single thread while the rest of the cluster looks idle. So keep scripts tiny, never loop over history, and plan the hot-key split from the capacity section.

Where it goes next: “And if a Redis node dies?”

“What happens to the counts if a Redis node dies?”

Redis copies data to its replicas a moment after it writes, not at the same instant. If a primary dies and a replica takes over, the last few updates can be lost, so some callers briefly get more than their limit. That is the same brief over-allow the Failure handling section accepts. The script itself does not need to handle it, but your alerts do.

Where it goes next: the interviewer may now leave the core path and ask about product edge cases.

“How do you charge an expensive call, like a 100 MB upload?”

Give each request a cost. A 100 MB upload might cost 10 tokens while a metadata GET costs 1, so tiny calls cannot burn the whole minute while heavy work starves (or the reverse). State the weight rule when they mention uploads or a GraphQL query. The refill basics live under Token bucket.

“If a mobile app retries, does every retry cost a token?”

Not necessarily. If every retry burns a token, a flaky network throttles a user who never finished the write. Ask whether duplicate POSTs with the same idempotency key should consume quota once or every time. A strong answer is to charge once when the backend recognises the key, and still charge distinct operations. Do not invent silent free retries without stating the product rule.

Say it in the interview: “One indivisible check-and-update in the shard that owns the key, using the store’s own clock, with the expiry set in the same step. The cost is that a failover can briefly over-allow, and I would alert on that.”

Hot keys and Lua CPU are covered in Key challenges. Pool wait and check latency belong in Observability and Bottlenecks and tradeoffs.

If you remember one thing: The deep dive is a conversation. GET then SET is the red flag, one atomic Redis step is the fix, and each answer is something you could have said before they asked.

Key challenges

The deep dive showed that the core step is simple once it is atomic. What makes the real system hard is everything around that step, and six hard parts show up again and again. Name them early in the interview, because each one is a place the interviewer is likely to push.

Correctness versus cost. An exact sliding window keeps every timestamp, and at 10 million checks a second that is a lot of memory and CPU. Most teams therefore use an approximation and say how wrong it can be. Pick a ceiling you can defend (a few percent is common) and offer to measure it, instead of claiming perfect counts.

Hot keys. One very busy client, such as a single partner’s API key or a whole office behind one address, can send so many checks that the one Redis node owning that client’s key runs at full CPU while the rest of the cluster looks healthy. The fix, splitting that client’s budget across several keys, belongs in the design from the start and not in the post-incident notes.

Layered limits. Real products stack a per-user limit, a per-address limit and a global limit. Checking them one after another costs one network round trip each. Keys that must be checked together should share a hash tag so one script can read them all, and the cheapest rejection should run first. Keys on different nodes cost one round trip per node, so keep the number of layers small.

Clocks. Use the shared store’s clock for window edges, never the client’s phone clock, which anyone can set. Across regions a small difference between server clocks can shift window edges slightly. A token bucket handles this well because it only needs “how much time passed since the last update” as measured by one clock. If an interviewer raises clock skew, you can say: “Client clocks never enforce anything, and the store’s clock decides.”

Multi-tenant fairness. One noisy customer on a shared Redis cluster can raise latency for everyone else. Teams answer with per-tenant quotas, key prefixes that make heavy tenants easy to find, and a dedicated cluster for the largest customers.

Multi-region consistency. If each region keeps its own Redis, a user who moves between regions can get roughly the limit times the number of regions for a short while. With 3 regions, a limit of 100 per minute and no sticky routing, a hopping client can briefly reach about 300 per minute. You can keep each user on one region’s counters (sticky routing), size each region’s budget for that drift, or use one central authority for strict global caps and accept the extra latency. Say the over-allow window you are willing to accept instead of claiming perfect global exactness.

If you remember one thing: Hot keys and layered limits are design requirements—not production surprises.

Scaling the system

Those hard parts get worse as traffic grows, so the next question is what happens when the number of callers multiplies. You do not have to wait for the interviewer to ask. Right after you finish the design, say the scaling story yourself, then be ready for the follow-ups.

What you can say upfront: “The gateways scale by adding more of them, because they keep no important state. The shared counts scale by splitting the key space across many Redis nodes, each owning a slice. At about 10 million checks a second that is on the order of a hundred or more primaries. Sharding spreads many different callers well, but it does not fix one very hot caller, so I would split that caller’s budget across several sub-keys. I would send increments to the primary, not to replicas, and across regions I would either keep a user sticky to one region or accept a stated over-allow window.”

That one paragraph answers most of what follows. Here are the follow-ups, in the order they usually come.

Interviewer: “Traffic doubles next quarter. What do you change?”

You: “The gateways first, because they are the easy half. They hold no important state, since the real counts live in Redis and the rules are a cached copy, so I add more of them behind the load balancer and nothing else has to change. Then I check the Redis side: shard count, memory, and the busiest keys. If the cluster has headroom, nothing changes there either.”

Interviewer: “How do you scale the Redis side?”

You: “I split the key space across many primary nodes by hashing the key, so each node owns a slice. With millions of different callers the load spreads evenly on its own. When the cluster is large enough that running it is the real cost, some teams move to a dedicated limiter tier that batches many checks into one network call. I would size it from the capacity numbers and confirm with a load test.”

Interviewer: “One partner sends 40,000 checks a second under a single API key. What happens?”

You: “That is the case sharding does not solve. Every check for that key goes to the one node that owns it, and Redis runs one script at a time per node, so that node saturates while the rest of the cluster looks idle, and other customers whose keys live there get slow checks. I would split that partner’s budget across several sub-keys so the work lands on several nodes, give very large tenants their own quota or cluster, and add a small local limit on the gateway for obvious abusers. The cost is that exact counting gets a little looser, because each sub-key holds only a share of the budget.”

Interviewer: “Can we read from replicas to spread the load?”

You: “Not for increments. A replica lags behind its primary, so it would hand out stale counts and let extra requests through. I keep increments on the primary unless I am deliberately building an approximate local cache and can state the error.”

Interviewer: “What about more than one region?”

You: “Counts that are slightly stale across regions are fine for many products, since being off by a few requests for a few seconds rarely matters. A strict global cap needs one central authority or sticky routing, and I would say plainly that I am trading latency for accuracy.”

If you remember one thing: Gateways scale statelessly; Redis scales until you need sharding and abuse isolation.

Failure handling

Scaling answers “what if there is more traffic”. This section answers “what if something breaks”. Again, say it first. Failures here come from the shared store, from rules changing, or from clients reacting badly to being limited, and what impresses an interviewer is that the answer was decided before the failure happened.

What you can say upfront: “Every failure has a decision made in advance. If Redis is slow or down, each route has its own rule: closed for login and password reset, open with a small local brake for catalog reads, and a circuit breaker so gateways stop waiting on a dead store. If Redis fails over and loses a few updates, some callers briefly get more than their limit. I accept that and alert on it. Rules are versioned and rolled out to a few gateways first. Every 429 carries a Retry-After with a little random jitter so clients do not all return at once.”

The follow-ups are about the details of those decisions.

Interviewer: “Redis times out. What does the gateway do?”

You: “It follows the route’s rule rather than guessing. For login and password reset it denies, because a short outage must not become a free window for password guessing. For catalog reads it allows, with a small in-memory limit on that gateway as a safety brake, so the site stays up. After repeated failures it uses a circuit breaker, which trips like a fuse so the gateway stops waiting on a dead dependency and decides immediately. Users would see slower responses or more 429s, which is degraded service, not an outage. An outage is gateways returning errors because they have no rule to apply, and I avoid that above all on login.”

Interviewer: “Redis fails over to a replica. What did you lose?”

You: “Redis copies each write to its replicas slightly after it happens. If a primary dies and a replica is promoted, the last few updates are gone, so some callers get more than their limit for a short time. Nobody is locked out, but protection is weaker for a moment. I accept that, tighten limits briefly if the backend is fragile, and alert when the 429 rate drops while traffic rises.”

Interviewer: “You push a new rule. What can go wrong?”

You: “During a deploy some gateways run the old rule and some run the new one, so the same caller can see two different limits. That is harmless for a minute and confusing for an hour. I put the rule version in the metrics, roll out to a small slice of gateways first, warm the rule cache before sending full traffic, and keep a one-step rollback.”

Interviewer: “A client keeps retrying the moment it gets a 429. What then?”

You: “That is a retry storm, sometimes called a thundering herd: many clients return at the same second. I always send Retry-After and add jitter, a small random extra delay, so they spread out. I also count denied requests as their own load, because a flood of rejections can use up gateway CPU and Redis connections, and for repeat offenders I would block cheaply at the edge for a short time.”

Interviewer: “Every gateway’s cached rules expire at the same time. What happens?”

You: “They all call the config service together and flood it, which is a cache stampede. I stagger the expiry, refresh in the background before the hard expiry, and when many requests miss at once I let only one fetch go to the config service. That last trick is called singleflight.”

ScenarioBad outcomeMitigation
Redis timeoutUnbounded traffic (fail open) or an outage (fail closed)Per-route fail mode; circuit breaker; small local brake
Redis failover or wipeCounters reset, brief over-allowAccept, tighten limits briefly, alert on a sudden drop in 429s
Gateway deployMixed rule versionsVersion in metrics, gradual rollout, quick rollback
Retry storm after 429All clients return togetherRetry-After with jitter, edge block for repeat offenders
Policy cache stampedeConfig service floodedStaggered expiry, background refresh, singleflight

If you remember one thing: Fail-open vs fail-closed is a product/security choice—never “we’ll decide later.”

Observability (what you measure)

Once the system is live, you need a way to tell whether it is healthy, and that takes more than one number. Do not watch only how often callers get HTTP 429 (“too many requests” — the number of times they were told to wait). That misses the failure mode where everything is slow and almost nobody is throttled.

SignalWhy it matters
limiter.check p99 / p999 (how long the slowest 1 in 100 and 1 in 1,000 checks take)The tax every request pays on the critical path; averages hide the slow tail
Redis error rate + pool waitDistinguishes “Redis down” from “pool starved”
Top keys by QPS / Lua CPUFinds hot keys before users blame “the app”
429 rate by route and policy versionCatches bad deploys and mixed versions
Allow/deny sampling (async)Debug without blocking the hot path

Alert ideas: limiter.check p99 above budget; single key above shard CPU threshold; sudden drop in 429s during a traffic spike (possible fail-open or counter reset).

Security and abuse

A limiter is also a security tool, and people will try to get around it, so this section covers how to keep it honest. Check who the caller is before you apply a per-user limit, so a stranger cannot burn someone else’s allowance. Prefer an API key or user id over a raw network address when many people share one office address (this shared-address setup is called NAT); still keep a coarse address or edge limit for floods from callers who are not signed in. Stolen keys need a fast revoke path in the rules system so operators can cut them off. Do not treat the app limiter as a replacement for a web firewall (WAF) or network flood scrubbing (DDoS protection)—those sit further out and buy different protection. On login and password-reset paths, fail closed (block when the counter store is unreachable) so a brief Redis outage does not become an open window for password guessing.

Cost awareness

Every choice so far also has a price, and interviewers like to hear that you noticed. Every allow that should have been a deny wastes backend CPU, database work, and support time. Every deny that should have been an allow wastes sales and trust. Redis memory and the CPU spent running short scripts are real dollars at about 10 million checks a second—so approximate counts and a small local cap in front of the shared store exist to buy protection without paying for exactness everywhere. Returning 429 without calling the real backend is the first cost win; isolating hot customers is how you stop one tenant from making the shared cluster expensive for everyone else.

Green Dashboard. Angry Users. The Production Mindset System Design Interviews Expect.

Everything so far describes how the limiter should work on a good day. This section is about what happens on a bad day, told as short stories from the people who carry the pager (the on-call engineers). Textbooks teach boxes and arrows. On-call teaches something else: a limiter can look healthy while real users are stuck, and it can look busy while it protects nothing.

A low 429 count can mean “nobody is abusing us” or “the limiter stopped counting”. A calm Redis dashboard can mean “all is well” or “every check is timing out and falling open”. Interviewers like this gap because it separates a diagram from someone who has run the thing.

Four nightmares that look fine on a dashboard:

What the dashboard showsWhat is actually happening
429s are near zero and the backend is on fireRedis failed over, the counters reset, and the gateways are failing open
Redis cluster CPU averages 30%One node owning a partner’s key sits at 100% and its checks are timing out
429 rate is steadyA bad deploy gave half the gateways a stricter rule, and real customers are blocked
Checkout latency is up and nobody is throttledEvery request waits on the limiter’s connection pool

Scene 1: Quiet 429s, loud backend

The moment: It is a flash-sale morning. The backend pager goes off, order latency doubles, and the limiter dashboard says nothing is being throttled.

The trap: Redis lost its primary overnight and a replica took over a few minutes ago. Its counters were behind or empty, so every caller looked brand new. On top of that, catalog routes fail open when Redis times out, so the first slow minutes let everything through. The metric that lied was “429 count”, because a limiter that cannot count also cannot deny.

What to build:

  • Alert on a sudden drop in 429s while total traffic rises, not only on high 429s.
  • Alert on the Redis error rate and on check latency, so you learn about the failover before the backend does.
  • Keep the small local brake on gateways so a counter reset cannot send the full flood at the backend.
  • After any failover, tighten limits for a few minutes on fragile routes.

Scene 2: One partner, one overloaded node

The moment: A large partner launches a bulk job. Their integration sends 40,000 checks a second under one API key. Support starts hearing “your API is slow”, but only from some customers.

The trap: Every check for that key goes to the same Redis node. That node’s single thread is saturated, so every other customer whose key happens to live on it also gets slow checks. The cluster-wide average CPU looks fine because 99 other nodes are idle.

What to build:

  • Track the top keys by checks per second and by script time, so the hot key shows up in minutes.
  • Split that partner’s budget across several sub-keys, and give very large tenants their own cluster or a clear quota.
  • Put a short in-memory limit in the gateway in front of the shared check for keys that are clearly abusive.

Scene 3: The deploy that blocked paying customers

The moment: A new rule goes out: free users get 100 requests a minute, paid users 1,000. Twenty minutes later paid customers say they are being throttled at 100.

The trap: Half the gateways have the new rule and half still have the old one. A bug in the new rule’s plan lookup sends paid users to the free plan’s rule on the updated half. Overall 429 rates look normal because the free users are fine.

What to build:

  • Put the rule version in every limiter metric and compare 429 rates by rule version and plan.
  • Roll rules out to a few gateways first, watch those numbers, then continue.
  • Keep a one-step way to roll back a rule without redeploying the gateways.

Scene 4: The retry storm after a 429

The moment: A popular mobile app ships a bug where it retries instantly on any error. Every user who hits the limit comes back immediately, and again, and again.

The trap: The 429 responses had no Retry-After, so the clients had nothing better to do than hammer. The limiter is doing its job, but the denial traffic alone is now loud enough to use up gateway CPU and Redis connections, and the check latency rises for everyone.

What to build:

  • Always send Retry-After, with a little jitter so clients do not all return in the same second.
  • Count denied requests as their own load: watch gateway CPU and connection-pool waits during a spike.
  • For repeat offenders, move them to a cheap edge block for a short time so they stop reaching Redis at all.

In the interview: Pick one scene and tell it like a short story: what people felt, which metric lied, and what you would build next. You do not need all four. You need to sound like someone who knows that green does not mean done.

Bottlenecks and tradeoffs

Those stories all trace back to a small number of choices where you cannot have both things you want. Interviewers listen for whether you can name the tension, say what breaks, and pick a side on purpose. Here are the three that matter most for a rate limiter.

Exactness vs throughput

The tension — A sliding window that looks at the last full minute feels fair to callers, but keeping every timestamp exact costs more Redis work on every check.

What breaks — At about 10 million checks a second, the slowest checks for Check (the p99) blow the sub-millisecond budget you promised.

What teams do — Use an approximate sliding window, or a token bucket that refills inside a short Redis script, and say out loud how wrong the count is allowed to be.

Say in the interview — Name the trade as fairness versus Redis CPU—not only “we use sliding window.”

Central store vs edge pre-check

The tension — A check that lives only on the gateway edge is fast, but a true global cap needs a round trip to a shared store.

What breaks — Pure edge counting under-enforces (each machine multiplies the limit). Pure central counting adds network time on every request.

What teams do — Use two layers (sometimes called hierarchical limits): a tight local hard cap on this gateway, then a Redis global refine, ideally in one short script pipeline.

Say in the interview — Draw two layers when they ask about 50 gateways sharing one product promise.

Availability vs abuse on failure

The tension — Fail-open (let traffic through when Redis is dark) keeps the site earning money; fail-closed (block when Redis is dark) stops password guessing during a blip.

What breaks — The wrong default on login during a Redis outage becomes a security incident.

What teams do — Choose the policy per route, trip a circuit breaker so gateways stop waiting on a dead store, and only cache a short local decision where it is safe.

Say in the interview — “Catalog reads may fail open; login fails closed”—and mean it.

If you remember one thing: The limiter can become the bottleneck that protects nothing if Redis or the gateway queue is ignored.

Interview tips

You now have a full design and the tradeoffs behind it, so the last thing to practice is defending it when the interviewer pushes back. These five are the pushes that come up most after you finish the design:

Each gateway keeps its own count — your limit multiplies by the fleet

In the room

You might say: “Each API server keeps its own count in memory for every client. That is fast and simple.”

Interview Push: “You have 50 gateways and a published limit of 100 requests per minute per user—how many requests can that user actually get through?”

Land here: Do the math out loud so the gap is obvious. Each gateway allows up to 100 before its counter fills. Spread across 50 machines, that is roughly 50 × 100 = 5,000 requests per minute—fifty times the product promise. Local memory is not shared; Gateway A never sees what Gateway B already allowed. Put the real counter in Redis (or another shared store every gateway can reach) and update it in one indivisible step. A tight local hard cap is fine as a shock absorber only if a global Redis check still enforces the real budget.

Redis is down—do you let everyone through?

In the room

You might say: “If Redis fails, we fail open—we let traffic through so the site stays up.”

Interview Push: “Is that OK for password reset and login?”

Land here: Letting everyone through when the counter store is down (fail-open) keeps the site available but invites abuse—password guessing loves an open door. Blocking everyone (fail-closed) protects sign-in paths but can look like an outage for catalog reads. Strong answers pick different policies per route: fail-closed on login and password reset; maybe fail-open on read-heavy catalog with a tight local safety cap. Never treat one global default as “neutral.”

A bare 429 teaches clients to hammer harder

In the room

You might say: “We return HTTP 429 when the user is over limit.”

Interview Push: “What should the client do next—and what happens if you give no guidance?”

Land here: A bare “too many requests” answer is not enough. Return a Retry-After header (seconds or an HTTP date) plus remaining, limit, and reset headers when you can. Without that wait hint, well-meaning clients and broken retry loops retry immediately, which turns a throttle into a flood of retries (a thundering herd). Distinct status codes matter too: 429 means “slow down”; 503 means “our service is unhealthy”—clients should not treat them the same.

Every retry burns a token—even when the first try succeeded

In the room

You might say: “Every request counts against the limit, including retries.”

Interview Push: “A phone on a bad network sends the same payment twice because the first response got lost. Should the second one cost quota?”

Land here: Counting every retry punishes a user for a network problem, and it can lock them out right after a successful write. The usual answer is an idempotency key, a unique id the client sends so the backend can recognise “this is the same operation again”. If the backend has already seen that key, the retry should not spend quota again, and it should not be processed again either. Also say what a rate limiter does not do: it slows a flood, but only the idempotency key prevents twenty separate charges. A good answer names both and says where each one lives.

One hot client, one hot Redis node

In the room

You might say: “We spread Redis keys across many nodes by hashing the key, so the load spreads evenly.”

Interview Push: “One partner sends 40,000 requests a second under a single API key. What happens to the node that owns that key?”

Land here: Spreading keys across nodes (sharding) helps when you have many different keys, not one very busy key. All 40,000 checks a second still go to the single node that owns that key, and Redis runs one command or script at a time per node, so that node saturates while the rest of the cluster looks idle. Other customers whose keys live on that node get slow checks too. The fix is to split that client’s budget across several sub-keys so the work lands on several nodes, to give very large tenants their own quota or cluster, and to add a small local limit on the gateway for obvious abusers. Say the cost out loud: sub-keys make exact counting a little looser, because each sub-key holds only a share of the budget.

Red flags to avoid without correcting yourself: “GET then SET,” “each node counts locally for a global cap,” “429 with no Retry-After,” “one fail-open policy for every route.”

After each push, close with one real part of the design you would build—a shared Redis key, a Lua check, a route-level fail mode—not “we will scale it later.”

What should stick

After working through the requirements, the decisions and the depth sections, you should be able to explain:

  1. Shared state — A product-wide limit needs one authoritative counter per caller key (or a clear two-layer hierarchy with a local brake). Counters kept only in each gateway’s memory multiply the published limit by the number of machines.
  2. Algorithm choice — A fixed clock-minute window is cheap but spiky at the edges; a token bucket fits product bursts; a sliding window fits boundary fairness; a leaky bucket fits a steady drain. Details: Rate limiting algorithms.
  3. Atomicity — One short Lua script (a small Redis-side program) does the read, the check, the update, and the expiry in a single round trip. Never read then write in two steps, and never run INCR and EXPIRE as two separate commands.
  4. HTTP contract — Denied callers get 429 plus Retry-After and limit headers, which is different from 503 (“we are broken”). Deny before the real backend does any work.
  5. Failure is policy — Choose fail-open or fail-closed per route; plan how to split hot keys across nodes; watch check latency (limiter.check p99) and top keys, not only the 429 count.

Tell it in the room: “Every request: the gateway builds tier:route:principal, runs one atomic Redis check, then either returns 200 with limit headers or 429 with Retry-After without calling upstream. I’d use token bucket when the product needs bursts. Login fails closed if Redis times out; catalog may fail open with a tight local shadow cap. One hot client id means I shard that counter key so one Redis slot cannot take down the cluster.”

Quick check before a mock: (1) 50 gateways × 100/min local = ? (2) Why is GET-then-SET wrong? (3) Login fail mode when Redis is down? → (1) ~5,000/min (2) two gateways both allow the last request (3) fail-closed.

Practice next: Open Design a Rate Limiter on Practice System Design and run a timed attempt using clarifying questions and one request story—without peeking until you have a draft on the board.

Key Takeaways

  • Every gateway must share one authoritative count per caller; private counts multiply the limit by the number of servers (50 gateways × 100 per minute allows about 5,000).
  • Update the count in one indivisible step, using a short Lua script that checks, spends and sets the expiry together; never read then write, and never INCR then EXPIRE as two commands.
  • Pick the counting style from the burst shape: token bucket for product bursts, sliding window for boundary fairness, fixed window for coarse caps.
  • Answer with 429 plus Retry-After before the backend does any work, and keep 429 (“slow down”) separate from 503 (“we are broken”).
  • Choose fail-open or fail-closed per route: closed for login and password reset, possibly open with a small local brake for catalog reads.
  • Plan for roughly a hundred or more Redis shards at 10 million checks a second, and plan how to split one very hot key.
  • Watch check latency, Redis errors, top keys and 429s by rule version, not just the 429 count.

Quick revision notes

The problem in your own words. A rate limiter answers one question before every request: is this caller still under their allowance? The hard part is that many gateways must agree on one count, and two of them must never both approve the last request.

The design in one breath. The gateway identifies the caller, builds a key from plan, route and caller, reads its cached rule, and asks Redis to run one short script that refills, checks and spends in a single step. Allowed requests carry limit headers and go on to the backend. Denied ones get a 429 with a retry time and never reach the backend.

The choices you must be able to defend. Token bucket when the product allows short bursts, sliding window when boundary fairness matters, fixed window only for coarse caps. Fail closed on login, and possibly fail open with a small local brake on catalog reads. A few percent of over-allow is acceptable if you name the number and watch it.

The numbers worth remembering. 50 gateways × 100 per minute allows about 5,000. About 10 GB of counters for 100 million clients, and a hundred or more Redis shards for 10 million checks a second. One key lives on one node, so one very hot client needs its budget split.

The mistakes that cost points. Counting per gateway, reading then writing in two steps, calling INCR and EXPIRE as two commands, sending a 429 with no Retry-After, using one fail mode for every route, and ignoring what happens after a Redis failover.

Frequently asked follow-ups

  • Token bucket vs sliding window—when do you pick each?
  • How do you implement distributed rate limiting without crushing Redis?
  • What happens when two gateway nodes race on the same key?
  • How do you handle a hot key for one abusive client?
  • Fail open or fail closed—what do you choose for an API vs a login endpoint?

Deep-dive questions and strong answer outlines

How does a token bucket work for HTTP APIs?

Tokens refill at a steady rate; each request consumes one (or weighted). Allows smooth bursts up to bucket capacity. Contrast with fixed window where a client can spike at window boundaries.

Where do you store counters at 10M checks/s?

In-memory per process is wrong for global limits. Use a fast remote store (often Redis) with sharding by key, pipelining, and TTL aligned to windows. Mention hot-key mitigation (sub-keys, regional budgets) if pushed.

How do you make increments correct under concurrency?

INCR on its own is atomic, but INCR followed by a separate EXPIRE is two commands, and a crash between them leaves a counter that never expires. Put both steps in one small Lua script (set the expiry only when the count is 1), or use a script for check-and-set in one round trip. For sliding windows, use sorted sets or approximate counters and admit the error bound if they ask for scale.

What do you return when limited?

HTTP 429, meaningful body, Retry-After seconds or timestamp, optional X-RateLimit-Remaining and reset time. Helps clients backoff without hammering.

How do layered limits (per user + per IP + global) compose?

Check innermost budget first for cheap rejection, then outer caps—often nested keys or a pipeline of checks in one Redis round trip with Lua. Define precedence when budgets disagree (e.g. authenticated user id beats IP for fairness).

How do you test rate limits without flaky tests?

Deterministic clock injection for windows, unit tests on pure counter math, and integration tests with controlled Redis or an embedded fake with the same semantics—not wall-clock sleeps in CI.

AI feedback on your design

After a practice session, InterviewCrafted summarizes strengths, gaps, and interviewer-style expectations—similar to a written debrief. See a static example report, then practice this problem to get feedback on your own answer.

FAQs

Q: Do I need exact counts or is approximate OK?

A: Many production systems accept a small error (for example sliding-window approximations) so they can stay fast at huge scale. Say the tradeoff out loud: exact Lua scripts cost more CPU per check; approximate structures and a little local leakage cost less and still protect backends.

Q: Is Redis always the answer?

A: Often, because of atomic ops and TTL. Alternatives include a dedicated limiter service, an edge SDK with sync, or hierarchical limits (a tight local cap plus a global Redis refine). Show why you picked one for this product, not that Redis is a default stamp.

Q: How is this different from a queue?

A: Rate limiting rejects or delays excess traffic on the request path; queues buffer work for later. You may combine them (429 vs 503 with a queue upstream), but the usual interview question is synchronous allow or deny.

Q: How do I talk about global vs per-region limits?

A: Exact global counts across regions are hard because of latency. Options include sticky routing, regional budgets that can drift for a short window, or a central authority for strict global caps—name consistency versus latency.

Q: What happens if each gateway keeps its own rate limit counter?

A: Local counters multiply the published limit by the number of gateways. With 50 gateways and a 100/min cap, a user can reach about 5,000/min. Global limits need one shared authoritative counter (usually Redis) updated atomically.

Q: Why is GET then SET wrong for distributed rate limiting?

A: Two gateways can both read the same remaining count and both allow the last request, over-allowing the cap. Use one atomic step instead: a small Lua script that increments and sets the expiry together for fixed windows, or one that refills and spends tokens for a token bucket, in a single Redis round trip.

Practice interactively

Open the practice session to use the canvas and stages, then review AI feedback.