← Back to practice catalog

System design interview guide

Distributed Cache System Design

TL;DR:

Picture a shop's website during a flash sale. One discounted product ends up in almost every shopper's cart, so every page asks the shop's database for the same product details, thousands of times a second. The shop would rather keep ready-made copies of those popular answers in memory on a group of machines, so most pages never wake the database. That group is called a distributed cache. It only works if the copies stay useful: when every read for that one product lands on a single cache machine, or when a missing copy sends one more identical question on to the database, the cache machines can still look calm while the database melts.

Overview diagram for Distributed Cache System Design

Problem statement

Busy applications ask the same questions of a database over and over—product pages, profiles, carts. Design a shared in-memory cache for that traffic: a group of machines that keep copies of frequently read data so most reads never reach the database.

The cache has to decide which machine holds which item, keep a second copy so one machine dying does not hurt, drop old values when memory fills up, and stay useful when one item becomes extremely popular or when many requests miss the same item at once. The database remains the official record (the source of truth) the whole time.

Interviewers care that you can explain how a key finds its machine, what happens when a machine is added, how a database change clears the cached copy, and what protects the database when the cache misses.

Introduction

The problem: one product, every cart

Picture a shop that runs a flash sale at noon. One discounted pair of headphones ends up in almost every shopper's cart, and every time a shopper opens a page, the shop's website asks its database for that product's name, price, and stock. The database is the official record of everything the shop sells, and it was built to keep that record safe, not to answer the same question a million times in a few minutes. At noon the questions pile up, the database slows down, pages stop loading, and checkout fails for everyone, including people who are buying something completely different, because all of them share that one database.

The usual way out is to keep ready-made copies of popular answers somewhere much faster, so the database only gets asked when no copy exists. That place is called a cache. One machine cannot hold or serve a whole large shop's worth of copies, so the copies are spread across a group of machines that act as one, and that group is called a distributed cache. Redis and Memcached are the two names you will hear most often for the software that runs on those machines.

A cache looks like a small addition to a system, and that is exactly why interviewers like it. The simple version takes a minute to describe, and the real version is full of choices about missing copies, outdated copies, popular items, and machines that fail. This page walks through those choices in the order you would make them in the room: first what we are solving and why, then what the system must do, then how big it is, and only after that the design itself.

Why teams bother with a distributed cache

Almost every busy website you use leans on a cache, even if you never see it. A news feed, a product page, or a profile page loads quickly because most of what it needs was already sitting in memory. Behind that speed are a few plain reasons.

Speed. Reading from memory on a nearby machine takes a fraction of a millisecond, while a trip to a database that has to look through a lot of stored data takes noticeably longer. A page that needs twenty small lookups feels very different at those two speeds.

Protecting the database. A database is the place that must stay correct, so it is the hardest and most expensive part of a system to make bigger. Most websites read far more than they write, so if copies answer most of the reads, the database only has to deal with the rest.

Cost. Adding memory machines that serve copies is usually cheaper than adding database capacity to answer the same number of reads.

Surviving spikes. A flash sale, a viral post, or a morning rush sends a sudden burst at a few items. A cache soaks up that burst so the database sees a gentle slope instead of a cliff.

A small safety margin. If the database gets slow for a minute, the most popular answers can still be served from copies while the database recovers.

The flash sale at noon is simply the sharpest version of all five reasons at once.

What problem are we solving?

Before the application asks the database for anything, it asks the cache one question:

“Do you have a copy of the value stored under this key?”

A key is a short name for one piece of data, such as product:9001. The cache answers with the value if it has one, which is called a hit, or with “not here”, which is called a miss. On a hit, the application uses the copy and the database never hears about the request. On a miss, the application has to get the value from the database and ideally leave a copy behind for the next person who asks.

Success looks like this. The large majority of reads are answered from memory in under a millisecond, the database sees only a small fraction of the reads, a product that has just changed in the database does not show its old price for long, and the whole thing keeps working when a cache machine breaks or when the shop adds machines to cope with growth.

The hard part is that a cache only holds some of the data, and everything it holds is a copy, and a copy can disagree with the original. How to keep copies useful, fresh, and spread fairly across many machines is what the rest of this page is about. We will not guess at the answers yet. First we need to be clear about what the system must do and how well, so that every design choice later has a reason behind it.

This page is the interview deep dive in the system design interview guides hub. If the idea of caching is completely new, skim Caching Strategies first.

How to approach

In an interview you do not start by drawing machines. You start by turning a short, vague prompt into a clear contract: what is being cached, how fresh it must be, how big the system is, what happens when something fails, and what is out of scope. Only then do you size the problem, make the design decisions one at a time, and draw the result. The list below is the order this page follows.

How to design a distributed cache in an interview

  1. Clarify the problem: what is stored, whether the cache may be empty after a restart, how fresh reads must be after a write, the read and write mix, how skewed the traffic is, and how big the system is.
  2. Write the functional requirements (what the cache does) and the non-functional requirements (how fast, how big, how reliable).
  3. Estimate the size: operations per second, number of stored values, value size, and how much traffic one very popular key can attract.
  4. Make the design decisions in order: where the copies live, who owns the truth, how a database change reaches the copies, which machine holds each key, what happens when a machine dies, what happens when memory is full, what to do about a popular key, and what to do when many requests miss at once.
  5. Draw the picture those decisions produce, then walk one read and one write through it.
  6. Go deeper where the interviewer pushes: the miss path, the invalidation race, adding a machine, hot keys, failover, eviction, and one production failure.

Clarifying questions (say these in the room)

In a real interview the prompt is only a few words: “Design a distributed cache.” Before you design anything, ask questions until the problem is clear. Each answer below changes the design, which is why it is worth asking.

You: “What are we storing in the cache: individual items looked up by a key, like a product or a profile, or whole pages, or the results of arbitrary database queries?”

Interviewer: “Individual items by key. Think product rows and user profiles.”

That answer fixes the unit you will store. Each item has a key such as product:9001 and a value, and one cached copy corresponds to one database row or a small bundle of rows. Caching arbitrary query results is much harder to keep fresh, so you can park it.

You: “Does the cache itself need to survive a restart, or can it come back empty and fill up again from the database?”

Interviewer: “Ephemeral is fine. The database is the source of truth.”

This lets you leave persistence digressions to the side. If the cache is allowed to be empty after a crash, it is a speed layer and not a store of record, which shapes almost every later choice about writes and failure.

You: “After someone saves a change, such as a new display name or a new price, may reads show the old value for a moment, or must the very next read see the new value?”

Interviewer: “Mostly eventual. A profile name after a save should not flip back and forth if you can help it.”

So brief staleness is acceptable for most fields, but a value should not keep jumping between old and new. Write down the phrase “after a database change, clear the cached copy.” That is called invalidation, and it is the core of keeping copies honest.

You: “Is the traffic mostly reads with occasional writes?”

Interviewer: “Read-heavy. Loading on a miss is the common path.”

That tells you copies are filled lazily, when someone asks, instead of being written to the cache and the database together on every change.

You: “How uneven is the traffic: are all keys about equally popular, or does one product take a huge share during a sale?”

Interviewer: “Very uneven. Call out hot keys.”

You must plan for more than “add more machines,” because one very popular key still lives on one machine.

You: “Roughly how big is this: how many operations per second, how many stored values, and how fast must a read be?”

Interviewer: “Assume about 10 million operations a second, about 100 million values, over 100 GB in total, and about 99 out of 100 hot reads under a millisecond.”

Those numbers feed the capacity estimate in the next section. They also tell you early that no single machine will be enough.

You: “How available must the cache be, and during a network split between machines, would you rather answer with a slightly old value or refuse to answer?”

Interviewer: “About 99.9% uptime. During a split, answering with something slightly old is better than failing.”

That means the cache favors staying available over being perfectly fresh, and it tells you the design should have a second copy of each slice of data ready to take over.

You: “I want to keep edge caching of static files, caching of arbitrary SQL results, and multi-region caches that accept conflicting writes out of scope, so we can focus on an application key-value cache. Does that match?”

Interviewer: “Yes, focus on the distributed cache layer.”

If the interviewer later asks about hash rings, store brands, or eviction flags, you do not need to answer yet. Park those until the design decisions, where each one has a natural place.

Functional requirements

Functional requirements describe what the system does. From the answers above, the cache must:

  1. Read one value by key, and treat a miss as a normal answer. A miss is not an error. It simply tells the application to go and fetch the value from the database.
  2. Save a value under a key, with an optional lifetime. The application leaves a copy behind after a miss, and a lifetime keeps old copies from lingering forever.
  3. Delete a value. After the database changes, the application must be able to throw away the cached copy so the next read loads the fresh one.
  4. Let values expire by themselves. Expiry gives every copy a built-in end date, which is a safety net if someone forgets to delete a copy.
  5. Drop the least useful values when memory is full. The cache holds only part of the data, so it must decide what to remove when new values arrive.
  6. Support a one-step counter increase on a single key. Counters such as page views or stock holds must not lose updates when many callers add at the same time.
  7. Help callers find the right machine for any key, even while machines are added or removed. Without that, a bigger cluster would be unusable.
  8. Optionally save a snapshot of memory to disk. This lets a restarted machine warm up faster, but it never replaces the database.

Non-functional requirements

Non-functional requirements describe how well the system must do its job. They are the numbers and promises that shape the design.

  1. Speed. About 99 out of 100 hot reads should finish in under a millisecond. The cache sits in front of every product page, so if it is slow, the whole site is slow.
  2. Scale. About 10 million operations a second, about 100 million values, and more than 100 GB of data. Reading is the heavy path.
  3. Availability. About 99.9% uptime. One cache machine failing must not turn into an outage, and it must not push all of its traffic onto the database at once.
  4. Behavior during a network split. The cache keeps answering, and some answers may be slightly old. Brief staleness is better than refusing everyone.
  5. Freshness after a change. A value that changed in the database must not stay out of date for long, and it must not flip back and forth between old and new.
  6. Growth without disruption. Adding a machine must move only a small slice of the data, so that scaling up does not cause a wave of misses.
  7. Fairness under uneven traffic. One very popular key must not take down the machine that holds it, because other keys on that machine would suffer too.
  8. Visibility. The team must be able to tell whether the cache is protecting the database, not just whether the cache machines are up.

Out of scope (and why you park these)

Write a short out-of-scope list on the board so the interviewer sees you can bound the problem.

CDN and browser caching. A content delivery network keeps copies of files, such as images and scripts, at the edge of the internet close to users. It matters for global speed but does not replace an application-tier cache for changing data like catalog and session state. Acknowledge it, then design the in-memory cluster the application servers talk to.

Caching arbitrary SQL query results. A cache of query results has to be cleared whenever any table underneath changes, which is fragile at scale. The interview usually wants entity caching (a product, a profile) with an explicit clear on write.

Active-active multi-region cache with conflicting writes. If two regions accept writes to the same key without coordination, they will eventually disagree and need a merge rule. Most designs keep read copies per region with slightly old data, or send writes to one region, unless the interviewer expands the scope.

Full transactions across many keys. A cache is not a database. Changing several keys on different machines atomically is out of scope for a typical round. Single-key atomic operations on one machine, such as increment, are enough.

That is the requirements stage. If you want to practice it before reading on, try the requirements stage on Practice System Design (about 5–10 minutes), then come back for the sizing and the decisions.

Minute pacing (about 40 minutes)

MinutesFocus
0–5Clarify the problem; write functional and non-functional requirements; agree out of scope
5–12Capacity, then the design decisions and the high-level boxes with one read story
12–25The miss path, invalidation after a write, and adding a machine (the deep dive)
25–35Hot keys, failover, replication lag, observability
35–40One production war story and questions

In the room (how you open): “I’d start by agreeing what we are caching and how fresh it must be: individual items by key, whether the cache may come back empty after a restart, how stale a read may be after a change, how read-heavy the traffic is, how uneven it is, and how big the system is. I’ll write the requirements on the board, size the problem, and then make the design decisions one at a time, so every box I draw has a reason.”

If you remember one thing: Ask first, write the requirements down, and only then design. Tell one read story and one write story before you recite eviction policies.

Capacity estimation

Once the requirements are clear, the next step is to find out how big the problem is, because the size decides how many machines the cache needs and where it is likely to break. Every number below is an estimate to say out loud and then confirm with a load test.

TopicAnchor (estimate)Implication
Working set~100 GB aggregateMemory will fill; some values will be thrown out; a few hot keys matter more than the cold majority
Aggregate operations~10M ops/sEach machine has a ceiling, so that ceiling decides how many machines you need, plus connection pooling and batching
Distinct keys~100MMemory per key plus bookkeeping; check how evenly keys spread across machines
Average value size~1 KB100M × 1 KB ≈ 100 GB; watch network bandwidth; large values need compression or splitting
Hot keysTop 0.01% of keys (about 10,000 keys)One logical key can saturate one machine, so plan replicas, an app-side copy, or a split key

Worked skew sketch. Ten million operations a second sounds evenly spread only on paper. If one product during a flash sale absorbs even one percent of read traffic, that is about 100,000 operations a second on one key. A single cache machine commonly handles on the order of a hundred thousand simple operations a second, so that one key could use up one machine's entire budget. Adding machines does not split one key unless you copy or split the key itself.

What the numbers imply

  • Throughput, not memory, decides the machine count. At about 100,000 operations a second per machine, 10 million operations a second means on the order of 100 machines holding primary data, which is only about 1 GB each for 100 GB of data. Say that the count comes from the operation rate, and that a load test would confirm it.
  • Every extra copy multiplies memory. If each slice has two replicas, 100 GB becomes about 300 GB of RAM before overhead. Size the working set on purpose instead of caching everything.
  • Eviction must be explicit. When the working set does not fit in memory, the miss rate climbs and the database quietly becomes the cache.
  • Bandwidth is fine on average. 10M operations × about 1 KB is roughly 10 GB/s across the whole cluster, and spread over about 100 machines that is about 100 MB/s each, which is easy for a normal network link. A single hot key is the exception, not the average.
  • Connection counts matter. Thousands of application servers each opening connections to about 100 machines means a very large number of connections, so clients must pool them or go through a proxy.

Checkpoint: What decides your machine count, and what does one very popular key do to the average?

If you remember one thing: Capacity math exists to expose skew. Averages lie when one key owns the sale.

Design decisions

We now know what the cache must do, how fast it must be, and roughly how big it is. Now we choose. Each decision below answers one question that the requirements raised, and each one rules out an easier idea that would not survive those requirements. Read them in order, because every decision builds on the one before it.

Decision 1: Where do the fast copies live?

The requirements asked for reads in under a millisecond and for the database to be spared from most reads. The simplest idea is for every application server to keep its own copies of popular values in its own memory. That is very fast because there is no network trip, and it needs no new system at all.

It breaks against several requirements. With, say, 500 application servers, each keeps its own copy of the same hot values, so the database loads the same product 500 times, and memory is wasted on duplicates. The full 100 GB cannot live in every server. When a product changes, every one of those 500 private copies must be cleared, and a delete sent to one server leaves 499 old copies behind, which breaks the freshness requirement. A server that restarts starts empty and sends all of its misses to the database.

There are two ways out of this.

  • One very large cache machine. It is easy to reason about, but 10 million operations a second, 100 GB of data, and 99.9% uptime are more than one machine can give. If it dies, every read goes to the database at once.
  • A shared cluster of cache machines that every application server uses over the network. Each key has one home, so copies are not duplicated 500 times, one delete clears the one copy, and the load and the memory are spread out.

We choose the shared cluster, and that is what makes it a distributed cache. A tiny in-memory copy inside each application server can still sit in front of it for a handful of extremely hot values, and Decision 7 returns to that.

Decision 2: Who owns the truth, and who fills the cache?

Now that the copies live in a shared cluster, we must say which one is the official record and who puts values in. The simplest idea is to treat the cache as the main store: write the value to the cache and read it from the cache. It is quick, but it breaks the requirements we agreed on. Cache machines keep data in memory, so a restart empties them. The cache must also drop values when memory is full, which no real record is allowed to do. And we agreed that the cache may come back empty.

So the database stays the source of truth, and the cache holds copies that may disappear at any time. That leaves the question of who fills the cache. There are four common patterns, compared in Core design approaches later on this page: the application fills the cache on a miss (called cache-aside), the cache fills itself from the database (read-through), writes go to the cache and the database together (write-through), or writes go to the cache first and reach the database later (write-behind).

Because the traffic is read-heavy and very uneven, and because the application already talks to the database, we choose cache-aside: the application looks in the cache first; on a miss it loads the database, saves a copy with a lifetime, and returns the value. The price is that clearing old copies becomes the application's job, which is exactly what Decision 3 deals with.

Decision 3: How does a database change reach the copies?

When a product's price changes in the database, every cached copy of the old price is wrong. The simplest idea is to write the new price to both places: update the database, and update the cache too. It sounds tidy, but two separate writes are not one step. If the application crashes between them, the two places disagree. Worse, two writers can race: writer A sets the price to 100, writer B sets it to 80, but their cache writes land in the opposite order, so the cache ends with 100 while the database says 80, and it stays wrong until the copy expires.

The second simple idea is to wait for the lifetime of the copy to run out. That fails the freshness requirement, because a price should not stay out of date for an hour.

The better idea is to commit to the database first, and only then delete the cached copy. A delete cannot leave the wrong value behind, because the next read simply misses and loads the fresh row. A second option is to keep a version number in both the database row and the cached value, so an old copy can be recognised and rejected. A third is to have the database publish change events that a small service turns into deletes, which helps when many applications write to the same table.

We choose commit first, then delete, with a lifetime on every copy as a backstop for a delete that gets lost. One more detail matters in practice: a single database row may feed several cached keys, such as the product page, a search snippet, and a cart preview, so the key naming scheme should make it easy to list every key a row owns. The race that remains, a slow reader writing an old value back after the delete, is covered in the deep dive.

Decision 4: Which machine holds each key, and how does the caller find it?

With many cache machines, each key needs exactly one home so that a read and the earlier write agree. The simplest idea is to hash the key into a number and take the remainder after dividing by the number of machines, so that hash(key) % N picks the machine.

It works until the cluster changes size, and the requirements said growth must be cheap. With four machines and then a fifth, the divisor changes from 4 to 5, and a key stays where it was only when both remainders happen to match, which is true for roughly one key in five. About 80% of all keys suddenly have a new home, and every one of those keys looks like a miss. The database receives a wave of questions at exactly the moment you were trying to add capacity.

Before: hash(k) % 4  →  keys spread on 4 machines
After:  hash(k) % 5  →  ~80% of keys remap → miss wave → database overload

There are three ways out of this.

  • A lookup table that says which machine owns each key. It moves exactly what you tell it to move, but with 100 million keys the table itself is large, every read needs an extra step to consult it, and the table becomes something else to keep alive.
  • Fixed slots. Hash every key into one of a fixed number of slots (Redis Cluster uses 16,384) and assign slots to machines. Adding a machine moves a few whole slots. Many systems use this.
  • A hash ring. Put both machines and keys on a circle. A key belongs to the first machine clockwise from its position. Adding a machine moves only the keys in the stretch between its neighbour and itself.

The ring and the slots are two versions of the same idea, which is that a change should move a small, nameable piece and leave everything else alone. We choose consistent hashing, the ring, with virtual nodes: each physical machine owns many small stretches of the ring instead of one, so load stays even when there are only a few machines and a machine that leaves hands its load to many neighbours instead of one.

Adding a node with consistent hashing moves only the wedge of keys near the new token; modulo hashing reshuffles nearly the entire keyspace

Figure: Consistent hashing migrates a wedge; modulo hash reshuffles almost everything.

The caller also has to find the owner. The simplest idea is to hard-code a list of cache addresses in every application, which fails the moment a machine is added or replaced, because every application would need a redeploy. The better answer is a smart client: a small library inside the application that holds a copy of the ring and sends each request directly to the owner. When its copy of the ring is out of date, the cluster answers with a redirect (Redis Cluster calls them MOVED and ASK) and the client updates itself. The alternative is a proxy, a stateless middle layer with one address that does the routing for the application. It is simpler for the application and helpful when many languages or a huge number of connections are involved, but it adds a hop. We start with the smart client and choose the proxy when connection counts or language variety make it worthwhile.

Decision 5: What happens when a machine dies?

A cache machine will die eventually, and the requirement is 99.9% uptime. The simplest idea is to do nothing special: when a machine dies, its keys are simply missing, and the database fills them back in. That keeps the design small, but it fails the database-protection requirement. A machine that owns one hundredth of the keys vanishes, a hundredth of all reads become misses at the same moment, and the database must absorb that burst until the new machine warms up.

There are three ways out of this.

  • Copy writes to a second machine and wait for it before replying (synchronous replication). Nothing is lost, but every write is slower, and for data that is only a copy of the database that cost rarely pays off.
  • Copy writes to a second machine without waiting (asynchronous replication). Writes stay fast, and the second machine is almost up to date, so it can take over quickly. The price is that the last few writes can be lost if the first machine dies at the wrong moment.
  • Keep no second copy and throttle the refill. Cheaper, but the failure window is a database brownout.

We choose a primary with one or more replicas, copied asynchronously. When health checks decide the primary is dead, a replica is promoted, and the promotion carries a new epoch number, a counter that only goes up, so an old primary that wakes up later knows it is out of date and stops accepting writes. Losing the last few writes is acceptable because the database holds the truth and the lost copies are rebuilt on the next miss. Reads go to the primary by default, and a replica may serve a read when slightly old data is fine. During a network split the cache keeps answering from whichever side it can reach, possibly with older data, because the requirement said brief staleness beats refusing everyone.

Decision 6: What happens when memory is full?

The cache holds only part of the data, so memory will fill up. The simplest idea is to refuse new values once memory is full. That fails right away: a newly popular product could never enter the cache, and the cache would slowly fill up with yesterday's favourites. The opposite idea is to grow memory without limit, which breaks the cost requirement.

The usual answer is to evict, which means to throw out values the cache thinks are least useful. Common rules are least recently used (LRU, remove what has not been touched for the longest time), least frequently used (LFU, remove what is touched least often), first in first out, or random. LRU is the usual starting point because recent use predicts near-future use. LFU is better when a long tail of one-time keys would otherwise push out popular ones.

Eviction and TTL (time to live, the lifetime we put on each copy) are two separate tools that work together. A TTL removes a value when its time is up, even if memory is plentiful. Eviction removes a value when memory is short, even if its TTL has not run out. We choose LRU-style eviction with a TTL on every key, and we watch the eviction count, because a high eviction count together with a falling hit ratio means the working set no longer fits, and the database is quietly taking the load back.

Hashing spreads many different keys evenly, so the simplest idea is to trust it and assume the load is spread. It fails for one key. A key has exactly one home, so a million reads a second for one product all arrive at one machine, which may use up its whole budget while the rest of the cluster sits nearly idle. Other keys that happen to live on the same machine get slow too.

There are four ways out of this.

  • A bigger machine for that slice. It buys time but does not change the shape of the problem.
  • Read from replicas for that key. The primary has copies, so spread that key's reads across them. The price is slightly old reads.
  • A tiny in-memory copy inside each application server (often called an L2, or second-level, cache) with a very short lifetime, such as a second or two. Most of the million reads never leave the application server.
  • Split the key into several keys such as product:9001:0 to product:9001:7, so the eight copies live on up to eight machines. Reads pick one at random, and a write must update or delete all eight.

We choose to detect hot keys by watching per-key rates, then put the tiny in-process copy in front of them first, and read from replicas second. Key splitting is for values that almost never change. The cost to say out loud is staleness: a copy with a two-second lifetime can be up to two seconds old.

Decision 8: What if many requests miss at the same moment?

Whenever a value is missing, someone has to load it. The simplest idea is to let every request that misses go to the database and fill the cache. That is fine when misses are spread out, but it fails when a popular value disappears all at once, for instance when its lifetime runs out. Ten thousand requests arrive at the same instant, all see a miss, and all send the same question to the database. This is called a thundering herd, and it is the usual way a healthy-looking cache still melts a database.

T=3600s  key expires on its primary
T=3600s  10k apps → 10k identical database SELECTs for the same row
         database connections maxed; cache dashboard still "healthy"

There are three tools, and they combine well.

  • Single flight. Let one request do the loading while the others wait for the result.
  • TTL jitter. Give each copy a lifetime of base + a small random amount, so that copies loaded together do not expire together.
  • Serve slightly old data while refreshing (stale-while-revalidate). Keep returning the last good value for a moment while one request refreshes it in the background, if the product allows it.

We choose all three, and the deep dive shows how to make single flight work across many application servers instead of only inside one.

Putting the decisions together

Taken together, the eight decisions are the design. A shared cluster of cache machines holds copies. The database stays the source of truth. The application fills the cache on a miss and deletes a copy after a database change. A hash ring with virtual nodes decides which machine owns each key, and a smart client sends requests there. Each slice has a primary and replicas copied asynchronously, with an epoch number to prevent two primaries. Memory is bounded by LRU-style eviction and a TTL on every key. Popular keys get extra copies, and a miss is loaded once, not thousands of times.

In the room (a fuller opening you can actually say): “I’d treat the database as the source of truth and the cache as a shared cluster of copies, so the application reads the cache first, and on a miss loads the database once and stores the value with a lifetime. On a write I’d commit to the database first and then delete the cached key, with the lifetime as a backstop. I’d spread keys across machines with a hash ring and virtual nodes, so adding a machine moves only a slice of keys, not almost all of them. Each slice has a primary and replicas, copied asynchronously, with an epoch number so an old primary cannot accept writes. Memory is bounded with LRU-style eviction. For one very popular key I’d add a tiny short-lived copy inside each application server and read from replicas, and for expiry I’d add jitter and make sure only one request reloads a missing value.”

The next section draws these decisions as one picture and shows who owns what.

High-level architecture

The decisions above were made one at a time. This section puts them on one page as a picture: which boxes exist, who owns what, and how one read travels through them. The deeper mechanics come later, so keep this at the level of a map.

Clients are either a smart client (a library inside the application that knows the ring and sends each request to the owning machine) or a proxy (a stateless middle layer that does the routing). The cluster exposes a keyspace split into slots or token ranges on a ring. Each range is owned by a primary with one or more replicas. A configuration service or gossip protocol (machines telling each other who is alive) publishes membership and the current epoch. Persistence (RDB snapshots, AOF logs) is optional and usually off the critical read path, because the database remains the truth.

Who owns what:

  • Routing layer (client library or proxy) — Turns each key into a position on the ring, then into the machine that owns that slice. When the map of machines is out of date, the cluster answers with a redirect (Redis Cluster calls these MOVED and ASK) so the client can refresh who owns the key.
  • Shard primary — The one machine that accepts writes for its slice of keys. It serves reads too, or hands a read to a replica when slightly old data is fine, and it enforces the memory limit and eviction.
  • Replicas — Keep a slightly delayed copy of the primary. When health checks decide the primary is dead, a replica is promoted, and that promotion carries a rising generation number (epoch) so an old primary cannot accept writes again.
  • Control plane — Adds and removes machines, throttles how fast keys move during a rebalance, spreads placement across racks and availability zones, and raises alerts when one key is hotter than the rest.
[ App servers ]
       |
       |  GET / SET / DEL (sync, sub-ms target on hit)
       v
[ Smart client or Proxy ]  ----membership----► [ Config / gossip ]
       |
       |  consistent hash: key → token → primary
       v
+------------------+     async replication    +------------------+
| Primary A        | ----------------------► | Replica A'         |
| (token range T1) |                         | (read optional)  |
+------------------+                         +------------------+
+------------------+                         +------------------+
| Primary B        | ----------------------► | Replica B'       |
| (token range T2) |                         |                  |
+------------------+                         +------------------+

  Rebalance (background): new node C → migrate token ranges → throttle → HANDOFF

In the room: Walk one GET: hash the key → route to the primary → memory lookup → hit or miss. Then one failover: primary dead → replica promoted → clients refresh the ring. Mention that rebalance is asynchronous and throttled, not instant free capacity.

If you remember one thing: A GET is hash → primary (or a replica, with stale-read rules). Rebalance is an operations event with a migration protocol, not a magic scale button.

Core design approaches

Decision 2 chose cache-aside. These patterns answer who loads on a miss and who updates on a write, and the interviewer may ask you to compare them. Most read-heavy designs land on cache-aside, but knowing the others lets you explain why.

Cache-aside (lazy loading)

The application checks the cache first. On a miss, it loads the database, then saves the value into the cache (SET) with a lifetime (TTL). On a database update, the app commits the write, then deletes the related cache keys (or bumps a version in the value). It is simple and common, and clearing old copies (invalidation) is the application's job.

When to pick: Read-heavy workloads where slightly old data is acceptable between a lifetime and a delete, and the database is clearly the source of truth.

Read-through

The cache module itself loads the database on a miss, so the application only talks to the cache API (the method names callers use). Clearing old copies can still be triggered by the app or by a feed of database changes (change-data-capture) on write.

When to pick: You want one shared cache API and central miss handling, and teams accept more logic inside the cache layer or a helper process beside it.

Write-through

Writes go to the cache and the database together before the reply comes back. Reads always see what was written through that combined path.

When to pick: Stale reads after a write are unacceptable and you accept higher write latency. It is less common at huge scale than cache-aside with delete-on-write.

Write-behind (write-back)

Writes are acknowledged after the cache update and flushed to the database later in the background. Writes are fast, but there is a loss risk if the cache dies before the flush. It is rare in generic cache interviews unless explicitly scoped.

When to pick: Write-heavy buffering where the product tolerates that short loss window. Say the durability tradeoff out loud.

PatternWho loads on missInvalidation / writeConsistencyTypical latency
Cache-asideApplicationApp DELETE after DB commitEventual + TTLFast reads; app owns races
Read-throughCache layerCache or app on DB changeEventualUniform API
Write-throughN/A (write path)Sync to DB on every writeStrongerSlower writes
Write-behindCacheAsync DB flushWeakest durabilityFastest writes

If you remember one thing: With cache-aside the app owns invalidation, and the database is the truth. Never pretend the cache alone is authoritative.

Data model

The design needs somewhere to keep keys, values, and the map of who owns what, so the next question is what exactly gets stored. Think in three layers: how keys map to machines, what lives in each slot, and how the cluster knows who owns what.

Key → slot / token (routing)

ConceptPurpose
Logical key (user:42:profile)What the application GETs and SETs
Hash(key) → token on ringStable mapping to a token range
Token range → primary nodeOwnership for reads and writes
Virtual nodes (vnodes)Many tokens per physical machine so load stays even when the node count is small

Clients compute the path from key to ring position to owning machine locally (a smart client) or ask a proxy. During a rebalance, a key may temporarily redirect (MOVED/ASK in Redis Cluster terms) until the move completes.

Value + TTL + version (payload)

FieldRole
Value bytes (JSON, protobuf, string)Cached representation of a database row or computed object
TTL (seconds)Automatic expiry; safety net when invalidation misses a key
Version or updated_at (optional)Detect stale writes on SET; compare on read after a database update
Flags (compressed, null marker)Operational metadata

Interview land: On a database update, prefer deleting the cached key or bumping a version over blindly saving old data back into the cache. Ordering races can put a stale value back if a slow read finishes after your write.

Membership (cluster topology)

StoreContents
Ring configNode id → token ranges; epoch generation
Health statePrimary/replica role, last heartbeat
Migration mapKeys mid-handoff from old owner to new owner

Gossip or a small config service pushes updates, and clients cache the topology with a TTL and refresh on redirect storms.

If you remember one thing: Draw key → token → primary separately from value + TTL. Routing and payload are different failure domains.

Detailed design

We now know what is stored and where the ownership map lives, so we can put the pieces in motion. This section follows two everyday moments: a shopper opens a product page while the flash sale is hot, and later the shop changes that product's price in the database. The cache only helps if both moments are handled on purpose. After the two stories comes the API: the calls the application makes, what comes back, and the small admin calls operators use. The exact races and the follow-up questions come in the deep dive right after this section.

Read path (cache-aside)

The application asks for a key, say product:9001. The smart client hashes the key and sends the GET to the machine that owns that slice of the keyspace. If the value is in memory, the call returns in well under a millisecond on a healthy cluster, and the shopper's page loads. That is the happy path, and the database never hears about it.

If the key is missing or has expired, that is a cache miss, which is not the same as “the cache is broken.” The application loads the row from the database, writes the result back into the cache with a TTL, and returns it to the shopper. The dangerous case is thousands of requests missing the same viral key at the same time, because each one would send the same question to the database. To prevent that, only one request is allowed to do the loading while the others wait and then read the fresh copy, and the TTLs carry a little random jitter so that expiry times do not line up.

Batch helpers such as MGET reduce round trips when you need many keys. Keep a miss and an error separate in your metrics: a timeout should not automatically become “load the database no matter what” unless you have a deliberate fallback with limits.

Write path (database update, then invalidate)

When someone updates the database row, the application commits that write first. After the commit succeeds, it deletes (or version-bumps) every cache key derived from that row. Deleting after the commit is safer than updating the cache before the database is sure. Some teams also put a version number in both the database row and the cached value, so a slow reader that filled the cache with old data cannot keep it there forever.

One race is worth naming in the room. A reader misses and starts a slow database read of the old price. Meanwhile an update commits a new price and deletes the cache key, which finds nothing to delete. The slow reader then finishes and saves the old price into the cache. Short TTLs, version checks on SET, and correct commit-then-delete ordering shrink that window. The deep dive shows the timeline.

Atomic operations and memory pressure

Operations like increment or compare-and-set work well when one primary owns the key, because that one machine can apply them one at a time. They are a poor place to invent cross-machine transactions, so push multi-key atomicity to the database. When memory fills up, the cache evicts keys under its policy (often LRU-style) even if their TTL has not expired. A rising eviction count together with a falling hit ratio usually means the working set no longer fits, and adding machines without fixing hot keys will not save you.

In the room (spoken): “Reads are cache-aside with one loader per missing key. Writes commit to the database, then delete the cache keys. I treat the cache as ephemeral acceleration, not the system of record.”

API design: operations, errors and cluster routing

A cache is not usually a public API of its own. It is a service the application servers call from inside the system, so the API design is about three things: the operations the application can call, what the application receives back, and how operators change the cluster. Caches usually speak a compact binary protocol (RESP, for Redis) or sit behind a thin HTTP wrapper. Clarity matters more than opcode trivia.

The internal calls. The application uses a small set of operations.

OpSemantics
GET keyReturn the value, or a miss (nil)
SET key value [EX ttl]Save the value, with an optional TTL
DEL key [key ...]Clear one or many keys
INCR key / DECR keyAtomic counter on a single key
MGET keys[]Batch read to reduce round trips
EXPIRE key secondsChange the TTL of an existing key

What the application receives. A hit returns the value. A miss returns “not present” and is the signal to start the load path. An error (a timeout, or the cluster being down) is a different thing from a miss, and it must not make every request hammer the database. For routing, a smart client follows redirects when topology changes, and a proxy hides all of that behind one address.

ModeBehavior
Smart clientComputes the hash locally; follows MOVED/ASK redirects when topology changes
ProxySingle endpoint; forwards to the correct shard

Which status code means what. If the cache sits behind HTTP, use 404 for “that key is not there” and 503 for “the cache is unavailable,” and document the mapping so that clients treat the two differently. A 404 should trigger a load from the database. A 503 should trigger a bounded fallback with timeouts, not an unlimited one.

How operators change the cluster. A small internal admin API lets the team add or remove nodes, throttle rebalance, and inspect hot keys without touching application code.

GET    /v1/admin/topology                  # nodes, token ranges, epoch
POST   /v1/admin/nodes                     # add a node (starts a throttled rebalance)
DELETE /v1/admin/nodes/{id}                # drain and remove a node
PUT    /v1/admin/rebalance/throttle        # cap migration bandwidth
GET    /v1/admin/hot-keys?limit=10         # top keys by operations per second

Request flow (GET, cache-aside):

App --GET key--> Client lib --hash--> Primary shard
                              |
                              hit --> return value
                              miss --> one loader --> DB --> SET + TTL --> value

If you remember one thing: A miss triggers a controlled refill path, not an unbounded parallel database query per client.

Deep dive: the questions an interviewer asks next

You have drawn the design and walked a read and a write through it. This is the point where a good interviewer starts digging, and the questions usually come in a fairly predictable order, because each answer raises the next one. Read this section as that conversation. Each question is followed by an answer you can give, plus a hint of where the interviewer is likely to go next. If the follow-up is obvious from your answer, say it before they ask.

“Thousands of requests miss the same key at the same moment. What exactly happens?”

Start by walking one miss from the application to the owning machine and then to the database, then show where that path goes wrong when thousands of requests arrive together.

  1. The client hashes product:9001 and routes to its primary P.
  2. P returns a miss (the key is absent or expired).
  3. Exactly one request is chosen to load the database. The others wait briefly.
  4. The loader reads the row from the database and SETs product:9001 with a TTL on P.
  5. The waiting requests retry the GET and hit.

If step 3 is skipped, 10,000 concurrent misses become 10,000 identical database queries, and the database runs out of connections. The common tool for step 3 is called singleflight: inside one application process, the first request for a key does the work and every other request for that key in the same process waits for that one answer.

There is a catch that is easy to miss. Singleflight inside one process only helps within that process. With 500 application servers, you still get up to 500 database queries for the same row, one per server. To get one load for the whole fleet, the loader first claims a short lock key in the cache, and everyone else waits and retries the GET. A sketch of the idea:

value = cache.GET(key)
if hit: return value

if cache.SET("lock:" + key, my_id, NX, EX = 5):   # NX = only if it does not exist yet
    row = db.read(key)
    cache.SET(key, row, EX = base_ttl + random(0, jitter))
    cache.DEL("lock:" + key)
    return row
else:
    wait a short random time, then retry GET (a few times)
    # if it still misses, fall back to a limited database read or to a stale copy

The lock key has its own short lifetime, so a loader that crashes cannot block everyone forever. Add TTL jitter (base + random) so a batch of values loaded together do not expire together, probabilistic early refresh (a small chance that a request refreshes the value shortly before it expires), and stale-while-revalidate where the product allows a slightly old value during refill.

Even with perfect coalescing, one hot key still lives on one primary, so herd control alone does not fix a hot shard. It has to be combined with a hot-key plan.

Where it goes next: the interviewer will probably ask what keeps the copy correct after a write.

“After a database write, how do you stop the cache serving the old value, and what can still go wrong?”

The rule is commit first, then delete. The application updates the products row for id 9001 and commits. Then it deletes product:9001 and any related keys such as product:9001:inventory. The next GET misses, reloads the fresh row, and saves it.

The bad pattern is updating the cache before the database commit. If the commit then fails or the application crashes, the cache holds a value the database never accepted. The pattern of deleting by wildcard (such as product:9001*) is also a trap, because scanning a large cluster for matching keys does not scale. Keep an explicit list of the keys a row owns, or a version stamp inside the value.

Even with the right order, one race remains. Here is the timeline:

t0  Reader A: GET product:9001 → miss
t1  Reader A: reads database → price 100 (old)
t2  Writer B: UPDATE price = 80, commit
t3  Writer B: DEL product:9001 (nothing there to delete)
t4  Reader A: SET product:9001 = 100   ← the old price is back

Reader A started before the write and finished after the delete, so it put stale data back. The ways to shrink this window are a short TTL on that key, a version check on SET (only save if the version is newer than what is stored, which needs a small server-side script), a second delete a moment after the first, or a lease (the cache hands the missing reader a token, and a delete cancels any outstanding tokens, so A's late SET is refused).

A second failure is a crash between the commit and the delete. The copy then stays old until its TTL runs out. For fields where that is not acceptable, publish the change as an event from the database (often called an outbox) and let a small worker perform the delete and retry it until it succeeds.

Interview land: “Invalidate by delete after commit. The TTL is a backup, not the main consistency tool for something like a price.”

Where it goes next: the interviewer will now ask about changing the size of the cluster.

“You add a fifth machine. What happens to the keys?”

Start with the bad design, because it is the one people reach for first. With four machines, slot = hash(key) % 4. Adding a fifth changes the divisor, and about 80% of keys map to a different machine. Every remapped key now looks like a miss, so the database absorbs a wave of questions at the moment you meant to add capacity.

The great design is a ring. Keys and machines both sit on a circle, and a key belongs to the first machine clockwise from its position. Add machine C, and only keys in the wedge between C's predecessor and C move to C. Every other arc stays put. Each physical machine owns many positions on the ring (virtual nodes), so load stays even with a small number of machines, and the wedge that moves comes from many neighbours instead of one.

Moving the wedge is still a careful operation. Rebalance runs in the background with a throttle on migration bandwidth, so copying data does not starve live traffic. A handoff protocol decides who answers for a key while it is mid-move: reads and writes for a moving key follow redirect rules (the old owner redirects to the new one), so two machines never answer with different values for the same key.

Interview land: “Adding a node moves a wedge, not the universe, and the move is throttled and explicit.”

Where it goes next: “And if one key is the problem, not the number of machines?”

“One product gets a million reads a second. Adding machines does not help, so what do you do?”

Hashing spreads many different keys across machines. It does nothing for one dominant key, because that key still has one home. A million reads a second all go to one primary, whose single thread and network card become the bottleneck while the cluster average looks calm. Other keys on that machine slow down too.

The remedies, in the order I would apply them, are these. First, a tiny in-process copy inside each application server with a one- or two-second lifetime, so most of the million reads never leave the server. Second, reads from the primary's replicas for that key, which spreads the load at the cost of slightly older answers. Third, for a value that almost never changes, split the key into several (product:9001:0 to product:9001:7), read one at random and write or delete all of them. A fourth option, bigger hardware for that slice, buys time but does not change the shape of the problem.

To use any of these, you first have to see the hot key. Track the top keys by operations per second and alert on per-machine CPU and queries, not on the cluster average.

Interview land: “Sharding spreads many keys. A single hot key needs copies of its own.”

Where it goes next: the interviewer will probably ask what happens when the primary itself dies.

“The primary dies. What do you lose, and where do reads go?”

Health checks notice that the primary has stopped answering, and a replica is promoted with a new epoch number that is higher than the old one. Clients learn the new owner by refreshing the topology or by following a redirect. If the old primary comes back, its epoch is lower, so it knows it has been replaced and refuses writes. That is fencing, and it is how you avoid two primaries accepting writes for the same keys.

Replication was asynchronous, so the replica may be a few updates behind. Those last few writes are lost. That is acceptable for a cache, because the database holds the truth and any value that was lost simply misses and is reloaded. Say that out loud, because it also means you must never keep the only copy of anything important in the cache.

For reads, the default is the primary. A replica is allowed only when slightly old data is fine. A user who just saved their profile and then reads from a lagging replica would see the old name, so for that user's next read, go to the primary, route them to a replica that has caught up, or compare a version number stored in the value and retry.

On a network split, the usual choice is to keep answering from the side you can reach and accept stale reads, which matches the requirement that availability beats freshness. The side that cannot reach a majority stops accepting writes, so two sides do not diverge.

Interview land: “Promote a replica with a higher epoch, accept a short loss window, and send critical reads to the primary.”

Where it goes next: “Memory is full, so what gets thrown away?”

“Memory is full. What gets thrown out, and how does that interact with TTL?”

Two things remove values, and they are independent. A TTL removes a value when its time is up, whether or not memory is full. Eviction removes a value when memory is short, even if its TTL has not run out. LRU (least recently used) is a good default; LFU (least frequently used) helps when a long tail of one-off keys would otherwise flush the popular ones.

The failure to watch for is eviction thrash: the working set is larger than memory, so the cache evicts values that are asked for again soon, the miss rate climbs, and the database becomes the cache. The signal is a rising eviction count with a falling hit ratio. The remedy is more memory for that slice, shorter lifetimes or smaller values for the long tail, or removing key types that do not deserve to be cached.

Say it in the interview: “The database is truth. A miss loads once per key, a database write is followed by a delete, a ring move touches a wedge, a hot key gets its own copies, and a failover promotes a replica with a higher epoch. The cost is a short window of staleness and a few lost writes, and I would monitor for both.”

Hot keys and eviction thrash are summarised again in Key challenges. Replication lag and pool wait belong in Observability and Bottlenecks and tradeoffs.

If you remember one thing: The deep dive is a conversation. One loader per missing key, delete after commit, a wedge not a reshuffle, and separate copies for a hot key are each things you could have said before they asked.

Key challenges

The deep dive showed that each step is simple on its own. What makes the real system hard is how those steps interact under load, and a handful of hard parts show up again and again. Name them early in the interview, because each one is a place the interviewer is likely to push.

Hot keys. One logical key lives on one primary, so CPU, memory bandwidth, and the single thread of that primary can run out long before the cluster average looks stressed. The plan for a hot key belongs in the design from the start.

Rebalancing. Moving keys without dual ownership bugs, where the wrong value is served mid-move, needs a handoff protocol and a throttled copy. Adding capacity can itself cause a latency spike if the move is not throttled.

Replication lag. A user saves a profile, reads from a replica, and sees the old name. Applications must know which replica they are reading from and whether read-your-writes is required.

Invalidation mapping. One database row may feed dozens of cache keys, and wildcard deletes do not scale. Version stamps and explicit key lists help.

Eviction thrash. If the working set is larger than memory, the cache evicts values it needs again soon, the miss rate rises, and the database takes the load.

Keys that do not exist. Requests for values that are not in the database at all, such as invalid product ids, always miss, so every one reaches the database. Cache a short-lived “not found” marker for those keys, and rate limit callers who ask for many keys that do not exist.

Split brain after a partition. Two primaries accepting writes is prevented with quorum, epoch, and fencing. Caches usually favor availability with stale reads.

If you remember one thing: Hot keys and rebalance hurt more in production than choosing LRU versus LFU on day one.

Scaling the system

Those hard parts get worse as traffic grows, so the next question is what happens when the number of requests multiplies. You do not have to wait for the interviewer to ask. Right after you finish the design, say the scaling story yourself, then be ready for the follow-ups.

What you can say upfront: “More capacity comes from adding machines, and with keys placed on a hash ring (consistent hashing) only a wedge of keys moves each time, with the move throttled and done off-peak. Read throughput can be scaled with replica reads where slightly old data is fine, while the primary serves the paths that must be fresh. Spreading keys across machines helps many different keys, but it does not fix one very hot key, so I would put a tiny in-process copy in front of hot keys and split or replicate the ones that need it. Across regions I would keep read copies per region with a stated staleness, not accept conflicting writes in two places.”

That one paragraph answers most of what follows. Here are the follow-ups, in the order they usually come.

Interviewer: “Traffic doubles next quarter. What do you change?”

You: “First I check what is actually short: operations per second, memory, or one hot machine. If it is throughput, I add machines, and the ring means only a wedge of keys moves to each new one. I add them off-peak, with migration throttled, because moving data uses network and CPU that live traffic needs.”

Interviewer: “Why not just buy bigger machines?”

You: “Bigger machines are sometimes the cheaper first step, especially for one slice that holds a monster key. But each machine still has one thread of work and one network link, so past a point more machines help more than bigger ones. And a bigger machine is a bigger loss when it fails.”

Interviewer: “Can we read from replicas to spread the load?”

You: “Yes, for keys where slightly old data is fine, such as product descriptions. Replicas lag behind the primary, so I keep reads on the primary for paths that need to see a write immediately, like a user checking their just-saved profile. I say which fields are which, instead of treating every key the same.”

Interviewer: “One product gets a share of the traffic that is larger than any machine can handle. What then?”

You: “That is the case sharding does not solve, because one key has one home. I add a tiny short-lived copy inside each application server, read from replicas for that key, and if the value rarely changes, split it into several keys that live on different machines. The cost is staleness bounded by that short lifetime and a write that has to touch every copy.”

Interviewer: “What about more than one region?”

You: “Active-active caches that accept writes in two regions will conflict, so I avoid them. I keep a cache cluster per region that reads from the local database copy or replica, and clear keys in each region after a write. Between regions a copy may be a few seconds old, which I state out loud, or I send all writes for a key to one region.”

If you remember one thing: Replica reads scale throughput, primary reads win when stale is unacceptable, and one hot key needs its own plan.

Failure handling

Scaling answers “what if there is more traffic.” This section answers “what if something breaks.” Again, say it first. Failures in a cache show up as a dead machine, a lagging replica, or a slow cluster that pushes everyone onto the database, and what impresses an interviewer is that the answer was decided before the failure happened.

What you can say upfront: “Every failure has a decision made in advance. If a primary dies, a replica is promoted with a higher generation number (epoch) so the old one cannot accept writes, and I accept that the last few updates may be lost because the database holds the truth. If a replica lags, critical reads go to the primary. If the whole cache is slow or down, the application falls back to the database through a trip switch (circuit breaker) with timeouts and a cap on parallel loads, so the fallback does not turn into a stampede. A cold cache is warmed at a controlled rate with staggered lifetimes. Slightly old data is acceptable for most fields, but a wrong price or balance is not, so those fields use the primary or short lifetimes with strict clearing of old copies.”

The follow-ups are about the details of those decisions.

Interviewer: “A primary crashes. What does the user see?”

You: “For a few seconds, some requests to that slice fail or are slow, until a replica is promoted. The application retries with backoff, and a miss falls through to the database for a moment. Users see degraded speed or a brief error on those items, not an outage of the whole site. I accept that the last few writes to that machine are lost, because the database is the truth and they reload on the next miss.”

Interviewer: “A user saves their display name and sees the old one when they refresh. Why?”

You: “The read hit a replica that was a little behind the primary. For that user's next read I would go to the primary, or send them to a replica that has caught up, or keep a version in the cached value and retry when it is older than the version they just saved. I would also say which fields can be old for a few seconds and which cannot.”

Interviewer: “The network splits the cluster in two. What happens?”

You: “Most caches choose to keep answering, accepting some slightly old reads, because that is better than refusing everyone. The side that cannot reach a majority stops accepting writes, so two primaries do not diverge, and epochs fence off an old primary when the split heals. For money paths I read from the primary or the database.”

Interviewer: “The cache is slow but not down. What goes wrong?”

You: “That is the dangerous case, because simple up-or-down checks do not catch it. Applications time out, fall back to the database in parallel, and the database overloads, which makes cache refills even slower. I put a circuit breaker and a timeout on the fallback, cap how many requests may reload the same key, and alert on cache p99 and connection-pool waits, not only whether the machine answers a ping.”

Interviewer: “You must add a node during peak traffic. Is that safe?”

You: “I avoid it, because the migration uses network and CPU the live traffic needs. If I must, I throttle the migration, pre-warm the replicas, and watch per-machine imbalance and rebalance duration. Adding capacity is an operations event, not an instant relief.”

Interviewer: “The whole cache cluster restarts from empty. What happens?”

You: “Every request misses, so the database takes the full load at once. I warm the cache from the database at a controlled rate, starting with the most popular keys, stagger TTLs so expiry does not arrive in a wave, and keep the one-loader-per-key rule and the fallback limits in place while it fills.”

FailureUser-visibleMitigation
Primary downErrors or brief failover latencyHealth checks, automatic replica promotion, epoch bump, client retry with backoff
Replica lag spikeStale reads, flip-flopping UIRoute critical reads to the primary; sticky routing; version in value
Network partitionSplit brain riskMajority side keeps writing; minority stops accepting writes; epoch fencing
Node add during peakLatency during migrationThrottle rebalance; off-peak adds; pre-warm replicas
Cache cluster unavailable or slowSlow path or errorsCircuit breaker to the database with a timeout and a bulkhead (a cap on parallel loads); unbounded fallback is a stampede
Full cluster cold startMiss storm on the databaseWarm the cache from the database at a controlled rate; stagger TTLs

Degraded UX: Slightly stale data is often acceptable. A wrong price or balance is not, so route those reads to the primary or use a short TTL with strict invalidation.

If you remember one thing: Under a partition, most caches choose availability and stale reads, unless the interviewer scopes money paths to primary reads.

Observability (what you measure)

Once the system is live, you need a way to tell whether it is protecting the database, and that takes more than a single hit ratio (the share of reads answered from the cache). One global hit ratio hides miss storms and hot machines, so measure what proves the cache is doing its job.

MetricWhy it matters
Hit ratio (global and per key prefix)The trend breaks when invalidation or eviction fails
Miss rate spike correlated with database QPSHerd on expiry or rebalance
p99 GET latency per machineA hot key or slow primary shows up before the cluster average moves
Per-machine QPS and CPUOne machine melting while the cluster looks green
evicted_keys / memory usageWorking set exceeds RAM; eviction thrash
Rebalance progress / keys migratingLatency during a node add
Replication lag (bytes or seconds)Stale read incidents after a write
Database query rate on the cache miss pathProves whether the cache protects the database
Single-flight wait timeWhether herd control is working or overloaded

Alert ideas: Top 10 keys by QPS; miss rate 10× within five minutes; database connection pool above 80% while cache CPU is calm; rebalance duration over its target.

If you remember one thing: Chart database QPS from the cache miss path next to the hit ratio. A green cache with a rising database load means the design is failing.

Security and abuse

A cache holds copies of real data and sits close to every application, so this section covers how to keep it safe. Keep the cache ports off the public internet, use a private network, encrypt traffic in transit where the software supports it (TLS), and give each application only the access it needs. Do not log secrets or full values at debug level, because keys often embed user ids and values often mirror database rows, so treat them as sensitive. The cache does not replace authorization: any application with access to the cluster can read its keys unless values are encrypted, which is uncommon for a generic cache, so authorization stays in the application.

An attacker can also try to fill memory by asking for endless made-up keys (user:random:*), which both evicts useful data and sends misses to the database. Cap key length, rate limit the creation of keys, monitor memory per key prefix, and cache short “not found” markers for ids that do not exist. If values are deserialized into objects, use a safe format such as JSON with a schema, not arbitrary object serialization.

If you remember one thing: The cache is a performance layer, not a security boundary. Authorization stays in the application.

Cost awareness

Every choice so far also has a price, and interviewers like to hear that you noticed. Memory is the main bill: 100 GB copied three ways is about 300 GB of RAM before overhead, so right-size what you keep instead of caching everything. Copying every write across availability zones is real money, and a hot write key costs network as well as CPU. Caching cold data pushes out hot data, so short lifetimes and least-frequently-used eviction reduce waste. Moving keys during peak costs engineer time and customer latency, so schedule migrations. The sneakiest cost is database fallback: a miss storm turns cache savings into a database scaling emergency, and it is cheaper to fund lifetime jitter and one-loader-per-key than to buy a bigger database in a hurry.

Say in the interview: Cache size is a cost decision tied to a speed goal, not “as big as possible.”

Green Dashboard. Angry Users. The Production Mindset System Design Interviews Expect.

Everything so far describes how the cache should work on a good day. This section is about what happens on a bad day, told as short stories from the people who carry the pager (the on-call engineers). Textbooks teach boxes and arrows. On-call teaches this: the cache cluster can look healthy while the database melts, or while users see data flip back and forth after a failover.

A cache is not just a faster database. It is a place where machine layout, lifetimes, clearing of old copies, and fallback paths all interact. The interesting failures are miss herds, a ring move done wrong, replica lag after a save, and an unbounded database refill when the cache slows down.

Four nightmares that look fine on a dashboard:

What the dashboard showsWhat is actually happening
Cache CPU averages 30% and the hit ratio looks fineOne key's lifetime expired together with thousands of others and the database is drowning
Capacity was added and the cluster looks biggerThe ring move is saturating the network and latency is worse than before
Failover completed successfully in the logClients still talk to the old primary and users see old data
The cache machines answer pingsThe cache is too slow to be useful and applications are stampeding the database

Scene 1: Tail latency jumps right after you “fixed” capacity

The moment: A shopping app has been slow every evening for shoppers browsing and checking out, so the team adds six cache machines before the weekend sale, hoping the extra machines will bring latency down. For hours afterwards, scrolling and checkout get slower, not faster. Support reports timeouts while the cluster CPU average still looks fine, and the slowest one percent of requests (p99) is worse than before the change.

The trap: The rebalance moved huge ranges of keys at once. Network cards were saturated copying data, clients still talked to old owners, and replicas lagged behind their primaries. Adding capacity triggered a migration storm, not instant relief.

What to build: Throttle how fast keys are copied during a move, schedule node adds off-peak, pre-warm replicas before the cutover, and alert on per-machine imbalance and how long a rebalance takes, not only on average CPU.

Scene 2: Database load spikes while cache dashboards stay green

The moment: Overnight, a shop job loads fifty thousand product entries into the cache and gives every entry the same one-hour lifetime. An hour later, shoppers hit brief checkout errors. Database connections are maxed out, yet the share of reads answered from the cache still looks fine on the global graph.

The trap: A thundering herd on the popular keys at expiry: thousands of requests miss together and each one asks the database the same question. The lifetimes were aligned by the bulk load, and there was no one-loader rule. The global hit ratio hid that a few keys were driving the incident.

What to build: Slightly randomized lifetimes (TTL jitter) on bulk loads, a one-loader-per-key rule across the fleet, serving a slightly old copy while one request refreshes where the product allows, and a view of keys thrown out, miss rate per key family, and database queries per second on the miss path.

Scene 3: Inconsistent reads after failover

The moment: A shopper changes their profile name and opens the app in two browser tabs to check the result. One tab shows the new name and the other shows the old name, and a “lost” change reappears after a refresh. On-call sees successful failovers in the log and assumes the cluster is healthy.

The trap: Clients had cached an out-of-date map of who owns which keys and sent writes to a dead primary. The replica that was promoted was not caught up, and there was a brief window where two machines both thought they were primary before a rising generation number (epoch) fenced the old one. Users also expected to read their own write from replicas, which nobody had built.

What to build: Push ownership-map updates quickly, read from the primary for critical keys after a write, bump the epoch on every promotion, and document that replicas do not give read-your-writes unless you build sticky routing or a version check.

Scene 4: Cache slow, then database stampede

The moment: At the start of a busy shopping hour, shoppers see timeouts on product pages and checkout. The database is overloaded, and the team insists “the cache cluster is up” because every machine still answers a health ping.

The trap: One very popular key saturated a single primary that can only do one piece of work at a time, the pool of connections from the apps to the cache filled up, and applications used unbounded parallel database refills on every miss or timeout. A slow cache is not a down cache, so up-or-down alerts never fired.

What to build: A circuit breaker on the database fallback (stop calling when the cache is too slow), a bulkhead (a cap on concurrent refills per key), and alerts on slow cache reads (GET p99) and time spent waiting for a connection, not only on whether the primary answers a ping.

[ Apps spike GETs ] → [ Cache primaries at CPU max ]
       → latency ↑, timeouts
       → clients fall back to DB in parallel → DB overload → worse cache refill

In the interview: Pick one scene. Tell what users felt, which metric lied, and what you would ship: one loader per key, jitter, a throttled rebalance, or a primary read after a write. You do not need all four. You need to sound like someone who knows green is not the same as correct.

Bottlenecks and tradeoffs

Those stories all trace back to a small number of choices where you cannot have both things you want. Interviewers listen for whether you can name the tension, say what breaks, and pick a side on purpose. Here are the four that matter most for a distributed cache.

Memory versus hit ratio

The tension — A bigger cluster answers more reads from memory and spares the database, but RAM is not free and every extra replica multiplies the memory bill.

What breaks — Past a point you pay more for little gain, and when the working set no longer fits you get eviction thrash: cold keys push out hot ones, misses rise, and the database quietly becomes the cache.

What teams do — Track how many keys are thrown out and how slow the worst reads are, keep only the hottest data in the fastest tier, right-size each machine, and use least-frequently-used eviction or short lifetimes for the long tail.

Say in the interview — Cache size is a cost decision tied to a speed goal, not “as big as possible.”

Strong consistency versus cache speed

The tension — Applications want a user who just saved something to see the new value on the next read, but caches favor speed and availability, and replicas lag behind the primary.

What breaks — Promising “always fresh everywhere” without reading the primary or waiting for synchronous copies.

What teams do — Keep the database as the truth, delete the cached copy on write, use short lifetimes on contested keys, and send critical reads to the primary.

Say in the interview — Most caches are eventually a little stale. Say what your application tolerates and what it does not.

Rebalance versus availability

The tension — Adding machines should increase capacity, but moving keys while traffic is live risks serving the wrong value and spiking latency.

What breaks — Two machines both answering for the same key mid-move, a storm of redirects, and a wave of database misses if you used modulo hashing.

What teams do — Place keys on a ring so only a wedge moves, use a handoff protocol, throttle copying, and have the old owner redirect to the new one for keys that are mid-move.

Say in the interview — Adding a node is an operations event, not a free horizontal scale button.

Cache-aside versus write-through

The tension — Cache-aside is simple but can race when clearing copies, while write-through keeps the cache and database in step but slows every write.

What breaks — A stale cache after a write with no delete, or write-through hurting write latency at scale.

What teams do — Cache-aside plus delete after commit for most read-heavy products, and write-through for small subsets that must stay strongly consistent.

Say in the interview — Pick the pattern from the read/write mix and how fresh reads must be, not from buzzwords.

If you remember one thing: A cache can fail open to the database in the worst way unless you engineer miss and fallback behavior.

Interview tips

You now have a full design and the tradeoffs behind it, so the last thing to practice is defending it when the interviewer pushes back. These five are the pushes that come up most after you finish the design:

Modulo hashing — your cluster reshuffles when you scale

In the room

You might say: “We hash the key and take the remainder after dividing by the number of servers to pick which machine owns it.”

Interview Push: “You have four nodes and add a fifth—what happens to every existing key?”

Land here: That remainder trick uses the machine count as the divisor, so when the count changes from four to five, almost all keys remap to a different machine. The cluster then sees a mass miss event: database queries per second spike and latency jumps, exactly when you thought you were scaling out. Consistent hashing places keys and nodes on a ring, and adding a node only moves keys in the wedge near the new position. Virtual nodes (many small stretches per physical machine) spread load evenly. Rebalance runs in the background with throttling and handoff so two owners never serve different values for the same key during migration.

Invalidate on update — one row, many cache keys

In the room

You might say: “When the database updates, we clear the cached copy so the next read loads the fresh row.”

Interview Push: “One product row feeds the product page, a search snippet, and a cart preview—how do you clear all of those?”

Land here: Start from the naming scheme (product:{id}, product:{id}:summary) and either keep an explicit list of keys to clear on write, echo a version column from the database into cached values, or publish change events to a small clearing service. Wildcard deletes (KEYS product:9001*) do not scale on large clusters. Delete the affected keys after the database commit, not before. A lifetime on each key (TTL) is a safety net for a missed clear, not the main consistency tool for price or inventory.

Read-after-write — replicas lie politely

In the room

You might say: “We read from replicas for speed on every GET so the primary is not overloaded.”

Interview Push: “A user saves their display name, refreshes immediately, and still sees the old name—why?”

Land here: The copy on the replica had not caught up yet—that delay is replication lag. The write landed on the primary, and the read hit a replica that was behind by milliseconds or seconds. For a user who must see their own write, route that user's next read to the primary, stick them to a replica that has caught up, or embed a version in the value and retry on a mismatch. Say which fields tolerate staleness (a display name for a few seconds) and which do not (an account balance). The cache is not a magic fix; routing and the freshness rule are.

Thundering herd — TTL is a scheduled database spike

In the room

You might say: “We set a lifetime on each key so outdated copies expire on their own.”

Interview Push: “A viral product key expires at the top of the hour—what hits the database?”

Land here: Every cached copy expires together if those lifetimes were aligned when the values were first loaded. Thousands of concurrent misses each issue a database query unless one request does the loading while the others wait, and remember that a one-loader rule inside one process still allows one query per application server, so use a short lock key in the cache to get one load for the whole fleet. Add a little random spread on lifetimes (base + random), a small chance of refreshing early, and serving a slightly old copy while one request refreshes where the product allows. This is why expiry is a design problem, not an operations surprise.

Cache versus database — who owns truth

In the room

You might say: “The cache is our source of truth—it's faster than the database.”

Interview Push: “A primary crashes with no persistence—what did we lose, and where do reads go?”

Land here: Anything kept only in memory can disappear at any time (ephemeral), so the database is the truth. After a failover or cold start, the cache misses and refills from the database, and anything not yet written to the database is gone. Optional on-disk snapshots or append-only logs (RDB/AOF) trade a chance of surviving a restart for slower tail latency when large heaps are copied, so mention that cost if you bring persistence up, and do not confuse cache snapshots with database reliability. A strong answer is: “We never write product inventory only in the cache without the database commit path.”

Red flags to avoid without correcting yourself: “Cache is source of truth,” “modulo hash when nodes change,” “invalidate with KEYS *,” “replica read right after write with no plan,” “miss means fire unlimited DB queries.”

After each interview push, close with one real part of the design you would build—consistent hashing with virtual nodes, delete-after-commit, one loader per key, or primary reads on critical keys—not “we'll optimize later.”

What should stick

After working through the requirements, the decisions and the depth sections, you should be able to explain:

  1. Same key, same machine — Route each key to one owner so reads and writes agree. When you add a machine, only a slice of keys moves, because a hash ring or slots keep the move small.
  2. Hit / miss / clear — On a miss, load the database once and store a copy. After a database write, delete that copy (or bump a version) so readers do not keep yesterday's value.
  3. Database is the source of truth — The cache is a speed layer that can disappear. A cold start or failover refills from the database.
  4. Expiry can stampede the database — Many clients missing the same key at once burns the database, so stagger expiry and load once per key across the fleet.
  5. One viral key can melt one machine — Extra copies, replica reads, or an app-side buffer help. “Add more nodes” alone does not.

Tell it in the room: “The database is truth. A GET hits the fast copy or loads the database once and stores the value with a lifetime. On a write I commit to the database and then delete the cached copy. A hash ring with virtual nodes means adding a server moves only a wedge of keys. I’d call out one hot key and many clients missing the same key at once, and handle both on purpose.”

Practice next: Open Design a Distributed Cache on Practice System Design and run a timed attempt using clarifying questions and one read story and one write story, without peeking until you have a draft on the board.

Key Takeaways

  • The database is the source of truth, and the cache is a shared cluster of copies that can disappear; never keep the only copy of something important in the cache.
  • Use cache-aside: read the cache first, load the database once on a miss, and after a database write commit first and then delete the cached key, with a TTL as a backstop.
  • Place keys with a hash ring and virtual nodes (or slots) so adding a machine moves only a wedge of keys, instead of the roughly 80% that modulo hashing remaps.
  • Give each slice a primary and asynchronous replicas, promote with a higher epoch, and accept a short window of staleness and a few lost writes.
  • One very popular key needs its own plan (a short-lived copy in each application server, replica reads, or a split key), because sharding spreads many keys, not one.
  • Protect the database from misses with one loader per key across the whole fleet, TTL jitter, and stale-while-revalidate where the product allows.
  • Watch database QPS on the miss path, per-machine QPS and CPU, evictions, and replication lag, not only the global hit ratio.

Quick revision notes

The problem in your own words. A distributed cache keeps fast copies of popular data on a group of machines so the database does not have to answer every read. The hard part is that copies can be missing, out of date, or concentrated on one machine, and the database is the thing that suffers when any of those happens.

The design in one breath. The application hashes the key, sends the read to the machine that owns it on a hash ring, and on a miss loads the database once and stores the value with a jittered lifetime. On a database write it commits and then deletes the cached key. Each slice has a primary and asynchronous replicas, promoted with a higher epoch when the primary dies, and memory is bounded by LRU-style eviction.

The choices you must be able to defend. Cache-aside because the traffic is read-heavy and the database is the truth. Delete after commit instead of updating the cache. A ring with virtual nodes instead of modulo hashing. Asynchronous replicas, accepting a few lost writes. Extra copies for a hot key, and one loader per key for a miss storm.

The numbers worth remembering. About 10 million operations a second means on the order of 100 primary machines at roughly 100,000 operations a second each, estimated and confirmed by a load test. 100 GB of data becomes about 300 GB of RAM with two replicas. One percent of traffic on one key is about 100,000 operations a second on a single machine. Adding a fifth machine with modulo hashing remaps about 80% of keys.

The mistakes that cost points. Calling the cache the source of truth, using modulo hashing, updating the cache before the database commit, clearing keys with wildcards, reading right after a write from a lagging replica, and letting every miss go straight to the database.

Frequently asked follow-ups

  • What happens when you add a node—modulo hash versus consistent hashing?
  • How do you handle a hot key on one shard?
  • How do you invalidate cache after a database update?
  • Can the cache be strongly consistent everywhere?
  • How is this different from just using Redis?

Deep-dive questions and strong answer outlines

Walk through a GET on a sharded cache cluster.

The client or proxy hashes the key to a token on the ring, routes to the owning primary (or a replica if stale reads are acceptable). On a hit, return the value from memory. On a miss in cache-aside, the application loads from the database, then SETs the key with a TTL. Mention singleflight on miss (one loader per key while others wait) so one viral expiry does not spawn thousands of identical database queries.

You add one server to the cluster—what happens to key placement?

With modulo hashing (hash(key) % N), almost every key moves when N changes—cache miss storm and database spike. With consistent hashing and virtual nodes, only keys in the wedge near the new token migrate; rebalance runs in the background with throttling and handoff so two owners do not serve conflicting values during migration.

Explain cache-aside and how you keep data fresh after a write.

The application reads the cache first; on miss it loads the database and populates the cache. On database update, the application commits the write then DELETEs affected cache keys (or bumps a version embedded in the value). Avoid updating the cache before the database without careful ordering—you can race and write stale data back. TTL is a safety net, not the primary invalidation strategy for critical fields.

The primary crashes—what happens to in-flight writes and reads?

Health checks detect failure; a replica promotes with an epoch or quorum bump so old primaries cannot accept writes. Clients refresh topology. In-flight writes may be lost depending on sync policy—say that out loud. Reads may hit stale replicas until catch-up; money paths often read the primary. The cache is ephemeral; the database is still source of truth.

One logical key gets a million reads per second—what do you do?

Hashing spreads average load but not one dominant logical key—all traffic still lands on one primary shard. Mitigate with read replicas for that key, an in-process L2 cache in app servers, splitting the key into sub-ranges, or replicating the hot value to multiple nodes. Alert on per-shard QPS and CPU, not only cluster averages.

A popular key's TTL expires at the same second—what hits the database?

Thundering herd: thousands of concurrent misses each query the database. Mitigate with singleflight inside each app process plus a short lock key in the cache so only one process across the fleet reloads, TTL jitter so expiries spread, probabilistic early refresh, and stale-while-revalidate where product allows slightly old data during refill.

AI feedback on your design

After a practice session, InterviewCrafted summarizes strengths, gaps, and interviewer-style expectations—similar to a written debrief. See a static example report, then practice this problem to get feedback on your own answer.

FAQs

Q: Do I need to memorize Redis protocol opcodes in the interview?

A: No. Interviewers care about behavior—what a get, save, delete, and lifetime mean, how an increment stays correct, and how clients follow redirects when ownership changes—not whether you recall byte layouts. Name the operations and what the application does on a hit versus a miss.

Q: Where does CAP fit for a distributed cache?

A: When the network splits the cluster, caches usually keep answering and accept slightly old reads rather than refusing every request. That matches most product needs—brief staleness beats an outage. Strong freshness on every read means reading the primary (or waiting for a majority), which costs extra latency.

Q: Is the cache the source of truth?

A: Almost always no. The database (or durable store behind the cache) owns the official record. The cache is a speed layer that can disappear. If a primary dies without saving memory to disk, you lose in-memory state and rebuild from the database on miss—design clearing and refill accordingly.

Q: How do I talk about persistence (RDB/AOF) without getting lost?

A: Saving memory to disk trades a chance of surviving a restart for operational cost—copying a large heap can spike the slowest reads, and how often the append log is flushed changes how much you can lose. For most cache interviews, say the database is truth and disk snapshots are optional for a warmer restart, not a substitute for the database.

Practice interactively

Open the practice session to use the canvas and stages, then review AI feedback.