Distributed Cache System Design

Introduction

A flash sale starts. One product id is in every cart. Your cache cluster shows healthy CPU—but the database catches fire and checkout fails for everyone.

A distributed cache is fast mutable state spread across machines. The interview is not “Redis versus Memcached trivia.” It is how clients find the right shard, what moves when a node joins, what replicas promise (and do not), and what happens when a hot key or TTL expiry turns the cache into a load generator for the database.

Weak answers say “we add Redis.” Strong answers separate routing (hash ring, virtual nodes) from the data plane (GET/SET on primaries), name cache-aside with explicit invalidation, and plan for thundering herd before the interviewer asks about a viral key expiring at noon.

The line that saves you: The cache is not the source of truth. The database is. The cache protects the database only if you design routing, eviction, invalidation, and miss behavior—not just hit ratio on a dashboard.

If you remember one thing: A green cache graph can coexist with a melting database when hot keys, rebalance, or herd on expiry are ignored.

How to approach

Spend the first few minutes turning “design a distributed cache” into a contract you can defend. Talk through clarifying questions out loud, then walk one GET on miss and one database update with invalidation before you name eviction flags.

Clarifying questions (say these in the room)

Start with whether the cache is ephemeral acceleration or must survive restarts, because that drives persistence talk and how hard you lean on the database on cold start.

You: “Is this a pure ephemeral cache in front of a database, or do we need the cache cluster itself to survive restarts with minimal cold miss storm?”

Interviewer: “Ephemeral is fine—database is source of truth.”

That answer lets you park long persistence digressions and focus on cache-aside and invalidation.

You: “What consistency bar do reads need—can product tolerate stale reads from replicas for a few hundred milliseconds, or do some keys need read-your-writes from the primary?”

Interviewer: “Mostly eventual; profile name after save should not flip-flop if you can help it.”

Now you owe replica versus primary routing rules and replication lag honesty—not “we always read replicas for speed.”

You: “Is the workload read-heavy with occasional writes, or do we need write-through semantics where every write must hit the database and cache together?”

Interviewer: “Read-heavy; cache-aside is the common path.”

That steers you toward lazy load on miss and delete on database update instead of write-through unless they push on strong consistency.

You: “How skewed is access—uniform keys, or do we expect hot keys like one product id during a sale?”

Interviewer: “Very skewed—call out hot keys.”

You must plan mitigation beyond “add more nodes,” because sharding spreads averages, not one viral logical key.

You: “When we add or remove nodes, can we tolerate a background rebalance with throttled migration, or do we need zero key movement?”

Interviewer: “Background rebalance is OK—explain what moves.”

That opens consistent hashing and handoff—not modulo hash that reshuffles everything.

You: “I want to leave full SQL query caching, CDN edge caching, and cross-region active-active cache writes out of scope so we can focus on application key-value cache sharding, replication, and invalidation. Does that match what you want?”

Interviewer: “Yes—focus on the distributed cache layer.”

Saying out of scope early stops you from burning half the interview on CDN or multi-master conflict resolution.

Minute pacing (about 40 minutes)

In the room (a fuller opening you can actually say): “I’d treat the cache as ephemeral acceleration—the database stays source of truth. Clients route keys with consistent hashing and virtual nodes so adding a server only moves a wedge of keys, not the whole keyspace. Reads are cache-aside: GET from the owning shard, on miss load the database once with singleflight, then SET with TTL. On database write, commit then DELETE affected keys—or bump a version in the value—so we don’t serve stale data after update. Replicas scale read-heavy keys where staleness is OK; critical keys read the primary. I’d plan for hot keys with replica reads or an app-level L2, TTL jitter, and singleflight so one expiry doesn’t stampede the database. Node add is a throttled rebalance with handoff, not a big-bang migration.”

If you remember one thing: Tell one GET miss story and one invalidation story before you recite LRU versus LFU.

Out of scope (and why you park these)

Write a short out-of-scope list on the board so the interviewer sees you can bound the problem.

CDN and browser caching. Edge caches static assets and some API responses at the network edge. They matter for global latency but do not replace an application-tier distributed cache for mutable session and catalog state. Acknowledge them; design the in-memory cluster the app servers talk to.

Caching arbitrary SQL query results. Query-result caches tie invalidation to table changes and are fragile at scale. The interview usually wants entity key-value caching (user profile, product row) with explicit invalidation on write.

Active-active multi-region cache with conflicting writes. Two regions accepting writes to the same key without coordination creates merge conflicts. Most designs use read replicas per region with async replication and stale SLAs, or route writes to one region—unless the interviewer explicitly expands scope.

Full transactional cache covering cross-shard ACID. A cache is not a database. Cross-key transactions on a sharded cache are out of scope for a typical round; per-key atomic ops (INCR, CAS) on one primary are enough.

Capacity estimation

Rough numbers anchor the whiteboard (tune in the room):

Worked skew sketch: Ten million operations per second sounds evenly spread only on paper. If one product id during a flash sale absorbs even one percent of read traffic, that is on the order of 100,000 ops/s on one logical key—one primary thread, one NIC, one hash slot. Adding nodes does not split one key unless you replicate or shard the key itself.

Implications:

• You shard to spread RAM and CPU across machines—but hot keys are a logical problem, not solved by fleet size alone.
• Eviction must be explicit when working set exceeds RAM; otherwise miss rate climbs and the database becomes the cache.
• Connection counts from thousands of app servers matter; smart clients or proxies pool connections to shards.

If you remember one thing: Capacity math exists to expose skew—averages lie when one key owns the sale.

High-level architecture

Clients either embed a smart client (knows the hash ring and token ownership) or talk to a proxy (stateless routing layer). The cluster exposes a logical keyspace partitioned into slots or token ranges on a ring; each range is owned by a primary with one or more replicas. A configuration service or gossip protocol publishes membership and epoch. Persistence (RDB snapshots, AOF logs) is optional and usually off the critical read path—the database remains truth.

Who owns what:

• Routing layer (client library or proxy) — Hash(key) → token → node; handles MOVED/ASK-style redirects when topology changes (Redis Cluster pattern).
• Shard primary — Serializes writes per key; serves reads or delegates to replica depending on consistency mode; enforces memory limit and eviction.
• Replicas — Async copy from primary; promoted on failover after health checks and epoch bump.
• Control plane — Add/remove nodes, rebalance throttles, placement across racks/AZs, hot key alerts.

[ App servers ]
       |
       |  GET / SET / DEL (sync, sub-ms target on hit)
       v
[ Smart client or Proxy ]  ----membership----► [ Config / gossip ]
       |
       |  consistent hash: key → token → primary
       v
+------------------+     async replication    +------------------+
| Primary A        | ----------------------► | Replica A'         |
| (token range T1) |                         | (read optional)  |
+------------------+                         +------------------+
+------------------+                         +------------------+
| Primary B        | ----------------------► | Replica B'       |
| (token range T2) |                         |                  |
+------------------+                         +------------------+

  Rebalance (background): new node C → migrate token ranges → throttle → HANDOFF

In the room: Walk one GET: hash key → route to primary → memory lookup → hit or miss. Then one failover: primary dead → replica promoted → clients refresh ring. Mention that rebalance is async and throttled—not instant free capacity.

If you remember one thing: GET is hash → primary (or replica with stale rules); rebalance is an ops event with migration protocol, not a magic scale button.

Core design approaches

These patterns answer who loads on miss and who updates on write. Most read-heavy interviews land on cache-aside; know the others to compare when the interviewer pushes on consistency.

Cache-aside (lazy loading)

The application checks the cache first. On miss, it loads the database, then SETs the cache with a TTL. On database update, the app commits the write, then DELETEs cache keys (or bumps a version in the value). Simple and common; invalidation is the app’s job.

When to pick: Read-heavy workloads where slightly stale data is OK between TTL and invalidation, and the database is clearly source of truth.

Read-through

The cache module itself loads the database on miss—the application only talks to the cache API. Invalidation can still be triggered by the app or change-data-capture on write.

When to pick: You want a uniform cache API and central miss handling; teams accept more logic inside the cache layer or sidecar.

Write-through

Writes go to cache and database together before ack. Reads always see what was written through the cache path.

When to pick: Stale reads after write are unacceptable and you accept higher write latency. Less common at huge scale than cache-aside plus delete-on-write.

Write-behind (write-back)

Writes ack after cache update; flush to database asynchronously. Fast writes; loss risk if the cache dies before persist—rare for generic cache interviews unless explicitly scoped.

When to pick: Write-heavy buffering where loss window is product-tolerated—say the durability tradeoff out loud.

If you remember one thing: Cache-aside means the app owns invalidation; the database is truth—never pretend the cache alone is authoritative.

Data model

Think in three layers: how keys map to shards, what lives in each slot, and how the cluster knows who owns what.

Key → slot / token (routing)

Clients compute hash(key) → token → owner locally (smart client) or ask a proxy. During rebalance, a key may temporarily redirect (MOVED/ASK) until migration completes.

Value + TTL + version (payload)

Interview land: On database update, prefer DELETE or version bump over blind SET of old data—ordering races can repopulate stale values if a slow read finishes after your write.

Membership (cluster topology)

Gossip or a small config service pushes updates; clients cache topology with TTL and refresh on redirect storms.

If you remember one thing: Draw key → token → primary separately from value + TTL—routing and payload are different failure domains.

Detailed design

Think of two everyday moments: an app reads a product page while the flash sale is hot, and later ops changes that product’s price in the database. The cache only helps if both paths are intentional.

Read path (cache-aside)

The application asks for a key—say a product cache entry. A smart client (or proxy) hashes that key and sends the GET to the machine that owns that slice of the keyspace. If the value is in memory, the call returns in well under a millisecond on a healthy cluster. That is the happy path.

If the key is missing or expired, you are looking at a cache miss, which is not the same as “the cache is broken.” The application then loads from the database (source of truth), writes the result back into the cache with a TTL, and returns it to the user. The dangerous part is when thousands of concurrent requests miss the same viral key at once: without coordination each one hits the database, which is how a healthy-looking cache still melts Postgres. The fix is singleflight (one loader per key; others wait or retry GET), plus TTL jitter so expiry times do not align like a marching band.

Batch helpers such as MGET reduce round trips when you need many keys. Keep miss and error separate in metrics: a timeout should not automatically become “load the database forever” unless you have a deliberate fallback policy and time limits.

Write path (database update, then invalidate)

When someone updates the database row, commit that write first. After the commit succeeds, delete (or version-bump) the cache keys that were derived from that row. Deleting after commit is safer than updating the cache before the database is sure. Some teams also put a version number inside both the database row and the cached value so a slow reader that filled cache with old data cannot overwrite a newer version forever.

Name a race in the room: Thread A misses and starts a slow database read of the old price. Thread B updates the price and deletes the cache key. Thread A then SETs the old price back into the cache. Short TTLs, version checks on SET, and correct delete-after-commit ordering shrink that window.

Atomic operations and memory pressure

Operations like increment or compare-and-set work well when one primary owns the key. They are a poor place to invent cross-shard transactions—push that work to the database if you need multi-key atomicity. When RAM fills up, the cache evicts keys under its policy (often LRU-style) even if their TTL has not expired yet. Rising eviction counts with a flat hit ratio usually means the working set no longer fits; adding nodes without fixing hot keys will not save you.

In the room (spoken): “Reads are cache-aside with singleflight on miss. Writes commit to the database, then delete cache keys. I treat the cache as ephemeral acceleration—not the system of record.”

Architectural dig: one GET on miss and one database invalidation

High-level boxes showed ownership. This dig is what happens inside the cluster when traffic spikes—where naive hashing and naive miss handling fail.

1) GET on miss (cache-aside)

Responsibility: Primary owns in-memory value for keys in its token range. Application owns refill logic and thundering herd control.

Numbered path:

1. Client hashes product:9001 → primary P.
2. P returns miss (key absent or expired).
3. App singleflight ensures one goroutine/thread loads the database.
4. Loader reads row from DB, SET product:9001 with TTL on P.
5. Concurrent waiters retry GET and hit.

What breaks first: Step 3 skipped—10,000 concurrent misses after TTL expiry → 10,000 identical database queries (thundering herd). Mitigate with singleflight, TTL jitter, stale-while-revalidate.

Interview land: “Miss path is one coalesced database read per key per expiry window—not one per waiting request.”

2) Database update → invalidation

Responsibility: Database is source of truth. Cache must not serve stale product price after commit.

Numbered path:

1. App updates products row id=9001 in DB; commit.
2. App DEL product:9001 (and related keys like product:9001:inventory if mapped).
3. Next GET misses, reloads fresh row, SETs cache.

Bad pattern: UPDATE cache before DB commit—crash leaves cache newer than DB. Better: commit first, then DELETE cache keys.

One product row → many cache keys: Document key naming (product:{id}, denormalized feed slices). Pattern DELETE (product:9001*) is expensive at scale—prefer explicit key list or version stamp in value.

Interview land: “Invalidate by delete after commit; TTL is backup, not the primary consistency mechanism for price changes.”

3) Node add: modulo hash versus consistent hashing

Bad design: hash(key) % N

You run four cache nodes. slot = hash(key) % 4. You add a fifth node. N changes from 4 to 5.

Almost every key maps to a different slot. The cluster looks like a mass invalidation: miss storm, database spike, latency cliff—right when you thought you were scaling up.

Before: hash(k) % 4  →  keys spread on 4 nodes
After:  hash(k) % 5  →  ~80%+ keys remap → miss wave → DB overload

Great design: consistent hashing + virtual nodes

Keys and nodes both sit on a ring. A key belongs to the first node clockwise from its hash. Add node C: only keys in the wedge between C’s predecessor and C move to C. Existing keys on other arcs stay put.

Figure: Consistent hashing migrates a wedge; modulo hash reshuffles almost everything.

Virtual nodes: Each physical machine owns many tokens on the ring so load stays balanced when node count is small. Rebalance runs in background with throttle and handoff so two primaries do not serve conflicting values mid-migration.

Interview land: “Add node moves a wedge, not the universe—rebalance is throttled and explicit.”

4) Thundering herd on TTL expiry

Bad design: aligned TTL, no coalescing

Ten thousand clients cache product:9001 with TTL = 3600 set at the same second during a sale warmup. At expiry, all keys vanish together. Every client sees miss; every app thread queries the database.

T=3600s  key expires on primary
T=3600s  10k apps → 10k DB SELECTs for same row
         DB connections maxed; cache “healthy”

Great design: jitter + singleflight + optional stale-while-revalidate

• TTL jitter: TTL = base + random(0, jitter) so expiries spread.
• Singleflight: one loader repopulates; others wait or serve stale briefly.
• Stale-while-revalidate: return last good value while async refresh runs (if product allows).

Scale inside the box: Even with coalescing, one hot key still hits one primary—shard CPU and network saturate. Combine herd control with hot key replication or app L2.

Interview land: “Expiry is a scheduled DDoS on your database unless jitter and singleflight are designed in.”

Numbered happy path (put it together)

1. GET → hash → primary → hit or miss.
2. Miss → singleflight → DB load → SET with jittered TTL.
3. DB write → commit → DEL cache keys.
4. Node add → wedge migration → throttled handoff → clients refresh ring.

If you remember one thing: The dig is wedge not reshuffle for topology, and one loader not N for miss storms.

Key challenges

• Hot keys: One logical key → one primary shard—CPU, memory bandwidth, and single-threaded primary limits bite before cluster average looks stressed.
• Rebalancing: Moving keys without dual ownership bugs—wrong value served during migration; need handoff protocol and throttled copy.
• Replication lag: User saves profile, reads replica, sees old name—apps must know which replica and whether read-your-writes is required.
• Invalidation mapping: One database row may map to dozens of cache keys; pattern deletes do not scale; version stamps help.
• Eviction thrash: Working set larger than RAM → constant eviction → rising miss rate → database overload.
• Split brain after partition: Two primaries accept writes—mitigate with quorum, epoch, fencing; caches usually favor availability with stale reads.

If you remember one thing: Hot keys and rebalance hurt more in production than picking LRU versus LFU on day one.

Scaling the system

• Horizontal: Add nodes; consistent hashing migrates adjacent wedges only; throttle migration during peak.
• Vertical: Bigger RAM on hot shards—sometimes cheaper than perfect algorithmic fairness for one monster key.
• Read scaling: Replica reads for read-heavy, stale-OK keys; primary for linearizable per-key sequence or read-after-write paths.
• Hot key mitigation: Replicate hot value to multiple nodes; in-process L2 in app tier; split logical key into sub-keys (product:9001:shard:{0..7}) with merge on read.
• Multi-region: Active-active cache writes are hard (conflicts); often read replicas per region with async replication and documented stale SLAs.

If you remember one thing: Replica reads scale throughput; primary wins when stale is unacceptable—and one hot key needs its own plan.

Failure handling

Degraded UX: Slightly stale data is often acceptable; wrong price or balance is not—route those reads to primary or short TTL plus strict invalidation.

If you remember one thing: Under partition, most caches choose availability + stale reads—unless the interviewer scopes money paths to primary reads.

API design

Caches expose binary protocols (RESP for Redis) or thin REST wrappers. Interview clarity matters more than opcode trivia.

Core operations:

Cluster routing (client view):

Error semantics: Distinguish miss (key not present—trigger cache-aside load) from error (timeout, CLUSTERDOWN—do not blindly hammer database). HTTP wrappers often use 404 for missing resource versus 503 for cache unavailable—document the mapping.

Request flow (GET, cache-aside):

App --GET key--> Client lib --hash--> Primary shard
                              |
                              hit --> return value
                              miss --> singleflight --> DB --> SET + TTL --> value

If you remember one thing: A miss triggers a controlled refill path—not an unbounded parallel database query per client.

Observability (what you measure)

Vanity hit ratio hides miss storms and hot shards. Measure what proves the cache protects the database.

Alerts that catch real incidents: Top 10 keys by QPS; miss rate ×10 in five minutes; DB connection pool >80% while cache CPU calm; rebalance duration over SLO.

If you remember one thing: Chart database QPS from cache miss alongside hit ratio—green cache plus rising DB means design failure.

Security and abuse

• Network isolation: Cache ports not on public internet; VPC, TLS in transit where supported, ACLs by app role.
• No secrets in cache keys or values logged at debug—keys often embed user ids; treat values as sensitive if they mirror database rows.
• AuthZ at app layer: Cache does not replace authorization—any app with cluster access can read keys unless encrypted at rest in value (uncommon for generic cache).
• Denial of service: Unbounded key space from attackers (user:random:*) fills memory—rate limit key creation, cap key length, monitor memory per tenant prefix.
• Injection: If values deserialize to objects, use safe formats (JSON with schema)—not arbitrary pickle.

If you remember one thing: The cache is a performance layer, not a security boundary—auth stays in the application.

Cost awareness

• RAM is the main bill: 100 GB replicated three ways is 300 GB of RAM before overhead—right-size working set versus “cache everything.”
• Cross-AZ replication traffic: Every write replicates—hot write keys cost network as well as CPU.
• Over-caching cold data: Storing long-tail keys evicts hot data—LFU or TTL discipline reduces waste.
• Rebalance and ops time: Node adds during peak cost engineer time and customer latency—schedule migrations.
• Database fallback cost: Miss storm turns cache savings into database scale emergency—cheaper to fund jitter and singleflight than emergency DB vertical scale.

Say in the interview: Cache size is a cost decision with an SLO, not “as big as possible.”

When users are stuck but the dashboards look fine

[ Apps spike GETs ] → [ Cache primaries at CPU max ]
       → latency ↑, timeouts
       → clients fall back to DB in parallel → DB overload → worse cache refill

Bottlenecks and tradeoffs

Memory versus hit ratio

The tension — Bigger cluster means fewer database hits; RAM is not free and replication multiplies footprint.

What breaks — Diminishing returns; eviction thrash when working set exceeds RAM; cold keys evict hot ones.

What teams do — Track evicted_keys and tail latency; tier hot data; right-size per shard; LFU or TTL on long-tail keys.

Say in the interview — Cache size is a cost decision with an SLO, not “as big as possible.”

Strong consistency versus cache speed

The tension — Apps want read-your-writes; caches favor speed and partition tolerance with stale replicas.

What breaks — Promising strong consistency everywhere without primary reads or sync replication.

What teams do — Database as truth; delete on write; short TTL on contested keys; primary read for critical paths.

Say in the interview — Most caches are eventually stale—say what your app tolerates and what it does not.

Rebalance versus availability

The tension — Adding nodes should increase capacity; migrating keys risks wrong values and latency spikes.

What breaks — Dual ownership during move; redirect storms; database miss wave if you used modulo hash.

What teams do — Consistent hashing wedge migration; handoff protocol; throttled copy; hinted handoff for reads during move.

Say in the interview — Adding a node is an ops event, not a free horizontal scale button.

Cache-aside versus write-through

The tension — Cache-aside is simple but invalidation races exist; write-through is stronger but slower on writes.

What breaks — Stale cache after write without delete; write-through melting write p99 at scale.

What teams do — Cache-aside plus delete after commit for most read-heavy products; write-through for small strongly consistent subsets.

Say in the interview — Pick pattern from read/write ratio and consistency bar, not from buzzwords.

If you remember one thing: Caches fail open to the database in the worst way unless you engineer miss and fallback behavior.

Interview tips

You have the full design in mind now. These five back-and-forths are how interviewers test whether you can defend it under pressure. Each one starts with a tempting shortcut, an interview push, and a clear answer.

Modulo hashing — your cluster reshuffles when you scale

You might say: “We hash the key modulo the number of servers to pick a shard.”

Interview Push: “You have four nodes and add a fifth—what happens to every existing key?”

Land here: Modulo changes the divisor—almost all keys remap. The cluster experiences a mass miss event: database QPS spikes, latency jumps, exactly when you thought you were scaling out. Consistent hashing places keys and nodes on a ring; adding a node only moves keys in the wedge near the new token. Virtual nodes spread load evenly across physical machines. Rebalance runs in the background with throttling and handoff so two owners never serve different values for the same key during migration.

Invalidate on update — one row, many cache keys

You might say: “When the database updates, we invalidate the cache.”

Interview Push: “One product row feeds the product page, search snippet, and cart preview—how do you invalidate?”

Land here: Name a key naming scheme (product:{id}, product:{id}:summary) and either maintain an explicit invalidation list on write, use a version column in the database echoed in cached values, or publish change events to a small invalidation service. Pattern deletes (KEYS product:9001*) do not scale on large clusters. DELETE affected keys after database commit—not before. TTL is a safety net for missed invalidations, not the primary consistency tool for price or inventory.

Read-after-write — replicas lie politely

You might say: “We read from replicas for speed on every GET.”

Interview Push: “User saves their display name, refreshes immediately, still sees the old name—why?”

Land here: Replication lag. The write landed on the primary; the read hit a replica behind by milliseconds or seconds. For read-your-writes, route that user’s next read to the primary, use sticky routing to a caught-up replica, or embed a version in the value and retry on mismatch. Say which fields tolerate staleness (display name for a few seconds) versus which do not (account balance). The cache is not a magic fix—routing and consistency mode are.

Thundering herd — TTL is a scheduled database spike

You might say: “We set TTL so data stays fresh automatically.”

Interview Push: “A viral product key expires at the top of the hour—what hits the database?”

Land here: Every cached copy expires together if TTLs were aligned at warm load. Thousands of concurrent misses each issue a database query unless singleflight coalesces to one loader. Add TTL jitter (base + random), probabilistic early refresh, and stale-while-revalidate where slightly old data during refill is product-OK. This is why expiry is a design problem, not an ops surprise.

Cache versus database — who owns truth

You might say: “The cache is our source of truth—it’s faster.”

Interview Push: “Primary crashes with no persistence—what did we lose, and where do reads go?”

Land here: In-memory state is ephemeral. Database is truth. After failover or cold start, cache misses refill from the database; anything not yet written to the database is gone. Optional RDB/AOF persistence trades durability for fork latency on large heaps—mention p99 impact if you bring it up, but do not confuse cache persistence with database reliability. Strong answers: “We never write product inventory only in cache without the database commit path.”

Red flags to avoid without correcting yourself: “Cache is source of truth,” “modulo hash when nodes change,” “invalidate with KEYS *,” “replica read right after write with no plan,” “miss means fire unlimited DB queries.”

After each interview push, close with one real part of the design you would build—consistent hashing with vnodes, delete-after-commit, singleflight, or primary read on critical keys—not “we’ll optimize later.”

What should stick

After reading this guide, you should be able to explain:

1. Consistent hashing + virtual nodes — Adding a server moves a wedge of keys, not the entire keyspace; rebalance is throttled and explicit.
2. Cache-aside + invalidation — Application loads on miss; delete (or version bump) after database commit; TTL as safety net.
3. Database is source of truth — Cache is ephemeral acceleration; failover and cold start refill from the database.
4. Thundering herd — One expiry can mean thousands of database hits; jitter + singleflight are mandatory patterns.
5. Hot keys — One logical key can saturate one shard; mitigate with replicas, L2, or key splitting—not only “add nodes.”

Tell it in the room: “Client hashes key to primary via consistent hashing with virtual nodes. GET hits memory or miss → singleflight → database → SET with jittered TTL. On database write, commit then DELETE cache keys. Replica for read-heavy stale-OK data; primary when read-after-write matters. Viral key: singleflight on miss, TTL jitter, watch per-shard QPS. Node add: throttled wedge rebalance with handoff—not modulo reshuffle.”

Key Takeaways

• Clarify first — Ephemeral versus durable cache, consistency bar, read/write pattern, hot keys, out of scope.
• Consistent hashing — Wedge migration on node add; virtual nodes for balance; modulo hash reshuffles everything.
• Cache-aside — App loads on miss; invalidate by DELETE after database commit; ordering races are real.
• Database is truth — Cache protects the database; it does not replace it.
• Thundering herd — Jitter, singleflight, stale-while-revalidate; expiry is a load test you schedule.
• Hot keys — Per-shard metrics; replicas, L2, or split keys—not fleet averages alone.
• Operate miss path — DB QPS from cache miss, per-shard CPU, rebalance duration, replication lag.
• Green is not correct — Healthy cache CPU with melting database means miss storm or hot key.

Related Topics

• Rate Limiter System Design - Shared in-memory state, hot keys, and atomic updates
• Facebook Feed System Design - Hot post keys, cache stampede, and TTL jitter on read paths
• WhatsApp System Design - Caching versus source of truth on messaging metadata
• Caching Strategies - Fundamentals behind cache-aside, read-through, and write-through
• Consistent Hashing - Ring mechanics and virtual nodes in depth

Practice this path: Distributed Cache System Design.

Quick revision notes

Use this as a last look before a mock interview, not as a substitute for the architectural dig above.

Clarify first. Agree the cache is ephemeral, the database is source of truth, how stale reads can be, whether the workload is read-heavy cache-aside, that hot keys will appear, and what you leave out of scope (CDN, SQL query cache, multi-master). Skipping this often means you draw Redis with no invalidation story.

Draw routing before flags. Consistent hashing with virtual nodes; key → token → primary. Adding a node moves a wedge, not the whole keyspace. Rebalance is background, throttled, with handoff.

Walk one GET miss. Hash to shard, miss, singleflight one database load, SET with jittered TTL. Reject unbounded parallel refill on the same key.

Walk one database write. Commit, then DELETE cache keys (or version bump). TTL backs up missed invalidations—it does not replace delete on write for critical fields.

Name hot key and herd defenses. Per-shard QPS alerts; replica or L2 for one logical key; TTL jitter; singleflight. Global hit ratio can lie.

Failover honesty. Replica lag causes stale reads; primary for read-after-write; epoch on promotion; cache loss is refill from database, not magic durability.

Practice this path: Distributed Cache System Design.