System design interview guide
Logging System Design
Picture an online shop on a busy night. Every time a customer pays, the shop's software writes a short note about what just happened, and engineers call each of those notes a log line. At 2am something breaks, and the engineer on call searches those notes to find out why. Now imagine the place those notes are sent has slowed down, and the checkout page is waiting for each note to be saved before it answers the customer. Payments stall, not because payments are broken, but because of the notes about payments. A logging system is the part that collects the notes from every service, keeps them searchable, and has to do all of that without ever slowing down the services it is watching.

Problem statement
Design a centralized logging platform: the one place where every service in a company sends the short notes it writes about what it is doing, so engineers can search them later. It collects logs from many services, buffers spikes, parses and redacts data, stores recent logs for fast search, moves old logs to cheaper retention tiers, and protects applications when the logging pipeline is slow. The main interview work is deciding what is durable, what can be dropped, how queries stay bounded, and how cost is controlled.
Introduction
The problem: the checkout that stalls at 2am
Picture an online shop on a busy night. Every time a customer pays, the shop's software writes a short note about what just happened: when it happened, which service wrote it, how serious it was, and a sentence of detail. Engineers call each of these notes a log line. One payment might leave a dozen of them behind, and the shop may have hundreds of services all doing the same, so the notes pile up by the million every second.
At 2am a bad release goes out and customers start seeing errors. The engineer on call opens a search screen and types the name of the payment service and the last ten minutes, hoping to read what the service said as it failed. That one search is the whole reason the notes were kept. Now picture two ways the night can go wrong. In the first, the search is slow, or half the lines are missing, so the engineer is guessing. In the second, which is worse, the place the notes are sent has slowed down and the checkout is waiting for each note to be saved before it replies, so customers cannot pay because of the notes about payments.
The part of the system that collects every service's notes, keeps them in one place, and lets people search them is called a logging system. It looks like plain bookkeeping, and that is exactly why interviewers like it. The simple version takes a minute to describe, and the real version is full of decisions about what to keep, what to drop, and what to pay for. This page follows those decisions in the order you would make them in a room: first what we are solving and why, then what the system must do, then how big it is, and only then the design itself.
Why teams bother with a central logging system
Before there is a central system, each service writes its notes to files on the machine where it runs. That works while a company has three machines. It stops working when there are thousands, because nobody can log in to a thousand machines and read files during an outage, and machines that crash or are replaced take their files with them. Teams build one shared place for several reasons.
Finding the cause of an incident. When something breaks, the answer is usually written down in a log line somewhere. A shared search turns “which of our two thousand machines said something odd?” into one query with a service name and a time range.
Following one customer's request across services. A single checkout may touch five services. If each service writes the same request id on its lines, one search shows the whole journey in order.
Proof of who did what. Some lines are not for debugging at all. A record that an administrator changed a permission, or that a payment was refunded, may be needed months later for a security review or an audit. Those lines must not be lost quietly.
Alerts and dashboards. Teams count certain lines, such as “payment failed”, to draw a chart or wake someone up, so the same stream feeds more than human search.
Keeping cost under control. Logs are text, and text at this volume costs real money to store and search. A central system is the only place where a company can see which service writes the most and put a limit on it.
What problem are we solving?
Many services write log lines all day. At any moment, an engineer might ask one question:
“What happened in the last few minutes for this service, and can I trust that the important lines are still there?”
To answer it, the system has to take lines from thousands of services without making any of them wait, keep recent lines searchable within a few seconds, keep older lines for months at a lower price, hide secrets, and stay useful on the exact night everyone searches at once. It also has to tell the truth about itself: if some lines were dropped, the people searching must be able to find that out.
That is the whole job. We will not guess at the design yet. First we need to be clear about what the system must do and how well, so every design choice later has a reason behind it.
This page is part of the system design interview guides hub. If the idea of keeping and searching logs is completely new, it is fine to keep reading from here, because every term is explained where it first appears.
How to approach
We now know what a logging system is, why teams build a shared one, and what success looks like from the searcher's point of view. In an interview you do not jump from that story to drawing boxes. You start by turning a short, vague prompt into a clear contract: which lines are in scope, what people need from search, what must never be lost, and how big the volume is. Only then do you size the problem, make the design decisions one at a time, and draw the result. The list below is the order this page follows.
How to design a logging system in an interview
- Clarify the problem: which kinds of lines are in scope, what search people need, what happens when the system is overloaded, whether secrets must be hidden, and how big the volume is.
- Write the functional requirements (what the system does) and the non-functional requirements (how fast, how big, how safe).
- Estimate the size: events per second, bytes per day, how many days stay fast to search, and what a typical search looks like.
- Make the design decisions in order: how a service hands over a line, whether every line is treated the same, what absorbs a flood, when secrets are removed, how recent and old lines are stored, which fields are searchable, and how searches are limited.
- Draw the picture those decisions produce, then follow one log line through it.
- Go deeper where the interviewer pushes: what happens when a buffer fills, how one noisy service is contained, how a parser bug is repaired, and one production failure.
Clarifying questions (say these in the room)
In a real interview the prompt is only a few words: “Design a logging system.” Before you design anything, ask questions until the problem is clear. Each answer below changes the design, which is why it is worth asking.
You: “What kinds of logs are in scope: debug messages from applications, audit records of who did what, security events, and web access logs, or only some of these?”
Interviewer: “Include all of them, but make the policy for each kind clear.”
That answer means you should not use one rule for everything. A debug line that says “cache miss for key 7” is cheap and can be sampled or dropped when things are tight. An audit line that says “admin changed the refund limit” may be needed for a compliance review, so it needs a stronger promise to be kept. We will call the kind of a line its class, and most of the later decisions depend on it.
You: “What do people need from search: only recent lines, or also old ones, and do they need live viewing during an incident?”
Interviewer: “Fast search for recent logs, slower access for old logs, and live tail during incidents.”
Live tail means watching new lines appear as they arrive, like watching a log file grow. This answer tells you the system has a fast tier for recent data and a cheaper, slower tier for old data, and that you should not promise instant search over years of logs.
You: “If the logging system is slow or full, is it acceptable for application requests to slow down too?”
Interviewer: “Applications should keep running, but audit loss must be visible.”
That fixes the most important rule of the whole system. The services being watched come first, and logging must never be the reason a customer cannot pay. It also means that when lines are dropped, the drop has to be counted and reported, especially for audit lines.
You: “Do we need to hide secrets such as passwords and card numbers, and must one team be unable to read another team's logs?”
Interviewer: “Yes. Users from one team or tenant should not see another team's logs, and secrets should not be broadly searchable.”
A tenant here means one team or customer whose data must be kept apart from the others. Because a secret that is already searchable is very hard to take back out, this answer puts secret removal before search, not after it.
You: “What scale should we design for?”
Interviewer: “Hundreds of terabytes per day, with spikes during incidents.”
A terabyte is a million megabytes, so this is a very large amount of text. Two things follow right away: one machine cannot hold it, and cost is part of the design, not an afterthought. Spikes during incidents matter because that is exactly when engineers search the hardest.
You: “I want to leave out a full metrics and tracing platform, a security analytics product, and dashboard design so we can focus on collecting, storing, and searching log lines. Does that match what you want?”
Interviewer: “Yes, focus on collecting, storing, and searching.”
If the interviewer later asks about a particular queue product or search product, you do not need to answer yet. Park those until the design decisions, where each one has a natural place.
Functional requirements
Functional requirements describe what the system does. From the answers above, a logging system must:
- Accept lines from every service without making the service wait. A service hands over its lines and carries on with its real work, even if the logging system is slow.
- Give every line a class and handle each class by its own rules. Debug lines may be sampled or dropped under pressure, while audit and security lines get a stronger promise to be kept, or a loud alert when that promise is at risk.
- Pull out fields and remove secrets. The system turns each line into named fields such as service, time, level and message, and strips passwords, tokens and card numbers before the line becomes searchable.
- Let people search recent lines with filters. Search by service, level, time range and a safe subset of text matching, with results in a few seconds.
- Support live viewing during an incident. An engineer can watch new lines from a service as they arrive.
- Keep old lines under a retention rule. Each tenant and class has a number of days in fast storage and a number of months or years in cheap storage, after which lines are deleted or placed on legal hold.
- Feed alerts and dashboards. Selected lines can be counted to drive charts and notifications.
- Keep teams apart. A person sees only the services and tenants they are allowed to see.
Non-functional requirements
Non-functional requirements describe how well the system must do its job. They are the numbers and promises that shape the design.
- Protect the applications. Writing a line must never block a customer request for long, and logging must not fill the disk of the machine it runs on.
- Search speed. A search that names a time range returns in a few seconds for recent data, so on-call can use it in the middle of an incident.
- Scale. Thousands of services, millions of events a second and hundreds of terabytes a day, with sudden spikes when something breaks.
- Honest durability. Each class has a clear promise. Delay is acceptable, but silent loss of audit lines is not, and every dropped line is counted.
- Isolation. One noisy service or one expensive search must not slow down everyone else.
- Safety. Secrets do not appear in search results, and access is limited by team.
- Affordable. Cost per terabyte kept, per field indexed and per search run is visible and capped, because at this volume cost is a real limit.
- Visibility. The platform tells its own operators when it is behind, dropping lines or failing to parse, because it is often the tool people rely on during someone else's outage.
Out of scope (and why you park these)
Write a short out-of-scope list on the board so the interviewer sees you can bound the problem.
A full metrics and tracing platform. Logs should connect to metrics and traces through a shared id such as trace_id, but designing the storage for numbers over time and for request traces across services would be a separate interview.
Every feature of a search language. A safe subset of keyword and field filters is enough. Arbitrary patterns over petabytes, joins between datasets and unbounded scans should be parked or moved to a slower, asynchronous export.
A complete security analytics product. Security tools can read audit and security lines from this system, but correlation rules and threat detection are extra scope unless the interviewer asks.
Dashboard design. Dashboards matter, but this interview is about collecting, storing, searching, protecting and operating the lines.
Perfect no-loss for every line. You can make audit lines very durable, but promising zero loss for all debug lines under every failure is expensive and can hurt the applications you are protecting.
That is the requirements stage. If you want to practice it before reading on, try the requirements stage on Practice System Design (about 5–10 minutes), then come back for the sizing and the decisions.
Minute pacing (about 40 minutes)
| Minutes | Focus |
|---|---|
| 0–5 | Clarify the problem; write functional and non-functional requirements; agree out of scope |
| 5–10 | Capacity: events per second, bytes per day, days kept fast, shape of a search |
| 10–20 | Design decisions, then the high-level boxes with one log line walked through |
| 20–32 | The deep dive: full buffers, noisy services, parser failures, secrets, bounded search |
| 32–40 | Scaling, failure, cost, one production story and questions |
In the room (how you open): “I’d start by agreeing what we are collecting and what people need from it: which kinds of lines are in scope, how quickly recent search must answer, what must never be lost, whether secrets must be hidden, and how big the volume is. I’ll write the requirements on the board, size the problem, and then make the design decisions one at a time, so every box I draw has a reason.”
If you remember one thing: Ask first, write the requirements down, and only then design.
Capacity estimation
Now that the requirements are clear, the next step is to find out how big the problem is, because the size decides what kind of buffer and storage you can afford. Logging is mostly a question of bytes: how many arrive each second, how long they must stay fast to search, and how often a person searches.
All numbers below are rough estimates to anchor the whiteboard. Tune them in the room.
| Axis | Interview-scale estimate | What it means for design |
|---|---|---|
| Ingest rate | Millions of events per second | Agents, collectors and the buffer must batch and split work across many machines |
| Daily volume | Hundreds of TB per day (for example 300 TB, about 3.5 GB every second on average) | Retention and compression decide monthly cost |
| Hot retention | 3 to 30 days | The fast search tier must be sized for recent incident debugging only |
| Cold retention | Months or years | Object storage and compressed column files are cheaper but slower |
| Query latency | Seconds for a recent, bounded search | Every search needs a time range and limits |
| Cardinality (how many different values a field can take) | Thousands of services, and many fields | Schema control prevents index explosions |
| Burst behavior | An incident or a bad release can multiply volume | Quotas and backpressure protect the platform and the applications |
What the numbers imply
- Not everything can stay fast. At 300 TB a day, seven days of raw text is already about 2 PB before compression and before the extra space a search index adds. That is why recent days live in the expensive fast tier and everything older moves to cheap storage.
- The buffer is measured in hours, not days. Holding a full day of traffic in a queue would mean about 300 TB, and with each message copied to three machines for safety, close to a petabyte of broker disk. A buffer for a few hours is enough to ride out a slow consumer and far cheaper.
- The number of queue partitions is part of the answer. If a partition (one ordered lane of the queue) comfortably carries something like 10 to 50 MB a second, then 3.5 GB a second needs on the order of 70 to 350 lanes at the very least. Real systems use many more, to leave headroom and to give big tenants their own lanes. Say this is an estimate you would confirm with a load test.
- Compression is a big lever. Logs are repetitive text, so they usually shrink a lot when compressed, but the exact ratio depends on the content. Say you would measure it on real data instead of quoting a number.
- A few services produce most of the bytes. One service that starts logging whole request bodies can double the total. That is why limits per service are a requirement, not a nicety.
- Spikes are the design case. The night everyone searches is also the night every service logs the most errors, so size for the spike, not for the average.
Checkpoint: Which part of the volume do you keep fast, and why not all of it?
If you remember one thing: Volume and retention decide the cost, so only recent data earns the expensive tier.
Design decisions
We now know what the system must do, how well it must do it, and roughly how big it is. Now we choose. Each decision below answers one question the requirements raised, and each one rules out an easier idea that would not survive the requirements. Read them in order, because every decision builds on the one before it.
Decision 1: How does a service hand over a line without waiting?
The first requirement says writing a line must never hold up a customer request. The simplest idea is for each service to send every line straight to the search system over the network as it writes it. It needs no extra pieces and it is easy to picture.
It fails the moment the search system is slow. Each write now waits for an answer, the service's threads pile up behind those waits, and a retry after a timeout sends the same line again, so the load grows exactly when things are already bad. The checkout stalls because of logging, which is the failure we named in the introduction.
There are a few ways out.
- Write to a local place and let something else ship it. The service writes the line to its own standard output or a local socket, which is nearly instant, and a separate program on the same machine picks it up and sends it onwards.
- Use a library inside the service that queues lines in memory. It is simple, but every language needs its own copy, and when the service crashes the lines still in memory are lost.
- Make the service retry until the logging system answers. This protects the lines but not the customer, so it brings back the original problem.
We choose the first one. A small program called an agent runs on every machine (or beside every service). It reads the lines, groups them into batches, compresses them, and sends them on. If the far end is slow, the agent keeps the batches in a bounded spool on local disk, and when that spool is full it follows a policy about what to drop. The service never waits on the network, and the cost of a slow pipe lands on a buffer that has a size limit and a rule. The rule itself is the next decision.
Decision 2: Is every line treated the same?
When a buffer fills, something has to give, so we must decide in advance what. The simplest idea is one rule for everything, such as “drop the newest lines” or “never drop anything”.
Neither survives the requirements. “Drop the newest” would throw away an audit record of a refund just as quickly as a debug message, which breaks the requirement that audit loss must be visible. “Never drop anything” means a flood of debug lines fills the machine's disk or blocks the service, which breaks the requirement to protect the applications.
So every line carries a class, and each class has its own promise.
| Class | Under pressure | If we cannot keep it |
|---|---|---|
| Debug and noisy | May be sampled, shortened or dropped first | Prefer keeping the application healthy |
| Application errors and warnings | Kept longer than debug, dropped only after debug | Count the drops |
| Access lines | Sampled if needed | Count the drops |
| Audit and security | Stronger path, stored separately from debug | Raise an alert; never lose silently |
This is a product and compliance decision, not just an engineering one, so it is worth saying out loud before you name any product. The key idea is that dropping a line is allowed only when it is counted, so the people searching can see how much is missing.
Decision 3: What absorbs a flood between the agents and the search system?
Even with agents, many thousands of them now send lines to one place that must index them. The simplest idea is to point every agent straight at the search cluster. Searching and indexing are expensive work, so when the cluster slows down during a spike, every agent's spool fills at once, and the lines are dropped at the worst moment.
The answer is a durable buffer in the middle: a queue, usually Kafka or Kinesis, that accepts lines quickly, writes them to several machines for safety, and lets the processors behind it read at their own speed. If the index is slow for twenty minutes, the queue holds the backlog and the processors catch up later. It also lets us replay: if we find a bug in how lines were parsed, we can read the same lines again from the queue.
Two details come with this choice. First, the queue is not infinite. It has disks and a retention window, so a long slowdown still ends in trouble, which the deep dive covers. Second, how we split the queue into lanes (partitions) decides who gets hurt in a storm. If all lines share one lane, one noisy service raises the delay for everyone. So we split by tenant, service and class, give very large producers their own lanes, and keep audit lines in their own lane so that turning on debug logging cannot starve security evidence. On top of that, each service gets an hourly or per-second quota at the agent or collector, so one release cannot flood the whole platform.
Decision 4: When do we remove secrets?
Lines carry things people forgot to protect: an authorization header in an error message, a card number in a request body. The simplest idea is to index everything and clean up later if someone reports a problem.
It fails the requirement that secrets never appear in search, because by the time anyone notices, the line has been copied to the search index, its replicas, snapshots and perhaps a dashboard. Deleting a secret from all of those is slow and rarely complete.
So we remove secrets before the line becomes searchable, in layers. The agent blocks fields it knows are secret, such as authorization headers and cookies. Then the stream processors, which read from the queue, run the full redaction (removing or replacing the value with a harmless token) before anything is written to the search index. If raw lines must be kept for legal reasons, they go to a separate, tightly controlled store. The cost is a little extra work on every line, and the benefit is that the search tier never sees the secret.
Decision 5: How do we keep recent lines fast and old lines cheap?
The requirements want search in seconds for recent lines and months or years of retention, and the capacity numbers showed we cannot afford both on the same disks. The simplest idea is one big search cluster that keeps everything. At hundreds of terabytes a day it would be enormously expensive, and its size would also make searches slower.
We use tiers. The hot tier is a search engine (OpenSearch, Elasticsearch, ClickHouse or similar) holding the last few days in indexes split by time, so a search for the last ten minutes touches only a small, recent slice. An optional warm tier keeps somewhat older data on cheaper disks. The cold tier stores compressed files in object storage, split into folders by tenant, service and date, so a slow scan can skip everything it does not need. A lifecycle job moves data down the tiers on a schedule set by the retention rule for each tenant and class.
The product has to be honest about the difference. A search over the last hour answers in seconds. A search over last quarter becomes a background job that returns a link, because pretending both cost the same would either be too expensive or too slow.
Decision 6: Which fields can be searched?
Search engines are fast because they build extra data structures for each field they index. The simplest idea is to index every field of every line so that people can search for anything.
That breaks the cost requirement. Some fields have a nearly unique value on every line, such as a request id or user id. Indexing them creates a structure as big as the data itself, memory use grows without bound, and merging the index slows everything down. This is called a cardinality problem, meaning too many different values.
So we control the schema. A small set of core fields is always indexed (time, service, level, tenant, class). Teams can add others only through an approved list. Everything else is stored with the line so it can be read back, but not indexed. If a team needs to count by a high-cardinality value, the better tool is a metric, not a log index. Lines with an unexpected shape go to a quarantine area instead of changing the live index layout.
Decision 7: How do we stop one search from hurting everyone?
A search box is a powerful tool and, left unlimited, a dangerous one. The simplest idea is to let anyone search any text over any time. During an incident one engineer asking for thirty days across every service with a wildcard can use more capacity than the original outage did.
So every search goes through a query service that enforces limits before touching the data. It requires a time range, checks that the person is allowed to see the tenant and service, estimates how much data the search would read, caps the number of rows and bytes returned, applies a timeout, and sends large or old searches to a background job instead of holding a web request open. Rejecting an over-broad search with a helpful message is not unfriendly. It is what keeps search alive for everyone else during an incident.
Putting the decisions together
Taken together, the seven decisions are the design. A service writes a line locally and never waits. An agent batches it, spools it within a limit and applies the rule for its class. A durable queue absorbs floods and splits lines by tenant, service and class. Stream processors parse the line and remove secrets before it is stored. Recent lines go to a hot index with a controlled set of fields, older lines move to cheap storage, and a query service puts bounds on every search.
In the room (a fuller opening you can actually say): “I’d start by agreeing which kinds of lines are in scope and giving each class its own promise: debug can be sampled or dropped under pressure, audit lines must be kept or raise an alert. A small agent on each machine takes lines off the service so it never waits on the network, batches them and spools to a limited local disk. Behind the agents I’d put a durable queue, split by tenant, service and class, so a slow search system or a noisy service cannot stall everyone. Stream processors read from the queue, pull out fields and remove secrets before anything becomes searchable. Recent days go to a hot, time-split index with only approved fields indexed; older lines move to cheap object storage and are searched through background jobs. Every search needs a time range and size limits. Finally, I’d count every dropped line by class, so loss is always visible.”
The next section draws these decisions as one picture and shows who owns what.
High-level architecture
The decisions above were made one at a time. This section puts them on one page as a picture: which boxes exist, who owns what, and how one line travels through them. The deeper mechanics come later, so keep this at the level of a map.
The architecture has five layers: collection, a durable buffer, processing, storage, and query.
[ App containers / VMs ]
|
v
[ Agent: tail, batch, local disk spool, quotas ]
|
v
[ Regional collector ]
|
v
[ Kafka / Kinesis: durable buffer, partitioned by tenant/service/class ]
|
+--> [ Stream processors: parse, enrich, redact PII, route ]
|
+--> [ Hot search index: recent days, interactive query ]
|
+--> [ Object storage: raw/Parquet, long retention ]
|
+--> [ Metrics / alert rules from selected logs ]
[ User ] --> [ Query API + RBAC + cost planner ] --> hot search or async cold job
Agents run close to the applications they watch. They read each service's printed output, log files, or a local socket, group lines into batches, shrink those batches, and write them to a limited local disk folder when the next hop is slow. Regional collectors receive those batches and write them to the durable queue. Stream processors then read from the queue, pull out fields, add useful context, remove sensitive values, choose the right storage path by team and class, and write many lines at once into storage.
Hot storage is built for recent search that people run while they wait. It might be OpenSearch, Elasticsearch, ClickHouse, or another product, but the design should talk about behavior rather than the brand name. Cold storage is built for cheap long keep and slower scans, often as files in object storage that are compressed into column-shaped layouts so a scan can skip what it does not need.
The query API is the door for people searching. It checks who is signed in, which teams they may see, requires a time range, caps rows and bytes, applies timeouts, and sends recent bounded searches to the hot tier while large or old searches become background export jobs or a slower analytics path.
Who owns what:
- Agent owns reading lines near the application, grouping and shrinking them, holding a limited local spool, first-pass sampling, and slowing down when the next hop cannot keep up.
- Collector owns regional intake, checking that agents are allowed to send, and writing batches into the durable queue.
- Queue owns holding lines safely for a while, letting you replay them after a fix, and separating writers from the slower indexing work.
- Stream processors own pulling out fields, adding context, removing secrets, choosing the storage path, and counting selected lines for alerts.
- Indexers own writing many lines at once into recent search storage, applying the approved field list, choosing how often new lines become searchable, and moving older data down.
- Query API owns who may search what, estimating cost, enforcing time and size limits, paging results, and starting background exports for old data.
In the room: Say where a line counts as safely kept for each class. For debug it may be the local spool or the queue acknowledgement; for audit it should be a durable queue with copies on several machines, plus a clear alert if that queue is unavailable.
Core design approaches
The map above shows one way to build each box. This section lays out the alternatives for the three choices interviewers probe most, so you can say why you picked one and when you would pick another.
Agent-based collection
Agents give you control near the source. They can batch lines, add host and Kubernetes metadata, apply first-pass size limits, spool to disk, and decide what happens under pressure. Without an agent, applications tend to own logging delivery themselves, which spreads retry and backpressure bugs across every service.
| Approach | Strength | Weakness | Pick it when |
|---|---|---|---|
| Agent on each node or beside each service | One place for batching, spooling and quotas; works for any language | One more program to run and upgrade | Many services and many languages (the default) |
| Logging library inside the service | No extra program | Each language needs its own copy; lines in memory are lost if the service crashes | A small number of services in one language |
| Service sends straight to the search system | Fewest parts | Slow search means a slow service | Almost never at this scale |
The agent should not block application threads forever. If an app writes to stdout, the runtime and node agent must have bounded buffers. If a service writes through a logging library, the library should have a small async queue and a policy for overflow.
Durable queue before indexing
A durable queue such as Kafka or Kinesis decouples producers from processors and search indexes. If the hot index slows down, agents and collectors can keep writing for a while, and consumers can catch up later. The queue also enables replay for reindexing after parser bugs or schema changes.
The queue is not infinite. Partition skew, broker disk, retention, and consumer lag still matter. Partition by tenant, service, class, or a composite key so one noisy producer does not block unrelated traffic.
Hot, warm, and cold storage
Hot storage serves recent interactive search. It uses time-based indexes, replicas, and refresh tuning. Warm storage may keep older searchable data with slower disks or fewer replicas. Cold storage keeps compressed files in object storage and supports slower analytics or export jobs.
| Tier | Holds | Typical question it answers | Latency to expect |
|---|---|---|---|
| Hot | Last days, fully indexed | “What did checkout say ten minutes ago?” | Seconds |
| Warm | Older searchable data, fewer replicas | “What happened last week?” | Seconds to tens of seconds |
| Cold | Compressed files in object storage | “Show every refund by this admin last quarter.” | Minutes, as a background job |
The user experience should match the tier. Recent incident queries can be fast. Month-old compliance searches may be asynchronous and slower.
Schema control and cardinality budgets
Structured logs are valuable when fields are predictable. They are dangerous when every team adds unique fields that become indexed dimensions. The platform should require core fields and limit what can become an indexed field.
High-cardinality fields such as request id, trace id, or user id may be stored for retrieval but not indexed for broad aggregation unless there is a clear product reason. Otherwise, mapping size and memory grow without bound.
Data model
The approaches above decided which stores exist, so the next question is what exactly each one holds. The same log event is stored in a different shape at each stage of the pipeline, and keeping those shapes straight is half of explaining the design.
| Entity or store | Important fields | Why it exists |
|---|---|---|
log_event | event_id, timestamp, tenant_id, service, level, message, trace_id, host, class, schema_version | Canonical event shape after collection |
agent_buffer | local batch id, offset, class, size, retry count | Lets the agent survive short upstream failures |
queue_topic | topic by tenant/service/class, partition, offset | Durable buffer and replay source |
parsed_event | normalized fields, redaction status, parser errors | Safe shape for indexing and routing |
hot_index | time bucket, selected indexed fields, stored message, routing fields | Recent interactive search |
cold_object | date/hour path, tenant, compressed Parquet or JSON, checksum | Long retention and cheaper scan |
retention_policy | tenant, class, hot days, cold days, delete or legal-hold rule | Controls lifecycle and cost |
query_job | user, tenant, query, time range, tier, byte estimate, status | Async cold searches and exports |
The event_id matters more than it looks. Delivery in this system is at least once, which means a line may arrive twice after a retry, and a unique id lets the index or the reader ignore the copy. Every log should also carry a class such as debug, application, audit, security, or access. That class decides sampling, retention, redaction strictness, and drop behavior.
Detailed design
We now know what is stored at each stage, so we can put the pieces in motion. This section follows one log line from the moment an application writes it until an engineer reads it on a search screen. The important design choices are not only the boxes. They are the points where the system buffers, redacts, indexes, drops, or slows down. After the story, it shows the calls people and agents make into the system. Read it once as a story. The follow-up questions come in the deep dive right after it.
Emitting and collecting
An application writes one tidy line of named fields (often as JSON text) to its printed output or to a local socket the agent listens on. The line includes a timestamp, service name, seriousness level, message, a shared request id if there is one, a team id when needed, and a class such as debug or audit. The application does not spend unbounded time waiting for the logging system. If it uses a logging library, the library puts the line on a small in-memory queue and returns quickly, with a written rule for what happens when that queue is full.
The agent on the same machine, or a small companion container next to the service, follows that stream as it grows, groups lines into batches, shrinks the batches, and writes them to a local disk spool when the regional collector is slow. The spool has a size limit because a full machine disk can hurt the application itself. When the limit is reached, the agent follows policy: drop or sample debug first, slow down noisy sources if possible, and alert loudly if audit or security lines are at risk.
Ingesting and processing
Regional collectors receive batches from agents and check that the agent is allowed to send. They write to the durable queue (Kafka or Kinesis) with enough copies for the log class. A queue acknowledgement, meaning the queue has accepted the write safely, is the durability point for important logs. Reader groups then pull from the queue, pull out named fields, add environment context, remove personal or secret data (often called PII redaction), and send each event to the right storage path.
Parsing should be strict enough to protect the search layout. Oversized messages, unexpected fields with nearly unique values, or broken JSON can go to a quarantine lane or a raw store instead of changing the live hot-search layout. Secret removal happens before the hot index because searchable secrets are hard to erase later.
Indexing and retention
Indexers bulk-write parsed events into time-based hot indexes. Bulk writes are more efficient than one write per log line. Index templates control which fields are indexed, which are stored only, and how many shards each time bucket gets. Refresh intervals balance freshness and write cost.
Lifecycle jobs move older data out of the hot tier. Some data may move to warm searchable storage. Older or high-volume data lands in object storage as compressed files. Retention policies delete or hold data by tenant and class. Audit logs may have longer retention and stricter immutability than debug logs.
Querying
A user searches through the query API. The API authenticates the user, checks tenant and service permissions, requires a time range, estimates query cost, and routes the request. Recent bounded searches go to hot indexes. Large or old searches become async jobs over cold storage with progress and download links.
The query engine enforces timeouts, row caps, byte caps, and pagination. If a query is too broad, the API rejects it with guidance rather than letting it scan the world.
- The application writes a structured line and returns to its real work.
- The agent batches, compresses and spools it, applying the class rule if the spool is full.
- A collector writes the batch to the durable queue, and the acknowledgement is the durability point.
- A processor parses the line, removes secrets and routes it by tenant and class.
- An indexer bulk-writes it to the hot index, and a lifecycle job later moves it to cheap storage.
- A person searches with a time range, the query API checks limits and permissions, and the result comes from the hot tier or a background job.
In the room (recap): “Applications write tidy log lines to a local agent. The agent groups them, holds them on a limited disk when needed, and follows the class rule if that disk fills. Collectors write batches into a durable queue. Stream processors pull out fields, remove secrets, and send each line to the right store. Recent days sit in a hot index for fast search; older days sit in object storage. The query API requires a time range and size limits so one search cannot take the shared platform down.”
API design: ingest, search, and background jobs
Most callers interact with the logging system in two ways: agents sending lines in, and people searching lines out. This part covers the call agents make inside the system, what the people-facing search returns, and the status codes and errors callers must handle.
The internal ingest call. Agents and collectors send lines in batches, not one at a time, because one network call per line would be far too slow at this volume.
| Surface | Role |
|---|---|
POST /v1/ingest/logs | Internal bulk ingest from agents or collectors |
GET /v1/logs | Search recent logs with time bounds and filters |
POST /v1/logs/query-jobs | Start an async cold or large query |
GET /v1/logs/query-jobs/{job_id} | Poll async query status and result location |
GET /v1/logs/tail | Open a bounded live stream of new lines for one service (server-sent events or a websocket) |
POST /v1/ingest/logs fields:
| Field | Role |
|---|---|
tenant_id | Tenant or org boundary |
service | Producer service |
class | Debug, app, audit, security, access |
events[] | Batched log events, each with an event_id so a retry does not create a duplicate |
agent_id | Source identity for auth and troubleshooting |
What people send and receive when they search. The search call must carry a time range and returns a capped page of lines with a cursor for the next page.
GET /v1/logs query params:
| Param | Role |
|---|---|
from, to | Required time range |
service | Filter by service |
level | Filter by severity |
q | Search expression using a safe subset |
limit | Server-capped page size |
cursor | Opaque pagination token |
User query with time range
|
v
[ Query API ] -> authz + cost estimate
|
+--> recent and bounded --> [ Hot index ]
|
+--> old or huge ---------> [ Async cold job ] -> object storage scan
Status codes and errors. A 400 means the time range is missing or the query is not supported, so the caller can fix it. A 401 means the caller is not signed in, and a 403 means they are signed in but not allowed to see that tenant. A 429 means an ingest or query quota has been exceeded, and the caller should wait and retry later. A 413 means an ingest batch was too large, and the agent should split it. For a big old search, the API replies that the job was accepted and returns a job id to poll, instead of holding an HTTP request open for minutes.
Deep dive: the questions an interviewer asks next
You have drawn the design and followed one line through it. This is the point where a good interviewer starts digging, and the questions usually come in a fairly predictable order, because each answer raises the next one. Read this section as that conversation. Each question is followed by an answer you can give, plus a hint of where the interviewer is likely to go next, so you can lead into that follow-up instead of waiting to be asked.
“You said the queue absorbs spikes. What happens when the pipeline is slow for a long time?”
This is the hardest part of logging, and the wrong answer is “Kafka absorbs it.” The queue absorbs it until partitions fall behind, broker disks fill, agents run out of spool, and applications start blocking or losing the exact lines needed for the incident.

Start with the tempting but dangerous design. A weak design lets every application send logs directly to the search cluster. When the index slows down, app threads block, retries multiply, and the production incident gets worse. Another weak design drops lines silently when buffers fill. That keeps the app alive but destroys trust in the logging platform.
The better design is a chain of buffers, each with a size limit and a rule. The agent absorbs short collector failures. The queue absorbs processor and index slowdowns. The delay between the newest line written and the newest line read (called consumer lag) tells you how far behind the pipeline is. When one buffer fills, the pressure moves back one step toward the source, and at each step the class decides what gives: debug is sampled, shortened or dropped first, application errors are kept longer, and audit lines take a stronger path and alert instead of vanishing. That policy is a product and compliance decision, not an implementation detail.
Partitioning decides who gets hurt during a storm. If all lines share one topic and one partition key, one noisy service can raise lag for everyone. Partition by tenant, service, and log class where useful. Give high-volume tenants quotas or dedicated topics. Keep audit separate from debug so “turning on DEBUG” does not starve security evidence.
Where it goes next: the interviewer will probably ask what the agent does, concretely, when the collector cannot be reached.
“What exactly does the agent do when the collector is unreachable?”
It keeps the batches in a spool on local disk and retries with a growing wait between attempts, so a recovering collector is not hit by every agent at once. The spool has a fixed size, because a full disk on the application's machine would hurt the application, which is the thing we are protecting. As an illustration, if a node produces 5 MB of compressed lines a second and the spool is 2 GB, it holds about seven minutes. Choose that number from real traffic, and say it is a trade between protecting lines and protecting the disk.
When the spool reaches its limit, the agent applies the class rules in a fixed order. It first stops reading low-value debug lines and samples the rest, then drops older debug batches before newer ones, and it keeps audit and security batches for as long as possible. Whatever it drops, it counts, by class, and sends that count on in a small separate message, because a count that is sent ahead is worth more than lines that cannot be.
Where it goes next: “Sampling sounds risky. How do you choose what to drop?”
“How do you decide which lines to drop without losing the important ones?”
Drop by class first, then by value inside a class. Inside debug, a common approach is to keep a fixed share of lines at random, say one in ten, but always keep lines that carry an error or that belong to a request already marked as failing. Keeping every line for a failing request while sampling the rest means the interesting requests stay complete.
Two guardrails make this safe. The dropped-line counter is itself never sampled, so you can always tell afterwards how much was lost. And the sampling rate is a dial an operator can change at runtime, so you can turn debug up for one service during an investigation and back down afterwards without redeploying anything.
Where it goes next: “Where, precisely, is a line durable? And do you promise exactly once?”
“Where is a line durable, and do you promise exactly-once delivery?”
For ordinary debug lines, a line counts as safe once the agent has spooled it to disk or the queue has acknowledged it, depending on the policy you chose. For audit and security lines, set a stronger point: the line is durable only after the queue has acknowledged it with copies on several machines. Until then the agent keeps it and keeps retrying.
Exactly once is more than this system needs. Retries mean a line can arrive twice, so we accept at least once delivery and give every line a unique event_id, so the index or the reader can ignore a copy. Losing an audit line is far worse than showing one twice.
Where it goes next: “One service just started logging ten times as much. What stops it from hurting everyone?”
“One service starts logging ten times as much. How do you keep it from hurting everyone else?”
Four layers, from the source inward. The agent enforces a bytes-per-second quota for each service and class, so the flood is cut at the machine. The collector enforces the same limit per tenant, in case many agents add up. Inside the queue, very large producers get their own topics or lanes, so a hot lane delays only that producer. And a dashboard shows bytes by service, so the team that caused the jump can be asked to turn the logging down.
One trap is worth naming. A queue can look healthy overall while one lane is overloaded, because the average hides the hot lane. This is why you watch lag per lane and class, not one global number.
Where it goes next: “Your parser has a bug and half the lines are coming out wrong. What now?”
“The parser has a bug and half the lines come out wrong. How do you recover?”
Because the queue keeps the original lines for a few hours, the fix is a replay. Lines that fail to parse, are too big, or carry unexpected fields go to a quarantine topic instead of the live index, so the search layout is protected. Once the parser is fixed, you start a new consumer at the point just before the bug and let it re-read from the queue, writing the corrected lines to the index. The event id makes the replay safe, because a line that was already indexed correctly is recognized as a repeat.
The limit is the queue's retention window. If the bug is found after the window has passed, only the raw copy in object storage is left, which is slower to reprocess. This is one reason to keep a raw copy of audit lines.
Where it goes next: “What if a secret gets through the redaction?”
“What if a secret gets through redaction and becomes searchable?”
That is the incident this design is built to make rare, so treat it like a security incident rather than a data cleanup. Block the stream that produced it, restrict access to the affected index, and rotate the secret, because a secret that was ever searchable should be considered known. Then repair the cause, which is usually a field the redaction rules did not know about, and add a test for it.
Cleaning the index is the slow part. Deleting from replicas, snapshots, exports and dashboards is slow and rarely complete, which is exactly why Decision 4 put redaction before indexing, not after. Redaction failures should page the platform team.
Where it goes next: “How do you stop one search from scanning the whole archive?”
“How do you stop one search from scanning the whole archive?”
At the query service, before any data is touched. It requires a time range, estimates how many bytes the search would read from the number of time-based indexes involved, and rejects or redirects it if the estimate is over the person's budget. A search over a few hours goes to the hot index. A search over months becomes an asynchronous job against cold storage, with a byte limit, a progress view, and approval for very large exports. Leading wildcards and heavy pattern matching are restricted, since they cannot use the index.
Say it in the interview: “Logging should protect the application first, but it must make loss visible and class-specific.” That sentence shows you understand why a logging outage can both hurt production and hide the evidence needed to fix production.
Hot partitions and cardinality are covered in Key challenges. Agent disk, lag by class and query cost belong in Observability and Bottlenecks and tradeoffs.
If you remember one thing: Every buffer needs a size limit and a rule for what gives first, and every dropped line must be counted.
Key challenges
The deep dive showed that the core idea is simple once each buffer has a rule. What makes a real logging platform hard is everything around those buffers, and five hard parts show up again and again. Name them early in the interview, because each one is a place the interviewer is likely to push.
Pressure flowing backward can break the app. If writing a log line blocks the threads that serve customers, or fills the machine's disk, the system built to explain an outage becomes part of the outage. Agents need size-limited buffers and a clear rule for what happens when those buffers fill.
Busy lanes create fake spare capacity. The whole cluster can look underused while one team, service, or queue lane is overloaded, because averages hide the hot lane. Splitting work into lanes and putting write limits on each producer is part of reliability, not a later tuning step.
Too many different field values can destroy the index. Treating request_id, user_id, or arbitrary labels as searchable fields can grow the search layout and memory until no amount of extra disk alone fixes it. That is why only approved fields are indexed.
Secrets and personal data spread quickly. Once passwords, tokens, or personal fields enter the hot index, its copies, snapshots, and exports, cleanup becomes slow and incomplete. Remove them before indexing.
Open-ended search can hurt everyone. A broad pattern search over a month of logs can cost more machine time than the original incident. Required time ranges and byte estimates are safety controls, not unfriendly friction.
If you remember one thing: Buffer pressure and field-value explosion are design requirements you state up front, not surprises you discover after launch.
Scaling the system
Those hard parts get worse as volume grows, so the next question is what happens when the number of services and the bytes per day multiply. You do not have to wait for the interviewer to ask. Right after you finish the design, say the scaling story yourself, then be ready for the follow-ups.
What you can say upfront: “Each layer scales on its own. Agents scale for free, because there is one per machine. Collectors and processors scale by adding more of them behind the queue, and the queue scales by adding partitions. Storage scales by tier: only recent days sit on the expensive hot index, which is split by time, and everything older moves to cheap object storage in folders by tenant, service and date. Search scales by routing, with recent bounded searches going to hot indexes and big scans becoming background jobs. The thing that stops it all from collapsing is quotas, on how much each service may write and how much each person may search, plus a limit on which fields may be indexed.”
That one paragraph answers most of what follows. Here are the follow-ups, in the order they usually come.
Interviewer: “Volume doubles next year. What do you change first?”
You: “The stateless parts first, because they are the easy half: more collectors and more processors consuming from the queue. Then I check the queue's partition count and broker disk, because adding partitions is easy at first and hard once a lane's order matters. Then I look at the hot index's shard count and disk. I would also look at the biggest producers first, because asking one team to stop logging full request bodies often saves more than another hundred machines.”
Interviewer: “How do you scale the hot search tier?”
You: “I split indexes by time, so each hour or day is its own set of shards. A search for the last ten minutes only touches the newest ones, and old indexes can be moved or deleted whole instead of line by line. I keep batching large enough to reduce overhead and the refresh interval long enough that writes stay cheap, and I accept that new lines may take a few seconds to show up in search. I would size from the capacity numbers and confirm with a load test.”
Interviewer: “How do you keep one team's spike from hurting another team's search?”
You: “Quotas at every entry. Ingest quotas protect the queue and the index, query quotas protect the search cluster, and retention rules protect cost. A very large team may need its own topics, its own indexes or even its own cluster, which costs more but limits the blast radius.”
Interviewer: “Dashboards and alerts run searches every minute. Doesn't that add up?”
You: “Yes, and it is a common hidden load. For anything that runs on a schedule, I would compute a number once as lines arrive, as a counter, and chart the counter, instead of re-running a large log search each minute.”
Interviewer: “And more than one region?”
You: “I would collect and buffer in each region, so a regional outage does not stop the others, and keep the data there for local search. A search across regions fans out and merges results, which is slower, so I would say plainly that cross-region search is a convenience, not a fast path.”
If you remember one thing: Scale by tier and by quota, and look at the noisiest producers before adding machines.
Failure handling
Scaling answers “what if there is more traffic”. This section answers “what if something breaks”. Again, say it first. Failures here come from the network between machines, the queue, the processors, the index and the people searching, and what impresses an interviewer is that the answer was decided before the failure happened.
What you can say upfront: “Every failure has a rule decided in advance, and the main rule is the difference between visible delay and silent loss. If an agent cannot reach a collector, it spools to a limited disk and retries with backoff. If the queue lags, I throttle debug and watch lag by class. If the parser breaks, bad lines go to a quarantine topic and I replay after a fix. If the hot index is slow or red, searches fail fast with a clear message and old queries are routed away. If redaction fails, I block the stream and treat it as a security incident. A delayed line can still help later, but a silently dropped audit line may be a compliance failure.”
The follow-ups are about the details of those decisions.
Interviewer: “The network between the agents and the collectors goes down for ten minutes. What happens?”
You: “The agents keep reading the service's output and keep batches in their local spool, retrying with growing waits so they do not all return at the same second. The spool has a size limit, so if the outage were long enough to fill it, debug lines would be sampled and dropped first, and the agent would alert if audit lines were at risk. When the network returns, the spools drain in order, so search is late for ten minutes but mostly complete.”
Interviewer: “Queue lag keeps growing. What do you do?”
You: “First find out which lane. If it is one service, I cut its quota or turn its debug level down, because that is cheaper than anything else. If it is everything, I add consumers up to the number of partitions, and look at whether the index is the slow step. Throughout, I alert on lag by class, since a delay on debug is an annoyance and a delay on audit may be a problem.”
Interviewer: “The hot index turns red during an incident. What do searches see?”
You: “Engineers would see timeouts at the worst moment, so I use a circuit breaker, which trips like a fuse so the query service stops sending requests to a struggling index and returns an immediate, clear message. I reduce the refresh rate to cut write pressure, route older queries to the warm or cold tier, and keep the queue filling so nothing is lost while the index recovers. Lines are late, not gone.”
Interviewer: “A cold scan is far bigger than anyone expected. What stops the bill?”
You: “The query estimator rejects or flags it before it starts, large jobs have a byte limit, and anything above that needs approval. After the fact, cost by tenant and by query tells us who ran it.”
Interviewer: “Redaction fails after a release. Now what?”
You: “I stop the release, quarantine the affected stream, restrict access to the index, rotate whatever secret leaked and record who accessed it. Redaction failures page the platform team, because cleanup after indexing is slow.”
| Failure | What users or teams see | What to build |
|---|---|---|
| Agent cannot reach collector | Local disk spool grows | Retry with backoff, disk limits, class-specific drop policy, and spool alerts |
| Queue lag grows | Logs arrive late in search | Scale consumers, throttle debug, isolate noisy producers, and alert on lag by class |
| Parser bug rejects many logs | Missing fields or quarantine spike | Quarantine topic, parser error dashboards, replay from queue after fix |
| Hot index is red or slow | Search timeouts during incident | Circuit breaker, read-only mode, reduce refresh rate, route old queries away |
| Cold scan is too broad | Huge bill or long-running job | Query estimator, async job limits, approval for large exports |
| Redaction fails | Sensitive data becomes searchable | Block release, quarantine affected stream, rotate secrets, audit access |
If you remember one thing: Visible delay is acceptable, and silent loss is not, especially for audit lines.
Observability (what you measure)
Once the platform is running, you need a way to tell whether it is healthy, and a logging platform has a special problem: it is often the tool people use to debug someone else's incident, so it must be able to describe its own health without depending on itself. This section follows the path of a line and says what to watch at each stage, with enough detail that the numbers mean something on first read.
On each agent, watch how many bytes it reads and sends, how full the local spool disk is, how many lines it drops by class, how often it retries, and whether CPU, memory, or disk pressure is rising. On each collector, watch how many batches arrive, how often sign-in fails, how large the batches are, how long a queue write takes, and how many batches are rejected.
On the queue, watch how fast producers write, how far behind each topic and class is reading (consumer lag), whether work is uneven across lanes, how full broker disks are, whether any lane's copies have fallen behind, and how old a replay would still be useful. One global lag number hides the fact that audit may be healthy while debug is delayed, or the other way around.
On processors, watch parser failures, how many lines land in quarantine because of unexpected shape, secret-removal failures, how long enrichment takes, and routing errors. On the hot index, watch how long bulk writes take, how often writes are rejected, how long a refresh takes before new lines appear in search, how hard merges are working, how close disks are to their limit, and how long the slowest shard responses take.
On search, watch typical and slow query times, timeouts, bytes scanned, rows returned, how often expensive queries are rejected, and which people cost the most. Track spend by team, service, class, storage tier, and keep window so teams can see what they generate.
Alert ideas: any dropped audit line; lag on any class above its budget; a sudden fall in lines from a service that is known to be running (silence can mean a broken agent); parser errors rising right after a release.
Security and abuse
We have talked about keeping lines searchable and affordable; this section is about keeping the platform honest, because a logging system holds the data people forgot to protect elsewhere and is a tempting place to read or flood. Lines often contain authorization headers, cookies, access tokens, passwords, card numbers, and other fields teams did not mean to share. Remove those values before broad indexing. Agents and processors should strip known secret fields early, and if a raw copy must be kept for legal reasons, it belongs in tighter storage with a stronger access record.
On the search door, use role-based access (who may do what) and keep teams apart, so a developer only sees services and teams they are allowed to see. Security and audit lines may need stricter roles than ordinary application lines.
Protect the write door too. Agents should prove who they are with certificates or signed credentials, and untrusted producers should be rejected so an attacker cannot flood the platform with fake lines or poison an investigation.
Protect search from abuse as well. Limit how often expensive searches may run, reject searches with no time range, restrict leading wildcards and heavy pattern matching, and record who accessed sensitive streams. A logging search tool can become a way to copy data out of the company if access is too broad.
Cost awareness
Every choice so far also has a price, and interviewers like to hear that you noticed. Logging cost is usually the sum of keeping the bytes, building searchable structures on top of them, and running the machines that answer searches. The cheapest line is the one never written, so teams need clear feedback about bytes by service and by seriousness level.
For noisy debug lines, sample some of them and let operators change the logging level without a redeploy. Keep error and audit rules separate so cost-saving on debug never quietly weakens evidence. Shorten oversized messages and, when it is safe, stop writing the same spam line again and again. Compression helps, but it does not fix an application that logs full request bodies at the level every environment keeps.
Index only the fields people really filter on. Store the full message so it can be read back, but do not make every key searchable. Fields with nearly unique values on every line raise memory, disk, merge work, and query cost when they are indexed.
Move data down on a schedule. Hot indexes keep recent days. Warmer or colder tiers keep older data with slower search expectations. Object storage with files split by team, service, and date is much cheaper for long keep than leaving everything in a search cluster.
Make the bill visible. Charts by team, service, class, and field help teams quiet noisy logging before finance or the platform team has to chase them.
Green Dashboard. Angry Users. The Production Mindset System Design Interviews Expect.
Everything so far describes how the platform should work on a good day. This section is about a bad day, told as short stories from the people who carry the pager (the on-call engineers). Textbooks teach boxes and arrows. On-call teaches something else: a logging platform can look healthy while the lines people need are missing, and it can look busy while it is quietly making an outage worse.
A low error count can mean “nothing is wrong” or “the agents stopped sending”. A calm search cluster can mean “all is well” or “nothing is arriving”. Interviewers like this gap because it separates a diagram from someone who has run the thing.
Four nightmares that look fine on a dashboard:
| What the dashboard shows | What is actually happening |
|---|---|
| Search cluster CPU is low and nothing is red | Lines have stopped arriving, because agents are spooling or dropping |
| Cluster looks underused | One lane or one tenant is saturated while the rest sits idle |
| Search is slow, but disk and CPU look fine | One field with a unique value per line has bloated the index |
| Everyone reports the logging tool is healthy | A secret has been searchable for hours |
Scene 1: Logging breaks the app during an incident
The moment: A team ships a new release of an order service on a Thursday afternoon. They want more detail, so the release logs the full body of every request at INFO level, which is the level every environment keeps. Within minutes each machine is producing many times its usual volume, and the same team is now watching a dashboard for problems with the release.
The trap: Agents spool to disk because the collectors cannot keep up. On some nodes, disk pressure makes the applications themselves unstable, and the logging system, built to help debug the incident, is adding to it. Infinite local buffering is not safe, blocking application threads forever is not safe, and dropping everything silently is not safe either.
What to build: Decide the overflow rule before the night arrives. Agents cap how much disk and memory they may use, sample or drop debug first when those caps fill, alert on every drop by class, and let operators turn the logging level down for one service without a redeploy. Audit streams stay on their own path or behind a stronger rule so a flood of debug cannot erase security evidence.
Scene 2: One high-cardinality field melts the hot index
The moment: A search engineer on the platform team gets a request from an application team that keeps needing to find one request across many lines. To help them, she makes request_id an indexed field on every line, and the next search by request id is wonderfully fast.
The trap: Over the following weeks, the index mapping grows, memory use rises, merges slow, and dashboards begin to time out. The application team sees slow search, and the platform team sees memory and shard pressure. Search engines are fast because they build extra structures for each field, and a value that is different on every line makes those structures as large as the data.
What to build: Treat the searchable field list as a product decision, not a free-for-all. A small set of required fields is always indexed, other fields may be stored with the line so they can be read back without being searchable, and fields with nearly unique values need approval or a different store. Lines with a surprising shape go to quarantine so they cannot suddenly change the live hot-search layout.
Scene 3: A secret appears in logs before anyone notices
The moment: A developer fixing a bug in a payment service adds a line to the error path that prints the whole request, to see what the other system sent back. The request includes an Authorization header with a live access token.
The trap: The line enters the hot index, its replicas, snapshots and a team dashboard. By the time someone notices, the secret has been searchable for hours. The trap is relying on teams to never log sensitive data, because they will make mistakes, especially in error handling and retries.
What to build: Remove secrets in layers before a line becomes searchable. Libraries and agents block known secret fields early, stream processors replace or strip sensitive values before indexing, and any raw copy that must be kept lives in tighter storage with an access record. Failures in that removal path should page the platform team, because cleanup after indexing is slow and incomplete.
Scene 4: A rogue query hurts everyone
The moment: During a major outage, dozens of engineers are searching the platform at once. One of them is looking for any hint of the cause, so she searches thirty days across every service with a wildcard.
The trap: The hot cluster begins rejecting legitimate incident queries, because one search is using a large share of its capacity. A search box without time bounds is a shared-cluster risk, and the incident is made longer by the very tool meant to shorten it.
What to build: Put a planner in front of every search. Require a start and end time, estimate how many bytes the search would read, cap how many rows come back, reject costly patterns, and move large or old scans to background jobs over cheap storage. That keeps incident search useful for everyone else in the room.
In the interview: Pick one scene and tell it like a short story: what people felt, which metric lied, and what you would build next. You do not need all four. You need to sound like someone who knows that green does not mean done.
Bottlenecks and tradeoffs
Those stories all trace back to a small number of choices where you cannot have both things you want. Interviewers listen for whether you can name the tension, say what breaks, and pick a side on purpose. Here are the four that matter most for a logging system.
Completeness vs application safety
The tension — Teams want every line, but the application must keep running. If logging blocks the app under pressure, observability becomes an outage amplifier.
What breaks — Request threads wait on logging, or node disks fill.
What teams do — Class-specific policy. Debug can sample or drop. Audit should be isolated and alert on risk.
Say in the interview — Say out loud which classes may be sampled or dropped under pressure, and which classes must be kept or raise an alert instead of vanishing quietly.
Hot search vs retention cost
The tension — Hot indexes make recent search fast but are expensive to keep for months. Object storage is cheap but slower.
What breaks — Either the bill, or the patience of the engineer waiting on a month-old search.
What teams do — Hot storage for recent incident debugging and cold storage for long retention. The API makes the difference visible instead of pretending every query has the same latency.
Say in the interview — “A search over the last hour should answer in seconds; a search over last quarter becomes a background job with a download link.”
Schema flexibility vs index stability
The tension — Teams want to add fields freely, but uncontrolled fields create mapping and cardinality explosions.
What breaks — Index memory, merge speed and search latency.
What teams do — Define a core schema, approve indexed fields, store extra fields safely, and quarantine dangerous schema drift.
Say in the interview — “Keeping a field with the line so people can read it back is cheap; making every field searchable is expensive and can melt the index.”
PII deletion vs append-only audit
The tension — Privacy rules may require deletion while audit systems may require immutability. One storage policy cannot satisfy every class.
What breaks — A legal request to erase a person's data meets a record that is not allowed to change.
What teams do — Class-specific retention, redaction before indexing, legal holds where required, and carefully scoped deletion or crypto-shredding (making data unreadable by deleting its encryption key) when policy demands it.
Say in the interview — Name the conflict out loud, then say which class wins when privacy deletion meets an audit keep rule.
If you remember one thing: The platform exists to help during an incident, so it must never become part of the incident.
Interview tips
You now have a full design and the tradeoffs behind it, so the last thing to practice is defending it when the interviewer pushes back. These five are the pushes that come up most after you finish the design.
Do not stop at “ELK”
In the room
You might say: “We send all logs to Elasticsearch and use Kibana to look at them.”
Interview Push: “What happens when ingestion spikes, when one field has huge cardinality, or when old data becomes too expensive?”
Land here: Walk the full path in plain order: a small agent next to each service, a limited local buffer, a durable queue, parsing, secret removal, a recent hot index, a cheap cold archive, a query API with limits, and a lifecycle that moves older data down. The search product is one box in that path, not the whole design.
Define the loss policy out loud
In the room
You might say: “We do not lose logs.”
Interview Push: “The agent disk is full and the queue is two hours behind. Does the app block? Do audit logs drop?”
Land here: Split lines by class before you talk about products. Debug may be sampled or dropped first when space runs out. Audit takes a stronger path and raises an alert if it is at risk. The application should not wait forever on logging, but every drop must be counted so loss is visible and chosen on purpose.
Put redaction before indexing
In the room
You might say: “If a secret ends up in the logs, we can delete it later.”
Interview Push: “After it has been indexed, copied to replicas, snapshotted, and exported?”
Land here: Remove or replace secrets before a line enters the hot index. Libraries and agents block obvious secret fields early, stream processors enforce the full policy, and search access is limited by role. Cleanup after a line has been broadly indexed is slow and rarely complete, which is why timing matters.
Bound every query
In the room
You might say: “Users can search the logs for any keyword.”
Interview Push: “Can they search all services for thirty days with a wildcard during an incident?”
Land here: Every search needs a time range, an estimate of how many bytes it would read, a cap on rows returned, and a timeout for expensive work. Large or old scans become background jobs over cold storage instead of holding the shared cluster open during an incident.
Talk about logging hurting production
In the room
You might say: “If the logging system is down, we just lose visibility.”
Interview Push: “What if agents fill disk, or app threads block while writing logs?”
Land here: Logging can break the applications it watches. Agents need size-limited buffers, local spool caps, a written overflow rule, and a way to turn logging down without a redeploy. Watch dropped lines by class and agent disk use, not only whether the search cluster's CPU looks fine.
After each push, close with one real part of the design you would build, such as a spool limit per class, a quarantine topic, or a query planner, not “we will scale it later.”
What should stick
After working through the requirements, the decisions and the depth sections, you should be able to explain:
- Follow one log line — from the application write, through the agent buffer, the durable queue, parsing, secret removal, the hot index, the cold archive, and the search result.
- Class decides policy — debug, application, audit, and security lines have different sampling, keep windows, and loss rules, and you say those rules out loud.
- Pressure on buffers is the real design — every buffer needs a size limit and a rule for what gives first when it fills, and every drop is counted.
- Recent search is expensive — keep only recent days on the fast tier, and move older data to cheaper storage with slower search expectations.
- Remove secrets early and bound every search — sensitive fields and open-ended scans are far easier to prevent than to clean up afterwards.
Tell it in the room: “Applications write tidy log lines to agents. Agents group and shrink them, hold them on a limited local disk, and follow a class-specific rule when that disk fills. Collectors write into Kafka as the durable buffer. Stream processors pull out fields, remove secrets, and route each line. Recent days go to hot time-split indexes; older days go to object storage. The query API checks who may search, requires a time range, estimates cost, and turns large old scans into background jobs. Under pressure, debug can be sampled first, and audit raises an alert instead of vanishing quietly.”
Key Takeaways
- Logging must never block the application: write locally, let an agent batch and spool within a size limit, and give every buffer a rule for what gives first.
- Give every line a class. Debug can be sampled or dropped under pressure, while audit and security lines take a stronger path and raise an alert instead of vanishing.
- Use a durable queue between agents and the search system, split by tenant, service and class, so a slow index or a noisy service cannot stall everyone; remember it holds hours, not days.
- Remove secrets before a line becomes searchable, because cleaning an index, its replicas and its snapshots afterwards is slow and rarely complete.
- Keep recent days in a time-split hot index with only approved fields indexed, and move older lines to cheap object storage searched through background jobs.
- Require a time range and size limits on every search, and send large or old scans to asynchronous jobs.
- Watch dropped lines by class, lag by class, agent disk, parser errors and query cost, not only search cluster CPU.
Related Topics
- Auth System Design - Audit logs, account events, and sensitive security data.
- Rate Limiter System Design - Query and ingest quotas under shared load.
- Distributed Cache System Design - Hot keys, partitions, and operational blast radius.
- Job Scheduler System Design - Background workers, retries, and dead-letter queues.
- Design WhatsApp - Privacy-aware logging when message content is sensitive.
Quick revision notes
The problem in your own words. A logging system collects the short notes every service writes about what it is doing, keeps recent ones searchable within seconds, keeps older ones cheaply, hides secrets, and does all of it without ever making the services it watches wait. The hard part is deciding what happens when the pipeline is slow or full.
The design in one breath. The service writes a line locally. An agent batches it and spools it to a limited disk, applying the rule for its class. A durable queue, split by tenant, service and class, absorbs floods. Processors parse the line and remove secrets, and indexers write it to a time-split hot index with only approved fields. A lifecycle job moves older lines to cheap object storage, and a query service enforces a time range and size limits on every search.
The choices you must be able to defend. Debug may be sampled or dropped, audit must be kept or alert, and every drop is counted. Delivery is at least once with a unique id per line. Redaction happens before indexing. High-cardinality fields are stored but not indexed. Old data is searched through background jobs.
The numbers worth remembering. Hundreds of terabytes a day is a few gigabytes every second, so seven days of raw text is about two petabytes, and a queue holds hours of traffic rather than days. Partition and storage figures here are estimates that you would confirm with a load test.
The mistakes that cost points. Stopping at the name of a search product, claiming “we never lose logs”, letting services send straight to the index, indexing every field, cleaning secrets after indexing, allowing unbounded searches, and watching only the search cluster while lines quietly stop arriving.
Frequently asked follow-ups
- How do agents collect and buffer logs?
- Where is the durable point in the pipeline?
- How do you search recent logs quickly?
- How do you control retention and cost?
- What happens when logging breaks applications?
Deep-dive questions and strong answer outlines
How does one log line travel through the system?
The application writes structured output to stdout or an agent socket. The agent batches and spools to disk if upstream is slow. Regional collectors write to Kafka or Kinesis with durability. Stream processors parse, enrich, redact PII, route by tenant and class, and bulk-write recent logs to a hot search index while archiving older or raw data to object storage.
Where is a log considered durable?
For normal debug logs, durability may start when the agent has spooled to disk or when Kafka acknowledges the write, depending on policy. For audit logs, define a stronger point, usually successful write to the durable queue with replication. Say which class can be dropped under pressure and which must alert instead.
How do you stop one service from overwhelming the platform?
Apply quotas by tenant, service, and log class at the agent or collector. Use dynamic log-level controls, sample debug logs first, isolate hot tenants with separate topics or partitions, and alert on bytes per service. Do not let a noisy debug stream share the same fate as audit logs.
How do you keep queries fast?
Require time bounds, route to time-based hot indexes, limit rows and bytes, reject or async-export large cold scans, and control indexed fields. Search recent data in a hot tier; query old data from object storage or a columnar engine with slower expectations.
How do you handle PII in logs?
Redact or tokenize sensitive fields in the stream before indexing. Block known secret fields at the agent when possible, enforce schemas, restrict query access with RBAC, and keep raw logs in tighter storage if they must be retained. Deleting PII after petabytes are indexed is slow and unreliable.
AI feedback on your design
After a practice session, InterviewCrafted summarizes strengths, gaps, and interviewer-style expectations—similar to a written debrief. See a static example report, then practice this problem to get feedback on your own answer.
FAQs
Q: Is ELK enough for the answer?
A: ELK or OpenSearch can be the hot search engine, but the design also needs agents, buffers, parsing, redaction, quotas, retention tiers, query limits, and failure policy.
Q: Should logs be exactly once?
A: Most logging systems accept at-least-once delivery with duplicate-tolerant event ids. Losing audit logs is worse than duplicating a line. Debug logs may accept sampling or loss under pressure if that policy is explicit.
Q: How is logging different from metrics and traces?
A: Logs are detailed events and messages. Metrics are aggregated numbers with low cardinality. Traces connect spans across services. A good platform links them through fields such as trace_id, but it does not store every trace dimension as an indexed log label.
Q: Why not index every field?
A: Every indexed field costs memory, disk, and merge work. High-cardinality fields such as request_id and user_id can explode index size if indexed blindly. Store them, but only index fields that teams really need to filter.
Practice interactively
Open the practice session to use the canvas and stages, then review AI feedback.