Logging System Design

Introduction

A bad deploy turns on DEBUG for one service. Within an hour the logging bill exceeds last month's total. On-call opens Kibana—and search times out because someone indexed user_id on every line.

Logging platforms lose interviews when candidates treat search as free. Strong answers tier data, bound cardinality, and shed load before Kafka backs up into application processes.

Weak answers stop at "Elasticsearch." Strong answers walk agent → buffer → process → hot index → cold archive and say what happens when a service printf-loops.

If you remember one thing: Logs are a product with SLOs and cost—not an infinite printf sink.

How to approach

Talk like you are following one log line, not naming observability vendors.

1. Ask scope — Audit vs debug classes? Multi-tenant? Retention and compliance (GDPR delete vs WORM)?
2. Producer → buffer — Agent batching, ack semantics, backpressure to the app.
3. Process → index — Enrichment, PII scrub, route by tenant/class.
4. Hot vs cold — Days in search index; months–years in object storage.
5. Query — Time-bounded API, RBAC, cost caps on large scans.

In the room: "I'll classify log types first, then walk one line from stdout to hot index and say when it ages to cold storage."

If you remember one thing: Define which hop is durable before you draw boxes.

Interview tips

Five common exchanges. Each has a trap answer, a pushback, and where to land.

Stopping at "Elasticsearch"

You: "We send everything to Elasticsearch and search in Kibana."

They ask: "One service logs request_id as a keyword on every line—what happens to heap in six months?"

Land here: Hot tier for recent interactive search with cardinality limits. Route high-cardinality fields to non-indexed storage or columnar cold tier. ILM moves old data to cheap object storage.

Ack semantics

You: "Once the app writes to stdout, the log is safe."

They ask: "The pod dies before the agent ships the line—did you lose the audit event?"

Land here: Agent acks to the app only after durable enqueue in Kafka (acks=all). Define which hop is "safe" for audit class vs debug.

Backpressure under burst

You: "Kafka absorbs any spike—we never drop."

They ask: "Kafka lag hits two hours and agent disks fill—then what?"

Land here: Backpressure to producers: sample or throttle debug at the agent. Never silently drop audit without policy. Dynamic log level on the noisy deployment. Dedicated partitions for noisy tenants.

PII in logs

You: "We'll delete PII later with a batch job."

They ask: "GDPR request while petabytes are indexed—how long does that take?"

Land here: Scrub in stream at ingest—tokenize emails, strip auth headers. RBAC on query API. Retention-only policy vs true delete is a product/legal call—name the tension.

Audit vs debug

You: "We sample 1% of all logs to save cost."

They ask: "Finance needs every admin action for seven years—where did those lines go?"

Land here: Separate streams or classes. No sampling for audit. WORM or append-only store where required. Debug can sample; audit cannot.

If you remember one thing: After each push, name one control—quota, cardinality cap, tier, ack hop—not "we'll scale the cluster."

Capacity estimation

So we cannot: index every debug field at full cardinality forever. We cannot treat cold-tier scans like hot search without cost estimators and timeouts. We cannot let one tenant fill shared buffers without quotas.

If you remember one thing: Hot = fast and expensive; cold = cheap and slow—APIs must reflect that.

High-level architecture

What breaks if ingest has no guardrails

A loop logs in a tight for. Kafka lag grows. Agent disks fill. The app blocks on backpressure—or OOMs. Audit lines drop silently with debug because everything shared one pipe.

What works: buffer, classify, tier

Agents (Fluent Bit, Vector, etc.) run on nodes or as sidecars. They tail stdout or receive gRPC, batch, compress, and spool to disk if upstream is slow. Regional collectors produce to Kafka / Kinesis. Stream processors enrich (Kubernetes metadata, trace id normalization), mask PII, and route by tenant and log class. Indexers bulk-write to OpenSearch/Elasticsearch hot tier (e.g. 7 days). Lifecycle jobs move to S3 Parquet with Athena/Glue or ClickHouse for analytics.

Who does what:

• Agent — Backpressure to the app; never block app threads forever.
• Kafka — Durable buffer; replay for reindex.
• Indexer — Bulk API; segment merges cost CPU.
• Query API — AuthZ/RBAC; query cost estimate and timeouts.

[ App ] --> stdout / socket --> [ Agent ] --> [ Regional collector ]
                                                  |
                                                  v
                                            [ Kafka / Kinesis ]
                                                  |
                    +-----------------------------+-----------------------------+
                    v                             v                             v
            [ Stream: PII scrub ]           [ Stream: route ]            [ Stream: metrics ]
                    |                             |
                    v                             v
              [ Hot search index ]          [ Object store Parquet ]
              (days, interactive)         (months–years, async query)

In the room: Say explicitly: audit streams may skip sampling and land in WORM or stricter retention.

If you remember one thing: The buffer protects producers; quotas protect the cluster—both are required.

Core design approaches

Structured log model

JSON with required fields: timestamp, service, level, trace_id, message (bounded size). Reject or quarantine oversize lines and schema drift.

If you remember one thing: Unbounded message size and unknown fields are DoS vectors—cap them at ingest.

Hot / warm / cold tiering

Hot: sub-hour query latency for on-call—search index with ILM.

Warm/cold: minutes to hours for compliance or analytics—object storage, columnar engines.

If you remember one thing: Query API should default to hot and require explicit opt-in for expensive cold scans.

Detailed design

Walk one log line from emit to search.

Ingest path

1. App writes structured JSON to stdout or agent socket.
2. Agent buffers in memory; spools to disk if Kafka is slow; applies tenant quota.
3. Produce to Kafka with acks=all (durability vs latency tradeoff).
4. Consumer group enriches, scrubs PII, routes to indexers; bulk-write to hot cluster.

If you remember one thing: Audit and debug should diverge before the index— not after everything is mixed.

Query path

1. User submits time range plus filters (service, level, text query).
2. Planner picks time partitions on hot tier (or triggers async cold export).
3. Execute with timeout and row/byte cap; return cursor for pagination.

In the room: "Search without from/to is rejected—unbounded queries burn money."

If you remember one thing: Every query needs time bounds and a cost ceiling.

Key challenges

• Burst after bad deploy — Rate limit per service at agent; dynamic log level down; isolate partition.
• Schema drift — Reject unknown high-cardinality fields or quarantine to raw topic.
• Multi-tenant noisy neighbor — Quota ingest EPS and query cost; separate clusters for prod vs staging.
• GDPR delete vs append-only — Retention TTL vs async compaction; legal hold exceptions.

If you remember one thing: Cardinality and volume are the enemies—not disk size alone.

Scaling the system

• Partition Kafka by service or tenant; dedicated topics for audit.
• Horizontal indexers; frozen indices for old hot segments; ILM to cold.
• Separate clusters for prod vs staging; optional per-tenant dedicated index for whales.
• Regional collectors when data residency requires local ingest before federation.

If you remember one thing: Scale consumers with lag alerts—not only index node count.

Failure handling

Real outage = cannot investigate prod (search down). Silent loss = audit gaps—worse for compliance.

If you remember one thing: Alert on dropped lines by class and ingest lag—not only index CPU.

API design

GET /v1/logs query params:

Diagram (query hot path):

User --> Query API --> AuthZ --> Search cluster (hot, time partitions)
                           |
                           +--> async export job --> cold (Parquet) for large scans

Errors: 429 ingest or query quota exceeded; 400 missing time range. RBAC denies cross-tenant reads.

In the room: Walk ingest ack hop, then GET /v1/logs with mandatory from/to.

If you remember one thing: Public query APIs must bound work—time, limit, timeout.

Production angles

[ Log storm ] --> Kafka lag --> consumers fall behind
       --> agents buffer --> disk full --> drop or block producer

Bottlenecks and tradeoffs

Completeness vs cost

The tension — Engineers want every debug line; finance wants a finite bill.

What breaks — Cluster melt or forced sampling that accidentally includes audit.

What teams do — Explicit sampling for debug only; quotas; dynamic levels.

Say in the interview — "100% debug capture is a policy choice with a price tag."

Search vs analytics storage

The tension — Inverted index excels at keyword needle search; columnar excels at aggregations over PB.

What breaks — Using one engine for both at petabyte scale.

What teams do — Hot OpenSearch for on-call; cold Parquet + Athena/ClickHouse for analytics.

Say in the interview — Match store to query pattern, not vendor habit.

Immutability vs GDPR delete

The tension — Audit wants append-only proof; privacy wants erase.

What breaks — Expensive reindex or legal hold conflicts.

What teams do — Retention TTL by class; crypto-shredding or scoped delete jobs; legal hold overrides.

Say in the interview — Name the tension; do not pretend one index satisfies all compliance modes.

If you remember one thing: Logging tradeoffs are cardinality, tier cost, and audit vs debug—not "which ELK vendor."

What should stick

You do not need to memorize every box. After this guide, you should be able to:

1. Backpressure chain — Agent → Kafka → index; shed debug before the app or audit suffers.
2. Hot vs cold — Recent logs in search index; old logs cheap in object storage—different query SLAs.
3. Cardinality budget — High-cardinality labels destroy index cost; block at agent.
4. Audit vs debug — Separate streams; no sampling on audit; WORM where required.
5. Ack hop — Durability means Kafka ack, not stdout write.

Tell it in the room: "Apps emit JSON; agents batch to Kafka with acks=all. Stream processors scrub PII and route audit separately. Indexers write hot tier for seven days; ILM rolls to Parquet in S3. Query API requires time bounds and quotas. Under burst, we throttle debug at the agent and never silently drop audit without alerting."