Job Scheduler System Design

Introduction

Midnight UTC hits. A million cron jobs wake at once. Billing runs. Then billing runs again — because the scheduler leader failed over and the new leader did not know the first batch already fired. Support finds duplicate charges. Marketing emails went out ten times.

That is what users feel when "run at T" is treated like a single-server cron tab on one VM.

Interviewers care about three failures: duplicate execution, lost jobs, and stuck jobs after a worker dies halfway through. The product wants billing once and emails once — not a lecture on TCP.

Your design should center a state machine, leases, and idempotency. Weak answers draw cron on one box. Strong answers separate when (time index + leader or shard) from who runs (workers + claim with lease).

If you remember one thing: Scheduling is durable timers at scale — separate "fire at T" from "execute safely with at-least-once delivery."

How to approach

Spend the first few minutes turning a vague prompt into a contract. Walk one job from scheduled → claimed → executed → acked or retried before you name queue brands. The goal is not to sound clever — it is to stop yourself from promising exactly-once on the wire.

Clarifying questions (say these in the room)

You: "Are jobs one-off, recurring cron, or dependency chains? If they mention DAGs, I will park full workflow orchestration and focus on enqueue, lease, retry — unless they explicitly want Airflow-level deps."

Interviewer: "Mostly one-off and cron; mention deps briefly if you have time."

That answer tells you the MVP is schedule + execute, not a full workflow engine.

You: "What execution guarantee does the product need? I assume at-least-once delivery on the wire and idempotent handlers for business effects — billing cannot rely on 'we never retry.'"

Interviewer: "Yes — retries are expected; side effects must be safe."

Now you can defend leases and DLQ instead of pretending exactly-once HTTP callbacks.

You: "How long do jobs run — sub-second webhooks or hour-long batch exports? That drives lease TTL, heartbeat, and whether long jobs get their own queue."

Interviewer: "Mix — most are seconds; some run ten to thirty minutes."

Plan separate lease policies and heartbeat for the long tail.

You: "Do we need strict priority — payment before marketing email — or is FIFO enough with tenant quotas?"

Interviewer: "Priority matters; starvation of low priority is a concern."

You will need aging or weighted fair queueing, not a single strict heap forever.

You: "Should the API return after durable persist only, or block until execution completes?"

Interviewer: "Async — 202 after persist."

That sets the async boundary: HTTP does not hold the worker.

You: "I will keep full DAG workflow engines, stream processing, and client-side scheduling out of scope so we can focus on durable enqueue, cron materialization, leases, and DLQ. Does that match?"

Interviewer: "Yes — core scheduler."

Saying out of scope early stops you from burning half the interview on Airflow internals.

Minute pacing (about 45–50 minutes)

Minutes	Focus
0–5	Clarify guarantees, job types, long vs short, priority, out of scope
5–12	Capacity + high-level boxes + one job lifecycle
12–22	Data model + write/claim paths + DB-as-queue vs external queue
22–32	Architectural dig: worker crash mid-job (lease + idempotency)
32–40	Cron jitter, failure table, API, observability
40–48	One production scene + bottlenecks
48–50	Questions

In the room (a fuller opening you can actually say): "I will assume at-least-once delivery with idempotent handlers for side effects — not exactly-once on the wire. Clients POST jobs; the API persists with next_run_at and returns 202. A scheduler leader ticks due rows into a ready queue without double-inserting the same cron occurrence — unique on cron_id and occurrence time. Workers claim with a lease and heartbeat for long jobs, call the customer webhook, ack on success or retry with exponential backoff; poison goes to DLQ. Cron gets jitter so :00 is not a stampede. I will measure queue age, not only CPU."

If you remember one thing: Guarantee delivery attempts separately from business effects once.

Out of scope (and why you park these)

Full DAG workflow orchestration (Airflow-class). Dependency graphs, backfill UI, and operator libraries are a different product layer. Acknowledge level-order deps as an extension; design the simple enqueue + lease path first.

Stream processing (Flink, Kafka Streams). Continuous event pipelines solve a different problem than durable delayed execution. Your scheduler fires discrete jobs at known times.

Client-only scheduling. Phones and browsers sleep, lie about time, and disappear. The server owns next_run_at and materialization.

Exactly-once side effects without idempotency. Say you cannot promise it on HTTP callbacks; park the debate by committing to idempotency keys and dedupe tables up front.

Capacity estimation

Rough numbers to anchor the whiteboard (tune in the room):

Load	Order of magnitude	Design implication
Future scheduled jobs	100M+ rows	Time-indexed queries; partition by tenant or time — avoid full table scan each tick
Executions per day	~10M	Worker pool sizing; batch dispatch from ready queue
Peak enqueue rate	~500/s at :00 without jitter	Horizontal workers; scheduler shard by tenant/hash; jitter spreads peak
Execution history	~TB/year	TTL archive to cold storage; retain failures longer than successes

Implications:

• The scheduler tick path must be O(due batch), not O(all jobs) — index on (status, next_run_at) or equivalent.
• Executions table grows fast — partition by time; do not store full payload blobs in every attempt row if you can reference object storage.
• Peak :00 cron without jitter can enqueue a minute's worth of work in seconds — size workers and downstream callbacks for age SLO, not average CPU.

If you remember one thing: Index for next due, plan execution history growth, and treat :00 as a capacity event unless jitter spreads it.

High-level architecture

API creates and updates jobs in a durable store (SQL or NoSQL with strong writes). Scheduler process(es) — often leader-elected — run a tick loop: query jobs where next_run_at <= now (with index), enqueue to a ready queue (Kafka, SQS, Rabbit) or direct claim if DB-as-queue. Workers pull or receive push, claim with lease (locked_until, worker_id), execute user payload (HTTP callback, container, script), ack or fail with backoff. History rows append attempt records. DLQ holds poison after max retries.

Who owns what:

• Scheduler leader — Advances cron occurrences, hydrates due jobs into ready queue; must not double-insert same occurrence — unique (cron_id, occurrence_ts) or fencing.
• Ready queue — Buffers runnable jobs; decouples tick rate from worker CPU; exposes backpressure via depth and age.
• Workers — Stateless; horizontal scale; respect lease; extend via heartbeat for long jobs.
• Execution store — Audit trail, status, error blobs; may share DB or stream to analytics.

[ Clients / services ]
        |
        | POST /jobs (sync: persist only → 202)
        v
[ Job API ] ──write──► [ Job metadata DB ]
        |                    ^
        |                    | next_run_at index
[ Scheduler Leader ] --------┘
        |
        | tick: due rows → enqueue (async)
        v
[ Ready queue: Kafka / SQS / DB SKIP LOCKED ]
        |
        v
[ Worker pool ] ──lease + heartbeat──► [ User workload ]
        |                                    |
        | fail max retries                   v
        v                              [ DLQ + alert ]

  Cron: scheduler expands next occurrence → same pipeline (with dedupe + jitter)

In the room: Say the async boundary clearly: HTTP returns 202 after persist — execution is later. Then describe lease, retry, and DLQ before diving into cron sharding.

If you remember one thing: API persists; scheduler enqueues; workers claim with lease.

Core design approaches

Database-as-queue vs external queue

DB polling with SELECT … FOR UPDATE SKIP LOCKED (Postgres) or equivalent can work at moderate scale — simpler ops, one system of record, harder to scale when tick and claim contend on the primary.

External queue adds moving parts but smooths bursts, standardizes retry visibility timeout and DLQ patterns, and decouples scheduler tick from worker throughput.

Approach	When to pick	What breaks first
DB-as-queue	MVP, moderate QPS, strong consistency appetite	Primary hot on tick + claim
External ready queue	Burst at :00, many workers, need queue-native DLQ	Ops complexity, dual-write without outbox
Hybrid	Metadata in DB, ready work in queue	Outbox or transactional enqueue discipline

Say in the interview: Start DB-as-queue if scale is modest; move ready work to Kafka/SQS when tick duration or claim contention hurts — not on day one by default.

Single leader vs sharded schedulers

Single leader tick is easy to reason about for dedupe and cron materialization. Shard by hash(tenant_id) or key range when one leader cannot scan enough due rows per second — each shard owns a subset of crons and must still dedupe within its shard.

Push vs pull workers

Pull (workers poll queue or DB) scales horizontally and naturally backpressures when workers are full. Push (scheduler dispatches to known workers) can reduce latency but needs careful double dispatch guards when the pusher retries.

Priority and fairness

Strict priority queues can starve low priority forever. Mitigations: aging (boost priority over wait time), weighted fair queueing, separate lanes (critical vs batch), per-tenant concurrency caps.

Dependency execution (brief)

Level-order: run jobs whose parents are complete; on parent success, unblock children — event-driven wake beats polling deps every second. Full DAG engines are out of scope unless the interviewer opens them.

If you remember one thing: DB-as-queue is simpler until the tick becomes hot; external queue adds ops but smooths :00 bursts and DLQ.

Data model

Without tables and fields, "we use a queue" is not a design.

jobs (schedule + payload metadata)

Column	Role
job_id	Primary key; correlation id for callbacks
tenant_id	Fairness, sharding, quotas
status	PENDING, RUNNING, COMPLETE, FAILED, CANCELLED
schedule_type	once, cron, interval
cron_expr / interval_sec	Recurrence definition
next_run_at	Indexed — scheduler tick queries due rows
priority	Queue lane or ordering hint
payload_ref	Pointer to blob in object storage — not multi-MB inline
max_attempts	Cap before DLQ
idempotency_key	Client-supplied dedupe on create
created_at, updated_at	Audit

Cron dedupe: When materializing an occurrence, insert or enqueue with unique (cron_id, occurrence_ts) so leader failover does not create two runnable rows for the same minute.

executions (attempt history)

Column	Role
execution_id	Per attempt
job_id	FK
attempt	Monotonic retry count
worker_id	Who claimed
lease_version / fencing_token	Stale worker detection
started_at, finished_at	Latency, SLO
status	RUNNING, SUCCESS, FAILED, LEASE_EXPIRED
error_code, error_blob	DLQ forensics

Partition by time — this table grows fastest.

Lease fields (on job row or claim table)

Field	Role
locked_until	Wall time or DB time — claim expires after this
worker_id	Current owner
lease_version	Increment on each claim — callback with old version rejected

Heartbeat: UPDATE … SET locked_until = now() + interval WHERE job_id = ? AND worker_id = ? AND lease_version = ?.

cron_next_run (optional normalization)

Some teams store last_occurrence_at and computed next_run_at on the cron definition row; tick advances next_run_at after successful materialization. Jitter: store jitter_sec and set first fire to base_time + random(0, jitter_sec).

DLQ

Field	Role
dlq_id	Entry id
job_id, final_attempt	Link back
reason	Poison, max attempts, manual kill
payload_snapshot	For replay forensics
replayed_at	Guard against accidental DLQ loops

If you remember one thing: next_run_at index, lease columns, cron dedupe key, and DLQ as first-class tables — not implied by the queue product.

Detailed design

This section follows one job from “customer clicked schedule” to “worker finished or gave up,” so you can hear how the pieces talk—not only what tables they touch.

Write path (schedule a job)

A client submits work: what to run (payload or webhook target), when (immediate, delayed, or cron), how important it is (priority), and an idempotency key. That key is how you survive mobile retries and “submit” double-clicks. If the same key arrives twice, the API returns the same job id and does not create a second row that will fire twice.

When the request is new, the API writes a durable job in a waiting state (PENDING), computes the first next_run_at, and answers quickly—often 202 Accepted—so the client does not wait for a worker to finish. Recurring jobs get their next fire time from the cron expression, preferably with a little random jitter so thousands of “every hour on the hour” jobs do not all wake at the same second.

If the product allows DAGs (parent jobs before children), store dependency links and only mark a child ready after every parent finishes successfully. Keep that story brief unless the interviewer expands scope.

When the durable store and the ready queue are separate systems, use an outbox: in one database transaction write both the job row and an “please enqueue me” outbox row; a small relay publishes to the queue afterward. That pattern exists because “saved in the database but never handed to workers” is a classic crash bug if you publish before the commit finishes, or ack the HTTP response before both persist.

Scheduler tick (turning “due” into “ready”)

Something must wake work when next_run_at arrives. That can be a leader process, a sharded set of tickers, or a delay queue—name your choice. On each tick, load a bounded batch of due rows (next_run_at <= now(), limited page size) so one tick cannot enqueue millions of messages and melt itself.

For each due job, create a ready-queue message or an execution attempt record. For cron, use a dedupe key such as (cron_id, occurrence_timestamp) with a uniqueness rule, so a leader failover cannot materialize “midnight billing” twice. Then update that cron’s next next_run_at (again with jitter if you use it) and advance the tick cursor. If you use a single leader, protect ticks with a fencing token or lease so an old leader that lost the election cannot keep writing.

Worker claim (taking ownership of one attempt)

Workers either pull from the external queue or poll the database. Either way, claiming must be exclusive. A common database pattern is: find one ready row whose lease has expired (or is unowned), lock it (FOR UPDATE SKIP LOCKED), and mark it RUNNING with locked_until = now + lease_ttl, a worker_id, and a rising lease_version.

The worker then performs the real side effect—often an HTTP callback to the customer’s system—passing identifiers such as job id, attempt number, and lease version so the receiver can ignore delayed duplicates. If the callback succeeds, the worker marks the job COMPLETE, records a successful execution, and acknowledges the queue message so it will not reappear.

If the callback fails in a way that might be temporary (timeouts, 5xx), the worker increments the attempt count, schedules next_run_at with exponential backoff, and returns the job to waiting—or lets the queue’s retry policy do that job. After too many attempts, the job moves to a dead-letter queue (DLQ) with an alert. Humans decide whether to fix and replay; automatic silent retries forever are how you burn vendor APIs overnight.

Long-running jobs

Some work lasts minutes or hours. Give those jobs a longer lease and require the worker to send heartbeats that push locked_until forward. If heartbeats stop, the lease expires and another worker may claim the job—so the handler still needs an idempotency story (or checkpoints) so a second claim does not double-charge or double-email. Keep long jobs on a separate pool if they would starve short interactive work.

Client POST /jobs → persist PENDING → 202 + job_id
        ↓
Scheduler tick → ready queue (deduped for cron)
        ↓
Worker claims with lease → optional heartbeats
        ↓
Customer callback → success: COMPLETE / failure: backoff → or DLQ

In the room (spoken): “I persist first, then materialize due work in bounded ticks with cron dedupe. Workers claim with a lease and heartbeat for long jobs. Delivery is at-least-once, so handlers are idempotent. Poison messages go to a DLQ with an alert—not infinite retries.”

Architectural dig: worker crash mid-job

This is the interview meat: what happens when the worker dies after the callback succeeded but before ack — and why fire-and-forget fails.

1) Bad design: fire-and-forget dispatch

Shape: Scheduler pushes job to worker over HTTP; no lease; no persistent RUNNING state; "done" means the HTTP call returned.

Concurrency scene:

t0  Worker W1 receives job J, starts 8-minute export
t1  W1 crashes at minute 4 — no heartbeat, no locked_until
t2  Scheduler thinks J is lost OR re-dispatches blindly
t3  Worker W2 starts J again — customer webhook runs twice
t4  Duplicate charges, duplicate emails

Why it fails: No authoritative claim; no way to distinguish "still running" from "never started"; retry creates duplicate side effects because the handler is not idempotent.

Interview land: "Fire-and-forget cannot survive at-least-once — you need lease + ack and idempotent handlers."

2) Great design: lease + heartbeat + idempotency

Owns: Worker holds lease on job row; scheduler/queue respects locked_until; customer handler dedupes on job_id + effect key.

Internals:

1. Claim sets RUNNING, locked_until = now + 5m, lease_version = 3.
2. Worker heartbeats every 60s: extend locked_until if still worker_id matches.
3. Callback includes X-Job-Id, X-Attempt, X-Lease-Version.
4. Customer stores (job_id, effect) unique — second POST is no-op success.
5. Worker acks only after durable COMPLETE write; message deleted from queue.

Concurrency scene (crash after callback 200, before ack):

t0  W1 claims J, lease_version=3
t1  W1 POST callback → customer returns 200, commits idempotent row
t2  W1 crashes before ack
t3  lease expires; W2 claims J, lease_version=4
t4  W2 POST callback → customer dedupe hit → 200 no-op
t5  W2 acks → COMPLETE once from business view

Figure: Lease bounds retry; idempotency bounds business effects.

Rejected alternative: "Exactly-once queue" — most managed queues are at-least-once; visibility timeout without heartbeat recreates the same duplicate window.

What breaks first inside the box: Heartbeat storms on thousands of long jobs — batch lease extend or longer TTL with fewer heartbeats.

Failure inside the box: Customer returns 200 slowly; worker must not ack until confirmed — or use outbox on customer side; stale worker with old lease_version must be rejected if it tries to commit late.

Interview land: "Wire is at-least-once; lease narrows duplicate execution window; idempotency makes duplicates safe."

3) Numbered happy path (defend in one breath)

1. Persist job with next_run_at.
2. Tick enqueues with dedupe.
3. Worker claims with lease.
4. Heartbeat while running.
5. Idempotent callback.
6. Ack + COMPLETE, or retry with backoff, or DLQ.

Key challenges

• Double dispatch: Two schedulers enqueue same cron tick — unique (cron_id, occurrence_ts), leader fencing, deterministic job id per scheduled minute.
• Lost work: Crash after API 202 but before enqueue — outbox pattern for DB + queue atomicity.
• Stuck RUNNING: Worker dies — lease expires; another worker claims — business sees at-least-once unless handler is idempotent.
• Thundering herd at :00: Millions of crons fire same minute — jitter within window; separate tiers; pre-scale workers.
• Priority inversion: Strict priority starves batch — aging or weighted fair queueing.
• Long vs short jobs: Shared pool lets one export block thousands of webhooks — separate queues and concurrency pools.
• Multi-tenant noisy neighbor: One tenant fills queue — per-tenant concurrency caps and fair dispatch.

If you remember one thing: Name double dispatch, lost enqueue, stuck RUNNING, and :00 herd — each with a mechanism, not a product name.

Scaling the system

• Workers: Horizontal; auto-scale on queue lag (oldest message age), not only CPU — age is user-visible pain.
• Scheduler: Shard tick by tenant or hash range; cap batch size per tick; read replicas for reporting — not for claim path without careful locking.
• DB: Partition jobs by tenant or time; isolate hot tenants to dedicated shards or queues.
• Cron jitter: Spread materialization — reduces synchronized downstream load (DB, customer APIs).
• Global schedules: Region-local schedulers for data residency; cross-region cron authority is rare — clarify which region owns next_run_at.

If you remember one thing: Scale workers on queue age; shard scheduler tick before one leader melts the primary.

Failure handling

Failure	Effect	Mitigation
Worker crash mid-job	Duplicate run after lease expiry	Idempotent handler; dedupe table; lease_version on callback
Queue backlog	Growing lag, late cron	Scale workers; shed low-priority; delay cron materialization when backlogged
Scheduler leader dies	Tick pauses seconds	Failover; dedupe prevents double fire; rate-limit catch-up backfill
User callback 500	Retry storm	Exponential backoff, max attempts, DLQ
Poison message	Infinite retry loop	DLQ after max attempts; alert; manual replay with gate
DB primary slow	Tick and claim lag	Index tune; shard; move ready queue off hot primary

Degraded UX: Jobs late during backlog — acceptable with honest SLO. Lost jobs or unbounded retry — unacceptable; durability and DLQ discipline are the product.

fail attempt → backoff → retry (attempt++)
                │
                └── attempt >= max → DLQ + page on-call

If you remember one thing: Worker death → lease expiry → retry is normal; poison → DLQ, not infinite loops.

API design

Endpoint	Role
POST /v1/jobs	Create; body: schedule, payload, priority, retry_policy; header Idempotency-Key
GET /v1/jobs/{id}	Status, next_run_at, attempts, last error
DELETE /v1/jobs/{id}	Cancel future runs; optional signal running worker (SIGTERM / lease revoke story)
POST /v1/jobs/{id}/trigger	Manual run (admin)
GET /v1/jobs/{id}/executions	Attempt history for support

Webhook callback (worker → customer system):

Header / field	Role
X-Job-Id	Correlation
X-Attempt	Retry count
X-Lease-Version	Stale worker detection
X-Scheduler-Signature	HMAC verify payload

Errors: 202 on create (accepted, not executed); 409 on idempotency conflict with mismatched body; 404 unknown job; 429 tenant quota exceeded.

POST /v1/jobs ──► 202 Accepted + job_id
                      │
Scheduler tick ──► ready queue ──► Worker ──► POST https://customer/callback
                                                  │
                                            200 ─► ack
                                            5xx ─► retry w/ backoff ─► DLQ

If you remember one thing: Create returns 202; execution is async; Idempotency-Key on create; signed callbacks on execute.

Observability

Do not watch only worker CPU. A green CPU graph can hide hours of queue age.

Signal	Why it matters
Queue depth + oldest runnable job age	User-visible time-to-run; beats CPU for cron SLO
Scheduler tick duration	Primary melt early warning
Lease expiry rate without success	Stuck or crashing workers
Retry rate + DLQ ingress	Poison or downstream outage
Cron materialization lag	now - next_run_at for due rows still pending
Attempt histogram by tenant	Noisy neighbor detection
Tick batch size vs enqueue rate	:00 herd sizing

Alert ideas: Oldest job age above SLO; DLQ rate spike; tick duration p99 high; duplicate execution detector (customer reports) correlated with lease expiry.

Security and abuse

Authenticate job submit APIs — anonymous enqueue is a DoS vector. Per-tenant quotas on create rate and concurrent RUNNING jobs. Validate payload size; store blobs in object storage, not unbounded DB rows. Sign callbacks (HMAC) so customers trust the scheduler. Do not log full payloads if they contain PII — reference ids only. Admin trigger and DLQ replay require elevated roles and audit log. Rate-limit manual replay from DLQ — replay storms have taken down production downstreams.

Cost awareness

Every duplicate webhook can waste downstream money (SMS, payment API fees, support time). Every stuck retry loop burns worker CPU and customer goodwill. Execution history at TB/year is real storage cost — TTL successes to cold tier, retain failures longer. External queue per-message fees add up at millions/day — DB-as-queue saves money until tick contention forces the move. Jitter is cheap engineering that avoids :00 over-provisioning. DLQ replay without gates is an expensive incident multiplier.

Production scenes

Textbooks teach boxes and arrows. On-call teaches this: the graph can look fine while jobs are late or duplicated.

Scene 1: The :00 cron comb — queue age, not CPU, goes red

The moment: Every hour on the hour, p99 time-to-first-execution spikes. Worker CPU graphs show a neat comb. Payment reconciliation waits behind a marketing email batch that materialized 50K messages in ten seconds.

The trap: Millions of crons share 0 * * * * with no jitter. Materialization returns a huge batch at once. Autoscaling lags the spike. Teams alert on CPU — CPU eventually catches up while queue age already breached SLO.

What to build: Jitter within a window (for example ±5 minutes). Separate queues by tier. Pre-scale workers before known peaks. Alert on oldest runnable job age before CPU.

Scene 2: Stuck RUNNING — the worker died but the row still says busy

The moment: Support reports jobs "stuck for hours." Dashboard shows thousands of RUNNING. No worker logs match those ids.

The trap: No lease or lease TTL longer than job runtime without heartbeat. Worker died; row never returned to claimable state. Or heartbeat bug stopped extending while process hung.

What to build: Mandatory locked_until with heartbeat extend. Reaper job marks LEASE_EXPIRED and requeues when locked_until < now(). Metrics on RUNNING rows past lease. Idempotent handlers so reaper-induced retry is safe.

Scene 3: Double dispatch on leader failover — midnight billing twice

The moment: Scheduler leader died at 00:00:02. New leader started. Billing ran twice. Duplicate charges.

The trap: Two leaders both materialized the same cron occurrence — no unique (cron_id, occurrence_ts) on enqueue. Or stale leader without fencing inserted after new leader.

What to build: Dedupe constraint on materialization. Fencing token on leader epoch. Deterministic job_id per occurrence for extra safety. Failover drills on cron boundary.

Scene 4: DLQ flood — replay without a gate

The moment: Downstream API returned 503 for twenty minutes. Every job retried, hit max attempts, landed in DLQ. Engineer clicked "replay all." Downstream died again. DLQ refilled.

The trap: Treating DLQ as a free retry button without rate limit, canary replay, or root-cause fix.

What to build: DLQ alert with sample errors. Replay N at a time with circuit breaker. Fix downstream before bulk replay. Separate poison (bad payload) from transient (503) in DLQ reason codes.

[ Enqueue rate >> process rate ]
        → ready queue depth ↑
        → age SLO red before CPU red
        → scale workers OR shed load OR delay cron materialization

In the interview: Pick one scene. Tell what users felt, which metric lied, what you would ship — idempotency keys, jitter, lease reaper, or gated DLQ replay.

Bottlenecks and tradeoffs

Simplicity vs scale (DB-as-queue vs external queue)

The tension — DB polling has fewer moving parts until the tick becomes the bottleneck.

What breaks — Primary melts on next_run_at scans and SKIP LOCKED contention; dispatch lag grows.

What teams do — Start DB-as-queue; move ready work to external queue when tick or claim contention hurts; outbox for consistency.

Say in the interview — Name when you add Kafka/SQS — not on day one by default.

Strong timing vs queue latency

The tension — "Run exactly at T" conflicts with queue-based execution.

What breaks — Jobs arrive seconds or minutes late during backlog.

What teams do — SLO as "within N seconds of T"; jitter cron to avoid herds; delay materialization when backlogged.

Say in the interview — Honest latency bar beats pretending nanosecond cron on a queue.

At-least-once vs exactly-once rhetoric

The tension — Product language wants "once"; physics gives retries.

What breaks — Duplicate side effects when handlers are not idempotent.

What teams do — Idempotency keys, dedupe tables, outbox on customer; never promise exactly-once on the wire.

Say in the interview — "Delivery attempts are at-least-once; effects are idempotent."

History retention cost

The tension — Detailed execution logs help debug but cost money at TB/year.

What breaks — Storage bills climb; queries slow.

What teams do — TTL archive successes; retain failures longer; partition by time.

If you remember one thing: Every tradeoff ties back to tick cost, queue age, or duplicate effects.

Interview tips

You have the full design in mind now. These back-and-forths test whether you have operated a scheduler — not only drawn one.

At-least-once vs exactly-once

You might say: "We guarantee exactly-once execution."

Interview Push: "Worker crashes after your callback returned 200 but before ack — what happens?"

Land here: The wire is at-least-once. Exactly-once effects come from idempotent handlers, dedupe tables keyed by job_id, or external systems that treat duplicate POSTs as success. Say that before they push on crashes — leases limit duplicate attempts; idempotency limits duplicate effects.

Visibility timeout without heartbeat

You might say: "Workers pull from SQS and delete the message when done."

Interview Push: "Job takes ten minutes; visibility is thirty seconds — do you lose work or run twice?"

Land here: Without heartbeat extend, the message becomes visible again while W1 still runs — W2 may duplicate. Use a lease on the job row (locked_until, heartbeat). When lease expires, another worker may claim — duplicate unless the handler is safe. That is visibility timeout done correctly.

Cron on leader failover

You might say: "One cron daemon on a VM."

Interview Push: "Leader dies at :00 and the backup starts — does midnight billing fire twice?"

Land here: Dedupe key (cron_id, occurrence_ts) unique on materialization, or fencing tokens so stale leaders cannot enqueue. Mention deterministic job id per scheduled minute. Failover drills on cron boundaries are production reality.

Clock skew across scheduler nodes

You might say: "Run when now() on the host passes T."

Interview Push: "NTP drift across fifty scheduler nodes — jobs early or late?"

Land here: Trust stored next_run_at evaluated in DB time or a single leader clock — not each host's wall clock. Admit schedules are "within N seconds," not nanosecond perfect.

Backfill after downtime

You might say: "Catch up everything that was due while we were down."

Interview Push: "Six hours offline — millions of jobs due at once?"

Land here: Rate-limit catch-up. Separate priority lanes for new work vs backfill. Spread with jitter so recovery does not recreate a :00 stampede.

Red flags to avoid: "Exactly-once HTTP callbacks," fire-and-forget without lease, cron at :00 without jitter, DLQ replay all with no gate.

After each push, close with one mechanism — lease, dedupe key, DLQ, jitter — not "we will scale workers."

What should stick

After reading this guide, you should be able to explain:

1. When vs who — Scheduler decides when; workers claim and run with leases and heartbeat.
2. At-least-once wire, idempotent effects — Retries are normal; business dedupe is mandatory.
3. Cron dedupe on failover — Unique (cron_id, occurrence_ts) or fencing — not two leaders firing midnight billing.
4. Leases for stuck work — Worker dies → lease expires → retry; poison → DLQ.
5. Measure queue age — Depth and oldest-message age beat CPU for user-visible cron pain.

Tell it in the room: "Jobs persist with next_run_at. A leader tick enqueues due work with dedupe. Workers claim with a lease and heartbeat, call the customer webhook, ack or retry with backoff. Cron uses jitter. Handlers must be idempotent because delivery is at-least-once."

Key Takeaways

• Clarify guarantee — At-least-once delivery; idempotent handlers for side effects.
• Async API — POST returns 202 after durable persist; execution is later.
• Lease + heartbeat — Survive worker crash and long jobs; reaper for stuck RUNNING.
• Cron jitter + dedupe — Avoid :00 herd and double dispatch on failover.
• Retry + DLQ — Exponential backoff, max attempts, gated replay.
• DB-as-queue vs external queue — Simple until tick hot; then ready queue + outbox.
• Observability — Queue age, tick duration, DLQ rate — not only CPU.
• Practice — Job Scheduler practice.

Related Topics

• Rate Limiter System Design — Admission control and fair-share patterns for enqueue APIs
• Notification Service System Design — Async delivery, retries, and idempotent sends after job completion
• Distributed Cache System Design — Hot metadata, coordination, and lease-like TTL patterns
• Job Scheduler practice workspace — Whiteboard this problem with interviewer prompts

Quick revision notes

Use this as a last look before a mock — not a substitute for the full guide.

Clarify first. Agree async 202, at-least-once vs idempotent effects, cron vs one-shot, long vs short jobs, priority needs, and what you park (full DAG, exactly-once rhetoric).

Draw two paths. Scheduler path: tick → due index → enqueue with dedupe + jitter. Worker path: claim with lease → heartbeat → callback → ack or backoff → DLQ.

Defend the dig. Bad: fire-and-forget. Great: lease + heartbeat + idempotency key on customer. Crash after 200 before ack is the canonical push.

Production story. Pick one: :00 herd (jitter + queue age), stuck RUNNING (lease reaper), double dispatch (dedupe key), DLQ flood (gated replay).

Close with metrics. Oldest runnable job age, tick p99, DLQ ingress — green CPU is not enough.