Job Scheduler System Design

Introduction

Midnight UTC hits. A million cron jobs wake at once. Billing runs. Then billing runs again—because the scheduler leader failed over and the new leader did not know the first batch already fired. Support finds duplicate charges. Marketing emails went out ten times.

That is what users feel when “run at T” is treated like a single-server cron tab.

Interviewers care about three failures: duplicate execution, lost jobs, and stuck jobs after a worker dies halfway. The product wants billing once and emails once—not a lecture on TCP.

Your design should center a state machine, leases, and idempotency. Weak answers draw cron on one VM. Strong answers separate when (time index + leader or shard) from who runs (workers + claim).

If you remember one thing: Scheduling is durable timers at scale—separate “fire at T” from “execute safely.”

How to approach

Talk like you are walking one job from “scheduled” to “done”—not listing queue brands.

Define execution guarantee first: almost always at-least-once on the wire, effectively-once for business side effects via idempotency keys or external dedupe. Then sketch tables (jobs, executions, optional deps). Walk one due job → claim → run → ack or retry. Only then cron and DAG depth.

In the room: “I’ll pick at-least-once delivery with idempotent handlers, then walk one job through claim and lease before cron and dependencies.”

If you remember one thing: Guarantee delivery attempts separately from business effects once.

Interview tips

Five exchanges that show whether you have operated a scheduler—not just drawn one.

At-least-once vs exactly-once

You: “We guarantee exactly-once execution.”

They ask: “Worker crashes after your callback returned 200 but before ack—what happens?”

Land here: The wire is at-least-once. Exactly-once effects come from idempotent handlers, dedupe tables, or external systems keyed by job_id. Say that out loud.

Visibility timeout and leases

You: “Workers pull from SQS and delete the message when done.”

They ask: “Job takes ten minutes; visibility is thirty seconds—do you lose work or run twice?”

Land here: Use a lease (locked_until, heartbeat extend). When lease expires, another worker may claim—duplicate unless the handler is safe. That is the visibility-timeout pattern.

Cron on leader failover

You: “One cron daemon on a VM.”

They ask: “Leader dies at :00 and the backup starts—does midnight billing fire twice?”

Land here: Single leader tick with dedupe key (cron_id, occurrence_ts) unique, or fencing tokens so stale leaders cannot enqueue. Mention deterministic job id per scheduled minute.

Clock skew

You: “Run when now() on the host passes T.”

They ask: “NTP drift across fifty scheduler nodes—jobs early or late?”

Land here: Trust stored next_run_at in DB time, not each host’s wall clock. Admit schedules are “within N seconds,” not nanosecond perfect.

Backfill after downtime

You: “Catch up everything that was due while we were down.”

They ask: “Six hours offline—millions of jobs due at once?”

Land here: Rate-limit catch-up. Separate priority lanes for new work vs backfill. Spread cron with jitter so :00 is not a coordinated stampede.

If you remember one thing: After each push, name one mechanism—lease, dedupe key, DLQ, jitter—not “we’ll scale workers.”

Capacity estimation

Implications: The scheduler tick path must be O(due batch) not O(all jobs). Executions table grows fast—partition by time.

If you remember one thing: Index for next due, not full scans—and plan for execution history growth.

High-level architecture

API creates/updates jobs in durable store (SQL or NoSQL with strong writes). Scheduler process(es)—often leader-elected—run a tick loop: query jobs where next_run_at <= now (with index), enqueue to ready queue (Kafka/SQS/Rabbit) or direct claim if DB-based queue. Workers pull or receive push, claim with lease (locked_until, worker_id), execute user payload (HTTP callback, container, script), ack or fail with backoff. History rows append attempt records.

Who owns what:

• Scheduler leader — Advances cron occurrences, hydrates due jobs into ready queue; must not double-insert same occurrence—use unique constraints or dedupe keys.
• Ready queue — Buffers runnable jobs; decouples tick rate from worker CPU; enables backpressure visibility (depth, age).
• Workers — Stateless; horizontal scale; respect lease; extend lease for long jobs (heartbeat).
• Execution store — Audit trail, status, error blobs; may be same DB or analytics sink.

[ Clients / services ]
        |
        | POST /jobs (sync: persist only)
        v
[ Job API ] ──write──► [ Job metadata DB ]
        |                    ^
        |                    | next_run_at index
[ Scheduler Leader ] --------┘
        |
        | tick: due rows → enqueue (async)
        v
[ Ready queue: Kafka / SQS ] ──► [ Worker pool ]
                                      |
                                      | lease + execute
                                      v
                              [ User workload + DLQ on poison ]

  Cron / deps: scheduler expands next occurrence → same pipeline

In the room: Say async boundary clearly: HTTP returns 202 after persist—execution is later. Then describe lease and retry.

If you remember one thing: API persists; scheduler enqueues; workers claim with lease.

Core design approaches

Database-as-queue vs external queue

DB polling with SKIP LOCKED (Postgres) can work at moderate scale—simpler ops, harder to scale tick. External queue adds moving parts but smooths bursts and standardizes retry/DLQ.

Single leader vs sharded schedulers

Single leader tick is easy to reason about; shard by hash(tenant_id) when one leader cannot scan enough rows—each shard owns subset of crons.

Dependency execution

Level-order: run ready jobs with no pending parents; on parent complete, unblock children—event-driven wake is better than polling deps every second.

If you remember one thing: DB-as-queue is simpler until the tick becomes hot; external queue adds ops but smooths bursts.

Detailed design

Write path (schedule job)

1. Client submits job with payload, schedule, idempotency key.
2. API inserts row PENDING, computes first next_run_at, returns job_id.
3. For DAG jobs, dependency rows link to parents—all parents done before enqueue.

Read path (worker claim)

1. Worker receives message or polls DB with locked_until < now().
2. UPDATE … WHERE id=? AND lease expired → set RUNNING, locked_until=now+lease.
3. Execute; on success COMPLETE; on failure retry with backoff or DLQ.

Cron evaluation

Leader runs every second (or coarser): for each cron row, materialize next occurrence if due—insert into ready queue with dedupe (cron_id, occurrence_ts) unique.

If you remember one thing: Write path persists; worker path claims with lease; cron materializes with dedupe.

Key challenges

• Double dispatch: Two schedulers both enqueue same cron tick—unique keys and fencing mitigate.
• Lost work: Crash before persist—ack only after durable enqueue; outbox pattern for API → queue atomicity.
• Stuck RUNNING: Worker dies—lease expires; job retries—at-least-once visible to business (idempotency).
• Thundering herd at :00: Many crons fire same minute—jitter spreads load.
• Ordering vs fairness: Strict priority can starve low priority—aging or weighted fair queueing.

If you remember one thing: Double dispatch, lost work, stuck RUNNING, and :00 herds are the named tensions—not “pick Kafka.”

Scaling the system

• Workers: Horizontal; auto-scale on queue lag (oldest message age).
• Scheduler: Shard tick by key range; read replicas for reporting—not for claim path without careful locking.
• DB: Partition jobs table by tenant or time; hot tenants isolated.
• Global schedules: Region-local schedulers for data residency; cross-region cron is rare—clarify authority.

If you remember one thing: Scale workers on queue age; shard scheduler tick before one leader melts the primary.

Failure handling

Degraded UX: Jobs late; outage is lost jobs or unbounded retry—unacceptable—durability is the product.

If you remember one thing: Worker death → lease expiry → retry is normal; poison goes to DLQ, not infinite loops.

API design

Headers: Idempotency-Key on create; X-Request-ID for tracing.

Webhook callback (worker → user system):

Flow diagram:

POST /v1/jobs ──► 202 Accepted + job_id
                      │
Scheduler tick ──► ready queue ──► Worker ──► POST https://customer/callback
                                                  │
                                            200 ─► ack
                                            5xx ─► retry w/ backoff

If you remember one thing: Create returns 202; execution is async with idempotency keys and signed callbacks.

Production angles

[ Enqueue rate >> process rate ]
        → ready queue depth ↑
        → age SLO red before CPU red
        → scale workers OR shed load OR delay cron materialization

Bottlenecks and tradeoffs

Simplicity vs scale

The tension — DB polling scheduler has fewer moving parts until the tick becomes the bottleneck.

What breaks — Primary melts on next_run_at scans; dispatch lag grows.

What teams do — Start DB-as-queue; move ready work to external queue when tick or claim contention hurts.

Say in the interview — Name when you would add Kafka/SQS—not on day one by default.

Strong timing vs queue latency

The tension — “Run exactly at T” conflicts with queue-based execution.

What breaks — Jobs arrive seconds late during backlog.

What teams do — SLO as “within N seconds of T”; jitter cron to avoid herds.

Say in the interview — Honest latency bar beats pretending nanosecond cron on a queue.

History retention cost

The tension — Detailed execution logs help debug but cost money at TB/year.

What breaks — Storage bills climb; queries slow.

What teams do — TTL archive to cold storage; retain all failures longer than successes.

Say in the interview — Partition executions by time; sample success detail if needed.

If you remember one thing: Every tradeoff ties back to tick cost, queue age, or duplicate effects.

What should stick

You do not need to memorize every box. After this guide, you should be able to:

1. When vs who — Scheduler decides when; workers claim and run with leases.
2. At-least-once wire, idempotent effects — Retries are normal; business dedupe is mandatory.
3. Cron dedupe on failover — Unique (cron_id, occurrence_ts) or fencing—not two leaders firing midnight billing.
4. Leases for stuck work — Worker dies → lease expires → retry; poison → DLQ.
5. Measure queue age — Depth and oldest-message age beat CPU for user-visible pain.

Tell it in the room: “Jobs persist with next_run_at. A leader tick enqueues due work. Workers claim with a lease, call the customer webhook, ack or retry with backoff. Cron materializes each occurrence once. Handlers must be idempotent because delivery is at-least-once.”