The On-Call Fatigue That Led to a Critical Bug

This is a story about how on-call fatigue led to a mistake that made an outage worse. It's also about why taking care of your team matters as much as taking care of your systems, and how we learned to prevent on-call burnout.

Related reading on this site: For connection pool sizing and exhaustion signals, see Connection Pooling and The Cache Stampede That Took Down Our API. For metrics, alerts, and debugging under pressure, read Monitoring & Observability. When human error amplifies a technical failure, compare The Circuit Breaker That Didn't Break—misconfiguration plus stress extends outages.

---

Context

We were running a small engineering team (5 engineers) with a shared on-call rotation. Each engineer was on-call for one week, then off for four weeks. The system had occasional incidents, but nothing critical.

Original On-Call Setup:

• Rotation: 5 engineers, 1 week on-call each
• Coverage: 24/7 on-call coverage
• Incidents: ~2-3 incidents per month
• Response Time: < 15 minutes SLA

Assumptions Made:

• 1 week on-call every 5 weeks was sustainable
• Engineers could handle occasional incidents
• On-call wasn't too stressful

---

The Incident

---

Symptoms

What We Saw:

• Initial Issue: PostgreSQL connection pool exhausted (active_connections / max_connections at 100%)
• Wrong Remediation: On-call engineer restarted the primary database at 2:10 AM
• Extended Outage: 15-minute pool-sizing fix became a 2-hour total outage (restart + recovery + re-triage)
• Error Rate: API 5xx jumped from <0.1% to ~40% during restart; stayed elevated until 3:00 AM
• Team Impact: New engineer on first solo rotation; guilt, sleep debt, morale dip for two weeks
• User Impact: Full checkout downtime for ~2 hours vs expected ~15 minutes

How We Detected It:

• PagerDuty alert: db_connection_pool_utilization > 95% for 3 minutes
• Engineer E acknowledged within 5 minutes but acted without runbook
• Senior engineer noticed SLO breach still open 30 minutes later and joined the bridge

Monitoring Gaps:

• No alert on time-to-escalate when primary on-call is stuck
• No runbook-linked dashboard for pool exhaustion (only a raw metric)
• No on-call fatigue or page-frequency tracking
• No post-incident support workflow—debrief happened a week late

---

Root Cause Analysis

Primary Cause: On-call fatigue and lack of incident response guidance turned a recoverable pool exhaustion into a destructive restart.

How fatigue extended a 15-minute fix into two hours:

The technical failure was bounded: connection pool at max, queries waiting, API timing out. The correct fix was increase pool size (or kill long-running queries) and trace the leak—roughly 15 minutes once someone familiar with the schema joined.

The human failure chain started earlier: a new engineer, first solo week, paged at 2:05 AM after three heavy on-call weeks across the team. Without a runbook, "database connection pool exhausted" mapped to "restart the database"—a plausible-sounding action that dropped all connections, forced crash recovery, and reset the symptom without fixing pool sizing. After restart, the pool exhausted again; total time on bridge: 2 hours.

pool exhausted → tired engineer → restart DB → 15m downtime → pool exhausts again → senior joins → resize pool → recovered

What Happened:

1. Engineer E was on-call for first time, new to team
2. Paged at 2:00 AM (woken up from sleep)
3. Tired, stressed, and unfamiliar with system
4. Saw database connection pool error
5. Made quick decision: restart database (seemed like reasonable fix)
6. Database restart caused 15-minute downtime
7. Service still down after restart (original issue not fixed)
8. Senior engineer called, identified correct fix
9. Total outage: 2 hours (should have been 15 minutes)

Why It Was So Bad:

• On-call fatigue: Engineer tired, stressed, not thinking clearly
• Lack of guidance: No runbook or incident response procedures
• Wrong decision: Restarting database made outage worse
• No escalation: Engineer didn't know when to escalate
• Team impact: Engineer felt guilty, team morale affected

Contributing Factors:

• New engineer on-call without proper training
• No incident response runbooks
• No escalation procedures
• On-call rotation too frequent for small team
• No post-incident support or debrief

---

Fix & Mitigation

Immediate Fix:

1. Corrected the issue: Increased database connection pool size
2. Service recovered: Back online after 2 hours
3. Post-incident review: Identified on-call fatigue as root cause

Long-Term Improvements:

Strategy	What it does	Best when
Primary + secondary on-call	Backup engineer joins if no progress in N minutes	Small teams; nights and holidays
Runbooks with "do not" list	Step-by-step fixes plus explicit anti-patterns (no DB restart for pool alerts)	Common alerts; new rotations
Escalation timers	Auto-page secondary if severity-1 open > 15 min	Prevent single-engineer heroics
Rotation spacing + compensation	Longer off-call windows, PTO credit, blameless debriefs	Sustaining 24/7 with <10 engineers

1. On-Call Practices:

   • Changed rotation to 1 week on-call every 6 weeks (more rest)
   • Added primary and secondary on-call (backup support)
   • Added on-call training for new engineers
   • Added on-call compensation (extra PTO days)

2. Incident Response:

   • Created runbooks for common incidents
   • Added escalation procedures (when to call senior engineer)
   • Added incident response training
   • Added "stop and think" checklist before major actions

3. Team Health:

   • Added post-incident debriefs (blameless)
   • Added on-call fatigue monitoring
   • Added mental health support
   • Added on-call rotation feedback process

4. Process Improvements:

   • Required runbook review before on-call
   • Added "test" environment for incident response practice
   • Created incident response playbook
   • Added on-call handoff procedures

---

Architecture After Fix

Key Changes:

• Primary and secondary on-call rotation
• Incident response runbooks
• Escalation procedures
• On-call training and support

---

Key Lessons

1. On-call fatigue is real: Tired engineers make mistakes. Ensure adequate rest and rotation.

2. Provide guidance: Runbooks and incident response procedures help engineers make better decisions under stress.

3. Know when to escalate: Engineers should know when to call for help, not try to fix everything alone.

4. Support your team: Post-incident debriefs, mental health support, and compensation matter.

5. Learn from incidents: Every incident is a learning opportunity. Review and improve processes.

---

Interview Takeaways

Common Questions:

• "How do you handle on-call?"
• "What is on-call fatigue?"
• "How do you prevent incidents?"

What Interviewers Are Looking For:

• Understanding of on-call practices
• Knowledge of incident response
• Awareness of team health
• Experience with process improvement

---

What a Senior Engineer Would Do Differently

From the Start:

1. Better on-call rotation: More rest between on-call weeks
2. Provide runbooks: Clear guidance for common incidents
3. Add escalation procedures: Know when to call for help
4. Train new engineers: On-call training before going on-call
5. Support team health: Mental health support, compensation, debriefs

The Real Lesson: Taking care of your team is as important as taking care of your systems. On-call fatigue leads to mistakes, which lead to longer outages.

---

How I'd answer in interviews

"We had a connection pool exhaustion alert at 2 AM. A tired, first-time on-call engineer restarted the database instead of resizing the pool or finding the leak—that added a 15-minute restart on top of the original problem and stretched a 15-minute fix into two hours. I'd fix the ops side with runbooks that say what not to do, primary-secondary rotation with auto-escalation timers, and blameless debriefs. On the systems side I'd alert on pool utilization early, link dashboards to runbooks, and invest in self-healing limits so on-call is not the only safety net."

---

Related reading on this site

• Monitoring & Observability — metrics, logs, traces, and alerts that shorten mean time to diagnose.
• Connection Pooling — sizing pools and recognizing exhaustion before restart seems tempting.
• The Cache Stampede That Took Down Our API — another story where connection pool saturation was the technical trigger.
• The Circuit Breaker That Didn't Break — when misconfiguration plus incident stress compounds outage duration.
• The N+1 Query Problem That Slowed Down Our API — a gradual pool/CPU drain that also needed better alerts and runbooks.

---

FAQs

Q: What is on-call fatigue?

A: On-call fatigue is exhaustion and stress from being on-call, especially when woken up frequently or dealing with stressful incidents. It leads to poor decision-making and mistakes.

Q: What's the difference between on-call fatigue and burnout?

A: On-call fatigue is acute—poor decisions after broken sleep or a heavy page week. Burnout is chronic disengagement from sustained overload. Fatigue caused our 2 AM restart mistake; untreated fatigue across rotations drives burnout. Both need rotation spacing, compensation, and blameless learning—not hero culture.

Q: How do you prevent on-call fatigue?

A: Ensure adequate rest between on-call weeks, provide runbooks and guidance, add backup on-call support, compensate on-call time, and provide mental health support.

Q: How do you handle on-call for a small team?

A: Use longer rotation cycles (more rest), add primary and secondary on-call, provide clear runbooks, and know when to escalate to external support.

Q: What should be in an incident response runbook?

A: Common incidents, step-by-step resolution procedures, escalation contacts, rollback procedures, and "stop and think" checklists before major actions.

Q: When should you escalate an incident?

A: Escalate when you're unsure of the fix, when the incident is getting worse, when you're tired or stressed, or when the incident exceeds SLA.

Q: How do you support engineers after incidents?

A: Hold blameless post-incident reviews, provide mental health support, recognize their effort, and learn from incidents to improve processes.

Q: Should on-call be compensated?

A: Yes. On-call is additional work and stress. Compensate with extra PTO, bonuses, or reduced regular work hours. This shows you value your team's time and health.