The Circuit Breaker That Didn't Break

This is a story about how a circuit breaker that was supposed to prevent failures actually made them worse. It's also about why understanding resilience patterns matters, and how we learned that configuration is as important as implementation.

Related reading on this site: For circuit breaker states, thresholds, and fallbacks in fundamentals form, see Circuit Breakers. For the full resilience interview walkthrough, use Circuit Breaker Pattern. When cache cold starts overload the origin, read The Cache Stampede That Took Down Our API. For queue backpressure as another cascade pattern, see The Message Queue Lag That Overwhelmed Our Order Processing.

---

Context

We had a microservices architecture with a payment service that processed transactions. The payment service called an external payment gateway API. We implemented a circuit breaker to prevent cascading failures if the gateway was down.

Original Architecture:

Technology Choices:

• Payment Service: Node.js microservice
• Circuit Breaker: Opossum library (Node.js)
• External Gateway: Third-party payment API
• Database: PostgreSQL for transaction records

Assumptions Made:

• Circuit breaker would prevent cascading failures
• Configuration was correct (we copied from documentation)
• Circuit breaker would open quickly when gateway failed

---

The Incident

---

Symptoms

What We Saw:

• Payment Service: Stopped accepting new requests—health check passed but thread pool saturated
• Error Rate: Payment path jumped from 0.1% to 100% in ~8 minutes
• Response Time: p50 went from 120ms to timeout (5s); gateway p99 at 6–8s while breaker stayed closed
• Thread Pool: 100/100 worker threads blocked on callGateway()—no capacity for new checkout requests
• Checkout Cascade: API Gateway 502/504 rose to 35% as payment service stopped dequeuing
• User Impact: ~50K failed checkout attempts in 15 minutes; estimated six-figure revenue at risk

How We Detected It:

• Alert: payment service request success rate < 50% for 2 minutes
• Dashboard showed external gateway status page reporting degraded performance
• On-call correlated thread dump (all blocked on HTTP client) with breaker state still CLOSED

Monitoring Gaps:

• No alert on circuit breaker state transitions (open/half-open/closed)
• No thread pool utilization alert separate from CPU/memory
• No dashboard overlay: gateway latency vs breaker error window size
• No chaos test proving breaker opens under slow-dependency scenarios

---

Root Cause Analysis

Primary Cause: Misconfigured circuit breaker threshold—too small a rolling window for 100+ concurrent calls, so the breaker never opened while threads starved.

How a closed circuit amplified gateway slowness:

The breaker was meant to fail fast when the external payment gateway degraded. Instead, errorThresholdPercentage: 50 on a 10-request rolling window could not represent 100 in-flight calls. Timeouts stacked in worker threads; the service stopped dequeuing new work even though CPU looked idle. Checkout did not get graceful degradation—it got total stall because nothing told callers to stop waiting.

gateway slow (5s+) → 100 threads blocked → no workers left → payment API wedged → checkout 100% fail → breaker still CLOSED

The Bug:

// BAD CONFIGURATION
const circuitBreaker = new CircuitBreaker(callGateway, {
  timeout: 5000, // 5 second timeout
  errorThresholdPercentage: 50, // Open after 50% errors
  resetTimeout: 30000, // Reset after 30 seconds
  // PROBLEM: Only 2 requests needed to trigger 50% threshold!
  // If 1 request fails, next request opens circuit
  // But we have 100 concurrent requests...
});

What Happened:

1. External gateway started experiencing high latency (5+ seconds)
2. Payment service had 100 concurrent requests to gateway
3. Circuit breaker error threshold was 50% (too low for high concurrency)
4. First 50 requests timed out, circuit should have opened
5. But circuit breaker only tracked last 10 requests (default window)
6. With 100 concurrent requests, circuit breaker couldn't track properly
7. Circuit breaker never opened, all requests kept trying
8. All payment service threads blocked waiting for gateway
9. Payment service stopped processing new requests
10. Cascading failure propagated to API Gateway

Why It Was So Bad:

• Wrong threshold: 50% error threshold too low for high concurrency
• Small window: Only tracking last 10 requests, not all concurrent requests
• No thread pool limits: All threads blocked, no capacity for new requests
• No fallback: No graceful degradation when circuit should open

Contributing Factors:

• Configuration copied from documentation without understanding
• No testing of circuit breaker under failure scenarios
• No monitoring of circuit breaker state
• High concurrency (100+ requests) not considered in configuration

---

Fix & Mitigation

Immediate Fix:

// FIXED CONFIGURATION
const circuitBreaker = new CircuitBreaker(callGateway, {
  timeout: 2000, // 2 second timeout (fail fast)
  errorThresholdPercentage: 80, // Open after 80% errors (higher threshold)
  resetTimeout: 60000, // Reset after 60 seconds
  volumeThreshold: 20, // Need at least 20 requests before opening
  rollingCountTimeout: 10000, // 10 second window
  rollingCountBuckets: 10, // 10 buckets for better tracking
  // Add fallback
  fallback: () => {
    return { error: 'Payment service temporarily unavailable' };
  }
});

Long-Term Improvements:

Strategy	What it does	Best when
Tuned breaker (volume + rolling window)	Opens only after enough samples; window matches concurrency	High fan-out services; external SaaS dependencies
Fallback / degraded mode	Return cached quote, queue for retry, or "pay later" UX	Revenue paths where hard fail is worse than delay
Bulkhead / thread pool caps	Isolate gateway calls from core API threads	Prevent one slow dependency from wedging the service
Timeouts + retry with jitter	Fail fast per call; bounded retries for transients	Pair with breaker—retries for blips, breaker for sustained failure (Circuit Breakers)

1. Circuit Breaker Configuration:

   • Increased error threshold to 80% (more appropriate for high concurrency)
   • Added volume threshold (need minimum requests before opening)
   • Added proper rolling window configuration
   • Added fallback mechanism

2. Thread Pool Management:

   • Added thread pool size limits
   • Added thread pool monitoring
   • Added alert for thread pool exhaustion

3. Monitoring & Alerting:

   • Added circuit breaker state monitoring
   • Added alert when circuit opens
   • Added external service latency monitoring
   • Added fallback usage tracking

4. Process Improvements:

   • Required circuit breaker testing in staging
   • Added resilience testing to CI/CD
   • Created runbook for circuit breaker incidents
   • Added configuration review process

---

Architecture After Fix

Key Changes:

• Proper circuit breaker configuration
• Thread pool limits
• Fallback mechanism
• Circuit state monitoring

---

Key Lessons

1. Circuit breaker configuration matters: Wrong thresholds can make failures worse. Understand your concurrency patterns before configuring.

2. Test resilience patterns: Don't just implement circuit breakers—test them under failure scenarios. Staging should simulate production failures.

3. Monitor circuit state: Know when your circuit breaker opens and closes. Alert on state changes.

4. Have fallbacks: When circuit opens, have a graceful degradation strategy. Don't just fail requests.

5. Consider concurrency: High concurrency changes how circuit breakers behave. Configure accordingly.

---

Interview Takeaways

Common Questions:

• "How do circuit breakers work?"
• "How do you configure circuit breakers?"
• "What happens when a circuit breaker opens?"

What Interviewers Are Looking For:

• Understanding of circuit breaker pattern
• Knowledge of configuration parameters
• Experience with resilience patterns
• Awareness of failure isolation strategies

---

What a Senior Engineer Would Do Differently

From the Start:

1. Understand configuration: Don't copy configs blindly. Understand what each parameter does.
2. Test under failure: Test circuit breakers with actual failures, not just happy paths.
3. Monitor circuit state: Track when circuits open/close and alert on state changes.
4. Add fallbacks: Always have a graceful degradation strategy.
5. Consider concurrency: Configure circuit breakers for your actual concurrency patterns.

The Real Lesson: Resilience patterns are powerful, but misconfiguration can make failures worse. Test, monitor, and understand your configuration.

---

How I'd answer in interviews

"Our payment service wrapped an external gateway with a circuit breaker, but we copied defaults: fifty percent errors on a ten-request window while running a hundred concurrent checkout calls. Gateway latency hit five seconds, threads blocked, the service stopped dequeuing, and checkout went to one hundred percent failure—while the breaker stayed closed. I'd configure volume threshold and rolling windows for real concurrency, add a fallback response, cap the gateway thread pool as a bulkhead, alert on breaker state changes, and chaos-test slow dependencies in staging before every resilience change ships."

---

Related reading on this site

• Circuit Breakers — states, thresholds, half-open probes, and fallback design in tutorial form.
• Circuit Breaker Pattern — full interview walkthrough for resilient microservices.
• The Cache Stampede That Took Down Our API — another cascade when a protection layer fails to trip.
• The Message Queue Lag That Overwhelmed Our Order Processing — when downstream slowness propagates upstream without a fail-fast boundary.
• Monitoring & Observability — alert on breaker state, pool saturation, and dependency latency together.

---

FAQs

Q: How do circuit breakers work?

A: Circuit breakers monitor request success/failure rates. When error rate exceeds a threshold, the circuit "opens" and stops sending requests to the failing service. After a timeout, it "half-opens" to test if the service recovered, then closes if successful.

Q: How do you configure circuit breaker thresholds?

A: Error threshold should be high enough (70-80%) to avoid false positives but low enough to catch real failures. Volume threshold ensures you have enough data before opening. Consider your concurrency patterns when configuring.

Q: What happens when a circuit breaker opens?

A: The circuit stops sending requests to the failing service and immediately returns an error or calls a fallback function. This prevents cascading failures and gives the failing service time to recover.

Q: Should you always use circuit breakers?

A: Circuit breakers are useful for external service calls, but not always necessary for internal services. Use them when you want to prevent cascading failures and have a fallback strategy.

Q: How do you test circuit breakers?

A: Simulate failures in staging: slow down external services, return errors, or time out requests. Verify that circuits open correctly and fallbacks work. Test recovery scenarios.

Q: What's the difference between circuit breaker and retry?

A: Retries attempt the same request multiple times on transient failures—they can amplify load if the dependency is down. Circuit breakers stop sending traffic after sustained failures and fail fast or call a fallback. Use retries for blips; use breakers (with bulkheads) when the dependency is persistently unhealthy.

Q: What's the difference between a circuit breaker and a bulkhead?

A: A circuit breaker stops calls to a failing dependency based on error rates. A bulkhead limits concurrent calls (thread pool, semaphore) so one slow dependency cannot exhaust all workers. We needed both: the breaker should have opened, and a bulkhead would have left threads for health checks and fallbacks.

Q: How do you choose circuit breaker timeouts?

A: Timeout should be shorter than your request timeout (fail fast). Error threshold should reflect your error tolerance. Reset timeout should give the service enough time to recover. Test and adjust based on actual behavior.