Fault Tolerance: Concepts, Trade-offs & Failure Modes

Fault tolerance is the ability of a system to continue operating correctly even when some components fail.

Failure Modes

Crash failures: Node stops responding (most common)

Byzantine failures: Node behaves arbitrarily (malicious or buggy)

Omission failures: Node fails to send/receive messages

Timing failures: Node responds too slowly or too fast

Fault Tolerance Techniques

Redundancy

Replication: Multiple copies of data/services

Active-active: All replicas handle requests

Active-passive: Standby replicas take over on failure

1 

Circuit Breaker

Prevents cascading failures by stopping requests to failing services.

1 

Graceful Degradation

System continues with reduced functionality.

1 

Examples

Database Replication

1 

Common Pitfalls

• Single point of failure: One component brings down system. Fix: Add redundancy
• No failure detection: Don't know when components fail. Fix: Health checks, timeouts
• Cascading failures: One failure causes others. Fix: Circuit breakers, rate limiting
• Not testing failures: System untested under failure. Fix: Chaos engineering
• Ignoring partial failures: System fails completely. Fix: Graceful degradation

Interview Questions

Beginner

Q: What is fault tolerance and why is it important?

A: Fault tolerance is the ability of a system to continue operating correctly even when components fail.

Why important:

• High availability: System stays up even with failures
• Reliability: Users can depend on the system
• Resilience: System recovers from failures
• User experience: Failures don't disrupt users

Example: If one database server fails, system should continue using other servers.

Intermediate

Q: How do you design a fault-tolerant distributed system?

A:

Key techniques:

1. Redundancy: Multiple copies of critical components
2. Failure detection: Health checks, timeouts, monitoring
3. Automatic recovery: Failover, restart failed components
4. Isolation: Failures don't cascade
5. Graceful degradation: Continue with reduced functionality

Example design:

• Load balancer with multiple backend servers
• Database replication (primary + replicas)
• Circuit breakers to prevent cascading failures
• Health checks to detect failures quickly
• Automatic failover when primary fails

Senior

Q: Design a fault-tolerant microservices architecture. How do you handle service failures, database failures, and network partitions?

A:

Architecture:

• Service redundancy: Multiple instances of each service
• Database replication: Primary + replicas
• Circuit breakers: Prevent cascading failures
• Health checks: Detect failures quickly
• Service mesh: Handle communication resilience

Design:

1 

Failure Handling:

1. Service failures: Circuit breaker, retry with backoff, failover to backup
2. Database failures: Read from replicas, promote replica to primary
3. Network partitions: Continue in degraded mode, sync when partition heals