Heartbeats & Health Checks

Heartbeats and health checks are mechanisms to detect node failures and monitor system health in distributed systems.

Heartbeats

Periodic messages sent to indicate a node is alive.

Implementation

1 

Health Checks

Endpoints that report node health status.

Liveness vs Readiness

Liveness: Is the process running?

Readiness: Can the process handle requests?

1 

Examples

Kubernetes Health Checks

1 

Leader Election with Heartbeats

1 

Common Pitfalls

• Too frequent heartbeats: Network overhead. Fix: Balance frequency with detection time
• Not handling network delays: False positives. Fix: Use timeout > 3x interval
• Single health check: May miss issues. Fix: Check multiple components
• Not failing fast: Unhealthy nodes continue serving. Fix: Remove from load balancer
• No graceful shutdown: Health check fails during shutdown. Fix: Implement graceful shutdown

Interview Questions

Beginner

Q: What are heartbeats and health checks used for?

A:

Heartbeats: Periodic messages to indicate a node is alive. Used for failure detection.

Health checks: Endpoints that report node health. Used to determine if node can handle requests.

Purpose:

• Failure detection: Know when nodes fail
• Load balancing: Route traffic only to healthy nodes
• Auto-recovery: Restart or replace failed nodes
• Monitoring: Track system health

Intermediate

Q: How do you implement health checks for a microservice? What's the difference between liveness and readiness?

A:

Liveness: Is the process running?

• Simple check: Process is alive
• Use: Kubernetes will restart if fails

Readiness: Can the process handle requests?

• Comprehensive check: Database, cache, dependencies all working
• Use: Load balancer routes traffic only if ready

Implementation:

1 

Senior

Q: Design a failure detection system for a distributed system with 1000+ nodes. How do you detect failures quickly while minimizing network overhead?

A:

Design:

1 

Optimizations:

• Hierarchical: Reduce network messages (nodes → cluster → region)
• Gossip: O(log n) messages instead of O(n)
• Adaptive: Adjust frequency based on failure rate
• Sampling: Check subset of nodes, rotate