← Back to distributed systems

Distributed Systems Topic

Replication Lag: Concepts, Trade-offs & Failure Modes

Learn about replication lag in distributed databases and how to handle it.

Intermediate8 min read

Replication lag is the delay between when data is written to the primary and when it's available on replicas.


What is Replication Lag?

Definition: Time difference between primary write and replica update.

Causes:

  • Network latency
  • Replica processing time
  • High write load
  • Network congestion

Impact

Stale reads: Reading from replica may return old data

Inconsistency: Different replicas may have different data

User experience: Users may not see their own writes immediately


Measuring Lag

TypeScript
Python
Java
1

Handling Lag

Read Your Writes

Route user's reads to primary after their writes.

TypeScript
Python
Java
1

Monotonic Reads

Ensure user always sees newer data (not older).

TypeScript
Python
Java
1

Examples

Database Replication

TypeScript
Python
Java
1

Common Pitfalls

  • Ignoring lag: Users see stale data. Fix: Monitor and handle lag
  • Not routing critical reads: Important reads go to stale replica. Fix: Route to primary
  • No lag monitoring: Don't know when lag is high. Fix: Monitor continuously
  • Assuming zero lag: Replicas always have some lag. Fix: Design for lag

Interview Questions

Beginner

Q: What is replication lag and why does it matter?

A: Replication lag is the delay between when data is written to the primary database and when it's available on replicas.

Why it matters:

  • Stale reads: Reading from replica may return old data
  • Inconsistency: Users may not see their own writes
  • User experience: Confusing when data doesn't appear immediately

Example: User updates profile, but when they refresh, old data appears (read from replica that hasn't updated yet).


Intermediate

Q: How do you handle replication lag in a read replica setup?

A:

Strategies:

  1. Read your writes: Route user's reads to primary after their writes
  2. Monotonic reads: Ensure user always sees newer data
  3. Lag-aware routing: Route critical reads to primary if lag is high
  4. Wait for replication: Wait for replica to catch up before reading

Implementation:

  • Track user's last write time
  • If recent write, read from primary
  • Otherwise, read from replica
  • Monitor lag and adjust routing

Senior

Q: Design a system that handles replication lag for a social media platform. Users post content that must be visible to followers. How do you ensure consistency while maintaining performance?

A:

Design:

TypeScript
Python
Java
1

Optimizations:

  • Caching: Cache recent posts to reduce replica load
  • Fan-out: Write to follower timelines on post (not on read)
  • Eventual consistency: Accept that followers may see posts slightly later

Key Takeaways

Replication lag is inevitable: Network and processing delays cause lag

Monitor lag: Continuously measure and alert on high lag

Read-your-writes: Route user's reads to primary after their writes

Lag-aware routing: Adjust read routing based on lag

Trade-offs: Consistency vs performance (primary vs replica reads)

Design for lag: Assume replicas are always slightly behind

Keep exploring

Partial failure and consistency show up together in real systems. Continue with the next hub topic that stresses the same idea.