Partition Tolerance: Concepts, Trade-offs & Failure Modes

Partition tolerance is the ability of a distributed system to continue operating despite network partitions that split the system into isolated groups.

What is a Network Partition?

A network partition occurs when network failures cause nodes to be unable to communicate, splitting the system into disconnected groups.

Example: Data center A can't reach data center B, but both continue operating independently.

CAP Theorem

You can only guarantee 2 of 3:

• Consistency: All nodes see same data
• Availability: System responds to requests
• Partition tolerance: System continues despite partitions

During partition:

• CP: Choose consistency (block writes, maintain consistency)
• AP: Choose availability (allow writes, sacrifice consistency)
• CA: Not possible in distributed systems (partitions will happen)

Handling Partitions

CP Systems (Consistency + Partition Tolerance)

Behavior: Block operations during partition to maintain consistency.

Example: Traditional databases with strong consistency.

1 

AP Systems (Availability + Partition Tolerance)

Behavior: Continue operating, accept eventual consistency.

Example: Dynamo, Cassandra, CouchDB.

1 

Examples

Dynamo-style Partition Handling

1 

Common Pitfalls

• Assuming no partitions: Partitions will happen, must design for them
• Choosing wrong CAP trade-off: Not matching system requirements
• Not handling conflicts: AP systems need conflict resolution
• Ignoring split-brain: Multiple leaders during partition
• Not testing partitions: Must test partition scenarios

Interview Questions

Beginner

Q: What is partition tolerance in the context of CAP theorem?

A: Partition tolerance means the system continues operating despite network partitions that split nodes into isolated groups.

CAP theorem: You can only guarantee 2 of 3:

• CP: Consistency + Partition tolerance (block during partition)
• AP: Availability + Partition tolerance (continue, accept inconsistency)
• CA: Not possible (partitions will happen in distributed systems)

Example: During partition, CP system blocks writes to maintain consistency. AP system allows writes but may have conflicts to resolve later.

Intermediate

Q: How does a system handle network partitions? Compare CP and AP approaches.

A:

CP Approach (Consistency + Partition Tolerance):

• Behavior: Block operations if no quorum
• Trade-off: Sacrifice availability for consistency
• Use when: Strong consistency required (financial systems)
• Example: Traditional SQL databases, Zookeeper

AP Approach (Availability + Partition Tolerance):

• Behavior: Continue operating, allow writes
• Trade-off: Sacrifice consistency for availability
• Use when: High availability required (social media, content delivery)
• Example: Dynamo, Cassandra, CouchDB

During partition:

• CP: Minority partition blocks, majority continues
• AP: Both partitions continue, resolve conflicts when partition heals

Senior

Q: Design a partition-tolerant distributed database. How do you handle writes during partitions, resolve conflicts, and ensure data consistency when partitions heal?

A:

Design: AP System with Conflict Resolution

1 

Conflict Resolution Strategies:

1. Last-write-wins: Simple, but may lose data
2. Vector clocks: Detect concurrent writes, require application resolution
3. CRDTs: Automatic conflict resolution for certain data types
4. Application-level: Let application decide how to merge