Request Batching: Performance, Rate Limits & Trade-offs

Request Batching

Why Engineers Care About This

Request batching combines multiple operations into a single request, reducing overhead (network round trips, connection setup, authentication). This improves performance for bulk operations (creating multiple records, updating multiple items). But batching adds complexity—batch size optimization, error handling, and partial failures. Understanding batching helps you design efficient APIs.

When your API is slow for bulk operations, or clients make many small requests, or network overhead dominates processing time, you're hitting problems that batching solves. These problems compound. Without batching, bulk operations require many requests (slow, wasteful). With poor batching (too large, too small), performance doesn't improve or becomes worse. Good batching optimizes performance for bulk operations.

In interviews, when someone asks "How would you optimize this API for bulk operations?", they're really asking: "Do you understand batching? Do you know when batching improves performance? Do you understand batch size optimization?" Most engineers don't. They process everything individually (slow for bulk) or batch everything (complex, not always beneficial).

Core Intuitions You Must Build

• Batching reduces overhead, not processing time. Batching combines multiple operations into one request, reducing network overhead (one round trip instead of many), connection overhead (one connection instead of many), and authentication overhead (one auth check instead of many). But batching doesn't reduce processing time—operations still take the same time. Batching is beneficial when overhead dominates (network latency, connection setup), not when processing dominates.

• Batch size optimization balances performance and latency. Small batches (10 items) have low latency (fast responses) but high overhead (many requests). Large batches (1000 items) have low overhead (few requests) but high latency (slow responses). Optimal batch size balances these—large enough to reduce overhead, small enough to maintain acceptable latency. Also, batch size depends on use case—real-time updates need small batches, bulk imports can use large batches.

• Batch APIs should support both individual and batch operations. Some operations need individual requests (real-time updates, single item operations). Some operations benefit from batching (bulk imports, bulk updates). Design APIs that support both—individual endpoints for single operations, batch endpoints for bulk operations. Don't force batching for everything—it adds complexity when not needed.

• Batch error handling requires careful design. Batch operations can have partial failures (some items succeed, some fail). Design error handling—return success/failure per item, or fail entire batch if any item fails (all-or-nothing). Choose based on use case—all-or-nothing for transactions (consistency), partial success for bulk operations (efficiency). Also, return detailed error information for failed items.

• Batching works well with async processing. Batch operations are often slow (processing many items). Make batch endpoints async—accept batch request, return job ID, process in background, notify when complete. This keeps APIs responsive (don't block on batch processing) and enables reliable processing (retries, error handling). Don't make batch endpoints synchronous—they block for too long.

• Batching requires idempotency for reliability. Batch operations can be retried (network errors, timeouts). Make batch processing idempotent—check if items were already processed (use item IDs, check database) before processing. This prevents duplicate processing on retries. Also, use idempotency keys (unique per batch) to detect duplicate batches.

Subtopics (Taught Through Real Scenarios)

When Batching Improves Performance

What people usually get wrong:

Engineers often batch everything, or nothing. But batching is beneficial when overhead dominates (network latency, connection setup), not when processing dominates. If processing is slow (complex operations, external API calls), batching doesn't help much. If overhead is slow (network latency, connection setup), batching helps significantly. Understand when batching improves performance—it's about overhead, not processing.

How this breaks systems in the real world:

A service batched all operations, even single-item operations. Batching added complexity (batch API, error handling) without providing benefits (single operations don't benefit from batching). The service was harder to use and maintain. The fix? Use batching only for bulk operations—individual operations use individual endpoints, bulk operations use batch endpoints. Now service is simpler and more efficient. But the real lesson is: batching is beneficial when overhead dominates, not always. Use batching selectively.

What interviewers are really listening for:

They want to hear you talk about when batching improves performance, overhead vs processing, and selective batching. Junior engineers say "just batch everything" or "never batch." Senior engineers say "batching reduces overhead (network, connections, auth) but not processing time—use batching when overhead dominates, not when processing dominates." They're testing whether you understand that batching is about overhead reduction, not processing optimization.

Batch Size Optimization

What people usually get wrong:

Engineers often use fixed batch sizes (e.g., 100 items) without understanding trade-offs. But batch size affects performance and latency. Small batches have low latency but high overhead. Large batches have low overhead but high latency. Optimal batch size balances these—large enough to reduce overhead, small enough to maintain acceptable latency. Also, batch size depends on use case—real-time needs small batches, bulk imports can use large batches.

How this breaks systems in the real world:

A service used fixed batch size (1000 items) for all operations. For real-time updates, batches were too large (high latency, poor UX). For bulk imports, batches were too small (high overhead, slow processing). The fix? Use different batch sizes for different use cases—small batches (10-50 items) for real-time, large batches (100-1000 items) for bulk operations. Now performance is optimized for each use case. But the real lesson is: batch size optimization depends on use case. Don't use fixed batch sizes.

What interviewers are really listening for:

They want to hear you talk about batch size optimization, latency vs overhead trade-offs, and use-case-specific batch sizes. Junior engineers say "just use batch size 100." Senior engineers say "optimize batch size based on use case—small batches for real-time (low latency), large batches for bulk operations (low overhead)—balance latency and overhead." They're testing whether you understand that batch size is a trade-off, not a fixed value.

Batch Error Handling

What people usually get wrong:

Engineers often design batch APIs that fail entirely if any item fails (all-or-nothing). But batch operations can have partial failures (some items succeed, some fail). All-or-nothing is simple but wasteful (reprocess all items if one fails). Partial success is efficient but complex (must handle mixed results). Choose based on use case—all-or-nothing for transactions (consistency), partial success for bulk operations (efficiency).

How this breaks systems in the real world:

A service used all-or-nothing batch processing. If one item failed (invalid data, constraint violation), the entire batch failed and had to be retried. This was wasteful—99 items succeeded but were reprocessed because 1 item failed. The fix? Use partial success—process all items, return success/failure per item. Now only failed items need to be retried. But the real lesson is: batch error handling depends on use case. All-or-nothing for consistency, partial success for efficiency.

What interviewers are really listening for:

They want to hear you talk about batch error handling, all-or-nothing vs partial success, and use-case-specific error handling. Junior engineers say "just fail the entire batch if any item fails." Senior engineers say "choose error handling based on use case—all-or-nothing for transactions (consistency), partial success for bulk operations (efficiency)—return detailed error information for failed items." They're testing whether you understand that batch error handling is about trade-offs, not just "failing."

---

• Batching reduces overhead, not processing time—beneficial when overhead dominates
• Batch size optimization balances performance and latency—small batches for real-time, large for bulk
• Batch APIs should support both individual and batch operations—don't force batching
• Batch error handling requires careful design—all-or-nothing for consistency, partial success for efficiency
• Batching works well with async processing—don't block on batch processing
• Batching requires idempotency for reliability—prevent duplicate processing on retries
• Good batching optimizes performance for bulk operations without adding unnecessary complexity

---

• API Design - Designing batch endpoints
• Background Jobs - Processing batches asynchronously
• Error Handling & Logging - Handling batch errors