Webhooks: Delivery Guarantees, Retries & Security

Webhooks

Why Engineers Care About This

Webhooks enable event-driven architectures—servers push events to clients instead of clients polling for updates. This is efficient (no polling overhead) and real-time (events delivered immediately). But webhooks are unreliable—delivery can fail, clients can be down, networks can partition. Understanding webhooks helps you build reliable event-driven systems.

When you need real-time event delivery, or polling creates too much overhead, or you need to notify external systems of events, you're hitting problems that webhooks solve. These problems compound. Polling wastes bandwidth and creates latency. Without webhooks, clients must poll frequently to get updates. Webhooks provide real-time delivery, but require careful implementation—retries, security, and idempotency.

In interviews, when someone asks "How would you notify external systems of events?", they're really asking: "Do you understand webhooks? Do you know how to implement reliable webhook delivery? Do you understand webhook security and retries?" Most engineers don't. They implement webhooks without retries (unreliable) or without security (vulnerable).

Core Intuitions You Must Build

• Webhooks are push notifications, not pull requests. Webhooks push events to clients (servers initiate delivery), unlike APIs where clients pull data (clients initiate requests). This enables real-time delivery—events are delivered immediately when they occur. But webhooks are unreliable—delivery can fail, clients can be down, networks can partition. Implement retries and idempotency to handle failures.

• Webhook delivery is at-least-once, not exactly-once. Webhooks can be delivered multiple times (retries, network issues, client timeouts). Clients must handle duplicate deliveries. Make webhook processing idempotent—check if event was already processed (use event IDs, check database) before processing. Also, use idempotency keys (unique per event) to detect duplicates.

• Webhook security requires signing and validation. Webhooks are HTTP requests from your server to clients. Without security, attackers could send fake webhooks. Implement webhook signing (HMAC, shared secret) and validation (verify signature on client side). This ensures webhooks are authentic and haven't been tampered with. Don't send webhooks without signing—they're vulnerable to attacks.

• Webhook retries require exponential backoff. When webhook delivery fails, retry with exponential backoff (wait longer between retries). This prevents overwhelming failing clients and gives them time to recover. Also, implement retry limits (e.g., retry 5 times, then move to dead letter queue). Don't retry forever—some failures are permanent (client endpoint doesn't exist).

• Webhook delivery monitoring is critical. Webhooks are asynchronous—you don't know if delivery succeeded immediately. Implement webhook delivery monitoring—track delivery status (pending, delivered, failed), delivery time, and failure rates. Alert on high failure rates or stuck webhooks. This helps you catch problems early and debug delivery issues.

• Webhook payloads should be minimal and extensible. Webhook payloads should include only necessary data (event type, event ID, relevant data). Don't include full resource data—clients can fetch it if needed. Also, design payloads to be extensible (add fields without breaking clients). Use versioning if payload structure changes significantly.

Subtopics (Taught Through Real Scenarios)

Webhook Delivery and Retries

What people usually get wrong:

Engineers often implement webhooks without retries. When webhook delivery fails (client down, network issue), the webhook is lost. This makes webhooks unreliable. Implement retries with exponential backoff—retry failed webhooks, waiting longer between retries. Also, implement retry limits (e.g., retry 5 times, then move to dead letter queue). This makes webhook delivery reliable.

How this breaks systems in the real world:

A service sent webhooks without retries. When a client was temporarily down (deployment, network issue), webhooks were lost. Clients missed events and had to poll to catch up. The fix? Implement webhook retries with exponential backoff—retry failed webhooks, waiting longer between retries. Now webhooks are delivered even when clients are temporarily down. But the real lesson is: webhook delivery is unreliable. Implement retries to make it reliable.

What interviewers are really listening for:

They want to hear you talk about webhook delivery, retries, and exponential backoff. Junior engineers say "just send webhooks." Senior engineers say "webhook delivery is unreliable—implement retries with exponential backoff, set retry limits, and move failed webhooks to dead letter queues after retries exhausted." They're testing whether you understand that webhook delivery requires reliability mechanisms.

Webhook Security and Signing

What people usually get wrong:

Engineers often send webhooks without security. But webhooks are HTTP requests that could be faked by attackers. Implement webhook signing (HMAC with shared secret) and validation (verify signature on client side). This ensures webhooks are authentic and haven't been tampered with. Don't send webhooks without signing—they're vulnerable to attacks.

How this breaks systems in the real world:

A service sent webhooks without signing. An attacker sent fake webhooks to clients, causing incorrect behavior (fake payment notifications, fake user events). Clients trusted webhooks without validation, leading to security issues. The fix? Implement webhook signing (HMAC with shared secret) and require clients to validate signatures. Now webhooks are secure. But the real lesson is: webhooks are HTTP requests that can be faked. Implement signing and validation.

What interviewers are really listening for:

They want to hear you talk about webhook security, signing, and validation. Junior engineers say "webhooks are just HTTP requests." Senior engineers say "webhooks are vulnerable to attacks—implement signing (HMAC with shared secret) and require clients to validate signatures to ensure authenticity." They're testing whether you understand that webhook security is critical.

Webhook Idempotency

What people usually get wrong:

Engineers often process webhooks without idempotency checks. But webhooks can be delivered multiple times (retries, network issues, client timeouts). If processing isn't idempotent, duplicate deliveries cause problems (process payment twice, create duplicate records). Make webhook processing idempotent—check if event was already processed (use event IDs, check database) before processing.

How this breaks systems in the real world:

A service processed webhooks without idempotency. A webhook was delivered, processed, then delivered again (retry). The event was processed twice, causing duplicate records. The fix? Make webhook processing idempotent—check if event was already processed (use event ID, check database) before processing. Now duplicate deliveries don't cause duplicate processing. But the real lesson is: webhooks can be delivered multiple times. Make processing idempotent.

What interviewers are really listening for:

They want to hear you talk about webhook idempotency, event IDs, and duplicate detection. Junior engineers say "just process webhooks." Senior engineers say "webhooks can be delivered multiple times—make processing idempotent by checking if event was already processed (use event IDs, check database) before processing." They're testing whether you understand that webhook delivery is at-least-once, not exactly-once.

---

• Webhooks are push notifications—servers push events to clients, enabling real-time delivery
• Webhook delivery is at-least-once—implement idempotency to handle duplicate deliveries
• Webhook security requires signing and validation—use HMAC with shared secret
• Webhook retries require exponential backoff—retry failed webhooks, waiting longer between retries
• Webhook delivery monitoring is critical—track delivery status, time, and failure rates
• Webhook payloads should be minimal and extensible—include only necessary data
• Webhooks enable event-driven architectures—but require careful implementation for reliability

---

• API Design - Designing webhook endpoints
• Message Queues - Using message queues for webhook delivery
• Authentication & Authorization - Webhook security and signing