Webhook Reliability Patterns Every SaaS Learns the Hard Way
Webhooks look simple until they aren't. Duplicate deliveries, silent retries, and out-of-order events break more integrations than any other cause.
Webhooks are one of those parts of a SaaS stack that look done on day one and haunt you for years. Duplicate deliveries. Silent retries. Events arriving out of order. The bugs usually show up in production, always at the worst time, and always in front of a paying customer. This is what we've learned shipping webhook systems across a bunch of projects.
The pattern below is what we consider table stakes for any B2B SaaS that sends or receives webhooks.
The failure modes nobody warns you about
Every webhook integration eventually hits these:
Duplicate delivery
The sender retries because it didn't see your acknowledgment in time. You process the event twice. If the event is "create invoice," the customer gets billed twice. Every major webhook provider — Stripe, GitHub, Shopify — will send duplicates. Not may. Will.
Out-of-order delivery
Event A fires at 10:00:00. Event B fires at 10:00:01. B arrives first. Your system now shows the "after" state before the "before" state. Timestamps in the payload help; timestamps of receipt don't.
Signature bypass attempts
Someone probes your webhook endpoint with forged payloads. If you're not verifying signatures on every request, this is how account takeovers happen.
Slow handler backpressure
Your webhook takes 4 seconds to process. The sender times out at 3 seconds and retries. You now have two workers processing the same event, competing for the same database row.
Silent failure
The webhook fails, gets retried 5 times, gives up, and nobody ever notices. Two weeks later a customer asks why their integration is broken.
Idempotency — the single most important pattern
Every webhook handler must be idempotent. Processing the same event twice must produce the same result as processing it once. This isn't optional. It's how you survive the reality that webhooks are always retried.
How to make a handler idempotent
Every webhook event has a unique ID from the sender. Stripe sends evt_1234. Shopify sends an X-Shopify-Webhook-Id. Use it.
On receipt, insert the event ID into a processed_events table with a unique constraint. If the insert fails on conflict, you've already seen this event — return 200 without doing anything else.
If the insert succeeds, process the event, commit both the business logic and the event insert in a single transaction. That way, if the handler crashes halfway through, the event isn't marked as processed and the next retry will actually do the work.
Signature verification — do it in the first line
The very first thing your handler should do is verify the signature. Not authentication middleware. Not request parsing. Signature verification, using the raw request body, with a constant-time comparison.
Every major provider documents this. Stripe uses HMAC-SHA256. GitHub uses HMAC-SHA256 with a specific header format. Slack uses a signing secret with a timestamp to prevent replay. Read the specific provider's docs. Don't roll your own scheme.
Two mistakes we see often. First, parsing the JSON body before verifying — because most frameworks parse for you, and the signature is over the raw bytes. Second, using regular string comparison instead of constant-time, which leaks the signature via timing. Both are silent bugs; both make your endpoint effectively unauthenticated.
Return 200 fast, do work async
The pattern that solves 80% of webhook problems: acknowledge in under 500ms, then do the actual work in a background job.
The handler's job on receipt:
- Verify signature.
- Check the event ID against
processed_events. - Store the raw payload in a queue table or job system.
- Return 200.
A background worker pulls from the queue, does the real business logic, and marks the event as processed. This decouples "reliable receipt" from "correct processing."
The benefit is enormous: your handler stays fast no matter what the downstream business logic does. Sender timeouts and retries drop to near zero. You can also replay failed jobs from the queue without asking the provider to resend.
Our note on background jobs covers the queue side of this in more detail.
Handling out-of-order events
Two patterns work.
Fetch the current state
When you receive an event, don't apply the delta from the payload. Fetch the current state from the provider's API. If you receive "subscription.updated" for subscription X, don't trust that the payload has the latest data — call the API and get the truth.
Slower but correct. This is what Stripe recommends for their webhooks, and it's the pattern most of our client integrations end up on.
Sequence numbers and timestamps
If the provider includes a sequence number or reliable timestamp per resource, compare on write. Only apply the event if its sequence is higher than what you've already applied. Requires provider support and careful implementation, but avoids the extra API call.
Retry logic if you're sending webhooks
The other side. If your SaaS sends webhooks to customers, they'll hit the same failure modes. Design accordingly.
Exponential backoff with jitter
Retry with increasing delay — 1 minute, 5 minutes, 30 minutes, 2 hours, 6 hours, 12 hours, 24 hours. Add random jitter to avoid thundering herds when your service recovers from an outage. Cap total retry duration at 72 hours — anything longer is unlikely to succeed and clutters your queues.
Circuit breaker per endpoint
If a customer's webhook endpoint has failed 20 times in a row, stop trying for a while. A dead endpoint doesn't need to be hammered every hour. Notify the customer's admin that their integration is failing.
Delivery logs the customer can see
Every webhook you send should have a log entry the customer can view — timestamp, event type, response status, response body. When a customer says "the webhook didn't fire," this is your defense.
Replay endpoint
Give customers a "replay" button in their dashboard. When their handler was broken and events piled up, they need a way to re-receive them without contacting support.
Testing webhooks locally
The tooling has improved. ngrok, Cloudflare Tunnel, and Tailscale funnel let you expose localhost to the internet for provider testing. Stripe CLI can forward events directly to a local port. Use these — the alternative is testing in production, which is how bad bugs ship.
Common mistakes
Processing synchronously. Handlers that do database writes, API calls, and email sends inline will eventually time out and cause retries. Acknowledge fast, work async.
Parsing before verifying. Some frameworks parse JSON automatically. If they do, capture the raw body before parsing. Signature verification depends on the raw bytes.
Trusting the payload as truth. Payloads represent a moment in time. If you need the current state, fetch it from the API.
Silent failure. A webhook that fails and gets swallowed is a customer support ticket in your future. Every failed job should alert someone, or at minimum surface in a dashboard.
No dead letter queue. Events that fail all retries need to go somewhere you can inspect. Deleting them loses the audit trail.
Trade-offs
Async processing adds latency between "webhook received" and "state updated." For most workflows that's fine — 5-10 seconds isn't user-visible. For real-time features, you'll need to be careful about where the sync boundary lives.
Idempotency also adds a database write for every event, including duplicates. At scale, the processed_events table can grow fast. Partition it by month, prune old entries after 30 days, and index it well.
Common misconceptions
"Webhooks are reliable if the provider says they are." Every provider retries. Every provider will send duplicates. Their reliability is your idempotency.
"HTTPS is enough — I don't need signature verification." HTTPS protects the transport. Signatures protect the origin. Without signatures, anyone with your webhook URL can forge events.
"We'll add retries later." The bugs from missing retries usually show up during a customer outage, when they're least fun to debug. Build retries in from day one.
Frequently asked questions
Final take
Webhook reliability isn't a single feature — it's a set of small habits that compound. Verify signatures. Acknowledge fast, work async. Store event IDs. Retry with backoff. Log every delivery. Every one of these takes an hour to add on day one and days to add retroactively after a customer incident.
If you're building a webhook system and want a review of the design, book a call. We've shipped a lot of these and the failure modes are predictable.
FAQ
Frequently asked questions
Should webhook handlers respond synchronously or asynchronously?+
Asynchronously, in almost every case. Verify the signature, store the event, acknowledge with 200 in under 500ms, then process in a background worker. Synchronous handlers eventually time out under load and cause a cascade of retries that make the problem worse.
How do I test webhooks in local development?+
Use a tunneling tool — ngrok, Cloudflare Tunnel, or Tailscale funnel — to expose your localhost to the internet. Stripe CLI, GitHub's smee.io, and similar tools can forward test events directly to a local port. Testing webhooks in production is how bad bugs ship.
What's the right retry schedule for outbound webhooks?+
Exponential backoff with jitter, starting at 1 minute and capping at around 24 hours between attempts. Total retry window of 24-72 hours. Add a circuit breaker per endpoint so dead integrations don't get hammered indefinitely. Cap the total attempts at 8-12.
Do I really need idempotency if the provider promises exactly-once delivery?+
Yes. No provider actually delivers exactly-once — the industry term is at-least-once. Even providers that market 'exactly-once' rely on the consumer being idempotent. Design for duplicates as a certainty and you'll never be surprised.
How long should I retain processed webhook events?+
30-90 days is typical for idempotency deduplication. Longer if you need it for audit or replay. Partition the table by month and prune old partitions on a schedule — otherwise it grows unbounded and slows down every insert.
Building something similar?
Let's talk in 30 minutes.