Design a webhook delivery platform that notifies millions of merchant endpoints about payment events — where one slow merchant must never delay anyone else's notifications.
Practice against the follow-up probes
- What delivery guarantee do you offer, and what does the consumer contract require of merchants in return?
- One merchant's endpoint hangs for 30 seconds per request. Trace what protects everyone else.
- Retries: schedule, ceiling, and what happens after the ceiling?
- What ordering do you promise? What do you refuse to promise, and why?
- How do merchants debug "I never got the webhook," and how do they verify a webhook is really from you?
Show answer guide
What the interviewer is probing
Multi-tenant reliability engineering: the defining challenge is isolation — millions of endpoints with wildly varying health sharing one delivery system — solved with per-tenant queues/concurrency budgets, circuit breakers, and honest contracts (at-least-once, no strict ordering). Developer-experience details (signatures, replay, delivery logs) separate candidates who've operated APIs from those who've only called them.
Strong answer outline
- Contract first: at-least-once delivery (exactly-once to an external HTTP endpoint is impossible) → events carry unique IDs; merchants must dedupe and treat handlers as idempotent. No global ordering promised — events carry timestamps/sequence hints; merchants reconcile via API reads if order matters.
- Architecture: event bus → per-merchant (or per-endpoint) delivery queues → worker fleet with per-tenant concurrency caps and short timeouts. Slow endpoint consequence: only its own queue backs up; fairness scheduling ensures workers don't pool-starve.
- Failure handling: retries with exponential backoff + jitter over ~a day-scale horizon; circuit-breaker per endpoint (persistent failures → probation with reduced attempts); after ceiling → dead-letter with merchant-visible status, dashboard alerts, and manual/automatic replay once the endpoint recovers.
- Security/DX: HMAC signatures with rotating secrets and timestamped payloads (replay-attack windows); delivery logs queryable by merchants (attempts, response codes, latencies); test-mode events and a replay button — the debugging surface is a product feature.
- Isolation extras: per-tenant rate limits, payload size caps, and egress protections (SSRF checks on merchant URLs, blocking internal address space).
- Observability: per-endpoint success/latency SLIs, backlog age alarms, global vs. tenant-scoped dashboards so incidents are classified (us vs. one merchant) in seconds.
The underlying concept
Webhooks invert the usual client-server reliability relationship: the platform becomes a client of millions of servers it doesn't control, so the design center is isolation — one tenant's pathology must be contained by construction (dedicated queues, concurrency budgets, breakers), not by hoping. The contract follows from distributed-systems truth: across an unreliable network with non-transactional receivers, at-least-once + consumer idempotency is the only honest promise, and ordering guarantees cost more than they're worth. Mature platforms treat observability and replay as part of the API: deliverability is a shared debugging problem with the merchant.
Source
Distilled Prep canon — curated from Stripe's public work on webhooks and API platform reliability.
Source: Stripe engineering blog