Back to Systems
LaravelQueuesReliabilityAPIMonitoring

Webhook Dispatcher

A reliable, configurable webhook delivery system with retry logic, payload transformation, and delivery guarantees.

The Problem

The application needed to notify external systems when key events occurred — order completed, payment received, shipment updated. The original implementation was straightforward and fragile: an HTTP POST fired synchronously during request processing. If the receiving server was slow, our response times spiked. If it was down, the webhook was lost forever. There was no retry logic, no delivery log, and no way to know which webhooks had failed until a partner complained about missing data.

As the number of webhook consumers grew, so did the blast radius of any single slow endpoint. One unresponsive consumer could degrade the entire application’s performance.

Constraints

Webhook consumers ranged from enterprise partners with strict payload schemas to internal microservices with minimal validation. The system needed to support per-consumer configuration: custom headers, payload transformations, and different retry policies. It also had to be observable — the operations team needed to see delivery status, failure reasons, and retry history without digging through logs.

Solution Architecture

I extracted webhook delivery into a standalone dispatcher service with four key components:

  • Event Capture — When a domain event fired (e.g., OrderCompleted), a listener created a WebhookDelivery record with the target URL, serialized payload, and consumer-specific configuration. No HTTP call happened at this point.

  • Queued Dispatch — A dedicated queue worker picked up pending deliveries and executed the HTTP request with configurable timeouts. Each delivery attempt was logged with status code, response time, and response body. The queue used a separate connection to isolate webhook traffic from the application’s primary job processing.

  • Retry Engine — Failed deliveries were retried with exponential backoff and jitter: 30 seconds, 2 minutes, 8 minutes, 30 minutes, 2 hours. Each attempt was recorded. After exhausting retries, the delivery moved to a dead-letter state and triggered an alert. Consumers could also configure a maximum retry window instead of a fixed attempt count.

  • Signature Verification — Every outbound payload was signed with an HMAC-SHA256 signature using a per-consumer secret. The signature was sent in a header so consumers could verify authenticity and reject tampered payloads.

  • Monitoring Dashboard — A lightweight admin panel showed delivery success rates, active retries, dead-lettered deliveries, and per-consumer health. This turned “did the webhook go through?” from a support ticket into a self-service lookup.

Tech Stack

  • Backend: Laravel 10, PHP 8.2
  • Queue: Redis with isolated queue connections via Laravel Horizon
  • Storage: PostgreSQL for delivery logs and consumer configuration
  • Security: HMAC-SHA256 payload signing, per-consumer secret rotation
  • Monitoring: Custom dashboard built with Livewire, alerting via Slack integration

What I Learned

The most valuable pattern I took from this project was treating webhooks as a delivery guarantee problem rather than a fire-and-forget convenience. Idempotency keys on every delivery let consumers safely handle duplicates from retries. The dead-letter queue turned silent failures into actionable alerts.

I also learned that per-consumer configurability is worth the upfront investment. Different consumers have wildly different reliability characteristics, and a one-size-fits-all retry policy either retries too aggressively for healthy endpoints or gives up too quickly on flaky ones. Letting each consumer tune their own policy reduced noise and improved actual delivery rates.