The Problem
The operations platform sat at the center of several external systems: a delivery tracking partner sending webhook events when shipments completed, a customer engagement platform handling post-delivery notifications, a courier service completing proof-of-delivery in its own app, and an AI support agent that needed live order context.
Each integration arrived as a webhook endpoint or API connector. Early implementations treated those entry points as convenient places to do everything synchronously—update internal state, notify the customer, call a third-party API, play a sound in the browser.
That worked until it didn’t.
The delivery partner’s successful-delivery webhook began timing out in production. Confirmations were failing at the integration boundary—not because deliveries were wrong, but because a customer-notification API call ran inside the webhook request. When that external service was slow, the partner’s HTTP client exceeded its timeout and the event was lost or retried unpredictably.
Separately, post-delivery rating prompts were firing twice when both a webhook route and a background job were wired to the same business moment. Customers received duplicate messages; the fix required tracing two independent code paths rather than patching one handler.
The underlying issue wasn’t any single bug—it was missing rules for what a webhook handler is allowed to do.
Constraints
Partner systems own their timeout budgets. The operations platform had to respond quickly even when downstream notification or enrichment work was slow.
Delivery confirmations could not be ambiguous—operations teams depended on them for order state, customer care, and downstream billing triggers.
Not every customer notification belongs on the hot path of a delivery event. A rating request sent the instant a package is marked delivered is a different product decision than one sent 48 hours later, after the customer has actually used the item.
Retries are normal. Partner webhooks will be delivered more than once; side effects had to tolerate that without double-notifying or corrupting state.
Solution Architecture
Delivery Webhooks: Acknowledge Fast, Process Async
The production incident was traced to a single pattern: a customer-notification call executing synchronously inside a delivery-partner webhook handler.
The fix separated concerns:
- The webhook validates the incoming payload and records the delivery event.
- Notification work—including calls to the customer engagement platform—is dispatched to a queue worker.
- The HTTP response returns immediately, within the partner’s timeout window.
A follow-up improvement ensured notification sounds played only in the agent’s most recent browser tab, avoiding duplicate audio when multiple operations tabs were open.
This established a permanent rule: never make third-party API calls inline inside an inbound webhook handler.
Rating Notifications: Webhook to Scheduled Job
Post-delivery rating prompts were redesigned around timing, not immediacy.
Instead of chaining notifications directly to delivery webhooks, a scheduled job queries orders delivered exactly 48 hours ago—using UTC-normalized timestamps and a dedicated order query builder so the window is predictable, testable, and independent of webhook delivery timing.
The background job was reworked to accept full order objects rather than loosely coupled identifiers, making the notification pipeline easier to reason about and extend.
When investigation revealed duplicate rating messages, the root cause was overlap: both the old webhook route and the new scheduled job were still active. The redundant webhook route was disabled, leaving a single authoritative path.
Delivery-Time Metadata Ingestion
Beyond success/failure signals, the delivery partner webhook stream carried delivery-time metadata useful for operational timelines and customer care.
A dedicated ingestion path parsed and persisted this data without blocking the webhook response on downstream enrichment—keeping the handler thin while still capturing the full partner payload for operational use.
Courier Proof-of-Delivery Handoff
A courier platform needed drivers to complete proof-of-delivery in their own mobile tooling. Building a parallel confirmation flow would have duplicated state and created drift between systems.
Instead, the operations platform passed the delivery partner’s tracking link into the courier system’s custom field—one reliable handoff point so drivers landed in the right place to confirm delivery. Integration by simplification, not by rebuilding the partner’s workflow.
AI Support: Order Lookup by Phone
An AI customer-support agent needed structured order context during live conversations—not raw database access, but a stable integration surface.
I built a read-only API endpoint: phone number in, matching orders with shipment identifiers and current statuses out. The AI agent consumed this as its source of truth for delivery questions, keeping support automation decoupled from internal page logic and UI assumptions.
Technologies
- Backend: Laravel, PHP
- Database: MySQL
- Queues: Laravel queue workers for async webhook side effects and notification jobs
- Admin & UI: Livewire, Filament (operations notifications and agent workflows)
- Integrations: Delivery partner webhooks, customer engagement notifications, courier platform API, AI support connector
- Testing: Pest (feature tests), PHPUnit (unit tests)
Engineering Decisions
Webhooks are boundaries, not workflows
A webhook handler validates, persists intent, enqueues work, and responds. Orchestrating every downstream system inside the HTTP request window makes partner reliability depend on every third party you call—which is how delivery confirmations started failing.
Queue all third-party work
Any external API call inside a webhook is a latent timeout. Moving notification, engagement, and enrichment work to queue workers trades milliseconds of response latency for reliable acknowledgment and independently retryable jobs.
Pick the right trigger for each notification
Real-time webhooks suit operational state changes. Customer-facing prompts that depend on elapsed time belong on scheduled jobs with explicit query windows—not chained blindly to delivery events.
One authoritative path per outcome
Duplicate rating notifications came from two active triggers, not from a single flaky handler. Disabling the redundant route was cleaner than adding deduplication logic on top of overlapping designs.
Integration surfaces stay narrow
The AI support connector exposes only what external consumers need—phone lookup, order identifiers, shipment statuses—rather than leaking internal models. Narrow APIs age better than page-scoped shortcuts.
Difficult Problems
The silent timeout
Delivery confirmations failed with no obvious application error—the partner simply timed out waiting for a response. Root cause required reading the handler as a timeline of synchronous calls, not assuming the webhook logic itself was slow.
Dual firing
Duplicate customer notifications persisted until both code paths were mapped: an event-driven webhook route and a time-based job running in parallel. Fixing it meant choosing one owner for the outcome, not adding idempotency keys to both.
Realtime polish at scale
After async notification delivery, agents with multiple browser tabs heard duplicate sounds. The fix targeted only the most recently active tab—operational UX detail on top of architectural decoupling.
What I Learned
This work changed how I approach every new partner integration.
The difficult part is not receiving HTTP requests—it is drawing boundaries between systems you do not control. Partners retry, notification vendors lag, and couriers have their own apps. The operations platform’s job is to stay reliable at the seam.
The delivery-timeout incident produced a rule I now apply by default: webhooks acknowledge fast, queues do the talking to third parties. The rating-notification redesign reinforced a second rule: match the trigger to the product moment—immediate state change versus delayed customer outreach are different problems.
I also learned that duplicate side effects usually mean duplicate owners, not duplicate requests. Finding the second trigger is faster than hardening the first handler.
Across delivery webhooks, scheduled notifications, courier handoffs, and AI support connectors, the same doctrine held: decouple webhooks from side effects, queue external work, document vendor behavior as fact, and give every customer-facing outcome exactly one authoritative path.