Handling webhooks reliably: retries, duplicates, ordering and replay

A webhook handler that works in development is easy: receive the POST, update the order, return 200. A webhook handler that works in production has to survive being called twice, being called late, being called in the wrong order, and not being called at all. This article covers each of those, with the Paydude retry schedule and the Events API as they actually behave.

Verifying the signature is the first step and is covered separately in webhook signature verification. Everything here assumes the request is already verified.

Acknowledge first, work second

Each delivery attempt has a five-second timeout, and any response other than a 2xx counts as a failure. A handler that sends an email, updates three tables and calls the API before responding will eventually take longer than five seconds — and then the event is retried even though your handler did the work.

So split the handler in two. The request path stores the verified event and returns immediately; a worker does the processing.

export async function POST(request) {
  const raw = await request.text();
  if (!verifySignature(raw, request.headers.get("payments-signature"))) {
    return new Response("bad signature", { status: 400 });
  }
  const event = JSON.parse(raw);

  // The unique constraint on event_id is the deduplication. A retried
  // delivery of an event we already have is a no-op, and still gets a 200 —
  // otherwise it would be retried again.
  await db.query(
    `insert into webhook_events (event_id, type, object_id, payload, received_at)
     values ($1, $2, $3, $4, now())
     on conflict (event_id) do nothing`,
    [event.id, event.type, event.data?.id, raw]
  );

  // Hand off. A queue, a job table polled by a worker, or waitUntil() on a
  // platform that supports it — anything that is not "before the response".
  await queue.enqueue("process-webhook-event", { eventId: event.id });

  return new Response("ok", { status: 200 });
}

create table webhook_events (
  event_id     text primary key,     -- evt_…; the dedupe key
  type         text not null,
  object_id    text,                 -- pi_…, po_…, sub_…: what it is about
  payload      jsonb not null,
  received_at  timestamptz not null,
  processed_at timestamptz           -- null until the worker finishes
);

create index on webhook_events (processed_at) where processed_at is null;

Storing the raw payload is worth the space. When a customer disputes what happened, the events table is the exact record of what the processor told you, in the order you received it.

Retries mean duplicates

When a delivery fails, Paydude retries on this schedule:

The schedule is generous, which is good for outages and means duplicates are a normal part of operation rather than an edge case. Any delivery your server received but did not acknowledge in time — a slow response, a deploy that dropped the connection, a load balancer timeout — comes back. The on conflict do nothing above is what makes that harmless.

Retries also mean reordering

If event A fails its first delivery and event B, sent a second later, succeeds, you receive B about five minutes before A. Nothing about webhooks promises arrival order, and a retry schedule guarantees you will sometimes see the reverse of it.

There is a second, subtler effect. An event carries the object as it was when the event was sent, not as it was when the thing in its name happened. In the Paydude sandbox, one payout.created event arrived with data.status: "paid" — the payout had been created and paid within milliseconds, and payout.paid followed four milliseconds later.

• Do not derive state from the event name. payout.created does not mean the payout is pending.
• Treat terminal states as final. Once your record says succeeded or canceled, a late event cannot change it. Do not generalise this to "only move forward" — a declined payment intent legitimately moves back to requires_payment_method so the customer can try another card.
• When it matters, re-fetch. For anything that moves money or grants access, treat the event as a prompt to GET the object and act on what the API returns now.

const TERMINAL = new Set(["succeeded", "canceled"]);

async function processEvent(eventId) {
  const { payload } = await db.one(
    "select payload from webhook_events where event_id = $1", [eventId]);
  const event = JSON.parse(payload);

  if (event.type.startsWith("payment_intent.")) {
    // Ask the API for the current state rather than trusting the snapshot.
    const intent = await api.get(`/v1/payment-intents/${event.data.id}`);
    const order = await db.orders.findByIntent(intent.id);

    // The re-fetched status is the truth, except that a terminal state we
    // already recorded is never overwritten by a later, stale read.
    if (order && !TERMINAL.has(order.intent_status) && intent.status !== order.intent_status) {
      await db.orders.update(order.id, { intent_status: intent.status });
      if (intent.status === "succeeded") await fulfil(order);
    }
  }

  await db.query(
    "update webhook_events set processed_at = now() where event_id = $1", [eventId]);
}

fulfil must itself be safe to call twice, because the worker can crash after fulfilling and before marking the event processed. The usual fix is the same as everywhere else in payments: a unique constraint on the thing that must only happen once.

When your endpoint goes quiet

Paydude tracks the health of each endpoint, and a sustained outage changes what happens to your events:

Re-enable with PATCH /v1/webhook-endpoints/{id} and { "status": "active" }; the endpoint moves to pending and is promoted to active after its first successful delivery. Owners and admins are emailed when an endpoint changes state, which is worth routing somewhere a person actually reads.

The backstop: sweep the Events API

Webhooks are a notification mechanism, not a guarantee. The dependable design pairs them with a periodic sweep that asks the API what happened and processes anything the webhook path missed. GET /v1/events returns the last 30 days of events, newest first, offset-paginated.

async function sweepEvents() {
  const lte = new Date();
  const gte = new Date(lte.getTime() - 3 * 24 * 60 * 60 * 1000);   // 3 days back

  const seen = [];
  for (let offset = 0; ; offset += 100) {
    const page = await api.get("/v1/events?" + new URLSearchParams({
      // Both bounds are required together — see the note below.
      created_at_gte: gte.toISOString(),
      created_at_lte: lte.toISOString(),
      types: "payment_intent.succeeded,payment_intent.payment_failed,payout.paid",
      limit: "100",
      offset: String(offset),
    }));
    seen.push(...page.items);
    if (!page.has_more) break;
  }

  // Newest-first from the API; process oldest-first.
  for (const event of seen.reverse()) {
    const inserted = await db.query(
      `insert into webhook_events (event_id, type, object_id, payload, received_at)
       values ($1, $2, $3, $4, now())
       on conflict (event_id) do nothing
       returning event_id`,
      [event.id, event.type, event.data?.id, JSON.stringify(event)]
    );
    // Only events the webhook path never delivered get queued.
    if (inserted.rowCount) await queue.enqueue("process-webhook-event", { eventId: event.id });
  }
}

Because the sweep writes into the same table with the same conflict rule, it cannot double-process an event the webhook already delivered. Tested against the sandbox on 28 September 2026, three details of the Events API matter for writing it:

• The date bounds must be sent together. The reference says created_at_lte defaults to now, but created_at_gte on its own returns 400 with "created_at_lte should not be null or undefined". Send both.
• The types filter accepts a comma-separated list, and a repeated types parameter works too. object_id narrows the list to one payment, payout or subscription — useful for rebuilding the history of a single object.
• Do not build on the delivery status filter. The documentation suggests filtering by delivery_status to find missed events, but every value — succeeded, failed, pending, undelivered — returned a 500 in the sandbox, with and without an endpoint registered. A time-window sweep deduplicated against your own table gets you the same answer without depending on it.

A checklist

1. Verify the signature against the raw body before parsing.
2. Insert the event by its id with a unique constraint; on conflict, still return 200.
3. Respond within five seconds; process in a worker.
4. In the worker, re-fetch the object for anything that moves money or grants access.
5. Never overwrite a terminal state; otherwise trust the re-fetched object.
6. Make the side effects themselves idempotent — fulfilment, emails, credits.
7. Sweep the Events API on a schedule, deduplicated against the same table.
8. Alert on endpoint status changes, and on a growing count of unprocessed events.