Answer the webhook first, do the work from a table

A webhook endpoint that only verifies, stores and answers 200, with a Postgres inbox table that makes duplicates harmless and out-of-order events visible.

Sender Endpoint Signature check Inbox table Worker Ordering guard Handler Dead-letter view Rejected log answers 200 unique event ID parks early events after five attempts failed signature
From sender to handler through a signature check and an inbox table, with side exits for rejected requests and dead letters.
On this page
  1. The inbox is one table
  2. Twelve lines of endpoint
  3. Workers pull rows; nobody pushes to them
  4. When the update arrives before the create
  5. Birchline’s 46 double credits
  6. Ten checks before it goes live

A webhook sender knows one thing about your system: whether it answered in time. Miss the deadline and the same event keeps coming back for hours.

So I treat the endpoint as a receipt desk. It checks the signature, writes the raw event to a table, and says 200. Everything that touches the CRM or the ledger happens afterwards, from the table, on my schedule.

The inbox is one table

The design hangs on a unique key over the sender’s own event ID. The sender already promised that ID is stable across retries, so I let the database enforce the promise.

CREATE TABLE webhook_inbox (
  id          bigserial   PRIMARY KEY,
  source      text        NOT NULL,
  event_id    text        NOT NULL,
  event_type  text        NOT NULL,
  body        text        NOT NULL,
  received_at timestamptz NOT NULL DEFAULT now(),
  status      text        NOT NULL DEFAULT 'pending',
  attempts    int         NOT NULL DEFAULT 0,
  next_try_at timestamptz NOT NULL DEFAULT now(),
  claimed_at  timestamptz,
  last_error  text,
  UNIQUE (source, event_id)
);

CREATE INDEX webhook_inbox_due
  ON webhook_inbox (next_try_at) WHERE status = 'pending';

CREATE VIEW webhook_dead_letters AS
  SELECT id, source, event_id, event_type, attempts, last_error, received_at
  FROM webhook_inbox WHERE status = 'failed';

body is text on purpose. The signature was computed over the exact bytes the sender transmitted, and jsonb would reorder keys and drop whitespace. Keep the original and you can re-verify or replay it a year later.

Twelve lines of endpoint

handle POST /webhooks/billing:
  raw = request.raw_body                       # bytes, before any JSON parsing
  ts, sig = parse(request.header("X-Signature"))
  if abs(now() - ts) > 5 minutes:              log_rejected("stale");  return 400
  expected = hmac_sha256(SECRET, ts + "." + raw)
  if not constant_time_equal(expected, sig):   log_rejected("bad signature");  return 400
  event = json_parse(raw)
  db.execute("INSERT INTO webhook_inbox (source, event_id, event_type, body)
              VALUES ('billing', $1, $2, $3)
              ON CONFLICT (source, event_id) DO NOTHING",
             event.id, event.type, raw)
  return 200

Header names and the signed string differ by sender, so check their docs. The five-minute tolerance stops someone replaying a captured request next year, after the original row has been purged.

Rejected requests go to an ordinary log line with the reason. They don’t go into the inbox, because anyone on the internet can produce them.

A duplicate hits the unique key, inserts nothing, and still gets a 200. That’s the answer that makes the sender stop. The only time I want a retry is when my own insert fails, and then the endpoint returns a 500 and lets the sender do what it’s good at.

Workers pull rows; nobody pushes to them

Each worker runs this in a loop:

WITH due AS (
  SELECT id
  FROM webhook_inbox
  WHERE status = 'pending' AND next_try_at <= now()
  ORDER BY received_at
  LIMIT 20
  FOR UPDATE SKIP LOCKED
)
UPDATE webhook_inbox i
SET status = 'processing', claimed_at = now()
FROM due
WHERE i.id = due.id
RETURNING i.id, i.event_type, i.body;

SKIP LOCKED lets several workers run the same query without waiting on each other or grabbing the same rows. On success a row becomes done. On failure it goes back to pending with attempts + 1 and a later next_try_at, and after five attempts it becomes failed and shows up in the dead-letter view. A sweeper resets anything stuck in processing for ten minutes, in case a worker died mid-batch.

I’d rather have this than a queue for most webhook volumes. The retry policy is two columns I can read, and “what happened to event X” is a SELECT. I made the longer case in Queue, log or table.

When the update arrives before the create

Senders rarely guarantee order. The guard sits between the worker and the handler and needs one more small table: the last version applied per object. Use the version or sequence number in the payload if there is one, and the object’s updated timestamp if there isn’t.

for an event about object O carrying version V:
  last = last_applied_version(O)            # null if O has never been seen
  if last is not null and V <= last:
      mark 'skipped'                        # older than what we already have
  else if the event needs a predecessor we don't have:
      if now() - received_at < 60 seconds:
          set status 'pending', next_try_at = now() + 5 seconds
      else:
          fetch O from the sender's API, apply that state,
          record its version, mark 'done'
  else:
      apply the event, record V, mark 'done'

Parking doesn’t count as an attempt. Read and write that version in one transaction with the object’s row locked, or two workers holding events for the same object will race.

The fallback is what keeps the park from becoming a second backlog: after a minute I stop waiting for an event that may never come and ask the sender what the object looks like now. Current state is always a legitimate answer.

Birchline’s 46 double credits

Birchline Software is a company I invented for this post, and so are its numbers. It receives about 18,000 billing webhooks a day. Its old endpoint updated the CRM inline, with a median of 2.8 seconds. On slow days the sender timed out and retried, and the retry started the same work a second time. That put duplicates at 3.1 percent, roughly 560 events a day. Most were harmless overwrites. Credit notes weren’t, and in one month 46 credits were applied twice.

With the inbox, the endpoint answers in 35 ms, so its own slowness stops causing retries. The duplicates that still arrive fail the unique key and never reach a handler. The table also shows something the old code hid: 0.4 percent of updates land before their create event. Those park for up to 60 seconds, then fall back to the API fetch.

Ten checks before it goes live

  1. The signature is verified against the raw bytes, before any JSON parsing.
  2. The comparison is constant-time, and the timestamp tolerance is enforced.
  3. The signing secret can be rotated, with two valid secrets during the overlap.
  4. The unique key is on the sender’s event ID, scoped by source.
  5. The endpoint does one insert and nothing else: no CRM call, no outbound HTTP.
  6. A duplicate returns 200. A failed insert returns 500.
  7. Workers claim rows with SKIP LOCKED, and stuck claims get reset.
  8. Handlers are safe to run twice, because a crash after the write and before done will rerun them.
  9. Stale versions are skipped, early events are parked, and the park has a deadline with an API fetch behind it.
  10. Someone is alerted when the dead-letter view isn’t empty, and old done rows are deleted on a schedule.