Short answer
A webhook pipeline is a distributed system, and the network will deliver your events twice, out of order, or not at all. Resilience comes from four properties: acknowledge fast and process asynchronously, make every handler idempotent, retry with backoff into a dead-letter queue, and reconcile against the source of truth on a schedule. n8n gives you the orchestration; these properties are what you have to design in.
Webhook integrations fail in a characteristic way. They work for months, then a provider has a slow afternoon, retries pile up, and you discover that your handler was never idempotent – so a hundred duplicate orders appear in a downstream system and someone spends a weekend reconciling them by hand.
Everything below is aimed at that scenario.
Acknowledge fast, process later
The single most important rule: return 200 before you do the work.
Providers time out webhook deliveries aggressively – commonly between 5 and 10 seconds – and a timeout is treated as a failure, which triggers a redelivery. If your handler validates the payload, calls three APIs, writes to a database and then responds, you have built a system that generates duplicates under exactly the conditions where duplicates hurt most: when things are already slow.
The receiving endpoint should do three things and nothing else:
- Verify the signature.
- Persist the raw payload with its delivery id.
- Return 200.
Processing happens after, from the stored payload. In n8n this means the webhook node writes to a queue or a table and responds immediately, with the real workflow triggered separately.
Idempotency is not optional
Assume every event will arrive more than once, because it will. Providers retry on timeout, on 5xx, and sometimes on a network blip that never reached you at all. “At least once” is the delivery guarantee you actually have.
Every webhook carries a stable identifier – event_id, delivery_id, or a header equivalent. Store it with a unique constraint and let the database enforce deduplication:
-- The unique index is the deduplication mechanism.
-- Application-level "check then insert" has a race window; this does not.
CREATE TABLE webhook_events (
event_id TEXT PRIMARY KEY,
source TEXT NOT NULL,
payload JSONB NOT NULL,
received_at TIMESTAMPTZ NOT NULL DEFAULT now(),
processed_at TIMESTAMPTZ,
attempts INT NOT NULL DEFAULT 0,
last_error TEXT
);
-- A duplicate delivery is a no-op, not an error.
INSERT INTO webhook_events (event_id, source, payload)
VALUES ($1, $2, $3)
ON CONFLICT (event_id) DO NOTHING;Where the provider does not supply an id, derive one from stable payload fields – the object id plus its updated timestamp – and hash it. A content hash is not as good as a real delivery id, but it is far better than nothing.
Verify signatures, and verify them correctly
An unauthenticated webhook endpoint is a public API for writing to your database. Two mistakes are common and both defeat the purpose:
- Verifying against the parsed body. The signature covers the raw bytes. Re-serialising JSON changes key order and whitespace, so you must capture the raw body before any parsing middleware touches it.
- Comparing with
==. Use a constant-time comparison. String equality short-circuits on the first differing byte, which leaks the signature one character at a time.
Check the timestamp too, and reject anything outside a few minutes’ tolerance – otherwise a captured request can be replayed indefinitely.
Retries, backoff, and where failures go to be seen
A retry policy has three parts, and the third is the one that gets skipped:
| Element | Sensible default | Why |
|---|---|---|
| Backoff | Exponential, with jitter | Without jitter, every failed event retries in lockstep and re-creates the spike |
| Attempt cap | 5 to 7 attempts | Beyond that the failure is structural, not transient |
| Dead-letter queue | Always, with the error and attempt count | A failure that is silently dropped is a data-loss bug you find out about from a customer |
Distinguish retryable from terminal failures. A 503 from a downstream API is worth retrying; a 422 because the payload references a deleted record is not – it will fail identically seven more times and delay everything behind it. Route terminal failures straight to the dead-letter queue.
And make the dead-letter queue visible. A DLQ nobody looks at is the same as no DLQ, with extra storage cost. Alert on depth, not just on individual errors.
Ordering: stop trying to preserve it
Concurrent delivery means an order.updated event can arrive before the order.created it depends on. Two workable responses, in order of preference:
- Make handlers order-independent. Upsert rather than insert. If an update arrives for an object you have not seen, create it from the payload. This is almost always the right answer.
- Fetch current state instead of trusting the payload. Treat the webhook as a notification that something changed, then read the authoritative record from the provider’s API. Costs an extra call; eliminates an entire class of ordering bug.
Enforcing strict ordering with per-key serialisation is possible but expensive, and it converts a throughput problem into a latency problem. Reach for it only when the domain genuinely requires it.
Reconciliation: the safety net that catches what the pipeline missed
Even a well-built pipeline drops events. Providers have incidents, deployments drop in-flight requests, and a bug that runs for an hour before anyone notices will lose whatever passed through in that hour.
Run a scheduled job that pulls changed records from the source over a lookback window and compares them against your own state. Two things matter about it: the reconciliation must be idempotent – it will re-process events you already handled – and the count of discrepancies it finds is a metric worth watching. A number that has been zero for months and suddenly is not tells you a pipeline broke before any customer does.
Where AI classification fits, and where it does not
n8n makes it easy to drop a model call into a workflow, and there is a genuine use for it: classifying unstructured content that arrives through the pipeline – routing inbound messages, categorising free-text fields, extracting structure from documents.
What it should not do is sit in the synchronous path of a financial or inventory operation. Model calls add latency, can fail, and are non-deterministic. Classify asynchronously, store the result with a confidence score, and route low-confidence cases to a human. The validation and fallback patterns for that are in Integrating OpenAI & Claude APIs into Enterprise Web Applications.
The commerce side of these pipelines – catalogue synchronisation, inventory consistency, checkout events – is covered in E-Commerce Engineering: Shopify Custom Apps & Scalable WooCommerce Systems.
Questions people actually ask
- Should the webhook endpoint be in n8n or in my own application?
- Put the receiving endpoint wherever you can guarantee it responds in under a second and can persist the payload durably. n8n webhook nodes are fine for that when n8n is reliably available; if the workflow instance is doing heavy work, a thin endpoint in your own application writing to a queue is safer.
- How do I test a webhook pipeline properly?
- Replay real captured payloads, not hand-written fixtures – the edge cases live in the fields you did not know existed. Then test the three scenarios that actually break production: the same event delivered twice, an update arriving before its create, and a downstream API returning 500 mid-workflow.
- What if the provider does not send an event id?
- Derive one by hashing the stable identifying fields – typically the object id combined with its last-modified timestamp. It is weaker than a real delivery id because a genuine repeat change within the same timestamp resolution will be treated as a duplicate, but it prevents the far more common failure of processing the same delivery twice.
- How long should I keep raw webhook payloads?
- Long enough to replay an incident – 30 to 90 days covers most cases. Keep them separately from processed state so a bug in processing can be fixed and the events replayed. Apply the same retention and access controls you would to any store holding customer data, because that is what it is.
- Is n8n suitable for production volume?
- Yes, with queue mode and workers rather than a single instance, and with the persistence and retry properties described above designed in rather than assumed. What n8n does not do for you is idempotency, reconciliation or ordering tolerance – those are properties of how you model the work, not of the orchestrator running it.