Webhook Ingestion at Scale with BullMQ and Redis

The general rules for webhook pipelines (acknowledge fast, be idempotent, retry into a dead-letter queue, reconcile) are covered in Designing Resilient Multi-Platform Data Pipelines with n8n and Webhooks. This article is narrower: how to implement the ingestion tier in Node.js with BullMQ and Redis, which BullMQ features do the heavy lifting, and the configuration mistakes that lose events under load.

Code below assumes BullMQ 5.x, ioredis and Express on Node.js 20+. The same structure works with Fastify or a serverless receiver, as long as the receiver can hold a Redis connection.

The shape of the system

A provider retries a webhook when it does not get a 2xx response in time. So the receiver’s only job is to make the event durable as quickly as possible and say so. Everything slow moves behind the queue.

The important property is that the receiver and the workers scale independently. A burst of 50,000 deliveries becomes a deep queue that drains at the rate your CRM or ERP will tolerate, instead of 50,000 concurrent calls that trip its rate limiter and fail.

Redis settings that are not optional

BullMQ’s going-to-production guide is short and worth following line by line. The settings that matter most for ingestion:

• maxmemory-policy noeviction. The docs call it the only setting that guarantees correct queue behaviour. With any eviction policy, Redis under memory pressure silently deletes keys, and a deleted job key is a lost webhook. With noeviction, writes fail loudly instead, the receiver returns an error and the provider retries.
• AOF persistence. The guide suggests fsync about once per second is enough for most applications. Without persistence, a Redis restart drops everything you already acknowledged.
• Separate connection settings for producers and workers. Disable enableOfflineQueue on the receiver’s Queue so that queue.add fails fast while Redis is unreachable. Leave it on for workers, and keep maxRetriesPerRequest: null for them (the Worker default), so they wait for the connection to come back.
• Managed Redis caveats. Check that your provider lets you set the eviction policy. On Redis Cluster, BullMQ needs a hash tag in the queue prefix (for example {webhooks}) so a queue’s keys share a slot. See the Redis Cluster pattern.

The receiver: verify, enqueue, acknowledge

// receiver.ts: BullMQ 5.x, ioredis, Express
import express from 'express';
import crypto from 'node:crypto';
import IORedis from 'ioredis';
import { Queue } from 'bullmq';

const connection = new IORedis(process.env.REDIS_URL!, {
  enableOfflineQueue: false, // fail fast so we can return 503
});

export const webhooks = new Queue('webhooks', {
  connection,
  defaultJobOptions: {
    attempts: 8,
    backoff: { type: 'exponential', delay: 2000, jitter: 0.5 },
    removeOnComplete: { age: 24 * 3600 }, // keeps the jobId (and dedup) for 24h
    removeOnFail: { age: 14 * 24 * 3600 },
  },
});
webhooks.on('error', (err) => console.error('queue error', err));

// BullMQ job ids must not contain ':' or be digits only.
const jobIdFor = (source: string, deliveryId: string) =>
  `${source}_${crypto.createHash('sha256').update(deliveryId).digest('hex')}`;

const app = express();

app.post(
  '/webhooks/:source',
  express.raw({ type: '*/*', limit: '1mb' }), // raw bytes: needed for signatures
  async (req, res) => {
    const source = req.params.source;
    const raw = req.body as Buffer;

    if (!verifySignature(source, req.headers, raw)) return res.sendStatus(401);

    const deliveryId = getDeliveryId(source, req.headers, raw);
    try {
      await webhooks.add(
        source,
        { source, body: raw.toString('utf8'), receivedAt: Date.now() },
        { jobId: jobIdFor(source, deliveryId) },
      );
      return res.sendStatus(202);
    } catch (err) {
      // Not durable, so do not acknowledge. The provider will retry.
      return res.sendStatus(503);
    }
  },
);

verifySignature and getDeliveryId are per provider. Most send an HMAC or a public-key signature over the raw body plus a delivery or event id header. Two details matter. Verify against the raw bytes, not re-serialised JSON. And use a constant-time comparison. The Shopify app development guide has a worked HMAC example.

Returning 503 when queue.add throws is the line that makes the system safe. If the receiver returns 2xx before the job is in Redis, a crash or Redis outage at that moment loses the event. The provider has no reason to retry it.

Idempotency: what jobId does and does not give you

BullMQ ignores an add whose custom jobId already exists in the queue (see Job IDs). Using the provider’s delivery id as the jobId therefore collapses redeliveries into one job, but only while that job is still stored. Once auto-removal deletes it, the same id can be added again.

The deduplication option answers a different question from jobId. A jobId says “this exact delivery”. A deduplication id says “work for this entity”. For a stream of contact.updated events, debounce mode keyed on the contact id often removes most of the work:

await webhooks.add('contact-sync', { contactId }, {
  deduplication: { id: `contact_${contactId}`, ttl: 5000, extend: true, replace: true },
  delay: 5000,
});

Treat both as optimisations. The guarantee belongs in the worker: record the delivery id under a unique constraint in the same transaction as the side effect, or make the side effect an upsert. Retention bounds memory, and memory is exactly what you cannot let run out with noeviction. So the queue’s dedup window will always be shorter than a provider’s worst-case redelivery.

Workers: concurrency, rate limits and failure classes

// worker.ts
import IORedis from 'ioredis';
import { Worker, UnrecoverableError } from 'bullmq';

const connection = new IORedis(process.env.REDIS_URL!, { maxRetriesPerRequest: null });

const worker = new Worker('webhooks', async (job) => {
  const event = JSON.parse(job.data.body);
  const res = await deliverDownstream(event); // your API call, idempotent

  if (res.status === 429) {
    const retryAfterMs = Number(res.headers.get('retry-after') ?? 10) * 1000;
    await worker.rateLimit(retryAfterMs);   // pauses the whole queue
    throw Worker.RateLimitError();          // back to waiting, not a failed attempt
  }
  if (res.status >= 400 && res.status < 500) {
    throw new UnrecoverableError(`rejected ${res.status}`); // no retries
  }
  if (!res.ok) throw new Error(`downstream ${res.status}`); // retried with backoff
}, {
  connection,
  concurrency: 20,
  limiter: { max: 50, duration: 1000 }, // global across all workers on this queue
});

worker.on('failed', (job, err) => { /* alert when attemptsMade === opts.attempts */ });
worker.on('error', (err) => console.error('worker error', err));

for (const sig of ['SIGINT', 'SIGTERM'] as const) {
  process.on(sig, async () => { await worker.close(); process.exit(0); });
}

Points worth knowing from the rate limiting docs:

• The limiter is global per queue. Ten workers with max: 50, duration: 1000 still process 50 jobs a second in total. Set it from the downstream API’s published limit, not from your worker count.
• Rate-limited jobs stay waiting, they do not fail. Manual rate limiting with worker.rateLimit() only works when limiter is set on the worker.
• Group keys were removed in BullMQ 3.0. For per-tenant fairness in open-source BullMQ, use a queue per destination or tenant class. BullMQ Pro has groups that round-robin between tenants.

Classify failures before you retry. A 503 or a timeout is transient and deserves exponential backoff with jitter. A 422 because the payload references a deleted record will fail identically on every attempt, so UnrecoverableError sends it straight to the failed set. That failed set is your dead-letter queue. Keep failed jobs long enough to inspect and replay them, and alert on its size.

Graceful shutdown matters on Kubernetes and other orchestrators. A worker killed mid-job leaves that job stalled until another worker picks it up, by default after about 30 seconds. So the downstream call may run twice, which is one more reason the handler must be idempotent.

Ordering

With concurrency above 1, jobs finish out of order, and providers do not promise delivery order anyway. Prefer handlers that upsert from current state, or that treat the event as a hint and fetch the latest record. If an entity genuinely needs serial processing, the debounced deduplication pattern above means at most one pending job per entity. The same reasoning about contention appears in preventing inventory overselling, where the queue serialises writes to a hot row.

What to measure

• Queue depth and age of the oldest waiting job. Depth alone is ambiguous. Age tells you whether you are keeping up.
• Redis used_memory against maxmemory. With noeviction, reaching the limit means the receiver starts returning 503. That is safe but needs a page.
• Receiver 503 rate and p99 latency. The receiver should stay fast even when workers are behind.
• Failed-set growth by error class. A spike in one class usually means a downstream contract changed.
• Deduplicated adds. A sudden rise means a provider is retrying, often because your receiver got slow.

Checklist

• Receiver reads the raw body, verifies the signature in constant time, and does nothing else before queue.add.
• Receiver returns 2xx only after queue.add resolves, and 503 when it throws.
• Redis uses noeviction and AOF, and is not shared with a cache that expects eviction.
• jobId is derived from the provider delivery id, with no colons and not purely numeric.
• removeOnComplete and removeOnFail are set, because the default keeps every job forever.
• Workers set limiter from the downstream limit, handle 429 with worker.rateLimit, and throw UnrecoverableError for permanent 4xx failures.
• Side effects are idempotent at the database layer.
• Workers close on SIGTERM, and the failed set is alerted on and replayable.

If the downstream is a CRM with strict per-account limits, the worker design changes a little. Architecting resilient GoHighLevel CRM sync pipelines applies this pattern to HighLevel’s burst and daily limits.

Questions people actually ask

Should I use BullMQ’s jobId or the deduplication option for webhooks?

Use jobId set from the provider’s delivery id to collapse retries of the same delivery. Use deduplication when you want one pending job per entity regardless of how many events arrive. Neither replaces a unique constraint at the point where the side effect happens.

Can I use Redis with allkeys-lru for BullMQ?

No. BullMQ’s production guide says noeviction is the only policy that guarantees correct queue behaviour. An LRU policy will delete job keys under memory pressure, which means silently losing events you have already acknowledged.

Is BullMQ enough, or do I need Kafka or a managed webhook gateway?

BullMQ suits job-style processing, where each event is handled once with retries and rate limits. If you need long retention, replay from an offset, or several independent consumers of the same stream, a log such as Kafka fits better. A managed gateway trades operational work for cost and vendor dependency.