Background Jobs and Queues in Node.js

Some work should not block an HTTP response. Sending a welcome email, resizing an uploaded image, generating a PDF, calling a third-party webhook — all of these are better done out-of-band. Doing them inline adds latency, and a failure in the slow operation causes the whole request to fail. Job queues solve both problems.

The Core Problem

Consider a user registration endpoint that sends a welcome email:

ts
// Without a queue — the response waits for the email
export async function POST(request: Request) {
  const user = await createUser(request);
  await sendWelcomeEmail(user.email); // takes 800ms, can fail
  return Response.json({ id: user.id }, { status: 201 });
}

If the email provider is slow or down, the registration call fails or times out. The user creation succeeded but the response didn't. With a queue:

ts
// With a queue — enqueue and return immediately
export async function POST(request: Request) {
  const user = await createUser(request);
  await emailQueue.add('welcome', { userId: user.id, email: user.email });
  return Response.json({ id: user.id }, { status: 201 });
}

The response is fast and the email delivery is decoupled. If the email fails, the worker retries without affecting the user experience.

BullMQ: The Standard Choice for Node.js

BullMQ is the de-facto queue library for Node.js. It's backed by Redis and handles job persistence, retries, delayed jobs, repeatable jobs, and prioritization.

Setup

ts
// lib/queues.ts
import { Queue, Worker, QueueEvents } from 'bullmq';
import IORedis from 'ioredis';

const connection = new IORedis(process.env.REDIS_URL!, {
  maxRetriesPerRequest: null,
});

export const emailQueue = new Queue('email', { connection });

Adding Jobs

ts
await emailQueue.add(
  'welcome',
  { userId: user.id, email: user.email },
  {
    attempts: 3,
    backoff: { type: 'exponential', delay: 2000 },
    removeOnComplete: 100, // keep last 100 completed jobs
    removeOnFail: 200,
  }
);

Writing a Worker

ts
// workers/emailWorker.ts
import { Worker } from 'bullmq';
import IORedis from 'ioredis';

const connection = new IORedis(process.env.REDIS_URL!, {
  maxRetriesPerRequest: null,
});

const worker = new Worker(
  'email',
  async (job) => {
    if (job.name === 'welcome') {
      await sendWelcomeEmail(job.data.email);
    }
  },
  {
    connection,
    concurrency: 5, // process up to 5 jobs simultaneously
  }
);

worker.on('failed', (job, err) => {
  console.error(`Job ${job?.id} failed:`, err);
});

The worker is a separate process. In production, you run it alongside your web server — often as a separate container or dyno.

Retry Logic and Backoff

Transient failures (rate-limited APIs, temporary network issues) are common. BullMQ's built-in retry with exponential backoff handles these well:

• attempts: 3 — try the job up to 3 times total
• backoff: { type: 'exponential', delay: 2000 } — wait 2s, then 4s, then 8s between attempts

For permanent failures (invalid data, unrecoverable errors), throw an UnrecoverableError to skip retries immediately:

ts
import { UnrecoverableError } from 'bullmq';

async function processJob(job: Job) {
  const result = validateJobData(job.data);
  if (!result.success) {
    throw new UnrecoverableError(`Invalid data: ${result.error}`);
  }
  // ... process
}

This mirrors the error handling philosophy in error handling patterns in Node.js APIs — distinguish between retryable and fatal errors early.

Delayed and Scheduled Jobs

BullMQ handles delayed execution and cron-style recurring jobs natively:

ts
// Run once after a 24-hour delay
await emailQueue.add(
  'followup',
  { userId: user.id },
  { delay: 24 * 60 * 60 * 1000 }
);

// Recurring job — every day at 9am
await reportQueue.add(
  'daily-report',
  {},
  { repeat: { pattern: '0 9 * * *' } }
);

This replaces cron scripts that run node script.js on a schedule — the queue handles persistence and ensures jobs don't pile up if the worker is temporarily down.

Serverless Considerations

Long-running workers don't fit naturally in serverless environments. A few approaches work well:

Separate worker process: Deploy workers on a persistent host (a single EC2 instance, a Railway service, a Fly.io machine) while keeping the web tier serverless. Workers are stateful by nature and benefit from a persistent process.

Short jobs via serverless triggers: For jobs that complete in under a few seconds, you can trigger a serverless function (via a webhook or queue-backed trigger) rather than maintaining a long-running worker. Platforms like Inngest or Trigger.dev abstract this.

The mismatch between serverless and long-running workers is a recurring theme in the future of serverless architecture.

Observability

A job queue you can't observe is a liability. BullMQ has a companion dashboard, Bull Board, that provides job visibility:

ts
import { createBullBoard } from '@bull-board/api';
import { BullMQAdapter } from '@bull-board/api/bullMQAdapter';
import { ExpressAdapter } from '@bull-board/express';

const serverAdapter = new ExpressAdapter();
createBullBoard({
  queues: [new BullMQAdapter(emailQueue), new BullMQAdapter(reportQueue)],
  serverAdapter,
});

app.use('/admin/queues', authMiddleware, serverAdapter.getRouter());

At minimum, log job IDs, names, and failure reasons. Alert on abnormal failure rates. This fits into the broader observability picture described in monitoring and observability for web apps.

Conclusion

Job queues are one of the highest-leverage architectural additions you can make to a Node.js backend. The pattern is straightforward: enqueue in the request handler, process in a worker, retry on failure. BullMQ handles the hard parts — persistence, retries, scheduling — so you can focus on the business logic. Start with email and notifications, then expand to any slow or failure-prone work. If you're curious how this fits into a larger project structure, see how I structure large Next.js and Node.js projects or reach out via the contact page.