Background Jobs with BullMQ

In Caching with Redis in NestJS we used Redis to avoid recomputing things. This time we use it to avoid doing things right now. Once your API starts sending emails, generating PDFs, resizing images, or calling a slow third-party service, you hit a wall. Some work simply takes too long to do inside a request-response cycle. Forcing it there makes your API feel unreliable even when the actual business logic is correct.

Background jobs move that work off the request path. The API accepts the request, hands the slow part to a queue, and responds immediately. A worker process picks the job up whenever it's ready and does the actual work, with retries, backoff, and visibility built in instead of bolted on. BullMQ is the current standard for this in the Node.js ecosystem, and NestJS has first-class support for it through @nestjs/bullmq.

This is part 3 of the NestJS Production Guide series. Nothing here is exotic. It's the kind of infrastructure that quietly prevents a lot of production pain once you cross a certain scale, and it's worth understanding before you actually need it under pressure.

Why background jobs matter

A synchronous request handler ties three things together whether you want it or not: accepting the work, doing the work, and returning a result. That coupling is fine for a database lookup that takes a few milliseconds. It's a liability for anything that depends on an external system, does real CPU work, or might legitimately take seconds instead of milliseconds.

Consider a signup flow that sends a welcome email through a transactional email provider. If you call that provider inline, your signup endpoint's latency and reliability are now hostage to a service you don't control. If their API is slow, your signup is slow. If their API is down, your signup fails, even though the account was created successfully in your own database.

Decoupling fixes this cleanly. The request creates the user, enqueues a job, and returns. The worker sends the email whenever the provider cooperates, retrying with backoff if it doesn't. The user gets a fast signup response regardless of what's happening downstream, and the email either arrives a second later or a minute later after a retry. Either way, the core business action already succeeded.

The same pattern applies to report generation, image processing, webhook delivery, batch data exports, and anything else where the caller doesn't need to wait for the result synchronously.

Setting up BullMQ in NestJS

BullMQ is built on Redis and uses Redis's data structures (lists, sorted sets, hashes) to implement reliable queues with delayed jobs, retries, rate limiting, and job priorities. The NestJS integration wraps this in decorators and DI so it feels native to the framework rather than tacked on.

Install the packages:

npm install @nestjs/bullmq bullmq

Register the queue connection in a shared module. Point it at the same Redis instance you're already running for caching or sessions, or use a dedicated instance if job volume is high enough that you don't want queue traffic competing with cache traffic.

// queue.module.ts
import { Module } from '@nestjs/common';
import { BullModule } from '@nestjs/bullmq';

@Module({
  imports: [
    BullModule.forRoot({
      connection: {
        host: process.env.REDIS_HOST,
        port: Number(process.env.REDIS_PORT),
        password: process.env.REDIS_PASSWORD,
      },
    }),
    BullModule.registerQueue({
      name: 'email',
    }),
  ],
  exports: [BullModule],
})
export class QueueModule {}

Producing a job is a matter of injecting the queue and calling add. Keep the payload small and serializable: pass IDs, not full entities, since BullMQ stores the job data as JSON in Redis and you don't want stale or oversized objects sitting there.

// users.service.ts
import { Injectable } from '@nestjs/common';
import { InjectQueue } from '@nestjs/bullmq';
import { Queue } from 'bullmq';

@Injectable()
export class UsersService {
  constructor(@InjectQueue('email') private readonly emailQueue: Queue) {}

  async registerUser(email: string, name: string): Promise<void> {
    // create the user record first; the job only fires on success
    await this.emailQueue.add(
      'welcome-email',
      { email, name },
      {
        attempts: 5,
        backoff: { type: 'exponential', delay: 2000 },
        removeOnComplete: 1000,
        removeOnFail: 5000,
      },
    );
  }
}

The worker side is a @Processor, which is where the actual side effect happens. Keep processors idempotent wherever you can: a job that gets retried after a partial failure should not double-send an email or double-charge a card.

// email.processor.ts
import { Processor, WorkerHost } from '@nestjs/bullmq';
import { Job } from 'bullmq';
import { Logger } from '@nestjs/common';

@Processor('email')
export class EmailProcessor extends WorkerHost {
  private readonly logger = new Logger(EmailProcessor.name);

  async process(job: Job<{ email: string; name: string }>): Promise<void> {
    const { email, name } = job.data;

    switch (job.name) {
      case 'welcome-email':
        this.logger.log(`Sending welcome email to ${email}`);
        await this.sendWelcomeEmail(email, name);
        break;
      default:
        this.logger.warn(`Unhandled job name: ${job.name}`);
    }
  }

  private async sendWelcomeEmail(email: string, name: string): Promise<void> {
    // call your transactional email provider here
  }
}

Register the processor in the same module as the queue, and run it either in the same process as your API for low volume, or as a separate deployable service once job throughput justifies its own scaling profile.

// email.module.ts
import { Module } from '@nestjs/common';
import { BullModule } from '@nestjs/bullmq';
import { EmailProcessor } from './email.processor';

@Module({
  imports: [BullModule.registerQueue({ name: 'email' })],
  providers: [EmailProcessor],
})
export class EmailModule {}

Retries, backoff, and dead letter handling

The retry configuration on a job is not a formality. It's the difference between "the email provider had a five-second blip and the job recovered" and "the job silently failed and nobody noticed for a week." Exponential backoff with a handful of attempts covers most transient failures: rate limits, brief network partitions, a downstream service restarting.

What backoff does not solve is a job that fails deterministically, such as a malformed payload or a permanently invalid email address. Those jobs will exhaust their attempts and land in the failed state, and if nothing is watching that state, they just sit there. Treat the failed job list the same way you'd treat an error log: something needs to alert on it and something needs to look at it periodically.

// queue-events.processor.ts
import { Injectable, Logger } from '@nestjs/common';
import { QueueEventsListener, QueueEventsHost, OnQueueEvent } from '@nestjs/bullmq';

@Injectable()
@QueueEventsListener('email')
export class EmailQueueEvents extends QueueEventsHost {
  private readonly logger = new Logger(EmailQueueEvents.name);

  @OnQueueEvent('failed')
  onFailed({ jobId, failedReason }: { jobId: string; failedReason: string }) {
    this.logger.error(`Job ${jobId} failed permanently: ${failedReason}`);
    // ship this to your alerting/observability stack, not just the logs
  }
}

A common mistake is setting attempts very high to "make failures go away." That doesn't fix anything; it just delays the point where a genuinely broken job gets noticed, and in the meantime it keeps hammering a downstream service that's already struggling. A moderate retry count with real alerting on the failed state is more honest and more useful than an aggressive retry count with no observability.

Concurrency, rate limiting, and separating workers from the API

By default, a worker processes jobs one at a time per process. You can raise concurrency per processor, but concurrency should match what the job actually does, not an arbitrary number. A job that's mostly waiting on network I/O, like calling an external API, can usually run at higher concurrency than a job doing real CPU work like image transcoding. Push concurrency too high on CPU-bound work and it just contends for the same cores.

@Processor('email', { concurrency: 10 })
export class EmailProcessor extends WorkerHost {
  // ...
}

If your jobs call a rate-limited third-party API, configure the queue's rate limiter instead of relying on retries to smooth over 429 responses. BullMQ supports this natively, and it's a better fit than client-side throttling scattered across your codebase.

BullModule.registerQueue({
  name: 'email',
  limiter: {
    max: 100,
    duration: 1000,
  },
});

The other decision worth making early is whether your worker runs in the same Node process as your API or as a separate service. Running them together is simpler to deploy and is a reasonable default while job volume is low: one Docker image, one deployment, one thing to monitor. Once job volume grows to the point where worker CPU usage starts affecting API latency, or you need to scale workers independently of API replicas, split them into separate processes that both connect to the same Redis instance and queue names. This is a scaling decision, not an architectural one you need to get right on day one.

Idempotency and job design

The single most important property of a good job handler is idempotency: running the same job twice, with the same data, should not cause a different or worse outcome than running it once. This matters because BullMQ, like any at-least-once delivery system, can redeliver a job. A worker can crash after completing the actual work but before acknowledging the job, and the job will be retried on restart.

For a welcome email, running twice is mildly annoying but survivable. For a job that charges a payment method or decrements inventory, running twice without protection is a real bug. Guard against this with a deterministic operation key: store a record of "this job ID already completed" in your database, or use INSERT ... ON CONFLICT DO NOTHING semantics so a duplicate execution is a no-op rather than a duplicate effect.

CREATE TABLE processed_jobs (
  job_id TEXT PRIMARY KEY,
  processed_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

async process(job: Job): Promise<void> {
  const alreadyProcessed = await this.jobsRepo.exists(job.id);
  if (alreadyProcessed) {
    return;
  }

  await this.doTheActualWork(job.data);
  await this.jobsRepo.markProcessed(job.id);
}

This is more code than the naive version, but it's the difference between a queue you trust and one you have to babysit.