NestJS Lazy Loading Modules

By default, NestJS loads all modules eagerly at startup, potentially leading to slower initialization and increased memory consumption. However, when dealing with workers that handle different job types with unique dependencies, it’s often more efficient to load modules only when they are actually needed. This is where lazy loading comes in.

In this article, we’ll dive deep into lazy loading in NestJS, explore its benefits, and demonstrate how to implement it effectively within a NestJS worker that leverages the Bull queueing system to manage jobs. You’ll learn how to dynamically load job-specific modules on demand, resulting in faster startup times, reduced memory usage, and improved scalability.

Source Code is available at the end of the article

Lazy loading vs. Eager loading

Lazy loading vs. Eager loading

Choose Lazy Loading When

Large Application with Many Modules: If your application has a complex structure with numerous modules, the initial startup time can be significant if all modules are loaded eagerly. Lazy loading becomes beneficial here, as it only loads modules when they are actually needed, resulting in a faster initial startup.

Serverless Environment: Serverless platforms charge based on execution time. Lazy loading is crucial here because it reduces the cold start time (the time it takes for a function to initialize before executing). By only loading necessary modules for each request, you optimize performance and reduce costs.

Drawbacks:

• Increased Complexity: Requires additional code to manage the lazy loading process, which can make the application slightly more complex.
• Cold Start Delay: The first time a lazily loaded module is used, there might be a slight delay as the module is loaded into memory. This is known as a “cold start.”

LazyModuleLoader

This is a utility class provided by NestJS to facilitate lazy loading. It handles the dynamic loading of modules at runtime. When a lazily loaded module is requested, the LazyModuleLoader fetches it from the file system (or cache) and loads it into memory, making it available for use.

LazyModuleLoader

Scenario

Imagine you have a NestJS worker responsible for processing various types of jobs:

• EmailJob: Sends promotional emails to customers.
• DataProcessingJob: Analyzes large datasets for insights.
• NotificationJob: Pushes notifications to users.

Since each job type has its own dependencies and may not be executed frequently, eager loading all modules at startup could lead to unnecessary resource consumption. Instead, we can use lazy loading to load the required modules only when needed.

Worker responsible for processing various types of jobs

Solution

The provided diagram illustrates a NestJS application using lazy loading for processing various jobs through a queue. The AppModule configures and provides the Queue, which enqueues jobs using the AppService. The Queue then sends job data to the JobProcessorModule, which processes jobs using the JobProcessor. The JobProcessor dynamically loads necessary modules on demand with the help of LazyModuleLoader. These modules include EmailJobModule, DataProcessingJobModule, and NotificationJobModule, each providing their respective services: EmailJobService, DataProcessingJobService, and NotificationJobService. Each service executes its specific job, such as sending emails, processing data, or sending notifications, enhancing efficiency and resource management through lazy loading.

JobProcessor: The Coordinator

In our scenario, the JobProcessor serves as the core of the worker application, performing the following tasks:

• Queue Listening: Subscribes to the Bull queue, awaiting new jobs.
• Job Reception: Receives job data, including job type and payload.
• Job Type Identification: Determines the job type from the received data.
• Lazy Loading Modules: Utilizes LazyModuleLoader to dynamically load the appropriate module (e.g., EmailJobModule) based on the job type.
• Service Retrieval: Retrieves the corresponding service (e.g., EmailJobService) from the loaded module.
• Job Execution: Delegates job execution to the service, passing the job data for processing.

Implementation

src
├── app.controller.ts
├── app.module.ts
├── app.service.ts
├── job-processor
│   ├── job-processor.module.ts
│   └── job-processor.processor.ts
├── jobs
│   ├── data-processing-job
│   │   ├── data-processing-job.module.ts
│   │   └── data-processing-job.service.ts
│   ├── email-job
│   │   ├── email-job.module.ts
│   │   └── email-job.service.ts
│   └── notification-job
│       ├── notification-job.module.ts
│       └── notification-job.service.ts
└── main.ts

JobProcessor Module

The JobProcessorModule class is responsible for importing the JobProcessor class and the BullModule to create a queue for processing jobs.

// src/job-processor/job-processor.module.ts
import { Module } from '@nestjs/common';
import { BullModule } from '@nestjs/bull';
import { JobProcessor } from './job-processor.processor';

@Module({
  imports: [
    BullModule.registerQueue({
      name: 'jobQueue',
    }),
  ],
  providers: [JobProcessor],
})
export class JobProcessorModule {}

The JobProcessor class is responsible for processing different types of jobs. We use dynamic imports to load the required job services lazily when a job is processed.

// src/job-processor/job-processor.processor.ts
import { Process, Processor } from '@nestjs/bull';
import { Job } from 'bull';
import { LazyModuleLoader } from '@nestjs/core';

@Processor('jobQueue')
export class JobProcessor {
  constructor(private readonly lazyModuleLoader: LazyModuleLoader) {}

  @Process('EmailJob')
  async handleEmailJob(job: Job) {
    try {
      const { EmailJobModule } = await import(
        '../jobs/email-job/email-job.module'
        );
      const moduleRef = await this.lazyModuleLoader.load(() => EmailJobModule);
      const { EmailJobService } = await import(
        '../jobs/email-job/email-job.service'
        );
      const emailJobService = moduleRef.get(EmailJobService);
      emailJobService.handleJob(job.data);
    } catch (e) {
      // Do some retry logic here, handling the error
      console.log(e);
    }
  }

  @Process('DataProcessingJob')
  async handleDataProcessingJob(job: Job) {
    console.log('Processing DataProcessingJob...');
    try {
      const { DataProcessingJobModule } = await import(
        '../jobs/data-processing-job/data-processing-job.module'
        );
      const moduleRef = await this.lazyModuleLoader.load(
        () => DataProcessingJobModule,
      );
      const { DataProcessingJobService } = await import(
        '../jobs/data-processing-job/data-processing-job.service'
        );
      const dataProcessingJobService = moduleRef.get(DataProcessingJobService);
      dataProcessingJobService.handleJob(job.data);
    } catch (e) {
      // Do some retry logic here, handling the error
      console.log(e);
    }
  }

  @Process('NotificationJob')
  async handleNotificationJob(job: Job) {
    console.log('Processing NotificationJob...');
    try {
      const { NotificationJobModule } = await import(
        '../jobs/notification-job/notification-job.module'
        );
      const moduleRef = await this.lazyModuleLoader.load(
        () => NotificationJobModule,
      );
      const { NotificationJobService } = await import(
        '../jobs/notification-job/notification-job.service'
        );
      const notificationJobService = moduleRef.get(NotificationJobService);
      notificationJobService.handleJob(job.data);
    } catch (e) {
      // Do some retry logic here, handling the error
      console.log(e);
    }
  }
}

Jobs

Each job type has its own module and service class. The service class contains the logic for processing the job.

EmailJobModule

// src/jobs/email-job/email-job.module.ts
import { Module } from '@nestjs/common';
import { EmailJobService } from './email-job.service';

@Module({
  providers: [EmailJobService],
  exports: [EmailJobService],
})
export class EmailJobModule {}

EmailJobService

import { Injectable } from '@nestjs/common';

@Injectable()
export class EmailJobService {
  handleJob(data: any) {
    console.log(`Handling Email Job with data: ${JSON.stringify(data)}`);
    // Logic for sending promotional emails
  }
}

AppController

The AppController class is responsible for triggering the different types of jobs.

// src/app.controller.ts
import { Controller, Post } from '@nestjs/common';
import { AppService } from './app.service';

@Controller('jobs')
export class AppController {
  constructor(private readonly appService: AppService) {}

  @Post('trigger')
  async triggerJobs() {
    await this.appService.triggerJobs();
    return { message: 'Jobs have been added to the queue' };
  }
}

AppService

The AppService class is responsible for adding jobs to the queue.

// src/app.service.ts
import { Injectable } from '@nestjs/common';
import { InjectQueue } from '@nestjs/bull';
import { Queue } from 'bull';

@Injectable()
export class AppService {
  constructor(@InjectQueue('jobQueue') private readonly jobQueue: Queue) {}

  async triggerJobs() {
    // Adding jobs to the queue
    console.log('Adding jobs to the queue');
    try {
      await this.jobQueue.add('EmailJob', { data: 'some data for email job' });
      await this.jobQueue.add('DataProcessingJob', {
        data: 'some data for data processing job',
      });
      await this.jobQueue.add('NotificationJob', {
        data: 'some data for notification job',
      });
    } catch (error) {
      console.error('Error adding jobs to the queue', error);
    }
  }
}

TEST

Install Redis

If you don’t have Redis installed, you can install it using the following commands:

I am using macOS Homebrew:

brew install redis

Start Redis:

brew services start redis

Verify Redis Installation

redis-cli ping

Start Application

Start the application by running the following command:

npm run start

Trigger Jobs

POST http://localhost:3000/jobs/trigger

Response:

{
  "message": "Jobs have been added to the queue"
}

Logs: should be displayed in the console, indicating that the application has started successfully and is processing the different types of jobs:

[Nest] 45900  - 06/30/2024, 10:45:41 AM     LOG [LazyModuleLoader] EmailJobModule dependencies initialized
Handling Email Job with data: {"data":"some data for email job"}
Processing DataProcessingJob...
[Nest] 45900  - 06/30/2024, 10:45:41 AM     LOG [LazyModuleLoader] DataProcessingJobModule dependencies initialized
Handling Data Processing Job with data: {"data":"some data for data processing job"}
Processing NotificationJob...
[Nest] 45900  - 06/30/2024, 10:45:41 AM     LOG [LazyModuleLoader] NotificationJobModule dependencies initialized
Handling Notification Job with data: {"data":"some data for notification job"}

Conclusion

Lazy loading is beneficial for large, modular applications, optimizing performance and resource use. Eager loading suits smaller applications where critical features must be immediately available. Choosing the right approach depends on application size, usage patterns, and performance requirements. Each strategy has its own benefits and drawbacks to consider. Balancing lazy and eager loading can optimize application performance and resource utilization.

Full Source Code : https://github.com/kelvin-bz/nest-lazy-loading-modules

Read more articles about NestJS : https://kelvinbz.medium.com/list/nestjs-eafb7de3e562

Read more articles about AWS: https://kelvinbz.medium.com/list/aws-32c5d9525c7d