Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exception crashing the whole app when broker is killed #690

Closed
Tychus opened this issue Jun 1, 2022 · 3 comments
Closed

Exception crashing the whole app when broker is killed #690

Tychus opened this issue Jun 1, 2022 · 3 comments

Comments

@Tychus
Copy link

Tychus commented Jun 1, 2022

Opening this again as i'm having the same issue.

I'm handling close error and blocked everything works fine when the broker is closed properly and the app will keep trying to reconnect as expected. But when i kill the broker (i'm running it through docker) i get both the error and close events as described in the doc but the app will crash right after.

  private async connect(): Promise<void> {
    if (this.connected) {
      return;
    }
    await this.connectionLock.runExclusive(async () => {
      if (this.connected) {
        return;
      }
      await pRetry(
        async () => {
          logger.info('Attempting rabbitMQ connection');
          this.connection = await client.connect(this.connectionSettings.getConnectionString());
        },
        {
          onFailedAttempt: (error) => {
            logger.error(`RabbitMq connection failed. Attempt ${error.attemptNumber}. Error: ${error.message}`);
            if (error.retriesLeft === 0) {
              this.connected = false;
              throw Error(`Failed to establish rabbitMq connection after ${this.maxConnectionRetry} attempts`);
            }
          },
          factor: 2,
          minTimeout: 1000,
          maxTimeout: 5000,
          retries: this.maxConnectionRetry,
        },
      );
      logger.info('RabbitMq connection successful');
      this.setEventListeners();
      this.eventEmitter.emit('connected');
      this.connected = true;
    });
  }

private setEventListeners(): void {
    this.connection.on('close', async (message) => {
      logger.warn(`RabbitMq connection lost: ${message}`);
      this.eventEmitter.emit('disconnected');
      await this.handleErrorEvents();
    });

    this.connection.on('error', async (error) => {
      logger.error(`RabbitMq connection error: ${error}`);
      this.eventEmitter.emit('error');
      await this.handleErrorEvents();
    });

    this.connection.on('blocked', async (reason) => {
      logger.warn(`RabbitMq connection blocked: ${reason}`);
      this.eventEmitter.emit('blocked');
      await this.disconnect();
      await this.handleErrorEvents();
    });
}

private async handleErrorEvents(): Promise<void> {
    this.connected = false;
    await this.connect();
  }

async disconnect(): Promise<void> {
  await this.connection.close();
  this.eventEmitter.emit('disconnected');
  this.connected = false;
}

Exception looks something like this:

[Node] 2022-06-01T08:23:12.546Z [error]: RabbitMq connection error: Error: Unexpected close ( connection receiving error event)
[Node] 2022-06-01T08:23:12.549Z [warn]: RabbitMq connection lost: Error: Unexpected close (connection receiving close event)
[Node] 2022-06-01T08:23:12.551Z [warn]: Bus connection lost. saving events for publishing on reconnection (publisher handling disconnect)
[Node] 2022-06-01T08:23:12.555Z [info]: Attempting rabbitMQ connection
[Node] Error: Unhandled error. (undefined)
[Node]     at new NodeError (node:internal/errors:372:5)
[Node]     at EventEmitter.emit (node:events:516:17)
[Node]     at EventEmitter.emit (node:domain:475:12)
[Node]     at ChannelModel.<anonymous> (/home/ty/Projects/rmq/node_modules/rmqlib/dist/messaging/rabbitMqBus/rabbitMqConnection.js:80:31)
[Node]     at ChannelModel.emit (node:events:527:28)
[Node]     at ChannelModel.emit (node:domain:475:12)
[Node]     at Connection.emit (node:events:527:28)
[Node]     at Connection.emit (node:domain:475:12)
[Node]     at Connection.C.onSocketError (/home/ty/Projects/rmq/node_modules/amqplib/lib/connection.js:353:10)
[Node]     at Socket.emit (node:events:539:35)
[Node]     at Socket.emit (node:domain:475:12)
[Node]     at endReadableNT (node:internal/streams/readable:1345:12)
[Node]     at processTicksAndRejections (node:internal/process/task_queues:83:21)
[Node] [nodemon] app crashed - waiting for file changes before starting...

node -v -> v16.15.0
PS: excuse the code i'm no TS/JS dev just trying to make things work

Originally posted by @Tychus in #338 (comment)

@cressie176
Copy link
Collaborator

   this.connection.on('error', async (error) => {
      logger.error(`RabbitMq connection error: ${error}`);
      this.eventEmitter.emit('error'); // <-- This will crash your application
      await this.handleErrorEvents();
    });

You're re-emitting the the error event.

If an EventEmitter does not have at least one listener registered for the 'error' event, and an 'error' event is emitted, the error is thrown, a stack trace is printed, and the Node.js process exits.

https://nodejs.org/api/events.html#error-events

@Tychus
Copy link
Author

Tychus commented Jun 2, 2022

The more your know. I wasn't actually listening to that event in the start because the lib will send a disconnect after it but it kept crashing the app and i found issues here similar. Thanks for taking the time. Closing the issue

@Tychus Tychus closed this as completed Jun 2, 2022
@AmosSpark
Copy link

   this.connection.on('error', async (error) => {
      logger.error(`RabbitMq connection error: ${error}`);
      this.eventEmitter.emit('error'); // <-- This will crash your application
      await this.handleErrorEvents();
    });

You're re-emitting the the error event.

If an EventEmitter does not have at least one listener registered for the 'error' event, and an 'error' event is emitted, the error is thrown, a stack trace is printed, and the Node.js process exits.

https://nodejs.org/api/events.html#error-events

Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants