"Failed to get $dbstats for database XXX: canceled while checking out a connection from connection pool: context canceled; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 0" "Failed to get $dbstats for database XXX: context deadline exceeded"

[moluo0]

Hello, let me first say sorry. I am using Google Translate to ask questions. The description of the question may not be very clear. Please understand. My English is not that good.

Describe the bug
When I use the following mongodb-exporter-deployment to connect two other identical databases with different data volumes, no error will be reported if the data volume is small. I have created several Databases and Collections and they can be collected normally. However, when the amount of data is large, some indicators cannot be obtained, and an error is reported in the log:
"Failed to get $dbstats for database worksheetdb: canceled while checking out a connection from connection pool: context canceled; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 0"
"Failed to get $dbstats for database worksheetdb: context deadline exceeded"
To Reproduce
Steps to reproduce the behavior:

1. describe steps to reproduce the issue

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongodb-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: mongodb-exporter
  replicas: 1
  template:
    metadata:
      labels:
        app: mongodb-exporter
      annotations:
        prometheus.io/port: "9216"
        prometheus.io/scrape: "true"
    spec:
      containers:
        - name: mongodb-exporter
          imagePullPolicy: IfNotPresent
          image: percona/mongodb_exporter:0.40.0
          env:
            - name: MONGODB_URI
              valueFrom:
                secretKeyRef:
                  name: mongodb-secret-test
                  key: datasource
          args:
            - "--discovering-mode"
            - "--compatible-mode"
            - "--collect-all"
            - '--mongodb.collstats-colls=config.App,config.ApplicationOverride,config.Caller,config.CommonRules,config.HabilitationOverride,config.IconSvg,config.UpdateStatusEntity,config.feature'
            # - "--log.level=debug"
            - "--mongodb.connect-timeout-ms=30000"
          ports:
            - containerPort: 9216
              name: metric-port
          resources:
            limits:
              memory: "256Mi"
              cpu: "256m"
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/scrape: 'true'
    prometheus.io/port: "9216"
  name: mongodb-exporter-service
  namespace: monitoring
spec:
  selector:
    app: mongodb-exporter
  type: ClusterIP
  ports:
    - name: metric-port
      port: 9216
      protocol: TCP
      targetPort: 9216
      # nodePort: 31082
---
apiVersion: v1
kind: Secret
metadata:
    name: mongodb-secret-test
    namespace: monitoring
type: Opaque
stringData:
    datasource: "mongodb://USER:PASSWORD@{host1}:27017/admin" 

Expected behavior
The log does not report errors and all indicators can be collected normally.

Logs
I have omitted some duplicate logs to improve readability

level=info ts=2024-02-04T03:43:06.832Z caller=tls_config.go:274 msg="Listening on" address=[::]:9216
level=info ts=2024-02-04T03:43:06.833Z caller=tls_config.go:277 msg="TLS is disabled." http2=false address=[::]:9216
time="2024-02-04T03:43:15Z" level=error msg="cannot get $indexStats cursor for collection modeldb.system.profile: (Unauthorized) not authorized on modeldb to execute command { aggregate: \"system.profile\", pipeline: [ { $indexStats: {} } ], cursor: {}, lsid: { id: UUID(\"01b14739-f158-4aff-9d16-4fd9aa3ad3e8\") }, $db: \"modeldb\" }"
time="2024-02-04T03:43:16Z" level=error msg="cannot get $indexStats cursor for collection basedb.system.profile: (Unauthorized) not authorized on basedb to execute command { aggregate: \"system.profile\", pipeline: [ { $indexStats: {} } ], cursor: {}, lsid: { id: UUID(\"01b14739-f158-4aff-9d16-4fd9aa3ad3e8\") }, $db: \"basedb\" }"
time="2024-02-04T03:43:17Z" level=error msg="cannot get $indexStats cursor for collection formcenter.system.profile: (Unauthorized) not authorized on formcenter to execute command { aggregate: \"system.profile\", pipeline: [ { $indexStats: {} } ], cursor: {}, lsid: { id: UUID(\"01b14739-f158-4aff-9d16-4fd9aa3ad3e8\") }, $db: \"formcenter\" }"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection piledb.scenes: connection(192.168.1.145:37017[-13]) incomplete read of message header: context deadline exceeded"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection piledb.settings: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 0"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection piledb.taskentities: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 0"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection piledb.taskpackages: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 0"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection piledb.volumeweights: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 0"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection csidb.accesssettings: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 0"
···
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection airdb.weathers: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 1"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection airdb.winds: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 1"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection branchdb.branches: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 1"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection branchdb.branchtemplates: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 1"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection branchdb.branchtypes: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 1"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection branchdb.branchtypetemplates: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 1"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection branchdb.classifications: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 1"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection branchdb.classificationtemplates: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 1"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection branchdb.inspectionlots: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 1"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection branchdb.linkmodels: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 1"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection operatingdb.mqcollections: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 2"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection yzstore_private_serverdb.appkeys: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 2"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection yzstore_private_serverdb.mqcollections: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 2"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection yzstore_private_serverdb.template_buys: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 2"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection yzstore_private_serverdb.template_uploads: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 2"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection drawingdb.archivesettings: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 2"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection drawingdb.departmentgroups: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 2"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection drawingdb.drawingtypes: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 2"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection drawingdb.drawingunits: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 2"
time="2024-02-04T03:43:18Z" level=error msg="cannot get $indexStats cursor for collection drawingdb.exportrecords: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 2"
···
time="2024-02-04T03:43:20Z" level=error msg="cannot get $indexStats cursor for collection thirdpartyauthdb.system.profile: (Unauthorized) not authorized on thirdpartyauthdb to execute command { aggregate: \"system.profile\", pipeline: [ { $indexStats: {} } ], cursor: {}, lsid: { id: UUID(\"01d9a45c-b6ea-444d-a9d5-14d8ce78130e\") }, $db: \"thirdpartyauthdb\" }"
time="2024-02-04T03:43:22Z" level=error msg="cannot get $indexStats cursor for collection formcenter.system.profile: (Unauthorized) not authorized on formcenter to execute command { aggregate: \"system.profile\", pipeline: [ { $indexStats: {} } ], cursor: {}, lsid: { id: UUID(\"f5d21336-0a4c-45b7-92ce-8dd540f7c5a9\") }, $db: \"formcenter\" }"
time="2024-02-04T03:43:22Z" level=error msg="cannot get $indexStats cursor for collection workflowdb.system.profile: (Unauthorized) not authorized on workflowdb to execute command { aggregate: \"system.profile\", pipeline: [ { $indexStats: {} } ], cursor: {}, lsid: { id: UUID(\"f5d21336-0a4c-45b7-92ce-8dd540f7c5a9\") }, $db: \"workflowdb\" }"
time="2024-02-04T03:43:22Z" level=error msg="cannot get $indexStats cursor for collection mixindb.mqcollections: context deadline exceeded"
time="2024-02-04T03:43:22Z" level=error msg="cannot get $indexStats cursor for collection moduledb.customformdatas: context deadline exceeded"
time="2024-02-04T03:43:22Z" level=error msg="cannot get $indexStats cursor for collection moduledb.customtemplatedatas: context deadline exceeded"
time="2024-02-04T03:43:22Z" level=error msg="cannot get $indexStats cursor for collection moduledb.customtemplatefields: context deadline exceeded"
time="2024-02-04T03:43:22Z" level=error msg="cannot get $indexStats cursor for collection moduledb.forms: context deadline exceeded"
time="2024-02-04T03:43:22Z" level=error msg="cannot get $indexStats cursor for collection moduledb.fromtemplates: context deadline exceeded"
time="2024-02-04T03:43:22Z" level=error msg="cannot get $indexStats cursor for collection moduledb.mqcollections: context deadline exceeded"
time="2024-02-04T03:43:22Z" level=error msg="cannot get $indexStats cursor for collection moduledb.sheetformdatas: context deadline exceeded"
time="2024-02-04T03:43:22Z" level=error msg="cannot get $indexStats cursor for collection moduledb.sheettemplatedatas: context deadline exceeded"
···

Environment

• Centos 7.9 uname 3.10.0-1160.62.1.el7.x86_64 ,
• k8s v1.24.10
• MongoDB version v4.0.28

Additional context
Add any other context about the problem here.
I also tried to collect the database separately through the --collector.dbstats parameter. There was no error in the execution, but there were also database collection errors in the logs collected through the --collector.all parameter.

[achar007]

any workarounds or any plans when fix will be applied ?

[Reborns]

Failed to get $dbstats for database XXX: context deadline exceeded"
We have the same problem, we have about 600 databases in a sharded cluster.
Tested with version 0.41.0
Also, if we enable "--discovering-mode" we get another error
cannot get $collstats cursor for collection XXX: context deadline exceeded collector=collstats
cannot get $collstats cursor for collection XXX : timed out while checking out a connection from connection pool: context deadline exceeded; total connections: 1, maxPoolSize: 100, idle connections: 0, wait duration: 207ns collector=collstats

[marcofoyer]

I have the same problem with version 0.40.0.

[BupycHuk]

We recommend to increase scrape intervals or disable dbstats and colstats collectors