Error pinging mysqld" err="context deadline exceeded

[MilkTeaNo7]

I'm really pushed to the wall, it's driving me crazy.

Troubleshooting Steps:

1. In the Prometheus backend, the node is online, and Targets can be seen.
   • Upon clicking in, it returns around 2000 data points.
   • Adding scrape_interval: 5m, scrape_timeout: 1m in Prometheus configuration allows a small portion of data to appear, but the error in mysqld_exporter persists.
2. Service restarts are ineffective, as are server restarts.
   • mysqld_exporter
   • Prometheus
   • Grafana
3. /metrics data can be obtained, tested with curl http://10.0.0.9:9104/metrics
   • On the server, it can be obtained.
   • On the mysqld_exporter local machine, it can be obtained.
   • The data volume is normal with around 2000 metrics.
   • The response time takes several seconds.
4. The mysqld_exporter service configuration file .my.cnf is correct.
   • The same configuration works on other servers.
   • Increasing the timeout with connect_timeout=10 is ineffective.
   • Changing the local address to host=localhost is ineffective.
5. The database account is correct.
   • Remote tools test the account with no issues.
   • The account has % privileges as well as local privileges.
   • Creating a new account with root privileges does not resolve the issue.
   • The account password does not contain special characters such as #.
6. The server can Ping the client machine, and the firewalls are all disabled.
7. The port is normal, port number 9104, not occupied.
8. The problematic client and the non-problematic client have the same version, both are go1.21.5.

[aliensmily]

same
OS: Ubuntu 24.04.1 LTS
DB: MariaDB 10.6.18

Content of myexporter.cnf

[client]
socket=/opt/mysql/admin/TESTDB1/config/mysql.sock
port=3306
user=MON_USER1

[aliensmily]

What i found out till now:

1. its a timeout, query the mysql db
2. when i call curl http://myhost:9104/metrics | less at the db host, i got immediately a response... when i call curl http://myhost:9104/metrics?target=myhost:9104 | less (like Prometheus) the call does not come to an end.
3. --log.level=debug doesn't write out more then "level=error msg="Error pinging mysqld" err="context canceled""
4. Ports are open, checked it with sudo netstat -tulpn | grep LISTEN | grep exporter
5. connecting mysql --defaults-file=/my/config/is/here.cnf works fine

Exporter is started via: /opt/mysql/product/exporters/mysqld_exporter --config.my-cnf=/my/config/is/here.cnf --web.listen-address=:9104 --log.level=debug

Any suggestiones?

[aliensmily]

We found out the problem only occurs on mysqld_exporter version 0.15.0 and 0.15.1.
We downgraded to mysqld_exporter version 0.14.0 and it worked fine.

OS: Ubuntu 24.04.1 LTS
DB: MariaDB 10.6.18