Report here the timeline of incidents and interventions on ovh3 server. Keep things short or write a report.
Images not displaying anymore on the website due to SSL problem (signaled by Edouard, with alert by blackbox exporter) In syslog:
Dec 5 13:12:58 off2 certbot[385770]: Failed to renew certificate images.openfoodfacts.org with error: The requested nginx plugin does not appear to be installed
Checked /etc/letsencrypt/renewal/images.openfoodfacts.org.conf it uses nginx authenticator.
To resolve:
apt install python3-certbot-nginx
certbot renewalso mv /etc/letsencrypt/renewal/ovh3.openfoodfacts.org.{conf,backup} as the domain is not served anymore
Nginx has a big static-access.log file (52G).
I changed /etc/logrotate.d/nginx to take into account /rpool/logs-nginx/*.log and launched /usr/sbin/logrotate /etc/logrotate.conf.
The day before, we switch back images.openfoodfacts.org to ovh3.
Alert was images.openfoodfacts.org down.
Trying to restart nginx, it failed and left behind zombie processes, that can't be killed even with kill -9 (meaning there a stuck waiting an I/O).
I could list a file (ls /rpool/off/images/products/376/002/924/8001/1.100.jpg)
but getting content wait indefinitely (cat /rpool/off/images/products/376/002/924/8001/1.json).
zpool status takes a lot of time to run and indicates 1 READ error on sdb.
We did a hard reboot.
see Slack thread.
See 2023-07-16 Disk sdc errors on OVH3
Nginx images not responding, as well as off.net.
ZFS dataset stop to work at 00:50 (UTC) in the morning. Add to do a hard reboot - no symptoms in log. slack thread
Hard reboot.
Hard reboot.