Week 34 2024 routine

[kiwixbot]

Check nodes free space

df -h / && df -h /data

• create a report in issue comment

Nodes system upgrades

apt update && apt upgrade

• run systematically the upgrade on bastion, stats, services, storage, demo nodes
• check for and apply important security upgrade on worker nodes asap (imager-worker, ondemand, sisyphus)

(regular workers updates are done separately on a monthly basis for worker nodes to not impact production)

Backups

• Ensure all borg repositories are being updated

k8s cluster

• Check Pod errors

k get pods -A -o wide|grep Error

• Check Pod restarts

k get pods -A -o wide | pyp -i 'print("\n".join([line for line in l if re.split(r"\s+", line)[4] != "0"]))'

• Check if k8s should/could be upgraded

curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER | jq ".version,.upgrade_available"
curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/versions | jq ".versions[].name"

• Upgrade k8s if applicable/possible

Stats

matomo - stats.kiwix.org

• Ensure download.kiwix.org stats are being recorded
• Check whether matomo should be upgraded

Grafana

• Alert list is normal
• Zimfarm dashboard is normal
• Jobs durations dashboard is normal
• There is no abnormal behaviors on cluster resources consumption
• All mdadm RAID arrays are OK (check all instances in the top left combobox)

Projects

• UptimeRobot has no alert
• zimit backlog is reasonable
• Cloud Code Signing Certificate usage is OK (look for Unused Signings under END ENTITY CERTIFICATES): we have 1,200 per year (August to August)
• Analyze zimit failed tasks
• PRs awaiting your review

Security

• Analyze/merge dependabot PRs

Note: this is an automatic reminder intended for the assignee(s).

[rgaudin]

Storage

Machine	Filesystem	Size	Used	Avail	Use%	Use change
bastion	/	37G	14G	22G	40%	-
stats	/	233G	108G	114G	49%	-3G
services	/	456G	304G	129G	71%	+5G
storage	/	33T	18T	14T	58%	-4T
imager-worker	/	1.9T	452G	1.4T	26%	don't care
sisyphus	/	233G	25G	196G	12%	don't care
ondemand	/	25G	9.7G	14G	42%	don't care
ondemand	/data	216G	205M	205G	1%	don't care
mirrors-qa	/	38G	3.2G	33G	9%	-
demo	/data	1.8T	920G	744G	56%	don't care

• 4T delta on storage due (I believe) to @benoit74's cleanup in prep of new storage server sync
• demo worker moved to a new 2TB server

misc

• Updated matomo to 5.1.1
• 152 signings/1,200 (kiwix-js trials and setup)
• updated reverse of zimit worker IP to worker.zimit.kiwix.org.

zimit

• 1 x net::ERR_INVALID_RESPONSE
• 2 x net::ERR_NAME_NOT_RESOLVED
• 2 x net::ERR_CONNECTION_REFUSED
• 1 x net::ERR_TIMED_OUT
• 1 x Seed Page Load Error (403)
• 4 x ValueError: Incorrect HttpUrl scheme in value openzim/warc2zim#332
• 7 x LookupError: unknown encoding: unicode openzim/warc2zim#331
• 1 x Relaxed filename is too relaxed openzim/zimfarm#1010
• 1 x Rewriter choke on a relative URL inside a querystring openzim/warc2zim#380
• 1 x Execution context was destroyed webrecorder/browsertrix-crawler#655
• 7 x Navigation timeout of 90000 ms exceeded
• 4 x ValueError: Title is too long
• 1 x ValueError: '..' segment in 'portalvwco.catalog.srv.br/common/nessComponents/xtree.html?css=../../prg' cannot be walked
• 5 x OOM ⚠️ OOM happened in warc2zim, not in the crawler. Only happened to the oldest tasks so I'm not opening a ticket this week
• Lots of requests for library.kiwix.org 🤨