Still consuming space in default data directory after configured new --data-dir

[James4Ever0]

Environmental Info:
K3s version:
k3s version v1.29.6+k3s2 (b4b156d)
go version go1.21.11

Node(s) CPU architecture, OS, and Version:
Linux redacted 5.16.0-kali7-amd64 #1 SMP PREEMPT Debian 5.16.18-1kali1 (2022-04-01) x86_64 GNU/Linux

Cluster Configuration:
1 server

Describe the bug:
After I configured new custom --data-dir in /etc/systemd/system/k3s.service at line ExecStart, it still saves some data under path /var/lib/rancher/k3s/data/current

4.0K	./etc
1.5M	./bin/aux
195M	./bin
195M	.

Steps To Reproduce:

• Normally use official command line to install K3s using default data directory
• Configure to the new data directory
• Rerun service
• Data mostly in new directory but left some under default directory

Expected behavior:

All data shall be stored in new directory.

Actual behavior:
Data being partly stored in old directory and mainly in new directory.

Additional context / logs:

After I used rsync -avh --progress /var/lib/rancher/k3s <new_data_directory> to migrate data, K3s still treats the old data like if it does not exist. K3s will download init data, pull kube-system images and create init containers.

This behavior is expected because Docker has similar bevariors too. I just want some explaination over this and how to reuse old data if possible.

[brandond]

I'm confused, did you expect k3s to move things for you when you changed the data-dir? It is not generally recommended to change the data-dir after the node has been started once, but it should work OK if you move things yourself. I'm not sure why you'd use rsync to copy things over instead of moving?

[James4Ever0]

The primary focus is that it still comsumes space at the default data storage location no matter I place rsynced data to the new location or not. I did purge the new location, rerun the K3s service and it still happened.

Rsync preserves symlinks. I use it to migrate conda installation folders with success.

I wonder if creating a symlink from the new folder to the default data folder will mitigate this issue.