Cross-volume cloning of storage vfolders

[lizable]

Motivation  

• Since we provide “model-store“ feature, which brings a frequent cloning operation from origin to destination that may have different storage host.
  Please refer to the context below:
  Jihyun Kang: 뭔가 유추하자면 서로 다른 storage 에 대해서는 지금 clone 대응이 안되어 있어서 그런것 같네요. posted in devops / http://Backend.AI Cloud on January 6, 2025 5:06 PM 
• Despite of “model-store“ feature, It’s obvious to support from storage-to-storage cloning feature.

Current Limitation

Cloning large vfolders (~hundreds GiB, TiB) across storage hosts (volumes) requires a special attention to resumability upon interrupts, as we frequently observe storage failures like inadvertent unmounts of storage volumes, etc. in high-load systems.

Also, it is impossible to utilize per-backend acceleration features like tree copies within a same filesystem, as the data must be transferred via networks.

Therefore, we currently do not support cross-volume vfolder cloning. This should be actively disabled/discouraged in the WebUI and Control Panel until we properly implement the feature.

Goals

• Define an interface to prepare, perform, and inspect the cloning procedure & progress.
  • We should also add the UI design to display such long-running progress-reporting resumable background jobs.
• Make it resumable.
• Enable progress tracking (though it may be difficult to implement “perfect” progress).

Considerations / Approaches

The simplest implementation would be shutil.copy_tree() in the storage proxy, where the host has the local mounts of both storage volumes. However, this has severe limitations on resumability and progress tracking.

• Idea: Incorporate rysnc to synchronize two local paths in the storage-proxy host, offloading our implementation effort for resuming and progress tracking. (ref: https://chatgpt.com/share/6784ad0b-3774-8000-8ace-50af8e0fb2a4)
  • Preferably, we could ship a statically built rsync binary together with the storage proxy.