GKE to Azure Migration using Velero

[abrahamarumbaka]

Hello All,

I m migrating from GKE to AKS using Velero. Velero version V1.12.0

I have GKE Cluster with a pod and a PVC. I configured velero using

velero install --provider gcp --plugins velero/velero-plugin-for-gcp:v1.5.2 --bucket testbucket --secret-file ~/velero-gcp-creds.json --backup-location-config serviceAccount=user-579@test-test-431805-c9.iam.gserviceaccount.com --snapshot-location-config region=us-central1-c

Velero backup command.

velero backup create my-backup --include-namespaces your-namespace

velero backup get - Gives me the successful backup and the storage bucket has the backup folder with files.

Now, I m trying to restore the backups to AKS. I have AKS env with a node. Install and configured Verelo on the target .

velero install \
    --provider azure \
    --plugins velero/velero-plugin-for-microsoft-azure:v1.5.0.

I tried to restore the back up using few approaches and nothing seems to be working.

Approach 1 -

Found a command to add gcp plugin in AKS. velero add --plugins velero/velero-plugin-for-gcp:v1.5.2 - and tried to reconfigure velero install on AKS by passing GCP bucket flag and GCP credentials file.
Unforunately did not help.

Approach 2 -

Tried to get the velero backups in the same PowerShell cmd line (switching between GKE and AKS config) and restored on the target. did not help.

Tried few other approaches, unfortunately it did not helped me. Not sure but in one scenario it did worked for, just the successful migration of application pod but not the PVC POD due to storage class mismatch between GKE and AKS.(I couldn't reproduce it later to debug)

Further research, I found that restic flag in velero will solve the problem. but the latest velero has --use-node-agent arg that will take snapshot based on the restic.

I have ton of other approaches unfortunately, couldn't proceed further.

Is there anyone who tried this solution? Or thoughts to guide me on next successful lift and shift?

(I was able to achieve it using cloudcasa, but since the app is very small, unfortunately I couldn't proceed with its licensing terms)

Thanks in advance,

[blackpiglet]

IMO, the first approach should work.
After adding the GCP plugin into the AKS environment, you should add a new BSL representing the GKE's backup repository.
After that BSL is available, Velero will sync the backups into the AKS cluster, then you can try to restore from them.
Just so you know, the cross-provider B/R can only support file system backup of the Volume Data. The snapshot-based volume backup cannot work in this scenario.

[abrahamarumbaka]

@blackpiglet thank you so much for your suggestions.
I have tried to backup in GKE - velero backup create testbackup --snapshot-volumes=false --include-namespaces demo --wait it was successful.
Configured in AKS - velero BSL and able to read the velero backups and restored using - velero restore create --from-backup testbackup --include-cluster-resources=false --wait , although it was able to restore the pods, pvc's etc.

The problem started with pvc being pending status. Further debugging found that storage class and provisioner do not match - >

GCP ->

standard kubernetes.io/gce-pd Delete Immediate true 9h
standard-rwo (default) pd.csi.storage.gke.io Delete

Azure ->
managed disk.csi.azure.com

Workaround 1 - created a configmap in AKS - ns velero - it maps the standard: managed, but not the provisioner .

apiVersion: v1
kind: ConfigMap
metadata:
  name: change-storage-class-config
  namespace: velero 
  labels:
    velero.io/plugin-config: ""
    velero.io/change-storage-class: RestoreItemAction
data:
  standard: managed

Workaround 2 - kubectl edit the pvc on target and removed - below annotations.
volume.beta.kubernetes.io/storage-provisioner: pd.csi.storage.gke.io
volume.kubernetes.io/storage-provisioner: pd.csi.storage.gke.io

unfortunately it did not helped. pvc still remains in pending state.

"storageClassName":"standard"}}
    volume.beta.kubernetes.io/storage-provisioner: pd.csi.storage.gke.io
    volume.kubernetes.io/storage-provisioner: pd.csi.storage.gke.io
  creationTimestamp: "2024-08-23T04:10:59Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    velero.io/backup-name: testbackup
    velero.io/restore-name: testbackup-20240822211101
  name: sample-pvc
  namespace: demo
  resourceVersion: "127012"
  uid: 70c258bd-d03a-4998-9244-fbf7f3198d0e
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: managed
  volumeMode: Filesystem
  volumeName: pvc-a5782015-1ee3-4b73-8239-92e6bf64ddac
status:
  phase: Pending

Could you please suggest any other workarounds?

[blackpiglet]

I suggest using the restore modifier feature to make it straight.
https://velero.io/docs/v1.14/restore-resource-modifiers/

[abrahamarumbaka]

Thank you @blackpiglet !