Can incremental snapshots be restored at all?

[2fst4u]

I'm a bit confused by the restore process. I was hoping to be able to schedule backups and have snapshots be taken, allowing the CSI snapshots to be restored in the regular way of creating a PVC by referencing the snapshot.

I understand velero now removes snapshots once a backup is done so that it's not accidentally removed if the namespace is deleted, which I understand.

But is there now no way for a snapshot to be restored? Is the only way to restore data to do a full PVC restore?

My concern there is I really have no way of knowing my backups are working because the only way to restore them to test them is to destroy the PVC, to restore over it - unless you have another cluster to restore into, which I don't have.

Am I missing something?

[kaovilai]

is there now no way for a snapshot to be restored?

No. VolumeSnapshotContent remains in the cluster. It is intentionally disconnected from VolumeSnapshot so it lives past namespace deletion.

VSC will only get deleted when backup gets deleted

On restore velero create a volumesnapshot object referencing the original volumesnapshotcontent and will be able to create a PVC using VolumeSnapshot as source.

I really have no way of knowing my backups are working

Check there are volumesnapshotcontents with your backup name in its labels/annotations.

[2fst4u]

But the volumesnapshots get deleted right?

[kaovilai]

VolumeSnapshots can be deleted without VSC getting deleted.
VS can be recreated from VSC and used later to restore.

[kaovilai]

velero/pkg/exposer/csi_snapshot.go

Lines 143 to 153 in 9dcfe16

	retained, err := csi.RetainVSC(ctx, e.csiSnapshotClient, vsc)
	if err != nil {
	return errors.Wrap(err, "error to retain volume snapshot content")
	}
	curLog.WithField("vsc name", vsc.Name).WithField("retained", (retained != nil)).Info("Finished to retain VSC")
	err = csi.EnsureDeleteVS(ctx, e.csiSnapshotClient, volumeSnapshot.Name, volumeSnapshot.Namespace, csiExposeParam.OperationTimeout)
	if err != nil {
	return errors.Wrap(err, "error to delete volume snapshot")
	}

Here we first make sure VSC is retained, then VS is deleted.
Thus you can simulate disaster by deleting NS of workload without VSC deletion and restore the NS later.

[kaovilai]

If we had NOT retained VSC manually in this case, then yes, a VS deletion = VSC deletion = poof no restore

[2fst4u]

I get that the VSC is retained which is good, but then what I'm confused about is don't you need the VS to restore a PVC from? Which has now since been deleted?

[kaovilai]

the only way to restore them to test them is to destroy the PVC, to restore over it

Not true.

You can create VolumeSnapshot pointing to existing VolumeSnapshotContent.
You can then create PVC with volumeSource set to the VolumeSnapshot.