backup fails with cipher error, why?

[ccna2ccda]

Hi everyone,

The scheduled backup throws errors in the log files. Many are similar to the one below. What does the error mean? The velero was upgraded from 1.3 to 1.4 recently. The indexes mentioned below are the data managed by velero, is it correct?

A few more questions:

1. Does the velero have initial full backup and then subsequent incremental backup concept?
2. Does the first run of backup take more time than any subsequent after the schedules are created.

time="2024-10-29T15:57:44Z" level=error msg="Error backing up item" backup=velero/sch-backup-sg39-prod-daily-20241029155734 error="failed to wait BackupRepository, errored early: backup repository is not ready: error to connect to backup repo: error to connect repo with storage: error to connect to repository: unable to create shared content manager: error loading indexes: error downloading indexes: error loading index blob xn27_eff9305fc39188c550929763f5be4be0-sd9daa255909ca8fb12d-c1: decrypt blob: error decrypting BLOB xn27_eff9305fc39188c550929763f5be4be0-sd9daa255909ca8fb12d-c1: unable to decrypt content: cipher: message authentication failed" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/repository/backup_repo_op.go:83" error.function=github.com/vmware-tanzu/velero/pkg/repository.GetBackupRepository logSource="pkg/backup/backup.go:510" name=pod0

Thanks.

[Lyndon-Li]

The repo data is encrypted automatically, this error means the decryption failed.
One question --- If you rollback to 1.13 (by 1.3 and 1.4, I think you are saying 1.13 and 1.14, right?), would the backup work?

[ccna2ccda]

Hi Lyndon,
Thanks for the quick reply. Yes, it is 1.14 upgraded from 1.13 on the velero server. I can't rollback to 1.13. Is there anyway to do a fresh backup using 1.14? The old backup's TTL is 30 days and most will be deleted soon.

[Lyndon-Li]

Does the velero have initial full backup and then subsequent incremental backup concept?

Yes

Does the first run of backup take more time than any subsequent after the schedules are created.

Most probably yes, but that depends on how many data have been changed since the last backup.

[ccna2ccda]

How do I know whether one backup is an initial full backup or incremental?

The scheduled backup had several BIG pod volumes in the backup; then I excluded them using the annotations. But surprisingly, the backup time does not change much. Then I remove the annotation, which means the big pod volumes will have to be included again. But this time, I still do not see the backup time increase much.

[sseago]

@ccna2ccda If you're using fs-backup or datamover then all volume backups will be incremental if there's still a prior non-expired backup which includes the volume. The backup itself (for the kubernetes metadata) is always full.

[ccna2ccda]

Thank you, Scott @sseago