velero backup delete is not deleting kopia data from s3

[todeb]

What steps did you take and what happened:
I created empty s3 repo and did a single backup, and then removing it.
I am expecting the all data that is not referenced by any other backup is deleted.
But on s3 i see the deleted was only content from /backup/testbackup/
but content in /kopia/ is not deleted.

./velero backup create testbackup --include-namespaces backuptest --snapshot-move-data

wait until it finish, check logs and s3.

./velero backup delete testbackup

What did you expect to happen:

all backup files and data volume chunks not referenced by any other backup to be deleted

Anything else you would like to add:

Environment:

• Velero version (use velero version): 1.13.2
• Velero features (use velero client config get features): velero-plugin-for-csi:v0.7.1 ; velero-plugin-for-aws:v1.9.2
• Kubernetes version (use kubectl version): v1.25.16
• Kubernetes installer & version: -
• Cloud provider or hardware configuration: -
• OS (e.g. from /etc/os-release): -

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "I would like to see this bug fixed as soon as possible"
• 👎 for "There are more important bugs to focus on right now"

[kaovilai]

Duplicate of #8365 which is unplanned.

#8364 in the works may help speed up deletion.

It is not expected to be immediate by design and have been discussed prior as impractical/dangerous with current architecture.

[todeb]

@kaovilai I'm not expecting this to be immediate, but even after 2 days i see kopia data in my S3 repo.
What is the schedule? What logs do I need to check why data was not deleted?

[kaovilai]

Once kopia data is created, some folder will remain even if all backups are deleted. Look at the remaining size if it was reduced.

[todeb]

Only few files were deleted. I did only one backup on that repo and I deleted it. So no references to other backups.
Expecting that all unreferenced files will gone. I did a backup of volume that contains just a few bytes file. So cannot say much about size decrease.

[kaovilai]

My team has tests and validated that size do reduce. So that is sufficient IMO. Concerns are if you have 1TB volume backup and when you delete its still taking up say 200MB.

[todeb]

Thank you for clarification. I'm also seeing that data chunks are being deleted (when did a test on bigger volumes). Although still that is a bit of mess when kopia is leaving all other files which are not referenced by anything in S3 repo. Expected would be that kopia cleaned this up.

[kaovilai]

velero do not delete kopia repos.. it init repos, add snapshots to it, delete snapshots as backups are deleted, but repos stay.