allow to exclude previews from backup upon instance backup

[devnoname120]

Follow up of: #5550

@szaimen I haven't tested the changes yet as I don't have the right setup… Is there any chance you could test them? Thanks!

[devnoname120]

Edit: I added Signed-off-by: Paul <[email protected]> to the end of the commit message and then force-pushed to make DCO pass.

[devnoname120]

@szaimen Is there a way to test the changes on my side on my production instance? I tried looking into building the whole AIO thing but there doesn't seem to be an easy way to do that. The actual docker build pipeline seem to be defined in the nextcloud-releases repo: https://github.com/nextcloud-releases/all-in-one/blob/main/.github/workflows/build_images.yml

I vaguely tried to pull the CI workflow change it to push to ghcr.io (to my own fork) instead of docker.io but the references to docker.io look farther reaching than just in the CI pipelines so I quickly gave up…

[szaimen]

Hi @devnoname120 thanks a lot for the contribution and sorry for my late reply!

I unfortunately have some problems with this as in my opinion, the backup should always represent a full backup with intact database and file system and basically including all data needed to restore the same state without exception. This also includes the previews as their path is also written into the database. Allowing to exclude the previews means that we always need to scan the database upon restore which is hacky imho.

Also, maybe we could find a better way that allows to exclude previews and more without having to add an option to the interface for each thing? This could probably be implemented like this: #4751. Would you be able to work on this?

Please let me know what you think about my thoughts!

[devnoname120]

@szaimen I understand your point of view and I think that it's very sensible. From my side the motivation underlying this PR is that every day my server is down for 20 minutes due to the daily backup, and the absolutely huge number of thumbnail files is responsible for a good part of it.

Furthermore, navigating my backups is extremely slow (almost a minute for the initial loading) — again due to these thumbnail files. Note that my Nextcloud instance and its data are fully stored on an SSD so there is not much I can do on that front.

I'm not entirely sure about the point of backing up and restoring thumbnails as they are generated on the fly anyway so I think most users would want to exclude them (note that I didn't make it the default).

While a more general approach for excluding things from backups would definitely be useful, I think that thumbnails are the killer use-case because of their massive footprint and unclear utility. I don't think of another obvious target that would have such a considerable impact.

Finally, given that the thumbnail exclusion on restore is already implemented I think that mirroring the implementation for the backup side is reasonable as it makes the experience consistent.

It does not mean that a more general exclusion feature won't be implemented at a later point, but this is additional work that I cannot take on my plate right now unfortunately.

What do you think?

Edit: I just ran the command find /data/appdata_oc95f75zaox2/preview | wc -l and it took 10 minutes and 20 seconds to complete! It returned 1081981 so 1M+ thumbnail files… You can see here how it's a huge problem on my side :-/

[szaimen]

I see. However this PR is very far from being finished as it still misses quite a few places in order to work correctly.

The much easier and more forward-thinking solution would be to allow .noaiobackup files to ignore folders. This then only needs to be adjusted inside the borgbackup container: https://github.com/nextcloud/all-in-one/tree/main/Containers/borgbackup and is much more reliable. See #4751 for some clues on the implementation.

[devnoname120]

However this PR is very far from being finished as it still misses quite a few places in order to work correctly.

Could you expand on what is missing? I was under the impression that I got everything right

[szaimen]

I will not give further hints on the implementation, as this feature is not going to be merged, sorry.

Please have a look at #4751 if you want to make it possible to ignore the folder during backup.

[devnoname120]

I feel very frustrated because of the considerable waste of time and I'm not sure anyone will implement what you're asking for anytime soon, so big opportunity cost here. I will close this PR but I think it's sad.