Moving Cloud Storage Provider / gocryptfs vs. rclone crypt

[loungebob]

since google wants to be massive ç"%R I'm moving my very large amount of data to another provider.

I figure server-side transfer is easier and faster and was wondering if I could just copy the encrypted data to the new provider if I set up gocryptfs there with the same password or what would be the way to getting around downloading/decrypting/encrypting/uploading?

also, can I create the new environment with the deterministic names setting enabled and still just server-side copy the files while dropping the dirivs and it'll work? probably not, already sound stupid. if not, is the only way to get rid of the diriv files the downloading/decrypting/encrypting/uploading way to a encryption created with that setting on?

I've been using gocryptfs for 6 years now and it's always performed and worked fine. and easily enough, for someone like me.

but, still, here's another question I couldnt really find a proper answer to or better yet, a comparison. I was looking into using rclone crypt. almost only for the simple reason it will let me mount the cloud folder on e.g. my shield and other linuxy devices. I cant do this with gocryptfs, right? assuming no, the only difference between those two methods would be, that with rclone (crypt) the shield would do the mounting and reading and decrypting and such and in the gocryptfs case, my computer's OS would do all that and then hand it over to the shield or plex or whatever?

one more: cryfs' incremental synchronization, isnt that a good thing and would speed up uploads/syncs? wouldnt everyone want to have that or aint it a big deal in reality? trying to find out if I understood this correctly....

[loungebob]

sorry, apparently I’ve already asked the first part 3 years ago and it was answered in issue #511 so please ignore the moving encrypted files between stores (so I use the same conf file or change the masterkey in the new store to match the one in the old one).

the question re: the deterministic filenames remains however.

and the other questions of course. unless I find out that those two were already asked (by me) and answered 🙈

[loungebob]

so #511 works just fine, tested it with a lot of files. so the moving "as is" question is done.

I tried to create a conf file with the deterministic names on and changed the masterkey to the old working (diriv) conf one (which worked just fine) but the resulting conf file would not work on my old data at all. not displaying anything when mounted. you're all probably laughing now because it's obvious to everyone that this could never work but I tried anyway.

does this mean, the only way to go from diriv to determ names is to actually decrypt and re-encrypt all data while moving it to a new store using a conf file with that flag on?

it's about 2PB of data and I really, really want to get rid of dirivs.

[kaoneko]

Hi, October 2024 here. As far as I understand gocryptfs, you are correct.

The data in gocryptfs.diriv files is used in encrypting/decrypting the file names. Using a config file that tells gocryptfs "we're using deterministic names now" doesn't change the fact that your file names are actually still encrypted using the data in gocryptfs.diriv. Without using it, the file names can't be decrypted, and the files don't show up on the mounted file system.

What you need(ed) is a way to re-encrypt all file names without re-encrypting all the data, which would be totally unnecessary and indeed very impractical with 2000 terabytes of data. It shouldn't be too hard to do, but I'm not sure if gocryptfs implements this functionality.

After having a quick look, I think you could achieve this with gocryptfs-xray. You would create an empty gocryptfs folder that uses the same master key but deterministic file names (i.e. copy over gocryptfs.conf and remove the DirIV flag). You would then mount both the folder holding your 2 PB of encrypted data and the empty folder using the -ctlsock option, e.g.

gocryptfs -ctlsock diriv.sock 2pbdir mountpoint1
gocryptfs -ctlsock deterministic.sock emptydir mountpoint2

Then you can use

printf "<diriv-encrypted-file-name>" | gocryptfs-xray -decrypt-paths diriv.sock | gocryptfs-xray -encrypt-paths deterministic.sock

to get the deterministic file/folder name.

Make a script that goes through your encrypted data folder, and for each file and folder obtains the deterministic name and renames it to that. Each renamed file or folder will disappear from the mounted file system. Once done, unmount everything, remove the DirIV flag from your gocryptfs.conf and mount the encrypted data folder again. All files and folders should show up. You can then remove all gocryptfs.diriv files.

Of course you want to be ridiculously sure your script works correctly before doing this with your real data. I'd probably also do it bit by bit, instead of running the script on the full 2 PB of data at once. You can use a conf file with the DirIV flag removed to mount a view of the files/folders that use deterministically encrypted names, and the regular conf file to mount a view of the files/folders that still use the diriv file.

In short, it's possible but not easy; you have to get your hands dirty and write and test some code yourself. Unless I'm missing something gocryptfs doesn't automate this for you.

What did you do in the end? Did you switch to deterministic names? If so, how?