Use source pool names as target pool name

[ryszard-suchocki]

As a user, I want to backup multiple pools to Backup Server that has the same pool schema, so I'm able to create a full replica ready to go in a disaster scenario.

Let's imagine the situation where you have 2 servers in equal hardware configuration, ZFS pools on those servers are divided on fast and slow storage (SSD/fs and HDD/fs) and you want to perform a backup.

Best regards!

[psy0rz]

Can you explain what you mean exactly?

We already support multiple pools automaticly, just set the autobackup: property like usual, but on multiple pools.

Edwin

[ryszard-suchocki]

1:1 replication, like using standard PVE built-in replication mechanism.
[host A]------------------[host B]
ssd1/vm-100-disk-0 => ssd1/vm-100-disk-0
hdd/vm-100-disk-0 => hdd/vm-100-disk-0

Currently, all FS with autobackup:property=true gives such a result:
[host A]------------------[host B]
ssd1/vm-100-disk-0 => ssd1/ssd1/vm-100-disk-0
hdd/vm-100-disk-0 => ssd1/hdd/vm-100-disk-0

Best regards

[psy0rz]

Ah i understand.

Can you try using --strip-path as workaround?

In your example a --strip-path 1 should result in:

[host A]------------------[host B]
ssd1/vm-100-disk-0 => ssd1/vm-100-disk-0
hdd/vm-100-disk-0 => ssd1/vm-100-disk-0

It strips the first "part" of the source name. In this case it strips ssd1 and hdd from it.

I assume the other vm has a different number than 100, so there should be no naming conflict.

Edwin

[ryszard-suchocki]

Well, you discovered the problem with PVE and VM disk naming when using ZFS storage (disk name is unique in storage scope, but not in whole VM). So stripping the first part of the source name is not an option because you will achieve a dangerous situation like below (this is the schema you created) :

[host A]------------------[host B]
ssd1/vm-100-disk-0 => ssd1/vm-100-disk-0
hdd/vm-100-disk-0 => ssd1/vm-100-disk-0

Additionally, I don't want to store virtual-disk which is assumed to contain lots of rarely used data (slow access is acceptable) on SSD storage.

The Strip-path parameter gives some possibilities, but it still not what I'm trying to achieve. I tried it earlier but to make full replication (1:1 between identical servers) I need 3 different commands and different properties for each of storage's type:

1. Set the common property to all storages used by VM to distinguish virtual machine drives among other i.e "mybackup-100"

2. Set property for SSD storage -> "mybackup-100-ssd"

3. Set property for HDD storage -> "mybackup-100-hdd"

4. Create snapshot (only snapshot, do not send) for all filesystems marked "mybackup-100" (--no-send and --allow-empty parameter ) [ here we have time consistent snapshot among all storages ]
   zfs-autobackup --ssh-source X mybackup-100 ThisIsNotImportant --verbose --keep-source X --keep-target X --no-hold --no-send --allow-empty

5. Pull snapshot of filesystem marked "mybackup-100-ssd" to SSD1
   zfs-autobackup --ssh-source X mybackup-100-ssd ssd1 --progress --verbose --strip-path 1 --keep-source X --keep-target X --no-hold --rollback --other-snapshots --destroy-incompatible --no-snapshot

6. Pull snapshot of filesystem marked "mybackup-100-hdd" to HDD
   zfs-autobackup --ssh-source X mybackup-100-hdd hdd --progress --verbose --strip-path 1 --keep-source X --keep-target X --no-hold --rollback --other-snapshots --destroy-incompatible --no-snapshot

What I want to achieve - preserve source pool name and replicate data to pool with the same name on another machine using the minimal count of commands

--preserve-pool-name | ignore destination pool name and use source pool name

[psy0rz]

You basically want something like rsync for zfs pools? :)

[ryszard-suchocki]

You basically want something like rsync for zfs pools? :)

yes, 1:1 replication with all benefits from zfs-autobackup (custom properties, autoprune).

[psy0rz]

Ok, first i have to fix some more stuff and then i'll figure out how to do this in a nice way.

[psy0rz]

update: the fixing some stuff turned into adding full regression testing and a lot of new features for 3.1. this one is still on the table however. :)

[psy0rz]

please have a look at #114 and let me know overthere what you think of it

[psy0rz]

In the mean time, if you want 1:1 replication on a per-pool basis, look here:

https://github.com/psy0rz/zfs_autobackup/wiki#making-source-and-target-paths-look-the-same

[psy0rz]

closing this, since its essentially fixed via that documentation. zfs-rsync is a different thing.