Storage Multipath

What is it?

"Multipath" is the situation where a storage device is reachable via more than one kernel block device. Writing to a block number using one of the block devices will be visible when reading the same block number from any of the other block devices.

This happens when a storage device is connected via multiple cables to a machine, to increase reliability and performance, or by accident.

This reliability and performance improvement is achieved by combining the multiple block devices via the device mapper into a new, single block device. However, the other block devices are still visible and accessible.

Why should Cockpit care?

Multipath devices must be used carefully since the normal exclusion mechanisms only work for single block devices.

For example, if one of the block devices of a multipath device is mounted as a filesystem somewhere, there is nothing in bare system that will stop (or even warn) someone from overwriting the storage device via another block device of the same multipath device.

As another example, if you create a filesystem via one block device, the udev properties of the other block devices are not updated.

Thus, Cockpit needs to be careful to always use the primary device mapper block device for a multipath device. If there is no such primary device, Cockpit needs to help setting it up.

Cockpit should also help with tweaking the parameters of existing multipath setups.

There might be more complicated scenarios, such as when a singlepath device turns into a multipath device when the second cable has finally been found at the bottom of that drawer that you really need to clean up one of these days.

What will be done

As a first step, Cockpit is made "multipath safe". It will show the multipath topology correctly, and will prevent people from using anything but the 'master' block device. This will work both when multipathd is running and correctly configured and also when it is not running or not properly configured.

Storaged already correctly identifies multiple paths to the same storage device by setting the Block.Drive property of all of them to point to the same Drive object. Previously, Cockpit has ignored the possibility that a Drive might have multiple Block objects to it, and has simply picked one of them at random.

Now Cockpit collects all Block objects that point to a single Drive object and displays them.

Cockpit identifies 'slaves' by looking for IdType=mpath_member. If there is exactly one non-slave, this is assumed to be the 'master' device mapper device. If there is more than one (or no) 'master', then things haven't been properly set up for this multipath device and Cockpit doesn't let people make filesystems on it, etc.

Cockpit should also show the state of multipathd, such as failed paths. It could also show performance metrics for the individual paths.

Cockpit should also help to properly setup multipath support. This needs input from the multipath guys about the things that should be offered for customization.

One thing is know already: multipathd needs to be setup and started. The easiest thing would be to have multipathd running by default in Fedora Server. The second best option would be to automatically and invisibly start it as needed.

Random notes

Virtio devices are silently blacklisted by default. Use SCSI.

Multipath support needs to be explicitly enabled:

# dnf install device-mapper-multipath
# mpathconf --enable
# systemctl restart multipathd
# udevadm trigger

Cockpit/storaged could depend on device-mapper-multipath. The final "udevadm trigger" should not be necessary and multipathd should trigger enough uevents. https://bugzilla.redhat.com/show_bug.cgi?id=1254583

Multipathd can be poked like so:

multipathd reconfigure
udevadm trigger

Again, the udevadm trigger shouldn't be necessary.

Multipathd ignores multipath devices that are already mounted. Thus, when Cockpit finds a improperly configured multipath device, it needs to unmount all of its partitions, and then poke multipathd to reconfigure (or enable/start it).

The multipath topology/state can be retrieved like so:

# multipathd list paths format '%d %m %o'
dev multipath dev_st 
sda [orphan]  running
sdb mpatha    running
sdc mpatha    running

It might be worthwhile to teach multipathd machine-readable output, maybe JSON.

Multipath members have this udev property:

ID_FS_TYPE=mpath_member

but there aren't enough uevents to ensure that it is always correctly set, see above.

sysfs also has topology in slaves/ and holders/. Storaged could export that generically.