Storage Multipath

What is it?

"Multipath" is the situation where a storage device is reachable via more than one kernel block device. Writing to a block number using one of the block devices will be visible when reading the same block number from any of the other block devices.

This happens when a storage device is connected via multiple cables to a machine, to increase reliability and performance, or by accident.

This reliability and performance improvement is achieved by combining the multiple block devices via the device mapper into a new, single block device. However, the other block devices are still visible and accessible.

Why should Cockpit care?

Multipath devices must be used carefully since the normal exclusion mechanisms only work for single block devices.

For example, if one of the block devices of a multipath device is mounted as a filesystem somewhere, there is nothing in bare system that will stop (or even warn) someone from overwriting the storage device via another block device of the same multipath device.

As another example, if you create a filesystem via one block device, the udev properties of the other block devices are not updated.

Question: If you put a UUID=xxx line into fstab, which block device does mount use?

Thus, Cockpit needs to be careful to always use the primary device mapper block device for a multipath device. If there is no such primary device, Cockpit needs to help setting it up.

Cockpit should also help with tweaking the parameters of existing multipath setups.

There might be more complicated scenarios, such as when a singlepath device turns into a multipath device when the second cable has finally been found at the bottom of that drawer that you really need to clean up one of these days.

What might be done

Storaged exports a o.s.Storaged.MultipathMember interface on block device objects with these properties:

Master (o)
Active (b)
Error (s)
...

Random notes

Virtio devices are silently blacklisted by default. Use SCSI.

Multipath support needs to be explicitly enabled:

# dnf install device-mapper-multipath
# mpathconf --enable
# systemctl restart multipathd
# udevadm trigger

Cockpit/storaged could depend on device-mapper-multipath. The final "udevadm trigger" should not be necessary and multipathd should trigger enough uevents. I'll file a bug.

The multipath topology/state can be retrieved like so:

# multipathd list paths format '%d %m %o'
dev multipath dev_st 
sda [orphan]  running
sdb mpatha    running
sdc mpatha    running

All changes to the topology/status seem to be done by multipathd, so it should be able to pass out notifications. Uevents on the block devices should be enough.

It might be worthwhile to teach multipathd machine-readable output, maybe JSON.

Multipath members have this udev property: