Storage Multipath

What is it?

"Multipath" is the situation where a storage device is reachable via more than one kernel block device. Writing to a block number using one of the block devices will be visible when reading the same block number from any of the other block devices.

This happens when a storage device is connected via multiple cables to a machine, to increase reliability and performance, or by accident.

This reliability and performance improvement is achieved by combining the multiple block devices via the device mapper into a new, single block device. However, the other block devices are still visible and accessible.

Why should Cockpit care?

Multipath devices must be used carefully since the normal exclusion mechanisms only work for single block devices.

For example, if one of the block devices of a multipath device is mounted as a filesystem somewhere, there is nothing in bare system that will stop (or even warn) someone from overwriting the storage device via another block device of the same multipath device.

As another example, if you create a filesystem via one block device, the udev properties of the other block devices are not updated.

Thus, Cockpit needs to be careful to always use the primary device mapper block device for a multipath device. If there is no such primary device, Cockpit needs to help setting it up.

Cockpit should also help with tweaking the parameters of existing multipath setups.

There might be more complicated scenarios, such as when a singlepath device turns into a multipath device when the second cable has finally been found at the bottom of that drawer that you really need to clean up one of these days.

What has been done

As a first step, Cockpit is made "multipath safe". It will show the multipath topology correctly, and will prevent people from using anything but the 'master' block device. This will work both when multipathd is running and correctly configured and also when it is not running or not properly configured.

Storaged already correctly identifies multiple paths to the same storage device by setting the Block.Drive property of all of them to point to the same Drive object. Previously, Cockpit has ignored the possibility that a Drive might have multiple Block objects to it, and has simply picked one of them at random.

Now Cockpit collects all Block objects that point to a single Drive object and displays them.

Cockpit identifies 'slaves' by looking for IdType=mpath_member. If there is exactly one non-slave, this is assumed to be the 'master' device mapper device. If there is more than one (or no) 'master', then things haven't been properly set up for this multipath device and Cockpit doesn't let people make filesystems on it, etc.

https://github.com/cockpit-project/cockpit/pull/2596

What still needs to be done

Cockpit should show the state of multipathd, such as failed paths. It could also show performance metrics for the individual paths.

Cockpit should help to properly setup multipath support. This needs input from the multipath guys about the things that should be offered for customization.

At the minimum: multipathd needs to be enabled and started. The easiest thing would be to have multipathd running by default in Fedora Server.

Enabling multipathd at runtime without a reboot might be tricky, for the same reasons that turning a singlepath device into a multipath device at runtime is tricky. Cockpit might thus play it safe and advice to reboot the system to enable multipathd.

And when I say Cockpit, I mean Storaged. The next step could be to add API for multipath to Storaged.

• org.storaged.Storaged.MultipathManager on /org/storaged/Storaged/Manager

  For global state/actions.

  Reconfigure() ForceQueueing() RestoreQueueing() ...

• org.storaged.Storaged.Multipath on /org/storaged/Storaged/blocks/*

  For state/actions of the device mapper device for a multipath device.

• org.storaged.Storaged.MultipathMember on /org/storaged/Storaged/blocks/*

  For state/actions for the individual paths of a multipath device.

Random notes

Virtio devices are silently blacklisted by default. Use SCSI.

Multipath support needs to be explicitly enabled:

# dnf install device-mapper-multipath
# mpathconf --enable
# multipathd reconfigure || systemctl start multipathd
# udevadm trigger

Cockpit/storaged could depend on device-mapper-multipath. The final "udevadm trigger" should not be necessary and multipathd should trigger enough uevents. https://bugzilla.redhat.com/show_bug.cgi?id=1254583

Multipathd ignores multipath devices that are already mounted, are a physical volume in a volume group with at least one active logical volume on it, or are otherwise in use. Thus, when Cockpit finds a improperly configured multipath device, it needs to unmount all of its partitions, deactivate all logical volumes, etc, and only then enable/start/reconfigure multipathd as above.

The multipath topology/state can be retrieved like so:

# multipathd list paths format '%d %m %o'
dev multipath dev_st 
sda [orphan]  running
sdb mpatha    running
sdc mpatha    running

It might be worthwhile to teach multipathd machine-readable output, maybe JSON.

Multipath members have this udev property:

ID_FS_TYPE=mpath_member

but there aren't enough uevents to ensure that it is always correctly set, see above.

sysfs also has topology in slaves/ and holders/. Storaged could export that generically.