Storage Multipath

What is it?

"Multipath" is the situation where a storage device is reachable via more than one kernel block device. Writing to a block number using one of the block devices will be visible when reading the same block number from any of the other block devices.

This happens when a storage device is connected via multiple cables to a machine, to increase reliability and performance, or by accident.

This reliability and performance improvement is achieved by combining the multiple block devices via the device mapper into a new, single block device. However, the other block devices are still visible and accessible.

Why should Cockpit care?

Multipath devices must be used carefully. Sometimes the members are protected, sometimes they are not.

A multipath member that is in active use (is part of a device mapper table) is reasonably protected against accidents. It can not be opened with O_EXCL, and thus mkfs, mount, wipefs, pvcreate, etc will refuse to touch it in the same way that they refuse to touch a block device with a mounted filesystem on it.

A multipath member that is not in active use, is not protected and it is dangerous to use it individually. Opening it with O_EXCL will succeed, and it is thus for example possible to mount the same storage device in two places with disastrous results.

Multipath members might not be in active use for a number of reasons:

• Multipath support isn't enabled yet.
• Multipathd couldn't set up a device mapper table because some of the members are already (and dangerously) in use for something, like being mounted somehwere or used as a physical volume.
• A new path has appeared at run-time and multipathd has not yet been poked to process it.
• A path has failed and has been removed from the device mapper table.
• A path has been manually removed from the table.

Thus, Cockpit needs to at least prevent people from using the individual members of a multipath device, both to avoid obscure error messages, and to prevent disasters.

Since multipath is an important subsystem and does not work fully autonomously in the background, Cockpit should have a UI for it in general.

In order of decreasing importance:

• Enable multipathd and start it.
• Managing the topology
• ...

What has been done

As a first step, Cockpit is made "multipath safe". It will show the multipath topology correctly, and will prevent people from using anything but the 'master' block device. This will work both when multipathd is running and correctly configured and also when it is not running or not properly configured.

Storaged already correctly identifies multiple paths to the same storage device by setting the Block.Drive property of all of them to point to the same Drive object. Previously, Cockpit has ignored the possibility that a Drive might have multiple Block objects to it, and has simply picked one of them at random.

Now Cockpit collects all Block objects that point to a single Drive object and displays them.

Cockpit identifies 'slaves' by looking for IdType=mpath_member. If there is exactly one non-slave, this is assumed to be the 'master' device mapper device. If there is more than one (or no) 'master', then things haven't been properly set up for this multipath device and Cockpit will not show its content at all.

https://github.com/cockpit-project/cockpit/pull/2596

As a second step, Cockpit detects multipath devices that are in a non-working state and offers two kinds of advices:

• When multipathd is not running, it explains that and shows a "Activate" button that will hopefully start multipathd. This is the state that people run into by default, since multipathd is not running by default.

• When multipathd is running, this is also exlained, but not corrective action is offered since there is nothing simple and obvious that needs doing.

https://github.com/cockpit-project/cockpit/pull/2601

What still needs to be done

Cockpit should show the state of multipathd, such as failed and excluded paths. It could also show performance metrics for the individual paths. It should allow adding and removing paths.

Random notes

Virtio devices are silently blacklisted by default. Use SCSI.

Multipath support needs to be explicitly enabled:

# dnf install device-mapper-multipath
# mpathconf --enable
# multipathd reconfigure || systemctl start multipathd
# udevadm trigger

Cockpit/storaged could depend on device-mapper-multipath. The final "udevadm trigger" should not be necessary and multipathd should trigger enough uevents. https://bugzilla.redhat.com/show_bug.cgi?id=1254583

Multipathd ignores multipath devices that are already mounted, are a physical volume in a volume group with at least one active logical volume on it, or are otherwise in use. Thus, when Cockpit finds a improperly configured multipath device, it needs to unmount all of its partitions, deactivate all logical volumes, etc, and only then enable/start/reconfigure multipathd as above.

The multipath topology/state can be retrieved like so:

# multipathd list paths format '%d %m %o'
dev multipath dev_st 
sda [orphan]  running
sdb mpatha    running
sdc mpatha    running

It might be worthwhile to teach multipathd machine-readable output, maybe JSON.

Multipath members have this udev property:

ID_FS_TYPE=mpath_member

but there aren't enough uevents to ensure that it is always correctly set, see above.

sysfs also has topology in slaves/ and holders/. Storaged could export that generically.