Add function to check if a BTRFS volume is complete

[vpodzime]

If BTRFS volume uses multiple devices, it's not easy for the user-space code (e.g. UDisks2) to determine if all the used devices are present in the system or not. There are ways to check it, but no API/library function. libblockdev-btrfs should cover this.

See also storaged-project/udisks#88

[vpodzime]

@cmurf, I'm trying to work on this with no success. I've created a RAID0 (data) btrfs volume on two loop devices and it's a mystery to me how to determine if the volume has all devices or not. With just one of the loop devices setup I'm getting this:

[root@localhost tmp]# wipefs /dev/loop0
offset               type
----------------------------------------------------------------
0x10040              btrfs   [filesystem]
                     UUID:  14872e8e-b34f-4021-a154-9f67695ab366

[root@localhost tmp]# udevadm info -qall /dev/loop0
P: /devices/virtual/block/loop0
N: loop0
S: disk/by-uuid/14872e8e-b34f-4021-a154-9f67695ab366
E: DEVLINKS=/dev/disk/by-uuid/14872e8e-b34f-4021-a154-9f67695ab366
E: DEVNAME=/dev/loop0
E: DEVPATH=/devices/virtual/block/loop0
E: DEVTYPE=disk
E: ID_BTRFS_READY=1
E: ID_FS_TYPE=btrfs
E: ID_FS_USAGE=filesystem
E: ID_FS_UUID=14872e8e-b34f-4021-a154-9f67695ab366
E: ID_FS_UUID_ENC=14872e8e-b34f-4021-a154-9f67695ab366
E: ID_FS_UUID_SUB=7067a06f-09ca-4a5d-a598-15e228841ccd
E: ID_FS_UUID_SUB_ENC=7067a06f-09ca-4a5d-a598-15e228841ccd
E: MAJOR=7
E: MINOR=0
E: SUBSYSTEM=block
E: TAGS=:systemd:
E: USEC_INITIALIZED=9699992580

[root@localhost tmp]# btrfs device usage /dev/loop0
ERROR: not a btrfs filesystem: /dev/loop0
[root@localhost tmp]# btrfs device scan /dev/loop0
Scanning for Btrfs filesystems in '/dev/loop0'
[root@localhost tmp]# echo $?
0
[root@localhost tmp]# mkdir mnt
[root@localhost tmp]# mount /dev/loop0 mnt/
mount: wrong fs type, bad option, bad superblock on /dev/loop0,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.
[root@localhost tmp]# dmesg|tail
[177023.844490] BTRFS: device fsid fda3d655-ae0b-45bd-82ba-86044c4735ca devid 2 transid 3 /dev/loop1
[177260.556201] BTRFS: device fsid c394edf9-63a3-4fb1-b4d7-11075633f050 devid 1 transid 3 /dev/loop0
[177260.559870] BTRFS: device fsid c394edf9-63a3-4fb1-b4d7-11075633f050 devid 2 transid 3 /dev/loop1
[177647.170877] BTRFS: device fsid 14872e8e-b34f-4021-a154-9f67695ab366 devid 1 transid 3 /dev/loop0
[177647.174403] BTRFS: device fsid 14872e8e-b34f-4021-a154-9f67695ab366 devid 2 transid 3 /dev/loop1
[178544.760069] BTRFS info (device loop0): disk space caching is enabled
[178544.760071] BTRFS info (device loop0): has skinny extents
[178544.760071] BTRFS info (device loop0): flagging fs with big metadata feature
[178544.760506] BTRFS error (device loop0): failed to read chunk tree: -5
[178544.794518] BTRFS error (device loop0): open_ctree failed

There seems to be nothing relevant to this issue under /sys/class/block/loop0/ and looking at /usr/include/linux/btrfs.h I don't see anything that could help either.

Any ideas?

[cmurf]

Yes, it's confusing for me as well. Here are some breadcrumbs.

btrfs device usage wants a mounted file system, and the mount is failing in this case I'm guessing because it can't find missing devices and the mount option doesn't have -o degraded.

/usr/lib/udev/rules.d/64-btrfs.rules I think ends up calling btrfs device ready or maybe directly calls BTRFS_IOC_DEVICES_READY. But the man page for btrfs device just says ready waits around. And in fact whether all devices are present or not I get the same command completion, and exit code even when a volume is missing a device. So I just posted a message to linux-btrfs@ about that.

When I strace btrfs fi show which lists all devices for Btrfs volumes and even shows if some are missing, BTRFS_IOC_FS_INFO shows how many devices per volume there are, and then BTRFS_IOC_DEV_INFO can find the path if found.

[vpodzime]

I'm afraid we are not able to resolve this issue now.

[cmurf]

What additional information is needed? Is there something upstream needs to provide or explain?

[vpodzime]

What additional information is needed? Is there something upstream needs to provide or explain?

Well, your previous comment (#244 (comment)) sounds to me like we have no option to do what this issue is about. If I understand it correctly, we would have to try to mount the volume, right?

[cmurf]

btrfs device ready acts on a block device, not on a mount point.

systemd-udev already has a wrapper for this in src/udev/udev-builtin-btrfs.c which is then used by rules/64-btrfs.rules. If a 2 device Btrfs volume is used for rootfs, this rule causes a hang during startup until both devices are found, systemd won't even try to mount until all devices are ready per this rule. So my question is why this udev rule doesn't also inhibit udisksd from trying to mount a Btrfs volume until all devices are ready.

If you just want information about a volume's members without mounting, that's btrfs filesystem show which by itself lists all Btrfs volumes and their devices regardless of whether they're mounted; and optionally it accepts any of [<path>|<uuid>|<device>|label]. But I figured that's too high level for what you're ultimately after because it returns human rather than machine readable information, which is why I mentioned the ioctl's it uses. BTRFS_IOC_FS_INFO and BTRFS_IOC_DEV_INFO do not require a mounted volume.

Also the problem of udisksd trying to mount an unready volume (individually trying to mount member devices as udisksd sees them appear) is not a big deal, it just results in the generic open_ctree error. The real problem is when asking e.g. in GUI, to umount multiple device Btrfs, udisks gets really confused, does not properly umount the filesystem before it starts deleting device nodes, and the result is fs corruption. https://bugs.freedesktop.org/show_bug.cgi?id=87277#c3

[vpodzime]

btrfs device ready acts on a block device, not on a mount point.

Yes, but it doesn't seem to work as we expect/need. Here's the example from a shell with RAID1 (both data and metadata) btrfs volume on /dev/loop0 and /dev/loop1 with /dev/loop1 removed:

[root@localhost tmp]# btrfs device ready /dev/loop0
[root@localhost tmp]# echo $?
0
[root@localhost tmp]# mkdir mnt
[root@localhost tmp]# mount /dev/loop0 mnt
mount: wrong fs type, bad option, bad superblock on /dev/loop0,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

(Also interesting that it's impossible to mount a RAID1 volume without the second disk, btw)

The only thing that seems to provide some useful data is:

[root@localhost tmp]# btrfs fi show /dev/loop0
warning, device 2 is missing
Label: none  uuid: 25426f60-0a0e-43b7-86f1-0c16cf1e362d
	Total devices 2 FS bytes used 112.00KiB
	devid    1 size 1.00GiB used 213.00MiB path /dev/loop0
	*** Some devices missing
[root@localhost tmp]# echo $?
0

except that it, of course (sarcasm), exits with code 0.

[cmurf]

Yes, but it doesn't seem to work as we expect/need.

The user space command doesn't seem to work as anyone expects, so I'm kinda beating up on upstream about what's going on because the ioctl does work as intended within the udev wrapper.
https://www.spinics.net/lists/linux-btrfs/msg67517.html

(Also interesting that it's impossible to mount a RAID1 volume without the second disk, btw)

It requires mount -o degraded <dev> There's no automatic degraded mount for Btrfs because it comes with excessive consequences, so the present behavior is to hard fail and make sure the user explicitly opts into degraded mounts.

except that it, of course (sarcasm), exits with code 0.

I expect only exit 0 from this command, or there's a bug. I'd sarcastically sooner expect a crash than non-zero.

[cmurf]

OK that upstream thread has been updated, and now I understand. Our tests are flawed due to misunderstanding how the devices_ready ioctl works. Once the kernel sees all devices, ready will exit 0. Even if a device subsequently goes missing, ready continues to exit 0. Only if the kernel has never seen all devices will it exit 1.

The kernel does not forget even if all devices are removed. If I re-add one of the devices, btrfs device ready still exits 0. Ick.

[vpodzime]

Funny story this one... Anyway, I take it the only reliable way to see if all devices from a volume are present is to use btrfs fi show and parse its output, right? (or the respective ioctl() of course)

[cmurf]

If mounting normally succeeds, by definition all devices are present. There's no harm in trying, but if they aren't all present, you get a generic mount failure message and therefore the failure reason is uncertain.

Otherwise, yes, and I'd say it's not worth doing the work to parse the output until something like this happens to unify the tools:
https://www.spinics.net/lists/linux-btrfs/msg67003.html