Machines upgraded from old FCOS releases use bootloaders denylisted in newer UEFI dbx

[dustymabe]

I ran into this when testing the new extended upgrade testing scaffolding:

• tests/kola: add extended upgrade test fedora-coreos-config#2310
• tests/kola: a few updates for the upgrade test fedora-coreos-config#2335
• jobs: add kola-upgrade test fedora-coreos-pipeline#847

Trying to start old < F34 systems with secure boot enabled using current coreos-assembler yields a non-booting system:

BdsDxe: failed to load Boot0001 "UEFI Misc Device" from PciRoot(0x0)/Pci(0x2,0x0): Access Denied
BdsDxe: No bootable option or device was found.                                          
BdsDxe: Press any key to enter the Boot Manager Menu.                                           

In this specific case this log came from trying to boot fedora-coreos-33.20201116.2.0-qemu.x86_64.qcow2

This is almost certainly because newer OVMF includes an updated dbx.

Documenting this here so we can point to it in the future.

[bgilbert]

At present we're automatically updating neither shim/GRUB nor the dbx denylist. That has some consequences:

1. Old installs will continue to work on old hardware as long as dbx isn't manually updated. We do ship fwupd and dbxtool (the dbx updater used by fwupd) and a user could invoke fwupdmgr update without updating their bootloader first. If fwupd allows this, it'll brick the bootloader. fwupd has code to refuse the update if the ESP has EFI binaries with old signatures, but we don't know how fwupd responds to FCOS's ESP quirks.
   • Investigate: in a FCOS VM with old UEFI firmware, upgraded from a very old FCOS release, and with an old bootloader, does fwupdmgr update correctly refuse to update dbx? What about on a system installed with boot_device.mirror?
2. Old install images will not boot in Secure Boot mode on UEFI firmware with newer dbx. Those images aren't supported anyway, so this seems fine.
3. By not automatically updating dbx, we're leaving older installs vulnerable to Secure Boot bypass via exploitation of a compromised bootloader (either the one we shipped, or a different one listed in newer dbx).
   • Investigate: in a FCOS VM with old UEFI firmware, upgraded from a very old FCOS release, and with a bootloader updated via bootupd, does fwupdmgr update correctly update dbx? What about on a system installed with boot_device.mirror?
   • Decide: how do we want to handle this? Pursue automatic bootloader + dbx updates? Document using fwupd to manually update dbx? We generally design FCOS not to require manual steps to maintain best-practice security.

[dustymabe]

We discussed this in the community meeting this week.

13:34:55   dustymabe | #agreed We've identified that we'd ultimately like to regularly
                     | update both the bootloader and dbx. For now there are some
                     | preliminary investigations that we can do to determine the scope
                     | and challenges related to the dbx updates and there are bootloader
                     | update safety features we'd like to investigate and implement in
                     | bootupd.

We agreed to open up new issues to track the feature requests:

• regular bootloader updates
• regular dbx updates

[dustymabe]

New issues were opened:

• Enable automatic bootloader updates #1468
• Regular updates for the UEFI revoked signatures database (dbx) #1478

I'm going to close this issue out because there is nothing for us to do for it outside of the two issues linked above.