TB in org.osbuild.rpm stage when trying to build edge-commit with microshift sources

[miabbott]

Describe the bug
After configuring osbuild to build an edge-commit according to the downstream microshift install instructions, trying to compose the commit results in a traceback.

Environment

• OS version (/etc/os-release and /etc/redhat-release):

$ cat /etc/os-release 
NAME="Red Hat Enterprise Linux"
VERSION="9.5 (Plow)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="9.5"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Red Hat Enterprise Linux 9.5 Beta (Plow)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_BUGZILLA_PRODUCT_VERSION=9.5
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.5 Beta"
$ cat /etc/redhat-release 
Red Hat Enterprise Linux release 9.5 Beta (Plow)

• osbuild-composer version (rpm -qi osbuild-composer)

$ rpm -qi osbuild-composer
Name        : osbuild-composer
Version     : 101
Release     : 1.el9
Architecture: x86_64
Install Date: Wed Apr 10 11:40:19 2024
Group       : Unspecified
Size        : 18399
License     : Apache-2.0
Signature   : RSA/SHA256, Tue Mar  5 05:12:56 2024, Key ID 199e2f91fd431d51
Source RPM  : osbuild-composer-101-1.el9.src.rpm
Build Date  : Mon Feb 26 07:26:12 2024
Build Host  : x86-64-05.build.eng.rdu2.redhat.com
Packager    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>
Vendor      : Red Hat, Inc.
URL         : https://github.com/osbuild/osbuild-composer
Summary     : An image building service based on osbuild
Description :

A service for building customized OS artifacts, such as VM images and OSTree
commits, that uses osbuild under the hood. Besides building images for local
usage, it can also upload images directly to cloud.

It is compatible with composer-cli and cockpit-composer clients.

To Reproduce

1. Follow the instructions to install microshift in an edge commit.
2. composer-cli compose start-ostree <blueprint> edge-commit

Expected behavior
Successful compose of edge-commit

Additional context
On my osbuild system, the microshift + fast-datapath sources I added were pointing at repos that are behind the Red Hat firewall/VPN, rather than using the CDN sources. It's possible (likely?) that I didn't configure the options in the TOML used for the sources correctly.

However, I would still expect that the compose would fail more gracefully.

See relevant osbuild-composer log snippet below:

Apr 10 12:02:51 rhel9imagebuilder osbuild-composer[4831]: 2024/04/10 12:02:51 POST /api/v1/compose
Apr 10 12:02:51 rhel9imagebuilder osbuild-composer[4831]: Distro rhel-8-beta has repositories defined, but it's not supported. Skipping.
Apr 10 12:02:51 rhel9imagebuilder osbuild-composer[4831]: Distro rhel-8-beta has repositories defined, but it's not supported. Skipping.
Apr 10 12:02:51 rhel9imagebuilder osbuild-composer[4831]: 2024/04/10 12:02:51 Please note that user customizations on "edge-commit" image type are deprecated and will be removed in the near future
Apr 10 12:02:51 rhel9imagebuilder osbuild-composer[4831]: time="2024-04-10T12:02:51-04:00" level=warning msg="Failed to load consumer certs: no consumer key found" func=github.com/osbuild/images/pkg/rhsm.LoadSystemSubscriptions file="/builddir/build/BUILD/osbuild-composer-101/_build/src/github.com/osbuild/osbuild-composer/vendor/github.com/osbuild/images/pkg/rhsm/secrets.go:104"
Apr 10 12:03:04 rhel9imagebuilder osbuild-worker[4832]: time="2024-04-10T12:03:04-04:00" level=info msg="Running job 'b1fb9df1-7b8e-4c7b-be39-d19f5ec2458e' (osbuild)\n"
Apr 10 12:04:15 rhel9imagebuilder kernel: loop: module loaded
Apr 10 12:04:39 rhel9imagebuilder osbuild-worker[4832]: time="2024-04-10T12:04:39-04:00" level=info msg="build pipeline results:\n" jobId=b1fb9df1-7b8e-4c7b-be39-d19f5ec2458e
Apr 10 12:04:39 rhel9imagebuilder osbuild-worker[4832]: time="2024-04-10T12:04:39-04:00" level=info msg="  org.osbuild.rpm success" jobId=b1fb9df1-7b8e-4c7b-be39-d19f5ec2458e
Apr 10 12:04:39 rhel9imagebuilder osbuild-worker[4832]: time="2024-04-10T12:04:39-04:00" level=info msg="  org.osbuild.selinux success" jobId=b1fb9df1-7b8e-4c7b-be39-d19f5ec2458e
Apr 10 12:04:39 rhel9imagebuilder osbuild-worker[4832]: time="2024-04-10T12:04:39-04:00" level=info msg="os pipeline results:\n" jobId=b1fb9df1-7b8e-4c7b-be39-d19f5ec2458e
Apr 10 12:04:39 rhel9imagebuilder osbuild-worker[4832]: time="2024-04-10T12:04:39-04:00" level=info msg="  org.osbuild.rpm failure:" jobId=b1fb9df1-7b8e-4c7b-be39-d19f5ec2458e
Apr 10 12:04:39 rhel9imagebuilder osbuild-worker[4832]: time="2024-04-10T12:04:39-04:00" level=info msg="    Failed to open file \"/sys/fs/selinux/checkreqprot\": Read-only file system" jobId=b1fb9df1-7b8e-4c7b-be39-d19f5ec2458e
Apr 10 12:04:39 rhel9imagebuilder osbuild-worker[4832]: time="2024-04-10T12:04:39-04:00" level=info msg="    dbpath set to '/usr/share/rpm'" jobId=b1fb9df1-7b8e-4c7b-be39-d19f5ec2458e
Apr 10 12:04:39 rhel9imagebuilder osbuild-worker[4832]: time="2024-04-10T12:04:39-04:00" level=info msg="    Signature check failed on sha256:e28029077732534cb644368e5c8f0dde798670429b6778e27cb21dd581168368, lookup package name in manifest." jobId=b1fb9df1-7b8e-4c7b-be39-d19f5ec2458e
Apr 10 12:04:39 rhel9imagebuilder osbuild-worker[4832]: time="2024-04-10T12:04:39-04:00" level=info msg="    Traceback (most recent call last):" jobId=b1fb9df1-7b8e-4c7b-be39-d19f5ec2458e
Apr 10 12:04:39 rhel9imagebuilder osbuild-worker[4832]: time="2024-04-10T12:04:39-04:00" level=info msg="      File \"/run/osbuild/bin/org.osbuild.rpm\", line 382, in <module>" jobId=b1fb9df1-7b8e-4c7b-be39-d19f5ec2458e
Apr 10 12:04:39 rhel9imagebuilder osbuild-worker[4832]: time="2024-04-10T12:04:39-04:00" level=info msg="        r = main(args[\"tree\"], args[\"inputs\"], args[\"options\"])" jobId=b1fb9df1-7b8e-4c7b-be39-d19f5ec2458e
Apr 10 12:04:39 rhel9imagebuilder osbuild-worker[4832]: time="2024-04-10T12:04:39-04:00" level=info msg="      File \"/run/osbuild/bin/org.osbuild.rpm\", line 292, in main" jobId=b1fb9df1-7b8e-4c7b-be39-d19f5ec2458e
Apr 10 12:04:39 rhel9imagebuilder osbuild-worker[4832]: time="2024-04-10T12:04:39-04:00" level=info msg="        subprocess.run([" jobId=b1fb9df1-7b8e-4c7b-be39-d19f5ec2458e
Apr 10 12:04:39 rhel9imagebuilder osbuild-worker[4832]: time="2024-04-10T12:04:39-04:00" level=info msg="      File \"/usr/lib64/python3.9/subprocess.py\", line 528, in run" jobId=b1fb9df1-7b8e-4c7b-be39-d19f5ec2458e
Apr 10 12:04:39 rhel9imagebuilder osbuild-worker[4832]: time="2024-04-10T12:04:39-04:00" level=info msg="        raise CalledProcessError(retcode, process.args," jobId=b1fb9df1-7b8e-4c7b-be39-d19f5ec2458e
Apr 10 12:04:39 rhel9imagebuilder osbuild-worker[4832]: time="2024-04-10T12:04:39-04:00" level=info msg="    subprocess.CalledProcessError: Command '['rpmkeys', '--dbpath', '/usr/share/rpm', '--root', '/run/osbuild/tree', '--checksig', 'sha256:e28029077732534cb644368e5c8f0dde798670429b6778e27cb21dd581168368']' returned non-zero exit status 1." jobId=b1fb9df1-7b8e-4c7b-be39-d19f5ec2458e
Apr 10 12:04:39 rhel9imagebuilder osbuild-worker[4832]: time="2024-04-10T12:04:39-04:00" level=info msg="    " jobId=b1fb9df1-7b8e-4c7b-be39-d19f5ec2458e
Apr 10 12:04:39 rhel9imagebuilder osbuild-worker[4832]: time="2024-04-10T12:04:39-04:00" level=error msg="osbuild job failed: osbuild build failed" jobId=b1fb9df1-7b8e-4c7b-be39-d19f5ec2458e
Apr 10 12:04:39 rhel9imagebuilder osbuild-worker[4832]: time="2024-04-10T12:04:39-04:00" level=error msg="failure details : [osbuild error: [123 34 116 121 112 101 34 58 32 34 111 114 103 46 111 115 98 117 105 108 100 46 101 114 114 111 114 46 115 116 97 103 101 34 44 32 34 100 101 116 97 105 108 115 34 58 32 123 34 115 116 97 103 101 34 58 32 123 34 105 100 34 58 32 34 101 52 101 98 54 97 56 100 54 97 98 57 50 49 102 101 49 48 52 98 50 53 56 55 100 97 55 51 55 102 56 57 99 54 48 99 101 57 56 102 101 49 99 101 98 53 101 49 51 57 51 49 50 50 101 101 57 56 102 57 97 97 98 53 34 44 32 34 116 121 112 101 34 58 32 34 111 114 103 46 111 115 98 117 105 108 100 46 114 112 109 34 44 32 34 111 117 116 112 117 116 34 58 32 34 70 97 105 108 101 100 32 116 111 32 111 112 101 110 32 102 105 108 101 32 92 34 47 115 121 115 47 102 115 47 115 101 108 105 110 117 120 47 99 104 101 99 107 114 101 113 112 114 111 116 92 34 58 32 82 101 97 100 45 111 110 108 121 32 102 105 108 101 32 115 121 115 116 101 109 92 110 100 98 112 97 116 104 32 115 101 116 32 116 111 32 39 47 117 115 114 47 115 104 97 114 101 47 114 112 109 39 92 110 83 105 103 110 97 116 117 114 101 32 99 104 101 99 107 32 102 97 105 108 101 100 32 111 110 32 115 104 97 50 53 54 58 101 50 56 48 50 57 48 55 55 55 51 50 53 51 52 99 98 54 52 52 51 54 56 101 53 99 56 102 48 100 100 101 55 57 56 54 55 48 52 50 57 98 54 55 55 56 101 50 55 99 98 50 49 100 100 53 56 49 49 54 56 51 54 56 44 32 108 111 111 107 117 112 32 112 97 99 107 97 103 101 32 110 97 109 101 32 105 110 32 109 97 110 105 102 101 115 116 46 92 110 84 114 97 99 101 98 97 99 107 32 40 109 111 115 116 32 114 101 99 101 110 116 32 99 97 108 108 32 108 97 115 116 41 58 92 110 32 32 70 105 108 101 32 92 34 47 114 117 110 47 111 115 98 117 105 108 100 47 98 105 110 47 111 114 103 46 111 115 98 117 105 108 100 46 114 112 109 92 34 44 32 108 105 110 101 32 51 56 50 44 32 105 110 32 60 109 111 100 117 108 101 62 92 110 32 32 32 32 114 32 61 32 109 97 105 110 40 97 114 103 115 91 92 34 116 114 101 101 92 34 93 44 32 97 114 103 115 91 92 34 105 110 112 117 116 115 92 34 93 44 32 97 114 103 115 91 92 34 111 112 116 105 111 110 115 92 34 93 41 92 110 32 32 70 105 108 101 32 92 34 47 114 117 110 47 111 115 98 117 105 108 100 47 98 105 110 47 111 114 103 46 111 115 98 117 105 108 100 46 114 112 109 92 34 44 32 108 105 110 101 32 50 57 50 44 32 105 110 32 109 97 105 110 92 110 32 32 32 32 115 117 98 112 114 111 99 101 115 115 46 114 117 110 40 91 92 110 32 32 70 105 108 101 32 92 34 47 117 115 114 47 108 105 98 54 52 47 112 121 116 104 111 110 51 46 57 47 115 117 98 112 114 111 99 101 115 115 46 112 121 92 34 44 32 108 105 110 101 32 53 50 56 44 32 105 110 32 114 117 110 92 110 32 32 32 32 114 97 105 115 101 32 67 97 108 108 101 100 80 114 111 99 101 115 115 69 114 114 111 114 40 114 101 116 99 111 100 101 44 32 112 114 111 99 101 115 115 46 97 114 103 115 44 92 110 115 117 98 112 114 111 99 101 115 115 46 67 97 108 108 101 100 80 114 111 99 101 115 115 69 114 114 111 114 58 32 67 111 109 109 97 110 100 32 39 91 39 114 112 109 107 101 121 115 39 44 32 39 45 45 100 98 112 97 116 104 39 44 32 39 47 117 115 114 47 115 104 97 114 101 47 114 112 109 39 44 32 39 45 45 114 111 111 116 39 44 32 39 47 114 117 110 47 111 115 98 117 105 108 100 47 116 114 101 101 39 44 32 39 45 45 99 104 101 99 107 115 105 103 39 44 32 39 115 104 97 50 53 54 58 101 50 56 48 50 57 48 55 55 55 51 50 53 51 52 99 98 54 52 52 51 54 56 101 53 99 56 102 48 100 100 101 55 57 56 54 55 48 52 50 57 98 54 55 55 56 101 50 55 99 98 50 49 100 100 53 56 49 49 54 56 51 54 56 39 93 39 32 114 101 116 117 114 110 101 100 32 110 111 110 45 122 101 114 111 32 101 120 105 116 32 115 116 97 116 117 115 32 49 46 92 110 34 44 32 34 101 114 114 111 114 34 58 32 110 117 108 108 125 125 125]]" jobId=b1fb9df1-7b8e-4c7b-be39-d19f5ec2458e
Apr 10 12:04:39 rhel9imagebuilder osbuild-worker[4832]: time="2024-04-10T12:04:39-04:00" level=info msg="Job 'b1fb9df1-7b8e-4c7b-be39-d19f5ec2458e' (osbuild) finished"

microshift source TOML

id = "rhocp-4.15"
name = "Red Hat OpenShift Container Platform 4.15 for RHEL 9"
type = "yum-baseurl"
url = "https://<internal hostname>/content/dist/layered/rhel9/x86_64/rhocp/4.15/os/"
check_gpg = true
check_ssl = true
system = false
rhsm = false

fast-datapath source TOML

id = "fast-datapath"
name = "Fast Datapath for RHEL 9"
type = "yum-baseurl"
url = "https://<internal hostname>/content/dist/layered/rhel9/x86_64/fast-datapath/os/"
check_gpg = true
check_ssl = true
system = false
rhsm = false

microshift blueprint

name = "microshift"
description = ""
version = "0.0.4"
modules = []
groups = []
distro = "rhel-94"

[[packages]]
name = "microshift"
version = "*"

[customizations.services]
enabled = ["microshift"]

[bcl]

That looks like it may be a gpg signature check failure (sorry it's so ugly). It looks like your sources have check_gpg on but no gpg key. If the packages are signed with a key the system doesn't know about it will fail. See https://osbuild.org/docs/on-premises/installation/managing-repositories -- although now that I look at that again it isn't clear that you can add gpg keys to the source. I'd first try turning off the check_gpg flag to see if that fixes things, then get the right key added.