edge-commit CS9 failure: ostree-remount.service fails to remount /sysroot and /var mountpoints

[mcattamoredhat]

Describe the bug
During edge-commit test case in CS9, our CI has detected that ostree-remount.service fails to remount /sysroot and /var mountpoints.
The output of the playbook task that checks journalctl of ostree-remount does not include the expected output Remounted rw: /sysroot nor Remounted rw: /var

TASK [assert] ******************************************************************
fatal: [192.168.100.50]: FAILED! => {
    "assertion": "'/sysroot' in result_remount_jounalctl.stdout",
    "changed": false,
    "evaluated_to": false,
    "msg": "/sysroot or /var are not remounted by ostree-remount"
}

Inside edge vm:

[admin@vm ~]$ sudo journalctl -u ostree-remount
Feb 12 18:16:57 localhost systemd[1]: Starting OSTree Remount OS/ Bind Mounts...
Feb 12 18:16:57 localhost systemd[1]: Finished OSTree Remount OS/ Bind Mounts.
-- Boot 750cc61b1dcf4a7597e93bc937705d6b --
Feb 12 18:17:12 localhost systemd[1]: Starting OSTree Remount OS/ Bind Mounts...
Feb 12 18:17:12 localhost systemd[1]: Finished OSTree Remount OS/ Bind Mounts.
-- Boot 102a413dd5714e5a8af5ce5b4e394f78 --
Feb 12 18:19:20 localhost systemd[1]: Starting OSTree Remount OS/ Bind Mounts...
Feb 12 18:19:20 localhost systemd[1]: Finished OSTree Remount OS/ Bind Mounts.
-- Boot 12671331afda4e9489744c5e3488f319 --
Feb 12 18:19:35 localhost systemd[1]: Starting OSTree Remount OS/ Bind Mounts...
Feb 12 18:19:35 localhost systemd[1]: Finished OSTree Remount OS/ Bind Mounts.
-- Boot 59099ebb918c4e0a8176227730157c34 --
Feb 12 18:19:50 localhost systemd[1]: Starting OSTree Remount OS/ Bind Mounts...
Feb 12 18:19:50 localhost systemd[1]: Finished OSTree Remount OS/ Bind Mounts.
-- Boot 519566393b414d16862bfc84599bbb30 --
Feb 12 18:20:03 localhost systemd[1]: Starting OSTree Remount OS/ Bind Mounts...
Feb 12 18:20:03 localhost systemd[1]: Finished OSTree Remount OS/ Bind Mounts.
Feb 12 18:37:27 vm systemd[1]: ostree-remount.service: Deactivated successfully.
Feb 12 18:37:27 vm systemd[1]: Stopped OSTree Remount OS/ Bind Mounts.
Feb 12 18:37:27 vm systemd[1]: Stopping OSTree Remount OS/ Bind Mounts...
Feb 12 18:37:27 vm systemd[1]: Starting OSTree Remount OS/ Bind Mounts...
Feb 12 18:37:27 vm systemd[1]: Finished OSTree Remount OS/ Bind Mounts.

[admin@vm ~]$ sudo journalctl -xeu ostree-remount
~
~
Feb 12 18:20:03 localhost systemd[1]: Starting OSTree Remount OS/ Bind Mounts...
░░ Subject: A start job for unit ostree-remount.service has begun execution
░░ Defined-By: systemd
░░ Support: https://access.redhat.com/support
░░ 
░░ A start job for unit ostree-remount.service has begun execution.
░░ 
░░ The job identifier is 159.
Feb 12 18:20:03 localhost systemd[1]: Finished OSTree Remount OS/ Bind Mounts.
░░ Subject: A start job for unit ostree-remount.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://access.redhat.com/support
░░ 
░░ A start job for unit ostree-remount.service has finished successfully.
░░ 
░░ The job identifier is 159.

The service seems to be running but it does not work even after manual restart.

Environment

• OS version (/etc/os-release and /etc/redhat-release):

 + source /usr/libexec/osbuild-composer-test/set-env-variables.sh
++ source /etc/os-release
+++ NAME='CentOS Stream'
+++ VERSION=9
+++ ID=centos
+++ ID_LIKE='rhel fedora'
+++ VERSION_ID=9
+++ PLATFORM_ID=platform:el9
+++ PRETTY_NAME='CentOS Stream 9'
+++ ANSI_COLOR='0;31'
+++ LOGO=fedora-logo-icon
+++ CPE_NAME=cpe:/o:centos:centos:9
+++ HOME_URL=https://centos.org/
+++ BUG_REPORT_URL=https://bugzilla.redhat.com/
+++ REDHAT_SUPPORT_PRODUCT='Red Hat Enterprise Linux 9'
+++ REDHAT_SUPPORT_PRODUCT_VERSION='CentOS Stream'

• osbuild-composer version (rpm -qi osbuild-composer)

 osbuild-composer                         x86_64  101-1.20240212git643be1c.el9      osbuild-composer   17 k
 osbuild-composer-core                    x86_64  101-1.20240212git643be1c.el9      osbuild-composer   10 M
 osbuild-composer-dnf-json                x86_64  101-1.20240212git643be1c.el9      osbuild-composer   11 k
 osbuild-composer-worker                  x86_64  101-1.20240212git643be1c.el9      osbuild-composer   17 M

To Reproduce
Steps to reproduce the behavior:

• git clone https://github.com/osbuild/osbuild.git && cd osbuild/
• make rpm
• sudo dnf -y install rpmbuild/RPMS/noarch/*
• cd ../
• git clone -b glitchtip-integration https://github.com/diaasami/osbuild-composer.git && cd osbuild-composer/
• make rpm
• sudo dnf -y install rpmbuild/RPMS/x86_64/*
• cd test/cases/
• EDGE_USER_PASSWORD=password ./ostree.sh

Expected behavior
ostree-remount.service remounts successfully /sysroot and /var mountpoints.

Additional context
In the steps to reproduce #3907 been used as an example. Although, this failure has been detected so far in several PRs such as:
#3928
#3939
#3828
#3907
#3886
#3934

[cgwalters]

Any test that's parsing output from a tool that's intended solely for human consumption is going to be fragile.

our CI has detected that ostree-remount.service fails to remount /sysroot and /var mountpoints.

But are the mounts actually not in the expected state? That's what the test should be checking right?

As noted in private chat I think what's going on here is we generally expect the mounts setup in the initramfs nowadays, not in this service.