ci: Run cockpit tests in PRs #1843
Conversation
|
You still need to actually enable packit on this project to make use of this. See #1820 (comment) Thanks! |
This comment was marked as resolved.
This comment was marked as resolved.
|
So, who would be able to enable packit for the selinux-policy project or the fedora-selinux org? @bachradsusi or @wrabcak? @zpytela told me that he's not permitted to do that. This is semi-urgent, as #1820 was merged without actually enabling packit, so there's no testing of the rpm build any more. |
|
It's Friday 8pm here. I'll take a look on Monday. Ok? |
|
Sure, thanks @bachradsusi ! (It's not that urgent of course, I just wouldn't like to stall it for another month) |
|
/packit test |
|
@martinpitt looks like it does things. Let me know if I missed a thing. |
|
Thanks @bachradsusi ! Yep, I'll wait for the results and then rebase this PR, as it's already a month old. If the RPM builds started to fail since #1820, I'll fix them separately first. |
|
The built COPR packages were uninstallable. Trying a rebase first, that also sorts out automatic notifications. |
|
Cockpit tests failed for commit 3bab283. @martinpitt, @jelly, @mvollmer please check. |
|
Hmm, this uninstallability a problem indeed.. Reproduces easily in fails with which is not quite "systemd, systemd-udev" as on the TF (as the container doesn't have them installed), but same root cause. One can even run In a Fedora 38 cloud VM, it's a bit more straightforward to investigate, as that already has (plus 6 more). So perhaps we need a newer policycoreutils in the game, that was rebuilt against selinux-policy from the 'rawhide' branch instead of from Fedora proper? @bachradsusi @zpytela how did you handle/test this before? There was only the build-rpm workflow before, but that didn't do any installation/validation. How did you test/install branch/PR builds manually before? Thanks! |
|
|
|
Disclaimer: I haven't followed this packit/copr/ci changes so I know almost nothing about it But libselinux conflicts with |
|
Packit used https://download.copr.fedorainfracloud.org/results/packit/fedora-selinux-selinux-policy-1843/fedora-38-x86_64/06440741-selinux-policy/selinux-policy.spec to build selinux-policy and there's already the version changed. Apparently, packit uses latest tag from this repo |
|
I guess you need to use https://packit.dev/docs/configuration#upstream_tag_template
|
|
Thanks @bachradsusi for spotting this! I'll see how to fix this. |
See https://cockpit-project.org/blog/tmt-cross-project-testing.html Drop the install-only tests, as TF only runs the default "install check" test if there are no plans, but now we have one. That will cover the installation/upgrade as preparation step.
|
@zpytela , @WOnder93 This is now working and ready for reviewing and discussion. We've had this running on https://github.com/containers/podman/ and https://github.com/storaged-project/udisks for a while already, and sorted out the initial quirks. In particular, automatic notifications. Nevertheless, please go over the points in the description to make sure we agree on the expectations. Thank you! |
|
@zpytela @bachradsusi : Gentle monthly ping -- can we discuss this at some point? Just yesterday we found https://bugzilla.redhat.com/show_bug.cgi?id=2244759 , such bugs would better be found right away in a PR rather than after landing. |
|
Looks good to me. But I'm not active maintainer of this repo. It needs to be merged by @zpytela |
|
@zpytela pretty please 👀 |
|
Thank you Martin, I am going to merge it now, sorry for the delay. |
|
Thanks @zpytela ! No worries, I actually expected that you still had some questions to discuss, but we can also do that on the life subject when trouble occurs (and it certainly will, as this is still a fairly new thing). Thank you for jumping in, let's see how this will work out! |
See https://cockpit-project.org/blog/tmt-cross-project-testing.html
Drop the install-only tests, as TF only runs the default "install check" test if there are no plans, but now we have one. That will cover the installation/upgrade as preparation step.
When done seriously, this is an ongoing commitment. In our recent years the Cockpit team has reported a couple dozen bug reports found by our tests, many of them were actual regressions. With this setup, we have a good chance of prevent landing such regressions.
This approach is fairly new at least for us in the Fedora world, so let's treat this as an experiment. We do need to talk about commitments and expectations, though. From our side, I propose:
We don't expect you to become experts in our tests. When they fail, it would be nice if you could have a quick look at the log and see if it's something obvious, especially an unexpected SELinux AVC in the log. But in general, we expect that someone from our team will investigate and discuss that with you. The PR where that happened provides a nice place to collect notes.
Whenever one of our tests fail, three of our team members will be notified by packit automatically, so that we can timely investigate them.
We don't expect you to block PRs on these tests, especially not if the testing farm has infrastructure problems . E.g. sometimes they to run into provisioning errors. Just ignore these then -- we see them too in our projects, and will usually prod #testing-farm then.
We would like you to at least give us a chance on that workday to look at a failed test (not infra failure, a "real" one) before you land a PR, so that this all makes sense. If something is urgent and you quick-land, then at least we still retain the benefit of having a trace in which PR tests started to fail, but then the damage is possibly already done. Note that this isn't a big new commitment from our side: we already spend many hours every week looking at regressions, and it takes a lot more effort to track them down weeks after they happened. So this will acually reduce both our and your time spent on hunting down regressions, because the context (PR) is fresh and small, as opposed to "whatever changed in Fedora in the last 4 weeks".
I set this up for Fedora latest and rawhide for the time being. If you think that rawhide has too much churn, of course feel free to drop that from .packit.yaml, or I do it here right away when you want to start slow. We have enabled rawhide in cockpit projects a while ago, and it does often fail due to unrelated reasons. Improving this is pretty much exactly why I start this initiative, and at least knowing about problems is still better IMHO than entirely ignoring rawhide, but it can get annoying.
Please let me know about any other question that you may have. I'm happy to discuss them here or in a gmeet for higher bandwidth.
Thanks!