Required Checks are not being respected when cancelling jobs

[hubertp]

We had recently at least 2 instances when not all Required Checks passed but were still merged. With automatic merge on (CI: Ready to merge label) we had:

• Fix a bad merge between #11926 and #11399 #12022
• Cleanup jline dependencies #12020

In those examples some PR checks failed, one manually triggered a re-run because of flakiness. Then one cancelled the run and the label got picked up and PR merged.
PRs should never be allowed to be merged with required checks not being run successfully.