OCPBUGS-30239: Prevent OVS-configuration to run before kdump #4213
Conversation
Kdump will generate it's initramfs based on the running state when kdump.service run If OVS has already run, the kdump fails to gather a working network config, which prevent network log exports, sush as SSH. See https://issues.redhat.com/browse/OCPBUGS-28239
|
/title OCPBUGS-28239: Prevent OVS-configuration to run before kdump |
|
/retitle OCPBUGS-28239: Prevent OVS-configuration to run before kdump |
openshift-ci
bot
changed the title
|
/retest |
|
/approve Seems safe, but also just in case /hold cc @trozet @cybertron in case there's any concerns, as the networking team generally manages that template |
openshift-ci
bot
added
the
do-not-merge/hold
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: jbtrystram, travier, yuqi-zhang The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
bot
added
the
approved
|
This PR looks good to me! |
|
/retest |
|
Makes sense. I have also run into issues using an ovs bridge in an initrd so I can see where that would be a problem. |
|
/jira refresh Not sure why the bot did not pick up the reference in the PR title. |
|
@cybertron: Jira Issue OCPBUGS-28239 is in a security level that is not in the allowed security levels for this repo.
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/retitle OCPBUGS-30239: Prevent OVS-configuration to run before kdump |
openshift-ci
bot
changed the title
openshift-ci-robot
added
jira/valid-reference
|
@jbtrystram: This pull request references Jira Issue OCPBUGS-30239, which is invalid:
Comment The bug has been updated to refer to the pull request using the external bug tracker. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/jira refresh |
|
@jbtrystram: This pull request references Jira Issue OCPBUGS-30239, which is invalid:
Comment In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/jira refresh |
openshift-ci-robot
added
jira/valid-bug
|
@jbtrystram: This pull request references Jira Issue OCPBUGS-30239, which is valid. The bug has been moved to the POST state. 3 validation(s) were run on this bug
Requesting review from QA contact: In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/retest |
|
Note that this requires a fresh build of RHCOS with this fix |
|
The PR has all the necessary labels, just seems that single node job is flaky. /test e2e-gcp-op-single-node |
|
/hold Revision e1770af was retested 3 times: holding |
openshift-ci
bot
added
the
do-not-merge/hold
|
Let's retry |
openshift-ci
bot
removed
the
do-not-merge/hold
|
@jbtrystram: The following test failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
openshift-merge-bot
bot
merged commit a24148c
into
openshift:master
|
@jbtrystram: Jira Issue OCPBUGS-30239: All pull requests linked via external trackers have merged: Jira Issue OCPBUGS-30239 has been moved to the MODIFIED state. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/cherry-pick release-4.15 |
|
@sinnykumari: new pull request created: #4259 In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
[ART PR BUILD NOTIFIER] This PR has been included in build ose-machine-config-operator-container-v4.16.0-202403141746.p0.ga24148c.assembly.stream.el8 for distgit ose-machine-config-operator. |
|
Fix included in accepted release 4.16.0-0.nightly-2024-03-21-152650 |
When overwriting a systemd unit with new content, we need to account for the case where the new unit content has a different `[Install]` section. If it does, then simply overwriting will leak the previous enablement symlinks and become node state. That's OK most of the time, but this can cause real issues as we've seen with the combination of openshift#3967 which does exactly that (changing `[Install]` sections) and openshift#4213 which assumed that those symlinks were cleaned up. More details on that cocktail in: https://issues.redhat.com/browse/OCPBUGS-33694?focusedId=24917003#comment-24917003 Fix this by always checking if the unit is currently enabled, and if so, running `systemctl disable` *before* overwriting its contents. The unit will then be re-enabled (or not) based on the MachineConfig.
When overwriting a systemd unit with new content, we need to account for the case where the new unit content has a different `[Install]` section. If it does, then simply overwriting will leak the previous enablement symlinks and become node state. That's OK most of the time, but this can cause real issues as we've seen with the combination of openshift#3967 which does exactly that (changing `[Install]` sections) and openshift#4213 which assumed that those symlinks were cleaned up. More details on that cocktail in: https://issues.redhat.com/browse/OCPBUGS-33694?focusedId=24917003#comment-24917003 Fix this by always checking if the unit is currently enabled, and if so, running `systemctl disable` *before* overwriting its contents. The unit will then be re-enabled (or not) based on the MachineConfig. Fixes: https://issues.redhat.com/browse/OCPBUGS-33694
When overwriting a systemd unit with new content, we need to account for the case where the new unit content has a different `[Install]` section. If it does, then simply overwriting will leak the previous enablement symlinks and become node state. That's OK most of the time, but this can cause real issues as we've seen with the combination of openshift#3967 which does exactly that (changing `[Install]` sections) and openshift#4213 which assumed that those symlinks were cleaned up. More details on that cocktail in: https://issues.redhat.com/browse/OCPBUGS-33694?focusedId=24917003#comment-24917003 Fix this by always checking if the unit is currently enabled, and if so, running `systemctl disable` *before* overwriting its contents. The unit will then be re-enabled (or not) based on the MachineConfig. Fixes: https://issues.redhat.com/browse/OCPBUGS-33694
When overwriting a systemd unit with new content, we need to account for the case where the new unit content has a different `[Install]` section. If it does, then simply overwriting will leak the previous enablement symlinks and become node state. That's OK most of the time, but this can cause real issues as we've seen with the combination of openshift#3967 which does exactly that (changing `[Install]` sections) and openshift#4213 which assumed that those symlinks were cleaned up. More details on that cocktail in: https://issues.redhat.com/browse/OCPBUGS-33694?focusedId=24917003#comment-24917003 Fix this by always checking if the unit is currently enabled, and if so, running `systemctl disable` *before* overwriting its contents. The unit will then be re-enabled (or not) based on the MachineConfig. Fixes: https://issues.redhat.com/browse/OCPBUGS-33694
When overwriting a systemd unit with new content, we need to account for the case where the new unit content has a different `[Install]` section. If it does, then simply overwriting will leak the previous enablement symlinks and become node state. That's OK most of the time, but this can cause real issues as we've seen with the combination of openshift#3967 which does exactly that (changing `[Install]` sections) and openshift#4213 which assumed that those symlinks were cleaned up. More details on that cocktail in: https://issues.redhat.com/browse/OCPBUGS-33694?focusedId=24917003#comment-24917003 Fix this by always checking if the unit is currently enabled, and if so, running `systemctl disable` *before* overwriting its contents. The unit will then be re-enabled (or not) based on the MachineConfig. Fixes: https://issues.redhat.com/browse/OCPBUGS-33694
When overwriting a systemd unit with new content, we need to account for the case where the new unit content has a different `[Install]` section. If it does, then simply overwriting will leak the previous enablement symlinks and become node state. That's OK most of the time, but this can cause real issues as we've seen with the combination of openshift#3967 which does exactly that (changing `[Install]` sections) and openshift#4213 which assumed that those symlinks were cleaned up. More details on that cocktail in: https://issues.redhat.com/browse/OCPBUGS-33694?focusedId=24917003#comment-24917003 Fix this by always checking if the unit is currently enabled, and if so, running `systemctl disable` *before* overwriting its contents. The unit will then be re-enabled (or not) based on the MachineConfig. Fixes: https://issues.redhat.com/browse/OCPBUGS-33694
When overwriting a systemd unit with new content, we need to account for the case where the new unit content has a different `[Install]` section. If it does, then simply overwriting will leak the previous enablement symlinks and become node state. That's OK most of the time, but this can cause real issues as we've seen with the combination of openshift#3967 which does exactly that (changing `[Install]` sections) and openshift#4213 which assumed that those symlinks were cleaned up. More details on that cocktail in: https://issues.redhat.com/browse/OCPBUGS-33694?focusedId=24917003#comment-24917003 Fix this by always checking if the unit is currently enabled, and if so, running `systemctl disable` *before* overwriting its contents. The unit will then be re-enabled (or not) based on the MachineConfig. Fixes: https://issues.redhat.com/browse/OCPBUGS-33694
When overwriting a systemd unit with new content, we need to account for the case where the new unit content has a different `[Install]` section. If it does, then simply overwriting will leak the previous enablement symlinks and become node state. That's OK most of the time, but this can cause real issues as we've seen with the combination of openshift#3967 which does exactly that (changing `[Install]` sections) and openshift#4213 which assumed that those symlinks were cleaned up. More details on that cocktail in: https://issues.redhat.com/browse/OCPBUGS-33694?focusedId=24917003#comment-24917003 Fix this by always checking if the unit is currently enabled, and if so, running `systemctl disable` *before* overwriting its contents. The unit will then be re-enabled (or not) based on the MachineConfig. Fixes: https://issues.redhat.com/browse/OCPBUGS-33694
When overwriting a systemd unit with new content, we need to account for the case where the new unit content has a different `[Install]` section. If it does, then simply overwriting will leak the previous enablement symlinks and become node state. That's OK most of the time, but this can cause real issues as we've seen with the combination of openshift#3967 which does exactly that (changing `[Install]` sections) and openshift#4213 which assumed that those symlinks were cleaned up. More details on that cocktail in: https://issues.redhat.com/browse/OCPBUGS-33694?focusedId=24917003#comment-24917003 Fix this by always checking if the unit is currently enabled, and if so, running `systemctl disable` *before* overwriting its contents. The unit will then be re-enabled (or not) based on the MachineConfig. Fixes: https://issues.redhat.com/browse/OCPBUGS-33694
Original description: daemon/update: disable systemd unit before overwriting When overwriting a systemd unit with new content, we need to account for the case where the new unit content has a different `[Install]` section. If it does, then simply overwriting will leak the previous enablement symlinks and become node state. That's OK most of the time, but this can cause real issues as we've seen with the combination of openshift#3967 which does exactly that (changing `[Install]` sections) and openshift#4213 which assumed that those symlinks were cleaned up. More details on that cocktail in: https://issues.redhat.com/browse/OCPBUGS-33694?focusedId=24917003#comment-24917003 Fix this by always checking if the unit is currently enabled, and if so, running `systemctl disable` *before* overwriting its contents. The unit will then be re-enabled (or not) based on the MachineConfig. Fixes: https://issues.redhat.com/browse/OCPBUGS-33694
Kdump will generate it's initramfs based on the running state when kdump.service run If OVS has already run, the kdump fails to gather a working network config, which prevent network log exports, sush as SSH.
See https://issues.redhat.com/browse/OCPBUGS-28239
Ideally we would need that to be backported to 4.14, otherwise I will add a systemd drop-in as a workaround in RHCOS