Jenkins sometimes fails to decrypt helm secrets (in the roll out stage in the orchestration pipeline

[matzehecht]

Describe the bug
Since we switched to helm our pipeline sometimes fails to decrypt our helm secrets as it can not find the key to decrypt it (added the error log under "log output").

I know that everything is configured as it should, because this happens only now and than. Also I can see in the logs, that it decrypts the secrets successfully in the helm secrets diff call but than fails to do so in the helm secrets upgrade. This is not urgent as rerunning the pipeline solves it. But at least the development team should be aware of it!

To Reproduce
Steps to reproduce the behavior:

1. Configure a ODS project (by adding helm secrets to a project, store the key in an openshift secret and sync it to jenkins).
2. Start an orchestration pipeline and hope you are lucky enough to encounter this bug.

Expected behavior
decrypting the secrets should work all the time.

Affected version (please complete the following information):

• OpenShift 4
• OpenDevStack 4.x

Log Output (ensure to remove any confidential information like tokens, project names, etc.

Failed to get the data key required to decrypt the SOPS file.

Group 0: FAILED
  <PRIVATE_KEY_FINGERPRINT>: FAILED
    - | could not decrypt data key with PGP key:
      | github.com/ProtonMail/go-crypto/openpgp error: Could not
      | load secring: open /home/jenkins/.gnupg/secring.gpg: no such
      | file or directory; GPG binary error: exit status 2

Recovery failed because no master key was able to decrypt the file. In
order for SOPS to recover the file, at least one key has to be successful,
but none were.

[matzehecht]

@serverhorror This is the issue for the bug we previously talked about.

[serverhorror]

rerunning the pipeline solves it

Tough one, sounds like some race condition. I'll try and take a look.

[serverhorror]

@matzehecht, @braisvq1996 sorry for not getting back earlier, did opendevstack/ods-quickstarters#946 fix this issue?

[matzehecht]

@matzehecht, @braisvq1996 sorry for not getting back earlier, did opendevstack/ods-quickstarters#946 fix this issue?

@serverhorror As mentioned in my comment on the mentioned PR those issues are not related. You can find all information in the other comment or the description of the respective issues.
In short: The other issue/PR affected the import of gpg keys. This issue happens while reading the successfully imported gpg key. Also this issue happens only some times and the other issue happened every time.
So no: The other issue/PR does not fix the issue.

[serverhorror]

Can someone, with the technical permissions, label this with help-wanted? I have no idea how to fix this :(