Config to automatically Re-trigger failed periodics

[jmguzik]

This PR introduces a new configuration field (along with the functionality in horologium component) for ProwJobs to support automatically re-triggering periodic jobs based on a specified policy. The new retrigger-failed-run field allows retrying a failed periodic job with customizable settings for the number of attempts and the interval between retries.

Example usage:

retrigger-failed-run:
  until_success: true
  attempts: 3
  interval: 6h

until_success: true alows to stop if success state is reached. Set to false it triggers until attempts reached.
sloves #268

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: jmguzik
Once this PR has been reviewed and has the lgtm label, please assign cjwagner for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• cmd/horologium/OWNERS
• pkg/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[netlify]

✅ Deploy Preview for k8s-prow ready!

Name	Link
🔨 Latest commit	62d9f69
🔍 Latest deploy log	https://app.netlify.com/sites/k8s-prow/deploys/6793d1a1acbb9d000884fc23
😎 Deploy Preview	https://deploy-preview-358--k8s-prow.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[Prucek]

/lgtm

[petr-muller]

/cc

[petr-muller]

/hold

Please provide a PR description of what the PR actually implements - the link to the issue is helpful but not sufficient, because even the issue contains some discussion about options for a vague idea, not a clear implementation direction.

[jmguzik]

@petr-muller done

[jmguzik]

/hold cancel
then?

[petr-muller]

I'll review tomorrow 🙏

[petr-muller]

One thing we need to be intentional about is the interaction between the next legit trigger and a retry, when both could occur, and also whether it is acceptable for retries to "push back" interval triggers.

As done here, next legit trigger takes precedence, basically interrupting the retry sequences, which may impact especially the stop on success = false case where we expect to deterministically get the configured amount of retries but we wont because the legit-triggerred job interrupts that.

That sounds acceptable to me but needs to be intentional, documented behavior.

[jmguzik]

@petr-muller thanks for the review. I agree with most of your comments but I will clear them when we reach common understanding.

As noted in the comments, every job marked with the new fields (in the configuration) will have associated label. First run will have value 1. This is because I wanted to keep other periodics unaffected and I make distinction on this field weather to process periodic as normal or repeatable.

One thing we need to be intentional about is the interaction between the next legit trigger and a retry, when both could occur, and also whether it is acceptable for retries to "push back" interval triggers.

I think the way it is implemented, normal trigger (in the case of this implementation first run value 1) will always have precedence because as soon as shouldTrigger is detected to be true, the code will not follow new section.

As done here, next legit trigger takes precedence, basically interrupting the retry sequences, which may impact especially the stop on success = false case where we expect to deterministically get the configured amount of retries but we wont because the legit-triggerred job interrupts that.

That's true

That sounds acceptable to me but needs to be intentional, documented behavior.

Makes sense.

[k8s-ci-robot]

New changes are detected. LGTM label has been removed.