The watcher Go application uses the Kubernetes client API to monitor jobs in the current namespace and send notifications for major lifecycle events.

See the README for additional details.

Original PR description follows:

Jobs need to be annotated in order to be eligible for notifications:

Use x-jobby.io/notify-channel: [slack|webhook] to determine the notification channel (HTTP webhook or Slack), and then control the destinations with x-jobby.io/slack-channel-ids (comma-separated) or x-jobby.io/webhook-urls (comma-separated) annotations.

If using the Slack notification channel, pass the Slack API token through the WATCHER_SLACK_API_TOKEN env variable.

Notification Channels

The code uses the notify package, which supports are variety of destinations.
So far, the following have been implemented:

• webhook: HTTP (POST) webhooks to a URL
• slack: Slack message to one or multiple channels, requires a Slack API token

Notification Types

The following notifications have been implemented (so far, only Kubernetes/Kueue Jobs are supported):

• Job goes from being suspended to running: implicated that the workload has been admitted by Kueue
  
• Job is completed: logs for all pods associated with the job are included in the message
  
• Job has any failed pods: logs the pods and their failed containers with additional information
  
• Workload preemption monitoring

Usage Example

The following job options set up Slack notifications for lifecycle events in a job:

@job(
    options=JobOptions(
        ...,
        labels={
           "x-jobby.io/notify-channel": "slack",
           "x-jobby.io/slack-channel-ids": "mlops-test",
        ),
    )
)
def my_job(): ...

Deployment

See watcher/README.md for details.