CloudEvents for container image registry updates?

[imjasonh]

I often find myself wanting a standard API for receiving notifications of new container images in an image registry. Docker Hub and Quay have their own webhook APIs, GCR has Pub/Sub, probably every registry has their own unique snowflake -- it doesn't make it very easy to build something generic, so things tend to fallback to polling 😢.

It'd be great if there was a standard CloudEvents update mechanism, which defined a standard message shape and let registries produce messages of those types however they want (HTTP, Pub/Sub, etc.), with standard client libraries to consume the stream however it reaches you.

Is specifying such a thing of interest to SIG Events? If so, we could try to spec something out and add it as a non-required surface of the OCI distribution spec, which covers the standard container image registry API (push, pull, tag, etc.). If registries adopted it, clients could consume the standard events from many registries without having to build a bunch of one-off notification implementations.

cc @afrittoli @jonjohnsonjr

[erkist]

I think this is very much in/line with what we are aiming to do, and I think the ArtifactPackaged and ArtifactPublished events should make sure to cover container images as well.

[m-linner-ericsson]

Agree we don't have an event specifically for containers in mind but for all type of deliverable artifacts.

[m-linner-ericsson]

When discussing this we should remember:

Known repository providers

• Docker Hub
• Quay
• AWS
• Google Cloud (GCR)
• Azure (Microsoft)
• Open Shift
• Artifactory (Jfrog)
• Nexus
• Maven repositories
• PYPI (Python)
• NPM (JavaScript)
• Harbor
• GEM (Ruby)
• NUGET (C#)
  For more see: https://goharbor.io/docs/2.3.0/install-config/harbor-compatibility-list/

Types

• Golang
• Linux repository packages
• Helm Charts

[m-linner-ericsson]

We also need to think of Security credentials for public versus private repositories

• When publishing something to a private repository who should get notified.
• How can the event lister listen to a private repository?
• How do handle a mix of public and private repositories?

[imjasonh]

These are great questions, which should definitely be considered.

One idea would be to have a "watch" endpoint for each repository, that gives events just for that repository. Advantages of this would be that authorization could be enforced at that level, and events could stop being sent when the subscriber loses access. A disadvantage would be that some clients would end up creating lots and lots and lots of subscriptions, both in order to be able to watch many repos, but also potentially to be able to watch multiple repos, as multiple users (or on behalf of many users), since they would want to retain their subscription when some other user loses access.

There's definitely a lot to consider and design, but I still think a generalized protocol for subscribing to updates that many registry implementations could have input on, and eventually implement, could be really great for the ecosystem.

[salaboy]

Does someone want to take the lead in creating some of those event emitters by using the Go SDK?

[m-linner-ericsson]

I guess the question has two sides

1. How should the protocol look like that the user subscribes to or gets notified with? Answers questions such as: Is it JSON or XML? What data does the trigger contain?
2. What is the transport of the protocol? Answers question such as: How do I listen for the trigger? Do I need to register myself?

For starters, we could define how the protocol would look like (what @salaboy was referring to above looking at https://github.com/cdfoundation/sig-events/blob/main/vocabulary-draft/continuous-integration-pipeline-events.md)