What encapsulates something runnable in a CD pipeline?

[erkist]

I'm having problems painting a mental picture of what exactly can be run in a CD pipeline as defined in the Continuous Delivery bucket PR #17

The PR states:

A pipeline, in the context of Continuous Delivery, is the definition of a set of tasks that needs to be performed to build, test, package, release and deploy software artifacts. A pipeline can be instanciated multiple times, for example to build different versions of the same artifact. We are refering to this instance as PipelineRun. It will have a unique Id and it will help us to track the build and release progress on a particular software artifact.

This matches quite well with a Jenkins pipeline, in my view, where a pipeline, which is runnable, consists of a set of stages (named Tasks in our spec).

I would call this a "push pipeline", e.g. there is a controlling system that starts an instance of the pipeline and step by step executes everything in it.

There is an alternative way of driving pipelines, which I would call a "pull pipeline" or "event-driven pipeline", where the pipeline consists of a bunch of distributed runnable tasks which (at least conceptually) all wait for "some value to be produced" that should cause them to run. These "values produced" could be anything from an artifact being released, to a test suite failing, to a new environment having been created.

In these scenarios, there isn't really a pipeline running in any system, and it doesn't make sense to talk about a pipeline run being finished, instead the focus is on what value has been produced, and what activities are or have been run as part of the pipeline instance.

Eiffel has support for this event-driven approach through various chaining concepts (e.g. CAUSE and CONTEXT), ESCI explicitly defines a Flow Run Id that is shared between all activities that are run as part of the same flow/pipeline.

I hope we can find a way to support both push and pull pipelines, but right now I can't figure out exactly what minimal parts are needed to support both. Does anyone have any suggestions?

[salaboy]

@erkist yes.. this is a great topic for a discussion, and I think that we should totally support both.. In my head, because we are defining events here you can do both, even if you are not running a pipeline. The event-driven approach only needs a correlation key between all the events to aggregate the data related to the value being produced (artifact, service or whatever the pipelines are doing).

What about TaskRuns? If all TaskRuns share the same correlation ID that can be used for the same purposes (if that correlation ID is the pipelineRun Id or just a correlation Id it should be technically the same). To your point, ESCI Flow Run Id is exactly that.. the correlation key shared between different activities.
Are we talking about the same? or is there something that I am missing from your perspective?

[erkist]

@salaboy I think we are talking about the same thing! If the spec was updated to say that TaskRuns could be queued, and that TaskRuns can have child TaskRuns, I think we could use TaskRun to represent event-driven pipelines.

[afrittoli]

I think I understand what you mean with push and pull pipelines. In my mind I think of "pull pipelines" as async pipelines.
I think the can coexist, and probably most workflows will end up being a combination of the two.

In the example of Jenkins, the Jenkins pipeline plugin would be your pipeline orchestrator. The trigger event might be someone pushing a button, or a cron trigger, or some other kind of event. Even if the orchestrator starts the execution of the pipeline, it does so as a reaction to an event of some kind. Once the pipeline is complete it may generate an event, and that might be the trigger to the next pipeline.

In case of Tekton, tekton is your pipeline orchestrator which takes care of executing through the various parts of the pipeline (tasks - in Tekton terminology). The execution of a pipeline in case of Tekton is initiated by submitting a PipelineRun into etcd. That can be done "manually" using kubectl or as a reaction to an event of some kind (Tekton support HTTP requests with a JSON payload). Once the PipelineRun is in etcd, Tekton will start the execution if the PipelineRun. When a pipeline is completed, it will generate an event, and that may trigger another pipeline.

"Push" pipelines usually have a static definition which allows to visualise and analyse them even before execution.
"Pull" (or async) pipelines are much more challenging in that respect. It may be hard to predict what the execution flow will look like.
My hope is that standard event will help tracing and visualising this kind of workflows (and combinations of the two) at runtime through events that may be generated by pipelines running in different platforms.

[m-linner-ericsson]

When reading https://github.com/cloudevents/spec/blob/v1.0.1/primer.md#cloudevents-concepts and especially the following part:

The goal of the CloudEvents specification is to define interoperability of event systems that allow services to produce or consume events, where the producer and consumer can be developed and deployed independently. A producer can generate events before a consumer is listening, and a consumer can express an interest in an event or class of events that is not yet being produced.

I am failing to paint the picture of a push pipeline that would fit into this picture. For me, a push pipeline needs to know all the steps beforehand and thus the different parts cannot be deployed independently.

Does this make sense and what do you think about this?

[TracyRagan]

This 'goal' definition gets a +1 from me. And yes, I believe events move us away from a push pipeline. But again, if we call it a 'pull' pipeline that may get confused with 'GitOps' even though GitOps really just relates to continuous deployments and IaC.

[erkist]

For me, a push pipeline needs to know all the steps beforehand and thus the different parts cannot be deployed independently.

True, but the push pipeline system could send events that other actors, that are deployed independently, could react to (as Andrea pointed out in a previous post). So I think our protocol should support both.

[m-linner-ericsson]

Maybe I am misunderstanding are we talking about orchestrated pipelines vs event-triggered pipelines?

• An orchestrated pipeline would send out events to tell what is happening but control of the execution is directed by the orchestrator
• An event-driven pipeline would both send out events and be controlled by the event it is listening for

We probably should draw an image instead to explain the different pipelines not to confuse each other.