Skip to content

Sunbird Build and Deployment Pipeline Overview

Jump to bottom
Deepak N edited this page Oct 16, 2017 · 3 revisions

Purpose

This document explains the Sunbird Deployment Pipeline, the tools and standards used to create the pipeline. The standards make is simple to discover the existing pipelines and also to setup pipelines for new services.

Tools used

Following open-source tools/frameworks have been used

  • Jenkins
  • Ansible
  • Docker Hub (Image Registry)

Please try these guided tours to learn about Jenkins.

Jenkins Setup Architecture

pipeline-architecture Link

  • Sunbird has two deployment environments, dev and staging.

  • Single Jenkins Master is used for both the environments.

  • Each environment has Jenkins agent service (jenkins agent which runs as a docker container) which is used by the Master to run jobs.

  • Depending on the number of jobs being executed Jenkins agent service could be either scaled up or down.

  • The default Jenkins agent is based on alpine distribution and the jobs are expected to install any dependencies as part of the job run.

  • There is also a Jenkins Build Agent which runs natively (not a docker container) on a VM in dev environment. This could be used for running jobs which cannot be run on Jenkins agent service (docker container).

    Note: As a best practice we recommend that jobs be run on the container agent, so that scaling is easy, if required. Running the job on a VM agent is discouraged and should be treated as a short term solution only.

  • Depending on the job, Jenkins agents will connect to swarm manager, other VMs (e.g. DBs like ES, Cassandra, etc) or external services like Docker Hub.

Sunbird Image Tags and Quality

Sunbird component images are versioned (e.g. 0.0.1) and each version is further tagged with a quality label.

  1. bronze: This certifies that the image is usable, but only on development testing environments. CI pipeline automatically tags image as bronze if the code compiles and all automated tests pass.
  2. silver: This certifies that the image is stable and has been tested to ensure that core functionalities are working fine. An image is tagged as silver after it has been tested on an integration environment and is found to be of good quality.
  3. gold: This certifies that the image is production-ready and could be used by deployments of sunbird. An image is tagged as gold after it has been extensively tested on at least 2 environments and is found to be of production grade quality.

Build Pipeline

build-pipeline Link Sunbird services have the following stages in build pipeline:

  1. Build: This stage compiles, runs unit and integration tests. It produces a docker image as an artifact which all the steps pass.
  2. Push to Image Registry: This stage pushes the image that was built in previous stage to Docker Hub and tags the image as bronze. It also publishes jenkins artefact which contains image metadata (name, version, etc), as metadata.json.
  3. Deploy to DEV: This stage is manually triggered and it deploys the service to DEV environment. It deploys the latest bronze image. It reads the metadata.json from previous stage to get the version of the image that needs to be deployed. It also publishes metadata.json for the next stage to consume.
  4. Tag as Silver: This stage is manually triggered and it stamps the image that is deployed on DEV as of silver quality (uses metadata.json from previous stage to determine the image version). Please see the guidelines above before stamping the image as silver. It also publishes metadata.json for the next stage to consume.
  5. Deploy to STAGING: This stage is manually triggered and it deploys the service to STAGING environment. It reads the metadata.json from previous stage to get the version of the image that needs to be deployed. It also publishes metadata.json for the next stage to consume.
  6. Tag as Gold: This stage is manually triggered and it stamps the image that is deployed on STAGING as of gold quality (uses metadata.json from previous stage to determine the image version). Please see the guidelines above before stamping the image as gold.

Footnotes

[1] For cases where installing dependencies as part of the job run is not possible, a custom image could be built with the required dependencies and this could be used to run a Jenkins agent service where the jobs run. More details could be found in this wiki.

Clone this wiki locally