day1

Day 1 Operations

Day 1 Operations are actions that users take to bootstrap a GitOps configuration.

Bootstrapping GitOps can be done with this command:

• kam bootstrap

The kam bootstrap command generates a functional GitOps setup including your first application.

This document describes how to bootstrap GitOps to deliver your first application.

You need to have the following installed in the OCP 4.x cluster.

• OpenShift GitOps Operator
• OpenShift Pipelines Operator

And, you will need these:

• An application source repository (taxi is used as an example in this document)
• The external image repository secret to authenticate image pushes on successful pipeline execution. To use quay.io, please follow prerequisites/quay.md
• The official kam binary downloaded
• A GitHub or GitLab access token (here are the steps to create the git access token for GitHub or GitLab)
• An SSH key connected to your GitHub or GitLab account (here are the steps to create an SSH key for GitHub or GitLab)

Bootstrapping the Manifest

$ kam bootstrap \
  --service-repo-url https://github.com/<your organization>/taxi.git \
  --gitops-repo-url https://github.com/<your organization>/gitops.git \
  --image-repo quay.io/<username>/<image-repo> \
  --dockercfgjson ~/Downloads/<username>-robot-auth.json \
  --git-host-access-token <your git access token> \
  --output <path to write GitOps resources> \
  --push-to-git=true

NOTE: Flag --push-to-git=true push the generated resources to your GitOps repository, this will execute git locally on the developer machine, which will in turn authenticate the push using your local SSH keys, this means that you need to be able to push to a Git repository from your local machine.

The kam bootstrap command also provides an interactive mode, which is triggered by running without any parameters, or by providing the --interactive flag, and will generate the GitOps directory and the required resources.

During an interactive mode session, choose to use default values or not. If default values are chosen, prompts will appear to allow you to enter any required values that haven't already been provided from the command line. This is the quickest way to generate a bootstrapped GitOps configuration.

In the event of using a self-hosted GitHub Enterprise or GitLab Community/Enterprise Edition if the driver name isn't evident from the repository URL, use the --private-repo-driver flag to select github or gitlab.

For more details see the Argo CD documentation.

The bootstrap process generates a fairly large number of files, including a pipelines.yaml describing your first application, and configuration for a complete CI pipeline and deployments from Argo CD.

A pipelines.yaml file (example below) is generated by the kam bootstrap command. This file is used by Day 2 commands such as kam service add to generate/update pipelines resources.

The bootstrap process creates two environments: dev and stage

Namespaces are generated for both of these environments.

The name of the app and service is derived from the last component of your service-repo-url e.g. if you bootstrap with --service-repo-url https://github.com/myorg/myproject.git this would bootstrap an app called app-myproject and a service called myproject.

Finally, the GitOps repository is created automatically if credentials are provided, this will create a private repository for pushing your generated resources, and the resources will be pushed to your git hosting service.

Secrets

By default, kam generates un-encrypted secrets into the secrets folder which is a sibling to your GitOps folder. Managing these un-encrypted secrets in git is insecure and is not recommended. kam only expects these secrets to be present in the cluster and is agnostic about the tool used to manage them.

Access Tokens

• The token is stored securely on the local filesystem using keyring. The keyring requires a username and service name to store the secret, the KAM tool stores the secret with the service name Kam and the username being the host name of the pertaining URL (e.g. --gitops-repo-url).

• If the token is set within the keyring, the keyring will not be prompted for in sucessive attempts of the Bootstrap and Webhook commands. The token can however be updated at the time of bootstrap in the keyring by passing the --save-token-keyring flag along with --git-host-access-token flag in the non-interactive mode of the bootstrap command.

• When a token is not provided in the command flag, an attempt is made to retrieve the token from the keyring. If unsuccessful, the cmd will look for the access token in an environment variable whose variable naming convention is as follows: The hostname (e.g. github.com) is extracted from the value passed to the repository URL (e.g. https://github.com/username/repo.git), where the . in the hostname is replaced by _ and concatenated with _TOKEN. In this case, the environment varaible name will be GITHUB_COM_TOKEN.

• In the event a token is not passed in the command, if the token is not found in the keyring or the environment variable with the specified name, the command will fail.

Private Repository

In case a private repository is used, register the repository with Argo CD either from the UI or CLI. Details on how to configure repositories in Argo CD can be found here.

Prefixing namespaces

By default, bootstrapping creates cicd, dev, and stage namespaces, these can be optionally prefixed by passing --prefix to the kam bootstrap command.

This is useful if you're working in a shared cluster, for example, with --prefix tst, the command will generate 3 namespaces called: tst-cicd, tst-dev and tst-stage.

Environment configuration

The dev environment is a very basic deployment

configuring pipelines

config:
  argocd:
    namespace: openshift-gitops
  pipelines:
    name: cicd
environments:
- apps:
  - name: app-taxi
    services:
    - name: taxi
      pipelines:
        integration:
          bindings:
          - dev-app-taxi-taxi-binding
          - gitlab-push-binding
      source_url: https://github.com/<your organization>/taxi.git
      webhook:
        secret:
          name: webhook-secret-dev-taxi
          namespace: cicd
  name: dev
  pipelines:
    integration:
      bindings:
      - gitlab-push-binding
      template: app-ci-template
- name: stage
gitops_url: https://github.com/<your organization>/gitops.git
version: 1

The pipelines key describes how to trigger an OpenShift Pipelines run, the integration binding and template are processed when a Pull Request is opened.

This is the default pipeline specification for the dev environment, you can find the definitions for these in these two files:

• config/cicd/base/06-templates/app-ci-build-from-push-template.yaml
• config/cicd/base/05-bindings/github-push-binding.yaml

By default, this triggers a PipelineRun of this pipeline

• config/cicd/base/04-pipelines/app-ci-pipeline.yaml

These files are not managed directly by the manifest, you're free to change them for your own needs, by default they use Buildah to trigger build, assuming that the Dockerfile for your application is in the root of your repository.

configuring services

- apps:
  - name: app-taxi
    services:
    - name: taxi
      pipelines:
        integration:
          bindings:
          - dev-app-taxi-taxi-binding
          - gitlab-push-binding
      source_url: https://github.com/<your organization>/taxi.git
      webhook:
        secret:
          name: webhook-secret-dev-taxi
          namespace: cicd

The YAML above defines an app called app-taxi, which has a reference to a service called taxi.

The configuration for these is written out to:

• environments/dev/apps/app-taxi/services/taxi/base/config/

The app-taxi app's configuration references the services configuration.

The source_url references the source code repository for the service.

The pipelines field describes the templates and bindings used for this service.

Thewebhook.secret is used to authenticate incoming hooks from Git host.

Argo CD Permissions

For each environment managed by kam, the namespace is labelled with argocd.argoproj.io/managed-by:<argocd-namespace> and the GitOps operator creates the necessary RBAC resources for the Argo CD instance to manage that namespace. In order to provide custom permissions for a namespace, update the role <argocd-instance>-argocd-application-controller in that namespace.

Bringing the bootstrapped environment up

Ignore these steps if the flag --push-to-git=true is part of your bootstrap command.

Create a new gitops un initialize private GitHub/GitLab repo, then from the root of your GitOps directory (with the pipelines.yaml), execute the following commands:

$ git init .
$ git add .
$ git commit -m "Initial commit."
$ git remote add origin <insert gitops repo>
$ git branch -M main
$ git push -u origin main

This will initialize the GitOps repository and is the start of your journey to deploying applications via Git.

Bringing the deployment infrastructure up

We'll bring up our deployment infrastructure, this is only necessary at the start. The configuration will be self-hosted thereafter.

$ oc apply -k config/argocd/

At this point, the apps in Argo CD should be synced and healthy. Apply the generated secrets from the secrets folder.

$ oc apply -f <secrets-folder>

Managing plain Kubernetes secrets in Git is not safe and hence it is recommended to encrypt them into a SealedSecret. Please skip the below section if you opt not to use Sealed Secrets.

Sealed Secrets

To use Bitnami Sealed Secrets, do the following:

1. Install the Helm Sealed Secrets Operator (Helm) in namespace kube-system with controller name of sealed-secrets
2. Ensure the kubectl context is set to the OpenShift cluster
3. Go to the secrets folder which is generated by kam. It is a sibling folder to the output or pipelines folder.
4. Run kubeseal on each secret. For example:

     cat gitops-webhook-secret.yaml | kubeseal \
     --controller-namespace kube-system \
     --controller-name sealed-secrets \
     --format yaml \
     > gitops-webhook-sealedsecret.yaml

5. Apply the sealed secrets, but be careful not to apply the unsealed secrets.

Alternatively instead of the above steps and to avoid installing the operator, you can use key certificates:

1. helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
2. helm install sealed-secrets --namespace kube-system --version 1.13.2 sealed-secrets/sealed-secrets
3. Create the cert

     kubeseal \
      --controller-name=sealed-secrets \
      --controller-namespace=kube-system \
      --fetch-cert > cert.pem

4. To encrypt the secret, run cat docker-config.yaml | kubeseal --cert cert.pem
5. Apply the sealed secrets, but be careful not to apply the unsealed secrets.

You can then check in the sealed secrets into Git. For more information see: https://github.com/bitnami-labs/sealed-secrets and https://engineering.bitnami.com/articles/sealed-secrets.html

Visualize your applications via the Argo CD UI

On installation of OpenShift GitOps, the operator sets up a ready-to-use Argo CD for cluster configuration. You can launch into this Argo CD instance from Console Application Launcher.

For the pre-created Argo CD instance under openshift-gitops project, you’ll find the password to log into the Argo CD UI under the Workloads > Secrets tab and selecting openshift-gitops-cluster. Once on that page, you can copy the secret at the bottom of the page to your clipboard and paste it into the Argo CD login page to use as the password. Use admin as the username.

Alternatively, you can fetch this password via the command line by running:

$ kubectl get secret openshift-gitops-cluster -n openshift-gitops -ojsonpath='{.data.admin\.password}' | base64 -d

You should now be logged in and able to see the Argo CD UI, and deployed applications should be healthy and in-sync.

Changing the initial deployment

The bootstrap creates a Deployment in environments/dev/apps/<app name>/services/<service name>/base/config/100-deployment.yaml. This should bring up nginx, and is purely for demo purposes, you'll need to change this to deploy your built image.

spec:
  containers:
  - image: nginxinc/nginx-unprivileged:latest
    imagePullPolicy: Always
    name: taxi

You'll want to replace this with the image for your application, once you've built and pushed it.

Your first CI run

Part of the configuration bootstraps a simple OpenShift Pipelines pipeline for building code when a pull-request is opened.

You will need to create a new webhook for your Git hosting service, this is used to trigger pipeline runs automatically on pushes to your repositories.

$ kam webhook create \
    --git-host-access-token <git host access token> \
    --env-name dev \
    --service-name taxi

Note: If the webhook creation fails with gitops-webhook-event-listener-route route not being present, login to the Argo CD UI to verify if the apps have been created and synced successfully (instructions on how to access the Argo CD UI is at the bottom of this guide)

Make a change to your application source, the taxi repo from the example, it can be as simple as editing the README.md and propose a change as a Pull Request.

This should trigger the PipelineRun:

Drilling into the PipelineRun we can see that it executed our single task:

And finally, we can see the logs that the build completed and the image was pushed:

Changing the default CI run

Before this next stage, we need to ensure that there's a webhook configured for the GitOps repo.

$ kam webhook create \
    --git-host-access-token <github user access token> \
    --cicd

This step involves changing the CI definition for your application code.

The default CI pipeline we provide is defined in the manifest file:

  pipelines:
    integration:
      bindings:
      - github-push-binding
      template: app-ci-template

This template drives a pipeline that is stored in this file:

• config/cicd/base/04-pipelines/app-ci-pipeline.yaml

An abridged version is shown below, it has a single task build-image, which executes the buildah task, which builds the source and generates an image and pushes it to your image-repo.

apiVersion: tekton.dev/v1alpha1
kind: Pipeline
spec:
  resources:
  - name: source-repo
    type: git
  tasks:
  - name: build-image
      inputs:
      - name: source
        resource: source-repo
    taskRef:
      kind: ClusterTask
      name: buildah

You will likely want to add additional tasks for running the tests for your application code.

Write the following Task to this file:

• config/cicd/base/03-tasks/go-test-task.yaml

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: go-test
  namespace: default
spec:
  resources:
    inputs:
      - name: source
        type: git
  steps:
    - name: go-test
      image: golang:latest
      command: ["go", "test", "./..."]

This is a simple test task for a Go application, it just runs the tests.

Append the newly added task to the existing kustomize file

• config/cicd/base/kustomization.yaml

Update the pipeline in this file:

• config/cicd/base/04-pipelines/app-ci-pipeline.yaml

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  creationTimestamp: null
  name: app-ci-pipeline
  namespace: cicd
spec:
  params:
  - name: REPO
    type: string
  - name: COMMIT_SHA
    type: string
  resources:
  - name: source-repo
    type: git
  - name: runtime-image
    type: image
  tasks:
  - name: go-ci
    resources:
      inputs:
      - name: source
        resource: source-repo
    taskRef:
      kind: Task
      name: go-test
  - name: build-image
    runAfter:
      - go-ci
    params:
    - name: TLSVERIFY
      value: "true"
    resources:
      inputs:
      - name: source
        resource: source-repo
      outputs:
      - name: image
        resource: runtime-image
    taskRef:
      kind: ClusterTask
      name: buildah

Commit and push this code, and open a Pull Request, you should see a PipelineRun being executed.

This validates that the YAML can be applied, by executing oc apply -k config/argocd/ --dry-run=client.