Add Function BackupDataAllUsingKopiaServer

[r4rajat]

Change Overview

Add New Kanister Function BackupDataAllUsingKopiaServer

Pull request type

Please check the type of change your PR introduces:

• 🚧 Work in Progress
• 🌈 Refactoring (no functional changes, no api changes)
• 🐹 Trivial/Minor
• 🐛 Bugfix
• 🌻 Feature
• 🗺️ Documentation
• 🤖 Test

Issues

• fixes #issue-number

Test Plan

• 💪 Manual
• ⚡ Unit test
• 💚 E2E

Manual Testing Steps

1) Create Images for Kanister and Repo Server controller

git tag -fa v21-repo-server-rajat -m "Testing"

bash build/gorelease.sh

docker tag ghcr.io/kanisterio/controller:v21-repo-server-rajat r4rajat/controller:v21-repo-server-rajat

docker tag ghcr.io/kanisterio/repo-server-controller:v21-repo-server-rajat r4rajat/repo-server-controller:v21-repo-server-rajat

docker push r4rajat/controller:v21-repo-server-rajat && docker push r4rajat/repo-server-controller:v21-repo-server-rajat

2) Install Kanister

helm install kanister ./helm/kanister-operator \
--namespace kanister \
--set image.repository=r4rajat/controller \
--set image.tag=v21-repo-server-rajat \
--set repositoryServerImage.repository=r4rajat/repo-server-controller \
--set repositoryServerImage.tag=v21-repo-server-rajat \
--set controller.parallelism=10 \
--create-namespace

3) Apply Repo Server CRD

kubectl apply -f pkg/customresource/repositoryserver.yaml -n kanister

4) Create Test Application [Time Logger]

kubectl create namespace time-logger

kubectl create -f ./examples/time-log/time-logger-deployment.yaml -n time-logger

5) Create OpenSSL Certificate

openssl req -newkey rsa:2048 -nodes -keyout key.pem -x509 -days 365 -out certificate.pem

6) Create S3 Location and Location Secret Config Files

• S3 Location Secret

vi s3_location_creds.yaml

apiVersion: v1
kind: Secret
metadata:
   name: s3-creds
   namespace: kanister
   labels:
      repo.kanister.io/target-namespace: monitoring
type: secrets.kanister.io/aws
data:
   # required: base64 encoded value for key with proper permissions for the bucket
   aws_access_key_id: <base64 encoded access key>
   # required: base64 encoded value for the secret corresponding to the key above
   aws_secret_access_key: <base64 encoded secret key>

• S3 Location

vi s3_location.yaml

apiVersion: v1
kind: Secret
metadata:
   name: s3-location
   namespace: kanister
   labels:
      repo.kanister.io/target-namespace: monitoring
type: Opaque
data:
   # required: specify the type of the store
   # supported values are s3, gcs, azure, and file-store
   type: czM=
   bucket: cmFqYXQtaW5mcmFjbG91ZA==
   # optional: used as a sub path in the bucket for all backups
   path: L3JlcG8tY29udHJvbGxlci8=
   # required, if supported by the provider
   region: dXMtZWFzdC0x
   # optional: if set to true, do not verify SSL cert.
   # Default, when omitted, is false
   #skipSSLVerify: false
   # required: if type is `file-store`
   # optional, otherwise
   #claimName: store-pvc

7) Apply Secrets

kubectl create secret tls repository-server-tls-cert --cert=certificate.pem --key=key.pem -n kanister

kubectl create secret generic repository-server-user-access -n kanister --from-literal=localhost=test1234

kubectl create secret generic repository-admin-user -n kanister --from-literal=username=admin@testpod1 --from-literal=password=test1234

kubectl create secret generic repo-pass -n kanister --from-literal=repo-password=test1234

kubectl apply -f s3_location_creds.yaml -n kanister

kubectl apply -f s3_location.yaml -n kanister

8) Create Repository

kopia --log-level=error --config-file=/tmp/kopia-repository.config --log-dir=/tmp/kopia-cache repository create --no-check-for-updates --cache-directory=/tmp/cache.dir --content-cache-size-mb=0 --metadata-cache-size-mb=500 --override-hostname=mysql.app --override-username=kanisterAdmin s3 --bucket=rajat-infracloud --prefix=/repo-controller/ --region=us-east-1 --access-key=<ACCESS_KEY> --secret-access-key=<SECRET_ACCESS_KEY>

9) Create Repository Server CR

vi repo-server-cr.yaml

apiVersion: cr.kanister.io/v1alpha1
kind: RepositoryServer
metadata:
  labels:
    app.kubernetes.io/name: repositoryserver
    app.kubernetes.io/instance: repositoryserver-sample
    app.kubernetes.io/part-of: kanister
    app.kuberentes.io/managed-by: kustomize
    app.kubernetes.io/created-by: kanister
  name: kopia-repo-server-1
  namespace: kanister
spec:
  storage:
    secretRef:
      name: s3-location
      namespace: kanister
    credentialSecretRef:
      name: s3-creds
      namespace: kanister
  repository:
    rootPath: /repo-controller/
    passwordSecretRef:
      name: repo-pass
      namespace: kanister
    username: kanisterAdmin
    hostname: mysql.app
  server:
    adminSecretRef:
      name: repository-admin-user
      namespace: kanister
    tlsSecretRef:
      name: repository-server-tls-cert
      namespace: kanister
    userAccess:
      userAccessSecretRef:
        name: repository-server-user-access
        namespace: kanister
      username: kanisteruser

kubectl apply -f repo-server-cr.yaml -n kanister

Wait till the status of Repository Server CR gets to ServerReady , You could check it by running following command

kubectl describe -n kanister repositoryserver.cr.kanister.io/kopia-repo-server-1

10) Create Blueprint

vi test-blueprint.yaml

apiVersion: cr.kanister.io/v1alpha1
kind: Blueprint
metadata:
  name: backupdate-bp
  namespace: kanister
actions:
    backup-all:
    outputArtifacts:
      timeLog:
        keyValue:
          path: '/repo-controller/time-logger/'
      backupIdentifier:
        keyValue:
          backupInfo: "{{ .Phases.backupToS3All.Output.BackupAllUsingKopiaServerInfo }}"
    phases:
      - func: BackupDataAllUsingKopiaServer
        name: backupToS3All
        args:
          namespace: "{{ .Deployment.Namespace }}"
          pods: "{{ index .Deployment.Pods 0 }}"
          container: kanister-sidecar
          includePath: /var/log

kubectl create -f test-blueprint.yaml -n kanister

11) Build kanctl with latest changes

go build -o kanctl cmd/kanctl/main.go 

12) Take Backup of the Application

./kanctl create actionset --action backup-all --namespace kanister --blueprint backupdate-bp --deployment time-logger/time-logger --repository-server=kopia-repo-server-1

actionset backup-all-4xr74 created

Check Status of the actionset

kubectl describe actionsets -n kanister backup-all-4xr74

Events:
  Type    Reason           Age   From                 Message
  ----    ------           ----  ----                 -------
  Normal  Started Action   8s    Kanister Controller  Executing action backup-all
  Normal  Started Phase    8s    Kanister Controller  Executing phase backupToS3All
  Normal  Ended Phase      2s    Kanister Controller  Completed phase backupToS3All
  Normal  Update Complete  2s    Kanister Controller  Updated ActionSet 'backup-all-4xr74' Status->completed

[github-actions]

Thanks for submitting this pull request 🎉. The team will review it soon and get back to you.

If you haven't already, please take a moment to review our project contributing guideline and Code of Conduct document.

[viveksinghggits]

Let's also mention the document that, if we have any, on how to create the repo server CR.

[viveksinghggits]

do we specify names of pods separated by comma (,).

[github-actions]

This PR is marked as stale due to inactivity. Add a new comment to reactivate it.

[github-actions]

This PR is marked as stale due to inactivity. Add a new comment to reactivate it.