Homework 3: CI CD, Kubernetes, and other components
- https://www.xsede.org/documents/10165/2374841/kubernetes.pdf/3e1f0d0e-9bb4-422b-a0de-d72770931d28
- Refer to the document here to deploy a kubernetes cluster and connect to it
Access our Jenkins here. Contact any of the team members for credentials.
-
Listed are the branch links :
-
Created Jenkinsfile for each microservice with respective configuration and stages :
-
As soon as the developer makes changes, the hook will trigger the Jenkins pipeline and the respective stages will start. Firstly, for some services, local build will take place, once the dependencies are correctly installed, and the build is successful, for some services, unit testing will take place, new docker image will be built and published to docker hub. The pipeline will then checkout with the Ansible Deployment GitHub branch and deployed to kubernetes clusters.
-
All Deployment yml files in the branch Ansible-K8s-deployments
-
Created playbook yml files for each service
-
Forecast : Cleanup - > checkout to another branch - > locally build - > Build docker image - > Publish to DockerHub - > Remove unused Docker image - > Checkout ansible deployments branch - > Deploy to Jetstreams
-
Gateway : Cleanup - > checkout to another branch - > locally build - > Run test cases - > Build docker image - > Publish to DockerHub - > Remove unused Docker image - > Checkout ansible deployments branch - > Deploy to Jetstreams
-
Ingestor : Cleanup - > checkout to another branch - > locally build - > Run test cases - > Build docker image - > Publish to DockerHub - > Remove unused Docker image - > Checkout ansible deployments branch - > Deploy to Jetstreams
-
Registry : Cleanup - > checkout to another branch - > Build docker image - > Publish to DockerHub - > Remove unused Docker image - > Checkout ansible deployments branch - > Deploy to Jetstreams
-
UI : Cleanup - > checkout to another branch - > Build docker image - > Publish to DockerHub - > Remove unused Docker image - > Checkout ansible deployments branch - > Deploy to Jetstreams
Access our Jenkins here. Contact any of the team members for credentials.
sudo apt update
sudo apt install openjdk-11-jdk
wget -q -O - https://pkg.jenkins.io/debian/jenkins.io.key | sudo apt-key add -
sudo sh -c 'echo deb http://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'
sudo apt update
sudo apt install jenkins
- Check Jenkins status
systemctl status jenkins
- Open port
sudo ufw allow 8080
- The user jenkins needs to be added to the group docker to successfully run docker commands:
sudo usermod -a -G docker jenkins
- Check if user is added to the group
grep docker /etc/group
- Restart Jenkins
sudo systemctl restart jenkins
- git.
- pipeline.
- CloudBees Docker Build and Publish.
- GitHub
- Error
Error fetching remote repo 'origin'
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --force --progress -- https://github.com/airavata-courses/CloudElves.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
- Solution
git config --global --unset credential.helper
git config --system --unset credential.helper
- NodeJS 14 or above:
- https://computingforgeeks.com/install-node-js-14-on-ubuntu-debian-linux/
sudo apt update
curl -sL https://deb.nodesource.com/setup_14.x | sudo bash -
cat /etc/apt/sources.list.d/nodesource.list
sudo apt -y install nodejs
node -v
-
Npm:
sudo apt install npm -
Python: already present on jetstream
- Use Jenkins Kubernetes
- Use Ansible playbook
- Use Helm charts
- Handle shell scripts manually
-
2,3,4 requires ssh ing into the K8s master
-
Option selected: Ansible
-
Ansible is used to deploy our application to Kubernetes as this is an agentless mechanism and doesn't depend upon configuration files (kubeconfig) or other dependencies.
- https://faun.pub/ci-cd-pipeline-using-jenkins-to-deploy-on-kubernetes-cf2fd5e185b8
- https://www.magalix.com/blog/create-a-ci/cd-pipeline-with-kubernetes-and-jenkins
- https://www.jenkins.io/doc/book/pipeline/docker/
sudo pip3 install ansible
- Verify installation (in new shell)
ansible --version
ansible-galaxy install geerlingguy.docker
ansible-galaxy install geerlingguy.jenkins
sudo pip install openshift pyyaml kubernetes
- Generate Public Private key on Jenkins Machine
ssh-keygen -t rsa
- Copy public key to Kubernetes Master VM Authorized Keys
scp ~/.ssh/id_rsa.pub username@host_ip
mkdir -p ~/.ssh
touch ~/.ssh/authorized_keys
cat ~/id_rsa.pub >> ~/.ssh/authorized_keys
- SSH into machine
ssh username@host
ssh -i ~/.ssh/old_keys/host2_key username@host
Ex - ssh -i elves_key [email protected]
- Path: /etc/ansible-host-inventory
[kubernetesMaster]
149.165.157.38 ansible_ssh_user=<user_name> ansible_python_interpreter="/usr/bin/python3"
- hosts: kubernetesMaster
gather_facts: false
vars:
version: "{{version}}"
ansible_ssh_private_key_file: "/home/exouser/.ssh/elves_key"
tasks:
- name: Update deploy
k8s:
state: present
validate_certs: no
definition: "{{ lookup('file', 'forecast.yml') | replace ('TAG',version) }}"
namespace: default
ansible-playbook -i /etc/ansible-host-inventory playbook.yml --extra-vars "version=3"
- To use a non-default private key
ansible-playbook -i /etc/ansible-host-inventory playbook.yml --extra-vars "version=3" --private-key=/home/exouser/.ssh/elves_key
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ModuleNotFoundError: No module named 'kubernetes'
- Use this to resolve:
- https://stackoverflow.com/questions/60866755/ansible-k8s-module-failed-to-import-the-required-python-library-openshift-on
sudo pip install openshift pyyaml kubernetes
or
sudo pip3 install openshift pyyaml kubernetes --user
fatal: [149.165.154.195]: UNREACHABLE! =>
{"changed": false, "msg": "Failed to connect to the host via ssh: Host key verification failed.", "unreachable": true}
- Add key to known hosts in Jenkins workspace at /var/lib/jenkins/.ssh/known_hosts
- SSH from VM/machine first. Copy key from .ssh/known_hosts to /var/lib/jenkins/.ssh/known_hosts
Alternate Solution
- Login as Jenkins user and ssh directly to the host
sudo su -s /bin/bash jenkins
- https://www.sergiosanchez.com/posts/deploy-eks/
- https://stackoverflow.com/questions/61489733/jenkins-pipeline-refering-to-ssh-keys-in-ansible-and-terraform
- https://serverfault.com/questions/1038540/how-to-run-jenkins-build-in-multiple-servers-with-ssh-key
- https://unix.stackexchange.com/questions/321364/what-is-exactly-is-going-on-with-my-jenkins-user
- https://blog.hsoi.com/2012/07/19/setting-up-jenkins-things-ive-learned-so-far/
We are using [Zenko Cloudserver] (https://www.zenko.io/cloudserver/), an S3 compatible object store service.
To run with a file backend, run the following command:
mkdir -m 700 $(pwd)/dataPath
mkdir -m 700 $(pwd)/metadataPath
docker run -d --name s3server -e S3DATAPATH="$(pwd)/dataPath" -e S3METADATAPATH="$(pwd)/metadataPath" -p 8000:8000 scality/cloudserver
To run with an in-memory backend, run the following command:
docker run -d --name s3server -p 8000:8000 scality/cloudserver-latest
Install AWS CLI from here
add the following into ~/.aws/creditials file
[default]
aws_access_key_id = accessKey1
aws_secret_access_key = verySecretKey1
region = us-east-1
After this, you can use the AWS CLI to access the local deployment of s3 using --endpoint=http://localhost:8000 option as follows:
aws s3 ls --endpoint-url http://localhost:8000
Use this script to deploy on kubernetes