Bare-metal CSI Operator

Kubernetes Operator to deploy and manage lifecycle of Bare-Metal CSI Driver

Supported environments

OpenShift: 4.6
RKE2r1: 1.21-1.23
Node OS:
- Ubuntu 18.04 / 20.04 LTS
- Red Hat Enterprise Linux 7.7 / 8.x
- Red Hat Enterprise Linux CoreOS 4.6
- CentOS Linux 7 / 8
Helm: 3.0

Installation process

Prerequisites
- lvm2 packet installed on the Kubernetes nodes
- helm v3+
- kubectl v1.16+

Add helm repository

helm repo add csi https://dell.github.io/csi-baremetal-operator
helm repo update
helm search repo csi -l

Setup environment variables

export REGISTRY=docker.io/objectscale
export DOCKER_REGISTRY_SECRET=dockerhub-pull-secret

Create docker registry secret

kubectl create secret docker-registry $DOCKER_REGISTRY_SECRET --docker-username=<USER NAME> \
--docker-password=<PASSWORD> --docker-email=<EMAIL>

Install CSI Operator

helm install csi-baremetal-operator csi/csi-baremetal-operator --set global.registry=$REGISTRY \
--set global.registrySecret=$DOCKER_REGISTRY_SECRET

Install CSI

Vanilla Kubernetes

helm install csi-baremetal csi/csi-baremetal-deployment --set scheduler.patcher.enable=true \
--set driver.drivemgr.type=halmgr --set global.registry=$REGISTRY --set global.registrySecret=$DOCKER_REGISTRY_SECRET

RKE2

helm install csi-baremetal csi/csi-baremetal-deployment --set scheduler.patcher.enable=true \
--set driver.drivemgr.type=halmgr --set platform=rke --set global.registry=$REGISTRY \
--set global.registrySecret=$DOCKER_REGISTRY_SECRET

OpenShift

helm install csi-baremetal csi/csi-baremetal-deployment --set scheduler.patcher.enable=true \
--set driver.drivemgr.type=halmgr --set platform=openshift --set global.registry=$REGISTRY \
--set global.registrySecret=$DOCKER_REGISTRY_SECRET

Note: In case of CSI deployment at Openshift platform in non default namespace, CSI ServiceAccounts
csi-node-sa and csi-baremetal-extender-sa will require Privileged SCC. In case there are no such permissions
CSI Operator will not deploy node and scheduler extender daemonsets. You can see the following events in such case:

65m         Warning   SecurityContextConstraintsVerificationFailed       deployment/csi-baremetal                              ServiceAccount csi-node-sa has insufficient securityContextConstraints, should have privileged
65m         Warning   SecurityContextConstraintsVerificationFailed       deployment/csi-baremetal                              ServiceAccount csi-baremetal-extender-sa has insufficient securityContextConstraints, should have privileged

To make them deployable - create the following role and rolebinding in deployed namespace:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: $ROLENAME
  namespace: $NAMESPACE
rules:
- apiGroups:
  - security.openshift.io
  resourceNames:
  - privileged
  resources:
  - securitycontextconstraints
  verbs:
  - use

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: role-binding
  namespace: $NAMESPACE
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: $ROLENAME
subjects:
- kind: ServiceAccount
  name: csi-baremetal-extender-sa
  namespace: $NAMESPACE
- kind: ServiceAccount
  name: csi-node-sa
  namespace: $NAMESPACE

After created the below roles and rolebindings CSI Operator will automatically catch the changes and deploy node and scheduler extender daemonsets.

Kind (for testing purposes only)

helm install csi-baremetal csi/csi-baremetal-deployment --set scheduler.patcher.enable=true \
--set driver.drivemgr.type=loopbackmgr --set driver.drivemgr.deployConfig=true --set global.registry=$REGISTRY \
--set global.registrySecret=$DOCKER_REGISTRY_SECRET

K3S

helm install csi-baremetal csi/csi-baremetal-deployment --set driver.drivemgr.type=halmgr \
--set global.registry=$REGISTRY --set global.registrySecret=$DOCKER_REGISTRY_SECRET

Note: Currently supports only manual patching - refer documentation

Not supported platform or system with third party Kubernetes scheduler extender - refer documentation for manual patching of Kubernetes scheduler configuration

helm install csi-baremetal csi/csi-baremetal-deployment --set driver.drivemgr.type=halmgr \
--set global.registry=$REGISTRY --set global.registrySecret=$DOCKER_REGISTRY_SECRET

Feature Supporting

CIS hardening

For supporting CIS Hardening we need to add the following settings while installing CSI and CSI Baremetal Operator:

While installing CSI Baremetal Operator set SecurityContext:

--set securityContext.enable=true --set securityContext.runAsNonRoot=true --set securityContext.runAsUser=1000

While installing CSI:

Set the following SecurityContexts at deployments.csi-baremetal.dell.com for Controller, NodeController components:

--set driver.controller.securityContext.enable=true --set driver.controller.securityContext.runAsNonRoot=true --set driver.controller.securityContext.runAsUser=1000 \
--set driver.securityContext.securityContext.enable=true --set driver.securityContext.securityContext.runAsNonRoot=true --set driver.securityContext.securityContext.runAsUser=1000

Create (if not existed) privileged PodSecurityPolicy:

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: privileged
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*'
spec:
  privileged: true
  allowPrivilegeEscalation: true
  allowedCapabilities:
  - '*'
  volumes:
  - '*'
  hostNetwork: true
  hostPorts:
  - min: 0
    max: 65535
  hostIPC: true
  hostPID: true
  runAsUser:
    rule: 'RunAsAny'
  seLinux:
    rule: 'RunAsAny'
  supplementalGroups:
    rule: 'RunAsAny'
  fsGroup:
    rule: 'RunAsAny'

Create Role for this policy usage:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: manager-role
  namespace: test-namespace
rules:
- apiGroups:
  - policy
    resourceNames:
  - privileged
    resources:
  - podsecuritypolicies
    verbs:
  - use

Create RoleBinding for this policy usage:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: manager-rolebinding
  namespace: test-namespace
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: manager-role
subjects:
- kind: ServiceAccount
  name: csi-node-sa
  namespace: test-namespace
- kind: ServiceAccount
  name: csi-baremetal-extender-sa
  namespace: test-namespace

Set the following SecurityContexts at deployments.csi-baremetal.dell.com for SchedulerExtender, SchedulerExtenderPatcher components:
```
--set scheduler.securityContext.enable=true --set scheduler.securityContext.privileged=true
```

Enable podSecurityPolicy at deployments.csi-baremetal.dell.com for Controller, Node, SchedulerExtender, SchedulerExtenderPatcher components:

--set driver.controller.securityContext.enable=true --set driver.controller.securityContext.runAsNonRoot=true --set driver.controller.securityContext.runAsUser=1000 \
--set driver.node.podSecurityPolicy.enable=true --set driver.node.podSecurityPolicy.resourceName=privileged \
--set scheduler.podSecurityPolicy.enable=true --set scheduler.podSecurityPolicy.resourceName=privileged

Usage

Storage classes
- Use storage class without lvg postfix if you need to provision PV bypassing LVM. Size of the resulting PV will be equal to the size of underlying physical drive.
- Use storage class with lvg postfix if you need to provision PVC based on the logical volume. Size of the resulting PV will be equal to the size of PVC.
To obtain information about:
- Node IDs assigned by CSI - kubectl get nodes.csi-baremetal.dell.com
- Local Drives discovered by CSI - kubectl get drives.csi-baremetal.dell.com
- Capacity available for allocation - kubectl get availablecapacities.csi-baremetal.dell.com
- Provisioned logical volume groups - kubectl get logicalvolumegroups.csi-baremetal.dell.com
- Provisioned volumes - kubectl get volumes.csi-baremetal.dell.com
- Storage groups - kubectl get sgs

Upgrade process

To upgrade please reference Installation process section but replace helm install by helm upgrade command.

See helm upgrade for command documentation.

Uninstallation process

Delete all Bare-metal-CSI-managed PVCs

pvcs=($(kubectl get pvc -o wide -A | grep csi-baremetal-sc | awk {'print $1" "$2'}))
for (( i=0; i<${#pvcs[@]} ; i+=2 ))
do
  kubectl delete pvc ${pvcs[i+1]} -n ${pvcs[i]}
done

Ensure there is no Bare-metal-CSI-managed PVC left

kubectl get pvc -o wide -A | grep csi-baremetal-sc

Delete custom resources

kubectl delete volumes --all -A
kubectl delete lvgs --all
kubectl delete sgs --all
kubectl delete csibmnodes --all

Delete helm releases

helm delete csi-baremetal
helm delete csi-baremetal-operator

Delete custom resource definitions

kubectl delete crd deployments.csi-baremetal.dell.com availablecapacities.csi-baremetal.dell.com \
availablecapacityreservations.csi-baremetal.dell.com logicalvolumegroups.csi-baremetal.dell.com \
volumes.csi-baremetal.dell.com drives.csi-baremetal.dell.com nodes.csi-baremetal.dell.com \
storagegroups.csi-baremetal.dell.com

Name		Name	Last commit message	Last commit date
Latest commit History 123 Commits
.github		.github
api/v1		api/v1
charts		charts
config		config
controllers		controllers
deploy		deploy
docs		docs
hack		hack
hook		hook
pkg		pkg
.gitignore		.gitignore
.golangci.yml		.golangci.yml
Dockerfile		Dockerfile
LICENSE		LICENSE
Makefile		Makefile
PROJECT		PROJECT
go.mod		go.mod
go.sum		go.sum
main.go		main.go
variables.mk		variables.mk

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Bare-metal CSI Operator

Supported environments

Installation process

Feature Supporting

CIS hardening

Usage

Upgrade process

Uninstallation process

About

Releases

Packages

Languages

License

yanxiucai/csi-baremetal-operator

Folders and files

Latest commit

History

Repository files navigation

Bare-metal CSI Operator

Supported environments

Installation process

Feature Supporting

CIS hardening

Usage

Upgrade process

Uninstallation process

About

Resources

License

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages