install_upi.md

Install: oVirt/RHV User-Provided Infrastructure

This User-Provisioned Infrastructure (UPI) process is based on several and customizable steps to allow the user to integrate into an existing infrastructure.

Creating and configuring oVirt/RHV resources is the responsibility of the user deploying OpenShift.

The OpenShift installer will still be used in several steps of the process to generate mandatory ignition files and to monitor the installation process itself.

Table of Contents

• Prerequisites
  • Ansible and oVirt roles
  • Network Requirements
    • Load Balancers
    • DNS
  • RHCOS image
• Getting Ansible playbooks
• Assets directory
• Inventory explained
  • General section
  • RHCOS section
  • Profiles and VMs
    • Profiles section
    • VMs section
• Install Config
  • Set compute replicas to zero
  • Set machine network
  • Set platform to none
• Manifests
  • Set control-plane nodes unschedulable
• Ignition configs
• Create templates and VMs
• Bootstrap
• Master nodes
• Wait for Control Plane
• OpenShift API
• Retire Bootstrap
• Worker nodes
  • Approve CSRs
• Wait for Installation Complete
• Destroy OpenShift cluster

Prerequisites

The inventory.yml file all the variables used by this installation can be customized as per user needs. The requirements for the UPI in terms of minimum resources are broadly the same as the IPI.

• oVirt/RHV account stored in the ovirt-config.yaml
  • this file is generated by the openshift-install binary installer following the CLI wizard.
• Name of the oVirt/RHV cluster to use
  • contained in the inventory.yml and input in the openshift-install.
• A base domain name of the OpenShift cluster
  • input in the openshift-install.
• A name of the OpenShift cluster
  • input in the openshift-install.
• OpenShift Pull Secret
  • input in the openshift-install.
• A DNS zone
  • to configure the resolution names for the OpenShift cluster base domain.
• LoadBalancers
  • for bootstrap and control-plane machines.
  • for machines running the ingress router (usually compute nodes).

Ansible and oVirt roles

To use the UPI process described here the following are required:

• Python3

Note:
Currently most of Linux distros provides Python 3 by default.

• Ansible

Note for CentOS users:
Depending on which version the system is running will be required epel-release repo enabled.

  $ sudo dnf install ansible

• python3-ovirt-engine-sdk4

  $ sudo dnf install python3-ovirt-engine-sdk4

• ovirt.image-template Ansible role (distributed as ovirt-ansible-image-template package on oVirt Manager)
• ovirt.vm-infra Ansible role (distributed as ovirt-ansible-vm-infra package on oVirt Manager)

  $ sudo ansible-galaxy install ovirt-ansible-vm-infra ovirt-ansible-image-template

To be sure to follow the UPI installation process, Ansible scripts and the binary openshift-install should be executed from the oVirt/RHV Manager or from a machine with access to the REST API of the oVirt/RHV Manager and with all the oVirt roles available (installed by default on the Manager machine).

Network Requirements

The UPI installation process assumes that the user satisfies some network requirements providing them through the existing infrastructure. During the boot the RHCOS based machines require an IP address in initramfs in order to establish a network connection to get their ignition config files. One of the recommended ways is to use a DHCP server to manage the machines in the long-term, maybe configuring the DHCP server itself to provide persistent IP addresses and host names to the cluster machines.

Network connectivity between machines should be configured to allow cluster components to communicate:

• Kubernetes NodePort Machines require connectivity to every other machine for OpenShift platform components through the port range 30000-32767 .

• OpenShift reserved Connectivity to reserved port ranges 10250-10259 and 9000-9999 should be granted on every machine.

• Machines to control-plane Connectivity to ports on ranges 2379-2380 (for etcd, peer and metrics) is required for control-plane machines and on port 6443 for Kubernetes API.

Load Balancers

Before installing the OpenShift Container Platform, two load balancers (layer-4) must be provided by the user infrastructure, one for the API and one for the Ingress Controller (to allow ingress to applications).

• Load balancer for port 6443 and 22623 on control-plane and bootstrap machines (the bootstrap can be removed after control-plane initialization completes). The 6443 must be both internal and external reachable and is needed by the Kubernetes API server. Port 22623 must be accessible to nodes within the cluster.

• Load balancer for port 443 and 80 for machines running the ingress router (usually worker nodes in the default configuration). Both ports must be accessible from within and outside the cluster.

NOTE: the rules above can also be set on the same load balancer server.

DNS

The UPI installation process requires the user to setup the existing infrastructure provided DNS to allow the correct resolution of the main components and services

• Kubernetes API DNS records api.<cluster_name>.<base_domain> (internal and external resolution) and api-int.<cluster_name>.<base_domain> (internal resolution) must be added to point to the Load balancer targeting the control plane machines.

• OpenShift routes A DNS record *.apps.<cluster_name>.<base_domain> must be provided to point to the Load balancer configured to manage the traffic for the ingress router (ports 443 and 80 of the compute machines).

NOTE: the DNS records above may also point to the same IP in case you are using only one load balancer configured with the rules described in the previous section.

RHCOS image

This UPI installation process requires a proper RHCOS (Red Hat Enterprise Linux CoreOS) image URL to be set in the inventory.yml file.

The RHCOS images can be found here and you have to choose the URL related to the OpenStack qcow2 image type, like in the example below

https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/4.6.0-0.nightly-2020-07-16-122837/rhcos-4.6.0-0.nightly-2020-07-16-122837-x86_64-openstack.x86_64.qcow2.gz

The version of the image should be choosen according to the OpenShift version you're about to install (in general less than or equal to the OCP version). Once you have the URL set in the inventory.yml a dedicated Ansible playbook will be in charge to download the qcow2.gz file, uncompress it in a specified folder and use it to create oVirt/RHV templates.

Getting Ansible playbooks

All the Ansible playbooks used in this UPI installation process are available here and can be downloaded with the following utility script

RELEASE="release-4.6"; \
curl -L -X GET https://api.github.com/repos/openshift/installer/contents/upi/ovirt\?ref\=${RELEASE} | 
grep 'download_url.*\.yml' | 
awk '{ print $2 }' | sed -r 's/("|",)//g' | 
xargs -n 1 curl -O

Different versions of the oVirt UPI playbooks can be downloaded changing the RELEASE environment variable to the desired branch (please be aware that this UPI work started with the release-4.6).

Assets directory

Before proceeding with the installation is better to set an environment variable with the path (absolute or relative according to your preferences) of the directory in which the openshift-install command will put all the artifacts and that we'll also refer to in the inventory.yml`.

$ export ASSETS_DIR=./wrk

Inventory Explained

This section shows an example of inventory.yml, used to specify the variables needed for the UPI installation process, with a brief explanation of the sections included.

---
all:
  vars:

    # ---
    # General section
    # ---
    ovirt_cluster: "Default" 
    ocp:
      assets_dir: "{{ lookup('env', 'ASSETS_DIR') }}"
      ovirt_config_path: "{{ lookup('env', 'HOME') }}/.ovirt/ovirt-config.yaml"

    # ---
    # RHCOS section
    # ---
    rhcos:
      image_url: "https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/latest-4.6/rhcos-4.6.0-0.nightly-2020-07-16-122837-x86_64-openstack.x86_64.qcow2.gz"
      local_cmp_image_path: "/tmp/rhcos.qcow2.gz"
      local_image_path: "/tmp/rhcos.qcow2"

    # ---
    # Profiles section
    # ---
    control_plane:
      cluster: "{{ ovirt_cluster }}"
      memory: 16GiB
      sockets: 4
      cores: 1
      template: rhcos_tpl
      operating_system: "rhcos_x64"
      type: high_performance
      graphical_console:
        headless_mode: false
        protocol:
        - spice
        - vnc
      disks:
      - size: 120GiB
        name: os
        interface: virtio_scsi
        storage_domain: depot_nvme
      nics:
      - name: nic1
        network: lab
        profile: lab

    compute:
      cluster: "{{ ovirt_cluster }}"
      memory: 16GiB
      sockets: 4
      cores: 1
      template: worker_rhcos_tpl
      operating_system: "rhcos_x64"
      type: high_performance
      graphical_console:
        headless_mode: false
        protocol:
        - spice
        - vnc
      disks:
      - size: 120GiB
        name: os
        interface: virtio_scsi
        storage_domain: depot_nvme
      nics:
      - name: nic1
        network: lab
        profile: lab

    # ---
    # VMs section
    # ---
    vms:
    - name: "{{ metadata.infraID }}-bootstrap"
      ocp_type: bootstrap
      profile: "{{ control_plane }}"
      type: server
    - name: "{{ metadata.infraID }}-master0"
      ocp_type: master
      profile: "{{ control_plane }}"
    - name: "{{ metadata.infraID }}-master1"
      ocp_type: master
      profile: "{{ control_plane }}"
    - name: "{{ metadata.infraID }}-master2"
      ocp_type: master
      profile: "{{ control_plane }}"
    - name: "{{ metadata.infraID }}-worker0"
      ocp_type: worker
      profile: "{{ compute }}"
    - name: "{{ metadata.infraID }}-worker1"
      ocp_type: worker
      profile: "{{ compute }}"
    - name: "{{ metadata.infraID }}-worker2"
      ocp_type: worker
      profile: "{{ compute }}"

General section

Variables in this section are mandatory and allow the user to specify

• ovirt_cluster: the name of the ovirt cluster in which you'll install the OCP cluster.
• ocp.assets_dir: is the path of the folder in which the openshift-install command will put all the files built in different stages.
• ocp.ovirt_config_path: path of the ovirt-config.yaml, generated by the openshift-install in the first stage, containing the ovirt credentials (necessary to interact with the oVirt/RHV Manager REST API).

RHCOS section

The rhcos variable contains the RHCOS public URL (image_url) for downloading the image in the local specified path (local_cmp_image_path) and uncompressing it (in a file described by local_image_path) before being able to use it.

  rhcos:
    image_url: "https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/4.6.0-0.nightly-2020-07-16-122837/rhcos-4.6.0-0.nightly-2020-07-16-122837-x86_64-openstack.x86_64.qcow2.gz"
    local_cmp_image_path: "/tmp/rhcos.qcow2.gz"
    local_image_path: "/tmp/rhcos.qcow2"

Please refer to the specific paragraph to learn more about which version of RHCOS image to choose and where to find it.

Profiles and VMs

The latest and important part of the inventory.yml is related to profiles and vms definition using the capabilities offered by the ovirt.vm-infra role, whose documentation can be found here. In the following paragraphs we'll explain the meaning of the basic parameters from the OCP point of view.

Profiles section

This section is mainly composed of two variable, control_plane and compute, that define the two different profiles used respectively for masters (and bootstrap) and workers.

    control_plane:
      cluster: "{{ ovirt_cluster }}"
      memory: 16GiB
      sockets: 4
      cores: 1
      template: rhcos_tpl
      operating_system: "rhcos_x64"
      type: high_performance
      disks:
      - size: 120GiB
        name: os
        interface: virtio_scsi
        storage_domain: depot_nvme
      nics:
      - name: nic1
        network: lab
        profile: lab

    compute:
      cluster: "{{ ovirt_cluster }}"
      memory: 16GiB
      sockets: 4
      cores: 1
      template: worker_rhcos_tpl
      operating_system: "rhcos_x64"
      type: high_performance
      disks:
      - size: 120GiB
        name: os
        interface: virtio_scsi
        storage_domain: depot_nvme
      nics:
      - name: nic1
        network: lab
        profile: lab

The user can customize the parameters of both profiles according to the needs and the minimum requirements.

• cluster: it's already set according to the value of the variable in the General Section.
• memory, sockets, cores: mandatory parameters necessary to define the common specs of the VMs generated from this profile.
• template: name of the template that the virtual machine will be based on (refer to this for further information about templates).
• operating_system: sets the vm OS type. With oVirt/RHV 4.4 it's mandatory to use the value rhcos_x64 to allow the ignition script to be correctly passed to the VM.
• type: it's the type that the VM will have once created.
• disks: in this section the specs of the disk must be set according to the basic requirements of OCP in terms of capacity and storage performances. It's possible to choose different storage-domains for control_plane and compute nodes.
• nics: defines the specs like the name of the nic and the network that the Vms will use. The virtual network interface profile can also be specified. The MAC address will be taken from the oVirt/RHV MAC pool.

VMs section

In this last section of the inventory.yml there's the definition of the vms variable, containing all the node instance that the user plans to create to deploy the OCP cluster (remember that there are minimum requirements in terms of number of master and worker nodes).

In the last section there's the list of all the vms that will be created and their role expressed by the ocp_type.

  vms:
    - name: "{{ metadata.infraID }}-bootstrap"
      ocp_type: bootstrap
      profile: "{{ control_plane }}"
      type: server
    - name: "{{ metadata.infraID }}-master0"
      ocp_type: master
      profile: "{{ control_plane }}"
    - name: "{{ metadata.infraID }}-master1"
      ocp_type: master
      profile: "{{ control_plane }}"
    - name: "{{ metadata.infraID }}-master2"
      ocp_type: master
      profile: "{{ control_plane }}"
    - name: "{{ metadata.infraID }}-worker0"
      ocp_type: worker
      profile: "{{ compute }}"
    - name: "{{ metadata.infraID }}-worker1"
      ocp_type: worker
      profile: "{{ compute }}"

As you can see above the vms variable is basically defined by a list of elements each one with at least three mandatory attributes

• name: name of the virtual machine to create.
• ocp_type: the role of the virtual machine in the OCP cluster (possible values are bootstrap, master, worker).
• profile: name of the profile (control_plane or compute) from which to inherit common specs

Additional attributes can be specified to override the ones defined in the inheriting profile

• type: is re-defined as server in the bootstrap vm

It's also possible to use all the attributes documented in the oVirt.vm-infra role (e.g.: fixed MAC address for each vm that could help to assign permanent IP through a DHCP).

Note: Looking at the vms attribute name setting, you can see that we are using the variable metadata.infraID whose value is obtained parsing the metadata.json file generated using the command openshift-install create ignition-configs (read more about it here). There's a specific set of Ansible tasks (common-auth.yml) included in all the UPI playbooks that contains the code to read the infraID from the specific file located in the ocp.assets_dir

---
- name: include metadata.json vars
  include_vars:
    file: "{{ ocp.assets_dir }}/metadata.json"
    name: metadata
  
  ...

Install config

Run the openshift-install to create the initial install-config using as assets directory the same that is specified in the inventory.yml (working_path).

$ openshift-install create install-config --dir $ASSETS_DIR
? SSH Public Key /home/user/.ssh/id_dsa.pub
? Platform <ovirt>
? Engine FQDN[:PORT] [? for help] <engine.fqdn>
? Enter ovirt-engine username <admin@internal>
? Enter password <******>
? oVirt cluster <cluster>
? oVirt storage <storage>
? oVirt network <net> 
? Internal API virtual IP <172.16.0.252>
? Ingress virtual IP <172.16.0.251>
? Base Domain <example.org>
? Cluster Name <ocp4>
? Pull Secret [? for help] <********>

Internal API and Ingress are the IPs added following the above DNS instructions

• api.ocp4.example.org: 172.16.0.252
• *.apps.ocp4.example.org: 172.16.0.251

Cluster Name (ocp4) and Base Domain (example.org) joint together will form the FQDN of the OCP cluster used to expose the API interface (https://api.ocp4.example.org:6443/) and the newly created applications (e.g. https://console-openshift-console.apps.ocp4.example.org).

You can obtain a new Pull secret from here.

The result of this first step is the creation of a install-config.yaml in the specified assets directory:

$ tree
.
└── wrk
    └── install-config.yaml

File $HOME/.ovirt/ovirt-config.yaml was also created for you by the openshift-install containing all the connection parameters needed to reach the oVirt/RHV engine and use its REST API.

NOTE: Some of the parameters added during the openshift-install workflow, in particular the Internal API virtual IP and Ingress virtual IP, will not be used because already configured in your infrastructure DNS (see DNS section). Other paramenters like oVirt cluster, oVirt storage, oVirt network, will be used as specified in the inventory.yml and removed from the install-config.yaml with the previously mentioned virtual IPs, using a script reported in a section below.

Set compute replicas to zero

Machine API will not be used by the UPI to create nodes, we'll create compute nodes explicitly with Ansible scripts. Therefore we'll set the number of compute nodes to zero replicas using the following python script:

$ python3 -c 'import os, yaml
path = "%s/install-config.yaml" % os.environ["ASSETS_DIR"]
conf = yaml.safe_load(open(path))
conf["compute"][0]["replicas"] = 0
open(path, "w").write(yaml.dump(conf, default_flow_style=False))'

NOTE: All the Python snippets in this document work with both Python 3 and Python 2.

Set machine network

OpenShift installer sets a default IP range for nodes and we need to change it according to our infrastructure. We'll set the range to 172.16.0.0/16 (we can use the following python script for this):

$ python3 -c 'import os, yaml
path = "%s/install-config.yaml" % os.environ["ASSETS_DIR"]
conf = yaml.safe_load(open(path))
conf["networking"]["machineNetwork"][0]["cidr"] = "172.16.0.0/16"
open(path, "w").write(yaml.dump(conf, default_flow_style=False))'

Set platform to none

The UPI for oVirt/RHV is similar to the bare-metal installation process and for now we don't need the specific ovirt platform section in the install-config.yaml, all the settings needed are specified in the inventory.yml. We'll remove the section

$ python3 -c 'import os, yaml
path = "%s/install-config.yaml" % os.environ["ASSETS_DIR"]
conf = yaml.safe_load(open(path))
platform = conf["platform"]
del platform["ovirt"]
platform["none"] = {}
open(path, "w").write(yaml.dump(conf, default_flow_style=False))'

Manifests

Editing manifests is an action required for the UPI and to generate them we can use again the binary installer

$ openshift-install create manifests --dir $ASSETS_DIR

$ tree
.
└── wrk
    ├── manifests
    │   ├── 04-openshift-machine-config-operator.yaml
    │   ├── cluster-config.yaml
    │   ├── cluster-dns-02-config.yml
    │   ├── cluster-infrastructure-02-config.yml
    │   ├── cluster-ingress-02-config.yml
    │   ├── cluster-network-01-crd.yml
    │   ├── cluster-network-02-config.yml
    │   ├── cluster-proxy-01-config.yaml
    │   ├── cluster-scheduler-02-config.yml
    │   ├── cvo-overrides.yaml
    │   ├── etcd-ca-bundle-configmap.yaml
    │   ├── etcd-client-secret.yaml
    │   ├── etcd-host-service-endpoints.yaml
    │   ├── etcd-host-service.yaml
    │   ├── etcd-metric-client-secret.yaml
    │   ├── etcd-metric-serving-ca-configmap.yaml
    │   ├── etcd-metric-signer-secret.yaml
    │   ├── etcd-namespace.yaml
    │   ├── etcd-service.yaml
    │   ├── etcd-serving-ca-configmap.yaml
    │   ├── etcd-signer-secret.yaml
    │   ├── kube-cloud-config.yaml
    │   ├── kube-system-configmap-root-ca.yaml
    │   ├── machine-config-server-tls-secret.yaml
    │   └── openshift-config-secret-pull-secret.yaml
    └── openshift
        ├── 99_kubeadmin-password-secret.yaml
        ├── 99_openshift-cluster-api_master-user-data-secret.yaml
        ├── 99_openshift-cluster-api_worker-user-data-secret.yaml
        ├── 99_openshift-machineconfig_99-master-ssh.yaml
        ├── 99_openshift-machineconfig_99-worker-ssh.yaml
        └── openshift-install-manifests.yaml

The command above will write manifests consuming the install-config.yaml and will show a warning message. If you plan on reusing the install-config.yaml file, back it up before you generate manifests.

INFO Consuming Install Config from target directory 
WARNING Making control-plane schedulable by setting MastersSchedulable to true for Scheduler cluster settings 

Set control-plane nodes unschedulable

Setting compute replicas to zero makes control-plane nodes schedulable which is something we don't want for now (router pods can run also on control-plane nodes but there are some Kubernetes limitation that will prevent those pods to be reachable by the ingress load balancer). Setting the control-plan as unschedulable means modifying the manifests/cluster-scheduler-02-config.yml setting masterSchedulable to False.

$ python3 -c 'import os, yaml
path = "%s/manifests/cluster-scheduler-02-config.yml" % os.environ["ASSETS_DIR"]
data = yaml.safe_load(open(path))
data["spec"]["mastersSchedulable"] = False
open(path, "w").write(yaml.dump(data, default_flow_style=False))'

Ignition configs

The next step is the one needed to build Ignition files from the manifests just modified. Ignition files have to be fetched by RHCOS machine initramfs to perform configurations that will bring to a live final node. We will use again the binary installer.

$ openshift-install create ignition-configs --dir $ASSETS_DIR

$ tree
.
└── wrk
    ├── auth
    │   ├── kubeadmin-password
    │   └── kubeconfig
    ├── bootstrap.ign
    ├── master.ign
    ├── metadata.json
    └── worker.ign

Other than the ignition files the installer generated

• auth folder containing the admin credentials necessary to connect to the cluster via the oc or kubectl CLI utilities.
• metadata.json with information like the OCP cluster name, OCP cluster ID and the infraID (generated for the current running installation).

The infraID will be used by the UPI Ansible playbooks as prefix for the VMs created during the installation process avoiding name clashes in case of multiple installations in the same oVirt/RHV cluster.

Note: certificates contained into ignition config files expire after 24 hours. You must complete the cluster installation and keep the cluster running for 24 hours in a non-degradated state to ensure that the first certificate rotation has finished.

Create templates and VMs

After having checked that all our variables in the inventory.yml fit the needs, we can run the first of our Ansible provisioning playbooks.

$ ansible-playbook -i inventory.yml create-templates-and-vms.yml

This playbook will use the connection parameters for the oVirt/RHV engine stored in the $HOME/.ovirt/ovirt-config.yaml reading also the metadata.json from the assets directory.

According to the variables

  rhcos:
    image_url: "https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/4.6.0-0.nightly-2020-07-16-122837/rhcos-4.6.0-0.nightly-2020-07-16-122837-x86_64-openstack.x86_64.qcow2.gz"
    local_cmp_image_path: "/tmp/rhcos.qcow2.gz"
    local_image_path: "/tmp/rhcos.qcow2"

the RHCOS image will be downloaded (in case not already existing locally), stored locally and extracted to be uploaded on the oVirt/RHV node and used for template creation. The user can check the RHCOS image for the OpenShift version he want to use from here.

Templates will be created according to the names specified in the inventory.yml for the control_plane and compute profile (in case of different names two different templates will be created). In case the user want to have different templates for each OCP installation in the same cluster, it is possible to customize the template name in the inventory.yml prepending the infraID (like it is done for VMs' names).

  control_plane:
    cluster: "{{ ovirt_cluster }}"
    memory: 16GiB
    sockets: 4
    cores: 1
    template: "{{ metadata.infraID }}-rhcos_tpl"
    operating_system: "rhcos_x64"
    ...

At the end of the execution the VMs specified will be created and left in stopped mode. This allows the user to fetch any information from the VMs that can help configuring other infrastructure elements (e.g.: getting MAC addresses to feed a DHCP server to assign permanent IPs).

Bootstrap

$ ansible-playbook -i inventory.yml bootstrap.yml

The playbook starts the bootstrap VM passing it the bootstrap.ign ignition file contained in the assets directory. That will allow the bootstrap node to configure itself and be ready to serve ignition files for the master nodes. The user can check the console inside oVirt/RHV UI or can connect to the VM via SSH. Running the following command from inside the bootstrap VM enables them to closely monitor the bootstrap process.

$ ssh core@<boostrap.ip>
[core@ocp4-lk6b4-bootstrap ~]$ journalctl -b -f -u release-image.service -u bootkube.service

Master nodes

$ ansible-playbook -i inventory.yml masters.yml

The masters.yml will start our control-plane made of three masters (but it can be customized) passing the master.ign ignition file to each of the VMs. master.ign ignition file contains a directive that instructs masters to fetch the ignition from the URL

https://api-int.ocp4.example.org:22623/config/master

targeted by the Load balancer that manages the traffic on port 22623 (accessible only inside the cluster) driving it to masters and bootstrap.

Wait for Control Plane

The user can monitor the control-plane bootstrap process with the following command:

$ openshift-install wait-for bootstrap-complete --dir $ASSETS_DIR

After some time the output of the command will be the following

INFO API v1.18.3+b74c5ed up
INFO Waiting up to 40m0s for bootstrapping to complete... 

After all the pods on master nodes and the etcd will be up and running the installer will show the following output:

INFO It is now safe to remove the bootstrap resources

OpenShift API

The OpenShift API can be accessed via the oc or kubectl using the admin credentials contained in the assets directory in the file auth/kubeconfig:

$ export KUBECONFIG=$ASSETS_DIR/auth/kubeconfig
$ oc get nodes
$ oc get pods -A

Retire Bootstrap

After the wait-for command says that the bootstrap process is complete, it is possible to remove the bootstrap VM

$ ansible-playbook -i inventory.yml retire-bootstrap.yml

and the user can remove it also from the Load balancer directives.

Worker nodes

$ ansible-playbook -i inventory.yml workers.yml

This is similar to what we did for masters but in this case workers won't automatically join the cluster, we'll need to approve their respective pending CSRs (Certificate Signing Requests).

Approve CSRs

CSRs for nodes joining the cluster will need to be approved by the administrator. The following command helps to list pending requests

$ oc get csr -A

Eventually one pending CSR per node will be shown

NAME        AGE    SIGNERNAME                                    REQUESTOR                                                                   CONDITION
csr-2lnxd   63m    kubernetes.io/kubelet-serving                 system:node:ocp4-lk6b4-master0.ocp4.example.org                             Approved,Issued
csr-hff4q   64m    kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Approved,Issued
csr-hsn96   60m    kubernetes.io/kubelet-serving                 system:node:ocp4-lk6b4-master2.ocp4.example.org                             Approved,Issued
csr-m724n   6m2s   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
csr-p4dz2   60m    kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Approved,Issued
csr-t9vfj   60m    kubernetes.io/kubelet-serving                 system:node:ocp4-lk6b4-master1.ocp4.example.org                             Approved,Issued
csr-tggtr   61m    kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Approved,Issued
csr-wcbrf   7m6s   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending

To filter and watch pending CSRs the following command can be used

$ watch "oc get csr -A | grep pending -i"

that refresh the output every two seconds

Every 2.0s: oc get csr -A | grep pending -i

csr-m724n   10m   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
csr-wcbrf   11m   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending

Every Pending request should be inspected

$ oc describe csr csr-m724n
Name:               csr-m724n
Labels:             <none>
Annotations:        <none>
CreationTimestamp:  Sun, 19 Jul 2020 15:59:37 +0200
Requesting User:    system:serviceaccount:openshift-machine-config-operator:node-bootstrapper
Signer:             kubernetes.io/kube-apiserver-client-kubelet
Status:             Pending
Subject:
         Common Name:    system:node:ocp4-lk6b4-worker1.ocp4.example.org
         Serial Number:  
         Organization:   system:nodes
Events:  <none>

and finally approved

$ oc adm certificate approve csr-m724n

After approving the first requests, another CSR per each worker will be issued and will need to be approved to have worker nodes not only joining the cluster, but also becoming Ready and having pods scheduled on them.

Wait for Installation Complete

The following command can now be run to follow the installation process till it's complete:

$ openshift-install wait-for install-complete --dir $ASSETS_DIR --log-level debug

Eventually it will give as output

• URL to reach the OpenShift Console (web UI).
• Username and password for the admin login.

Destroy OpenShift cluster

$ ansible-playbook -i inventory.yml \
    retire-bootstrap.yml \
    retire-masters.yml   \
    retire-workers.yml

Removing DNS added records, Load balancers and any other infrastructure configuration is left to the user.