Merge pull request #90 from stackhpc/deployment

Fixes and improvements to deployment

deployment/inventory

@@ -1,24 +1,40 @@
+# Example inventory for deployment to a single host (localhost).
+
+# HAProxy load balancer.
+# Should contain exactly one host.
 [haproxy]
 localhost ansible_connection=local
 
+# Jaeger distributed tracing UI.
+# Should contain at most one host.
 [jaeger]
 localhost ansible_connection=local
 
+# Minio object storage service (for test & development only).
+# Should contain at most one host.
 [minio]
 localhost ansible_connection=local
 
+# Prometheus monitoring service.
+# Should contain at most one host.
 [prometheus]
 localhost ansible_connection=local
 
+# Reductionist servers.
+# May contain multiple hosts.
 [reductionist]
 localhost ansible_connection=local
 
-[step:children]
-reductionist
-
+# Step Certificate Authority (CA).
+# Should contain exactly one host.
 [step-ca]
 localhost ansible_connection=local
 
+# Do not edit.
+[step:children]
+reductionist
+
+# Do not edit.
 [docker:children]
 haproxy
 jaeger

deployment/site.yml

@@ -186,7 +186,6 @@
       changed_when: false
       delegate_to: "{{ groups['step-ca'][0] }}"
       run_once: true
-      when: not step_stat.stat.exists
 
     - name: Bootstrap CA
       ansible.builtin.command: >
@@ -386,12 +385,36 @@
         volumes: "{{ reductionist_volumes }}"
         restart: true  # Load new certificates. TODO: Hot reload
 
+- name: Gather facts for HAProxy
+  hosts:
+    - reductionist
+    - "!haproxy"
+  tags:
+    - haproxy
+  gather_facts: true
+
 - name: Deploy HAProxy
   hosts: haproxy
   tags:
     - haproxy
   become: true
   tasks:
+    # Currently we are not deploying any failover mechanism such as keepalived,
+    # so limit to one HAProxy server.
+    - name: Assert that there is only one HAProxy server
+      ansible.builtin.assert:
+        that:
+          groups['haproxy'] | length == 1
+
+    - name: Wait for reductionist backends to be accessible from HAProxy host
+      ansible.builtin.uri:
+        url: "https://{{ hostvars[item].ansible_facts.default_ipv4.address }}:8081/.well-known/reductionist-schema"
+        # The certificates are only valid for the HAProxy frontend address (reductionist_host).
+        validate_certs: false
+      until: result is success
+      register: result
+      loop: "{{ query('inventory_hostnames', 'reductionist') }}"
+
     - name: Ensure /etc/haproxy directory exists
       ansible.builtin.file:
         path: /etc/haproxy
@@ -417,6 +440,6 @@
 
     - name: Wait for reductionist server to be accessible via HAProxy
       ansible.builtin.uri:
-        url: "https://{{ ansible_facts.default_ipv4.address }}:8080/.well-known/reductionist-schema"
+        url: "https://{{ reductionist_host }}:8080/.well-known/reductionist-schema"
       until: result is success
       register: result

docs/deployment.md

@@ -3,12 +3,6 @@
 The [deployment](https://github.com/stackhpc/reductionist-rs/tree/main/deployment) directory in the Reductionist Git repository contains an Ansible playbook to deploy Reductionist and supporting services to one or more hosts.
 The Ansible playbook allows for a secure, scale-out deployment of Reductionist, with an HAProxy load balancer proxying requests to any number of Reductionist backend servers.
 
-The following OS distributions are supported:
-
-* Ubuntu 20.04-22.04
-* CentOS Stream 8-9
-* Rocky Linux 8-9
-
 The following services are supported:
 
 * Docker engine
@@ -20,31 +14,74 @@ The following services are supported:
 * Reductionist
 * HAProxy (load balancer for Reductionist)
 
+## Prerequisites
+
+The existence of correctly configured hosts is assumed by this playbook.
+
+The following host OS distributions are supported:
+
+* Ubuntu 20.04-22.04
+* CentOS Stream 8-9
+* Rocky Linux 8-9
+
+Currently only a single network is supported.
+Several TCP ports should be accessible on this network.
+This may require configuration of a firewall on the host (e.g. firewalld, ufw) or security groups in a cloud environment.
+
+* SSH: 22
+* Reductionist backend: 8081
+* Reductionist frontend: 8080 (HAProxy host only)
+* Step CA: 9999 (Step CA host only)
+* Minio: 9000 (Minio host only)
+* Prometheus: 9090 (Prometheus host only)
+* Jaeger: 16686 (Jaeger host only)
+
+The Ansible control host (the host from which you will run `ansible-playbook`) should be able to resolve the hostnames of the hosts.
+If names are not provided by DNS, entries may be added to `/etc/hosts` on the Ansible control host.
+Issues have been reported when using Ansible with password-protected SSH private keys and SSH agent.
+
+It may be desirable to host the Reductionist API on a different address, such as a hostname or public IP exposed on the host running HAProxy.
+This may be configured using the `reductionist_host` variable.
+
 ## Configuration
 
 An example Ansible inventory file is provided in [inventory](https://github.com/stackhpc/reductionist-rs/blob/main/deployment/inventory) which defines all groups and maps localhost to them. For a production deployment it is more typical to deploy to one or more remote hosts.
 
 The following example inventory places HAProxy, Jaeger, Prometheus and Step CA on `reductionist1`, while Reductionist is deployed on `reductionist1` and `reductionist2`.
 
 ```ini
+# Example inventory for deployment to two hosts, reductionist1 and reductionist2.
+
+# HAProxy load balancer.
+# Should contain exactly one host.
 [haproxy]
 reductionist1
 
+# Jaeger distributed tracing UI.
+# Should contain at most one host.
 [jaeger]
 reductionist1
 
+# Prometheus monitoring service.
+# Should contain at most one host.
 [prometheus]
 reductionist1
 
+# Reductionist servers.
+# May contain multiple hosts.
 [reductionist]
 reductionist[1:2]
 
-[step:children]
-reductionist
-
+# Step Certificate Authority (CA).
+# Should contain exactly one host.
 [step-ca]
 reductionist1
 
+# Do not edit.
+[step:children]
+reductionist
+
+# Do not edit.
 [docker:children]
 haproxy
 jaeger
@@ -54,7 +91,7 @@ reductionist
 step-ca
 ```
 
-Some variables are provided to configure the deployment in the [group_vars](https://github.com/stackhpc/reductionist-rs/tree/main/deployment/group_vars) directory. Reductionist configuration options may be specified using environment variables.
+Some variables are provided to configure the deployment in the [group_vars](https://github.com/stackhpc/reductionist-rs/tree/main/deployment/group_vars) directory. Reductionist configuration options may be specified using environment variables specified using `reductionist_env`.
 
 ## Ansible control host setup