Custos deployment on Jetstream

We were able to successfully deploy the custos using the documentation from CloudElves, Scapsulators and DSDummies. We want to give all the credit to them for their documentation and continuous help through out our deployment. Thank you guys!

Deployment Hardware:

1. One Master and 2 - workers. (Each machine is from JS1 and a medium type)
2. Deployed Kubernetes 1.19.15 version using Rancher cloudman boot.

Keycloak endpoint: https://js-169-225.jetstream-cloud.org/ui/vault/auth

Here is our deployment of all the pods:

Testing API for creating a Tenant:

API endpoint: https://js-169-225.jetstream-cloud.org/tenant-management/v1.0.0/oauth2/tenant
Request type: POST

{
    "client_name":"test1",
    "requester_email":"[email protected]",
    "admin_username":"test",
    "admin_first_name":"test",
    "admin_last_name":"test",
    "admin_email":"[email protected]",
    "contacts":["[email protected]","[email protected]"],
    "redirect_uris":["http://localhost:8080/callback*",
    "https://js-169-225.jetstream-cloud.org/callback*"],
    "scope":"openid profile email org.cilogon.userinfo",
    "domain":"js-169-225.jetstream-cloud.org",
    "admin_password":"test",
    "client_uri":"https://js-169-225.jetstream-cloud.org/",
    "logo_uri":"https://js-169-225.jetstream-cloud.org/",
    "application_type":"web",
    "comment":"Custos super tenant for production"
}

Response:

{
    "client_id": "custos-udthvb74xlxarpeyvzje-10000002",
    "client_secret": "JYLz6eyd9orKyZ4NR1AzCvO7cNTGDIbsWeoAtzJz",
    "is_activated": false,
    "client_id_issued_at": 1651795740000,
    "client_secret_expires_at": 0,
    "registration_client_uri": "https://custos.scigap.org/apiserver/tenant-management/v1.0.0/oauth2/tenant?client_id=custos-udthvb74xlxarpeyvzje-10000002",
    "token_endpoint_auth_method": "client_secret_basic",
    "msg": "Use Base64 encoded clientId:clientSecret as auth token for authorization, Credentials are activated after admin approval"
}

Login to keycloak and set the supertenant to true for id with 10000002. Activating the tenant

{
"client_id":"custos-udthvb74xlxarpeyvzje-10000002",
"status":"ACTIVE",
"super_tenant":true,
"updatedBy":"user"
}

Response:

{
    "tenant_id": "10000002",
    "status": "ACTIVE"
}

Challenges:

• We faced many issues while building a stable kubernetes cluster. Sol: Installing Rancher, K8s on Jetstream 1 installation was smooth.
• Deploying Vault was tricky for the first time. Especially the steps needed to unseal the vault.
• Postgres server not getting launched. Sol: The solution was to use 'postgres' for both username as password. Using 'postgres' as name will give admin acesss. We really don't know why the other username and password combination was not working for us.
• Unresolved ACME pod. In js2 the certificates are failing because there are not able to be verified. In JS1 again we did not face this issue.
• Labelling the nodes to actually start the custos pods.
• Instead of creating the volumes, creating the directories helped resolved issues connecting to the DB.

And many more..

It was a great lesson for us to understand how important is the documentation is. We wish the documentation was more friendly. But thanks to Isuru for his time and the utility scripts for deploying in custos, without him it would have been an impossible task to all of us.