feat: add json file in terraform for Civo Multi account

[jokestax]

Description

1.Remove Civo Token from Vault Secrets:

The Civo API token has been removed from the secrets stored in Vault. The Crossplane Terraform provider will no longer utilize this token

2.Reference JSON Secret During Provisioning:

A JSON secret, created at the time of provisioning, will be referenced in the Crossplane system from Vault. This secret will be utilized during the provisioning process and subsequently deleted from the Crossplane namespace once the cluster has been provisioned.

Testing

pass the flag of "--gitops-template-branch civo-multi-account" when proivisoning a cluster and test the multiaccount provisioning by inserting appropriate keys for downstream accounts

[muse-sisay]

Hi @jokestax,

I am a bit confused with the changes you are proposing. Are you suggesting we use the API(kubefirst) to create and delete cluster creds file? If so, in my opinion external secrets should be responsible for creating the secret in the cluster. That way we have a reference in the gitops-template repository.

and subsequently deleted from the Crossplane namespace once the cluster has been provisioned.

What if a user wants to provision a cluster in the same downstream account?

[jokestax]

hey @muse-sisay,yes we need to use the api to create and delete the secrets file,we do store the creds in vault but when provisioning it needs the creds to be in specific format, thats why the create and delete of creds file, to provision in same account by default there will be a secret created in vault with default api key and will be displayed in the ui

[muse-sisay]

What format does Civo provider expect the json file to be?

To expand more on my second question. (or add more questions). What happens if a user makes an infrastructure change? Or what happens when Crossplane's Terraform provide runs a plan every 10 minutes to detect drift and the creds.json file is missing?

[jokestax]

the format is

{
   "apikeys": {
       "tf_key": "write-your-token-here"
   },
   "meta": {
       "current_apikey": "tf_key"
   }
}

i also had this doubt, so I tested it and it seems the cross-plane just stores it in its cache once it picks up the plan,so it doesn't look at secrets again and again,we don't delete it, we just update it when the user clicks to deploy a second cluster

[muse-sisay]

Thank you for testing. I am a little bit confused. Why does it need to be updated when deploying a second cluster?

Another what-if scenario, the crossplane tf provider pod crashes.

[jokestax]

if they used different account to provision it should change the secret (api key) previously present,
and crossplane pod getting crashed, it will spin up a new pod as argued will take care of that,and its very unlikely that in the 5-10 seconds of cross-plane coming pod coming back user will provision a new cluster

[muse-sisay]

There is no need to change the API key (secret) every time you want to spin up a cluster in a different account. The provider resource for each cluster should reference a uniqe secret. (Instead of civo-creds, for example, refer to <account-name>-civo-creds secret file. This elimnates the need to delete/update each time a cluster is provisioned.

crossplane pod getting crashed, it will spin up a new pod as argued will take care of that,and its very unlikely that in the 5-10 seconds of cross-plane coming pod coming back user will provision a new cluster

After a restart, at some point, the terraform provider is going to ran a plan against all the infra/cluster a user has. And morethan one workspace is going to complain it's not able to connect/access the resources.

[muse-sisay]

Apologies for the late reply Rishi. Kubecon has been hectic.

Can you also add an external secret that references creds from vault for the default account? It can be stored in civo-github/templates/mgmt/components/cloud-accounts.

[jokestax]

hi @muse-sisay,why to refer external-secret for default account?

[muse-sisay]

Evening Rishi,

We will be consistent on how we retrieve account creds. We won't have two ways of creating a an account secret.