Reshape values.yaml to be more ergonomic

[chrisseto]

I've heard from many that the shape of our values.yaml isn't the most inviting and I agree. Let's propose a more ergonomic shape and then utilize our upcoming versioning policy (#1093 ) to march towards it.

Must Haves

Layers

Values should have cascading "layers" that are defined in terms of the layer underneath them. The "bottom" layer would be the raw manifests that are being deployed to allow overriding of any fields but also to better document the output of the chart.

Omnipotent backdoors

While kustomize as a post rendering hook can fix most issues it's not always an option and there are some limitations. We should build a first class "backdoor" into the chart so users are never hard blocked on us implementing new features. Accepting a list of JSON patches or even a subset of Kustomize options could go a very long way here. The kustomize route would also be an excellent option for implementing all the "common____" options in a more understandable way.

Utilize well known types

Rather than defining our own types for everything, we should reach for off the self versions from within the Kubernetes API as much as possible. Doing so will allow us to point to external documentation that users are likely already familiar with rather than having to maintain our own set thereof.

Rely on patterns and rules of thumbs rather than heavy documentation

Rather than exhaustively explaining every field, we should try to define flexible patterns that can be applied at large. For example, any value that might be a large string (This TLS cert or secretes) should be able to be defined as a raw string, configmap reference, or secret reference.

Nice to Haves

• Doesn't rely on map[string]T types so that we don't have to worry about setting preserveUnknownFields.

JIRA Link: K8S-120

[c4milo]

I absolutely agree with the above direction. I would also try to keep it as close as possible to the shape of the configuration type used by Redpanda, replacing the bits that don't make sense in a k8s context for, as mentioned by Chris above:

1. References to configmaps for things like trust stores configurations, etc.
2. References to secrets
3. References to cert-manager are super useful for Redpanda Cloud but they might not be for self-hosted. So, maybe just references to configmaps and secrets for configuring TLS certs and keys.

General rule of thumb is to stay away from strong opinions because they are going to cause friction for self-hosted customers, and even for us in Cloud. More succinctly put, less is more.