Change Controller concurrency defaults based on available CPU

[clux]

Would you like to work on this feature?

yes

What problem are you trying to solve?

Am writing a doc on scaling controllers and realising there's a footgun in the default unlimited concurrency implemented by controller::Config, unlimited is great until you start having heavy load and THEN it starts breaking (as you hit apiserver limits via throttling, memory limits via OOMs).

Describe the solution you'd like

It is probably better if by default we make this dial use a default based on how much CPU is given to the pod running the controller.

This can be done by doing a multiple of the value received form num_cpus::get for instance (which reads cgroup limits). Maybe a default with a minimum of 2 or 4 would be good? Normally folloing num cores exactly is not great since we usually use way less than that for most controllers.

Solutions from the GO world allow accomplishing the same thing by injecting evars:
https://blog.howardjohn.info/posts/gomaxprocs/

It would be nicer to do this by default.

Describe alternatives you've considered

• maintain unlimited
• default 1 concurrency (feels sluggish in most cases)

Documentation, Adoption, Migration Strategy

must be noted in releases and kube.rs docs

Target crate for feature

kube-runtime