Application controller can run out of disk space on '/dev/shm', when deploying concurrently to a large number of clusters, or with many large manifests

[jgwest]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

At present, Argo CD (via gitops-engine) uses /dev/shm to store temporary files that are passed to kubectl code, such as manifests and cluster credentials. The temp directory logic can be found within pkg/utils/io/io.go of gitops-engine.

However, since /dev/shm is backed by RAM, it has limited size (but has the security advantage of avoiding writes of sensitive info to disk). For many use cases this is not an issue, but, when deploying to a large number of kubernetes clusters, or deploying large k8s manifests, /dev/shm can run out of disk space.

We can/should allow Argo CD users to customize the target output path for application-controller/gitops-engine generated manifests, to allow them to avoid this issue.

I propose we add a new configuration parameter to argocd-cmd-params-cm (and connected to Application Controller Deployment/StatefulSet YAML), that allow users to specify a custom application controller manifest destination path, which will allow application controller to writing to a different path with the container, such as '/tmp'.

See the linked issue in gitops-engine for further details.

This issue has a corresponding issue on gitops-engine: argoproj/gitops-engine#614
And PR: argoproj/gitops-engine#613

To Reproduce

Concurrently deploy to a large number of clusters, or deploy a large number of large manifests: if the concurrent KiB of generated resources exceed /dev/shm size, /dev/shm runs out of space, and an error occurs.

Expected behavior

Users should be able to specify custom Argo CD path/volume to store generated manifests, allowing them to avoid running out of disk space in /dev/shm.

Version

All