Crash when launching the same cluster in parallel

[cg505]

Repro steps:

• run sky jobs launch 100x in parallel
• some invocations crash

This seems to be a race condition related to the ray cluster yaml created when provisioning the controller cluster (and I assume this can apply to any cluster that you are launching many times in parallel).
The crash is here:

skypilot/sky/backends/backend_utils.py

Line 995 in 6dc386b

restored_yaml_content = _replace_yaml_dicts(

Since tmp_yaml_path is unique per cluster but not per invocation, it seems that multiple processes are writing to/reading from this file at the same time.
Note that this shouldn't affect the actual yaml_path since that is handled by a simple rename

skypilot/sky/backends/backend_utils.py

Line 1019 in 6dc386b

os.rename(tmp_yaml_path, yaml_path)

One possible solution would be to add some random value to the tmp_yaml_path.

Version & Commit info:

• sky -v: dev
• sky -c: forgot to check, I think f2991b144d4b15eac55dd7f759f361b6146033b3