global caches & cargo-sweep

[idelvall]

Summary

This PR:

• Runs cargo sweep after the user cargo sub-command
• Introduces @ijc RUN_WITH_CACHE UDC
• Caches $CARGO_HOME is a non-locking global cache (shared across all Earthfiles). This significantly reduces cache size

On global cache

In order to make the new global cache work with sharing=shared mode (that allows parallel builds to access its contents), I had to make sure that $CARGO_HOME/.package-cache was also being stored in the cache, since this the file used by Cargo as a lock to synchronize the access to $CARGO_HOME from parallel builds.

The simplest way I found to achieve this was by temporarily changing the whole $CARGO_HOME location to a folder within the cache, while running the same bin/cargo program.

[ijc]

LGTM.

[idelvall]

Ok, by using global caches (caches with id) I encounter this exact problem https://users.rust-lang.org/t/does-dockers-mount-type-cache-cause-problems/97073.
Now trying to understand why cargo locks don't work in the buildkit cache mount...

[idelvall]

I've checked and flock properly works on shared cache mounts, so still investigating why I am facing this rust-lang/cargo#11376

[idelvall]

This looks promising: rust-lang/cargo#11376 (comment)

[idelvall]

This last commit sets $CARGO_INSTALL_ROOT pointing to the original $CARGO_HOME (in the target filesystem, not the mount cache).

The reason is that, installed crates are not fetched on demand if not found (they need to be installed), so if BK GC removes them from the cache then the build will fail, while if they are stored in the target layers and BK GCed them, the build will rerun those commands and install them back.

This setting makes cargo install to store the crates in the target filesystem to avoid such scenario

[idodod]

Looks good, just a few minor comments.