ci: consolidate tasks and use sccache and nextest #1015
Conversation
Deploying with
|
| Latest commit: |
3542f85
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://93890caa.hydroflow.pages.dev |
| Branch Preview URL: | https://pr1015.hydroflow.pages.dev |
|
Should help speed up builds a lot by avoiding redundant compilation. |
|
Two main optimizations:
|
|
We split up CI testing in order to speed it up. Although some code gets compiled multiple times, the parallelism benefits the overall runtime - #944 What does sccache do? Combining clippy and check (and fmt?) is fine https://github.com/hydro-project/hydroflow/actions/runs/7404907733/job/20147104671?pr=1015 |
|
sccache caches compiled crates into GitHub's action cache, so that future builds can reuse those. Also locally this eliminates redundant builds by using more fine-grained cache keys than cargo. Looking at https://github.com/hydro-project/hydroflow/actions/runs/7404907733/job/20147104671, it seems that after consolidation the total time is only a little longer than the previous longest run 11m -> 14m. My thinking was that this will make it possible to enable Windows builds in PR since this makes it much harder to hit the concurrency limit. |
|
With nextest and some caching fixes (cidev breaks caching because crates build as part of tests are built with dev, so we use an environment variable), down to 9m 43s for all tests. |
shadaj
changed the title
shadaj
force-pushed
the
pr1015
branch
5 times, most recently
from
c5630e7 to
12b3475
Compare
| @@ -47,18 +47,14 @@ pub fn test_tick_loop() { | |||
| async fn test_persist_stratum_run_available() -> Result<(), Box<dyn Error>> { | |||
| let (out_send, out_recv) = hydroflow::util::unbounded_channel(); | |||
|
|
|||
| let handle = tokio::task::spawn_blocking(|| { | |||
There was a problem hiding this comment.
This test was flaky but doesn't actually need any concurrency?
shadaj
force-pushed
the
pr1015
branch
8 times, most recently
from
147ffb6 to
6c9c9a9
Compare
shadaj
force-pushed
the
pr1015
branch
2 times, most recently
from
0b3c87d to
d77c385
Compare
|
@MingweiSamuel bump on considering this? although it increases the max job time by ~1-2 minutes, I've been seeing a lot of builds blocked on a full queue which this would help a lot with |
|
I checked the stats being printed out and it seems to definitely be helping with shared compilations of dependencies. Might not help much with the in-tree sources though. |
ci: consolidate tasks and use sccache and nextest