-
Notifications
You must be signed in to change notification settings - Fork 64
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Parallel nix evaluation in github actions #947
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Back to draft: let's not merge this yet, I'll add one more change to this PR in a moment. |
Signed-off-by: Henri Rosten <[email protected]>
Signed-off-by: Henri Rosten <[email protected]>
Signed-off-by: Henri Rosten <[email protected]>
Signed-off-by: Henri Rosten <[email protected]>
Signed-off-by: Henri Rosten <[email protected]>
Signed-off-by: Henri Rosten <[email protected]>
70a3e72
to
bae51db
Compare
|
If you want to try out something quickly you can also go for: https://github.com/nix-community/nix-github-actions Don't know how this would scale for this project because we don't quite a few nixos closures here. https://github.com/DeterminateSystems/magic-nix-cache may work but don't know if we would run into too many rate limits here. |
filter="$3" | ||
echo "[+] Using filter: '$filter'" | ||
# Output all flake output names | ||
nix flake show --all-systems --json |\ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If this is just about evaluation, this will already evaluate everything. You shouldn't need nix-eval-jobs afterwards.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It will be also faster because nix-eval-jobs has to serialize drv files to disk.
Based on #947 (comment), we should not merge this. |
Description of changes
Do not merge this, see: #947 (comment)
Various updates to Ghaf github actions, most importantly:
eval.yml
, which runs the nix evaluation in parallel, splitting the evaluation to 8 subsets, each of which are executed on a separate github-hosted runner. In testing, this improves the eval action runtime from ~35 minutes down to ~7 minutes.eval.sh
allows filtering the evaluated targets via the-t
option: the initial version in this PR will evaluate all Ghaf flake outputs that match the default filter 'packages.*
'.eval
,build
, andcheck
to make it more apparent if one of those workflows would start failing after merge to main.Checklist for things done
x86_64
aarch64
riscv64
make-checks
and it passesnixos-rebuild ... switch
Instructions for Testing
Tested in fork: https://github.com/henrirosten/ghaf/actions/workflows/eval.yml