Cahce dependencies in CI to speedup workflows #471
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
savente93
changed the title
DirkEilander
approved these changes
Aug 10, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue addressed
Fixes #393
Explanation
If I'm honest I'm somewhat unsatisfied with this solution, but trying to optimise further is yielding all sorts of problems I don't quite understand, and it is actually faster, so I've decided to leave it where it is for now. If I gain new understanding or motivation I can come back to it later.
Because GitHub for now has a 10GB limit of caching, which is actually quite easy to bump up against if you're not careful, I've added some cache purging. Apparently it is quite easy for conda environments to get out of sync, so I've gone with what seems to be best practice: simply empty the cache at midnight. This might mean that the first Ci you run of the day might take longer, but after that it should be faster than usual. I think this is actually a decent mechanism to use our available cache at least somewhat effectively.
Checklist
main