[CI] Reduce the amount taking to run tests in the CI from 5h to 11min #1297
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.
|
The CI is failing for both unit and integraiton tests for Airflow 2.6 because it is taking a long time to try to resolve dependencies. Example: https://github.com/astronomer/astronomer-cosmos/actions/runs/11594745046/job/32330864858 This PR aims to resolve the issue.
…cncf-kubernetes is 2.7
tatiana
force-pushed
the
fix-ci-for-tests.py3.8-2.6
branch
from
c0b17c4
to
830bde5
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #1297 +/- ##
==========================================
+ Coverage 95.73% 95.84% +0.10%
==========================================
Files 67 67
Lines 3967 3973 +6
==========================================
+ Hits 3798 3808 +10
+ Misses 169 165 -4 ☔ View full report in Codecov by Sentry. |
dosubot
bot
added
size:XS
dosubot
bot
added
size:S
pankajkoti
approved these changes
Nov 5, 2024
There was a problem hiding this comment.
These are amazing fixes and great manoeuvring. Certainly this has a massive operational impact & contributors are going to the love the speedy CI now. Kudos to you @tatiana for being so patient with the acting CI, not giving up & taking it to completion 👏🏽
| @@ -2,7 +2,7 @@ name: test | |||
|
|
|||
| on: | |||
| push: # Run on pushes to the default branch | |||
| branches: [main] | |||
| branches: [main, fix-ci-for-tests.py3.8-2.6] | |||
There was a problem hiding this comment.
Suggested change
| branches: [main, fix-ci-for-tests.py3.8-2.6] | |
| branches: [main] |
gentle reminder comment to remove this before merging
dosubot
bot
added
size:L
dosubot
bot
added
the
area:performance
tatiana
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Context
Closes: #1299
Some CI jobs took an outrageous amount of time to run.
One of our CI test pipeline jobs was taking over five hours to run:
https://github.com/astronomer/astronomer-cosmos/actions/runs/11505558596
More recently, this same job started taking over 6 hours to run and started timing out in the CI, making Cosmos' main branch red for both unit and integration tests for Airflow 2.6. The underlying reason is that it took a long time to resolve dependencies. This seems to have happened since October 29, as seen on the commit 84c5fbd to main.
Example: https://github.com/astronomer/astronomer-cosmos/actions/runs/11594745046/job/32330864858
About this change
This PR solves the original issue by changing where and how we install Airflow dependencies, simplifying the previous setup. We manage Airflow test dependencies in the
pre-install-airflow.shfile, not inpyproject.toml, othersh, or the Github action. We are also being strict as we can dependent on the Airflow version. Where possible, we use constraints. Where different providers' dependencies conflict with previous versions of Airflow, we just ensure the expected version remains being used after the installation.Example of a successful run:
https://github.com/astronomer/astronomer-cosmos/actions/runs/11685652312
Bonus
I realised with this change that users who use K8s and want to define different paths for their dbt projects in Airflow and K8s were facing an issue. This problem was evident when running the k8s example DAG. I've fixed the problem as part of this PR.
Follow-up actions
Since this has been taking a long time to solve and our main branch is still red, I commented out two tasks that were failing tests - and I've logged a follow-up issue for us to address this:
#1304