Update hatch_build.py #302
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When we build PROD images in CI, we need constraints, to know which version of the depencies to use. Those constraints are build by earlier steps in the build and uploaded as artifacts so that they can be re-used by PROD image build. However depending on the type of PROD build (in main or in v2-*) different constraints are used: for main build we use source constraints because we are installing provider packages build from main sources, in v2* builds we are using PyPI constraints, because we are installing provider packages from PyPI. This made the dependency tree a bit complicated because PROD builds would have to wait for constraints generation rather than use quickly prepared source constraints and in apache#38533 we introduced an extra step in PROD build for v2* branches to additionally generate constraint in-the job that builds PROD images. This however has some consequences, such as potentially incosistencies between constraints used during other parts of the build, because PROD images can be potentially built much later than the generated constraints for the rest of the build. Also it introduced the need for the PROD builds to pull CI images to generate the constraints which adds an overhead in the process and makes PROD builds in v2 branches few minutes longer than they could be. This PR approaches it differently. Since now with `uv` constraints generation is way faster (few minutes, we could easily make PROD builds use always the same constraint artifacts generated in single job and pay a little delay penalty of starting the PROD builds but get it back at PROD build time (because constraints will be generated only once). Also that removes (back) the need of PROD builds to pull the CI images and removes the need to generate source constraints at CI image build time - which will decrease the build time a little. This PR makes necessary changes to make it happens: * removes source constraints generation in CI images * extracting generate-constraints job to be separate workflow and merging "docs" instead (we are hitting the limit of 20 workflows) * add dependency from all PROD build to wait for generate-constraints * remove CI image pulling and related steps from PROD image build * switches PROD to download and use the full set of constraints generated by `generate-constraints` job
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.