Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GHA: Publish release to test.pypi.org on demand #418

Merged
merged 6 commits into from
Jan 23, 2023
Merged

GHA: Publish release to test.pypi.org on demand #418

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
65fdd77
GHA: Auto publish release when triggered on main
jorisroovers Jan 23, 2023
4796967
Attemp to fix setting SETUPTOOLS_SCM_PRETEND_VERSION
jorisroovers Jan 23, 2023
9f7dc7d
fixup! Attemp to fix setting SETUPTOOLS_SCM_PRETEND_VERSION
jorisroovers Jan 23, 2023
4ba885c
Comments + attemp to use hatch publish
jorisroovers Jan 23, 2023
2e7f428
Run hatch version instead of hatch env create
jorisroovers Jan 23, 2023
a213a3a
fixup! Run hatch version instead of hatch env create
jorisroovers Jan 23, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 38 additions & 10 deletions .github/workflows/publish-release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,32 +15,60 @@ on:
default: "main"

jobs:
checks:
publish:
runs-on: "ubuntu-latest"
steps:
- name: Setup python
uses: actions/[email protected]
with:
python-version: ${{ matrix.python-version }}
python-version: "3.11"

- name: Install Hatch
run: python -m pip install hatch==1.6.3

- uses: actions/[email protected]
with:
ref: ${{ inputs.repo_release_ref }}
fetch-depth: 0 # checkout all history, needed for hatch versioning

- name: Set SETUPTOOLS_SCM_PRETEND_VERSION
run: echo "SETUPTOOLS_SCM_PRETEND_VERSION=$(hatch version | cut -d+ -f1)" >> $GITHUB_ENV
# Run hatch version once to avoid additional output ("Setting up build environment for missing dependencies")
# during the next step
- name: Hatch version
run: hatch version

- name: Test SETUPTOOLS_SCM_PRETEND_VERSION
run: echo $SETUPTOOLS_SCM_PRETEND_VERSION
# Hatch versioning is based on git (using hatch-vcs). If there is no explicit tag for the commit we're trying to
# publish, hatch versioning strings will have this format: 0.19.0.dev52+g9f7dc7d
# With the string after '+' being the 'g<short-sha>' of the commit.
#
# However, PyPI doesn't allow '+' in version numbers (no PEP440 local versions allowed on PyPI).
# To work around this, we override the version string by setting the SETUPTOOLS_SCM_PRETEND_VERSION env var
# to the version string without the '+' and everything after it.
# We then only actual publish such releases on the main branch to guarantee the dev numbering scheme remains
# unique.
# Note that when a tag *is* present (i.e. v0.19.0), hatch versioning will return the tag name (i.e. 0.19.0)
# and this step has no effect, ie. SETUPTOOLS_SCM_PRETEND_VERSION will be the same as `hatch version`.
- name: Set SETUPTOOLS_SCM_PRETEND_VERSION
run: |
echo "SETUPTOOLS_SCM_PRETEND_VERSION=$(hatch version | cut -d+ -f1)" >> $GITHUB_ENV

- name: Build (gitlint-core)
run: |
hatch build
run: hatch build
working-directory: ./gitlint-core

- name: Build (gitlint)
run: |
hatch build
run: hatch build
jorisroovers marked this conversation as resolved.
Show resolved Hide resolved

- name: Publish (gitlint-core)
run: hatch publish -r test
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't recommend using manual CLI invocations for publishing because you'll be missing out on the seamless GHA integrations.

For example, PyPI has recently implemented OIDC, and it's about to be enabled for private beta testing. It will allow to completely remove the GHA secrets and rely on secure short-lived tokens. Getting those tokens requires a few API requests to two different services that are specific to GHA, meaning the CLI tools will likely not implement this.

But the curated action https://github.com/marketplace/actions/pypi-publish will support it the moment OIDC is fully enabled on PyPI. With that, (Test)PyPI uploads will become zero-configuration for free, almost automatically.

Another point I'd like to make is that it's much more secure to perform publishing from a dedicated job — the secrets shouldn't be available to the build scripts or anything else that ends up being executed from the pre-publish jobs.

I also recommend using the environment: feature of GHA — it allows one to isolate the secrets, so they are not global and available to any job, but are only accessible from the job that has the proper environment set. Environments can also have branch protection — I usually make them require human approval for the job that publishes to PyPI. A real person needs to click a button on GitHub to allow such a job to start.

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, multiple suggestions in here :) I'll definitely consider and probably adopt them.

But the curated action https://github.com/marketplace/actions/pypi-publish will support it the moment OIDC is fully enabled on PyPI. With that, (Test)PyPI uploads will become zero-configuration for free, almost automatically.

Thanks, will check it out!

Another point I'd like to make is that it's much more secure to perform publishing from a dedicated job — the secrets shouldn't be available to the build scripts or anything else that ends up being executed from the pre-publish jobs.

Hmm, I hadn't considered this. Is this because people could be opening a PR that modifies the CI pipeline that runs on every PR and modify it to extract the secrets?

I also recommend using the environment: feature of GHA — it allows one to isolate the secrets, so they are not global and available to any job, but are only accessible from the job that has the proper environment set.

Yup, makes total sense. Facepalm for not considering this, I've used environments before elsewhere.

I usually make them require human approval for the job that publishes to PyPI. A real person needs to click a button on GitHub to allow such a job to start

Yes this makes sense and I've done this before. Would need to check how we can marry that with our auto publishing of dev builds to PyPI on every commit which is already going on.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jorisroovers

Hmm, I hadn't considered this. Is this because people could be opening a PR that modifies the CI pipeline that runs on every PR and modify it to extract the secrets?

TL;DR it is possible to smuggle some something that might access the secrets post-merge.

The secrets aren't available for runs triggered by the pull_request event. But they will be available for other events, post merge. If you use build tooling that has plugins, I imagine it might be possible to silently add a compromised plugin that would be loaded by said build tooling. It may be hard to track down if such third-party software does anything suspicious. And it's almost impossible to catch during reviews (unless you know to do audit of the entire transitive dependency tree being updated by such a change) — because it's expensive, and it's not the first thing most non-security-experienced reviewers would think of.

I've been seeing https://www.stepsecurity.io/products/harden-runner being integrated in many projects lately in an attempt to tighten the possibility of sneaking out sensitive data out of the CI. Maybe you should consider adding it too.

Yup, makes total sense. Facepalm for not considering this, I've used environments before elsewhere.

Yeah, they are not advertised as a match for the publishing use-case which is why it took me a while to come up with this approach. But I've started doing this in all of my projects a year (or maybe more) ago. I usually call the environments pypi and testpypi. Use the same secret name PYPI_API_TOKEN, meaning that I have two separate jobs publishing to different indexes. One environment needs approvals and the other is unconditional. Hopefully, with the OIDC support, the secrets bit will become unnecessary but the use for pre-approval use-case will still stand. I also like that I can render version URLs as environment params in CI, and they will be clickable in the workflow run summary graph. But this requires pre-calculating the expected version vars at the beginning of the workflow which is why I tend to have a whole separate job that runs first dedicated solely to calculating a number of variables that are shared across the jobs and are used for assertions sometimes.

Yes this makes sense and I've done this before. Would need to check how we can marry that with our auto publishing of dev builds to PyPI on every commit which is already going on.

Just a separate environment without protections, and a separate job for that too.

working-directory: ./gitlint-core
env:
HATCH_INDEX_USER: ${{ secrets.TEST_PYPI_GITLINT_CORE_USERNAME }}
jorisroovers marked this conversation as resolved.
Show resolved Hide resolved
HATCH_INDEX_AUTH: ${{ secrets.TEST_PYPI_GITLINT_CORE_PASSWORD }}
if: inputs.pypi_target == 'test.pypi.org' && inputs.repo_release_ref == 'main'

- name: Publish (gitlint)
run: hatch publish -r test
env:
HATCH_INDEX_USER: ${{ secrets.TEST_PYPI_GITLINT_USERNAME }}
HATCH_INDEX_AUTH: ${{ secrets.TEST_PYPI_GITLINT_PASSWORD }}
if: inputs.pypi_target == 'test.pypi.org' && inputs.repo_release_ref == 'main'
4 changes: 2 additions & 2 deletions .github/workflows/test-release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
name: Test release
run-name: "Test release (${{ inputs.gitlint_version }}, pypi_source=${{ inputs.pypi_source }}, repo_test_ref=${{ inputs.repo_test_ref }})"
name: Test Release
run-name: "Test Release (${{ inputs.gitlint_version }}, pypi_source=${{ inputs.pypi_source }}, repo_test_ref=${{ inputs.repo_test_ref }})"
on:
workflow_dispatch:
inputs:
Expand Down