Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

gitlint and gitlint-core v0.19.0 fail to build from sdist #460

Closed
jorisroovers opened this issue Mar 8, 2023 · 13 comments
Closed

gitlint and gitlint-core v0.19.0 fail to build from sdist #460

jorisroovers opened this issue Mar 8, 2023 · 13 comments
Assignees
@jorisroovers
Labels
bug User-facing bugs

Comments

@jorisroovers
Copy link
Owner

jorisroovers commented Mar 8, 2023 •
edited
Loading

This issue was first discovered and discussed in #367.

To reproduce for gitlint-core:

> wget https://files.pythonhosted.org/packages/29/59/c317a4a39657a4a76cd7dc0c9db4147cca57f13ebe11a469d8c6c0f88715/gitlint_core-0.19.0.tar.gz
> tar xvf gitlint_core-0.19.0.tar.gz
> cd gitlint_core-0.19.0/
> hatch build -t wheel                                                                                                                 
[wheel]
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/vscode/.local/share/hatch/env/virtual/gitlint-core/ovOkWHgP/gitlint-core-build/lib/python3.11/site-packages/hatchling/__main__.py", line 6, in <module>
    sys.exit(hatchling())
             ^^^^^^^^^^^
  File "/home/vscode/.local/share/hatch/env/virtual/gitlint-core/ovOkWHgP/gitlint-core-build/lib/python3.11/site-packages/hatchling/cli/__init__.py", line 26, in hatchling
    command(**kwargs)
  File "/home/vscode/.local/share/hatch/env/virtual/gitlint-core/ovOkWHgP/gitlint-core-build/lib/python3.11/site-packages/hatchling/cli/build/__init__.py", line 75, in build_impl
    for artifact in builder.build(
  File "/home/vscode/.local/share/hatch/env/virtual/gitlint-core/ovOkWHgP/gitlint-core-build/lib/python3.11/site-packages/hatchling/builders/plugin/interface.py", line 93, in build
    self.metadata.validate_fields()
  File "/home/vscode/.local/share/hatch/env/virtual/gitlint-core/ovOkWHgP/gitlint-core-build/lib/python3.11/site-packages/hatchling/metadata/core.py", line 243, in validate_fields
    _ = self.version
        ^^^^^^^^^^^^
  File "/home/vscode/.local/share/hatch/env/virtual/gitlint-core/ovOkWHgP/gitlint-core-build/lib/python3.11/site-packages/hatchling/metadata/core.py", line 128, in version
    self._version = self._get_version()
                    ^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/share/hatch/env/virtual/gitlint-core/ovOkWHgP/gitlint-core-build/lib/python3.11/site-packages/hatchling/metadata/core.py", line 222, in _get_version
    core_metadata = self.core
                    ^^^^^^^^^
  File "/home/vscode/.local/share/hatch/env/virtual/gitlint-core/ovOkWHgP/gitlint-core-build/lib/python3.11/site-packages/hatchling/metadata/core.py", line 174, in core
    metadata_hook.update(self.core_raw_metadata)
  File "/home/vscode/.local/share/hatch/env/virtual/gitlint-core/ovOkWHgP/gitlint-core-build/lib/python3.11/site-packages/hatch_vcs/metadata_hook.py", line 45, in update
    urls[key] = formatter.format(url)
                ^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.asdf/installs/python/3.11.0/lib/python3.11/string.py", line 190, in format
    return self.vformat(format_string, args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/share/hatch/env/virtual/gitlint-core/ovOkWHgP/gitlint-core-build/lib/python3.11/site-packages/hatchling/utils/context.py", line 127, in vformat
    result, _ = self._vformat(format_string, args, kwargs, used_args, 10)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.asdf/installs/python/3.11.0/lib/python3.11/string.py", line 247, in _vformat
    result.append(self.format_field(obj, format_spec))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/share/hatch/env/virtual/gitlint-core/ovOkWHgP/gitlint-core-build/lib/python3.11/site-packages/hatchling/utils/context.py", line 145, in format_field
    return self.__formatters[formatter](value, data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/share/hatch/env/virtual/gitlint-core/ovOkWHgP/gitlint-core-build/lib/python3.11/site-packages/hatch_vcs/metadata_hook.py", line 39, in <lambda>
    'commit_hash': lambda *args: vcs_utils.get_commit_hash(self.root),
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/share/hatch/env/virtual/gitlint-core/ovOkWHgP/gitlint-core-build/lib/python3.11/site-packages/hatch_vcs/vcs_utils.py", line 10, in get_commit_hash
    return subprocess.check_output(['git', 'rev-parse', 'HEAD'], cwd=root).decode('utf-8').strip()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.asdf/installs/python/3.11.0/lib/python3.11/subprocess.py", line 465, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.asdf/installs/python/3.11.0/lib/python3.11/subprocess.py", line 569, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['git', 'rev-parse', 'HEAD']' returned non-zero exit status 128

This issue is caused by a bug in hatch-vcs metadata hook. I’ll be opening a bug with hatch-vcs for this soon.

For gitlint-core, it’s this line that’s problematic (thanks for pointing this out @ofek):

gitlint/gitlint-core/pyproject.toml

Line 70 in 014bfae

'Source Commit' = "https://github.com/jorisroovers/gitlint/tree/{commit_hash}/gitlint-core"

The same issue exists for the gitlint package, caused by these 2 lines:

gitlint/pyproject.toml

Lines 48 to 49 in 014bfae

'Source Archive' = "https://github.com/jorisroovers/gitlint/archive/{commit_hash}.zip"
'Source Commit' = "https://github.com/jorisroovers/gitlint/tree/{commit_hash}"

The fix in both cases is to remove these lines from the respective pyproject.toml files.

However, after doing so, I ran into another issue with gitlint-core:

> hatch build -t wheel                                                                                                                   
[wheel]
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/vscode/.local/share/hatch/env/virtual/gitlint-core/ovOkWHgP/gitlint-core-build/lib/python3.11/site-packages/hatchling/__main__.py", line 6, in <module>
    sys.exit(hatchling())
             ^^^^^^^^^^^
  File "/home/vscode/.local/share/hatch/env/virtual/gitlint-core/ovOkWHgP/gitlint-core-build/lib/python3.11/site-packages/hatchling/cli/__init__.py", line 26, in hatchling
    command(**kwargs)
  File "/home/vscode/.local/share/hatch/env/virtual/gitlint-core/ovOkWHgP/gitlint-core-build/lib/python3.11/site-packages/hatchling/cli/build/__init__.py", line 75, in build_impl
    for artifact in builder.build(
  File "/home/vscode/.local/share/hatch/env/virtual/gitlint-core/ovOkWHgP/gitlint-core-build/lib/python3.11/site-packages/hatchling/builders/plugin/interface.py", line 93, in build
    self.metadata.validate_fields()
  File "/home/vscode/.local/share/hatch/env/virtual/gitlint-core/ovOkWHgP/gitlint-core-build/lib/python3.11/site-packages/hatchling/metadata/core.py", line 244, in validate_fields
    self.core.validate_fields()
  File "/home/vscode/.local/share/hatch/env/virtual/gitlint-core/ovOkWHgP/gitlint-core-build/lib/python3.11/site-packages/hatchling/metadata/core.py", line 1326, in validate_fields
    getattr(self, attribute)
  File "/home/vscode/.local/share/hatch/env/virtual/gitlint-core/ovOkWHgP/gitlint-core-build/lib/python3.11/site-packages/hatchling/metadata/core.py", line 509, in readme
    raise OSError(message)
OSError: Readme file does not exist: README.md

This is caused by the fact that in the source code, gitlint-core/README.md is a symlink to gitlint’s main README.md file (same for the gitlint-core/LICENSE file).

When creating the gitlint-core sdist, hatch includes the gitlint-core/README.md as a symlink file and not as the actual README.md the symlink is pointing to. When then running hatch build -t wheel on the sdist, hatch errors out trying to read the contents of the gitlint-core/README.md symlink file (now pointing nowhere).

This is easily resolved by replacing the symlink by the actual README.md file in gitlint-core in the source code.

I’ll be submitting a fix for this shortly.

CC: @ofek, @harens
@jorisroovers jorisroovers self-assigned this Mar 8, 2023
@jorisroovers jorisroovers added the bug User-facing bugs label Mar 8, 2023
@jorisroovers jorisroovers mentioned this issue Mar 8, 2023
Merged
jorisroovers added a commit that referenced this issue Mar 8, 2023
@jorisroovers
Remove hatch-vcs metadata hook that uses commit_hash
a2d1663 
Temporarily comment out dynamic URLs using the `commit_hash` variable in the
tool.hatch.metadata.hooks.vcs.urls section in pyproject.toml.

Required to fix builds from sdist.

Relates to #460
@jorisroovers jorisroovers mentioned this issue Mar 8, 2023
Closed
jorisroovers added a commit that referenced this issue Mar 8, 2023
@jorisroovers
Remove hatch-vcs metadata hook that uses commit_hash (#461)
1f8a16b 
Temporarily comment out dynamic URLs using the `commit_hash` variable in the
tool.hatch.metadata.hooks.vcs.urls section in pyproject.toml.

Required to fix builds from sdist.

Relates to #460
@jorisroovers
Copy link
Owner Author

@harens This should be fixed now - I just tried a hatch build -t wheel on the sdist of the latest dev builds and it worked.

Can you confirm? Latest dev builds:
https://pypi.org/project/gitlint/0.19.1.dev2/
https://pypi.org/project/gitlint-core/0.19.1.dev2/

@webknjaz
Copy link
Contributor

webknjaz commented Mar 9, 2023

I was just thinking about this: https://github.com/jorisroovers/gitlint/pull/418/files#r1131062741

@jorisroovers TL;DR to have a proper smoke-test, switch to invoking python -m build (without CLI args! it's important). It'll still use hatch's PEP 517 backend. This would build an sdist from Git checkout, followed by building a wheel from said sdist rather than from Git.
Additionally, you can make use of https://github.com/re-actors/checkout-python-sdist to run testing from sdist and not Git. This takes smoke-testing availability for the downstream packaging since they often use sdist as the source of truth.

@sigmavirus24
Copy link
Collaborator

@webknjaz would you be up for sending a PR to use that?

@webknjaz
Copy link
Contributor

webknjaz commented Mar 9, 2023

Which part? using pypa/build? or testing from sdist? The latter would probably require some agreement and restructuring, while the former is straightforward.

@ofek
Copy link

ofek commented Mar 9, 2023

There is also this which I've heard good things about (and of course it's Hynek) https://github.com/hynek/build-and-inspect-python-package

webknjaz added a commit to webknjaz/gitlint that referenced this issue Mar 9, 2023
@webknjaz
🧪🔧 Switch to using pypa/build in CI
8207faa 
This patch switches the CI/CD workflows to use the PyPA-endorsed
PEP 517 build front-end called `build`. It is called without
`--sdist` and `--wheel` CLI options, in which case `build` creates
an sdist from the Git checkout but the following wheel build is made
from that sdist tarball as opposed to using the Git checkout.

This has a side effect of smoke-testing that it's possible to build
wheels from the published sdists. And this is the flow that
`pip install` follows when it's requested to install from an sdist.

Ref: jorisroovers#460 (comment)
@webknjaz webknjaz mentioned this issue Mar 9, 2023
Merged
@webknjaz
Copy link
Contributor

webknjaz commented Mar 9, 2023 •
edited by jorisroovers
Loading

@ofek yes, I know about that action, but don't use it myself — it's too coupled for my taste.

FWIW I've made the small version of the PR: #463. https://github.com/re-actors/checkout-python-sdist can always be added later.

@harens
Copy link

harens commented Mar 9, 2023

@harens This should be fixed now - I just tried a hatch build -t wheel on the sdist of the latest dev builds and it worked.

Can you confirm? Latest dev builds: https://pypi.org/project/gitlint/0.19.1.dev2/ https://pypi.org/project/gitlint-core/0.19.1.dev2/

Seems to work fine, so I'll update the MacPorts portfile in the next stable release. Thanks for fixing @jorisroovers 👍

@jorisroovers
Copy link
Owner Author

Seems to work fine, so I'll update the MacPorts portfile in the next stable release. Thanks for fixing @jorisroovers 👍

Great. I'll try to release 0.19.1 with this fix later today so we can get the homebrew package out for 0.19.x and then look at the other suggestions afterwards.

@webknjaz
Copy link
Contributor

webknjaz commented Mar 9, 2023

@jorisroovers I recommend releasing after merging #463 so that the fix is validated in the CI, not just locally.

@jorisroovers
Copy link
Owner Author

@jorisroovers I recommend releasing after merging #463 so that the fix is validated in the CI, not just locally.

Ack. I'll do it tomorrow morning then (CET), rather than trying to rush it out tonight and inevitably make a mistake.

jorisroovers pushed a commit that referenced this issue Mar 10, 2023
@webknjaz
🧪🔧 Switch to using pypa/build in CI (#463)
61a69ea 
This patch switches the CI/CD workflows to use the PyPA-endorsed
PEP 517 build front-end called `build`. It is called without
`--sdist` and `--wheel` CLI options, in which case `build` creates
an sdist from the Git checkout but the following wheel build is made
from that sdist tarball as opposed to using the Git checkout.

This has a side effect of smoke-testing that it's possible to build
wheels from the published sdists. And this is the flow that
`pip install` follows when it's requested to install from an sdist.

Ref: #460 (comment)
@jorisroovers jorisroovers mentioned this issue Mar 10, 2023
Closed
jorisroovers added a commit that referenced this issue Mar 10, 2023
@jorisroovers
0.19.1 release
c1dff87 
- Fix issue that prevented homebrew packages from being built (#460)
- Switch to using pypa/build in CI (#463)

Full Release details in CHANGELOG.md.
@jorisroovers jorisroovers mentioned this issue Mar 10, 2023
Merged
jorisroovers added a commit that referenced this issue Mar 10, 2023
@jorisroovers
0.19.1 release (#469)
acc9d9d 
- Fix issue that prevented homebrew packages from being built (#460)
- Switch to using pypa/build in CI (#463)

Full Release details in CHANGELOG.md.
@jorisroovers
Copy link
Owner Author

@harens just published 0.19.1! Can you retry?

Thanks!

@harens
Copy link

harens commented Mar 10, 2023

Already done :)

macports/macports-ports@3243c9c

Homebrew/homebrew-core#125315

@jorisroovers
Copy link
Owner Author

Awesome, thanks!

I created separate issues for everything else that was suggested here - thanks everyone. Closing this one!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jorisroovers jorisroovers

Labels
bug User-facing bugs
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@sigmavirus24 @jorisroovers @webknjaz @ofek @harens