"pip install ." for Jupytext requires nodejs #706

mwouts · 2021-01-10T22:26:57Z

We are working on a pre-commit-hook for Jupytext with @JohnPaton and @Skylion007 at #698.

I was giving a try at this .pre-commit-config.yaml file:

repos:
  - repo: https://github.com/JohnPaton/jupytext
    rev: pre-commit-hooks
    hooks:
    - id: jupytext

in order to test the PR.

Unfortunately, the installation of Jupytext fails with this error:

E                   Building wheels for collected packages: jupytext
E                     Building wheel for jupytext (PEP 517): started
E                     Building wheel for jupytext (PEP 517): finished with status 'error'
E                   Failed to build jupytext
E                   
E               stderr:
E                     ERROR: Command errored out with exit status 1:
E                      command: /home/marc/.cache/pre-commit/repoxfmha9gm/py_env-python3.8/bin/python /home/marc/.cache/pre-commit/repoxfmha9gm/py_env-python3.8/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmp8eukjuc8
E                          cwd: /tmp/pip-req-build-4fs4zglz
E                     Complete output (54 lines):
E                     running bdist_wheel
E                     running jsdeps
E                     Installing build dependencies with npm.  This may take a while...
E                     > jlpm install
E                     Traceback (most recent call last):
E                       File "/tmp/pip-build-env-i5vf53rm/overlay/bin/jlpm", line 8, in <module>
E                         sys.exit(main())
E                       File "/tmp/pip-build-env-i5vf53rm/overlay/lib/python3.8/site-packages/jupyterlab/jlpmapp.py", line 43, in main
E                         execvp('node', ['node', YARN_PATH] + argv)
E                       File "/tmp/pip-build-env-i5vf53rm/overlay/lib/python3.8/site-packages/jupyterlab/jlpmapp.py", line 24, in execvp
E                         cmd = which(cmd)
E                       File "/tmp/pip-build-env-i5vf53rm/overlay/lib/python3.8/site-packages/jupyterlab_server/process.py", line 59, in which
E                         raise ValueError(msg)
E                     ValueError: Please install Node.js and npm before continuing installation. You may be able to install Node.js from your package manager, from conda, or directly from the Node.js website (https://nodejs.org).
E                     Traceback (most recent call last):
E                       File "/home/marc/.cache/pre-commit/repoxfmha9gm/py_env-python3.8/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
E                         main()
E                       File "/home/marc/.cache/pre-commit/repoxfmha9gm/py_env-python3.8/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
E                         json_out['return_val'] = hook(**hook_input['kwargs'])
E                       File "/home/marc/.cache/pre-commit/repoxfmha9gm/py_env-python3.8/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 204, in build_wheel
E                         return _build_backend().build_wheel(wheel_directory, config_settings,
E                       File "/tmp/pip-build-env-i5vf53rm/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 216, in build_wheel
E                         return self._build_with_temp_dir(['bdist_wheel'], '.whl',
E                       File "/tmp/pip-build-env-i5vf53rm/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 202, in _build_with_temp_dir
E                         self.run_setup()
E                       File "/tmp/pip-build-env-i5vf53rm/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 145, in run_setup
E                         exec(compile(code, __file__, 'exec'), locals())
E                       File "setup.py", line 72, in <module>
E                         setup(
E                       File "/tmp/pip-build-env-i5vf53rm/overlay/lib/python3.8/site-packages/setuptools/__init__.py", line 153, in setup
E                         return distutils.core.setup(**attrs)
E                       File "/home/marc/miniconda3/envs/notebook_hooks/lib/python3.8/distutils/core.py", line 148, in setup
E                         dist.run_commands()
E                       File "/home/marc/miniconda3/envs/notebook_hooks/lib/python3.8/distutils/dist.py", line 966, in run_commands
E                         self.run_command(cmd)
E                       File "/home/marc/miniconda3/envs/notebook_hooks/lib/python3.8/distutils/dist.py", line 985, in run_command
E                         cmd_obj.run()
E                       File "/tmp/pip-build-env-i5vf53rm/overlay/lib/python3.8/site-packages/jupyter_packaging/setupbase.py", line 503, in run
E                         [self.run_command(cmd) for cmd in cmds]
E                       File "/tmp/pip-build-env-i5vf53rm/overlay/lib/python3.8/site-packages/jupyter_packaging/setupbase.py", line 503, in <listcomp>
E                         [self.run_command(cmd) for cmd in cmds]
E                       File "/home/marc/miniconda3/envs/notebook_hooks/lib/python3.8/distutils/cmd.py", line 313, in run_command
E                         self.distribution.run_command(command)
E                       File "/home/marc/miniconda3/envs/notebook_hooks/lib/python3.8/distutils/dist.py", line 985, in run_command
E                         cmd_obj.run()
E                       File "/tmp/pip-build-env-i5vf53rm/overlay/lib/python3.8/site-packages/jupyter_packaging/setupbase.py", line 274, in run
E                         c.run()
E                       File "/tmp/pip-build-env-i5vf53rm/overlay/lib/python3.8/site-packages/jupyter_packaging/setupbase.py", line 379, in run
E                         run(npm_cmd + ['install'], cwd=node_package)
E                       File "/tmp/pip-build-env-i5vf53rm/overlay/lib/python3.8/site-packages/jupyter_packaging/setupbase.py", line 225, in run
E                         return subprocess.check_call(cmd, **kwargs)
E                       File "/home/marc/miniconda3/envs/notebook_hooks/lib/python3.8/subprocess.py", line 364, in check_call
E                         raise CalledProcessError(retcode, cmd)
E                     subprocess.CalledProcessError: Command '['/tmp/pip-build-env-i5vf53rm/overlay/bin/jlpm', 'install']' returned non-zero exit status 1.
E                     ----------------------------------------
E                     ERROR: Failed building wheel for jupytext
E                   ERROR: Could not build wheels for jupytext which use PEP 517 and cannot be installed directly

Another (simpler) way to reproduce the error is to try pip install . in the jupytext-dev environment.

I also note that the problem disappears when I install nodejs in that environment, however, I don't think we can or should take a dependency on nodejs! Especially in the context of pre-commit hooks, where we want to install only the CLI part of Jupytext.

The text was updated successfully, but these errors were encountered:

JohnPaton · 2021-01-11T10:37:22Z

Do you know where the dependency is coming from? I'm also having the issue after I rebased the latest changes to master but I don't really understand it. Is it because you added jupyter-packaging as a requirement?

mwouts · 2021-01-11T10:54:13Z

Yes, the dependency is coming from jupyter-packaging, which we use to package the (TypeScript) extension for Jupyter Lab. A temp workaround is to comment out that part. Tonight I'll see how to fix that, #705 seems to provides some hint to avoid recompiling that extension too often...

Skylion007 · 2021-01-11T15:40:26Z

Yeah, I can confirm this issue doesn't exist on the only version of JuPyText, I've been using for pre-commit hooks.

JohnPaton · 2021-01-12T11:48:14Z

This issue marked as closed now, but it's not solved, at least for me. Full output from a clean virtualenv:

$ pip install git+ssh://[email protected]/mwouts/jupytext@master
Collecting git+ssh://****@github.com/mwouts/jupytext@master
  Cloning ssh://****@github.com/mwouts/jupytext (to revision master) to /private/var/folders/ch/6nbm4wyn01j4gcjjxgq2r0sr0000gp/T/pip-req-build-a5hiqof3
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting markdown-it-py~=0.6.0
  Using cached markdown_it_py-0.6.1-py3-none-any.whl (81 kB)
Collecting attrs<21,>=19
  Using cached attrs-20.3.0-py2.py3-none-any.whl (49 kB)
Collecting mdit-py-plugins~=0.2.1
  Using cached mdit_py_plugins-0.2.4-py3-none-any.whl (39 kB)
Collecting nbformat>=4.0.0
  Using cached nbformat-5.0.8-py3-none-any.whl (172 kB)
Collecting ipython-genutils
  Using cached ipython_genutils-0.2.0-py2.py3-none-any.whl (26 kB)
Collecting jsonschema!=2.5.0,>=2.4
  Using cached jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)
Requirement already satisfied: setuptools in ./lib/python3.8/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.0.0->jupytext==1.9.1+dev) (51.0.0)
Collecting jupyter-core
  Using cached jupyter_core-4.7.0-py3-none-any.whl (82 kB)
Collecting pyrsistent>=0.14.0
  Using cached pyrsistent-0.17.3-cp38-cp38-macosx_10_15_x86_64.whl
Collecting pyyaml
  Using cached PyYAML-5.3.1-cp38-cp38-macosx_10_15_x86_64.whl
Collecting six>=1.11.0
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting toml
  Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting traitlets>=4.1
  Using cached traitlets-5.0.5-py3-none-any.whl (100 kB)
Building wheels for collected packages: jupytext
  Building wheel for jupytext (PEP 517) ... error
  ERROR: Command errored out with exit status 1:
   command: /Users/cw0504/.virtualenvs/tmp-90cf416c5e0740c/bin/python /Users/cw0504/.virtualenvs/tmp-90cf416c5e0740c/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /var/folders/ch/6nbm4wyn01j4gcjjxgq2r0sr0000gp/T/tmp0wt8k8e5
       cwd: /private/var/folders/ch/6nbm4wyn01j4gcjjxgq2r0sr0000gp/T/pip-req-build-a5hiqof3
  Complete output (51 lines):
  running bdist_wheel
  running jsdeps
  Installing build dependencies with npm.  This may take a while...
  > jlpm install
  Fail to get yarn configuration. dyld: Library not loaded: /usr/local/opt/icu4c/lib/libicui18n.64.dylib
    Referenced from: /usr/local/bin/node
    Reason: image not found

  dyld: Library not loaded: /usr/local/opt/icu4c/lib/libicui18n.64.dylib
    Referenced from: /usr/local/bin/node
    Reason: image not found
  Traceback (most recent call last):
    File "/Users/cw0504/.virtualenvs/tmp-90cf416c5e0740c/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
      main()
    File "/Users/cw0504/.virtualenvs/tmp-90cf416c5e0740c/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/Users/cw0504/.virtualenvs/tmp-90cf416c5e0740c/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 204, in build_wheel
      return _build_backend().build_wheel(wheel_directory, config_settings,
    File "/private/var/folders/ch/6nbm4wyn01j4gcjjxgq2r0sr0000gp/T/pip-build-env-gjimzwy2/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 216, in build_wheel
      return self._build_with_temp_dir(['bdist_wheel'], '.whl',
    File "/private/var/folders/ch/6nbm4wyn01j4gcjjxgq2r0sr0000gp/T/pip-build-env-gjimzwy2/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 202, in _build_with_temp_dir
      self.run_setup()
    File "/private/var/folders/ch/6nbm4wyn01j4gcjjxgq2r0sr0000gp/T/pip-build-env-gjimzwy2/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 145, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 87, in <module>
      setup(
    File "/private/var/folders/ch/6nbm4wyn01j4gcjjxgq2r0sr0000gp/T/pip-build-env-gjimzwy2/overlay/lib/python3.8/site-packages/setuptools/__init__.py", line 153, in setup
      return distutils.core.setup(**attrs)
    File "/Users/cw0504/.pyenv/versions/3.8.3/lib/python3.8/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/Users/cw0504/.pyenv/versions/3.8.3/lib/python3.8/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/Users/cw0504/.pyenv/versions/3.8.3/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/private/var/folders/ch/6nbm4wyn01j4gcjjxgq2r0sr0000gp/T/pip-build-env-gjimzwy2/overlay/lib/python3.8/site-packages/jupyter_packaging/setupbase.py", line 503, in run
      [self.run_command(cmd) for cmd in cmds]
    File "/private/var/folders/ch/6nbm4wyn01j4gcjjxgq2r0sr0000gp/T/pip-build-env-gjimzwy2/overlay/lib/python3.8/site-packages/jupyter_packaging/setupbase.py", line 503, in <listcomp>
      [self.run_command(cmd) for cmd in cmds]
    File "/Users/cw0504/.pyenv/versions/3.8.3/lib/python3.8/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/Users/cw0504/.pyenv/versions/3.8.3/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/private/var/folders/ch/6nbm4wyn01j4gcjjxgq2r0sr0000gp/T/pip-build-env-gjimzwy2/overlay/lib/python3.8/site-packages/jupyter_packaging/setupbase.py", line 274, in run
      c.run()
    File "/private/var/folders/ch/6nbm4wyn01j4gcjjxgq2r0sr0000gp/T/pip-build-env-gjimzwy2/overlay/lib/python3.8/site-packages/jupyter_packaging/setupbase.py", line 379, in run
      run(npm_cmd + ['install'], cwd=node_package)
    File "/private/var/folders/ch/6nbm4wyn01j4gcjjxgq2r0sr0000gp/T/pip-build-env-gjimzwy2/overlay/lib/python3.8/site-packages/jupyter_packaging/setupbase.py", line 225, in run
      return subprocess.check_call(cmd, **kwargs)
    File "/Users/cw0504/.pyenv/versions/3.8.3/lib/python3.8/subprocess.py", line 364, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['/private/var/folders/ch/6nbm4wyn01j4gcjjxgq2r0sr0000gp/T/pip-build-env-gjimzwy2/overlay/bin/jlpm', 'install']' died with <Signals.SIGABRT: 6>.
  ----------------------------------------
  ERROR: Failed building wheel for jupytext
Failed to build jupytext
ERROR: Could not build wheels for jupytext which use PEP 517 and cannot be installed directly

mwouts · 2021-01-12T17:47:12Z

I see... so my attempt to find out whether npm is available was not enough - your example shows a case where npm available but still the typescript packaging fails.

I'll propose another approach based on an environment variable. In addition I expect a significant improvement in terms of speed. Yesterday I measured the following times for pip install -e .:

1 min for the full package, including the JS extensions
17 secs for the Python package when jupyterlab and jupyter-packaging are added to pyproject.toml
5 secs for the Python package when jupyterlab and jupyter-packaging are removed from pyproject.toml.

mwouts · 2021-01-12T19:02:54Z

@JohnPaton , now we'll do the switch based on the environment variables BUILD_JUPYTERLAB_EXTENSION or CI. Could you give it a try and let us know if the issue is over for you as well? Thanks

gunnhildsp · 2024-03-01T07:02:23Z

I am having this issue when trying to install the jupytext pre-commit hook. I am running poetry run pre-commit install and then poetry run pre-commit run --all-files which tries to install the pre-commit env. I assume pre-commit uses the python version from the poetry virtual env, which is 3.10.12. Do you know any workarounds? This is my pre-commit hook config:

---
default_language_version:
  python: python3
repos:
  - repo: https://github.com/mwouts/jupytext
    rev: v1.16.1
    hooks:
      - id: jupytext
        args: [ --sync ]
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.5.0
    hooks:
      - id: check-json
      - id: check-merge-conflict
      - id: check-toml
      - id: check-yaml
      - id: debug-statements
      - id: detect-private-key
      - id: end-of-file-fixer
      - id: name-tests-test
        args: [--pytest-test-first]
      - id: trailing-whitespace
  - repo: https://github.com/astral-sh/ruff-pre-commit
    rev: v0.1.14
    hooks:
      - id: ruff
        args: [--fix, --exit-non-zero-on-fix]
      - id: ruff-format

mwouts · 2024-03-02T22:42:52Z

Hey @gunnhildsp , yes we are aware of this issue in the context of the pre-commit hook, cf. #1210.

The idea would be to set an environment variable HATCH_BUILD_HOOKS_ENABLE=false when installing the hook (that will remove the node dependency), however we have not yet figured out how to do this (if anyone knows how to do this, please comment on the other issue, thanks!)

This was referenced Jan 10, 2021

Enable integration with pre-commit #698

Merged

Use skip_if_exists from jupyter-packaging #705

Closed

mwouts mentioned this issue Jan 11, 2021

Don't build the jupyterlab-jupytext extension if npm is not available #708

Merged

mwouts closed this as completed in #708 Jan 11, 2021

mwouts reopened this Jan 12, 2021

mwouts mentioned this issue Jan 12, 2021

Don't build the jupyterlab-jupytext extension unless BUILD_JUPYTERLAB_EXTENSION (or CI) is True #710

Merged

mwouts closed this as completed in #710 Jan 12, 2021

mwouts mentioned this issue Jan 12, 2021

Use BUILD_JUPYTERLAB_EXTENSION in the PyPi-publish step #711

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"pip install ." for Jupytext requires nodejs #706

"pip install ." for Jupytext requires nodejs #706

mwouts commented Jan 10, 2021

JohnPaton commented Jan 11, 2021 •

edited

Loading

mwouts commented Jan 11, 2021

Skylion007 commented Jan 11, 2021

JohnPaton commented Jan 12, 2021 •

edited

Loading

mwouts commented Jan 12, 2021

mwouts commented Jan 12, 2021

gunnhildsp commented Mar 1, 2024

mwouts commented Mar 2, 2024

"pip install ." for Jupytext requires nodejs #706

"pip install ." for Jupytext requires nodejs #706

Comments

mwouts commented Jan 10, 2021

JohnPaton commented Jan 11, 2021 • edited Loading

mwouts commented Jan 11, 2021

Skylion007 commented Jan 11, 2021

JohnPaton commented Jan 12, 2021 • edited Loading

mwouts commented Jan 12, 2021

mwouts commented Jan 12, 2021

gunnhildsp commented Mar 1, 2024

mwouts commented Mar 2, 2024

JohnPaton commented Jan 11, 2021 •

edited

Loading

JohnPaton commented Jan 12, 2021 •

edited

Loading