Better separation of jupyterlite-sphinx's dependencies and that of the JupyterLite environment

[agriyakhetarpal]

This issue aims to track the problem of jupyterlite-sphinx's dependencies also being available for JupyterLite when building – which have the potential to mislead users, such as warnings in downstream projects (#236 (comment)), or Jupytext showing up as fully available/compatible in the JupyterLite distribution at a time when it is not (#236 (comment)), as pointed out by @jtpio.

A transcript of the conversation, which describes a solution, has been added below for easier record-keeping :

---

Ah, I understand; yes, it would be nice not to inject jupytext into the JupyterLite environment and not dilute any extensions. I did suggest keeping it as an optional dependency in #191 (comment) under a flag, but no one talked about it in the code review, so I suppose it was not addressed.

It makes me think that we are constructing JupyterLite instances with shared dependencies (such as from a project's [docs] extras), which also creates the possibility of dependency conflicts that are not exposed to the user, besides such corruptions in the JupyterLite environment that you shared. As a solution, how would you feel about "separating" the Sphinx docs builds and the JupyterLite builds better – say, through constructing a virtual environment with virtualenv and activating it, installing JupyterLite and any extensions into that environment separately, and then discarding said environment at the end of the Sphinx build when we have all the files? This way, jupytext would be retained for converting notebooks from Markdown to IPyNB, but wouldn't have a role with JupyterLite, as it would not exist when the static files are built – further limiting its usage.

We could then have configuration options to control the version of JupyterLite, and other dependencies (such as juptyerlite-pyodide-kernel, etc.) including their versions from conf.py itself:

jupyterlite_version = "0.5.0"

# or maybe construct a conda env and add "jupyterlite_conda_dependencies" as well?
jupyterlite_pypi_dependencies = [
"jupyterlab_materialdarker>=0.6.0",
"jupytext==1.14.0"
]

# maybe? or is the recommended way to install nowadays just "pip"?
jupyterlite_npm_dependencies = [
"base16-gruvbox-dark"
]

It might look like a bit much, but could be more correct in the long run. Still, it might be possible to find a middle ground in terms of the complexity.

Originally posted by @agriyakhetarpal in #236 (comment)