Ideas for new IDE for Data Science

[pplonski]

I'm pasting here some parts of my discussion with @abalter (cited text) about the new IDE for Data Science.

I do strongly believe in not reinventing the wheel. There is one license where you can use a FOSS for closed projects with certain stipulations. I can't remember which one. But maybe theia is ok with that.

I've read about Theia. The project was started by GitPod.io company because they need a web browser IDE similar to VS Code. They started work on Theia in 2017. Theia was used in their service (online IDE with CI/CD). In April 2021, they dropped Theia and replace it with VS Code. So Theia won't have support in the long term. (There is still some development, but there are rather problems with VS Code extensions integrations).

Jupyter: Serialized is impossible to human read. It contains both the code AND information for how to present it. It also contains the results. Markdown comes in multiple cells. Multiple languages handled by "magics" which only exist in python notebooks. An R notebook can't run python code. However, does maintain environment between code blocks of a given language---they run in the same kernel.

R Notebook: Serialized is basically markdown, and completely agnostic to presentation which can be anything pandoc translates to. Has a poor UI presentation in code form, and is not editable in html_notebook form. Does not record results. Is language agnostic. Chunks can be in virtually any language. However, only R chunks maintain their environment.
I think the mission is to combine the pluses and eliminate the minuses. I have many ideas. I think everything is actually very simple except the issue of serializing the results. However, I'm pretty sure I know how to handle that as well.

Two additional notes:

1. Are you familiar with jupytext? This allows you to open Rmarkdown notebooks in Jupytelab as "Notebooks" (notebook UI) and run them as-is. It is actually two-way: You can maintain synchronized .ipynb and .Rmd files. Therefore, you can (in theory---it "mostly" works) have an .ipynb and download it as a .Rmd to run in RStudio.

I heard about them but never tried (will try tomorrow). Similar package https://github.com/gpoore/codebraid - they claim that multiple languages can be used in one document.

2. Are you familiar with R Reticulate? This is a package that gives you access to conda environments within an R Notebook and maintains a python environment. You can access variables between the environments cleanly. I would love to test how this plays in a Jupyter notebook.

Essentially, I think between those two you essentially have the ideas needed for a language-agnositc notebook.

So there is a lot to think about here. I'll take a little time and sketch this out in a more readable way.

To summarize, I have a strong feeling that a lot can be done in this space. I would like to wisely select the silver bullet.

I'm going to take a look at your mockup. Just a favor--could you make conda packages rather than pip? It's infinitely better, and becoming more and more the industry standard.

From my initial tests, conda works better between different OS (works really well on Windows), but I mostly use pip (I'm on Ubuntu).

[pplonski]

After many checks, I think the most flexible environment is JupyterLab. It has already many extensions available. Everything can be improved/changed with extension.

I've found a nice example of IDE on top of JupyterLab https://github.com/elyra-ai/elyra - it has many extensions (autocomplete, linting, git-integration, visual pipeline editor, scheduling, hybrid runtime, Python and R support). Looks very nice. It is a web-based IDE.

There is also a possibility to create a desktop app from JupyterLab, https://github.com/jupyterlab/jupyterlab-desktop