Source of truth for MyST specification and relationship to MyST-parser?

[noirbizarre]

Hi 👋🏼

I've been working on a Myst implementation from the beginning of Myst.
I never published it, but I'd like to do it.

When I started, I was parsing the .md files from Myst-parser documentation as a source of trust for the spec.
Since then, there has been multiple new potential sources of trust and each one has different specification and scope:

• the "specifications" section of the mystmd site which doesn't seem to be a parsing specification but an internal DSL or AST specification
• the published spec and test suite on unpkg which doesn't seem to match the mystmd.org documentation
• the Authoring guide section from the site
• The spec repository which docs/example seems to match what is published on unpkg
• The Myst-Parser website and its repository but which seems to have differences with what is documented on mystmd.org

Some of those sources do not include both source and expected rendering, the actual rendering being done on publication by an actual implementation (the TS one for mystmd.org if I understand it well), making it difficult to use it.

Can you explain the relationship between those sources/references and tell me what should be taken as official parsing/rendering specification source?

[noirbizarre]

Given there is another dedicated discussion space, I just found https://github.com/orgs/executablebooks/discussions/1194 which is basically about the same confusion.
I think it needs clarifications between those project and their relationship and maybe a consolidation of both jupiterbooks and executablebooks organization, repositories and discussion spaces (which seems to cover the same scope as executablebooks tagline is "An open collaboration to create executable books with Jupyter" and later on the same page: "JupyterBooks: Build beautiful, publication-quality books and documents from computational material like Jupyter Notebooks.")

[choldgraf]

Thanks @noirbizarre for opening this and for making the cross-link. I wanted to quickly share some context below to help clarify. We're also using the following issue to track overall Jupyter Book v1 to v2 information in case that's helpful:

• 🚨 Jupyter Book 2.0 planning and information: what to expect here jupyter-book#2281

Historical context

Originally, MyST was envisioned as a markdown syntax that exposed the functionality of docutils/Sphinx with Markdown syntax. The myst-parser repository is the project that defined the MyST markdown syntax, and defined a bridge to docutils/Sphinx. It was funded by a Sloan Foundation grant, and developed by the team behind that grant. Here's a blog post announcing Jupyter Book 1.0, which was built on Sphinx. That grant team did most of the work building Jupyter Book 1.0, and maintained the Sphinx-based stack defined at https://executablebooks.org. However, a year ago the project leads decided that a different technical direction would provide a better long-term pathway for the project, and we made a pivot (described next).

Expanding the vision of MyST and incorporating under Jupyter

A year ago, we expanded the goals of MyST to define a document standard rather than just a markdown syntax, and also focused the implementation on a new JavaScript-based engine at https://mystmd.org. We decided the best path forward was to split off this effort from the original grant-funded (and Sphinx-based) technical stack. As part of this effort, we created a new GitHub organization (jupyter-book) and moved the TypeScript-based technologies there.

The Jupyter Enhancement Proposal to incorporate jupyter-book has a lot of good context for rationale behind this, and the pre-proposal issue has good conversation around this.

Moving forward, mystmd.org is the source of truth

Moving forward, the https://mystmd.org stack is the "source of truth", and the myst-parser can be considered an independently-maintained implementation of MyST markdown for Python (a better name might be myst-sphinx, and perhaps we'll get to that renaming eventually but have a lot of other stuff to tackle as well).

But there may discrepancies because we're low on resources

Realistically speaking, you're going to find discrepancies between sources of truth defined on mystmd.org, because the project has little resources to work with and is executing a fairly complex shift in its technical priorities. Here are a few ideas for our targets thought:

• mystmd.org/spec will be the source of truth for the MyST markdown syntax as well as the AST for the document structure. However, it may be lagging behind the primary implementation of MyST markdown (next bullet point).
• mystmd.org/guide is the MyST document engine, a TypeScript-based CLI that builds MyST documents. This is the thing getting the most attention right now, with the goal of bringing it up to feature parity with Jupyter Book 2. As such, it is probably closer to a "reference implementation" of the MyST spec and is currently a better source of truth than what's defined at /spec.
• Moving forward, our goal is to put resources back into making sure the /spec information is aligned with the MyST document engine implementation, however realistically it might be some time because we're very low on resources.

We are interested in guiding others to contribute

We are also interested in growing the developer-base for the MyST document engine and Jupyter Book 2, so if you're interested in helping out with any of the work that needs to happen, we'd certainly be interested in providing guidance to help you or others get started.

I hope this helps explain a little bit about where the project is at right now and where to look for information.

[noirbizarre]

Thanks for the detailed answer and the links. Things are a lot much clearer now.

I totally get the low resources' constraint (I am a single, not funded open-source developer with a family and only night and weekends to contribute and/or maintain).

Now that I have the right precedence of repositories, I will be able to advance and also to contribute on specs (basically contribute back to spec rendering from features that I am implementing from the guide).

If I may, I think that making the guide the real spec by providing the expected example rendering would lower the workload.
Actually, the first thing I did on my project is 2 pytest plugins accepting JSON or MD specs as test suite without any modification as long as examples are provided with the expected rendering (I successfully ingest, CommonMark specs, both json and md, gfm specs and published Myst specifications). I plan to release everything very soon (both plugin are part of the same project).

Anyway, thanks for the answer, thanks for the (very much needed) Myst initiative and talk to you soon as I expect top contribute.

[Mathnerd314]

a better name might be myst-sphinx, and perhaps we'll get to that renaming eventually but have a lot of other stuff to tackle as well

I appreciate the response and understand that the team has many competing priorities. That said, clarity around the project's goals and scope is critical for building confidence in its long-term viability. For instance, renaming the MyST-Parser repository to something like myst-sphinx-legacy and updating the README to remove the statement "This repository serves as the reference implementation of MyST Markdown" could be quick tasks—perhaps taking only about 10 minutes for someone with sufficient access. These small adjustments would help set clearer expectations for users. I also outlined other small steps in my ticket that could collectively take about an hour but would significantly improve the project's organization and communication.

I recognize this may seem less urgent compared to broader technical shifts, but for many users, these details serve as the "source of truth" for the standard. Without clear signals from the project, there is a risk of further fragmentation, as humorously noted in the Standards xkcd. I hope the project can take steps toward greater clarity and coordination to strengthen user trust and engagement. As it is, although I have enjoyed using MyST, the Quarto project looks very attractive.

[choldgraf]

Thanks for providing your perspective @Mathnerd314 - I agree with most of your points and will think about them.

You're right that Quarto is another nice option for similar kinds of workflows. They are pretty different project structures (MyST is maintained by an open community and Quarto by a company with a unified product team / strategy) and they are at very different stages of development (MyST is only a year or two old depending on how you count, and Quarto has been around for several years now), so adjust expectations accordingly (both approaches have pros and cons, so I'm not making any value judgments here).