Investigate the use of shiki as the library for syntax highlighting

[Nova38]

Proposal

Use shiki, https://shiki.style/, as the syntax highlighting library for mystmd. It uses the same syntax highlighting engine as vscode and has very good tooling.

• It uses TextMate grammars for its language definitions. Which is the same definitions that are used by vscode and GitHub's linquist among others.
  • The output it produces is very good.
  • there is a large number of high-quality language definitions already defined.
  • It also allows reusing vscode color themes.
• It is supports being run in a compilation step for static generation and at runtime in the end users browser.
• It has a extensive API for defining transformations and decorations

Additional notes

It is used in a lot of open source projects

• Astro
• Nuxt
• vitepress

[agoose77]

I recall seeing this mentioned somewhere ... turns out it was you! https://discord.com/channels/1083088970059096114/1083088970059096117/1295937103846314004

[Nova38]

Yep! I finally have some free time to work on some contributions to a few projects, so I wanted to open the floor to see if this is something that would be desired.

I recall seeing this mentioned somewhere ... turns out it was you! https://discord.com/channels/1083088970059096114/1083088970059096117/1295937103846314004

I also forgot to add that it is really easy to add custom languages to shiki as the consumer of the library. I think that might be useful if someone wanted to have syntax highlight for a custom languages (such as if it was a book about language design/compilers and they made up a simple language) (See https://shiki.style/guide/load-lang)

[choldgraf]

Thanks for this idea @Nova38 ! I wanted to share a few quick questions to help us triage this idea a bit. I promise they're not coming from a place of skepticism - just trying to understand cost/benefit!

• What's the benefit of using this over the current system we use for syntax highlighting? (in functionality, maintenance cost, long-term outlook, UX, etc?)
• How much work would it be to implement and then maintain this?
• Why should this be the most important thing to use maintainer time implementing/reviewing/etc?

[Nova38]

No worries, I completely understand. Those are excellent questions! I will work on writing up those and should have them in a short bit.

To aid the comparison is there a specific package that handles the code directive/syntax highlighting directive? I used the GitHub search to find where https://github.com/react-syntax-highlighter/react-syntax-highlighter was being used, but I am unfamiliar with the codebase and want to make sure I am comparing it to the correct spot. If you know of the top of your head could you point me in the right direction please? If not don't worry about it I'll just keep going through it.

Thanks!

[rowanc1]

The syntax highlighter is used in the myst-theme package here:

https://github.com/jupyter-book/myst-theme/blob/main/packages/myst-to-react/src/code.tsx#L106

[Nova38]

Here are some of my initial thoughts

• What's the benefit of using this over the current system we use for syntax highlighting? (in functionality, maintenance cost, long-term outlook, UX, etc?)

  • An example of what is possible, here is it intigrated with typescripts Shiki Twoslash: Static Code Samples for JS Projects - https://shikijs.github.io/twoslash/

  • The UX of adding new transfomations is a lot more consise

  • Functionality:

    • Uses .tmlanguage definitions, which has more languages that prism/highlight and its definitions are better.
    • Allows access to preform transforms using hast https://github.com/syntax-tree/hast
    • It provides a cleaner api than the current one for adding decorations Decorations | Shiki - https://shiki.style/guide/decorations

    import { codeToHtml } from 'shiki'
    const html = await codeToHtml(code, {
    theme: 'vitesse-light',
    lang: 'ts',
    decorations: [ 
    {
      // line and character are 0-indexed
      start: { line: 1, character: 0 },
      end: { line: 1, character: 11 },
      properties: { class: 'highlighted-word' }
    }
    ]

  • It has some built in transformations that could improve code readability such as the @shikijs/colorized-brackets | Shiki - https://shiki.style/packages/colorized-brackets and Shiki Magic Move Demo - https://shiki-magic-move.netlify.app/ Usng

  • Longterm outlook

    • Language definitions are likely to keep being developed that are compatible as it is used by a lot of tools
    • Built on top of some of the libraries used by vscode (oniguruma)
    • Integrated in to multiple large projects/frameworks (Nuxt, Astro, vitepress, etc... )'
    • They have a rough roadmap Future | Shiki - https://shiki.style/guide/future
    • Some of the core developers also are core developers on things like vitest, nuxt, etc...

  • Maintenance cost

    • Shiki has a diffrent way of parsing what they call as 'meta' strings for customizing the front ends.
      • @shikijs/transformers | Shiki - https://shiki.style/packages/transformers
      • // [!code highlight]
      • We could still use the css and other parts of their transform and just add the classes in the transform that we get from the myst directive props.
      • I am pretty sure that any of these have to be manually enabled so we don't want them to work it should be simple to just not add them

• How much work would it be to implement and then maintain this?

  • Honestly I am not sure how long it would take to implement. It doesn't look like it would take much to add it. To maintain it after the initial setup would probably be minimal. I can make a POC to see what the code will look like. I Think that it might be less work maintaining the current code of the code highlighter.

• Why should this be the most important thing to use maintainer time implementing/reviewing/etc?

  • It could be used to further expand what mystmd can do with code annotations

[Nova38]

Is there a way to override an existing component and inject different project specific component?

[stevejpurves]

@Nova38 no not on the renderer/theme/react side of things, this is part of the plugin framework that has yet to be fleshed out.

+1 for shikijs being pure ESM though. We've had to wrangle with react-syntax-highlighter at times due to the way it is packaged, trying to be a cjs/esm distribution.

[Nova38]

I am not sure that I entirely follow the question. Are you wanting how it by itself can be adapted into a plugin itself, how it could fit in by themselves exposing parts of it that other plugins could then be able to expand on it and add new features to the code blocks, or is it something else?

[Nova38]

Do you mean about how it allow things like decorations to code blocks be added by plugins. If so then there is actually a really nice api that could definitely be hooked into by the plugin system. They have some documentation about it here https://shiki.style/guide/decorations. I have also included an example they have of using that API directly.

import { codeToHtml } from 'shiki'

const code = `
const x = 10
console.log(x)
`.trim()

const html = await codeToHtml(code, {
  theme: 'vitesse-light',
  lang: 'ts',
  decorations: [ 
    {
      // line and character are 0-indexed
      start: { line: 1, character: 0 },
      end: { line: 1, character: 11 },
      properties: { class: 'highlighted-word' }
    }
  ]
})

And here is the rendered result:

A good example of using what this can allow can be seen in the first party twoslash plugin. https://shiki.style/packages/twoslash

The integrating it into the plugins system could be done by allowing plugins could generate some of the objects passed into the decoration array or by allowing the definition of something like their transformation API (https://shiki.style/packages/transformers)

@Nova38 no not on the renderer/theme/react side of things, this is part of the plugin framework that has yet to be fleshed out.

+1 for shikijs being pure ESM though. We've had to wrangle with react-syntax-highlighter at times due to the way it is packaged, trying to be a cjs/esm distribution.

[Nova38]

Also I was wondering, would it be better to include it in the myst-to-html or its own exetention instead on the react myst theme as it can render to html without needing to load the libraries in the browser. That way it will reduce the amount of JavaScript on the end users page

https://github.com/jupyter-book/mystmd/tree/main/packages/myst-to-html

The syntax highlighter is used in the myst-theme package here:

https://github.com/jupyter-book/myst-theme/blob/main/packages/myst-to-react/src/code.tsx#L106

[choldgraf]

Just wanted to note that we have very limited bandwidth on the project right now, so there's a good chance that discussions like this will take a while to play out. We appreciate all of this research and consideration here, there's just too much to do and not much time to work on the project :-)

To your last point, I think that this issue might be relevant to the point you're making:

• Enable and document how to render MyST as part of other applications or as a standalone component #1698