Curious: why make something Typora specific?

[vassudanagunta]

The whole point of Markdown is to be generic. Writing something for Typora specifically seems to go against the grain.

Why not make a general purpose tool that works with CommonMark (Typora purports to be CommonMark compliant, at least as its goal), and then add those extensions that Typora adds (e.g. Mermaid diagrams embedded in code blocks)?

Also, did you implement your own Markdown parser from scratch? That seems odd. Why not use one of the many existing parsers out there? If your implementation has some unique features, you should publish it as a library!

[PegasisForever]

Thanks for asking ;)

Why not make a general purpose tool that works with CommonMark, and then add those extensions that Typora adds?

I actually looked into some existing parsers to see if I can build typora parser on top of it. It turns out they don't provide enough expansibility for typora-flavoured markdown. Use marked.js as an example:

• Typora allows consecutive empty paragraphs (the spec disallows it). Marked's extension system is not low-level enough for me to customise this (and it should not be).
• Table of content is a similar story, only top-level headers are shown in the ToC. I can't get this information using marked's extension system.
• Also how typora deals with repeated header id is different than marked, I have no way to customise it using extension either.
• And a lot more

There's just a lot of small things I can't do with existing frameworks.

I also looked into forking marked.js and adding these features, however it's basically 3000 lines of undocumented & extremely complex code. It'll probably take more time for me to understand its code than writing a parser on my own.

Also, did you implement your own Markdown parser from scratch?

This is my first time writing a proper parser, I thought it would be challenging and fun :P (turns out it frys my brain) (also performance is not great)

[vassudanagunta]

understood.

(turns out it frys my brain) (also performance is not great)

Have you tested your parser/renderer against the CommonMark spec? All CommonMark parsers pass these tests. It's not trivial to do so. You can borrow the spec.txt test parser/runner code from CommonMark.js or from Markdown-it or others.

If your parser fails on more than a small number of tests, I'd make that clear in the README, especially as you've given it a non-alpha version number of 1.0+

[PegasisForever]

When i'm coding the parser I do use the spec as a guide, however the parser is not planned to be 100% compliant to the spec:

1. Typora itself does not follow the spec, especially on how it parses emphasises & auto url detection.
2. For typora-specific syntaxes, there's no spec at all.
3. The time spend on following the spec is just not worth the outcome imo.

However my parser do have some problems in handling special characters, this will be addressed in future versions. And yes I should mention in the readme this parser is not spec compliant.

[vassudanagunta]

i get it. makes sense.

but FYI, @abnerlee promised that Typora would be CommonMark and GFM compatible, so you should submit an issue when you come across discrepancies. Typora's is a superset of GFM, just as GFM is a superset of CommonMark. For example, support for diagrams is all through fenced code blocks, so it completely in line with CommonMark; It's just that Typora has a custom rendering for fenced code blocks with certain info strings, which is still consistent with CommonMark's spec, just as syntax highlighting code blocks is.

Have you looked at markdown-it? It's highly extendible.

I believe Typora is built on top of CodeMirror. You might want to look at this thread, in particular this comment.

The reason I'm interested in this stuff is because I'm writing a "universal" structured plain text parser/renderer, where you can define the syntax declaratively (i.e. with no code) with what I call Plain Text Stylesheets (ptss). For example, I have a CommonMark.ptss (stylesheet) that my parser fully supports and passes all CommonMark tests. And since it's just a "stylesheet", you can extend it easily, e.g. define a syntax for admonitions in a few lines of declaration. If I were only supporting CommonMark I'd release it already, but I'm going much further.

If you're interested in being updated, star/watch this repo (I just now made it as a placeholder) 😉

[PegasisForever]

By typora-specific syntaxes, I mean the emoji (: smile :), the math block ($$\nlatex\n$$), and the front matter block. Regarding on the diagram support, the main problem is these diagram libraries only run in the browser, so I need to fork each of them to make them work in node.

Have you looked at markdown-it?

Yes I did, unfortunately not to the task.

I believe Typora is built on top of CodeMirror.

I think only the fenced code part is built on top of code mirror. However code mirror doesn't work in node at all, I had to use highlight.js to replace it.

in particular this comment

Wow I didn't know this library before, I feel like I might be reinventing the wheels...(still good learning experience though)

---

Your ptss idea sounds very cool to me, however I don't have a lot of experience writing parsers so don't expect any constructive opinions from me :P. I feel like there are already a lot of parser generators out there in the public, can they do the same job?

[vassudanagunta]

Plain Text Stylesheets (PTSS) isn't a parser generator. It's a spec for stylesheets that define structured plain text presentation styles such as Markdown, analogous to CSS which is a spec for stylesheets that define graphical layout and typographical presentation styles. But it is applied in the opposite direction. With CSS, the source code (HTML) is rendered to the graphical presentation style it describes. With PTSS, the plain text presentation style it describes is unrendered to an abstract form, which becomes the source that can then be re-rendered into any other form.

"What? Markdown is not a structured plain text presentation style, it's source code! It doesn't need to be un-rendered", you might say. But if you think about it, the key to Markdown's success is that it is designed for reading as-is. In other words, a presentation, that happens to be plain text. When we render Markdown to HTML or some other format, you can look at it as un-rendering it from one presentation style so it can be rendered into another.

I've written a reference implementation PTSS un-renderer -- which is a parser because that's what un-rendering plaintext involves (un-rendering the graphical output of CSS would need OCR and more). So instead of a world where we have not only separate Markdown, RST and AsciiDoc parsers, but also a multitude of incompatible Markdown variant parsers, imagine one with a spec for universal human-oriented structured text parsers¹. To extend a format you don't write a new parser, you just change the stylesheet definition. There are even more benefits, such as enabling an ecosystem for plain text the way LLVM enables an ecosystem for programming languages, rather than the very siloed world we have today.

For instance, you or someone would create a Typora.ptss describing Typora's styles. That would enable parsing any Typora-Mardown docs into the abstract form. That abstract form (a tree) would have nodes for math expressions, labelled with a kind (e.g. Mathjax). You would configure the system to call the Mathjax rendering library of your choice to handle those nodes on rendering.

Writing you about this gives me practice in trying to explain something that is hard because it is a massive paradigm shift.

---

Footnotes

1. universal parser sounds impossible, which is probably why you thought "parser generator". But when you limit it to "human-oriented" plain text, you find out that the design space is pretty constrained, and that there are only so many kinds of patterns to recognize. You just have to generalize them. ↩