mistletoe is a Markdown parser in pure Python, designed to be fast, modular and fully customizable.
mistletoe is not simply a Markdown-to-HTML transpiler. It is designed, from the start, to parse Markdown into an abstract syntax tree. You can swap out renderers for different output formats, without touching any of the core components.
Remember to spell mistletoe in lowercase!
-
Fast: mistletoe is as fast as the fastest implementation currently available: that is, over 4 times faster than Python-Markdown, and much faster than Python-Markdown2.
See the performance section for details.
-
Modular: mistletoe is designed with modularity in mind. Its initial goal is to provide a clear and easy API to extend upon.
-
Customizable: as of now, mistletoe can render Markdown documents to LaTeX, HTML and an abstract syntax tree out of the box. Writing a new renderer for mistletoe is a relatively trivial task.
mistletoe requires Python 3.3 and above, including Python 3.7, the current development branch. It is also tested on PyPy 5.8.0. Install mistletoe with pip:
pip3 install mistletoeAlternatively, clone the repo:
git clone https://github.com/miyuchina/mistletoe.git
cd mistletoe
pip3 install -e .See the contributing doc for how to contribute to mistletoe.
Here's how you can use mistletoe in a Python script:
import mistletoe
with open('foo.md', 'r') as fin:
rendered = mistletoe.markdown(fin)mistletoe.markdown() uses mistletoe's default settings: allowing HTML mixins
and rendering to HTML. The function also accepts an additional argument
renderer. To produce LaTeX output:
import mistletoe
from mistletoe.latex_renderer import LaTeXRenderer
with open('foo.md', 'r') as fin:
rendered = mistletoe.markdown(fin, LaTeXRenderer)Finally, here's how you would manually specify extra tokens and a renderer
for mistletoe. In the following example, we use HTMLRenderer to render
the AST, which adds HTMLBlock and HTMLSpan to the normal parsing
process.
from mistletoe import Document, HTMLRenderer
with open('foo.md', 'r') as fin:
with HTMLRenderer() as renderer:
rendered = renderer.render(Document(fin))pip installation enables mistletoe's commandline utility. Type the following directly into your shell:
mistletoe foo.mdThis will transpile foo.md into HTML, and dump the output to stdout. To save
the HTML, direct the output into a file:
mistletoe foo.md > out.htmlYou can pass in custom renderers by including the full path to your renderer
class after a -r or --renderer flag:
mistletoe foo.md --renderer custom_renderer.CustomRendererRunning mistletoe without specifying a file will land you in interactive
mode. Like Python's REPL, interactive mode allows you to test how your
Markdown will be interpreted by mistletoe:
mistletoe [version 0.5.3] (interactive)
Type Ctrl-D to complete input, or Ctrl-C to exit.
>>> some **bold text**
... and some *italics*
... ^D
<html>
<body>
<p>some <strong>bold text</strong> and some <em>italics</em></p>
</body>
</html>
>>>
The interactive mode also accepts the --renderer flag.
mistletoe is the fastest Markdown parser implementation available in pure Python; that is, on par with mistune. Try the benchmarks yourself by running:
python3 test/benchmark.pyOne of the significant bottlenecks of mistletoe compared to mistune, however, is the function overhead. Because, unlike mistune, mistletoe chooses to split functionality into modules, function lookups can take significantly longer than mistune.
To boost the performance further, it is suggested to use PyPy with mistletoe. Benchmark results show that on PyPy, mistletoe is about twice as fast as mistune:
$ pypy3 test/benchmark.py mistune mistletoe
Test document: test/samples/syntax.md
Test iterations: 1000
Running tests with mistune, mistletoe...
========================================
mistune: 13.524028996936977
mistletoe: 6.477352762129158The above result was achieved on PyPy 5.8.0-beta0, on a 13-inch Retina MacBook Pro (Early 2015).
Here's an example to add GitHub-style wiki links to the parsing process, and provide a renderer for this new token.
GitHub wiki links are span-level tokens, meaning that they reside inline,
and don't really look like chunky paragraphs. To write a new span-level
token, all we need to do is make a subclass of SpanToken:
from mistletoe.span_token import SpanToken
class GithubWiki(SpanToken):
passmistletoe uses regular expressions to search for span-level tokens in the
parsing process. As a refresher, GitHub wiki looks something like this:
[[alternative text | target]]. We define a class variable, pattern,
that stores the compiled regex:
class GithubWiki(SpanToken):
pattern = re.compile(r"\[\[ *(.+?) *\| *(.+?) *\]\]")
def __init__(self, match_obj):
passFor spiritual guidance on regexes, refer to xkcd classics. For an actual representation of this author parsing Markdown with regexes, refer to this brilliant meme by Greg Hendershott.
mistletoe's span-level tokenizer will search for our pattern. When it finds a match, it will pass in the match object as argument into our constructor. We have defined our regex so that the first match group is the alternative text, and the second one is the link target.
Note that alternative text can also contain other span-level tokens. For
example, [[*alt*|link]] is a GitHub link with an Emphasis token as its
child. To parse child tokens, simply pass match_obj to the super
constructor (which assumes children to be in match_obj.group(1)),
and save off all the additional attributes we need:
from mistletoe.span_token import SpanToken
class GithubWiki(SpanToken):
pattern = re.compile(r"\[\[ *(.+?) *\| *(.+?) *\]\]")
def __init__(self, match_obj):
super().__init__(match_obj)
self.target = match_obj.group(2)There you go: a new token in 7 lines of code.
Adding a custom token to the parsing process usually involves a lot
of nasty implementation details. Fortunately, mistletoe takes care
of most of them for you. Simply pass your custom token class to
super().__init__() does the trick:
from mistletoe.html_renderer import HTMLRenderer
class GithubWikiRenderer(HTMLRenderer):
def __init__(self):
super().__init__(GithubWiki)We then only need to tell mistletoe how to render our new token:
def render_github_wiki(self, token):
template = '<a href="{target}">{inner}</a>'
target = token.target
inner = self.render_inner(token)
return template.format(target=target, inner=inner)Cleaning up, we have our new renderer class:
from mistletoe.html_renderer import HTMLRenderer, escape_url
class GithubWikiRenderer(HTMLRenderer):
def __init__(self):
super().__init__(GithubWiki)
def render_github_wiki(self, token):
template = '<a href="{target}">{inner}</a>'
target = escape_url(token.target)
inner = self.render_inner(token)
return template.format(target=target, inner=inner)It is preferred that all mistletoe's renderers be used as context managers. This is to ensure that your custom tokens are cleaned up properly, so that you can parse other Markdown documents with different token types in the same program.
from mistletoe import Document
from contrib.github_wiki import GithubWikiRenderer
with open('foo.md', 'r') as fin:
with GithubWikiRenderer() as renderer:
rendered = renderer.render(Document(fin))For more info, take a look at the base_renderer module in mistletoe.
The docstrings might give you a more granular idea of customizing mistletoe
to your needs.
For me, the question becomes: why not mistune? My original motivation really has nothing to do with starting a competition. Here's a list of reasons I created mistletoe in the first place:
- I am interested in a Markdown-to-LaTeX transpiler in Python.
- I want to write more Python.
- "How hard could it be?"
- "For fun," says David Beazley.
Here's two things mistune inspired mistletoe to do:
- Markdown parsers should be fast, and other parser implementations in Python leaves much to be desired.
- A parser implementation for Markdown does not need to restrict itself to one flavor of Markdown.
Here's two things mistletoe does differently from mistune:
- Per its readme, mistune will always be a single-file script. mistletoe breaks its functionality into modules.
- mistune, as of now, can only render Markdown into HTML. It is relatively trivial to write a new renderer for mistletoe.
- Unlike mistune, mistletoe is pushing for some extent of spec compliance with CommonMark.
The implications of these are quite profound, and there's no definite this-is-better-than-that answer. Mistune is near perfect if one wants what it provides: I have used mistune extensively in the past, and had a great experience. If you want more control, however, give mistletoe a try.