Home

Awesome

goldmark

https://pkg.go.dev/github.com/yuin/goldmark https://github.com/yuin/goldmark/actions?query=workflow:test https://coveralls.io/github/yuin/goldmark https://goreportcard.com/report/github.com/yuin/goldmark

A Markdown parser written in Go. Easy to extend, standards-compliant, well-structured.

goldmark is compliant with CommonMark 0.31.2.

Motivation

I needed a Markdown parser for Go that satisfies the following requirements:

golang-commonmark may be a good choice, but it seems to be a copy of markdown-it.

blackfriday.v2 is a fast and widely-used implementation, but is not CommonMark-compliant and cannot be extended from outside of the package, since its AST uses structs instead of interfaces.

Furthermore, its behavior differs from other implementations in some cases, especially regarding lists: Deep nested lists don't output correctly #329, List block cannot have a second line #244, etc.

This behavior sometimes causes problems. If you migrate your Markdown text from GitHub to blackfriday-based wikis, many lists will immediately be broken.

As mentioned above, CommonMark is complicated and hard to implement, so Markdown parsers based on CommonMark are few and far between.

Features

Installation

$ go get github.com/yuin/goldmark

Usage

Import packages:

import (
    "bytes"
    "github.com/yuin/goldmark"
)

Convert Markdown documents with the CommonMark-compliant mode:

var buf bytes.Buffer
if err := goldmark.Convert(source, &buf); err != nil {
  panic(err)
}

With options

var buf bytes.Buffer
if err := goldmark.Convert(source, &buf, parser.WithContext(ctx)); err != nil {
  panic(err)
}
Functional optionTypeDescription
parser.WithContextA parser.ContextContext for the parsing phase.

Context options

Functional optionTypeDescription
parser.WithIDsA parser.IDsIDs allows you to change logics that are related to element id(ex: Auto heading id generation).

Custom parser and renderer

import (
    "bytes"
    "github.com/yuin/goldmark"
    "github.com/yuin/goldmark/extension"
    "github.com/yuin/goldmark/parser"
    "github.com/yuin/goldmark/renderer/html"
)

md := goldmark.New(
          goldmark.WithExtensions(extension.GFM),
          goldmark.WithParserOptions(
              parser.WithAutoHeadingID(),
          ),
          goldmark.WithRendererOptions(
              html.WithHardWraps(),
              html.WithXHTML(),
          ),
      )
var buf bytes.Buffer
if err := md.Convert(source, &buf); err != nil {
    panic(err)
}
Functional optionTypeDescription
goldmark.WithParserparser.ParserThis option must be passed before goldmark.WithParserOptions and goldmark.WithExtensions
goldmark.WithRendererrenderer.RendererThis option must be passed before goldmark.WithRendererOptions and goldmark.WithExtensions
goldmark.WithParserOptions...parser.Option
goldmark.WithRendererOptions...renderer.Option
goldmark.WithExtensions...goldmark.Extender

Parser and Renderer options

Parser options

Functional optionTypeDescription
parser.WithBlockParsersA util.PrioritizedSlice whose elements are parser.BlockParserParsers for parsing block level elements.
parser.WithInlineParsersA util.PrioritizedSlice whose elements are parser.InlineParserParsers for parsing inline level elements.
parser.WithParagraphTransformersA util.PrioritizedSlice whose elements are parser.ParagraphTransformerTransformers for transforming paragraph nodes.
parser.WithASTTransformersA util.PrioritizedSlice whose elements are parser.ASTTransformerTransformers for transforming an AST.
parser.WithAutoHeadingID-Enables auto heading ids.
parser.WithAttribute-Enables custom attributes. Currently only headings supports attributes.

HTML Renderer options

Functional optionTypeDescription
html.WithWriterhtml.Writerhtml.Writer for writing contents to an io.Writer.
html.WithHardWraps-Render newlines as <br>.
html.WithXHTML-Render as XHTML.
html.WithUnsafe-By default, goldmark does not render raw HTML or potentially dangerous links. With this option, goldmark renders such content as written.

Built-in extensions

Attributes

The parser.WithAttribute option allows you to define attributes on some elements.

Currently only headings support attributes.

Attributes are being discussed in the CommonMark forum. This syntax may possibly change in the future.

Headings

## heading ## {#id .className attrName=attrValue class="class1 class2"}

## heading {#id .className attrName=attrValue class="class1 class2"}
heading {#id .className attrName=attrValue}
============

Table extension

The Table extension implements Table(extension), as defined in GitHub Flavored Markdown Spec.

Specs are defined for XHTML, so specs use some deprecated attributes for HTML5.

You can override alignment rendering method via options.

Functional optionTypeDescription
extension.WithTableCellAlignMethodextension.TableCellAlignMethodOption indicates how are table cells aligned.

Typographer extension

The Typographer extension translates plain ASCII punctuation characters into typographic-punctuation HTML entities.

Default substitutions are:

PunctuationDefault entity
'&lsquo;, &rsquo;
"&ldquo;, &rdquo;
--&ndash;
---&mdash;
...&hellip;
<<&laquo;
>>&raquo;

You can override the default substitutions via extensions.WithTypographicSubstitutions:

markdown := goldmark.New(
    goldmark.WithExtensions(
        extension.NewTypographer(
            extension.WithTypographicSubstitutions(extension.TypographicSubstitutions{
                extension.LeftSingleQuote:  []byte("&sbquo;"),
                extension.RightSingleQuote: nil, // nil disables a substitution
            }),
        ),
    ),
)

Linkify extension

The Linkify extension implements Autolinks(extension), as defined in GitHub Flavored Markdown Spec.

Since the spec does not define details about URLs, there are numerous ambiguous cases.

You can override autolinking patterns via options.

Functional optionTypeDescription
extension.WithLinkifyAllowedProtocols[][]byte | []stringList of allowed protocols such as []string{ "http:" }
extension.WithLinkifyURLRegexp*regexp.RegexpRegexp that defines URLs, including protocols
extension.WithLinkifyWWWRegexp*regexp.RegexpRegexp that defines URL starting with www.. This pattern corresponds to the extended www autolink
extension.WithLinkifyEmailRegexp*regexp.RegexpRegexp that defines email addresses`

Example, using xurls:

import "mvdan.cc/xurls/v2"

markdown := goldmark.New(
    goldmark.WithRendererOptions(
        html.WithXHTML(),
        html.WithUnsafe(),
    ),
    goldmark.WithExtensions(
        extension.NewLinkify(
            extension.WithLinkifyAllowedProtocols([]string{
                "http:",
                "https:",
            }),
            extension.WithLinkifyURLRegexp(
                xurls.Strict(),
            ),
        ),
    ),
)

Footnotes extension

The Footnote extension implements PHP Markdown Extra: Footnotes.

This extension has some options:

Functional optionTypeDescription
extension.WithFootnoteIDPrefix[]byte | stringa prefix for the id attributes.
extension.WithFootnoteIDPrefixFunctionfunc(gast.Node) []bytea function that determines the id attribute for given Node.
extension.WithFootnoteLinkTitle[]byte | stringan optional title attribute for footnote links.
extension.WithFootnoteBacklinkTitle[]byte | stringan optional title attribute for footnote backlinks.
extension.WithFootnoteLinkClass[]byte | stringa class for footnote links. This defaults to footnote-ref.
extension.WithFootnoteBacklinkClass[]byte | stringa class for footnote backlinks. This defaults to footnote-backref.
extension.WithFootnoteBacklinkHTML[]byte | stringa class for footnote backlinks. This defaults to &#x21a9;&#xfe0e;.

Some options can have special substitutions. Occurrences of “^^” in the string will be replaced by the corresponding footnote number in the HTML output. Occurrences of “%%” will be replaced by a number for the reference (footnotes can have multiple references).

extension.WithFootnoteIDPrefix and extension.WithFootnoteIDPrefixFunction are useful if you have multiple Markdown documents displayed inside one HTML document to avoid footnote ids to clash each other.

extension.WithFootnoteIDPrefix sets fixed id prefix, so you may write codes like the following:

for _, path := range files {
    source := readAll(path)
    prefix := getPrefix(path)

    markdown := goldmark.New(
        goldmark.WithExtensions(
            NewFootnote(
                WithFootnoteIDPrefix(path),
            ),
        ),
    )
    var b bytes.Buffer
    err := markdown.Convert(source, &b)
    if err != nil {
        t.Error(err.Error())
    }
}

extension.WithFootnoteIDPrefixFunction determines an id prefix by calling given function, so you may write codes like the following:

markdown := goldmark.New(
    goldmark.WithExtensions(
        NewFootnote(
                WithFootnoteIDPrefixFunction(func(n gast.Node) []byte {
                    v, ok := n.OwnerDocument().Meta()["footnote-prefix"]
                    if ok {
                        return util.StringToReadOnlyBytes(v.(string))
                    }
                    return nil
                }),
        ),
    ),
)

for _, path := range files {
    source := readAll(path)
    var b bytes.Buffer

    doc := markdown.Parser().Parse(text.NewReader(source))
    doc.Meta()["footnote-prefix"] = getPrefix(path)
    err := markdown.Renderer().Render(&b, source, doc)
}

You can use goldmark-meta to define a id prefix in the markdown document:

---
title: document title
slug: article1
footnote-prefix: article1
---

# My article

CJK extension

CommonMark gives compatibilities a high priority and original markdown was designed by westerners. So CommonMark lacks considerations for languages like CJK.

This extension provides additional options for CJK users.

Functional optionTypeDescription
extension.WithEastAsianLineBreaks...extension.EastAsianLineBreaksStyleSoft line breaks are rendered as a newline. Some asian users will see it as an unnecessary space. With this option, soft line breaks between east asian wide characters will be ignored. This defaults to EastAsianLineBreaksStyleSimple.
extension.WithEscapedSpace-Without spaces around an emphasis started with east asian punctuations, it is not interpreted as an emphasis(as defined in CommonMark spec). With this option, you can avoid this inconvenient behavior by putting 'not rendered' spaces around an emphasis like 太郎は\ **「こんにちわ」**\ といった.

Styles of Line Breaking

StyleDescription
EastAsianLineBreaksStyleSimpleSoft line breaks are ignored if both sides of the break are east asian wide character. This behavior is the same as east_asian_line_breaks in Pandoc.
EastAsianLineBreaksCSS3DraftThis option implements CSS text level3 Segment Break Transformation Rules with some enhancements.

Example of EastAsianLineBreaksStyleSimple

Input Markdown:

私はプログラマーです。
東京の会社に勤めています。
GoでWebアプリケーションを開発しています。

Output:

<p>私はプログラマーです。東京の会社に勤めています。\nGoでWebアプリケーションを開発しています。</p>

Example of EastAsianLineBreaksCSS3Draft

Input Markdown:

私はプログラマーです。
東京の会社に勤めています。
GoでWebアプリケーションを開発しています。

Output:

<p>私はプログラマーです。東京の会社に勤めています。GoでWebアプリケーションを開発しています。</p>

Security

By default, goldmark does not render raw HTML or potentially-dangerous URLs. If you need to gain more control over untrusted contents, it is recommended that you use an HTML sanitizer such as bluemonday.

Benchmark

You can run this benchmark in the _benchmark directory.

against other golang libraries

blackfriday v2 seems to be the fastest, but as it is not CommonMark compliant, its performance cannot be directly compared to that of the CommonMark-compliant libraries.

goldmark, meanwhile, builds a clean, extensible AST structure, achieves full compliance with CommonMark, and consumes less memory, all while being reasonably fast.

BenchmarkMarkdown/Blackfriday-v2-8                   302           3743747 ns/op         3290445 B/op      20050 allocs/op
BenchmarkMarkdown/GoldMark-8                         280           4200974 ns/op         2559738 B/op      13435 allocs/op
BenchmarkMarkdown/CommonMark-8                       226           5283686 ns/op         2702490 B/op      20792 allocs/op
BenchmarkMarkdown/Lute-8                              12          92652857 ns/op        10602649 B/op      40555 allocs/op
BenchmarkMarkdown/GoMarkdown-8                        13          81380167 ns/op         2245002 B/op      22889 allocs/op

against cmark (CommonMark reference implementation written in C)

----------- cmark -----------
file: _data.md
iteration: 50
average: 0.0044073057 sec
------- goldmark -------
file: _data.md
iteration: 50
average: 0.0041611990 sec

As you can see, goldmark's performance is on par with cmark's.

Extensions

List of extensions

Loading extensions at runtime

goldmark-dynamic allows you to write a goldmark extension in Lua and load it at runtime without re-compilation.

Please refer to goldmark-dynamic for details.

goldmark internal(for extension developers)

Overview

goldmark's Markdown processing is outlined in the diagram below.

            <Markdown in []byte, parser.Context>
                           |
                           V
            +-------- parser.Parser ---------------------------
            | 1. Parse block elements into AST
            |   1. If a parsed block is a paragraph, apply 
            |      ast.ParagraphTransformer
            | 2. Traverse AST and parse blocks.
            |   1. Process delimiters(emphasis) at the end of
            |      block parsing
            | 3. Apply parser.ASTTransformers to AST
                           |
                           V
                      <ast.Node>
                           |
                           V
            +------- renderer.Renderer ------------------------
            | 1. Traverse AST and apply renderer.NodeRenderer
            |    corespond to the node type

                           |
                           V
                        <Output>

Parsing

Markdown documents are read through text.Reader interface.

AST nodes do not have concrete text. AST nodes have segment information of the documents, represented by text.Segment .

text.Segment has 3 attributes: Start, End, Padding .

(TBC)

TODO

See extension directory for examples of extensions.

Summary:

  1. Define AST Node as a struct in which ast.BaseBlock or ast.BaseInline is embedded.
  2. Write a parser that implements parser.BlockParser or parser.InlineParser.
  3. Write a renderer that implements renderer.NodeRenderer.
  4. Define your goldmark extension that implements goldmark.Extender.

Donation

BTC: 1NEDSyUmo4SMTDP83JJQSWi1MvQUGGNMZB

License

MIT

Author

Yusuke Inuzuka