Home

Awesome

Prepare your markdown for easy diff'ing!

<!-- vim-markdown-toc GFM --> <!-- vim-markdown-toc -->

About

This is mdslw, the MarkDown Sentence Line Wrapper, an auto-formatter that prepares your markdown for easy diff'ing.

Motivation

Markdown documents are written for different purposes. Some of them are meant to be read in plain text, while others are first rendered and then presented to the reader. In the latter case, the documents are often kept in version control and edited with the same workflows as other code.

When editing source code, software developers do not want changes in one location to show up as changes in unrelated locations. Now imagine a markdown document like this:

# Lorem Ipsum

Lorem ipsum dolor sit amet. Consectetur adipiscing elit. Sed do eiusmod tempor
incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam.

Adding the new sentence Excepteur sint occaecat cupidatat non proident. after the second one and re-arranging the text as a block would result in a diff view like this that shows changes in several lines:

3,4c3,5
< Lorem ipsum dolor sit amet. Consectetur adipiscing elit. Sed do eiusmod tempor
< incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam.
---
> Lorem ipsum dolor sit amet. Consectetur adipiscing elit. Excepteur sint occaecat
> cupidatat non proident. Sed do eiusmod tempor incididunt ut labore et dolore
> magna aliqua. Ut enim ad minim veniam.

Now imagine the original text had a line break after every sentence, i.e. it had looked like this:

# Lorem Ipsum

Lorem ipsum dolor sit amet.
Consectetur adipiscing elit.
Sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
Ut enim ad minim veniam.

For text formatted like this, a diff would only show up for the sentences that are actually affected, simplifying the review process:

4a5
> Excepteur sint occaecat cupidatat non proident.

Most rendering engines treat a single linebreak like a single space. Thus, both documents would be identical when presented to the reader even though the latter is significantly nicer to keep up to date with version control. The tool mdslw aims to auto-format markdown documents in exactly this way.

Working principle

The tool mdslw operates according to a very simple process that can be described as follows:

In contrast to most other tools the author could find, mdslw does not parse the entire document into an internal data structure just to render it back because that might result in changes in unexpected locations. Instead, it adjusts only those areas that do contain text that can be wrapped. That is, mdslw never touches any parts of a document that cannot be line-wrapped automatically. That includes, for example, code blocks, HTML blocks, and pipe tables.

Caveats

About markdown extensions

There are quite a lot of markdown extensions out there. It is not possible for mdslw to support all of them. Instead, mdslw aims at supporting CommonMark as well as some extensions used by its users. A new extension can be supported if supporting it does not negatively impact CommonMark support and if support can be added relatively easily. Please feel free to suggest support for a new extension as a contribution.

Command reference

Call as:

mdslw [OPTIONS] [PATHS]...

A PATH can point to a file or a directory. If it is a file, then it will be auto-formatted irrespective of its extension. If it is a directory, then mdslw will discover all files ending in .md recursively and auto-format those. If you do not specify any path, then mdslw will read from stdin and write to stdout.

The following is a list of all supported command line arguments. Note that you can also configure mdslw via environment variables or config files. Values are resolved in the following order:

Command Line Arguments

Automatic file discovery

This tool uses the ignore crate in its default settings to discover files when given a directory as a PATH. Details about those defaults can be found here. Briefly summarised, the following rules apply when deciding whether a file shall be ignored:

If you wish to format a file that is being ignored by mdslw, then pass it as an argument directly. Files passed as arguments are never ignored and will always be processed.

Environment Variables

Instead of or in addition to configuring mdslw via command line arguments or config files, you can configure it via environment variables. For any command line option --some-option=value, you can instead set an environment variable MDSLW_SOME_OPTION=value. For example, instead of setting --end-markers=".?!", you could set MDSLW_END_MARKERS=".?!" instead. When set, the value specified via the environment variable will take precedence over the default value and a value taken from config files. When set, a command line argument will take precedence over the environment variable. Take a call like this for example:

export MDSLW_EXTENSION=".markdown"
export MDSLW_MODE=both
mdslw --mode=check .

This call will search for files with the extension .markdown instead of the default .md. Furthermore, files will only be checked due to --mode=check, even though the environment variable MDSLW_MODE=both has been set. Defaults will be used for everything else.

Config Files

Instead of or in addition to configuring mdslw via command line arguments or environment variables, you can configure it via config files. Such a file has to have the exact name .mdslw.toml and affects all files in or below its own directory. Multiple config files will be merged. Options given in config files closer to a markdown file take precedence.

Configuration files are limited to options that influence the formatted result. They cannot influence how mdslw operates. For example, the option --mode cannot be set via config files while --max-width can. The following example shows all the possible options that can be set via config files. Note that all entries are optional in config files, which means that any number of them may be left out. The following is a full config file containing all the default values.

<!-- cfg-start --> 
max-width = 80
end-markers = "?!:."
lang = "ac"
suppressions = ""
ignores = ""
upstream = ""
case = "ignore"
features = ""
<!-- cfg-end -->

When set, the value specified via the config file will take precedence over the default value. When set, an environment variable or a command line argument will take precedence over a value taken from config files.

Installation

Go to the project's release page, select the correct binary for your system, and download it. See below for how to select the correct one. Rename the downloaded binary to mdslw (or mdslw.exe on Windows) and move it to a location that is in your $PATH such as /usr/local/bin (will be different on Windows). Moving it there will likely require admin or root permissions, e.g. via sudo. On Unix systems, you also have to make the binary executable via the command chmod +x mdslw, pointing to the actual location of mdslw. From now on, you can simply type mdslw in your terminal to use it!

The naming of the release binaries uses the LLVM target triple. You can also use the following list to pick the correct binary for your machine:

Building From Source

First, install rust, including cargo, via rustup. Then, make sure you have git installed, too. Once you have both cargo and git, execute the following commands in a terminal:

git clone https://github.com/razziel89/mdslw
cargo install --locked --path mdslw

That way, you will only get the default suppression list. If you want additional suppression lists such as the ones bundled with the pre-compiled binaries, you also require the tools jq, make, and curl. Once you have them installed, run make -C mdslw build-language-files before running the cargo install command to retrieve the suppression lists. The install command will pick them up automatically.

Editor Integration

Contributions describing integrations with more editors are welcome!

neovim

The recommended way of integrating mdslw with neovim is through conform.nvim. Simply install the plugin and modify your init.vim like this to add mdslw as a formatter for the markdown file type:

require("conform").setup({
  formatters_by_ft = {
    markdown = { "mdslw" },
  },
  formatters = {
    mdslw = { prepend_args = { "--stdin-filepath", "$FILENAME" } },
  },
})

Alternatively, you can also use the vim-like integration shown below.

vim

Add the following to your ~/.vimrc to have your editor auto-format every .md document before writing it out:

function MdFormat()
  if executable("mdslw")
    set lazyredraw
    " Enter and exit insert mode to keep track
    " of the cursor position, useful when undoing.
    execute "normal! ii\<BS>"
    let cursor_pos = getpos(".")
    %!mdslw --stdin-filepath "%"
    if v:shell_error != 0
      u
    endif
    call setpos('.', cursor_pos)
    set nolazyredraw
  endif
endfunction

autocmd BufWritePre *.md silent! :call MdFormat()

VS Code

Assuming you have mdslw installed and in your PATH, you can integrate it with VS Code. To do so, install the extension Run on Save and add the following snippet to your settings.json:

{
  "emeraldwalk.runonsave": {
    "commands": [
      {
        "match": ".*\\.md$",
        "cmd": "mdslw '${file}'"
      }
    ]
  }
}

From now on, every time you save to an existing markdown file, mdslw will auto-format it. This snippet assumes an empty settings.json file. If yours is not empty, you will have to merge it with the existing one.

Tips and Tricks

Non-Breaking Spaces

The following codepoints are recognised as non-breking spaces by default:

How to insert a non-breaking space depends on your operating system as well as your editor. The below will cover the non-breaking space U+00A0.

vim/neovim

Adding this to your ~/.vimrc or init.vim will let you insert non-breaking spaces when pressing CTRL+s in insert mode and also show them as +:

" Make it easy to insert non-breaking spaces and show them by default.
set list listchars+=nbsp:+
inoremap <C-s> <C-k>NS
" Alternatively, you can use this if your neovim/vim does not support this
" digraph. Note that your browser might not copy the non-breaking space at the
" end of the following line correctly.
inoremap <C-s>  

❗Tips for how to add and show non-breaking spaces in other editors are welcome.

Disabling Auto-Formatting

You can tell mdslw to stop auto-formatting parts of your document. Everything between the HTML comments <!-- mdslw-ignore-start --> and <!-- mdslw-ignore-end --> will not be formatted. For convenience, mdslw also recognises prettier's range ignore directives <!-- prettier-ignore-start --> and <!-- prettier-ignore-end -->.

How to contribute

If you have found a bug and want to fix it, please simply go ahead and fork the repository, fix the bug, and open a pull request to this repository! Bug fixes are always welcome.

In all other cases, please open an issue on GitHub first to discuss the contribution. The feature you would like to introduce might already be in development. Please also take note of the intended scope of mdslw.

Licence

GPLv3

If you want to use this piece of software under a different, more permissive open-source licence, please contact me. I am very open to discussing this point.