Home

Awesome

lit

a little preprocessor for literate programming

fire

inverts code and comments in any language to make a codebase documentation-first

Quick Start

By default, extracts code from Markdown files:

# strip out Markdown content from literate programming
# files in the current directory, leaving only
# executable code in parallel files
$ ./lit.sh

Alternatively, and highly recommended for development purposes, you can just comment out the Markdown, thereby preserving the original line numbers for more accurate debugging:

# comment out Markdown content from literate programming
# files in the current directory with hash style inline
# comments as used in e.g. Python, Ruby, or Bash
$ ./lit.sh --before "#"

As above, but also immediately execute the code in a Markdown file:

# use Python to execute the fenced code blocks inside script.py.md
$ python $(./lit.sh --input "script.py.md" --before "#")

Overview

Literate programming is the delightful idea popularized by Donald Knuth that source code should be written and/or richly annotated for clarity to human readers instead of mercilessly optimized for computing efficiency. One easy way to move in this direction is to write your source code into Markdown documents, with the code set aside in Markdown code blocks sprinkled throughout the written explanations. This inverts the usual relationship between code and comments: everything is assumed to be a comment until a special delimiter for marking the code is found, instead of the other way around as with most source code. In addition, your documentation can then use links, pictures, diagrams, embeds, iframes, or anything else allowed in Markdown and HTML.

This script is a tiny text preprocessor built with bash and awk which allows you to write all your source code in GitHub Flavored Markdown and then quickly send the content of the code blocks into parallel executable files. For a quick illustration, compare the annotated source of the included hello-world script to its processed output.

Installation?!?

Feel free to clone this repository, but you can also just download the script. It is self contained and there are no dependencies beyond bash and awk, which you almost certainly already have.

A one-liner for command line installation:

# install lit.sh
$ curl https://raw.githubusercontent.com/vijithassar/lit/master/lit.sh > lit.sh && chmod +x lit.sh

Usage

To compile literate source code, simply run the script. By default, it will process any Markdown files it finds in the current directory.

# compile literate code files
$ ./lit.sh

Filenames must contain two extensions in order for the script to operate on them. The first should be the regular extension for the language you're compiling to (.py, .js, etc), and the second should be the Markdown .md extension.

For example, these filenames would be processed:

These would not be processed:

Arguments

Input

The --input or -i arguments can be used to specify a path or file glob within which to find files to process. File globs must be quoted strings. If this argument is omitted, all literate code files in the current working directory will be processed.

# compile literate code files in the ./source directory
$ ./lit.sh --input "./source"
# compile literate Python files in the ./source directory
$ ./lit.sh --input "./source/*.py.md"

Output

The --output or -o arguments can be used to specify a path where the processed files will be written. If this argument is omitted, the processed files will be written in the same directory as the input files.

# compile literate code files in the current directory
# and write to the ./build directory
$ ./lit.sh --output ./build
# compile literate code files in the ./source directory
# and write to the current working directory
$ ./lit.sh --input "./source" --output .

Comments

Rather than simply stripping Markdown content entirely, it can be advantageous to just comment it out instead, since that means all code in the output file appears on the same line as in the original literate Markdown source, and thus errors and messages can be accurately reported by debuggers, loggers, compilers, and other such development tools. To comment out Markdown content instead of stripping it, use the --before or -b arguments, followed by the character(s) used to denote inline code comments for the language you are compiling.

This is definitely the recommended way to use this tool, but it can't be the default behavior because you need to specify the inline comment delimiter for your language in order for it to work.

# comment out Markdown content from literate programming
# files in the current directory with hash style inline
# comments as used in e.g. Python, Ruby, or Bash
$ ./lit.sh --before "#"

You can also comment out Markdown content using a block commenting style by supplying the --after or -a arguments, followed by the characters used to denote the end of a block comment in the language you are compiling. However, inline comments are preferable, because block comments can be broken if any of your Markdown content includes the character sequence that denotes the end of a block comment. This option is included mostly to allow "literate CSS" files, since CSS does not have a single-line comment syntax.

# comment out Markdown content from "literate CSS" files
# in the current directory using block comment syntax
$ ./lit.sh --input "./*.css.md" --before "/*" --after "*/"

Logging

As it processes files, the script echoes out the filenames of the resulting executable (non-Markdown) code files. If you want, you can capture these in a subshell and use them in further downstream logic, thereby totally abstracting away even the filename of the resulting processed code and creating a completely documentation-first workflow.

For example, to compile a single file, capture the filename with a subshell, and immediately execute the result:

# compile script.js.md to script.js and
# immediately execute with Node.js using a
# subshell
$ node $(./lit.sh --input "script.js.md" --before "//")
# compile script.py.md to script.py and
# immediately execute with Python using a
# subshell
$ python $(./lit.sh --input "script.py.md" --before "#")

Concise and subshell-friendly output is the most useful default, but you can also enable more verbose human-readable messages using the --verbose or -v arguments.

# compile all files in the current directory
# and output verbose log messages
$ ./lit.sh --verbose

Logging is naturally disabled when you use stdin or stdout with the --stdio or -s flags, in which case the printed output is the processed code.

stdin and stdout

The --stdio or -s arguments can be used to read from stdin for the Markdown content to be processed and send the processed code content to stdout.

# compile annotated.py.md to code.py by routing over stdio
$ cat annotated.py.md | ./lit.sh --stdio > code.py

Hidden Files

In addition, the --hidden or -h arguments can be used to prepend a dot . to the output filename. This is useful because files that start with a dot are hidden by default on most UNIX-like file systems. This behavior lets you hide the artifacts of compiling your Markdown into executable files.

For example, combining with subshells as described above:

# compile script.py.md to a hidden file and immediately execute it
$ python $(./lit.sh --input script.py.md --hidden)

If you're using Git, you may also want to add script.js or script.py to your .gitignore file in this scenario.

Process Substitution

You can avoid the artifact of creating a hidden file by using process substition to treat stdout as a file.

# compile script.py.md and execute the output
# with python as though it is a file
$ python <(cat script.py.md | ./lit.sh --stdio)

Advantages

Disadvantages

Tests

Unit tests are written with bats.

# run unit tests with bats
$ bats test.bats

Ecosystem

Pedantry

This isn't quite true to the original conception of literate programming, which also advocated for nonlinear compiling so source code can be structured for humans and then reorganized for execution.

However:

Miscellaneous

This lil guy: