Home

Awesome

unified-engine

Build Coverage Downloads Sponsors Backers Chat

unified engine to process multiple files, lettings users configure from the file system.

Contents

What is this?

This package is the engine. It’s what you use underneath when you use remark-cli or a language server. Compared to unified, this deals with multiple files, often from the file system, and with configuration files and ignore files.

When should I use this?

You typically use something that wraps this, such as:

You can use this to make such things.

Install

This package is ESM only. In Node.js (version 16+), install with npm:

npm install unified-engine

Use

The following example processes all files in the current folder with a markdown extension with remark, allows configuration from .remarkrc and package.json files, ignoring files from .remarkignore files, and more.

/**
 * @import {Callback} from 'unified-engine'
 */

import process from 'node:process'
import {remark} from 'remark'
import {engine} from 'unified-engine'

engine(
  {
    color: true,
    extensions: ['md', 'markdown', 'mkd', 'mkdn', 'mkdown'],
    files: ['.'],
    ignoreName: '.remarkignore',
    packageField: 'remarkConfig',
    pluginPrefix: 'remark',
    processor: remark,
    rcName: '.remarkrc'
  },
  done
)

/** @type {Callback} */
function done(error, code) {
  if (error) throw error
  process.exitCode = code
}

API

This package exports the identifiers Configuration and engine. There is no default export.

engine(options, callback)

Process.

Parameters
Returns

Nothing (undefined).

Configuration

Internal class to load configuration files.

Exposed to build more complex integrations.

Parameters
Fields

Completer

Completer (TypeScript type).

Type
type Completer = (CompleterCallback | CompleterRegular) & {
    pluginId?: string | symbol | undefined
}

type CompleterCallback = (set: FileSet, next: CompleterCallbackNext) => undefined
type CompleterCallbackNext = (error?: Error | null | undefined) => undefined
type CompleterRegular = (set: FileSet) => Promise<undefined> | undefined

Callback

Callback called when done (TypeScript type).

Called with a fatal error if things went horribly wrong (probably due to incorrect configuration), or a status code and the processing context.

Parameters
Returns

Nothing (undefined).

ConfigResult

Resolved configuration from Configuration (TypeScript type).

Fields

ConfigTransform

Transform arbitrary configs to our format (TypeScript type).

Parameters
Returns

Our config format (Preset).

Context

Processing context (TypeScript type).

Fields

FileSet

A FileSet is created to process multiple files through unified processors (TypeScript type).

This set, containing all files, is exposed to plugins as the second parameter.

Parameters

None.

Fields

Options

Configuration (TypeScript type).

👉 Note: options.processor is required.

Fields

Preset

Sharable configuration, with support for specifiers (TypeScript type).

Specifiers should not be used in actual presets (because they can’t be used by regular unified), but they can be used in config files locally, as those are only for the engine.

They can contain plugins and settings.

Type
import type {
  Plugin as UnifiedPlugin,
  PluginTuple as UnifiedPluginTuple,
  Preset as UnifiedPreset,
  Settings
} from 'unified'

type Preset = {
  plugins?: PluggableList | PluggableMap | undefined
  settings?: Settings | undefined
}

type Pluggable = Plugin | PluginTuple | UnifiedPreset
type PluggableList = Array<Pluggable>
type PluggableMap = Record<string, unknown>
type Plugin = UnifiedPlugin | string
type PluginTupleSupportingSpecifiers =
  | [plugin: string, ...parameters: Array<unknown>]
  | UnifiedPluginTuple

ResolveFrom

How to resolve (TypeScript type).

Type
type ResolveFrom = 'cwd' | 'dir';

VFileReporter

Transform arbitrary configs to our format (TypeScript type).

This is essentially the interface of vfile-reporter, with added support for unknown fields in options and async support.

Parameters
Returns

Report (Promise<string> or string).

Config files

unified-engine accepts configuration through options and through configuration files (rc files).

Explicit configuration

One configuration file can be given through options.rcPath, this is loaded regardless of options.detectConfig and options.rcName.

Implicit configuration

Otherwise, configuration files are detected if options.detectConfig is turned on, depending on the following options:

The first file that is searched for in a folder is used as the configuration. If no file is found, the parent folder is searched, and so on.

The schema (type) of rc files is Preset.

Examples

An example rc file could look as follows:

{
  "plugins": [
    "remark-inline-links",
    "remark-lint-recommended"
  ],
  "settings": {
    "bullet": "*",
    "ruleRepetition": 3,
    "fences": true
  }
}

Another example, rc.js, could look as follows:

exports.plugins = [
  './script/natural-language.js',
  'remark-lint-recommended',
  'remark-license'
]

exports.settings = {bullet: '*'}

When using ESM (ECMAScript modules), rc.mjs could look as folows:

export default {
  plugins: [
    './script/natural-language.js',
    'remark-lint-recommended',
    'remark-license'
  ],
  settings: {bullet: '*'}
}

Another example, rc.yaml, could look as follows:

plugins:
  - 'rehype-document'
  - 'rehype-preset-minify'
settings:
  preferUnquoted: true
  quote: "'"
  quoteSmart: true
  verbose: true

Ignore files

unified-engine accepts patterns to ignore when searching for files to process through ignore files.

Explicit ignoring

One ignore file can be given through options.ignorePath, this is loaded regardless of options.detectIgnore and options.ignoreName.

Implicit ignoring

Otherwise, ignore files are detected if options.detectIgnore is turned on and options.ignoreName is given.

The first file named $ignoreName in the parent folder of a checked file is used. Or, if no file is found, the parent folder if searched, and so on.

Extra ignoring

In addition to explicit and implicit ignore files, other patterns can be given with options.ignorePatterns. The format of each pattern in ignorePatterns is the same as a line in an ignore file. Patterns and files are resolved based on the current working folder.

It is also possible to ignore files that do not have an associated detected configuration file by turning on options.ignoreUnconfigured.

Ignoring

Ignoring is used when searching for files in folders. If paths (including those expanded from globs) are passed in that are ignored, an error is thrown. These files can be silently ignored by turning on options.silentlyIgnore.

Normally, files are ignored based on the path of the found ignore file and the patterns inside it. Patterns passed with options.ignorePatterns are resolved based on the current working directory.

Patterns in an explicit ignore file passed in with options.ignorePath can be resolved from the current working directory instead, by setting options.ignorePathResolveFrom to 'cwd' instead of 'dir' (default).

If paths or globs to folders are given to the engine, they will be searched for matching files, but node_modules are normally not searched. Pass paths (or globs) to the node_modules you want to include in options.files to search them.

The format for ignore files is the same as .gitignore, so it’s possible to pass a .gitignore in as options.ignorePath.

node-ignore is used under the hood, see its documentation for more information.

Examples

An example ignore file could look as follows:

# Ignore files in `.github`.
.github/

# Bower.
bower_components/
# Duo dependencies.
components/

# Fixtures.
test/{input,tree}/

If we had an ignore file folder/.remarkignore, with the value: index.txt, and our file system looked as follows:

folder/.remarkignore
folder/index.txt
index.txt

Then folder/index.txt would be ignored but index.txt would not be.

Plugins

Normally, unified plugins receive a single options argument upon attaching (an Object users can provide to configure the plugin).

If a plugin is attached by unified-engine, a second argument is given: FileSet.

Examples

unified-engine can be configured extensively by engine authors.

options.alwaysStringify

This example shows how you can use options.alwaysStringify when you don’t want the engine to write to the file system, but still want to get the compiled results. One example that does this is unified-engine-gulp.

import {remark} from 'remark'
import {engine} from 'unified-engine'
import {VFile} from 'vfile'

const file = new VFile({path: 'example.md', value: '_hi_'})

engine(
  {alwaysStringify: true, files: [file], processor: remark},
  function (error, code, context) {
    if (error) throw error
    console.log(context?.files.map((d) => String(d)))
  }
)

Yields:

example.md: no issues found

[ '*hi*\n' ]

options.configTransform

To support custom rc files, that have a different format than what the engine supports, pass as ConfigTransform.

This example processes readme.md and loads options from custom (from a package.json). configTransform is called with those options and transforms it to configuration unified-engine understands.

import {remark} from 'remark'
import {engine} from 'unified-engine'

engine(
  {
    configTransform,
    files: ['readme.md'],
    packageField: 'custom',
    processor: remark()
  },
  function (error) {
    if (error) throw error
  }
)

function configTransform(config) {
  return {settings: (config || {}).options}
}

Where package.json contains:

{
  "name": "foo",
  "private": true,
  "custom": {
    "options": {
      "bullet": "+"
    }
  }
}

options.defaultConfig

This example processes readme.md. If package.json exists, that config is used, otherwise the configuration at defaultConfig is used.

import {remark} from 'remark'
import {engine} from 'unified-engine'

engine(
  {
    defaultConfig: {settings: {bullet: '+'}},
    files: ['readme.md'],
    packageField: 'remarkConfig',
    processor: remark()
  },
  function (error) {
    if (error) throw error
  }
)

Where package.json contains:

{
  "name": "foo",
  "private": true,
  "remarkConfig": {
    "settings": {
      "bullet": "-"
    }
  }
}

options.detectConfig

This example processes readme.md but does not allow configuration from .remarkrc or package.json files, as detectConfig is false.

import {remark} from 'remark'
import {engine} from 'unified-engine'

engine(
  {
    detectConfig: false,
    files: ['readme.md'],
    processor: remark(),
    packageField: 'remarkConfig',
    rcName: '.remarkrc'
  },
  function (error) {
    if (error) throw error
  }
)

options.detectIgnore

This example processes files in the current working directory with an md extension but does not ignore file paths from the closest .remarkignore file, because detectIgnore is false.

import {remark} from 'remark'
import {engine} from 'unified-engine'

engine(
  {
    detectIgnore: false,
    extensions: ['md'],
    files: ['.'],
    ignoreName: '.remarkignore',
    processor: remark()
  },
  function (error) {
    if (error) throw error
  }
)

options.extensions

This example reformats all files with md, markdown, and mkd extensions in the current folder.

import {remark} from 'remark'
import {engine} from 'unified-engine'

engine(
  {
    extensions: ['md', 'mkd', 'markdown'],
    files: ['.'],
    output: true,
    processor: remark
  },
  function (error) {
    if (error) throw error
  }
)

options.filePath

This example shows that streamIn is named as filePath:

import {PassThrough} from 'node:stream'
import {remark} from 'remark'
import remarkPresetLintRecommended from 'remark-preset-lint-recommended'
import {engine} from 'unified-engine'

const streamIn = new PassThrough()

streamIn.write('doc')

setImmediate(function () {
  streamIn.end('ument')
})

engine(
  {
    filePath: '~/alpha/bravo/charlie.md',
    out: false,
    plugins: [remarkPresetLintRecommended],
    processor: remark(),
    streamIn
  },
  function (error) {
    if (error) throw error
  }
)

Yields:

~/alpha/bravo/charlie.md
  1:1  warning  Missing newline character at end of file  final-newline  remark-lint

⚠ 1 warning

options.files

This example processes LICENSE and all files with an md extension in doc.

import {remark} from 'remark'
import {engine} from 'unified-engine'

engine(
  {
    extensions: ['md'],
    files: ['LICENSE', 'doc/'],
    processor: remark
  },
  function (error) {
    if (error) throw error
  }
)

options.frail

This example uses remark-lint to lint readme.md and exits with the given exit code. Normally, only errors turn the code to 1, but in frail mode lint warnings result in the same.

import process from 'node:process'
import {remark} from 'remark'
import {engine} from 'unified-engine'

engine(
  {
    files: ['readme.md'],
    frail: true,
    plugins: ['remark-preset-lint-recommended'],
    processor: remark()
  },
  function (error, code) {
    process.exitCode = error ? 1 : code
  }
)

options.ignoreName

This example processes files in the current working directory with an md extension, and is configured to ignore file paths from the closest .remarkignore file.

import {remark} from 'remark'
import {engine} from 'unified-engine'

engine(
  {
    extensions: ['md'],
    files: ['.'],
    ignoreName: '.remarkignore',
    processor: remark()
  },
  function (error) {
    if (error) throw error
  }
)

options.ignorePath

This example processes files in the current working directory with an md extension and ignores file paths specified in .gitignore.

import {remark} from 'remark'
import {engine} from 'unified-engine'

engine(
  {
    extensions: ['md'],
    files: ['.'],
    ignorePath: '.gitignore',
    processor: remark()
  },
  function (error) {
    if (error) throw error
  }
)

options.ignorePathResolveFrom

This example processes files in the current working directory with an md extension and takes a reusable configuration file from a dependency.

import {remark} from 'remark'
import {engine} from 'unified-engine'

engine(
  {
    extensions: ['md'],
    files: ['.'],
    ignorePath: 'node_modules/my-config/my-ignore',
    ignorePathResolveFrom: 'cwd',
    processor: remark()
  },
  function (error) {
    if (error) throw error
  }
)

options.ignorePatterns

This example processes files in the current working directory with an md extension, except for readme.md:

import {remark} from 'remark'
import {engine} from 'unified-engine'

engine(
  {
    extensions: ['md'],
    files: ['.'],
    ignorePatterns: ['readme.md'],
    processor: remark()
  },
  function (error) {
    if (error) throw error
  }
)

options.ignoreUnconfigured

This example processes files in the current working directory with an md extension, but only if there is an explicit .remarkrc config file near (upwards) to them:

import {remark} from 'remark'
import {engine} from 'unified-engine'

engine(
  {
    extensions: ['md'],
    files: ['.'],
    ignoreUnconfigured: true,
    processor: remark(),
    rcName: '.remarkrc'
  },
  function (error) {
    if (error) throw error
  }
)

options.inspect

This example shows a module which reads and parses doc.md, then remark-unlink transforms the syntax tree, the tree is formatted with unist-util-inspect, and finally written to stdout(4).

import {remark} from 'remark'
import remarkUnlink from 'remark-unlink'
import {engine} from 'unified-engine'

engine(
  {
    files: ['doc.md'],
    inspect: true,
    plugins: [remarkUnlink],
    processor: remark()
  },
  function (error) {
    if (error) throw error
  }
)

Where doc.md looks as follows:

[foo](https://example.com)

Yields:

root[1] (1:1-2:1, 0-27)
└─ paragraph[1] (1:1-1:27, 0-26)
   └─ text: "foo" (1:2-1:5, 1-4)

options.out

This example uses remark-lint to lint readme.md, writes the report, and ignores the serialized document.

import {remark} from 'remark'
import remarkPresetLintRecommended from 'remark-preset-lint-recommended'
import {engine} from 'unified-engine'

engine(
  {
    files: ['readme.md'],
    out: false,
    plugins: [remarkPresetLintRecommended],
    processor: remark()
  },
  function (error) {
    if (error) throw error
  }
)

options.output

This example writes all files in src/ with an md extension compiled to dest/.

import {remark} from 'remark'
import {engine} from 'unified-engine'

engine(
  {
    extensions: ['md'],
    files: ['src/'],
    output: 'dest/',
    processor: remark()
  },
  function (error) {
    if (error) throw error
  }
)

options.packageField

This example processes readme.md, and allows configuration from remarkConfig fields in package.json files.

import {remark} from 'remark'
import {engine} from 'unified-engine'

engine(
  {
    files: ['readme.md'],
    packageField: 'remarkConfig',
    processor: remark()
  },
  function (error) {
    if (error) throw error
  }
)

options.pluginPrefix

This example processes readme.md and loads the preset-lint-recommended plugin. Because pluginPrefix is given, this resolves to remark-preset-lint-recommended (from node_modules/) if available.

import {remark} from 'remark'
import {engine} from 'unified-engine'

engine(
  {
    files: ['readme.md'],
    pluginPrefix: 'remark',
    plugins: ['preset-lint-recommended'],
    processor: remark()
  },
  function (error) {
    if (error) throw error
  }
)

options.plugins

This example processes readme.md and loads the remark-preset-lint-recommended preset.

import {remark} from 'remark'
import {engine} from 'unified-engine'

engine(
  {
    files: ['readme.md'],
    plugins: ['remark-preset-lint-recommended'],
    processor: remark()
  },
  function (error) {
    if (error) throw error
  }
)

options.processor

This example reformats stdin(4) using remark, writes the report to stderr(4), and formatted document to stdout(4).

import {remark} from 'remark'
import {engine} from 'unified-engine'

engine({processor: remark}, function (error) {
  if (error) throw error
})

options.quiet

This example uses remark-lint to lint readme.md. Nothing is reported if the file processed successfully.

import {remark} from 'remark'
import {engine} from 'unified-engine'

engine(
  {
    files: ['readme.md'],
    plugins: ['remark-preset-lint-recommended'],
    processor: remark(),
    quiet: true
  },
  function (error) {
    if (error) throw error
  }
)

options.rcName

This example processes readme.md and allows configuration from .remarkrc, .remarkrc.json, .remarkrc.yml, .remarkrc.yaml, .remarkrc.js, .remarkrc.cjs, and .remarkrc.mjs files.

import {remark} from 'remark'
import {engine} from 'unified-engine'

engine(
  {files: ['readme.md'], processor: remark(), rcName: '.remarkrc'},
  function (error) {
    if (error) throw error
  }
)

options.rcPath

This example processes readme.md and loads configuration from config.json.

import {remark} from 'remark'
import {engine} from 'unified-engine'

engine(
  {files: ['readme.md'], processor: remark(), rcPath: 'config.json'},
  function (error) {
    if (error) throw error
  }
)

options.reporter and options.reporterOptions

This example processes all HTML files in the current folder with rehype, configures the processor with .rehyperc files, and prints a report in JSON using vfile-reporter-json with reporter options.

import {rehype} from 'rehype'
import {engine} from 'unified-engine'

engine(
  {
    extensions: ['html'],
    files: ['.'],
    processor: rehype(),
    rcName: '.rehyperc',
    reporter: 'json',
    reporterOptions: {pretty: true}
  },
  function (error) {
    if (error) throw error
  }
)

options.settings

This example processes readme.md and configures the compiler (remark-stringify) with bullet: '+'.

import {remark} from 'remark'
import {engine} from 'unified-engine'

engine(
  {files: ['readme.md'], processor: remark(), settings: {bullet: '+'}},
  function (error) {
    if (error) throw error
  }
)

options.silent

This example uses remark-lint to lint readme.md but does not report any warnings or success messages, only fatal errors, if they occur.

import {remark} from 'remark'
import {engine} from 'unified-engine'

engine(
  {
    files: ['readme.md'],
    plugins: ['remark-preset-lint-recommended'],
    processor: remark(),
    silent: true
  },
  function (error) {
    if (error) throw error
  }
)

options.streamError

This example uses remark-lint to lint readme.md and writes the report to report.txt.

import fs from 'node:fs'
import {remark} from 'remark'
import remarkPresetLintRecommended from 'remark-preset-lint-recommended'
import {engine} from 'unified-engine'

engine(
  {
    files: ['readme.md'],
    out: false,
    plugins: [remarkPresetLintRecommended],
    processor: remark(),
    streamErr: fs.createWriteStream('report.txt')
  },
  function (error) {
    if (error) throw error
  }
)

options.streamIn

This example uses remark-lint to lint an incoming stream.

import {PassThrough} from 'node:stream'
import {remark} from 'remark'
import remarkPresetLintRecommended from 'remark-preset-lint-recommended'
import {engine} from 'unified-engine'

const streamIn = new PassThrough()

streamIn.write('doc')

setImmediate(function () {
  streamIn.end('ument')
})

engine(
  {
    out: false,
    plugins: [remarkPresetLintRecommended],
    processor: remark(),
    streamIn
  },
  function (error) {
    if (error) throw error
  }
)

Yields:

<stdin>
  1:1  warning  Missing newline character at end of file  final-newline  remark-lint

⚠ 1 warning

options.streamOut

This example reads readme.md and writes the serialized document to readme-two.md. This can also be achieved by passing output: 'readme-two.md' instead of streamOut.

import fs from 'node:fs'
import {remark} from 'remark'
import {engine} from 'unified-engine'

const streamOut = fs.createWriteStream('readme-two.md')

engine(
  {files: ['readme.md'], processor: remark(), streamOut},
  function (error) {
    if (error) throw error
  }
)

options.tree

This example reads tree.json, then remark-unlink transforms the syntax tree, and the transformed tree is written to stdout(4).

import {remark} from 'remark'
import remarkUnlink from 'remark-unlink'
import {engine} from 'unified-engine'

engine(
  {
    files: ['tree.json'],
    plugins: [remarkUnlink],
    processor: remark(),
    tree: true
  },
  function (error) {
    if (error) throw error
  }
)

Where tree.json looks as follows:

{
  "type": "paragraph",
  "children": [{
    "type": "link",
    "url": "https://example.com",
    "children": [{
      "type": "text",
      "value": "foo"
    }]
  }]
}

Yields:

{
  "type": "paragraph",
  "children": [{
    "type": "text",
    "value": "foo"
  }]
}

options.treeIn

This example reads tree.json, then remark-unlink transforms the syntax tree, the tree is serialized, and the resulting document is written to stdout(4).

import {remark} from 'remark'
import remarkUnlink from 'remark-unlink'
import {engine} from 'unified-engine'

engine(
  {
    files: ['tree.json'],
    plugins: [remarkUnlink],
    processor: remark(),
    treeIn: true
  },
  function (error) {
    if (error) throw error
  }
)

Where tree.json looks as follows:

{
  "type": "paragraph",
  "children": [{
    "type": "link",
    "url": "https://example.com",
    "children": [{
      "type": "text",
      "value": "foo"
    }]
  }]
}

Yields:

foo

options.treeOut

This example shows a module which reads and parses doc.md, then remark-unlink transforms the syntax tree, and the tree is written to stdout(4).

import {remark} from 'remark'
import remarkUnlink from 'remark-unlink'
import {engine} from 'unified-engine'

engine(
  {
    files: ['doc.md'],
    plugins: [remarkUnlink],
    processor: remark(),
    treeOut: true
  },
  function (error) {
    if (error) throw error
  }
)

Where doc.md looks as follows:

[foo](https://example.com)

Yields:

{
  "type": "paragraph",
  "children": [{
    "type": "text",
    "value": "foo"
  }]
}

Types

This package is fully typed with TypeScript. It exports the additional types Completer, Callback, ConfigResult, ConfigTransform, Context, FileSet, Options, Preset, ResolveFrom, and VFileReporter.

Compatibility

Projects maintained by the unified collective are compatible with maintained versions of Node.js.

When we cut a new major release, we drop support for unmaintained versions of Node. This means we try to keep the current release line, unified-engine@^11, compatible with Node.js 16.

Security

unified-engine loads and evaluates configuration files, plugins, and presets from the file system (often from node_modules/). That means code that is on your file system runs. Make sure you trust the workspace where you run unified-engine and be careful with packages from npm and changes made by contributors.

Contribute

See contributing.md in unifiedjs/.github for ways to get started. See support.md for ways to get help.

This project has a code of conduct. By interacting with this repository, organization, or community you agree to abide by its terms.

License

MIT © Titus Wormer

<!-- Definitions -->