Home

Awesome

Hostlist compiler

NPM

This is a simple tool that makes it easier to compile a hosts blocklist compatible with AdGuard Home or any other AdGuard product with DNS filtering.

<a name="usage"></a> Usage

First of all, install the hostlist-compiler:

npm i -g @adguard/hostlist-compiler

After that you have two options.

Quick hosts conversion

Convert and compress a /etc/hosts-syntax blocklist to AdGuard syntax.

hostlist-compiler -i hosts.txt -i hosts2.txt -o output.txt

Build a configurable blocklist from multiple sources

Prepare the list configuration (read more about that below) and run the compiler:

hostlist-compiler -c configuration.json -o output.txt

All command line options

Usage: hostlist-compiler [options]

Options:
  --config, -c      Path to the compiler configuration file             [string]
  --input, -i       URL (or path to a file) to convert to an AdGuard-syntax
                    blocklist. Can be specified multiple times.          [array]
  --input-type, -t  Type of the input file (/etc/hosts, adguard)        [string]
  --output, -o      Path to the output file                  [string] [required]
  --verbose, -v     Run with verbose logging                           [boolean]
  --version         Show version number                                [boolean]
  -h, --help        Show help                                          [boolean]

Examples:
  hostlist-compiler -c config.json -o       compile a blocklist and write the
  output.txt                                output to output.txt
  hostlist-compiler -i                      compile a blocklist from the URL and
  https://example.org/hosts.txt -o          write the output to output.txt
  output.txt

<a name="configuration"></a> Configuration

Configuration defines your filter list sources, and the transformations that are applied to the sources.

Here is an example of this configuration:

{
  "name": "List name",
  "description": "List description",
  "homepage": "https://example.org/",
  "license": "GPLv3",
  "version": "1.0.0.0",
  "sources": [
    {
      "name": "Local rules",
      "source": "rules.txt",
      "type": "adblock",
      "transformations": ["RemoveComments", "Compress"],
      "exclusions": ["excluded rule 1"],
      "exclusions_sources": ["exclusions.txt"],
      "inclusions": ["*"],
      "inclusions_sources": ["inclusions.txt"]
    },
    {
      "name": "Remote rules",
      "source": "https://example.org/rules",
      "type": "hosts",
      "exclusions": ["excluded rule 1"]
    }
  ],
  "transformations": ["Deduplicate", "Compress"],
  "exclusions": ["excluded rule 1", "excluded rule 2"],
  "exclusions_sources": ["global_exclusions.txt"],
  "inclusions": ["*"],
  "inclusions_sources": ["global_inclusions.txt"]
}

Here is an example of a minimal configuration:

{
  "name": "test list",
  "sources": [
    {
      "source": "rules.txt"
    }
  ]
}

Exclusion and inclusion rules

Please note, that exclusion or inclusion rules may be a plain string, wildcard, or a regular expression.

<a name="command-line"></a> Command-line

Command-line arguments.

Usage: hostlist-compiler [options]

Options:
  --version      Show version number                                   [boolean]
  --config, -c   Path to the compiler configuration file     [string] [required]
  --output, -o   Path to the output file                     [string] [required]
  --verbose, -v  Run with verbose logging                              [boolean]
  -h, --help     Show help                                             [boolean]

Examples:
  hostlist-compiler -c config.json -o       compile a blocklist and write the
  output.txt                                output to output.txt

<a name="api"></a> API

Install: npm i @adguard/hostlist-compiler or yarn add @adguard/hostlist-compiler

JavaScript example:

const compile = require("@adguard/hostlist-compiler");

;(async () => {
    // Compile filters
    const result = await compile({
        name: 'Your Hostlist',
        sources: [
            {
                type: 'adblock',
                source: 'https://adguardteam.github.io/AdGuardSDNSFilter/Filters/filter.txt', // or local file
                transformations: ['RemoveComments', 'Validate'],
            },
        ],
        transformations: ['Deduplicate'],
    });

    // Write to file
    writeFileSync('your-hostlist.txt', result.join('\n'));
})();

TypeScript example:

import compile from '@adguard/hostlist-compiler';
import { writeFileSync } from 'fs';

;(async () => {
    // Compile filters
    const result = await compile({
        name: 'Your Hostlist',
        sources: [
            {
                type: 'adblock',
                source: 'https://adguardteam.github.io/AdGuardSDNSFilter/Filters/filter.txt',
                transformations: ['RemoveComments', 'Validate'],
            },
        ],
        transformations: ['Deduplicate'],
    });

    // Write to file
    writeFileSync('your-hostlist.txt', result.join('\n'));
})();

or:

import HostlistCompiler, { IConfiguration as HostlistCompilerConfiguration } from '@adguard/hostlist-compiler';
import { writeFileSync } from 'fs';

;(async () => {
    // Configuration
    const config: HostlistCompilerConfiguration = {
        name: 'Your Hostlist',
        sources: [
            {
                type: 'adblock',
                source: 'https://adguardteam.github.io/AdGuardSDNSFilter/Filters/filter.txt',
                transformations: ['RemoveComments', 'Validate'],
            },
        ],
        transformations: ['Deduplicate'],
    };

    // Compile filters
    const result = await HostlistCompiler(config);

    // Write to file
    writeFileSync('your-hostlist.txt', result.join('\n'));
})();

<a name="transformations"></a> Transformations

Here is the full list of transformations that are available:

  1. RemoveComments
  2. Compress
  3. RemoveModifiers
  4. Validate
  5. ValidateAllowIp
  6. Deduplicate
  7. InvertAllow
  8. RemoveEmptyLines
  9. TrimLines
  10. InsertFinalNewLine

Please note that these transformations are are always applied in the order specified here.

<a name="remove-comments"></a> RemoveComments

This is a very simple transformation that simply removes comments (e.g. all rules starting with ! or #).

<a name="compress"></a> Compress

[!IMPORTANT] This transformation converts hosts lists into adblock lists.

Here's what it does:

  1. It converts all rules to adblock-style rules. For instance, 0.0.0.0 example.org will be converted to ||example.org^.
  2. It discards the rules that are now redundant because of other existing rules. For instance, ||example.org blocks example.org and all it's subdomains, therefore additional rules for the subdomains are now redundant.

<a name="remove-modifiers"></a> RemoveModifiers

By default, AdGuard Home will ignore rules with unsupported modifiers, and all of the modifiers listed here are unsupported. However, the rules with these modifiers are likely to be okay for DNS-level blocking, that's why you might want to remove them when importing rules from a traditional filter list.

Here is the list of modifiers that will be removed:

[!CAUTION] Blindly removing $third-party from traditional ad blocking rules leads to lots of false-positives.

This is exactly why there is an option to exclude rules - you may need to use it.

<a name="validate"></a> Validate

This transformation is really crucial if you're using a filter list for a traditional ad blocker as a source.

It removes dangerous or incompatible rules from the list.

So here's what it does:

If there are comments preceding the invalid rule, they will be removed as well.

<a name="validate-allow-ip"></a> ValidateAllowIp

This transformation exactly repeats the behavior of Validate, but leaves the IP addresses in the lists.

<a name="deduplicate"></a> Deduplicate

This transformation simply removes the duplicates from the specified source.

There are two important notes about this transformation:

  1. It keeps the original rules order.
  2. It ignores comments. However, if the comments precede the rule that is being removed, the comments will be also removed.

For instance:

! rule1 comment 1
rule1
! rule1 comment 2
rule1

Here's what will be left after the transformation:

! rule1 comment 2
rule1

<a name="invertallow"></a> InvertAllow

This transformation converts blocking rules to "allow" rules. Note, that it does nothing to /etc/hosts rules (unless they were previously converted to adblock-style syntax by a different transformation, for example Compress).

There are two important notes about this transformation:

  1. It keeps the original rules order.
  2. It ignores comments, empty lines, /etc/hosts rules and existing "allow" rules.

Example:

Original list:

! comment 1
rule1

# comment 2
192.168.11.11   test.local
@@rule2

Here's what we will have after applying this transformation:

! comment 1
@@rule1

# comment 2
192.168.11.11   test.local
@@rule2

<a name="removeemptylines"></a> RemoveEmptyLines

This is a very simple transformation that removes empty lines.

Example:

Original list:

rule1

rule2


rule3

Here's what we will have after applying this transformation:

rule1
rule2
rule3

<a name="trimlines"></a> TrimLines

This is a very simple transformation that removes leading and trailing spaces/tabs.

Example:

Original list:

rule1
   rule2
rule3
		rule4

Here's what we will have after applying this transformation:

rule1
rule2
rule3
rule4

<a name="insertfinalnewline"></a> InsertFinalNewLine

This is a very simple transformation that inserts a final newline.

Example:

Original list:

rule1
rule2
rule3

Here's what we will have after applying this transformation:

rule1
rule2
rule3

RemoveEmptyLines doesn't delete this empty row due to the execution order.

<a name="how-to-build"></a> How to build