Awesome

AdGuard Filters Compiler

Filters compiler package is a tool for compiling ad blocking filters into a supported format. It is used in FiltersRegistry.

• Usage
• Tests
• Development
• Filters metadata
• @include directive

Usage

This package is suggested to be used with filters repository with directory structure presented in tests here.

The package could be run with the following command:

const whitelist = [1, 3];
const blacklist = [2];

const path = require('path');
const compiler = require("adguard-filters-compiler");

const filtersDir = path.join(__dirname, './filters');
const logPath = path.join(__dirname, './log.txt');
const reportPath = path.join(__dirname, './report.txt');

const platformsPath = path.join(__dirname, './platforms');

const customPlatformsConfig = {
    // Here you can redefine some of the platforms from platforms.json
    // or add new platforms if you need it.
    "MAC_V3": {
        "platform": "mac",
        "path": "mac_v3",
        "configuration": {
            "ignoreRuleHints": false,
            "removeRulePatterns": [
                "^\\/.*" // remove regex rules for some reason.
            ],
            "replacements": [
                {
                    "from": "regex",
                    "to": "repl"
                }
            ]
        },
        "defines": {
            "adguard": true,
            "adguard_app_mac": true
        }
    },
};

compiler.compile(filtersDir, logPath, reportPath, platformsPath, whitelist, blacklist, customPlatformsConfig);

Tests

yarn test

Development

In order to add support for new scriptlets and redirects, you should update @adguard/tsurlfilter with updated scriptlets.

For fixing scriptlets converting or validation you should update scriptlets.

Filters metadata

Description of the filters metadata is available in the FiltersRegistry repository.

<a name="include-directive"></a> @include directive and its options

The @include directive provides the ability to include content from the specified address.

Syntax:

@include <filepath> [<options>]

where:

• <filepath> — required, URL or same origin relative file path to be included;

• <options> — optional, a list of options separated by spaces. Available options:

  • /stripComments removes AdBlock-style syntax comments from the included file — lines which start with !;
  • /notOptimized adds a !+ NOT_OPTIMIZED hint to the rules;
  • /exclude="<filepath>" excludes from the included file rules listed in the exceptions file available by filepath;
  • /addModifiers="<modifiers>" adds the specified modifiers (string as is) to the rules in the included file. The addModifiers option can also work with the host-rule format files. In this case, host-file comments are to be replaced # by AdBlock-style syntax comments !;
  • /ignoreTrustLevel disables the check of the trust level of the included file. Allowed only for the same origin files.
  • /optimizeDomainBlockingRules remove redundant rules for domain blocking of the included file. Base rules with modifiers and rules of other format will be ignored. Comments are not checked separately and may not be relevant after optimization.

[!IMPORTANT] The content of the included file is formatted by the options due to the order of their mention in the directive, except /ignoreTrustLevel.

Examples

• Include a file with domains, add modifiers to the rules, exclude some rules, add a hint to the rules, and remove comments from the prepared rules:

  @include ../input.txt /addModifiers="script" /exclude="../exclusions.txt" /notOptimized /stripComments /optimizeDomainBlockingRules
  

  The order of execution of the options is as follows:

  1. @include ../input.txt: Includes the content of the file named input.txt from the parent directory.

     # comment
     example.com
     example.org
     

  2. /addModifiers="script": Adds the $script modifier to all rules in the included file.

     Result of adding modifiers:

     ! comment
     example.com$script
     example.org$script
     

     Used to restrict rules with modifiers when blocking the entire domain would result in a breakage. issue example

  3. /exclude="../exclusions.txt": Excludes rules listed in the exception list from the file named exclusions.txt, if they match.

     Due to the content of exclusions.txt:

     example.com$script
     example2.com$script
     

     Result of excluding:

     ! comment
     example.org$script
     

     Used to exclude problematic rules in the filter

  4. /notOptimized: Adds the !+ NOT_OPTIMIZED hint to the rules.

     Result of adding the hint:

     ! comment
     !+ NOT_OPTIMIZED
     example.org$script
     

     Used in cases where the filter is designed for mobile site layout and some rules may be removed, due to the lack of ability to collect statistics on mobile platforms.

  5. /stripComments: Removes comments in AdBlock style from the included file.

     !+ NOT_OPTIMIZED
     example.org$script
     

  6. /optimizeDomainBlockingRules: Remove only domain blocking redundant rules from the included file.

     Due to the optimization:

     ||example.com^
     ||sub.example.com^
     ||domain.com^
     ||test.com^$script
     

     Result of optimization:

     ||example.com^
     ||domain.com^
     ||test.com^$script
     

• Ignore the trust level of the filter list (specified in the metadata) during the file including — include the file rules as is:

  @include ./input.txt /ignoreTrustLevel