Home

Awesome

DuckDuckGo Tracker Radar Collector

🕸 Modular, multithreaded, puppeteer-based crawler used to generate third party request data for the Tracker Radar.

How do I use it?

Use it from the command line

  1. Clone this project locally (git clone git@github.com:duckduckgo/tracker-radar-collector.git)
  2. Install all dependencies (npm i)
  3. Run the command line tool:
npm run crawl -- -u "https://example.com" -o ./data/ -v

Available options:

Use it as a module

  1. Install this project as a dependency (npm i git+https://github.com:duckduckgo/tracker-radar-collector.git).

  2. Import it:

// you can either import a "crawlerConductor" that runs multiple crawlers for you
const {crawlerConductor} = require('tracker-radar-collector');
// or a single crawler
const {crawler} = require('tracker-radar-collector');

// you will also need some data collectors (/collectors/ folder contains all build-in collectors)
const {RequestCollector, CookieCollector, …} = require('tracker-radar-collector');
  1. Use it:
crawlerConductor({
    // required ↓
    urls: ['https://example.com', {url: 'https://duck.com', dataCollectors: [new ScreenshotCollector()]}, …], // two formats available: first format will use default collectors set below, second format will use custom set of collectors for this one url
    dataCallback: (url, result) => {…},
    // optional ↓
    dataCollectors: [new RequestCollector(), new CookieCollector()],
    failureCallback: (url, error) => {…},
    numberOfCrawlers: 12,// custom number of crawlers (there is a hard limit of 38 though)
    logFunction: (...msg) => {…},// custom logging function
    filterOutFirstParty: true,// don't save any first-party data (false by default)
    emulateMobile: true,// emulate a mobile device (false by default)
    proxyHost: 'socks5://myproxy:8080',// SOCKS proxy host (none by default)
    antiBotDetection: true,// if anti bot detection script should be injected (true by default)
    chromiumVersion: '843427',// Chromium version that should be downloaded and used instead of the default one
    maxLoadTimeMs: 30000,// how long should crawlers wait for the page to load, defaults to 30s
    extraExecutionTimeMs: 2500,// how long should crawlers wait after page loads before collecting data, defaults to 2.5s
});

OR (if you prefer to run a single crawler)

// crawler will throw an exception if crawl fails
const data = await crawler(new URL('https://example.com'), {
    // optional ↓
    collectors: [new RequestCollector(), new CookieCollector(), …],
    log: (...msg) => {…},
    urlFilter: (url) => {…},// function that, for each request URL, decides if its data should be stored or not
    emulateMobile: false,
    emulateUserAgent: false,// don't use the default puppeteer UA (default true)
    proxyHost: 'socks5://myproxy:8080',
    browserContext: context,// if you prefer to create the browser context yourself (to e.g. use other browser or non-incognito context) you can pass it here (by default crawler will create an incognito context using standard chromium for you)
    runInEveryFrame: () => {window.alert('injected')},// function that should be executed in every frame (main + all subframes)
    executablePath: '/some/path/Chromium.app/Contents/MacOS/Chromium',// path to a custom Chromium installation that should be used instead of the default one
    maxLoadTimeMs: 30000,// how long should the crawler wait for the page to load, defaults to 30s
    extraExecutionTimeMs: 2500,// how long should crawler wait after page loads before collecting data, defaults to 2.5s
});

ℹ️ Hint: check out crawl-cli.js and crawlerConductor.js to see how crawlerConductor and crawler are used in the wild.

Output format

Each successfully crawled website will create a separate file named after the website (when using the CLI tool). Output data format is specified in crawler.js (see CollectResult type definition). Additionally, for each crawl metadata.json file will be created containing crawl configuration, system configuration and some high-level stats.

Data post-processing

Example post-processing script, that can be used as a template, can be found in post-processing/summary.js. Execute it from the command line like this:

node ./post-processing/summary.js -i ./collected-data/ -o ./result.json

ℹ️ Hint: When dealing with huge amounts of data you may need to increase nodejs's memory limit e.g. node --max_old_space_size=4096.

Creating new collectors

Each collector needs to extend the BaseCollector and has to override following methods:

Additionally, each collector can override following methods:

There are couple of built-in collectors in the collectors/ folder. CookieCollector is the simplest one and can be used as a template.

Each new collector has to be added in two places to be discoverable:

You can also add types to define the structure of the data exported by your collector. These should be added to the CollectorData type in collectorsList.js. This will add type hints to all places where the data is used in the code.