Home

Awesome

<p align="center"> <a href="https://github.com/bda-research/node-crawler"> <img alt="Node.js" src="https://raw.githubusercontent.com/bda-research/node-crawler/master/crawler_primary.png" width="400"/> </a> </p>

npm package

CircleCI NPM download Package Quality

Crawler v2 : Advanced and Typescript version of node-crawler

Features:

If you have prior experience with Crawler v1, for fast migration, please proceed to the section Differences and Breaking Changes.

Quick start

Install

Requires Node.js 18 or above.

IMPORTANT: If you are using a Linux OS, we currently recommend sticking with Node.js version 18 for the time being, rather than opting for higher versions (even if some dependencies suggest 20 or later). Our unit tests have encountered stability issues on Linux with higher versions of Node.js, which may be caused by more profound underlying reasons. However, at present, we do not have the resources to address these issues.

$ npm install crawler

Warning: Given the dependencies involved (Especially migrating from request to got) , Crawler v2 has been designed as a native ESM and no longer offers a CommonJS export. We would also like to recommend that you convert to ESM. Note that making this transition is generally not too difficult.

Usage

Execute asynchronously via custom options

import Crawler from "crawler";

const c = new Crawler({
    maxConnections: 10,
    // This will be called for each crawled page
    callback: (error, res, done) => {
        if (error) {
            console.log(error);
        } else {
            const $ = res.$;
            // $ is Cheerio by default
            //a lean implementation of core jQuery designed specifically for the server
            console.log($("title").text());
        }
        done();
    },
});

// Add just one URL to queue, with default callback
c.add("http://www.amazon.com");

// Add a list of URLs
c.add(["http://www.google.com/", "http://www.yahoo.com"]);

// Add URLs with custom callbacks & parameters
c.add([
    {
        url: "http://parishackers.org/",
        jQuery: false,

        // The global callback won't be called
        callback: (error, res, done) => {
            if (error) {
                console.log(error);
            } else {
                console.log("Grabbed", res.body.length, "bytes");
            }
            done();
        },
    },
]);

// Add some HTML code directly without grabbing (mostly for tests)
c.add([
    {
        html: "<title>This is a test</title>",
    },
]);

please refer to options for detail.

Slow down

Use rateLimit to slow down when you are visiting web sites.

import Crawler from "crawler";

const c = new Crawler({
    rateLimit: 1000, // `maxConnections` will be forced to 1
    callback: (err, res, done) => {
        console.log(res.$("title").text());
        done();
    },
});

c.add(tasks); //between two tasks, minimum time gap is 1000 (ms)

Custom parameters

Sometimes you have to access variables from previous request/response session, what should you do is passing parameters in options.userParams :

c.add({
    url: "http://www.google.com",
    userParams: {
        parameter1: "value1",
        parameter2: "value2",
        parameter3: "value3",
    },
});

then access them in callback via res.options

console.log(res.options.userParams);

Raw body

If you are downloading files like image, pdf, word etc, you have to save the raw response body which means Crawler shouldn't convert it to string. To make it happen, you need to set encoding to null

import Crawler from "crawler";
import fs from "fs";

const c = new Crawler({
    encoding: null,
    jQuery: false, // set false to suppress warning message.
    callback: (err, res, done) => {
        if (err) {
            console.error(err.stack);
        } else {
            fs.createWriteStream(res.options.userParams.filename).write(res.body);
        }
        done();
    },
});

c.add({
    url: "https://raw.githubusercontent.com/bda-research/node-crawler/master/crawler_primary.png",
    userParams: {
        filename: "crawler.png",
    },
});

preRequest

If you want to do something either synchronously or asynchronously before each request, you can try the code below. Note that direct requests won't trigger preRequest.

import Crawler from "crawler";

const c = new Crawler({
    preRequest: (options, done) => {
        // 'options' here is not the 'options' you pass to 'c.queue', instead, it's the options that is going to be passed to 'request' module
        console.log(options);
        // when done is called, the request will start
        done();
    },
    callback: (err, res, done) => {
        if (err) {
            console.log(err);
        } else {
            console.log(res.statusCode);
        }
    },
});

c.add({
    url: "http://www.google.com",
    // this will override the 'preRequest' defined in crawler
    preRequest: (options, done) => {
        setTimeout(() => {
            console.log(options);
            done();
        }, 1000);
    },
});

Direct request

Support both Promise and callback

import Crawler from "crawler";

const crawler = new Crawler();

// When using directly "send", the preRequest won't be called and the "Event:request" won't be triggered
const response = await crawler.send("https://github.com/");
console.log(response.options);
// console.log(response.body);

crawler.send({
    url: "https://github.com/",
    // When calling `send`, `callback` must be defined explicitly, with two arguments `error` and `response`
    callback: (error, response) => {
        if (error) {
            console.error(error);
        } else {
            console.log("Hello World!");
        }
    },
});

Table

Content

Work with Http2

Now we offer hassle-free support for using HTTP/2: just set http2 to true, and Crawler will operate as smoothly as with HTTP (including proxies).

Note: As most developers using this library with proxies also work with Charles, it is expected to set rejectAuthority to false in order to prevent the so-called 'self-signed certificate' errors."

crawler.send({
    url: "https://nghttp2.org/httpbin/status/200",
    method: "GET",
    http2: true,
    callback: (error, response) => {
        if (error) {
            console.error(error);
        }
        console.log(`inside callback`);
        console.log(response.body);
    },
});

Work with rateLimiters

Control the rate limit. All tasks submit to a rateLimiter will abide the rateLimit and maxConnections restrictions of the limiter. rateLimit is the minimum time gap between two tasks. maxConnections is the maximum number of tasks that can be running at the same time. rateLimiters are independent of each other. One common use case is setting different rateLimiters for different proxies. One thing is worth noticing, when rateLimit is set to a non-zero value, maxConnections will be forced to 1.

import Crawler from "crawler";

const c = new Crawler({
    rateLimit: 2000,
    maxConnections: 1,
    callback: (error, res, done) => {
        if (error) {
            console.log(error);
        } else {
            const $ = res.$;
            console.log($("title").text());
        }
        done();
    },
});

// if you want to crawl some website with 2000ms gap between requests
c.add("http://www.somewebsite.com/page/1");
c.add("http://www.somewebsite.com/page/2");
c.add("http://www.somewebsite.com/page/3");

// if you want to crawl some website using proxy with 2000ms gap between requests for each proxy
c.add({
    url: "http://www.somewebsite.com/page/1",
    rateLimiterId: 1,
    proxy: "proxy_1",
});
c.add({
    url: "http://www.somewebsite.com/page/2",
    rateLimiterId: 2,
    proxy: "proxy_2",
});
c.add({
    url: "http://www.somewebsite.com/page/3",
    rateLimiterId: 3,
    proxy: "proxy_3",
});
c.add({
    url: "http://www.somewebsite.com/page/4",
    rateLimiterId: 4,
    proxy: "proxy_1",
});

Normally, all ratelimiters instances in the limiter cluster of crawler are instantiated with options specified in crawler constructor. You can change property of any rateLimiter by calling the code below. Currently, we only support changing property 'rateLimit' of it. Note that the default rateLimiter can be accessed by crawler.setLimiter(0, "rateLimit", 1000);. We strongly recommend that you leave limiters unchanged after their instantiation unless you know clearly what you are doing.

const crawler = new Crawler();
crawler.setLimiter(0, "rateLimit", 1000);

Class: Crawler

Event: 'schedule'

Emitted when a task is being added to scheduler.

crawler.on("schedule", options => {
    options.proxy = "http://proxy:port";
});

Event: 'limiterChange'

Emitted when limiter has been changed.

Event: 'request'

Emitted when crawler is ready to send a request.

If you are going to modify options at last stage before requesting, just listen on it.

crawler.on("request", options => {
    options.searchParams.timestamp = new Date().getTime();
});

Event: 'drain'

Emitted when queue is empty.

crawler.on("drain", () => {
    // For example, release a connection to database.
    db.end(); // close connection to MySQL
});

crawler.add(url|options)

Add a task to queue and wait for it to be executed.

crawler.queueSize

Size of queue, read-only

Options

You can pass these options to the Crawler() constructor if you want them to be global or as items in the crawler.add() calls if you want them to be specific to that item (overwriting global options)

Global only options

silence

maxConnections

priorityLevels

rateLimit

skipDuplicates

homogeneous

userAgents

Crawler General options

url | method | headers | body | searchParams...

forceUTF8

jQuery

encoding

rateLimiterId

retries

retryInterval

timeout

priority

skipEventRequest

html

proxies

const ProxyManager = {
    index: 0,
    proxies: JSON.parse(fs.readFileSync("../proxies.json")),
    setProxy: function (options) {
        let proxy = this.proxies[this.index];
        this.index = ++this.index % this.proxies.length;
        options.proxy = proxy;
        options.rateLimiterId = Math.floor(Math.random() * 15);
    },
};

crawler.on("schedule", options => {
    // options.proxy = "http://127.0.0.1:8000";
    ProxyManager.setProxy(options);
});

proxy

http2

referer

userParams

preRequest

Callback

Work with Cheerio

Crawler by default use Cheerio. We are temporarily no longer supporting jsdom for certain reasons, may be later.

Differences and Breaking Changes

renaming

Options list here are renamed but most of the old ones are still supported for backward compatibility.

Crawler Options

options.priorityRange → options.priorityLevels

options.uri → options.url

options.json → options.isJson (Boolean. The "json" option is now work completely different.)

options.limiter → options.rateLimiterId

options.retryTimeout → options.retryInterval

crawler.direct → crawler.send

crawler.queue → crawler.add

crawler.setLimiterProperty → crawler.setLimiter

Origin Request Options

incomingEncoding → encoding

qs → searchParams

strictSSL → rejectUnauthorized

gzip → decompress

jar → cookieJar (accepts tough-cookie jar)

jsonReviver → parseJson

jsonReplacer → stringifyJson

Behavior Changes

Some practices that were acceptable and offen used in version 1 but not in version 2：

How to test

Crawler uses nock to mock http request, thus testing no longer relying on http server.

$ pnpm test