Awesome
Scrape
CLI utility to scrape emails from websites
Features
- Asynchronous scraping
- Recursive link follow
- External link follow
- Cloudflare email obfuscation decoding
- Client side rendered pages support through headless
chromium
load awaits - Simple, grepable output
Install
-
MacOS:
brew tap lawzava/scrape https://github.com/lawzava/scrape brew install scrape
-
Linux:
sudo snap install scrape
Usage
Sample call:
scrape -w https://lawzava.com
Depends on chromium
or google-chrome
being available in path if --js
is used
Parameters:
--async Scrape website pages asynchronously (default true)
--debug Print debug logs
-d, --depth int Max depth to follow when scraping recursively (default 3)
--follow-external Follow external 3rd party links within website
-h, --help help for scrape
--js Enables EnableJavascript execution await
--output string Output type to use (default 'plain', supported: 'csv', 'json') (default "plain")
--output-with-url Adds URL to output with each email
--recursively Scrape website recursively (default true)
--timeout int If > 0, specify a timeout (seconds) for js execution await
-w, --website string Website to scrape (default "https://lawzava.com")
Note about scraper package
For those that are looking for scraper
package - this repository was intended as a cli-use only thus the scraper package was moved to lawzava/emailscraper.
The scrape
utility will be maintained as a CLI implementation of emailscraper
package.