Home

Awesome

subscrape

This is a growing library of Python facilities to scrape Substrate and Moonbeam-based chains. Substrate chains are scraped using Subscan while EVM-based chains use Moonscan.io and Blockscout.io.

Roadmap: https://app.subsocial.network/@alice_und_bob/subscrape-releasing-v1-0-and-roadmap-for-v2-0-32075

The basic workflow of bin/scrape.py considers the configuration presented in data/scrape_config.json to traverse through the given chains and perform the operations for each chain. For Substrate, scraping extrinsics and transfers is supported.

Data is stored locally using SubscanDB and can be used to use the data and transform it. The application works in a way that subsequent runs will only fetch deltas.

bin/transfers_all_chains.py takes addresses and chains from data/transfers_config.json to create a CSV of all transfers associated with that accounts.

Links

Documentation

Limitations

Error handling is not very sophisticated, so if the scrape is interrupted by an uncaught exception, the delta might be incomplete and subsequent runs might miss some data. To remedy the issue, the delta must be deleted and the scraper run again.

Usage

Installation

virtualenv venv venv\scripts\activate pip install -Ur .\PipRequirements.txt

API Keys

If you have a Subscan API key, drop it in a file named config/subscan-key. If you have a Moonscan.io API key, note that they are network-specific, so place it either in a file named config/moonscan-moonriver-key or config/moonscan-moonbeam-key. Blockscout does not need an API key.

Example applications

Here are several specific examples of how the subscrape library has been used in the past. There are several example application scripts in the /bin/ folder.

Using scrape.py as an application

Consuming scrape.py as a helper