Awesome
Ethereum Light Crawler
This tool uses Ethereum's peer-discovery protocol to measure the size of the Ethereum network (testnets included)
How it works?
The light-crawler
creates an Ethereum node
with a new generated ecdsa Private Key to launch a Discovery5
service.
Once the node and the service are running, the light-crawler will request random ENRs from the network, indexing in an SQL database all the different atributes from the discovered Nodes (we index all the available ENRs in the network, or in other words, all the Ethereum networks).
Requirements
- (recomended) Go1.17+
- (must) make
- (must) local or remote postgreSQL database
How to run it?
Once the repo is cloned, you can compile it and generate a binary running:
$ make build
Once the binary was generated, you can execute it by running:
USAGE:
./build/eth-light-crawler [subcommands] [arguments]
COMMANDS:
discv5 crawl Ethereum's public DHT thought the Discovery 5.1 protocol
help, h Shows a list of commands or help for one command
GLOBAL OPTIONS:
--help, -h show help (default: false)
The current crawler only support the discovery5.1 protocol, so to crawl it your can run the following command:
USAGE:
./build/eth-light-crawler discv5 [arguments...]
OPTIONS:
--log-level value verbosity of the logs that will be displayed [debug,warn,info,error] (default: "info") [$IPFS_CID_HOARDER_LOGLEVEL]
--db-endpoint value login endpoint to the database (default: "postgres://test:password@localhost:5432/eth_light_crawler") [$IPFS_CID_HOARDER_DB_ENDPOINT]
--port value port number that we want to use/advertise in the Ethereum network (default: 9001)
--reset-db reset the content of the db tables (default: false)
--help, -h show help (default: false)
NOTE: the light-crawler
will require to have a postgreSQL database created before running it, it will only create the required tables to run.
Maintainers
Miga Labs / @cortze
Contributions
This tool is open for any kind of feedback and contribution, so please feel free to approach us!