Awesome
NEWS 2019-Feb-23 - video!
Explaining software v55 video released on youtube:
--> watch
chainhammer v59
TPS measurements of parity aura, geth clique, quorum, tobalaba, etc. It should work with any Ethereum type chain; we focused on PoA consensus.
instructions
video
The brand new release v55 is now presented & explained in a useful video on youtube.
folders
hammer/
- submits many transactions, while watching the recent blocksreader/
- reads blocks; visualizes TPS, blocktime, gas, bytes - see reader/README.mddocs/
- see esp. reproduce.md, cloud.md, FAQ.md, new: azure.mdresults/
- for each client one markdown file;results/runs/
- auto-generated pageslogs/
- check this first if problemsnetworks/
- network starters & external repos via install script, see belowscripts/
- installers and other iseful bash scriptsenv/
- Python virtualenv, created via install script, see belowtests/
- start whole integration test suite via./pytest.sh
chronology
See the results/ folder:
- log.md: initial steps; also tried Quorum's private transactions
- quorum.md: raft consensus, quorum is a geth fork
- tobalaba.md: parity fork of EnergyWebFoundation
- quorum-IBFT.md: IstanbulBFT, 2nd consensus algo in quorum
- geth.md: geth clique PoA algorithm
- parity.md: parity aura PoA algorithm, many attempts to accelerate
- eos.md: not begun yet
- substrate.md: not begun yet
results summary
Outdated table in which I had run each of the experiments manually in autumn 2018; soon re-done completely, using the below automation. So please contact me now, if you know how to accelerate any of these clients:
hardware | node type | #nodes | config | peak TPS_av | final TPS_av |
---|---|---|---|---|---|
t2.micro | parity aura | 4 | (D) | 45.5 | 44.3 |
t2.large | parity aura | 4 | (D) | 53.5 | 52.9 |
t2.xlarge | parity aura | 4 | (J) | 57.1 | 56.4 |
t2.2xlarge | parity aura | 4 | (D) | 57.6 | 57.6 |
t2.micro | parity instantseal | 1 | (G) | 42.3 | 42.3 |
t2.xlarge | parity instantseal | 1 | (J) | 48.1 | 48.1 |
t2.2xlarge | geth clique | 3+1 +2 | (B) | 421.6 | 400.0 |
t2.xlarge | geth clique | 3+1 +2 | (B) | 386.1 | 321.5 |
t2.xlarge | geth clique | 3+1 | (K) | 372.6 | 325.3 |
t2.large | geth clique | 3+1 +2 | (B) | 170.7 | 169.4 |
t2.small | geth clique | 3+1 +2 | (B) | 96.8 | 96.5 |
t2.micro | geth clique | 3+1 | (H) | 124.3 | 122.4 |
t2.micro SWAP | quorum crux IBFT | 4 | (I) SWAP! | 98.1 | 98.1 |
t2.micro | quorum crux IBFT | 4 | (F) | lack of RAM | |
t2.large | quorum crux IBFT | 4 | (F) | 207.7 | 199.9 |
t2.xlarge | quorum crux IBFT | 4 | (F) | 439.5 | 395.7 |
t2.xlarge | quorum crux IBFT | 4 | (L) | 389.1 | 338.9 |
t2.2xlarge | quorum crux IBFT | 4 | (F) | 435.4 | 423.1 |
c5.4xlarge | quorum crux IBFT | 4 | (F) test_getNearestEntry() | 536.4 | 524.3 |
Reproduce these results easily; for the config
column also see there.
Quickest reproduction with my Amazon AMI readymade image.
And see that bottom of parity.md and geth.md
and quorum-IBFT.md for the latest runs, issues, and additional details.
faster wider more
- how I initially got this faster, on Quorum, step by step, please do read the 1st logbook log.md
- then I improved per client, see each in #chronology above
- (possible TODOs - any other ideas?)
but not much more needed = the current version is already fully automated. Use it! May it help you to improve the speed of your Ethereum client!
you
Add yourself to other-projects.md using chainhammer, or projects which are similar to this.
(Especially if you work in one of the dev teams, you know your client code best - ) please try to improve the above results, e.g. by varying the CLI arguments with which the nodes are started; I don't see that as my job, you will be much more successful with that.
See parity PE#9393, parity SE#58521, geth GE#17447, quorum Q#479.
Please report back when you have done other / new measurements.
install and run
All this is developed and much tested on Debian, locally and in the AWS cloud. New: Ubuntu now also supported, see below.
quickstart
N.B.: Better do this on a disposable cloud, or virtualbox machine; because the installation makes lasting changes and needs sudo!
After unpacking a ZIP of the downloaded repo, or by
git clone https://github.com/drandreaskrueger/chainhammer drandreaskrueger_chainhammer
ln -s drandreaskrueger_chainhammer CH
cd CH
you now only need these two lines to prepare and run the 1st experiment!
scripts/install.sh
CH_TXS=1000 CH_THREADING="sequential" ./run.sh $HOSTNAME-TestRPC testrpc
You will then have a diagram, and a HTML and MD page about this run!
(on Ubuntu instead: scripts/install.sh docker ubuntu
)
activate docker
Better now logout & login, or close the terminal, and open a new terminal, because the above scripts/install.sh might have enabled docker for the the first time for this user. Then:
All supported clients in one go:
For the full integration test, run each client for a short moment:
export CH_MACHINE=yourChoice
./run-all_small.sh
For detailed instructions, please see docs/, esp. reproduce.md, and for troubleshooting FAQ.md and github issues.
benchmarking a remote node
Chainhammer can now be stripped down to its pure benchmarking abilities, i.e. without the installation of docker and without the three local network starters (parity-deploy, geth-dev, quorum-crux). It was successfully used to benchmark the Microsoft Azure blockchain-as-a-service product. The essential difference is to start the installation with the switch nodocker
:
scripts/install.sh nodocker
So, if you just want to benchmark your existing Ethereum node or network, have a look at the manual docs/azure.md .
unittests
./pytest.sh
enables the virtualenv,
then starts a testrpc-py
Ethereum simulator on http://localhost:8545 in the background,
logging into tests/logs/
;
then runs ./deploy.py andtests
;
and finally runs all the unittests, also logging into tests/logs/
.
(Instead of testrpc-py) if you want to run tests with another node,
just start that; and run pytest
manually:
source env/bin/activate
py.test -v --cov
There were 98 tests on January 23rd, all 98 PASSED
(see this logfile -->
cat tests/logs/*.ansi
because colors) on these different Ethereum providers:
- testrpc instantseal (
testrpc-py
) 13 seconds - geth Clique (
geth-dev
) 63 seconds - quorum IBFT (
blk-io/crux
) 59 seconds - parity instantseal (
parity-deploy
) 8 seconds - parity aura (
parity-deploy
) 72 seconds
credits
Please credit this as:
benchmarking scripts "chainhammer"
maintainer: Dr Andreas Krueger 2018-2020
https://github.com/drandreaskrueger/chainhammer
Consider to submit your improvements & usage as pull request. Thanks.
development was supported by
v01-v35 financed by Electron.org.uk 2018
v40-v55 financed by Web3Foundation 2018-2019
v58-v59 financed by Microsoft Azure 2019
Thank you very much!
short summary
The open source tools 'chainhammer' submits a high load of smart contract transactions to an Ethereum based blockchain, then 'chainreader' reads the whole chain, and produces diagrams of TPS, blocktime, gasUsed and gasLimit, and the blocksize. https://github.com/drandreaskrueger/chainhammer
# The following diagrams are outdated! Just make your own, new ones, with:
CH_MACHINE=yourChoice ./run-all_large.sh
chainhammer: hammer --> reader --> diagrams
examples:
geth clique on AWS t2.xlarge
geth.md = geth (go ethereum client), "Clique" consensus.
50,000 transactions to an Amazon t2.xlarge machine.
Interesting artifact that after ~14k transactions, the speed drops considerably - but recovers again. Reported.
reader/img/geth-clique-50kTx_t2xlarge_tps-bt-bs-gas_blks12-98.png
quorum IBFT on AWS t2.xlarge
quorum-IBFT.md = Quorum (geth fork), IBFT consensus, 20 millions gasLimit, 1 second istanbul.blockperiod; 20000 transactions multi-threaded with 23 workers. Initial average >400 TPS then drops to below 300 TPS, see quorum issue)
quorum raft
OLD RUN on a desktop machine.
quorum.md = Quorum (geth fork), raft consensus, 1000 transactions multi-threaded with 23 workers, average TPS around 160 TPS, and 20 raft blocks per second)
tobalaba
OLD RUN on a desktop machine.
tobalaba.md = Public "Tobalaba" chain of the EnergyWebFoundation (parity fork), PoA; 20k transactions; > 150 TPS if client is well-connected.
parity aura v1.11.11 on AWS t2.xlarge
parity.md#run-18 = using parity-deploy.sh dockerized network of 4 local nodes with increased gasLimit, and 5 seconds blocktime; 20k transactions; ~ 60 TPS on an Amazon t2.xlarge machine.
N.B.: Could not work with parity v2 yet because of bugs PD#76 and PE#9582 --> everything still on parity v1.11.11
parity-v1.11.11-aura_t2xlarge_tps-bt-bs-gas_blks5-85.png
Calling all parity experts: How to improve these too slow TPS results?
See issue PE#9393, and the detailed log of what I've tried already, and the 2 shortest routes to reproducing the results: reproduce.md.
Thanks.