Home

Awesome

Robot Framework Solidity Testing Toolkit

Toolkit Checks

Security Checks

CodeQL

Alchemy

Robot

Disclaimer

By using this software you understand the risks of Web3 and blockchain technologies. This toolkit is intended for testing and learning exciting concepts in the Web3 and blockchain industry with a security-minded self-education approach. The blockchain and DeFi ecosystems do not show mercy in any way to even the smallest mistakes. I recommend that you self-educate yourself first and read MetaMask's security knowledge base before experimenting with this repository. You are responsible for your own personal cryptocurrency funds and Web3 wallet private keys, and it is your personal responsibility to educate yourself to protect them. Any misuse or mishandling of this software on any of the EVM compatible blockchain mainnets or testnets is at your own risk.

Also just a friendly warning, DO NOT ATTEMPT TO SEND FUNDS to any of the 20 test accounts provided by Hardhat.

Ethereum set the standard, but cross-chain is the future

The capabilities of the Solidity programming language and the Ethereum platform in general are vast and growing every day. One of the common usecases for Ethereum has been to approach it as a gigantic Web3 backend system. The very simple OpenZeppelin ERC20 TestToken and other smart contract examples included in this toolkit are only tiny slices of the bigger picture in comparison to what is being used today in Web3 and DeFi production environments. In general when you approach people on the street and they hear the word Ethereum, the reactions you would probably get from them is "oh that's the thing I can buy on Coinbase or Binance and send to my parents on the other side of the world" or "that's the blockchain network that has all those worthless poo-coins right?". In reality, Ethereum was built for much immensely grander designs than the simple monetary usecases that most people are familiar with. Vitalik Buterin and the other Ethereum co-founders originally sought to create a "World Computer". As you browse through this documentation you will notice that I am not attempting to try to create "generic keywords" for the Solidity language or the "World Computer" that smart contracts are deployed to, and you will see a lot of mostly non-generic transparent strategies in this repository. As a busy person who is trying to actively learn blockchain technologies and is worried about the security of his own Web3 wallet (also concerned for the safety of all of yours as well), I will leave the generic blockchain automation keyword development for others to pursue. An important trend in the Ethereum ecosystem to pay more attention to is the rising popularity of bridges into many other types of blockchains, and the current Web3 protocols that are evolving towards cross-chain communication. I have usecases of my own for cross-chain EVM (Ethereum Virtual Machine) projects and prefer to focus on my ideas for this toolkit, but I wanted to share my high-level strategies here (also discussed with TestGuild's Joe Colantonio on YouTube) and possibly hear your thoughts too if you want to reach out to me on Robot Framework's Slack Group.

Solidity clean-room environment testing is safe and repeatable

This multichain toolkit contains keyword-driven automation that helps locally test EVM compatible Solidity smart contracts, deploys them using a multichain approach, and has the basic building blocks for creating blockchain event monitoring bots. One of the primary components is a custom-made robotframework-hardhat-remote-library.js that was built using Hardhat and ethers.js libraries for Web3 development, and combined together using comick's node-robotremoteserver. This toolkit has been designed to create local clean-room environment mainnet forks through Hardhat Network for deploying smart contracts on multiple types of blockchains that support the EVM. I have included small checks and basic RPA automation scripts that work for Ethereum, Fantom, Avalanche, Polygon, but they can be adapted to cover many others as well by simply adjusting minor Hardhat CLI options and altering the provided config files. Also all of the automated checks in this repository are spending ZERO actual gas because they are mostly reading or interacting with simulated blockchains running on a local machine, and not sending testnet transactions (scroll down to see risks of frequently doing that) or touching anything on mainnets.

"Ethereum is a Dark Forest" - Dan Robinson and Georgios Konstantopoulos

You may be wondering. What's so special about Hardhat Network, Trufflesuite's Ganache, and "clean-room environment testing"? Isn't testing on any of the Ethereum Testnets enough since ETH on a testnet has zero real-world value? Technically all blockchain dApps and complex integrations with multiple existing smart contracts should always be deeply tested with testnets first before deploying them to a mainnet, but relying ONLY on testnets for testing is not always a good idea. For example, here are some disadvantages and possibly severe risks for testing on both Ethereum Testnets and Ethereum Mainnet.

Some Testnet testing disadvantages and risks:

Some Mainnet testing disadvantages and risks:

Stay vigilant and protect yourself:

There will be special cases and complex projects where frequent testing and smart contract deployments to Ethereum Testnet are unavoidable. I recommend setting up a "testing-only Metamask wallet" on a different browser (or maybe on a completely different laptop if possible), and also look into investing in a high-quality and well tested VPN service. Always assume you are exposed and try your best to protect your internet connection and your personal workstation from exposure to scammers, malicious hackers, and other bad actors that are frequently found in the blockchain industry. Also don't let this dark side of the industry scare you away from these exciting technologies. Once you set up proper protections and establish good practices you should be ok.

Why Robot Framework? Why not MochaJS etc.?

MochaJS is a good and flexible test framework, but it's not easy to figure out how it can handle the following types of tests...

A lot of the automation in this repository is influenced by my Robot-Framework-Lone-Tester-Strategies-RoboCon-2019, Tool-Strategies-Lone-Testers-Test-Leadership-Congress-2019, and Getting-Started-Robotframework-AppiumLibrary-RoboCon-2021-And-2022 workshops. More information about these RoboCon workshops can be found on robocon.io or watch this YouTube video (click here) to see an example. Also just a general side note, I created this toolkit to demonstrate that there is more than one way to approach Web3, dApp, or blockchain-related testing and to encourage QA professionals who are interested in RPA tools. I also intended this to help suggest ideas to bridge Robot Framework into Web3 and blockchain, as a means to experiment with new or cutting edge testing technologies focused on the security of EVM compatible smart contracts.

Toolkit Roadmap

In the future I plan to expand this toolkit into the following areas.

Technical Requirements

The following are the basic technical requirements needed to run these automated checks. Please note that this entire toolkit was developed on a MacOS machine, but should also work for most Linux users (preferably most current Ubuntu LTS versions etc.) and I would keep an eye on whatever the Hardhat or Truffle (source: https://github.com/trufflesuite/ganache-ui/releases) communities recommend for Linux distributions.

Quick Start Guide

Before running any of the automation first make sure that all of the basic technical requirements are installed and working, then you can run bash ./start-solidity-qa-workflows.sh Install-Tools-On-MacOS-Or-Linux. After the installation script successfully finishes you should be able to run any of the following commands without problems.

Current Toolkit Capabilities

You can view just this help menu again (without triggering any automation) by running 'bash ./start-solidity-qa-workflows.sh -h' or 'bash ./start-solidity-qa-workflows.sh --help'.

---->>>> Local Solidity Test Environment Setup Commands <<<<----
bash ./start-solidity-qa-workflows.sh Install-Tools-On-MacOS-Or-Linux
bash ./start-solidity-qa-workflows.sh Install-Solidity-Security-Testing-Tools-On-MacOS
bash ./start-solidity-qa-workflows.sh Start-Default-Hardhat-Network-And-Robotremoteserver
bash ./start-solidity-qa-workflows.sh Start-Multichain-Hardhat-Network-And-Robotremoteserver
bash ./start-solidity-qa-workflows.sh Start-Default-Truffle-Develop-And-Robotremoteserver
bash ./start-solidity-qa-workflows.sh Start-Default-Foundry-Anvil-Node-And-Wagmi-dApp
bash ./start-solidity-qa-workflows.sh Start-Forked-Foundry-Anvil-Node-And-Wagmi-dApp
bash ./start-solidity-qa-workflows.sh Stop-Local-Blockchain-Nodes-And-Delete-Logs

---->>>> Solidity Static Analysis And Security Testing Commands <<<<----
bash ./start-solidity-qa-workflows.sh Run-Solidity-Static-Analysis

---->>>> Solidity Vulnerability Scanning Security Testing Commands <<<<----
bash ./start-solidity-qa-workflows.sh Stop-And-Clean-Up-Docker-Security-Tests
bash ./start-solidity-qa-workflows.sh Run-Parallel-Datadriven-Smart-Contract-Security-Tests

---->>>> Alchemy SDK and Infura Websocket Docker Container Bots Powered by Robot Framework RPA <<<<----
bash ./start-solidity-qa-workflows.sh Stop-And-Clean-Up-Docker-Bot-RPA-Tasks
bash ./start-solidity-qa-workflows.sh Run-Alchemy-SDK-Bot-RPA-Tasks
bash ./start-solidity-qa-workflows.sh Run-Infura-Websocket-Bot-RPA-Tasks

---->>>> Hardhat Deployment Commands Powered by Robot Framework RPA (NOTE: These scripts are capable of deploying to Mainnets or Testnets!) <<<<----
bash ./start-solidity-qa-workflows.sh Run-Smart-Contract-Hardhat-Ethereum-RPA-Deployment
bash ./start-solidity-qa-workflows.sh Run-Smart-Contract-Hardhat-Fantom-RPA-Deployment
bash ./start-solidity-qa-workflows.sh Run-Smart-Contract-Hardhat-Polygon-RPA-Deployment
bash ./start-solidity-qa-workflows.sh Run-Smart-Contract-Hardhat-Avalanche-RPA-Deployment

---->>>> Truffle Deployment Commands Powered by Robot Framework RPA (NOTE: These scripts are capable of deploying to Mainnets or Testnets!) <<<<----
bash ./start-solidity-qa-workflows.sh Run-Smart-Contract-Truffle-Ethereum-RPA-Deployment

---->>>> Interactive Tests For Deployed Smart Contracts Using Hardhat (NOTE: These tests can run on multiple EVM Compatible Blockchains!) <<<<----
bash ./start-solidity-qa-workflows.sh Run-Smart-Contract-Tests-Default-NonForked-Hardhat-Network
bash ./start-solidity-qa-workflows.sh Run-Smart-Contract-Tests-Forked-Mainnet-Hardhat-Network 

---->>>> Interactive Tests For Deployed Smart Contracts Using Truffle Suite <<<<----
bash ./start-solidity-qa-workflows.sh Run-Smart-Contract-Tests-Default-Truffle-Develop-Instance

---->>>> Interactive Tests For Deployed Smart Contracts Using Foundry <<<<----
bash ./start-solidity-qa-workflows.sh Run-Parallel-Smart-Contract-Foundry-And-Playwright-Browser-Tests

[Example Workflow 1] -> Deploy then test a smart contract using the built-in Hardhat Network Ethereum Node and node-robotremoteserver (on MacOS or Linux)

From the commands posted above, here is an example workflow (with screenshots) below that will work reliably as long as both the basic technical requirements and the bash ./start-solidity-qa-workflows.sh Install-Tools-On-MacOS-Or-Linux command have been completed without issues.

[Example Workflow 2] -> Fork Fantom Mainnet, deploy, then test an existing smart contract using the Hardhat Network Fantom fork and node-robotremoteserver (on MacOS or Linux)

From the commands posted above, here is an example workflow (with screenshots) below that will work reliably as long as both the basic technical requirements and the bash ./start-solidity-qa-workflows.sh Install-Tools-On-MacOS-Or-Linux command have been completed without issues.

[Example Workflow 3] -> Use Robot Framework RPA to do only the deployment using Hardhat (on MacOS or Linux)

From the commands posted above, here is an example workflow below that will work reliably as long as both the basic technical requirements and the bash ./start-solidity-qa-workflows.sh Install-Tools-On-MacOS-Or-Linux command have been completed without issues.

Results for the example workflows above can be found in the ...-log.html files stored in the logs folder.

Windows User Advice

All Windows users should please read my NOTES.md file for more details.