Home

Awesome

auction-keeper

Build Status codecov

The DAI Stablecoin System incentivizes external agents, called keepers, to automate certain operations around the Ethereum blockchain. The purpose of auction-keeper is to:

Check out the <a href="https://youtu.be/wevzK3ADEjo?t=733">July 23rd, 2019 community meeting</a> for some more information about MCD auctions and the purpose of this component.

auction-keeper can participate in clip and flip (collateral sale), flap (MKR buy-and-burn) and flop (MKR minting) auctions. Its unique feature is the ability to plug in external bidding models, which tell the keeper when and how high to bid. This keeper can be safely left running in background. The moment it notices or starts a new auction it will spawn a new instance of a bidding model for it and then act according to its instructions. Bidding models will be automatically terminated by the keeper the moment the auction expires. The keeper also automatically deals expired auctions if it's us who won them.

This keeper is intended to be a reference implementation. It may be used as-is, or pieces borrowed to develop your own auction trading bot.

https://chat.makerdao.com/channel/keeper

Architecture

auction-keeper directly interacts with Clipper, Flipper, Flapper and Flopper auction contracts deployed to the Ethereum blockchain. Decisions which involve pricing are delegated to bidding models.

Bidding models are simple processes, external to the main auction-keeper process. As they do not have to know anything about blockchain and smart contracts, they can be implemented in basically any programming language. The only thing they need to do is to read and write JSON documents they exchange with auction-keeper. The simplest example of a bidding model is a shell script which echoes a fixed price.

Monitoring ongoing auctions and discovering new ones

The main task of this keeper, as already outlined above, is to constantly monitor all ongoing auctions, discover new ones, ensure that an instance of bidding model is running for each auction, provide these instances of the current status of their auctions and bid according to decisions taken by them.

The way the auction discovery and monitoring mechanism works at the moment is simplistic for illustration purposes. It basically operates as a loop which kicks in on every new block enumerating all auctions from 1 to kicks. Bidding models are checked periodically and submitted where appropriate.

Starting and stopping bidding models

auction-keeper maintains a collection of child processes, as each bidding model is its own dedicated process. New processes (new bidding model instances) are spawned by executing a command according to the --model commandline parameter. These processes are automatically terminated (via SIGKILL) by the keeper shortly after their associated auction expires.

Whenever the bidding model process dies, it gets automatically respawned by the keeper.

Example:

bin/auction-keeper --model '../my-bidding-model.sh' [...]

Communicating with bidding models

auction-keeper communicates with bidding models via their standard input and standard output.

Straight away after the process gets started, and every time the auction state changes, the keeper sends a one-line JSON document to the standard input of the bidding model process. Sample message sent from the keeper to the model looks like:

{"id": "6", "flapper": "0xf0afc3108bb8f196cf8d076c8c4877a4c53d4e7c", "bid": "7.142857142857142857", "lot": "10000.000000000000000000", "beg": "1.050000000000000000", "guy": "0x00531a10c4fbd906313768d277585292aa7c923a", "era": 1530530620, "tic": 1530541420, "end": 1531135256, "price": "1400.000000000000000028"}

The meaning of individual fields:

Bidding models should never make an assumption that messages will be sent only when auction state changes. It is perfectly fine for the auction-keeper to periodically send the same messages to bidding models.

At the same time, the auction-keeper reads one-line messages from the standard output of the bidding model process and tries to parse them as JSON documents. Then it extracts two fields from that document:

A sample message sent from the model to the keeper may look like:

{"price": "750.0"}

Whenever the keeper and the model communicate in terms of prices, it is the MKR/DAI price (for flap and flop auctions) or the collateral price expressed in DAI e.g. DGX/DAI (for clip or flip auctions).

Any messages writen by a bidding model to stderr will be passed through by the keeper to its logs. This is the most convenient way of implementing logging from bidding models.

No facility is provided to prevent you from bidding an unprofitable price. Please ensure you understand how your model produces prices and how prices are consumed by the keeper for each of the auction types in which you participate.

Simplest possible bidding model

If you just want to bid a fixed price for each auction, this is the simplest possible bidding model you can use:

#!/usr/bin/env bash

while true; do
  echo "{\"price\": \"723.0\"}" # put your desired price amount here
  sleep 120                      # locking the price for n seconds
done

The stdout provides a price for the collateral (for clip and flip auctions) or MKR (for flap and flop auctions). The sleep locks the price in place for the specified duration, after which the keeper will restart the price model and read a new price.
Consider this your price update interval. To conserve system resources, take care not to set this too low.

Other bidding models

Thanks to our community for these examples:

Limitations

Installation

This project uses Python 3.6.6.

In order to clone the project and install required third-party packages please execute:

git clone https://github.com/makerdao/auction-keeper.git
cd auction-keeper
git submodule update --init --recursive
pip3 install -r requirements.txt

For some known Ubuntu and macOS issues see the pymaker README.

Usage

Run bin/auction-keeper -h without arguments to see an up-to-date list of arguments and usage information.

To participate in all auctions, a separate keeper must be configured for clip or flip of each collateral type, as well as one for flap and another for flop. Collateral types (ilks) combine the name of the token and a letter corresponding to a set of risk parameters. For example, ETH-A and ETH-B are two different collateral types for the same underlying token (WETH). Regardless of whether a keeper is configured for --clip or --flip, the keeper will choose the appropriate collateral liquidation contract for the specified --ilk and addresses configured in pymaker.

Configure --from-block to the block where MCD was deployed. One way to find this is to look at the MCD_DAI contract of the deployment you are using and determine the block in which it was deployed.

example list of keepers

Please note collateral types in the table above are provided for illustrative purposes, and should not be interpreted as an endorsement of which collaterals should be deployed to mainnet, which will be determined by an appropriate governance process. A complete list of ilks for a deployment may be gleaned from the addresses.json.

Gas price strategy

Auction keeper can use one of several sources for the initial gas price of a transaction:

If neither --oracle-gas-price nor --fixed-gas-price is configured, or if gas oracles are not producing prices, the keeper will choose a starting gas price determined by your node. When not using --fixed-gas-price, --gas-initial-multiplier (default 1.0, or 100%) allows you to configure a more aggressive initial gas price.

Auction keeper periodically attempts to increase gas price when transactions are queueing. After a few blocks, a transaction's gas price will be multiplied by --gas-reactive-multiplier (default 1.125, an increase of 12.5%) until it is mined or --gas-maximum (default 2000 Gwei) is reached.
Note that Parity, as of this writing, requires a minimum gas increase of 1.125 to propagate transaction replacement; this should be treated as a minimum value unless you want replacements to happen less frequently.

This gas strategy is used by keeper in all interactions with chain. When sending a bid, this strategy is used only when the model does not provide a gas price. Unless your price model is aware of your transaction status, it is generally advisable to allow the keeper to manage gas prices for bids, and not supply a gasPrice in your model.

Accounting

Key points:

The keeper provides facilities for managing Vat balances, which may be turned off to manage manually. To manually control the amount of Dai in the Vat, pass --keep-dai-in-vat-on-exit and --keep-gem-in-vat-on-exit, set --return-gem-interval 0, and do not pass --vat-dai-target.

Warnings: Do not use an eth-from account on multiple keepers as it complicates Vat inventory management and will likely cause nonce conflicts. Using an eth-from account with an open vault is also discouraged.

Dai

All auction contracts exclusively interact with Dai (for all auctions) in the Vat. --vat-dai-target may be set to the amount you wish to maintain, or all to join your account's entire token balance. Rebalances do not account for Dai moved from the Vat to an auction contract for an active bid. Dai is rebalanced per --vat-dai-target when:

To avoid transaction spamming, small "dusty" Dai balances will be ignored (until the keeper exits, if so configured).
By default, all Dai in your eth-from account is exited from the Vat and added to your token balance when the keeper is terminated normally. This feature may be disabled using --keep-dai-in-vat-on-exit.

Collateral (clip and flip auctions)

Won collateral is periodically exited by setting --return-gem-interval to the number of seconds between balance checks. Collateral is exited from the Vat when the keeper is terminated normally unless --keep-gem-in-vat-on-exit is specified.

Other tools

Alternatively, mcd-cli can be used to manually manage Vat balances. Here is an example to join 6000 Dai on a testnet, and exit 300 Dai on Kovan, respectively:

mcd -C testnet dai join 6000
mcd -C kovan dai exit 300

mcd-cli requires installation and configuration; view the mcd-cli README for more information.

Managing resources

Minimize load on your node

To start clip and flip auctions, the keeper needs a list of urns and the collateralization ratio of each urn.
There are two ways to retrieve the list of urns:

To start flop auctions, the keeper needs a list of bites to queue debt. To manage performance, periodically adjust --from-block to the block where the first bite which has not been flogged.

The --min-auction argument arbitrarily ignores older completed auctions, such that the keeper needn't check their status. The --max-auctions argument allows you to limit the number of bidding models created to handle active auctions. Both switches help reduce the number of requests (not just transactions) made to the node.

Transaction management

Bid management can be sharded across multiple keepers by auction id. To do this, configure --shards with the number of keepers you will run, and a separate --shard-id for each keeper, counting from 0. For example, to configure three keepers, set --shards 3 and assign --shard-id 0, --shard-id 1, --shard-id 2 for the three keepers. Kicks are not sharded; for an auction contract, only one keeper should be configured to kick.

If you are sharding across multiple accounts, you may wish to have another account handle all your deals. The --deal-for argument allows you to specify a space-delimited list of accounts for which you'll deal auctions. You may disable dealing auctions by specifying --deal-for NONE in each of your shards. If you'd like to donate your gas to deal auctions for all participants, --deal-for ALL is also supported. Unlike kicks, deals are sharded, so remove sharding configuration if running a dedicated deal keeper.

Too many pending transactions can fill up the transaction queue, causing a subsequent transaction to be dropped. By waiting a small --bid-delay after each bid, multiple transactions can be submitted asynchronously while still allowing some time for older transactions to complete, freeing up the queue. Many parameters determine the appropriate amount of time to wait. For illustration purposes, assume the queue can hold 12 transactions, and gas prices are reasonable. In this environment, a bid delay of 1.2 seconds might provide ample time for transactions at the front of the queue to complete. Etherscan.io can be used to view your account's pending transaction queue.

Upon startup, the keeper will clear its transaction queue. This helps recover from insufficiently-aggressive gas configuration and reduces gas-wasting transactions.

Hardware and operating system resources

Infrastructure

This keeper connects to the Ethereum network using Web3.py and interacts with the Dai Stablecoin System (DSS) using pymaker. A connection to an Ethereum node (--rpc-host) is required. Parity and Geth nodes are supported over HTTP. Web3.py's WebsocketProvider does not support multiple threads awaiting a response from the websocket. A full or archive node is required; light nodes are not supported.

If you don't wish to run your own Ethereum node, third-party providers are available. This software has been tested with ChainSafe. When using Infura and kicking off clip or flip auctions, reduce --chunk-size to 1/10th of the default.

Testing

This project uses pytest for unit testing. Testing depends upon on a Dockerized local testchain included in lib\pymaker\tests\config.

In order to be able to run tests, please install development dependencies first by executing:

pip3 install -r requirements-dev.txt

You can then run all tests with:

./test.sh

License

See COPYING file.

Disclaimer

YOU (MEANING ANY INDIVIDUAL OR ENTITY ACCESSING, USING OR BOTH THE SOFTWARE INCLUDED IN THIS GITHUB REPOSITORY) EXPRESSLY UNDERSTAND AND AGREE THAT YOUR USE OF THE SOFTWARE IS AT YOUR SOLE RISK. THE SOFTWARE IN THIS GITHUB REPOSITORY IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. YOU RELEASE AUTHORS OR COPYRIGHT HOLDERS FROM ALL LIABILITY FOR YOU HAVING ACQUIRED OR NOT ACQUIRED CONTENT IN THIS GITHUB REPOSITORY. THE AUTHORS OR COPYRIGHT HOLDERS MAKE NO REPRESENTATIONS CONCERNING ANY CONTENT CONTAINED IN OR ACCESSED THROUGH THE SERVICE, AND THE AUTHORS OR COPYRIGHT HOLDERS WILL NOT BE RESPONSIBLE OR LIABLE FOR THE ACCURACY, COPYRIGHT COMPLIANCE, LEGALITY OR DECENCY OF MATERIAL CONTAINED IN OR ACCESSED THROUGH THIS GITHUB REPOSITORY.