Home

Awesome

DEX live trading strategy deployment example

This is an example repository for setting up a docker-compose.yml for a minimal live trade-executor.

Preface

This is an example repository to do partial deployment for live trading strategies.

You need

This preproduction set up can be more straightforward than the actual production deployment, as we shortcut a few things here:

Step 1: Develop a trading strategy

See Getting started if you do not have a trading strategy yet.

The usual deliverables of developing a trading strategy include:

  1. Initial backtest notebook
  2. Optimiser that crunches through multiple parameters
  3. Final backtest notebook with fixed parameters you are prepared to take to the live trading

This repository comes with a simple example ETH-USDC strategy.

For this example we have

  1. Initial backtest - somewhere in Getting started repo
  2. Optimiser notebook. See here
  3. Final backtest. See here

Note: Github does not view notebooks fully - you need to open them locally.

Note: This strategy might be too simple and overfit for real-world trading.

Step 2: Give the strategy id

You are going to have a lot of strategies. You need to have a systematic way to keep track of them.

This example uses id hotwallet-polygon-eth-usdc-breakout. It's a mouthful, but self-explanatory.

This id is used in

Step 3: Extract strategy as a Python module

We convert the final backtest notebook to a Python module. We use (3) from the step above as the starting point.

Rules to convert

trading_strategy_engine_version = "0.5"
name = "ETH-BTC-USDC momentum"  # Optional: Frontend metadata
tags = {StrategyTag.beta, StrategyTag.live}  # Optional: Frontend metadata
icon = ""  # Optional: Frontend metadata
short_description = ""  # Optional: Frontend metadata
long_description = ""  # Optional: Frontend metadata

We also edit the strategy to dynamically decide which trading universe to use

Example modification here:

def get_strategy_trading_pairs(execution_mode: ExecutionMode) -> list[HumanReadableTradingPairDescription]:
    if execution_mode.is_backtesting():
        # Need longer history
        trading_pairs = [
            (ChainId.centralised_exchange, "binance", "ETH", "USDT"),
        ]

    else:
        # For live trading we do Uniswap v3 on Polygon
        trading_pairs = [
            (ChainId.polygon, "uniswap-v3", "WETH", "USDC", 0.0005),
        ]
    return trading_pairs


def create_trading_universe(
    timestamp: datetime.datetime,
    client: Client,
    execution_context: ExecutionContext,
    universe_options: UniverseOptions,
) -> TradingStrategyUniverse:
    """Create the trading universe."""
    trading_pairs = get_strategy_trading_pairs(execution_context.mode)

    if execution_context.mode.is_backtesting():
        # Backtesting - load Binance data to get longer history
        strategy_universe = create_binance_universe(
            [f"{p[2]}{p[3]}" for p in trading_pairs],
            candle_time_bucket=Parameters.candle_time_bucket,
            stop_loss_time_bucket=Parameters.stop_loss_time_bucket,
            start_at=universe_options.start_at,
            end_at=universe_options.end_at,
            forward_fill=True,
        )        

    else:
        # How many bars of live trading data needed
        required_history_period = Parameters.required_history_period

        dataset = load_partial_data(
            client=client,
            time_bucket=Parameters.candle_time_bucket,
            pairs=trading_pairs,
            execution_context=execution_context,
            universe_options=universe_options,
            liquidity=False,
            stop_loss_time_bucket=Parameters.stop_loss_time_bucket,
            required_history_period=required_history_period,
        )
        # Construct a trading universe from the loaded data,
        # and apply any data preprocessing needed before giving it
        # to the strategy and indicators
        strategy_universe = TradingStrategyUniverse.create_from_dataset(
            dataset,
            reserve_asset="USDC",
            forward_fill=True,
        )

    return strategy_universe

Step 4: Set up docker-compose.yml entry

Note: We use new style docker compose commands in this README, instead of docker-compose legacy command (with dash). Make sure you have the latest version.

Step 5: Set up an environment file

Each trade executor docker is configured using environment variable files using the normal Docker conventions.

Note: For the production deployment it is recommend against putting any highly sensitive material like private keys directly to these files/

The file comes with

Step 6: Launching Docker for the first time

Step 6a: Set Github container registry access

Docker images are distributed on Github Container Registry ghcr.io. The access is public, but you need to have an access token through your Github account.

To enable docker login to Github see how to set up Github access token to download Docker images from GHCR.

When you find your token you can do:

GITHUB_USERNAME=miohtama
# Your Personal Access Token (classic)
CR_PAT=ghp_mmc...
# This will save a config file locally with your GHCR access key
echo $CR_PAT | docker login ghcr.io -u $GITHUB_USERNAME --password-stdin

Docker login should reply:

Login Succeeded

Step 6b: Test Docker container start

We try to run and verify our Docker Compose launches.

First we need to choose the trade-executor release. The trade executor is under active development and may see multiple releases per day. You can find releases on Github

# Get the latest trade-executor version tag from Github to environment variable
source scripts/export-latest-trade-executor-version.sh

# Launch to show the command line help
docker compose run hotwallet-polygon-eth-usdc-breakout --help

You should see the command line help for trade-executor command:

Usage: trade-executor [OPTIONS] COMMAND [ARGS]...

Options:
  --install-completion [bash|zsh|fish|powershell|pwsh]
                                  Install completion for the specified shell.
  --show-completion [bash|zsh|fish|powershell|pwsh]
                                  Show completion for the specified shell, to copy it or customize the installation.
  --help                          Show this message and exit.
...

Step 7: Run backtest for the strategy module

This will run the strategy backtest using docker compose command.

You need to add your Trading Strategy API key in ./env/hotwallet-polygon-eth-usdc-breakout.env:

Now run the backtest:

docker compose run hotwallet-polygon-eth-usdc-breakout backtest

Note: Currently macOS and Apple Silicon run Intel architecture Docker images, slowing down the execution 5x - 10x. Github Actions do not support building macOS Docker images yet. When you run this command on macOS prepare to wait for ~20 minutes.

You are likely to encounter several Python bugs in this step, so keep fixing your Python strategy module.

The end of the output should look like:

Writing backtest data the state file: /usr/src/trade-executor/state/hotwallet-polygon-eth-usdc-breakout-backtest.json
Exporting report, notebook: state/hotwallet-polygon-eth-usdc-breakout-backtest.ipynb, HTML: state/hotwallet-polygon-eth-usdc-breakout-backtest.html

After the backtest is complete, you can view the HTML report:

# macOS way to open a HTML file from the command line
open state/hotwallet-polygon-eth-usdc-breakout-backtest.html

You should see the backtest results, as captured from the default backtest notebook template:

Step 8: Set up a hot wallet

You need a hot wallet

Step 8a: Generate a private key

To create a hot wallet for the executor you can do it from the command line:

head -c 32 /dev/urandom|xxd -ps -c 32

This will give you a private key (example - do not use this private key):

68f4e1be83e2bd242d1a5a668574dd3b6b76a29f254b4ae662eba5381d1fc3a6

Then

Note: Hot wallets cannot be shared across different trade-executor instances, because this will mess up accounting.

Step 8b: Add the private key environment variable file

Private key will be needed in the trade execution configuration file

Example:

# Do not use this private key 
PRIVATE_KEY=0x68f4e1be83e2bd242d1a5a668574dd3b6b76a29f254b4ae662eba5381d1fc3a6

Step 8c Check the wallet

trade-executor provides the subcommand check-wallet to check the hot wallet status.

This checks

docker compose run hotwallet-polygon-eth-usdc-breakout check-wallet

Output:

AssertionError: At least 0.1 native currency need, our wallet 0xABF10A4027B3e5D29c21292955DcC4ECCe05747A has 0.00000000

This gives you the address for your private key.

Step 8d: Fund the account

We need to fund the account on Polygon

You can use a service like Transak to get MATIC with a debit card. KYC needed. You can get MATIC swapped to USDC.e e.g. on KyberSwap.

Send MATIC and USDC.e to the address you saw above.

Note: Due to historical reasons we use USDC.e (bridged variant) over native USDC on Polygon, because it has better liquidity. These two tokens are not fungible as the writing of this.

Step 8e: Check the wallet again

Now you should have some MATIC and USDC.e in your hot wallet.

docker compose run hotwallet-polygon-eth-usdc-breakout check-wallet

We see the account is funded now:

Balance details
  Hot wallet is 0xABF10A4027B3e5D29c21292955DcC4ECCe05747A
  We have 10.000000 tokens left for gas
  The gas error limit is 0.100000 tokens
  Reserve asset: USDC 
Reading token balances for 1 tokens at block 59193849, last 
  Balance of USD Coin (PoS) (0x2791Bca1f2de4661ED88A30C99A7a9449Aa84174): 1.988353 USDC

Step 9: Check the trading universe

trade-executor provides the subcommand check-universe to ensure the market data feeds work correctly.

You can run this with configured docker-compose as:

docker compose run hotwallet-polygon-eth-usdc-breakout check-universe

This will print out the last available candle, based on the the strategy Parameters.candle_time_bucket:

Latest OHCLV candle is at: 2024-07-10 15:00:00, 0:19:41.794234 ago

Step 10: Init the strategy

This will write the initial state file for your trading strategy:

docker compose run hotwallet-polygon-eth-usdc-breakout init

It should output something like:

Saved state to state/hotwallet-polygon-eth-usdc-breakout.json, total 1,108 chars
All done: State deployment info is <Deployment chain:polygon address:0xABF10A4027B3e5D29c21292955DcC4ECCe05747A name:None token:None>

Step 11: Perform test trade

After you are sure that trading data and hot wallet are fine, you can perform a test trade from the command line.

Example:

docker compose run hotwallet-polygon-eth-usdc-breakout perform-test-trade

This will print something like:

Test trade report
  Gas spent: 0.016125835459386936
  Trades done currently: 2
  Reserves currently: 1.986876 USDC
  Reserve currency spent: 0.0014769999999999506 USDC
  Buy trade price, expected: 3113.8176546091504, actual: 3111.
  Sell trade price, expected: 3108.3790383082533, actual: 3110.0700549742683 (WETH-USDC)

Step 12: Launch the live trading strategy

Now you are ready to start the strategy.

We first suggest start on the foreground.

docker compose up hotwallet-polygon-eth-usdc-breakout

After the boot up dance you will see:

apscheduler.scheduler                              INFO     Scheduler started

This means the trade-executor is now running, with its scheduled tasks (decide trades, check stop losses, revalue the portfolio, etc.). Leave it running.

There is a lot of output (noisy). In some point you should catch the message about decide_trades:

 2024-07-11 08:08:27 tradeexecutor.strategy.runner                      INFO     No open 
 2024-07-11 08:08:27 tradeexecutor.strategy.runner                      INFO     No frozen 
 2024-07-11 08:08:27 tradeexecutor.strategy.runner                      TRADE    Portfolio 
 
 Total equity: $1.99, in cash: $1.99
 Life-time positions: 1, trades: 2
 …
 Reserves:
 
     1.99 USDC
 
 2024-07-11 08:08:27 tradeexecutor.utils.timer                          INFO     Starting task decide_trades at 2024-07-11 08:08:27.183694, context is {}

The given strategy rebalances every 1h. So you should see something working or not working within 1h. The rebalance does not happen exactly at 00 minutes, but a bit past, as the candle data needs to be properly formed first.

For the long-term running, use daemon option:

docker compose up -d hotwallet-polygon-eth-usdc-breakout

Step 13: Configure additional RPC providers (optional)

See the multi-RPC configuration documentation here.

Step 14: Set up Discord logging (optional)

In this example, we only output trades in the docker console.

The trade-executor supports output to different team chat mediums. like Discord.

To configure Discord logging

DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/12...

Now all trade-related logging messages will be displayed at the Discord channel as well.

Support and social media

Got any questions? Pop into our Discord.