Home

Awesome

Reinforcement Learning For Financial Trading :chart_with_upwards_trend:

How to use Reinforcement learning for financial trading using Simulated Stock Data using MATLAB. This project is split into two sections:

  1. Single Agent Learning
  2. Multiagent Learning

<a name="section-1"></a> Single Agent Learning Setup

To run:

  1. Open RL_trading_demo.prj
  2. Open workflow.mlx (MATLAB Live Script preferred) or workflow.m (MATLAB script viewable in GitHub)
  3. Run workflow.mlx

Environment and Reward can be found in: myStepFunction.m

Requires

Overview

The goal of the Reinforcement Learning agent is simple. Learn how to trade the financial markets without ever losing money.

Note, this is different from learn how to trade the market and make the most money possible.

Reinforcement Learning for Financial Trading

Lets apply some of the terminology and concepts of teaching a reinforcement learning agent to trade.

Case Study

Our environment consists of 3 stocks, $20000 cash & 15 years of historical data:

Stocks are:

Action Space: (buy, sell ,hold) for 3 stocks = 27 total actions

Observation Space:

Strategy

Reward

The reward system was what took the most time to develop and required several iterations.

The details are listed, but to simplify

A good reward is given when a trade results in a profit and a stock is bought/held/sold at the right time. For example buying a stock on the way up.

The reverse goes for giving a penalty except for one thing.

A large penalty is given if ANY trade of the 3 stocks is determined as a bad trade. WHY? In the first iteration of the reward system, this was not there. What was observed is that the agent quickly learnt the best of the 3 stocks to trade and ignored the other 2.

Training

Here is an overview of how long it took to learn. You might think 80hrs is a long time. But remember, how long do you think it takes a human to learn how to trade successfully over 12 years?

Results

Highlights

Histogram of 100 Simulations

Best Simulation

Worst Simulation

The histogram shows that for 100 simulations, not once did the agent ever lose money. So the goal was achieved!

However, you can see that the range varies quite a bit. If you inspect the plots on the right, just buying and holding 1 stock would returned a profit just as good if not better than the agent.

BUT – It’s easy to judge retrospectively. The agent was trading each day as it occurred. None of the simulations resulted in a loss.

Further Improvements

The case study did ignore some common things to consider when trading the market. Here a few areas that could improve the performance, and make the trained agent more robust:

Conclusion - Single Agent Case

The aim of this example was to show:

<a name="section-2"></a> Multiagent Learning Setup

The example is to show the use of Multiagent Reinforcement Learning for Financial Trading. In this example, agents compete to outperform each other. They are rewarded when they outcompete the other, and penalized when they underperform.

Setup

To run:

  1. Open RL_trading_demo.prj
  2. Open workflowMulti.mlx
  3. Run workflowMutli.mlx

Requires

Overview

MATLAB R2023b has introduced a new feature that allows for multiagent reinforcement learning, whereby multiple agents interact in the same environment. These agents can either collaborate towards a common goal or compete against each other.

In this example, two agents compete against each other to outperform each other in trading stocks. They attempt to learn a better strategy that results in a higher profit than their competitor.

These agents mimic the inherently competitive nature of trading - they fight against their foe to optimize their strategy!

Note that the environment is nearly identical to the Single Agent Example. These agents have the same action space and observation space. Each agent also shares the same reward function mentioned in the Single Agent Example, but two of the agents in this example extend this reward function as mentioned later.

Case Study

Define the three Proximal Policy Optimization (PPO) agents by defining an actor and a critic. These agents could be defined using the Deep Network Designer, although this example defines the networks programmatically. The network architecture for each agent follows:

actor critic

Observation Space: at each time step, each agent observes 19 different continuous values specified in the Single Agent Example.

Action Space: contains 27 possible actions, just like in the Single Agent Example.

Reward Function

All three agents share a common reward function, and agents 1 and 2 add on to this reward function with a specialized competitive reward.

Shared reward for all agents: Give the agent +1 as a reward if they made a profit or if they sold stocks while its indicators were suggesting a negative trajectory. Give the agent -1 otherwise. This reward is identical to the reward in the Single Agent Case.

Competitive reward for agents 1 and 2: Give the agent +0.5 as a reward if it has made more of a profit than its competitor, give the agent -0.5 otherwise.

Given that agent 3 does not contain the competitive reward, a greater performance in agents 1 and 2 would indicate that the competitive strategy aids in learning.

Training

Agents were trained for 2,500 episodes, where each episode contained 2,597 steps (length of training data). The results from the agent as shown in the Episode Manager are shown below:

training

As shown in this image, agents 1 and 2 constantly battle to outcompete each other.

Results

As shown in the results below, agents 1 and 2 outperform agent 3 -- their competitive nature aids in learning. Further, agents 1 and 2 over 1.5x their initial $20,000 on the test dataset.

These results demonstrate the effectiveness of utilizing competitive agents for the Stock Trading Problem. While the competing agents do not always outperform their solitary counterpart (Agent 3), this work proves the utility and possibility of using this multiagent paradigm!

In the future, this idea could be expanded to many other areas in finance -- multiagent reinforcement learning is currently an active area of research, as its competitive nature lends itself well to the inherently competitive world of finance.

Performance

For more information on Reinforcement Learning in MATLAB:

Free Reinforcement Learning Onramp - No downloads, or installation, just your browser and you!

Download a free trial

Getting Started with Reinforcement Learning (YouTube series)

Copyright 2020 The MathWorks, Inc.

View Reinforcement Learning for Financial Trading on File Exchange