Home

Awesome

<div align="center"> <img align="center" width="30%" alt="image" src="https://github.com/AI4Finance-Foundation/FinGPT/assets/31713746/e0371951-1ce1-488e-aa25-0992dafcc139"> </div>

ElegantRL “小雅”: Massively Parallel Deep Reinforcement Learning

Downloads Downloads Python 3.6 PyPI License

<br/> <a href="https://github.com/AI4Finance-Foundation/ElegantRL" target="\_blank"> <div align="center"> <img src="figs/icon.jpg" width="40%"/> </div> <!-- <div align="center"><caption>Slack Invitation Link</caption></div> --> </a> <br/>

“小雅”源于《诗经·小雅·鹤鸣》,旨在「他山之石,可以攻玉」。

Visitors

ElegantRL (website) is developed for users/developers with the following advantages:

ElegantRL implements the following model-free deep reinforcement learning (DRL) algorithms:

For more details of DRL algorithms, please refer to the educational webpage OpenAI Spinning Up.

ElegantRL supports the following simulators:

Contents

Tutorials

ElegantRL-Helloworld

<div align="center"> <img align="center" src=figs/File_structure.png width="800"> </div>

For beginners, we maintain ElegantRL-Helloworld as a tutorial. Its goal is to get hands-on experience with ELegantRL.

One sentence summary: an agent (agent.py) with Actor-Critic networks (net.py) is trained (run.py) by interacting with an environment (env.py).

File Structure

Experimental Demos

More efficient than Ray RLlib

Experiments on Ant (MuJoCo), Humainoid (MuJoCo), Ant (Isaac Gym), Humanoid (Isaac Gym) # from left to right

<div align="center"> <img align="center" src=figs/envs.png width="800"> <img align="center" src=figs/performance1.png width="800"> <img align="center" src=figs/performance2.png width="800"> </div>

ElegantRL fully supports Isaac Gym that runs massively parallel simulation (e.g., 4096 sub-envs) on one GPU.

More stable than Stable-baseline 3

Experiment on Hopper-v2 # ElegantRL achieves much smaller variance (average over 8 runs).

Also, PPO+H in ElegantRL completed the training process of 5M samples about 6x faster than Stable-Baseline3.

<div align="center"> <img align="center" src=figs/SB3_vs_ElegantRL.png width="640"> </div>

Testing and Contributing

Our tests are written with the built-in unittest Python module for easy access. In order to run a specific test file (for example, test_training_agents.py), use the following command from the root directory:

python -m unittest unit_tests/test_training_agents.py

In order to run all the tests sequentially, you can use the following command:

python -m unittest discover

Please note that some of the tests require Isaac Gym to be installed on your system. If it is not, any tests related to Isaac Gym will fail.

We welcome any contributions to the codebase, but we ask that you please do not submit/push code that breaks the tests. Also, please shy away from modifying the tests just to get your proposed changes to pass them. As it stands, the tests on their own are quite minimal (instantiating environments, training agents for one step, etc.), so if they're breaking, it's almost certainly a problem with your code and not with the tests.

We're actively working on refactoring and trying to make the codebase cleaner and more performant as a whole. If you'd like to help us clean up some code, we'd strongly encourage you to also watch Uncle Bob's clean coding lessons if you haven't already.

Requirements

Necessary:
| Python 3.6+     |
| PyTorch 1.6+    |

Not necessary:
| Numpy 1.18+     | For ReplayBuffer. Numpy will be installed along with PyTorch.
| gym 0.17.0      | For env. Gym provides tutorial env for DRL training. (env.render() bug in gym==0.18 pyglet==1.6. Change to gym==0.17.0, pyglet==1.5)
| pybullet 2.7+   | For env. We use PyBullet (free) as an alternative of MuJoCo (not free).
| box2d-py 2.3.8  | For gym. Use pip install Box2D (instead of box2d-py)
| matplotlib 3.2  | For plots.

pip3 install gym==0.17.0 pybullet Box2D matplotlib # or pip install -r requirements.txt

To install StarCraftII env,
bash ./elegantrl/envs/installsc2.sh
pip install -r sc2_requirements.txt

Citation:

To cite this repository:

@misc{erl,
  author = {Liu, Xiao-Yang and Li, Zechu and Zhu, Ming and Wang, Zhaoran and Zheng, Jiahao},
  title = {{ElegantRL}: Massively Parallel Framework for Cloud-native Deep Reinforcement Learning},
  year = {2021},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/AI4Finance-Foundation/ElegantRL}},
}
@article{liu2021elegantrl,
  title={ElegantRL-Podracer: Scalable and elastic library for cloud-native deep reinforcement learning},
  author={Liu, Xiao-Yang and Li, Zechu and Yang, Zhuoran and Zheng, Jiahao and Wang, Zhaoran and Walid, Anwar and Guo, Jian and Jordan, Michael I},
  journal={NeurIPS, Workshop on Deep Reinforcement Learning},
  year={2021}
}