Home

Awesome

ALF

<p align="center"> <img src="docs/images/logo.png" width = "500" alt="ALF-logo"/> </p>

CI

Agent Learning Framework (ALF) is a reinforcement learning framework emphasizing on the flexibility and easiness of implementing complex algorithms involving many different components. ALF is built on PyTorch. The development of previous version based on Tensorflow 2.1 has stopped as of Feb 2020.

Tutorial

A draft tutorial can be accessed on RTD. This tutorial is still under construction and some chapters are unfinished yet.

Documentation

Read the ALF documentation here.

Algorithms

AlgorithmTypeReference
A2COn-policy RLOpenAI Baselines: ACKTR & A2C
PPOOn-policy RLSchulman et al. "Proximal Policy Optimization Algorithms" arXiv:1707.06347
PPGOn-policy RLCobbe et al. "Phasic Policy Gradient" arXiv:2009.04416
DDQNOff-policy RLHasselt et al. "Deep Reinforcement Learning with Double Q-learning" arXiv:1509.06461
DDPGOff-policy RLLillicrap et al. "Continuous control with deep reinforcement learning" arXiv:1509.02971
QRSACOff-policy RLDabney et al. "Distributional Reinforcement Learning with Quantile Regression" arXiv:1710.10044
SACOff-policy RLHaarnoja et al. "Soft Actor-Critic Algorithms and Applications" arXiv:1812.05905
OACOff-policy RLCiosek et al. "Better Exploration with Optimistic Actor-Critic" arXiv:1910.12807
HEROff-policy RLAndrychowicz et al. "Hindsight Experience Replay" arXiv:1707.01495
TAACOff-policy RLYu et al. "TAAC: Temporally Abstract Actor-Critic for Continuous Control" arXiv:2104.06521
SEditorOff-policy/Safe RLYu et al. "Towards Safe Reinforcement Learning with a Safety Editor Policy" NeurIPS 2022
DIAYNIntrinsic motivation/ExplorationEysenbach et al. "Diversity is All You Need: Learning Diverse Skills without a Reward Function" arXiv:1802.06070
ICMIntrinsic motivation/ExplorationPathak et al. "Curiosity-driven Exploration by Self-supervised Prediction" arXiv:1705.05363
RNDIntrinsic motivation/ExplorationBurda et al. "Exploration by Random Network Distillation" arXiv:1810.12894
MuZeroModel-based RLSchrittwieser et al. "Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model" arXiv:1911.08265
BCOffline RLPomerleau "ALVINN: An Autonomous Land Vehicle in a Neural Network" NeurIPS 1988 <br> Bain et al. "A framework for behavioural cloning" Machine Intelligence 1999
Causal BCOffline RLSwamy et al. "Causal Imitation Learning under Temporally Correlated Noise" ICML2022
IQLOffline RLKostrikov, et al. "Offline Reinforcement Learning with Implicit Q-Learning" arXiv:2110.06169
MERLINUnsupervised learningWayne et al. "Unsupervised Predictive Memory in a Goal-Directed Agent"arXiv:1803.10760
MoNetUnsupervised learningBurgess et al. "MONet: Unsupervised Scene Decomposition and Representation" arXiv:1901.11390
Amortized SVGDGeneralFeng et al. "Learning to Draw Samples with Amortized Stein Variational Gradient Descent" arXiv:1707.06626
HyperNetworkGeneralRatzlaff and Fuxin. "HyperGAN: A Generative Model for Diverse, Performant Neural Networks" arXiv:1901.11058
MCTSGeneralGrill et al. "Monte-Carlo tree search as regularized policy optimization" arXiv:2007.12509
MINEGeneralBelghazi et al. "Mutual Information Neural Estimation" arXiv:1801.04062
ParticleVIGeneralLiu and Wang. "Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm" arXiv:1608.04471 <br> Liu et al. "Understanding and accelerating particle-based variational inference" arXiv:1807.01750
GPVIGeneralRatzlaff, Bai et al. "Generative Particle Variational Inference via Estimation of Functional Gradients" arXiv:2103.01291
SVGD optimizerGeneralLiu et al. "Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm" arXiv:1608.04471
VAEGeneralHiggins et al. "beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework" ICLR2017
RealNVPGeneralDinh et al. "Density estimation using Real NVP" arXiv:1605.08803
SpatialBroadcastDecoderGeneralWatters et al. "Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs" arXiv:1901.07017
VQ-VAEGeneralA van den Oord et al. "Neural Discrete Representation Learning" NeurIPS2017

Installation

OS software

The following installation was tested on Ubuntu22.04 with CUDA 11.8.

Python3.11 is currently supported by ALF. Note that some pip packages (e.g., pybullet) need python dev files, so make sure python3.11-dev is installed:

sudo apt install -y python3.11 python3.11-dev

Boost is also required by ALF for fast parallel environments.

sudo apt install libboost-all-dev

Python environment

Virtualenv is recommended for the installation. After creating and activating a virtual env, you can run the following commands to install ALF:

git clone https://github.com/HorizonRobotics/alf
cd alf
pip install pybind11
pip install -e . --extra-index-url https://download.pytorch.org/whl/cu118

For Nix Users

There is a built-in Nix-based development environment defined in flake.nix. To activate it, run

$ nix develop

in the root of your local repository.

Docker

We also provide a docker image of ALF for convenience. In order to use this image, you need to have docker and nvidia-docker (for ALF gpu usage) installed first.

docker run --gpus all -it horizonrobotics/cuda:11.8.0-py3.11-torch2.2-ubuntu22.04 /bin/bash

This will give you a shell that have all ALF and dependencies pre-installed.

The current docker image contains an ALF version on Feb 21, 2024. Regular version updates are expected in the future.

Examples

You can train any _conf.py file under alf/examples as follows:

python -m alf.bin.train --conf=CONF_FILE --root_dir=LOG_DIR

During training, we use tensorboard to show the progress of training:

tensorboard --logdir=LOG_DIR

After training, you can evaluate the trained model and visualize environment frames using the following command:

python -m alf.bin.play --root_dir=LOG_DIR

Deprecated

An older version of ALF used gin for job configuration. Its syntax is not as flexible as ALF conf (e.g., you can't easily do math computation in a gin file). There are still some examples with .gin under alf/examples. We are in the process of converting all .gin examples to _conf.py examples.

You can train any .gin file under alf/examples using the following command:

cd alf/examples; python -m alf.bin.train --gin_file=GIN_FILE --root_dir=LOG_DIR

Warning: When using gin, ALF has to be launched in the same directory with the gin file(s). If an error says that no configuration file is found, then probably you've launched ALF in a wrong place.

All the examples below are trained on a single machine Intel(R) Core(TM) i9-7960X CPU @ 2.80GHz with 32 CPUs and one RTX 2080Ti GPU.

A2C

PPO

PPG

DDQN

DDPG

SAC

ICM

RND

<img src="alf/examples/rnd_mrevenge.png" width="300" height="200" alt="mrevenge-training-curve"><img src="alf/examples/rnd_mrevenge.gif" width="150" height="200" alt="mrevenge-video">

DIAYN

BC

Merlin

MuZero

Citation

If you use ALF for research and find it useful, please consider citing:

@software{Xu2021ALF,
  title={{{ALF}: Agent Learning Framework}},
  author={Xu, Wei and Yu, Haonan and Zhang, Haichao and Hong, Yingxiang and Yang, Break and Zhao, Le and Bai, Jerry and ALF contributors},
  url={https://github.com/HorizonRobotics/alf},
  year={2021}
}

Contribute to ALF

You are welcome to contribute to ALF. Please follow the guideline here.