Awesome

Dreamer PyTorch

PA: This repository is in maintenance mode. No new features will be added but bugfixes and contributions are welcome. Please create a pull request with any fixes you have!

Dream to Control: Learning Behaviors by Latent Imagination

Paper: https://arxiv.org/abs/1912.01603
Project Website: https://danijar.com/project/dreamer/
TensorFlow 2 implementation: https://github.com/danijar/dreamer
TensorFlow 1 implementation: https://github.com/google-research/dreamer

Results

Task	Average Return @ 1M	Dreamer Paper @ 1M
Acrobot Swingup	69.54	~300
Cartpole Balance	877.5	~990
Cartpole Balance Sparse	814	~900
Cartpole Swingup	633.6	~800
Cup Catch	885.1	~990
Finger Turn Hard	212.8	~550
Hopper Hop	219	~250
Hopper Stand	511.6	~990
Pendulum Swingup	724.9	~760
Quadruped Run	112.4	~450
Quadruped Walk	52.82	~650
Reacher Easy	962.8	~950
Walker Stand	956.8	~990

Table 1. Dreamer PyTorch vs. Paper Implementation

1 random seed for PyTorch, 5 for the paper
Code @ commit ccea6ae
37H for 1M steps on P100, 20H for 1M steps on V100

Installation

Install Python 3.11
Install Python Poetry

# clone the repo with rlpyt submodule
git clone --recurse-submodules https://github.com/juliusfrost/dreamer-pytorch.git
cd dreamer-pytorch

# Windows
cd setup/windows_cu118

# Linux
cd setup/linux_cu118

# install with poetry
poetry install

# install with pip
pip install -r requirements.txt

Running Experiments

To run experiments on Atari, run python main.py, and add any extra arguments you would like. For example, to run with a single gpu set --cuda-idx 0.

To run experiments on DeepMind Control, run python main_dmc.py. You can also set any extra arguments here.

Experiments will automatically be stored in data/local/yyyymmdd/run_#
You can use tensorboard to keep track of your experiment. Run tensorboard --logdir=data.

If you have trouble reproducing any results, please raise a GitHub issue with your logs and results. Otherwise, if you have success, please share your trained model weights with us and with the broader community!

Testing

To run tests:

pytest tests

If you want additional code coverage information:

pytest tests --cov=dreamer

Code structure

main.py run atari experiment
main_dmc.py run deepmind control experiment
dreamer dreamer code
- agents agent code used in sampling
  - atari_dreamer_agent.py Atari agent
  - dmc_dreamer_agent.py DeepMind Control agent
  - dreamer_agent.py basic sampling agent, exploration, contains shared methods
- algos algorithm specific code
  - dreamer_algo.py optimization algorithm, loss functions, hyperparameters
  - replay.py replay buffer
- envs environment specific code
  - action_repeat.py action repeat wrapper. ported from tf2 dreamer
  - atari.py Atari environments. ported from tf2 dreamer
  - dmc.py DeepMind Control Suite environment. ported from tf2 dreamer
  - env.py base classes for environment
  - modified_atari.py unused atari environment from rlpyt
  - normalize_actions.py normalize actions wrapper. ported from tf2 dreamer
  - one_hot.py one hot action wrapper. ported from tf2 dreamer
  - time_limit.py Time limit wrapper. ported from tf2 dreamer
  - wrapper.py Base environment wrapper class
- experiments currently not used
- models all models used in the agent
  - action.py Action model
  - agent.py Summarizes all models for agent module
  - dense.py Dense fully connected models. Used for Reward Model, Value Model, Discount Model.
  - distribution.py Distributions, TanH Bijector
  - observation.py Observation Model
  - rnns.py Recurrent State Space Model
- utils utility functions
  - logging.py logging videos
  - module.py freezing parameters