Awesome

One Policy to Control Them All:<br/>Shared Modular Policies for Agent-Agnostic Control

ICML 2020

[Project Page] [Paper] [Demo Video] [Long Oral Talk]

Wenlong Huang¹, Igor Mordatch², Deepak Pathak^{3 4}

¹University of California, Berkeley, ²Google Brain, ³Facebook AI Research, ⁴Carnegie Mellon University<br/>

This is a PyTorch-based implementation of our Shared Modular Policies. We take a step beyond the laborious training process of the conventional single-agent RL policy by tackling the possibility of learning general-purpose controllers for diverse robotic systems. Our approach trains a single policy for a wide variety of agents which can then generalize to unseen agent shapes at test-time without any further training.

If you find this work useful in your research, please cite using the following BibTeX:

@inproceedings{huang2020smp,
  Author = {Huang, Wenlong and
  Mordatch, Igor and Pathak, Deepak},
  Title = {One Policy to Control Them All:
  Shared Modular Policies for Agent-Agnostic Control},
  Booktitle = {ICML},
  Year = {2020}
  }

Setup

Requirements

• Python-3.6
• PyTorch-1.1.0
• CUDA-9.0
• CUDNN-7.6
• MuJoCo-200: download binaries, put license file inside, and add path to .bashrc

Setting up repository

git clone https://github.com/huangwl18/modular-rl.git
cd modular-rl/
python3.6 -m venv mrEnv
source $PWD/mrEnv/bin/activate

Installing Dependencies

pip install --upgrade pip
pip install -r requirements.txt

Running Code

Train with existing environment

• Train both-way SMP on Walker++ (12 variants of walker):

python main.py --expID 001 --td --bu --morphologies walker

• Train both-way SMP on Humanoid++ (8 variants of 2d humanoid):

python main.py --expID 002 --td --bu --morphologies humanoid

• Train both-way SMP on Cheetah++ (15 variants of cheetah):

python main.py --expID 003 --td --bu --morphologies cheetah

• Train both-way SMP on Hopper++ (3 variants of hopper):

python main.py --expID 004 --td --bu --morphologies hopper

• To train both-way SMP for only one environment (e.g. walker_7_main), specify the full name of the environment without the .xml suffix:

python main.py --expID 005 --td --bu --morphologies walker_7_main

To run with one-way message passing, disable --td for bottom-up-only message passing or disable --bu for top-down-only message passing. To run without any message passing, disable both --td and --bu.

Train with custom environment

• Train both-way SMP for only one environment:

python main.py --expID 006 --td --bu --custom_xml <PATH_TO_XML_FILE>

• Train both-way SMP for multiple environments (xml files must be in the same directory):

python main.py --expID 007 --td --bu --custom_xml <PATH_TO_XML_DIR>

Note that the current implementation assumes all custom MuJoCo agents are 2D planar and contain only one body tag with name torso attached to worldbody.

Visualization

• To visualize all walker environments with the both-way SMP model from experiment expID 001:

python visualize.py --expID 001 --td --bu --morphologies walker

• To visualize only walker_7_main environment with the both-way SMP model from experiment expID 001:

python visualize.py --expID 001 --td --bu --morphologies walker_7_main

Provided Environments

Note that each walker agent has an identical instance of itself called flipped, for which SMP always flips the torso message passed to both legs (e.g. the message that is passed to the left leg in the main instance is now passed the right leg).

For the results reported in the paper, the following agents are in the held-out set for the corresponding experiments:

• Walker++: walker_5_main, walker_6_flipped
• Humanoid++: humanoid_2d_7_right_arm, humanoid_2d_7_lower_arms
• Cheetah++: cheetah_4_front, cheetah_5_balanced, cheetah_6_front
• Walker-Hopper++: walker_5_main, walker_6_flipped, hopper_3
• Walker-Hopper-Humanoid++: walker_5_main, walker_6_flipped, hopper_3, humanoid_2d_7_right_arm, humanoid_2d_7_lower_arms

All other agents in the corresponding experiments are used for training.

Acknowledgement

The TD3 code is based on this open-source implementation. The code for Dynamic Graph Neural Networks is adapted from Modular Assemblies (Pathak*, Lu* et al., NeurIPS 2019).