Awesome

VectorizedMultiAgentSimulator (VMAS)

<a href="https://pypi.org/project/vmas"><img src="https://img.shields.io/pypi/v/vmas" alt="pypi version"></a>

[!NOTE]
We have released BenchMARL, a benchmarking library where you can train VMAS tasks using TorchRL! Check out how easy it is to use it.

Welcome to VMAS!

This repository contains the code for the Vectorized Multi-Agent Simulator (VMAS).

VMAS is a vectorized differentiable simulator designed for efficient MARL benchmarking. It is comprised of a fully-differentiable vectorized 2D physics engine written in PyTorch and a set of challenging multi-robot scenarios. Scenario creation is made simple and modular to incentivize contributions. VMAS simulates agents and landmarks of different shapes and supports rotations, elastic collisions, joints, and custom gravity. Holonomic motion models are used for the agents to simplify simulation. Custom sensors such as LIDARs are available and the simulator supports inter-agent communication. Vectorization in PyTorch allows VMAS to perform simulations in a batch, seamlessly scaling to tens of thousands of parallel environments on accelerated hardware. VMAS has an interface compatible with OpenAI Gym, with Gymnasium, with RLlib, with torchrl and its MARL training library: BenchMARL, enabling out-of-the-box integration with a wide range of RL algorithms. The implementation is inspired by OpenAI's MPE. Alongside VMAS's scenarios, we port and vectorize all the scenarios in MPE.

Paper

The arXiv paper can be found here.

If you use VMAS in your research, cite it using:

@article{bettini2022vmas,
  title = {VMAS: A Vectorized Multi-Agent Simulator for Collective Robot Learning},
  author = {Bettini, Matteo and Kortvelesy, Ryan and Blumenkamp, Jan and Prorok, Amanda},
  year = {2022},
  journal={The 16th International Symposium on Distributed Autonomous Robotic Systems},
  publisher={Springer}
}

Video

Watch the presentation video of VMAS, showing its structure, scenarios, and experiments.

Table of contents

• VectorizedMultiAgentSimulator (VMAS)
  • Welcome to VMAS!
    • Paper
    • Video
  • Table of contents
  • How to use
    • Notebooks
    • Install
    • Run
      • RLlib
      • TorchRL
    • Input and output spaces
      • Output spaces
      • Input action space
  • Simulator features
  • Creating a new scenario
  • Play a scenario
  • Rendering
    • Plot function under rendering
    • Rendering on server machines
  • List of environments
    • VMAS
      • Main scenarios
      • Debug scenarios
    • MPE
  • Our papers using VMAS
  • TODOS

How to use

Notebooks

•   Using a VMAS environment. Here is a simple notebook that you can run to create, step and render any scenario in VMAS. It reproduces the use_vmas_env.py script in the examples folder.
•   Training VMAS in BenchMARL (suggested). In this notebook, we show how to use VMAS in BenchMARL, TorchRL's MARL training library.
•   Training VMAS in TorchRL. In this notebook, available in the TorchRL docs, we show how to use any VMAS scenario in TorchRL. It will guide you through the full pipeline needed to train agents using MAPPO/IPPO.
•   Training competitive VMAS MPE in TorchRL. In this notebook, available in the TorchRL docs, we show how to solve a Competitive Multi-Agent Reinforcement Learning (MARL) problem using MADDPG/IDDPG.
•   Training VMAS in RLlib. In this notebook, we show how to use any VMAS scenario in RLlib. It reproduces the rllib.py script in the examples folder.

Install

To install the simulator, you can use pip to get the latest release:

pip install vmas

If you want to install the current master version (more up to date than latest release), you can do:

git clone https://github.com/proroklab/VectorizedMultiAgentSimulator.git
cd VectorizedMultiAgentSimulator
pip install -e .

By default, vmas has only the core requirements. To install further dependencies to enable training with Gymnasium wrappers, RLLib wrappers, for rendering, and testing, you may want to install these further options:

# install gymnasium for gymnasium wrappers
pip install vmas[gymnasium]

# install rllib for rllib wrapper
pip install vmas[rllib]

# install rendering dependencies
pip install vmas[render]

# install testing dependencies
pip install vmas[test]

# install all dependencies
pip install vmas[all]

You can also install the following training libraries:

pip install benchmarl # For training in BenchMARL
pip install torchrl # For training in TorchRL
pip install "ray[rllib]"==2.1.0 # For training in RLlib. We support versions "ray[rllib]<=2.2,>=1.13"

Run

To use the simulator, simply create an environment by passing the name of the scenario you want (from the scenarios folder) to the make_env function. The function arguments are explained in the documentation. The function returns an environment object with the VMAS interface:

Here is an example:

 env = vmas.make_env(
        scenario="waterfall", # can be scenario name or BaseScenario class
        num_envs=32,
        device="cpu", # Or "cuda" for GPU
        continuous_actions=True,
        wrapper=None,  # One of: None, "rllib", "gym", "gymnasium", "gymnasium_vec"
        max_steps=None, # Defines the horizon. None is infinite horizon.
        seed=None, # Seed of the environment
        dict_spaces=False, # By default tuple spaces are used with each element in the tuple being an agent.
        # If dict_spaces=True, the spaces will become Dict with each key being the agent's name
        grad_enabled=False, # If grad_enabled the simulator is differentiable and gradients can flow from output to input
        terminated_truncated=False, # If terminated_truncated the simulator will return separate `terminated` and `truncated` flags in the `done()`, `step()`, and `get_from_scenario()` functions instead of a single `done` flag
        **kwargs # Additional arguments you want to pass to the scenario initialization
    )

A further example that you can run is contained in use_vmas_env.py in the examples directory.

With the terminated_truncated flag set to True, the simulator will return separate terminated and truncated flags in the done(), step(), and get_from_scenario() functions instead of a single done flag. This is useful when you want to know if the environment is done because the episode has ended or because the maximum episode length/ timestep horizon has been reached. See the Gymnasium documentation for more details on this.

RLlib

To see how to use VMAS in RLlib, check out the script in examples/rllib.py.

You can find more examples of multi-agent training in VMAS in the HetGPPO repository.

TorchRL

VMAS is supported by TorchRL and its MARL training library BenchMARL.

Check out how simple it is to use VMAS in BenchMARL with this notebook.

We provide a notebook which guides you through a full multi-agent reinforcement learning pipeline for training VMAS scenarios in TorchRL using MAPPO/IPPO.

You can find example scripts in the TorchRL repo here on how to run MAPPO-IPPO-MADDPG-QMIX-VDN using the VMAS wrapper.

Input and output spaces

VMAS uses gym spaces for input and output spaces. By default, action and observation spaces are tuples:

spaces.Tuple(
    [agent_space for agent in agents]
)

When creating the environment, by setting dict_spaces=True, tuples can be changed to dictionaries:

spaces.Dict(
  {agent.name: agent_space for agent in agents}
)

Output spaces

If dict_spaces=False, observations, infos, and rewards returned by the environment will be a list with each element being the value for that agent.

If dict_spaces=True, observations, infos, and rewards returned by the environment will be a dictionary with each element having key = agent_name and value being the value for that agent.

Each agent observation in either of these structures is either (depending on how you implement the scenario):

• a tensor with shape [num_envs, observation_size], where observation_size is the size of the agent's observation.

 def observation(self, agent: Agent):
        return torch.cat([agent.state.pos, agent.state.vel], dim=-1)

• a dictionary of such tensors

 def observation(self, agent: Agent):
        return {
            "pos": agent.state.pos,
            "nested": {"vel": agent.state.vel},
        }

Each agent reward in either of these structures is a tensor with shape [num_envs].

Each agent info in either of these structures is a dictionary where each entry has key representing the name of that info and value a tensor with shape [num_envs, info_size], where info_size is the size of that info for that agent.

Done is a tensor of shape [num_envs].

Input action space

Each agent in vmas has to provide an action tensor with shape [num_envs, action_size], where num_envs is the number of vectorized environments and action_size is the size of the agent's action.

The agents' actions can be provided to the env.step() in two ways:

• A List of length equal to the number of agents which looks like [tensor_action_agent_0, ..., tensor_action_agent_n]
• A Dict of length equal to the number of agents and with each entry looking like {agent_0.name: tensor_action_agent_0, ..., agent_n.name: tensor_action_agent_n}

Users can interchangeably use either of the two formats and even change formats during execution, vmas will always perform all sanity checks. Each format will work regardless of the fact that tuples or dictionary spaces have been chosen.

Simulator features

• Vectorized: VMAS vectorization can step any number of environments in parallel. This significantly reduces the time needed to collect rollouts for training in MARL.
• Simple: Complex vectorized physics engines exist (e.g., Brax), but they do not scale efficiently when dealing with multiple agents. This defeats the computational speed goal set by vectorization. VMAS uses a simple custom 2D dynamics engine written in PyTorch to provide fast simulation.
• General: The core of VMAS is structured so that it can be used to implement general high-level multi-robot problems in 2D. It can support adversarial as well as cooperative scenarios. Holonomic point-robot simulation has been chosen to focus on general high-level problems, without learning low-level custom robot controls through MARL.
• Extensible: VMAS is not just a simulator with a set of environments. It is a framework that can be used to create new multi-agent scenarios in a format that is usable by the whole MARL community. For this purpose, we have modularized the process of creating a task and introduced interactive rendering to debug it. You can define your own scenario in minutes. Have a look at the dedicated section in this document.
• Compatible: VMAS has wrappers for RLlib, torchrl, OpenAI Gym and Gymnasium. RLlib and torchrl have a large number of already implemented RL algorithms. Keep in mind that this interface is less efficient than the unwrapped version. For an example of wrapping, see the main of make_env.
• Tested: Our scenarios come with tests which run a custom designed heuristic on each scenario.
• Entity shapes: Our entities (agent and landmarks) can have different customizable shapes (spheres, boxes, lines). All these shapes are supported for elastic collisions.
• Faster than physics engines: Our simulator is extremely lightweight, using only tensor operations. It is perfect for running MARL training at scale with multi-agent collisions and interactions.
• Customizable: When creating a new scenario of your own, the world, agent and landmarks are highly customizable. Examples are: drag, friction, gravity, simulation timestep, non-differentiable communication, agent sensors (e.g. LIDAR), and masses.
• Non-differentiable communication: Scenarios can require agents to perform discrete or continuous communication actions.
• Gravity: VMAS supports customizable gravity.
• Sensors: Our simulator implements ray casting, which can be used to simulate a wide range of distance-based sensors that can be added to agents. We currently support LIDARs. To see available sensors, have a look at the sensors script.
• Joints: Our simulator supports joints. Joints are constraints that keep entities at a specified distance. The user can specify the anchor points on the two objects, the distance (including 0), the thickness of the joint, if the joint is allowed to rotate at either anchor point, and if he wants the joint to be collidable. Have a look at the waterfall scenario to see how you can use joints. See the waterfall and joint_passage scenarios for an example.
• Agent actions: Agents' physical actions are 2D forces for holonomic motion. Agent rotation can also be controlled through a torque action (activated by setting agent.action.u_rot_range at agent creation time). Agents can also be equipped with continuous or discrete communication actions.
• Action preprocessing: By implementing the process_action function of a scenario, you can modify the agents' actions before they are passed to the simulator. This is used in controllers (where we provide different types of controllers to use) and dynamics (where we provide custom robot dynamic models).
• Controllers: Controllers are components that can be appended to the neural network policy or replace it completely. We provide a VelocityController which can be used to treat input actions as velocities (instead of default vmas input forces). This PID controller takes velocities and outputs the forces which are fed to the simulator. See the vel_control debug scenario for an example.
• Dynamic models: VMAS simulates holonomic dynamics models by default. Custom dynamics can be chosen at agent creation time. Implementations now include DiffDriveDynamics for differential drive robots, KinematicBicycleDynamics for kinematic bicycle model, and Drone for quadcopter dynamics. See diff_drive, kinematic_bicycle and drone debug scenarios for examples.
• Differentiable: By setting grad_enabled=True when creating an environment, the simulator will be differentiable, allowing gradients flowing through any of its function.

Creating a new scenario

To create a new scenario, just extend the BaseScenario class in scenario.py.

You will need to implement at least make_world, reset_world_at, observation, and reward. Optionally, you can also implement done, info, process_action, and extra_render.

You can also change the viewer size, zoom, and enable a background rendered grid by changing these inherited attributes in the make_world function.

To know how, just read the documentation of BaseScenario in scenario.py and look at the implemented scenarios.

Play a scenario

You can play with a scenario interactively! Just execute its script!

Just use the render_interactively function in the interactive_rendering.py script. Relevant values will be plotted to screen. Move the agent with the arrow keys and switch agents with TAB. You can reset the environment by pressing R. If you have more than 1 agent, you can control another one with W,A,S,D and switch the second agent using LSHIFT. To do this, just set control_two_agents=True. If the agents also have rotational actions, you can control them with M,N for the first agent and with Q,E (for example in the diff_drive scenario).

On the screen you will see some data from the agent controlled with arrow keys. This data includes: name, current obs, current reward, total reward so far and environment done flag.

Here is an overview of what it looks like:

Rendering

To render the environment, just call the render or the try_render_at functions (depending on environment wrapping).

Example:

env.render(
    mode="rgb_array", # "rgb_array" returns image, "human" renders in display
    agent_index_focus=4, # If None keep all agents in camera, else focus camera on specific agent
    index=0, # Index of batched environment to render
    visualize_when_rgb: bool = False, # Also run human visualization when mode=="rgb_array"
)

You can also change the viewer size, zoom, and enable a background rendered grid by changing these inherited attributes in the scenario make_world function.

Plot function under rendering

It is possible to plot a function under the rendering of the agents by providing a function f to the render function.

env.render(
    plot_position_function=f
)

The function takes a numpy array with shape (n_points, 2), which represents a set of x, y values to evaluate f over for plotting. f outputs either an array with shape (n_points, 1), which will be plotted as a colormap, or an array with shape (n_points, 4), which will be plotted as RGBA values.

See the sampling.py scenario for more info.

Rendering on server machines

To render in machines without a display use mode="rgb_array". Make sure you have OpenGL and Pyglet installed. To use GPUs for headless rendering, you can install the EGL library. If you do not have EGL, you need to create a fake screen. You can do this by running these commands before the script:

export DISPLAY=':99.0'
Xvfb :99 -screen 0 1400x900x24 > /dev/null 2>&1 &

or in this way:

xvfb-run -s \"-screen 0 1400x900x24\" python <your_script.py>

To create a fake screen you need to have Xvfb installed.

List of environments

VMAS

Main scenarios

Debug scenarios

MPE

Our papers using VMAS

• VMAS features training of balance, transport, give_way, wheel
• HetGPPO features training of het_mass, give_way, joint_passage, joint_passage_size
• SND features training of navigation, joint_passage, joint_passage_size, wind
• TorchRL features training of navigation, sampling, balance
• BenchMARL features training of navigation, sampling, balance
• The Cambridge RoboMaster features training of navigation
• DiversityControl (DiCo) features training of navigation, sampling, dispersion, simple_tag

TODOS

TODOs are now listed here.

• Improve test efficiency and add new tests
• Implement 2D drone dynamics
• Allow any number of actions
• Improve VMAS performance
• Dict obs support in torchrl
• Make TextLine a Geom usable in a scenario
• Notebook on how to use torch rl with vmas
• Allow dict obs spaces and multidim obs
• Talk about action preprocessing and velocity controller
• New envs from joint project with their descriptions
• Talk about navigation / multi_goal
• Link video of experiments
• Add LIDAR section
• Implement LIDAR
• Rewrite all MPE scenarios
  • simple
  • simple_adversary
  • simple_crypto
  • simple_push
  • simple_reference
  • simple_speaker_listener
  • simple_spread
  • simple_tag
  • simple_world_comm