Awesome
Learning by Cheating
This repo is the implemention of paper Learning by Cheating in CARLA 0.9.6.
Learning by Cheating
Dian Chen, Brady Zhou, Vladlen Koltun, Philipp Krähenbühl,
Conference on Robot Learning (CoRL 2019)
arXiv 1912.12294
Code in CARLA 0.9.5 is coming soon.
Reference
If you find this repo to be useful in your research, please consider citing our work
@inproceedings{chen2019lbc,
author = {Chen, Dian and Zhou, Brady and Koltun, Vladlen and Kr\"ahenb\"uhl, Philipp},
title = {Learning by Cheating},
booktitle = {Conference on Robot Learning (CoRL)},
year = {2019},
}
Updates
- (March 2022) Checkout our latest follow-up work: LAV!
- (May 2021) Checkout our latest follow-up work: World on Rails
- (2020) Checkout our submission to the 2020 CARLA Challenge!
Video
For a summarization of this project, please checkout our video.
Installation
Please refer to INSTALL.md for setup instructions.
Quick start
We provide a quick script here in case you would like to skip compiling and directly use the official binary release:
# Download CARLA 0.9.6
wget http://carla-assets-internal.s3.amazonaws.com/Releases/Linux/CARLA_0.9.6.tar.gz
mkdir carla_lbc
tar -xvzf CARLA_0.9.6.tar.gz -C carla_lbc
cd carla_lbc
# Download LBC
git init
git remote add origin https://github.com/dianchen96/LearningByCheating.git
# rename the LICENSE file to avoid conflicts during the pull
mv LICENSE CARLA_LICENSE
git pull origin release-0.9.6
wget http://www.cs.utexas.edu/~dchen/lbc_release/navmesh/Town01.bin
wget http://www.cs.utexas.edu/~dchen/lbc_release/navmesh/Town02.bin
mv Town*.bin CarlaUE4/Content/Carla/Maps/Nav/
# Create conda environment
conda env create -f environment.yml
conda activate carla
# Install carla client
cd PythonAPI/carla/dist
rm carla-0.9.6-py3.5-linux-x86_64.egg
wget http://www.cs.utexas.edu/~dchen/lbc_release/egg/carla-0.9.6-py3.5-linux-x86_64.egg
easy_install carla-0.9.6-py3.5-linux-x86_64.egg
# Download model checkpoints
cd ../../..
mkdir -p ckpts/image
cd ckpts/image
wget http://www.cs.utexas.edu/~dchen/lbc_release/ckpts/image/model-10.th
wget http://www.cs.utexas.edu/~dchen/lbc_release/ckpts/image/config.json
cd ../..
mkdir -p ckpts/priveleged
cd ckpts/priveleged
wget http://www.cs.utexas.edu/~dchen/lbc_release/ckpts/privileged/model-128.th
wget http://www.cs.utexas.edu/~dchen/lbc_release/ckpts/privileged/config.json
cd ../..
Once you are done with that, you need to start the Carla Server and the LbC agent.
Running the Carla Server
- Open up a terminal.
- Inside the carla directory run
./CarlaUE4.sh -fps=10 -benchmark
.
Running the LbC Agent
-
Open up another terminal to run the LbC agent.
-
To run the LbC agent, your
PYTHONPATH
needs to be set correctly. Make sure[CARLA PATH]/PythonAPI
is in yourPYTHONPATH
If you are inside the carla_lbc directory (created above), you can run the following command.
export PYTHONPATH="`pwd`/PythonAPI:$PYTHONPATH"
- After ensuring your
PYTHONPATH
is set correctly, run this:
CUDA_VISIBLE_DEVICES="0" python benchmark_agent.py --suite=town2 --model-path=ckpts/image/model-10.th --show
- Now you can see the the image model drive in the testing town!
Benchmark Results (0.9.6 w/ pedestrians fix)
Since CARLA does not have an official 0.9+ version that supports pedestrian crossing, we modified the most up-to-date CARLA (0.9.6) to support pedestrian crossing to compare to the original benchmark.
All our changes are only on the client side, and summarized below:
- Modified navigation mesh, such that pedestrians walk and "cross" the streets.
- Modified navigation, such that pedestrians avoid cars.
- Added pedestrian teleport functionality, such that they are teleported if stuck and causing a traffic jam.
We are cleaning-up our CARLA 0.9.5 implementation, and the code is coming soon.
Autopilot
Training | New weather | New town | New town & weather | |
---|---|---|---|---|
CoRL Straight | 100 | 100 | 100 | 100 |
CoRL Turning | 100 | 100 | 100 | 100 |
CoRL Nav | 100 | 100 | 100 | 100 |
CoRL Nav Dynamic | 100 | 100 | 100 | 100 |
Priviledged(cheating) Agent
Training | New weather | New town | New town & weather | |
---|---|---|---|---|
CoRL Straight | 100 | 100 | 100 | 100 |
CoRL Turning | 100 | 100 | 100 | 100 |
CoRL Nav | 100 | 100 | 99 | 100 |
CoRL Nav Dynamic | 100 | 100 | 100 | 100 |
Stage 1(purely offline) Agent
Training | New weather | New town | New town & weather | |
---|---|---|---|---|
CoRL Straight | 100 | 100 | 100 | 100 |
CoRL Turning | 96 | 100 | 95 | 98 |
CoRL Nav | 94 | 98 | 94 | 98 |
CoRL Nav Dynamic | 95 | 92 | 88 | 90 |
Stage 2(online fine-tuned) Agent
Training | New weather | New town | New town & weather | |
---|---|---|---|---|
CoRL Straight | 100 | 100 | 100 | 100 |
CoRL Turning | 100 | 96 | 100 | 100 |
CoRL Nav | 100 | 100 | 98 | 100 |
CoRL Nav Dynamic | 100 | 100 | 99 | 100 |
Training Models
Data collection
python data_collector.py --dataset_path=[PATH]
Use --n_episodes
to select the number of trajectories you want to collect. Make sure [CARLA PATH]/PythonAPI
is in your python path, or add PYTHONPATH=[CARLA PATH]/PythonAPI
before the call.
Train a privileged agent
cd training
python train_birdview.py --dataset_path=[DATA PATH] --log_dir=[LOG DIR]
-
--dataset_path
expects a folder that contains atrain
and aval
subdirectory, where each of these should contain.lmdb
trajectory files collected from the data collection script. -
--log_dir
will store the model checkpoints, the hyperparameter configurations, and the training losses and visualizations. You can track your model usingtensorboard --log_dir [LOG_DIR]
to monitor the progress. You should expect a validation loss smaller than5e-3
for a well trained a privileged model.
<img src=figs/birdview.png height="40%" width="40%">
Train an image agent
Stage 0 (warm-up)
cd training
python train_image_phase0.py --dataset_path=[DATA PATH] --log_dir=[LOG DIR] --pretrained --teacher_path=[TEACHER PATH]
--teacher_path
expects the path to a privileged agent.th
checkpoint. Make sureconfig.json
from priveleged agent trainig lies in the same directory as the checkpoint
Stage 1
cd training
python train_image_phase1.py --dataset_path=[DATA PATH] --log_dir=[LOG DIR] --teacher_path=[TEACHER PATH] --ckpt=[CKPT PATH]
--ckpt
expects the path to the stage 0.th
checkpoint.
Stage 2 (dagger)
cd training
python train_image_phase2.py --teacher_path=[TEACHER PATH] --ckpt=[CKPT PATH] --log_dir=[LOG DIR]
--ckpt
expects the path to the stage 1.th
checkpoint.
For all stages you can track your model using tensorboard --log_dir [LOG_DIR]
to monitor the progress.
<img src=figs/image_phase1.png>
Note
Due to randomness, the retrained model will not be the same as the published, and you will likely need to retune the controller parameters.
Benchmarking models
- Start a CARLA server instance
./Carla.sh -fps=10 -benchmark -world-port=[PORT NUM]
- Run
python benchmark_agent.py --suite=[SUITE NAME] --port=[PORT NUM] --model_path=[MODEL PATH]
. This will create asummary.csv
in/benchmark
and benchmarking videos in/benchmark/[SUITE NAME]
. - Once benchmarking is done, use
python view_benchmark_results.py [MODEL_PATH]/benchmark/[MODEL NAME]
to print a results table like the ones shown below.
Note that CARLA is non-deterministic, since currently we cannot control the random seeds in the server. Our client-side random seed makes sure the other vehicles have deterministic initial positions, but the decision of whether to turn left or right at intersections is non-deterministic.
Detailed Benchmark Results
Autopilot
╔Performance of autopilot════════════╦═════════╦═══════╗
║ Suite Name ║ Success Rate ║ Total ║ Seeds ║
╠═════════════════════╬══════════════╬═════════╬═══════╣
║ FullTown01-v1 ║ 100 ║ 100/100 ║ 0 ║
║ FullTown01-v2 ║ 100 ║ 50/50 ║ 0 ║
║ FullTown01-v3 ║ 100 ║ 100/100 ║ 0 ║
║ FullTown01-v4 ║ 100 ║ 50/50 ║ 0 ║
║ FullTown02-v1 ║ 100 ║ 100/100 ║ 0 ║
║ FullTown02-v2 ║ 100 ║ 50/50 ║ 0 ║
║ FullTown02-v3 ║ 100 ║ 100/100 ║ 0 ║
║ FullTown02-v4 ║ 100 ║ 50/50 ║ 0 ║
║ NoCrashTown01-v1 ║ 100.0 ± 0.0 ║ 300/300 ║ 0,1,2 ║
║ NoCrashTown01-v2 ║ 100.0 ± 0.0 ║ 150/150 ║ 0,1,2 ║
║ NoCrashTown01-v3 ║ 98.7 ± 0.6 ║ 296/300 ║ 0,1,2 ║
║ NoCrashTown01-v4 ║ 99.3 ± 1.2 ║ 149/150 ║ 0,1,2 ║
║ NoCrashTown01-v5 ║ 86.3 ± 3.2 ║ 259/300 ║ 0,1,2 ║
║ NoCrashTown01-v6 ║ 82.7 ± 6.1 ║ 124/150 ║ 0,1,2 ║
║ NoCrashTown02-v1 ║ 100.0 ± 0.0 ║ 300/300 ║ 0,1,2 ║
║ NoCrashTown02-v2 ║ 100.0 ± 0.0 ║ 150/150 ║ 0,1,2 ║
║ NoCrashTown02-v3 ║ 99.0 ± 1.0 ║ 297/300 ║ 0,1,2 ║
║ NoCrashTown02-v4 ║ 98.0 ± 2.0 ║ 147/150 ║ 0,1,2 ║
║ NoCrashTown02-v5 ║ 60.0 ± 2.6 ║ 180/300 ║ 0,1,2 ║
║ NoCrashTown02-v6 ║ 58.7 ± 7.6 ║ 88/150 ║ 0,1,2 ║
║ StraightTown01-v1 ║ 100 ║ 100/100 ║ 0 ║
║ StraightTown01-v2 ║ 100 ║ 50/50 ║ 0 ║
║ StraightTown02-v1 ║ 100 ║ 100/100 ║ 0 ║
║ StraightTown02-v2 ║ 100 ║ 50/50 ║ 0 ║
║ TurnTown01-v1 ║ 100 ║ 100/100 ║ 0 ║
║ TurnTown01-v2 ║ 100 ║ 50/50 ║ 0 ║
║ TurnTown02-v1 ║ 100 ║ 100/100 ║ 0 ║
║ TurnTown02-v2 ║ 100 ║ 50/50 ║ 0 ║
╚═════════════════════╩══════════════╩═════════╩═══════╝
Priviledged(cheating) Agent
╔Performance of model-512════════════╦═════════╦═══════╗
║ Suite Name ║ Success Rate ║ Total ║ Seeds ║
╠═════════════════════╬══════════════╬═════════╬═══════╣
║ FullTown01-v1 ║ 100 ║ 100/100 ║ 0 ║
║ FullTown01-v2 ║ 100 ║ 50/50 ║ 0 ║
║ FullTown01-v3 ║ 100 ║ 100/100 ║ 0 ║
║ FullTown01-v4 ║ 100 ║ 50/50 ║ 0 ║
║ FullTown02-v1 ║ 100 ║ 100/100 ║ 0 ║
║ FullTown02-v2 ║ 100 ║ 50/50 ║ 0 ║
║ FullTown02-v3 ║ 99 ║ 99/100 ║ 0 ║
║ FullTown02-v4 ║ 100 ║ 50/50 ║ 0 ║
║ NoCrashTown01-v1 ║ 100.0 ± 0.0 ║ 300/300 ║ 0,1,2 ║
║ NoCrashTown01-v2 ║ 100.0 ± 0.0 ║ 150/150 ║ 0,1,2 ║
║ NoCrashTown01-v3 ║ 96.3 ± 3.1 ║ 289/300 ║ 0,1,2 ║
║ NoCrashTown01-v4 ║ 97.3 ± 3.1 ║ 146/150 ║ 0,1,2 ║
║ NoCrashTown01-v5 ║ 80.2 ± 4.9 ║ 239/298 ║ 0,1,2 ║
║ NoCrashTown01-v6 ║ 81.3 ± 5.8 ║ 122/150 ║ 0,1,2 ║
║ NoCrashTown02-v1 ║ 100.0 ± 0.0 ║ 300/300 ║ 0,1,2 ║
║ NoCrashTown02-v2 ║ 100.0 ± 0.0 ║ 150/150 ║ 0,1,2 ║
║ NoCrashTown02-v3 ║ 95.0 ± 1.0 ║ 285/300 ║ 0,1,2 ║
║ NoCrashTown02-v4 ║ 93.3 ± 2.3 ║ 140/150 ║ 0,1,2 ║
║ NoCrashTown02-v5 ║ 45.5 ± 8.4 ║ 135/297 ║ 0,1,2 ║
║ NoCrashTown02-v6 ║ 45.3 ± 9.5 ║ 68/150 ║ 0,1,2 ║
║ StraightTown02-v1 ║ 100 ║ 100/100 ║ 0 ║
║ StraightTown02-v2 ║ 100 ║ 50/50 ║ 0 ║
║ TurnTown02-v1 ║ 100 ║ 100/100 ║ 0 ║
║ TurnTown02-v2 ║ 100 ║ 50/50 ║ 0 ║
╚═════════════════════╩══════════════╩═════════╩═══════╝
Stage 1(purely offline) Agent
╔Performance of model-32═══════════╦═════════╦═══════╗
║ Suite Name ║ Success Rate ║ Total ║ Seeds ║
╠═══════════════════╬══════════════╬═════════╬═══════╣
║ FullTown01-v1 ║ 93 ║ 93/100 ║ 0 ║
║ FullTown01-v2 ║ 98 ║ 49/50 ║ 0 ║
║ FullTown01-v3 ║ 94 ║ 94/100 ║ 0 ║
║ FullTown01-v4 ║ 96 ║ 48/50 ║ 0 ║
║ FullTown02-v1 ║ 94 ║ 94/100 ║ 0 ║
║ FullTown02-v2 ║ 96 ║ 48/50 ║ 0 ║
║ FullTown02-v3 ║ 92 ║ 92/100 ║ 0 ║
║ FullTown02-v4 ║ 94 ║ 47/50 ║ 0 ║
║ NoCrashTown01-v1 ║ 88.3 ± 1.5 ║ 265/300 ║ 0,1,2 ║
║ NoCrashTown01-v2 ║ 91.3 ± 3.1 ║ 137/150 ║ 0,1,2 ║
║ NoCrashTown01-v3 ║ 74.3 ± 3.8 ║ 223/300 ║ 0,1,2 ║
║ NoCrashTown01-v4 ║ 71.3 ± 4.6 ║ 107/150 ║ 0,1,2 ║
║ NoCrashTown01-v5 ║ 27.7 ± 3.5 ║ 83/300 ║ 0,1,2 ║
║ NoCrashTown01-v6 ║ 24.7 ± 2.3 ║ 37/150 ║ 0,1,2 ║
║ NoCrashTown02-v1 ║ 85.0 ± 2.6 ║ 255/300 ║ 0,1,2 ║
║ NoCrashTown02-v2 ║ 80.7 ± 2.3 ║ 121/150 ║ 0,1,2 ║
║ NoCrashTown02-v3 ║ 64.3 ± 4.0 ║ 193/300 ║ 0,1,2 ║
║ NoCrashTown02-v4 ║ 60.0 ± 4.0 ║ 90/150 ║ 0,1,2 ║
║ NoCrashTown02-v5 ║ 11.7 ± 2.3 ║ 35/300 ║ 0,1,2 ║
║ NoCrashTown02-v6 ║ 11.3 ± 3.1 ║ 17/150 ║ 0,1,2 ║
║ StraightTown01-v1 ║ 100 ║ 100/100 ║ 0 ║
║ StraightTown01-v2 ║ 100 ║ 50/50 ║ 0 ║
║ StraightTown02-v1 ║ 100 ║ 100/100 ║ 0 ║
║ StraightTown02-v2 ║ 100 ║ 50/50 ║ 0 ║
║ TurnTown01-v1 ║ 96 ║ 96/100 ║ 0 ║
║ TurnTown01-v2 ║ 100 ║ 50/50 ║ 0 ║
║ TurnTown02-v1 ║ 97 ║ 97/100 ║ 0 ║
║ TurnTown02-v2 ║ 100 ║ 50/50 ║ 0 ║
╚═══════════════════╩══════════════╩═════════╩═══════╝
Note that this is with different controller parameters from what we submitted for CoRL2019, yielding slightly different numbers.
The original raw numbers are shown below
╔Performance of model-32═══════════╦═════════╦═══════╗
║ Suite Name ║ Success Rate ║ Total ║ Seeds ║
╠═══════════════════╬══════════════╬═════════╬═══════╣
║ FullTown01-v1 ║ 94 ║ 94/100 ║ 0 ║
║ FullTown01-v2 ║ 98 ║ 49/50 ║ 0 ║
║ FullTown01-v3 ║ 95 ║ 95/100 ║ 0 ║
║ FullTown01-v4 ║ 92 ║ 46/50 ║ 0 ║
║ FullTown02-v1 ║ 94 ║ 94/100 ║ 0 ║
║ FullTown02-v2 ║ 98 ║ 49/50 ║ 0 ║
║ FullTown02-v3 ║ 88 ║ 88/100 ║ 0 ║
║ FullTown02-v4 ║ 90 ║ 45/50 ║ 0 ║
║ StraightTown01-v1 ║ 100 ║ 100/100 ║ 0 ║
║ StraightTown01-v2 ║ 100 ║ 50/50 ║ 0 ║
║ StraightTown02-v1 ║ 100 ║ 100/100 ║ 0 ║
║ StraightTown02-v2 ║ 100 ║ 50/50 ║ 0 ║
║ TurnTown01-v1 ║ 96 ║ 96/100 ║ 0 ║
║ TurnTown01-v2 ║ 100 ║ 50/50 ║ 0 ║
║ TurnTown02-v1 ║ 95 ║ 95/100 ║ 0 ║
║ TurnTown02-v2 ║ 98 ║ 49/50 ║ 0 ║
╚═══════════════════╩══════════════╩═════════╩═══════╝
Stage 2(online fine-tuned) Agent
╔Performance of model-10══════════╦═════════╦═══════╗
║ Suite Name ║ Success Rate ║ Total ║ Seeds ║
╠══════════════════╬══════════════╬═════════╬═══════╣
║ FullTown01-v1 ║ 100 ║ 100/100 ║ 0 ║
║ FullTown01-v2 ║ 100 ║ 50/50 ║ 0 ║
║ FullTown01-v3 ║ 100 ║ 100/100 ║ 0 ║
║ FullTown01-v4 ║ 100 ║ 50/50 ║ 0 ║
║ FullTown02-v1 ║ 100 ║ 100/100 ║ 0 ║
║ FullTown02-v2 ║ 100 ║ 50/50 ║ 0 ║
║ FullTown02-v3 ║ 98 ║ 98/100 ║ 0 ║
║ FullTown02-v4 ║ 98 ║ 49/50 ║ 0 ║
║ NoCrashTown01-v1 ║ 99.0 ± 0.0 ║ 297/300 ║ 0,1,2 ║
║ NoCrashTown01-v2 ║ 89.3 ± 3.1 ║ 134/150 ║ 0,1,2 ║
║ NoCrashTown01-v3 ║ 97.3 ± 1.2 ║ 292/300 ║ 0,1,2 ║
║ NoCrashTown01-v4 ║ 95.3 ± 3.1 ║ 143/150 ║ 0,1,2 ║
║ NoCrashTown01-v5 ║ 70.0 ± 4.8 ║ 196/280 ║ 0,1,2 ║
║ NoCrashTown01-v6 ║ 62.7 ± 2.3 ║ 94/150 ║ 0,1,2 ║
║ NoCrashTown02-v1 ║ 99.0 ± 0.0 ║ 297/300 ║ 0,1,2 ║
║ NoCrashTown02-v2 ║ 84.7 ± 3.1 ║ 127/150 ║ 0,1,2 ║
║ NoCrashTown02-v3 ║ 93.3 ± 2.1 ║ 280/300 ║ 0,1,2 ║
║ NoCrashTown02-v4 ║ 70.0 ± 2.0 ║ 105/150 ║ 0,1,2 ║
║ NoCrashTown02-v5 ║ 46.1 ± 4.0 ║ 128/278 ║ 0,1,2 ║
║ NoCrashTown02-v6 ║ 32.7 ± 9.5 ║ 49/150 ║ 0,1,2 ║
╚══════════════════╩══════════════╩═════════╩═══════╝
Note that this is run with different controller parameters from what we submitted for CoRL2019, yielding slightly better numbers.
The original raw numbers are shown below
╔Performance of model-10═══════════╦═════════╦═══════╗
║ Suite Name ║ Success Rate ║ Total ║ Seeds ║
╠═══════════════════╬══════════════╬═════════╬═══════╣
║ FullTown01-v1 ║ 100 ║ 100/100 ║ 0 ║
║ FullTown01-v2 ║ 100 ║ 50/50 ║ 0 ║
║ FullTown01-v3 ║ 100 ║ 100/100 ║ 0 ║
║ FullTown01-v4 ║ 96 ║ 48/50 ║ 0 ║
║ FullTown02-v1 ║ 98 ║ 98/100 ║ 0 ║
║ FullTown02-v2 ║ 100 ║ 50/50 ║ 0 ║
║ FullTown02-v3 ║ 99 ║ 99/100 ║ 0 ║
║ FullTown02-v4 ║ 100 ║ 50/50 ║ 0 ║
║ NoCrashTown01-v1 ║ 97.0 ± 1.0 ║ 291/300 ║ 0,1,2 ║
║ NoCrashTown01-v2 ║ 86.7 ± 4.2 ║ 130/150 ║ 0,1,2 ║
║ NoCrashTown01-v3 ║ 93.3 ± 0.6 ║ 280/300 ║ 0,1,2 ║
║ NoCrashTown01-v4 ║ 87.3 ± 3.1 ║ 131/150 ║ 0,1,2 ║
║ NoCrashTown01-v5 ║ 70.7 ± 4.5 ║ 212/300 ║ 0,1,2 ║
║ NoCrashTown01-v6 ║ 63.3 ± 3.1 ║ 95/150 ║ 0,1,2 ║
║ NoCrashTown02-v1 ║ 99.7 ± 0.6 ║ 299/300 ║ 0,1,2 ║
║ NoCrashTown02-v2 ║ 70.0 ± 4.0 ║ 105/150 ║ 0,1,2 ║
║ NoCrashTown02-v3 ║ 94.0 ± 3.0 ║ 281/299 ║ 0,1,2 ║
║ NoCrashTown02-v4 ║ 62.0 ± 2.0 ║ 93/150 ║ 0,1,2 ║
║ NoCrashTown02-v5 ║ 51.3 ± 3.1 ║ 154/300 ║ 0,1,2 ║
║ NoCrashTown02-v6 ║ 38.7 ± 6.4 ║ 58/150 ║ 0,1,2 ║
║ StraightTown01-v1 ║ 100 ║ 100/100 ║ 0 ║
║ StraightTown01-v2 ║ 100 ║ 50/50 ║ 0 ║
║ StraightTown02-v1 ║ 100 ║ 100/100 ║ 0 ║
║ StraightTown02-v2 ║ 100 ║ 50/50 ║ 0 ║
║ TurnTown01-v1 ║ 100 ║ 100/100 ║ 0 ║
║ TurnTown01-v2 ║ 96 ║ 48/50 ║ 0 ║
║ TurnTown02-v1 ║ 100 ║ 100/100 ║ 0 ║
║ TurnTown02-v2 ║ 100 ║ 50/50 ║ 0 ║
╚═══════════════════╩══════════════╩═════════╩═══════╝
License
This repo is released under the MIT License (please refer to the LICENSE file for details). Part of the PythonAPI and the map rendering code is borrowed from the official CARLA repo, which is under MIT license. The image augmentation code is borrowed from Coiltraine which is released under MIT license.