Awesome

LoRAT

This is the official repository for ECCV 2024 Tracking Meets LoRA: Faster Training, Larger Model, Stronger Performance (LoRAT).

[Models] [Raw Results] [Poster]

Prerequisites

Environment

Assuming you have a working python environment with pip installed.

system packages (ubuntu)

apt update
apt install -y libturbojpeg

install pytorch

Can be skipped if using NGC container. PyTorch version should be >= 2.0.

pip install torch torchvision

extra python packages

pip install -r requirements.txt

This codebase should also work on Windows and macOS for debugging purposes.

Dataset

Download

Unzip

The paths should be organized as follows:

LaSOT
├── airplane
├── basketball
...
├── training_set.txt
└── testing_set.txt

LaSOT_Extension
├── atv
├── badminton
...
└── wingsuit

GOT-10k
├── train
│   ├── GOT-10k_Train_000001
│   ...
├── val
│   ├── GOT-10k_Val_000001
│   ...
└── test
    ├── GOT-10k_Test_000001
    ...
    
TrackingNet
├── TEST
├── TRAIN_0
...
└── TRAIN_11

COCO
├── annotations
│   ├── instances_train2017.json
│   └── instances_val2017.json
└── images
    ├── train2017
    │   ├── 000000000009.jpg
    │   ├── 000000000025.jpg
    │   ...
    └── val2017
        ├── 000000000139.jpg
        ├── 000000000285.jpg
        ...
TNL2K_TEST
├── advSamp_Baseball_game_002-Done
├── advSamp_Baseball_video_01-Done
...

Prepare `consts.yaml`

Copy consts.template.yaml as consts.yaml and fill in the paths.

LaSOT_PATH: '/path/to/lasot'
LaSOT_Extension_PATH: '/path/to/lasot_ext'
GOT10k_PATH: '/path/to/got10k'
TrackingNet_PATH: '/path/to/trackingnet'
COCO_2017_PATH: '/path/to/coco2017'
TNL2K_TEST_PATH: '/path/to/tnl2k_test'

Login to wandb (optional)

wandb login

Training & Evaluation

Note: Our code performs evaluation automatically when model training is complete.

Model weight is saved in /path/to/output/run_id/checkpoint/epoch_{last}/model.bin.
Performance metrics can be found on terminal output and wandb dashboard.
Tracking results are saved in /path/to/output/run_id/eval/epoch_{last}/.

Using run.sh helper script (Linux with NVIDIA GPU only)

# Train and evaluate LoRAT-B-224 model on all GPUs
./run.sh LoRAT dinov2 --output_dir /path/to/output
# Train and evaluate LoRAT-L-224 model on all GPUs
./run.sh LoRAT dinov2 --output_dir /path/to/output --mixin large
# Train and evaluate LoRAT-g-378 model on all GPUs
./run.sh LoRAT dinov2 --output_dir /path/to/output --mixin giant_378
# Train and evaluate LoRAT-L-224 model following GOT-10k protocol on all GPUs
./run.sh LoRAT dinov2 --output_dir /path/to/output --mixin large --mixin got10k
# Train and evaluate on specific GPUs
./run.sh LoRAT dinov2 --output_dir /path/to/output --device_ids 0,1,2,3
# Train and evaluate on multiple nodes
./run.sh LoRAT dinov2 --output_dir /path/to/output --nnodes $num_nodes --node_rank $node_rank --master_address $master_node_ip --date 2024.03.07-04.59.08-976343

You can set the default settings, e.g. output_dir, in run.sh.

Call main.py directly

# Train and evaluate LoRAT-B-224 model on single GPU
python main.py LoRAT dinov2 --output_dir /path/to/output

# Train and evaluate LoRAT-B-224 model on CPU
python main.py LoRAT dinov2 --output_dir /path/to/output --device cpu

# Train and evaluate LoRAT-B-224 model on all GPUs
python main.py LoRAT dinov2 --distributed_nproc_per_node $num_gpus --distributed_do_spawn_workers --output_dir /path/to/output

# Train and evaluate LoRAT-B-224 model on multiple nodes, run_id need to be set manually
python main.py LoRAT dinov2 --master_address $master_address --distributed_node_rank $node_rank distributed_nnodes $num_nodes --distributed_nproc_per_node $num_gpus --distributed_do_spawn_workers --output_dir /path/to/output --run_id $run_id

See python main.py --help for more options.

Note: If you encounter any issues with torch.compile, disable is with --mixin disable_torch_compile.

Note: You can disable wandb logging with --disable_wandb.

Evaluation

Our code performs evaluation automatically when model training is complete. You can run evaluation only with the following command:

# evaluation only, on all datasets, defined in config/_dataset/test.yaml
./run.sh LoRAT dinov2 --output_dir /path/to/output --mixin evaluation --weight_path /path/to/weight.bin

The evaluated datasets are defined in config/_dataset/test.yaml.

Note that, as defined in config/LoRAT/run.yaml, we evaluate LaSOT Extension dataset three times. The final performance is the average of the three evaluations.

TrackingNet evaluation

Once the full evaluation is done, result files are saved in /path/to/output/run_id/eval/epoch_{last}/TrackingNet-test.zip.

Submit this file to the TrackingNet evaluation server to get the result of TrackingNet test split.

Train and evaluate with GOT-10k dataset

# Train and evaluate LoRAT-B-224 model following GOT-10k protocol on all GPUs
./run.sh LoRAT dinov2 --output_dir /path/to/output --mixin got10k

Submit /path/to/output/run_id/eval/epoch_{last}/GOT10k-test.zip to the GOT-10k evaluation server to get the result of GOT-10k test split.

Evaluation only:

# evaluation only, on GOT-10k dataset
./run.sh LoRAT dinov2 --output_dir /path/to/output --mixin got10k --mixin evaluation --weight_path /path/to/weight.bin

Note that, as defined in config/LoRAT/_mixin/got10k.yaml, we evaluate GOT-10k dataset three times.

VOT toolkit integration

Install VOT toolkit

pip install vot-toolkit

Download VOT dataset

prepare the VOT dataset by running the following command:

cd /path/to/vot_workspace
vot initialize vot_stack(vots2024/main|tests/multiobject)

fill the path to the VOT dataset in consts.yaml

VOTS2023_PATH: '/path/to/vots2023_workspace/sequences'
VOT_TESTS_MULTIOBJECT_PATH: '/path/to/vot_tests_workspace/sequences'

Run VOT experiments

# Run VOT experiment (vots2024/main stack) on LoRAT-g-378 with SAM-H segmentation model
python vot_main.py vots2024/main LoRAT dinov2 /path/to/output --mixin giant_378 --mixin segmentify_sam_h --tracker_name LoRAT  --weight_path /path/to/lorat_model_weight.bin

Custom Dataset

This page describes how to create a custom dataset for training and evaluation.

Citation

@inproceedings{lorat,
  title={Tracking Meets LoRA: Faster Training, Larger Model, Stronger Performance},
  author={Lin, Liting and Fan, Heng and Zhang, Zhipeng and Wang, Yaowei and Xu, Yong and Ling, Haibin},
  booktitle={ECCV},
  year={2024}
}