Awesome
Multi-source Templates Learning for Real-time Aerial Object Tracking
This is the official code for the paper "Multi-source Templates Learning for Real-time Aerial Object Tracking".In this work, we present an efficient Aerial Object Tracking method via Multi-source Templates named MSTL.
Highlights
Real-Time Speed on edge platform.
Our tracker can run ~200fps on GPU, ~100fps on CPU, and ~20 on Nvidia Jetson Xavier NX platform. After tensorRT to accelerate, the speed can reach , ~60fps on Jetson Xavier NX, ~19 fps on Jetson Nano.
Opposing to previous aerial trackers which evaluate on high-end platform(like Jetson AGX/Orin Series), the proposed tracker can run on extremely cheap edge platform: Jetson Nano and Jetson Xavier NX.
Competitive performance.
| Year | Speed(fps) | UAV123(Prec.) | UAV123@10fpsUAV123(Prec.) | UAV20LUAV123(Prec.) | |
|---|---|---|---|---|---|
| Ours | 209 | 82.35 | 83.50 | 83.59 | |
| TCTrack | CVPR 2022 | 128 | 80.05 | 77.39 | 67.20 |
| HIFT | ICCV 2021 | 137 | 78.70 | 74.87 | 76.32 |
Demo
Quick Start
Environment Preparing
python 3.7.3
pytorch 1.11.0
opencv-python 4.5.5.64
Training
First, you need to set paths for training datasets in lib/train/admin/local.py.
Then, run the following commands for training.
python lib/train/run_training.py
Evaluation
First, you need to set paths for this project in lib/test/evaluation/local.py.
Then, run the following commands for evaluation on four datasets.
- UAV123
python tracking/test.py MSTL MSTL --dataset uav
- UAV20L
python tracking/test.py MSTL MSTL --dataset uavl
- UAV@10fps
python tracking/test.py MSTL MSTL --dataset uav10
- UAV -x
python tracking/test.py MSTL MSTL --dataset uavd
Trained model and Row results
The trained models, the training logs, and the raw tracking results are provided in the model zoo
MSTL framework for other transformer-based trackers.
To use our framework for other transformer-based trackers, we jointly trained the original tracker with an additional prediction head. The head takes outputs of the transformer encoder(or transformer-based structure) as inputs and predicts the bounding box of the target directly.
As an example, we use TransT(CVPR2021) with 4 feature integration layers (Each layer with 2 self-Attention and 2 Cross-Attention) to demonstrate how to implement the proposed decoupling strategy.
- During training:
- Step 1: Feed the outputs of the second layers into an additional cross-attention mechanism to fuse the search and template features.
- Step 2: Use the outputs of the cross-attention as input for prediction heads to locate the target.
- Step 3: Train the original model together with the additional cross-attention layer and prediction heads.
- During inference:
- We use the outputs of the second layer to locate the target.
| Tracker | Original Succ. (UAV123) | Original params | Succ. (After decoupling) | params(After decoupling) |
|---|---|---|---|---|
| TransT | 69.1 | 23.0M | - | 16.7M |
| STARK-S | 68.35 | 28.079M | 68.55 | 18.616M |
For further information, we will make the corresponding codes and pre-trained models available.
About the UAV platform
The platform mainly consists of four parts, i.e., a Pixhawk flight controller, a Figure Number transmissions, a visual camera and a Jetson Xavier NX onboard computer. The onboard computer can obtain the video flow through the USB port. The ground station computer can remotely access the onboard computer and select the target to be tracked through data transmission.
Acknowledgement
- Thanks for the PyTracking and stark Library, which helps us to quickly implement our ideas. We would like to thank their authors for providing great frameworks and toolkits.