Awesome
Code for Hierarchical Multi-modal Fusion Tracker in CVPR2022 paper Visible-Thermal UAV Tracking: A Large-Scale Benchmark and New Baseline, which is a strong baseline for RGB-T tracking.
Framework
Three complementary modules are introduced for multi-modal fusion
- Complementary Image Fusion(CIF): CIF aims to use a shared backbone to extract complementary information and Kullback–Leibler divergence loss is introduced to unify the feature distribution.
- Discriminative Feature Fusion(DFF): DFF aims to build individual representations for both modalities and learns a channel-wise modality weight to fuse them.
- Adaptive Decision Fusion(ADF): ADF is to adaptively provide the final response by considering the results of two branches and the modality confidence.
Model Zoo
The pretrained model are available at [GoogleDrive] and [BaiduDisk]
Get Started
Set up Anaconda environment
conda create -n HMFT python=3.6
conda activate HMFT
cd $Path_to_HMFT$
bash install.sh
Run demo sequence
cd $Path_to_HMFT$/mfDiMP/pytracking
python demo.py
Test on current benchmarks
cd $Path_to_HMFT$/mfDiMP/pytracking
python run_VTUAV.py
python run_GTOT.py
python run_RGBT210.py
python run_RGBT234.py
Training
Results
The results can be found at [GoogleDrive] and [BaiduDisk]
Citation
If you find our work useful, please cite
@InProceedings{Zhang_CVPR22_VTUAV, author = {Zhang Pengyu and Jie Zhao and Dong Wang and Huchuan Lu and Xiang Ruan}, title = {Visible-Thermal UAV Tracking: A Large-Scale Benchmark and New Baseline}, booktitle = {Proceedings of the IEEE conference on computer vision and pattern recognition}, year = {2022} }