Awesome

RGBD Tracking Overview

Related papers will be continuously updated. Please feel free to contact jinyu.yang96@outlook.com, zhe.li.cs@outlook.com.

BTW: Welcome to visit the SUSTech-VIP Lab as research assistants and visiting students. Looking forward to working with you! [加入我们] [Join us] [Arxiv]

:rocket: News

:boom: Track-Anything Development Underway: Give it a Try Now!

:boom: DMT is accepted by ECCV workshop 2022.

:boom: ProTrack is accepted by ACMMM 2022.

Overview

RGBD Tracking Express

2023

ViPT: Jiawen Zhu, Simiao Lai, Xin Chen, Dong Wang, Huchuan Lu. Visual Prompt Multi-Modal Tracking. In CVPR, 2023. [Paper] [Code].
SPT: Xue-Feng Zhu, Tianyang Xu, Zhangyong Tang, Zucheng Wu, Haodong Liu, Xiao Yang, Xiao-Jun Wu, Josef Kittler. RGBD1K: A Large-scale Dataset and Benchmark for RGB-D Object Tracking. In AAAI, 2023. [Paper] [Project]

2022

DMT: Shang Gao, Jinyu Yang, Zhe Li, Feng Zheng, Aleš Leonardis, Jingkuan Song. Learning Dual-Fused Modality-Aware Representations for RGBD Tracking. In ECCVW, 2022. [Paper]
ProTrack: Jinyu Yang, Zhe Li, Feng Zheng, Aleš Leonardis, Jingkuan Song. Prompting for Multi-Modal Tracking. In ACMMM, 2022. [Paper]

2021

DeT: Song Yan, Jinyu Yang, Jani Käpylä, Feng Zheng, Aleš Leonardis, Joni-Kristian Kämäräinen. DepthTrack: Unveiling the Power of RGBD Tracking. In ICCV, 2021. [Paper] [Code]
TSDM: Pengyao Zhao, Quanli Liu, Wei Wang and Qiang Guo. TSDM: Tracking by SiamRPN++ with a Depth-refiner and a Mask-generator. In ICPR, 2021. [Paper] [Code]
3s-RGBD: Feng Xiao, Qiuxia Wu, Han Huang. Single-scale siamese network based RGB-D object tracking with adaptive bounding boxes. In Neurocomputing, 2021. [Paper]

2020

DAL: Yanlin Qian, Alan Lukezic, Matej Kristan, Joni-Kristian Kämäräinen, Jiri Matas. DAL : A deep depth-aware long-term tracker. In ICPR, 2020. [Code]
RF-CFF: Yong Wang, Xian Wei, Hao Shen, Lu Ding, Jiuqing Wan. Robust fusion for RGB-D tracking using CNN features. In Applied Soft Computing Journal, 2020. [Paper]
SiamOC: Wenli Zhang, Kun Yang, Yitao Xin, Rui Meng. An Occlusion-Aware RGB-D Visual Object Tracking Method Based on Siamese Network. In ICSP, 2020. [Paper]
WCO: Weichun Liu, Xiaoan Tang, Chengling Zhao. Robust RGBD Tracking via Weighted Convlution Operators. In Sensors 20(8), 2020. [Paper]

2019

3DMS: Alexander Gutev, Carl James Debono. Exploiting Depth Information to Increase Object Tracking Robustness. In ICST 2019. [Paper]
CA3DMS: Ye Liu, Xiao-Yuan Jing, Jianhui Nie, Hao Gao, Jun Liu, Guo-Ping Jiang. Context-Aware Three-Dimensional Mean-Shift With Occlusion Handling for Robust Object Tracking in RGB-D Videos. In TMM 21(3), 2019. [Paper] [Code]
Depth-CCF: Guanqun Li, Lei Huang, Peichang Zhang, Qiang Li, YongKai Huo. Depth Information Aided Constrained correlation Filter for Visual Tracking. In GSKI 2018. [Paper]
ECO_TA: Yangliu Kuai, Gongjian Wen, Dongdong Li, Jingjing Xiao. Target-Aware Correlation Filter Tracking in RGBD Videos. In Sensors 19(20), 2019. [Paper]
H-FCN: Ming-xin Jiang, Chao Deng, Jing-song Shan, Yuan-yuan Wang, Yin-jie Jia, Xing Sun. Hierarchical multi-modal fusion FCN with attention model for RGB-D tracking. In Information Fusion [Paper]
OTR: Ugur Kart, Alan Lukezic, Matej Kristan, Joni-Kristian Kamarainen, Jiri Matas. Object Tracking by Reconstruction with View-Specific Discriminative Correlation Filters. In CVPR 2019. [Paper] [Code]
RGBD-OD: Yujun Xie, Yao Lu, Shuang Gu. RGB-D Object Tracking with Occlusion Detection. In CIS 2019. [Paper]

2018

CSR-rgbd: Uğur Kart, Joni-Kristian Kämäräinen, Jiří Matas. How to Make an RGBD Tracker? In ECCV Workshop 2018. [Paper] [Code]
DM-DCF: Uğur Kart, Joni-Kristian Kämäräinen, Jiří Matas, Lixin Fan, Francesco Cricri. Depth Masked Discriminative Correlation Filter. In ICPR 2018. [Paper]
OACPF: Yayu Zhai, Ping Song, Zonglei Mou, Xiaoxiao Chen, Xiongjun Liu. Occlusion-Aware Correlation Particle FilterTarget Tracking Based on RGBD Data. In Access (6), 2018. [Paper]
RT-KCF: Han Zhang, Meng Cai, Jianxun Li. A Real-time RGB-D tracker based on KCF. In CCDC 2018. [Paper]
SEOH: Jiaxu Leng, Ying Liu. Real-Time RGB-D Visual Tracking With ScaleEstimation and Occlusion Handling. In Access (6), 2018. [Paper]
STC: Jingjing Xiao, Rustam Stolkin, Yuqing Gao, Aleš Leonardis. Robust Fusion of Color and Depth Data for RGB-D Target Tracking Using Adaptive Range-Invariant Depth Models and Spatio-Temporal Consistency Constraints. In TC 48(8) 2018. [Paper] [Code]

2017

ODIOT: Wei-Long Zheng, Shan-Chun Shen, Bao-Liang Lu. Online Depth Image-Based Object Tracking with Sparse Representation and Object Detection. In Neural Process Letters, 2017. [Paper]
ROTSL: Zi-ang Ma, Zhi-yu Xiang. Robust Object Tracking with RGBD-based Sparse Learning. In ITEE (18), 2017. [Paper]

2016

DLS: Ning An, Xiao-Guang Zhao, Zeng-Guang Hou. Online RGB-D Tracking via Detection-Learning-Segmentation. In ICPR 2016. [Paper]
DS-KCF_shape: Sion Hannuna, Massimo Camplani, Jake Hall, Majid Mirmehdi, Dima Damen, Tilo Burghardt, Adeline Paiement, Lili Tao. DS-KCF: A Real-time Tracker for RGB-D Data. In RTIP (16), 2016. [Paper] [Code]
3D-T: Adel Bibi, Tianzhu Zhang, Bernard Ghanem. 3D Part-Based Sparse Tracker with Automatic Synchronization and Registration. In CVPR 2016. [Paper] [Code]
OAPF: Kourosh Meshgia, Shin-ichi Maedaa, Shigeyuki Obaa, Henrik Skibbea, Yu-zhe Lia, Shin Ishii. Occlusion Aware Particle Filter Tracker to Handle Complex and Persistent Occlusions. In CVIU (150), 2016. [Paper]

2015

CDG: Huizhang Shi, Changxin Gao, Nong Sang. Using Consistency of Depth Gradient to Improve Visual Tracking in RGB-D sequences. In CAC, 2015. [Paper]
DS-KCF: Massimo Camplani, Sion Hannuna, Majid Mirmehdi, Dima Damen, Adeline Paiement, Lili Tao, Tilo Burghardt. Real-time RGB-D Tracking with Depth Scaling Kernelised Correlation Filters and Occlusion Handling. In BMVC, 2015. [Paper] [Code]
DOHR: Ping Ding, Yan Song. Robust Object Tracking Using Color and Depth Images with a Depth Based Occlusion Handling and Recovery. In FSKD, 2015. [Paper]
ISOD: Yan Chen, Yingju Shen, Xin Liu, Bineng Zhong. 3D Object Tracking via Image Sets and Depth-Based Occlusion Detection. In SP (112), 2015. [Paper]
OL3DC: Bineng Zhong, Yingju Shen, Yan Chen, Weibo Xie, Zhen Cui, Hongbo Zhang, Duansheng Chen ,Tian Wang, Xin Liu, Shujuan Peng, Jin Gou, Jixiang Du, Jing Wang, Wenming Zheng. Online Learning 3D Context for Robust Visual Tracking. In Neurocomputing (151), 2015. [Paper]

2014

MCBT: Qi Wang, Jianwu Fang, Yuan Yuan. Multi-Cue Based Tracking. In Neurocomputing (131), 2014. [Paper]

2013

PT: Shuran Song, Jianxiong Xiao. Tracking Revisited using RGBD Camera: Unified Benchmark and Baselines. In ICCV, 2013. [Paper] [Project]

2012

AMCT: Germán Martín García, Dominik Alexander Klein, Jörg Stückler, Simone Frintrop, Armin B. Cremers. Adaptive Multi-cue 3D Tracking of Arbitrary Objects. In JDOS, 2012. [Paper]

Datasets

PTB: Shuran Song, Jianxiong Xiao. Tracking Revisited using RGBD Camera: Unified Benchmark and Baselines. In ICCV, 2013. [Paper] [Project] [Dataset]
STC: Jingjing Xiao, Rustam Stolkin, Yuqing Gao, Aleš Leonardis. Robust Fusion of Color and Depth Data for RGB-D Target Tracking Using Adaptive Range-Invariant Depth Models and Spatio-Temporal Consistency Constraints. In TC 48(8) 2018. [Paper] [Dataset]
CDTB:Alan Lukezic, Ugur Kart, Jani Kapyla, Ahmed Durmush, Joni-Kristian Kamarainen, Jiri Matas, Matej Kristan. CDTB: A Color and Depth Visual Object Tracking Dataset and Benchmark. In ICCV, 2019. [Paper] [Dataset]
DepthTrack: Song Yan, Jinyu Yang, Jani Käpylä, Feng Zheng, Aleš Leonardis, Joni-Kristian Kämäräinen. DepthTrack : Unveiling the Power of RGBD Tracking. In ICCV, 2021. [Paper] [Dataset]

Challenges

VOT-2019: [Website] [Paper]
VOT-2020: [Website] [Paper]
VOT-2021: [Website] [Paper]

<!-- A comprehensive survey on multimodal tracking [[Paper]](https://arxiv.org/pdf/2012.04176.pdf), including RGB-T and RGB-D tracking methods. This list will be long-term updating. If your related paper is missing in this review, feel free to contact [pyzhang@mail.dlut.edu.cn](mailto:pyzhang@mail.dlut.edu.cn). ![alt text](https://github.com/zhang-pengyu/Multimodal_tracking_survey/blob/master/Paper_list_with_taxonomy.png) # Citation If our paper and repositority are helpful for your work, please cite us, @article{Zhang_Arxiv20_MM_tracking_survey,\ author = {Pengyu Zhang and Dong Wang and Huchuan Lu},\ title = {Multi-modal Visual Tracking: Review and Experimental Comparison},\ journal={arXiv preprint arXiv:2012.04176},\ year={2020}\ } # Multimodal Tracking List ## RGB-D tracking ### 2020 * **WCO**: Weichun Liu, Xiaoan Tang, Chengling Zhao. Robust RGBD Tracking via Weighted Convlution Operators. In _Sensors_ 20(8), 2020. [[Paper]](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8950173/) ### 2019 * **3DMS**: Alexander Gutev, Carl James Debono. Exploiting Depth Information to Increase Object Tracking Robustness. In _ICST_ 2019. [[Paper]](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8861628/) * **OTR**: Ugur Kart, Alan Lukezic, Matej Kristan, Joni-Kristian Kamarainen, Jiri Matas. Object Tracking by Reconstruction with View-Specific Discriminative Correlation Filters. In _CVPR_ 2019. [[Paper]](https://openaccess.thecvf.com/content_CVPR_2019/papers/Kart_Object_Tracking_by_Reconstruction_With_View-Specific_Discriminative_Correlation_Filters_CVPR_2019_paper.pdf) [[Code]](https://github.com/ugurkart/OTR) * **TACF**: Yangliu Kuai, Gongjian Wen, Dongdong Li, Jingjing Xiao. Target-Aware Correlation Filter Tracking in RGBD Videos. In _Sensors_ 19(20), 2019. [[Paper]](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8752050) * **CA3DMS**: Ye Liu, Xiao-Yuan Jing, Jianhui Nie, Hao Gao, Jun Liu, Guo-Ping Jiang. Context-Aware Three-Dimensional Mean-Shift With Occlusion Handling for Robust Object Tracking in RGB-D Videos. In _TMM_ 21(3), 2019. [[Paper]](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8425768) * **OTOD**: Yujun Xie, Yao Lu, Shuang Gu. RGB-D Object Tracking with Occlusion Detection. In _CIS_ 2019. [[Paper]](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9023755) ### 2018 * **CSR-rgbd**: Uğur Kart, Joni-Kristian Kämäräinen, Jiří Matas. How to Make an RGBD Tracker? In _ECCV Workshop_ 2018. [[Paper]](https://link.springer.com/chapter/10.1007/978-3-030-11009-3_8) [[Code]](http://tracking.cs.princeton.edu/) * **DMDCF**: Uğur Kart, Joni-Kristian Kämäräinen, Jiří Matas, Lixin Fan, Francesco Cricri. Depth Masked Discriminative Correlation Filter. In _ICPR_ 2018. [[Paper]](https://arxiv.org/pdf/1802.09227.pdf) * **SEOH**: Jiaxu Leng, Ying Liu. Real-Time RGB-D Visual Tracking With ScaleEstimation and Occlusion Handling. In _Access_ (6), 2018. [[Paper]](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8353501) * **ARDM**: Jingjing Xiao, Rustam Stolkin, Yuqing Gao, Aleš Leonardis. Robust Fusion of Color and Depth Data for RGB-D Target Tracking Using Adaptive Range-Invariant Depth Models and Spatio-Temporal Consistency Constraints. In _TC_ 48(8) 2018. [[Paper]](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8026575) [[Code]](https://github.com/shine636363/RGBDtracker) * **OACPF**: Yayu Zhai, Ping Song, Zonglei Mou, Xiaoxiao Chen, Xiongjun Liu. Occlusion-Aware Correlation Particle FilterTarget Tracking Based on RGBD Data. In _Access_ (6), 2018. [[Paper]](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8463446) * **CCF**: Guanqun Li, Lei Huang, Peichang Zhang, Qiang Li, YongKai Huo. Depth Information Aided Constrained correlation Filter for Visual Tracking. In _GSKI_ 2018. [[Paper]](https://iopscience.iop.org/article/10.1088/1755-1315/234/1/012005) * **RTKCF**: Han Zhang, Meng Cai, Jianxun Li. A Real-time RGB-D tracker based on KCF. In _CCDC_ 2018. [[Paper]](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8407972) ### 2017 * **ROTSL**: Zi-ang Ma, Zhi-yu Xiang. Robust Object Tracking with RGBD-based Sparse Learning. In _ITEE_ (18), 2017. [[Paper]](https://link.springer.com/article/10.1631/FITEE.1601338) ### 2016 * **DLST**: Ning An, Xiao-Guang Zhao, Zeng-Guang Hou. Online RGB-D Tracking via Detection-Learning-Segmentation. In _ICPR_ 2016. [[Paper]](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7899805) * **DSKCF**: Sion Hannuna, Massimo Camplani, Jake Hall, Majid Mirmehdi, Dima Damen, Tilo Burghardt, Adeline Paiement, Lili Tao. DS-KCF: A Real-time Tracker for RGB-D Data. In _RTIP_ (16), 2016. [[Paper]](https://link.springer.com/content/pdf/10.1007/s11554-016-0654-3.pdf) [[Code]](https://github.com/mcamplan/DSKCF_JRTIP2016) * **3DT**: Adel Bibi, Tianzhu Zhang, Bernard Ghanem. 3D Part-Based Sparse Tracker with Automatic Synchronization and Registration. In _CVPR_ 2016. [[Paper]](https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Bibi_3D_Part-Based_Sparse_CVPR_2016_paper.pdf) [[Code]](https://github.com/adelbibi/3D-Part-Based-Sparse-Tracker-with-Automatic-Synchronization-and-Registration) * **OAPF**: Kourosh Meshgia, Shin-ichi Maedaa, Shigeyuki Obaa, Henrik Skibbea, Yu-zhe Lia, Shin Ishii. Occlusion Aware Particle Filter Tracker to Handle Complex and Persistent Occlusions. In _CVIU_ (150), 2016. [[Paper]](http://ishiilab.jp/member/meshgi-k/files/ai/prl14/OAPF.pdf) ### 2015 * **ISOD**: Yan Chen, Yingju Shen, Xin Liu, Bineng Zhong. 3D Object Tracking via Image Sets and Depth-Based Occlusion Detection. In _SP_ (112), 2015. [[Paper]](https://www.sciencedirect.com/science/article/pii/S0165168414004204) * **DSOH**: Massimo Camplani, Sion Hannuna, Majid Mirmehdi, Dima Damen, Adeline Paiement, Lili Tao, Tilo Burghardt. Real-time RGB-D Tracking with Depth Scaling Kernelised Correlation Filters and Occlusion Handling. In _BMVC_, 2015. [[Paper]](https://core.ac.uk/reader/78861956) [[Code]](https://github.com/mcamplan/DSKCF_BMVC2015) * **DOHR**: Ping Ding, Yan Song. Robust Object Tracking Using Color and Depth Images with a Depth Based Occlusion Handling and Recovery. In _FSKD_, 2015. [[Paper]](https://ieeexplore.ieee.org/document/7382068) * **CDG**: Huizhang Shi, Changxin Gao, Nong Sang. Using Consistency of Depth Gradient to Improve Visual Tracking in RGB-D sequences. In _CAC_, 2015. [[Paper]](https://ieeexplore.ieee.org/document/7382555) * **OL3DC**: Bineng Zhong, Yingju Shen, Yan Chen, Weibo Xie, Zhen Cui, Hongbo Zhang, Duansheng Chen ,Tian Wang, Xin Liu, Shujuan Peng, Jin Gou, Jixiang Du, Jing Wang, Wenming Zheng. Online Learning 3D Context for Robust Visual Tracking. In _Neurocomputing_ (151), 2015. [[Paper]](https://www.sciencedirect.com/science/article/pii/S0925231214013757) ### 2014 * **MCBT**: Qi Wang, Jianwu Fang, Yuan Yuan. Multi-Cue Based Tracking. In _Neurocomputing_ (131), 2014. [[Paper]](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.700.8771&rep=rep1&type=pdf) ### 2012 * **AMCT**: Germán Martín García, Dominik Alexander Klein, Jörg Stückler, Simone Frintrop, Armin B. Cremers. Adaptive Multi-cue 3D Tracking of Arbitrary Objects. In _JDOS_, 2012. [[Paper]](https://link.springer.com/chapter/10.1007/978-3-642-32717-9_36) ### Datasets * **PTB**: Shuran Song, Jianxiong Xiao. Tracking Revisited using RGBD Camera: Unified Benchmark and Baselines. In _ICCV_, 2013. [[Paper]](https://vision.princeton.edu/projects/2013/tracking/paper.pdf) [[Project]](https://tracking.cs.princeton.edu/index.html) [[Dataset]](https://tracking.cs.princeton.edu/dataset.html) * **STC**: Jingjing Xiao, Rustam Stolkin, Yuqing Gao, Aleš Leonardis. Robust Fusion of Color and Depth Data for RGB-D Target Tracking Using Adaptive Range-Invariant Depth Models and Spatio-Temporal Consistency Constraints. In _TC_ 48(8) 2018. [[Paper]](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8026575) [[Dataset]](https://beardatashare.bham.ac.uk/dl/fiVnhJRjkyNN8QjSAoiGSiBY/RGBDdataset.zip) * **CDTB**:Alan Lukezic, Ugur Kart, Jani Kapyla, Ahmed Durmush, Joni-Kristian Kamarainen, Jiri Matas, Matej Kristan. CDTB: A Color and Depth Visual Object Tracking Dataset and Benchmark. In _ICCV_, 2019. [[Paper]](https://arxiv.org/pdf/1907.00618.pdf) [[Project]](https://www.vicos.si/Projects/CDTB) [[Dataset]](https://www.votchallenge.net/vot2019/dataset.html) * **DepthTrack**: Song Yan, Jinyu Yang, Jani Käpylä, Feng Zheng, Aleš Leonardis, Joni-Kristian Kämäräinen. DepthTrack : Unveiling the Power of RGBD Tracking. In ArXiv, 2021. [[Paper]](https://arxiv.org/abs/2108.13962) * **RGBD1K**: Xue-Feng Zhu, Tianyang Xu, Zhangyong Tang, Zucheng Wu, Haodong Liu, Xiao Yang, Xiao-Jun Wu, Josef Kittler. RGBD1K: A Large-scale Dataset and Benchmark for RGB-D Object Tracking. In _AAAI_, 2023. [[Paper]](https://arxiv.org/pdf/2208.09787.pdf)