Home

Awesome

Mamba_FETrack

News

:dart: Abstract

RGB-Event based tracking is an emerging research topic, focusing on how to effectively integrate heterogeneous multi-modal data (synchronized exposure video frames and asynchronous pulse Event stream). Existing works typically employ Transformer based networks to handle these modalities and achieve decent accuracy through input-level or feature-level fusion on multiple datasets. However, these trackers require significant memory consumption and computational complexity due to the use of self-attention mechanism. This paper proposes a novel RGB-Event tracking framework, Mamba-FETrack, based on the State Space Model (SSM) to achieve high-performance tracking while effectively reducing computational costs and realizing more efficient tracking. Specifically, we adopt two modality-specific Mamba backbone networks to extract the features of RGB frames and Event streams. Then, we also propose to boost the interactive learning between the RGB and Event features using the Mamba network. The fused features will be fed into the tracking head for target object localization. Extensive experiments on FELT and FE108 datasets fully validated the efficiency and effectiveness of our proposed tracker. Specifically, our Mamba-based tracker achieves 43.5/55.6 on the SR/PR metric, while the ViT-S based tracker (OSTrack) obtains 40.0/50.9. The GPU memory cost of ours and ViT-S based tracker is 13.98GB and 15.44GB, which decreased about $9.5%$. The FLOPs and parameters of ours/ViT-S based OSTrack are 59GB/1076GB and 7MB/60MB, which decreased about $94.5%$ and $88.3%$, respectively. We hope this work can bring some new insights to the tracking field and greatly promote the application of the Mamba architecture in tracking.

FETrack Framework

<p align="center"> <img src="https://github.com/Event-AHU/Mamba_FETrack/blob/main/figures/Mamba_track_framework.jpg" alt="framework" width="700"/> </p>

Environment Settings

conda create -n mamba_fetrack python=3.10.13
conda activate mamba_fetrack

conda install cudatoolkit==11.8 -c nvidia
pip install torch==2.1.1 torchvision==0.16.1 torchaudio==2.1.1 --index-url https://download.pytorch.org/whl/cu118
conda install -c "nvidia/label/cuda-11.8.0" cuda-nvcc
conda install packaging
pip install -r vim_requirements.txt

Download the mamba-1.1.1 and source code and place it in the project path of Mamba_FETrack. Go to source code and install the corresponding environment.

cd mamba-1.1.1
pip install .

Download the casual-conv1d-1.1.3 and source code and place it in the project path of Mamba_FETrack. Go to source code and install the corresponding environment.

cd ..
cd causal-conv1d-1.1.3
pip install .

bash install.sh

python tracking/create_default_local_file.py --workspace_dir . --data_dir ./data --save_dir ./output

lib/train/admin/local.py  # paths about training
lib/test/evaluation/local.py  # paths about testing

Download Dataset

Download tracking datasets FELT OR FE108, and put it in ./data.

Download Checkpoint

Download pre-trained and put it under $/pretrained_models.

Download Trained Weights for Model

Download the trained model weights from Mamba_FETrack_ep0050.pth and put it under $/output/checkpoints/train/mamba_fetrack/mamba_fetrack_felt for test directly.

Training and Testing Script

# train
python tracking/train.py --script mamba_fetrack --config mamba_fetrack_felt --save_dir ./output --mode single --nproc_per_node 1 --use_wandb 0
python tracking/train.py --script mamba_fetrack --config mamba_fetrack_fe108 --save_dir ./output --mode single --nproc_per_node 1 --use_wandb 0

# test
python tracking/test.py mamba_fetrack mamba_fetrack_felt --dataset felt --threads 1 --num_gpus 1
python tracking/test.py mamba_fetrack mamba_fetrack_fe108 --dataset fe108 --threads 1 --num_gpus 1

Evaluation Toolkit

  1. Download the FELT_eval_toolkit from FELT_eval_toolkit (Passcode：AHUT), and open it with Matlab (over Matlab R2020).
  2. add your tracking results (Passcode：AHUT) in $/felt_tracking_results/ and modify the name in $/utils/config_tracker.m
  3. run Evaluate_FELT_benchmark_SP_PR_only.m for the overall performance evaluation, including AUC, PR, NPR.
  4. run plot_BOC.m for BOC score evaluation and figure plot.
  5. run plot_radar.m for attributes radar figrue plot.
  6. run Evaluate_FELT_benchmark_attributes.m for attributes analysis and figure saved in $/res_fig/.
  1. add your tracking results (Passcode：AHUT) in $/output/test/tracking_results/mamba_fetrack/
  2. run $/tracking/analysis_results.py for the overall performance evaluation, including AUC, PR, NPR.

Visualization

<p align="center"> <img src="https://github.com/Event-AHU/Mamba_FETrack/blob/main/figures/activationMAPs.jpg" alt="framework" width="700"/> </p>

Experimental Results

<p align="center"> <img src="https://github.com/Event-AHU/Mamba_FETrack/blob/main/figures/FE108.jpg" alt="framework" width="700"/> </p> <p align="center"> <img src="https://github.com/Event-AHU/Mamba_FETrack/blob/main/figures/FELT.png" alt="framework" width="700"/> </p>

Acknowledgment

[OSTrack] [Mamba] [FELT] [CEUTrack] [FE108]

:newspaper: Citation

If you think this paper is helpful, please feel free to leave a star ⭐️ and cite our paper:

@misc{huang2024mambafetrack,
      title={Mamba-FETrack: Frame-Event Tracking via State Space Model}, 
      author={Ju Huang and Shiao Wang and Shuai Wang and Zhe Wu and Xiao Wang and Bo Jiang},
      year={2024},
      eprint={2404.18174},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}