Awesome

MambaPro is a novel multi-modal object ReID framework that integrates CLIP's pre-trained capabilities with state-of-the-art multi-modal aggregation techniques. Using Parallel Feed-Forward Adapters (PFA), Synergistic Residual Prompts (SRP), and the innovative Mamba Aggregation (MA) mechanism, it achieves robust performance with reduced computational complexity. MambaPro sets new standards in handling long sequences and missing modalities.

News

• We released the MambaPro codebase and paper! 🚀 Paper
• Great news! Our paper has been accepted to AAAI 2025! 🎉

Table of Contents

• Introduction
• Contributions
• Results
• Visualizations
• Reproduction
• Citation

Introduction

Multi-modal object ReID leverages complementary data from diverse modalities (e.g., RGB, NIR, TIR) to overcome challenges like poor lighting and occlusion. MambaPro advances this field by:

1. PFA: Transferring CLIP's pre-trained knowledge to ReID tasks via parallel adapters.
2. SRP: Integrating modality-specific prompts with synergistic transformations.
3. MA: Efficiently modeling intra- and inter-modality interactions with linear complexity.

Contributions

• Introduced MambaPro, the first CLIP-based framework for multi-modal object ReID.
• Developed SRP for synergistic learning across modalities with residual refinements.
• Proposed MA, achieving linear complexity for long-sequence multi-modal interactions.
• Validated effectiveness on RGBNT201, RGBNT100, and MSVR310 benchmarks.

Results

Multi-Modal Object ReID

Multi-Modal Person ReID [RGBNT201]

Multi-Modal Vehicle ReID [RGBNT100 & MSVR310]

Ablation Studies [RGBNT201]

Hyperparameter Analysis [RGBNT201]

Visualizations

Feature Distribution (t-SNE)

GradCam Visualization

Reproduction

Datasets

• RGBNT201: Google Drive
• RGBNT100: Baidu Pan (Code: rjin)
• MSVR310: Google Drive

Pretrained Models

• CLIP: Baidu Pan (Code: 52fu)

Configuration

• RGBNT201: configs/RGBNT201/MambaPro.yml
• RGBNT100: configs/RGBNT100/MambaPro.yml
• MSVR310: configs/MSVR310/MambaPro.yml

Training

#!/bin/bash
# python = 3.10.13
# cuda = 11.8
source activate (your_env)
cd (your_path)
pip install -r requirements.txt
cd selective_scan && pip install .
python train_net.py --config_file configs/RGBNT201/MambaPro.yml

Note

• If you want to use a CLIP-based framework for multi-modal object ReID, DeMo is a better choice, the prompt/adapter tuning configuration in MambaPro is retained for users, besides, we provide detailed visualizations code in DeMo.
• Thanks for your attention and support!

Star History

Citation

If you find MambaPro helpful in your research, please consider citing:

@inproceedings{wang2025MambaPro,
  title={MambaPro: Multi-Modal Object Re-Identification with Mamba Aggregation and Synergistic Prompt},
  author={Wang, Yuhao and Liu, Xuehu and Yan, Tianyu and Liu, Yang and Zheng, Aihua and Zhang, Pingping and Lu, Huchuan},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  year={2025}
}