Home

Awesome

<p align="center"> <h1 align="center">Magic Tokens: Select Diverse Tokens for Multi-modal Object Re-Identification</h1> <p align="center"> <img src="https://github.com/924973292/EDITOR/assets/89966785/aca4a98d-2fd7-4725-9b76-7d932d2a06b1" alt="Description of the image" width="400" height="395"> <p align="center"> <a href="https://scholar.google.com/citations?user=MfbIbuEAAAAJ&hl=zh-CN" rel="external nofollow noopener" target="_blank"><strong>Pingping Zhang*</strong></a> 路 <a href="https://scholar.google.com/citations?user=WZvjVLkAAAAJ&hl=zh-CN" rel="external nofollow noopener" target="_blank"><strong>Yuhao Wang</strong></a> 路 <a href="https://dblp.org/pid/51/3710-66.html" rel="external nofollow noopener" target="_blank"><strong>Yang Liu</strong></a> 路 <a href="https://dblp.org/pid/138/5016.html" rel="external nofollow noopener" target="_blank"><strong>Zhengzheng Tu</strong></a> 路 <a href="https://scholar.google.com/citations?hl=zh-CN&user=D3nE0agAAAAJ" rel="external nofollow noopener" target="_blank"><strong>Huchuan Lu</strong></a> </p> <p align="center"> <a href="https://arxiv.org/abs/2403.10254" rel="external nofollow noopener" target="_blank">CVPR 2024 Paper</a>

CVPR2024_poster

Previous methods may be easily affected by irrelevant backgrounds and usually ignore the modality gaps. To address above issues, we propose a novel learning framework named EDITOR to sElect DIverse Tokens for multi-modal Object ReID. EDITOR prioritizes the selection of object-centric information, aiming to preserve the diverse features of different modalities while minimizing background interference. Our proposed EDITOR achieves competitive performance on three multi-modal object ReID benchmarks, i.e., RGBNT201, RGBNT100 and MSVR310.

News

Exciting news! Our paper has been accepted by the CVPR 2024! 馃帀

Table of Contents

Introduction

Multi-modal object ReID is crucial in scenarios where objects are captured through different image spectra, such as RGB, near-infrared, and thermal imaging. Previous multi-modal ReID methods typically adhere to the approach of extracting global features from all regions of images in different modalities and subsequently aggregating them. Nevertheless, these methods present two key limitations: (1) Within individual modalities, backgrounds introduce additional noise, especially in challenging visual environments. (2) Across different modalities, backgrounds introduce overhead in reducing modality gaps, which may amplify the difficulty in aggregating features. Hence, our method prioritizes the selection of object-centric information, aiming to preserve the diverse features of different modalities while minimizing background interference.

Contributions

Results

Multi-modal Object ReID

<img src="Results/Overall_Performance.png" alt="RGBNT201" width="1000px">

Performance comparison with different modules [RGBNT201銆丷GBNT100]

<img src="Results/Main_Ablation_RGBNT201.png" alt="Performance comparison with different modules" width="1000px"> <img src="Results/Main_Ablation_RGBNT100.png" alt="Performance comparison with different modules" width="1000px">

Parameter Analysis of EDITOR [RGBNT100]

<img src="Results/Parameter.png" alt="Performance comparison with different modules" width="1000px">

Visualizations

T-SNE

<img src="Results/TSNE.png" alt="T-SNE" width="1000px">

Similarity

<img src="Results/Similarity.png" alt="Grad-CAM" width="1000px">

Selection

<img src="Results/Select_Vis.png" alt="Grad-CAM" width="1000px">

Please check the paper for detailed information

Reproduction

Datasets

RGBNT201 link: https://drive.google.com/drive/folders/1EscBadX-wMAT56_It5lXY-S3-b5nK1wH
Market1501-MM link: https://drive.google.com/drive/folders/1EscBadX-wMAT56_It5lXY-S3-b5nK1wH
RGBNT100 link: https://pan.baidu.com/s/1xqqh7N4Lctm3RcUdskG0Ug code锛歳jin
MSVR310 link: https://drive.google.com/file/d/1IxI-fGiluPO_Ies6YjDHeTEuVYhFdYwD/view?usp=drive_link

Pretrained

ViT-B link: https://pan.baidu.com/s/1YE-24vSo5pv_wHOF-y4sfA code: vmfm

Configs

RGBNT201 file: EDITOR/configs/RGBNT201/EDITOR.yml
Market1501-MM file: EDITOR/configs/Market1501-MM/EDITOR.yml
RGBNT100 file: EDITOR/configs/RGBNT100/EDITOR.yml
MSVR310 file: EDITOR/configs/MSVR310/EDITOR.yml

Tips

Please ensure that your batch size matches the one specified in the configuration file. Typically, using a batch size of 128 and a number of instances of 16 is more robust compared to using a batch size of 64/32 and a number of instances of 8. However, it's important to note that a batch size of 128 consumes over 24GB of GPU memory. In future work, we will pay attention to GPU memory consumption to provide convenient reproducible configurations for more researchers! Thank you for your attention!

Bash

# python = 3.8
# cuda = 11.4
#!/bin/bash
source activate (your env)
cd ../(your path)
pip install -r requirements.txt
python train_net.py --config_file ../RGBNT201/EDITOR.yml

Star History

Star History Chart

Citation

If you find EDITOR useful in your research, please consider citing:

@InProceedings{Zhang_2024_CVPR,
    author    = {Zhang, Pingping and Wang, Yuhao and Liu, Yang and Tu, Zhengzheng and Lu, Huchuan},
    title     = {Magic Tokens: Select Diverse Tokens for Multi-modal Object Re-Identification},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2024},
    pages     = {17117-17126}
}