Awesome

2021-CVPR-MRL

Peng Hu, Xi Peng, Hongyuan Zhu, Liangli Zhen, Jie Lin, Learning Cross-modal Retrieval with Noisy Labels, IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 19-25, 2021. (PyTorch Code)

Abstract

Recently, cross-modal retrieval is emerging with the help of deep multimodal learning. However, even for unimodal data, collecting large-scale well-annotated data is expensive and time-consuming, and not to mention the additional challenges from multiple modalities. Although crowd-sourcing annotation, e.g., Amazon's Mechanical Turk, can be utilized to mitigate the labeling cost, but leading to the unavoidable noise in labels for the non-expert annotating. To tackle the challenge, this paper presents a general Multimodal Robust Learning framework (MRL) for learning with multimodal noisy labels to mitigate noisy samples and correlate distinct modalities simultaneously. To be specific, we propose a Robust Clustering loss (RC) to make the deep networks focus on clean samples instead of noisy ones. Besides, a simple yet effective multimodal loss function, called Multimodal Contrastive loss (MC), is proposed to maximize the mutual information between different modalities, thus alleviating the interference of noisy samples and cross-modal discrepancy. Extensive experiments are conducted on four widely-used multimodal datasets to demonstrate the effectiveness of the proposed approach by comparing to 14 state-of-the-art methods.

Framework

Usage

To train a model with 0.6 noise rate on Wikipedia, just run main_noisy.py:

python main_noisy.py --max_epochs 30 --log_name noisylabel_mce --loss MCE  --lr 0.0001 --train_batch_size 100 --beta 0.7 --noisy_ratio 0.6 --data_name wiki

You can get outputs as follows:

Epoch: 24 / 30
 [================= 22/22 ==================>..]  Step: 12ms | Tot: 277ms | Loss: 2.365 | LR: 1.28428e-05
 
Validation: Img2Txt: 0.480904	Txt2Img: 0.436563	Avg: 0.458733
Test: Img2Txt: 0.474708	Txt2Img: 0.440001	Avg: 0.457354
Saving..

Epoch: 25 / 30
 [================= 22/22 ==================>..]  Step: 12ms | Tot: 275ms | Loss: 2.362 | LR: 9.54915e-06
 
Validation: Img2Txt: 0.48379	Txt2Img: 0.437549	Avg: 0.460669
Test: Img2Txt: 0.475301	Txt2Img: 0.44056	Avg: 0.45793
Saving..

Epoch: 26 / 30
 [================= 22/22 ==================>..]  Step: 12ms | Tot: 276ms | Loss: 2.361 | LR: 6.69873e-06
 
Validation: Img2Txt: 0.482946	Txt2Img: 0.43729	Avg: 0.460118

Epoch: 27 / 30
 [================= 22/22 ==================>..]  Step: 12ms | Tot: 273ms | Loss: 2.360 | LR: 4.32273e-06
 
Validation: Img2Txt: 0.480506	Txt2Img: 0.437512	Avg: 0.459009

Epoch: 28 / 30
 [================= 22/22 ==================>..]  Step: 12ms | Tot: 269ms | Loss: 2.360 | LR: 2.44717e-06
 
Validation: Img2Txt: 0.481429	Txt2Img: 0.437096	Avg: 0.459263

Epoch: 29 / 30
 [================= 22/22 ==================>..]  Step: 12ms | Tot: 275ms | Loss: 2.359 | LR: 1.09262e-06
 
Validation: Img2Txt: 0.482126	Txt2Img: 0.437257	Avg: 0.459691
Evaluation on Last Epoch:
Img2Txt: 0.475	Txt2Img: 0.440	
Evaluation on Best Validation:
Img2Txt: 0.475	Txt2Img: 0.441

Comparison with the State-of-the-Art

Ablation Study

Citation

If you find MRL useful in your research, please consider citing:

@inproceedings{hu2021MRL,
   title={Learning Cross-modal Retrieval with Noisy Labels},
   author={Peng Hu, Xi Peng, Hongyuan Zhu, Liangli Zhen, Jie Lin},
   booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
   month={June},
   year={2021}
}