Awesome

PLIP

PLIP is a novel Language-Image Pre-training framework for generic Person representation learning which benefits a range of downstream person-centric tasks.

Also, we present a large-scale person dataset named SYNTH-PEDES to verify its effectiveness, where the Stylish Pedestrian Attributes-union Captioning method (SPAC) is proposed to synthesize diverse textual descriptions.

Experiments show that our model not only significantly improves existing methods on downstream tasks, but also shows great ability in the few-shot and domain generalization settings. More details can be found at our paper PLIP: Language-Image Pre-training for Person Representation Learning.

News

• 🔥[06.1] The SYNTH-PEDES is released. Welcome to download and use!
• 🔥[06.1] The code for CMPM/C fine-tuning is released! It leads to SOTA performance without bells and whistles!
• 🔥[05.31] The pre-trained model and zero-shot inference code are released !

SYNTH-PEDES

SYNTH-PEDES is by far the largest person dataset with textual descriptions without any human annotation effort. Every person image has 2 or 3 different texutal descriptions and 6 attribute annotations. The dataset is released at Baidu Yun.

Note that SYNTH-PEDES can only be used for research, any commercial usage is forbidden.

This is the comparison of SYNTH-PEDES with other popular datasets.

These are some examples of our SYNTH-PEDES dataset.

Annotation format:

{
    "id": 7,
    "file_path": "Part1/7/1.jpg",
    "attributes": [
        "man,black hair,black shirt,pink shorts,black shoes,unknown"
    ],
    "captions": [
        "A man in his mid-twenties with short black hair is wearing a black t-shirt over light pink trousers. He is also wearing black shoes.",
        "The man with short black hair is wearing a black shirt and salmon pink shorts. He is also wearing black shoes."
    ],
    "prompt_caption": [
        "A man with black hair is wearing a black shirt with pink shorts and a pair of black shoes."
    ]
}

Models

We utilize ResNet50 and Bert as our encoders. After pre-training, we fine-tune and evaluate the performance on three downstream tasks. The checkpoints have been released at Baidu Yun and Google Drive.

CUHK-PEDES dataset (Text Re-ID R@1/R@10)

ICFG-PEDES dataset (Text Re-ID R@1/R@10)

Market1501 & DukeMTMC (Image Re-ID mAP/cmc1)

Evaluate on PETA & PA-100K & RAP (PAR mA/F1)

Usage

Install Requirements

we use 4 RTX3090 24G GPU for training and evaluation.

Create conda environment.

conda create --name PLIP --file requirements.txt
conda activate PLIP

Datasets Prepare

Download the CUHK-PEDES dataset from here and ICFG-PEDES dataset from here.

Organize them in data folder as follows:

|-- data/
|   |-- <CUHK-PEDES>/
|       |-- imgs
|            |-- cam_a
|            |-- cam_b
|            |-- ...
|       |-- reid_raw.json
|
|   |-- <ICFG-PEDES>/
|       |-- imgs
|            |-- test
|            |-- train 
|       |-- ICFG_PEDES.json
|
|   |-- <SYNTH-PEDES>/
|       |-- Part1
|       |-- ...
|       |-- Part11
|       |-- synthpedes_dataset.json

Zero-shot Inference

Our pre-trained model can directly be transfered to downstream tasks, especially text-based Re-ID.

1. Run the python file and generate train/test/valid json files respectively.

python dataset_split.py

2. Then you can evaluate by running:

python zs_inference.py

Fine-tuning Inference

Almost all existing downstream person-centric methods can be improved through replacing the backbone with our pre-trained model. Taking CMPM/C as example:

1. Go to the CMPM/C root:

cd Downstreams/CMPM-C

2. Run the following to train and test. Note that you can modify the code yourself for single GPU training:

python dataset_split.py 
CUDA_VISIBLE_DEVICES=0,1,2,3 python -m torch.distributed.launch --nproc_per_node=4 train.py
python test.py

Evaluate on Other Methods and Tasks.

By simply replacing the visual backbone with our pre-trained model, almost all existing methods on downstream tasks make significant improvements. For example, you can try by the following repositories:

Text-based Re-ID: SSAN, LGUR

Image-based Re-ID: BOT, MGN, ABD-Net

Person Attribute Recognition: Rethink, Label2label, VTB

Reference

If you use PLIP in your research, please cite it by the following BibTeX entry:

@inproceedings{
zuo2024plip,
title={PLIP: Language-Image Pre-training for Person Representation Learning},
author={Jialong Zuo and Jiahao Hong and Feng Zhang and Changqian Yu and Hanyu Zhou and Changxin Gao and Nong Sang and Jingdong Wang},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024}
}