Home

Awesome

UDA-Animal-Pose

About

This is the source code for our paper

Chen Li, Gim Hee Lee. From Synthetic to Real: Unsupervised Domain Adaptation for Animal Pose Estimation. In CVPR2021.

The pseudo labels generated from model trained on synthetic data are noisy, which leads to degraded performance when directly used for training. In this work, we propose an online coarse-to-fine pseudo label update strategy to gradually replace noisy pseudo labels with more accurate ones.

<p align="center"> <img width="400" height="250" src="illustration.png"> </p>

Our network is a student-teacher architecture, where the student network consists of a multi-scale domain adaptation module(MDAM), a refinement block (RB) and a self-feedback loop. We conduct online coarse-to-fine pseudo label update through the inner loopand the outer loop, respectively.

<p align="center"> <img width="684" height="245" src="network.png"> </p>

For more details, please refer to our paper.

Bibtex

@inproceedings{li2021synthetic,
  title={From synthetic to real: Unsupervised domain adaptation for animal pose estimation},
  author={Li, Chen and Lee, Gim Hee},
  booktitle={Proceedings of the IEEE/CVF conference on computer vision and pattern recognition},
  pages={1482--1491},
  year={2021}
}

Dependencies

  1. Python 3.7.10
  2. Pytorch 1.2.0

Please refer to requirements.txt for more details on dependencies.

Download datasets

Data Preprocess

Train

If you do not want to train from scratch, you can download our pretrained model trained on the synthetic data (stage1), as well as the pseudo labels generated by this model. Move the folder psudo_labels to ./animal_data/ and move the pretrained model to ./checkpoint/all/. The pretrained model achieve a PCK@0.05 of 64.71 on the real animal dataset. Train our model by running:

CUDA_VISIBLE_DEVICES=0 python train/train_refinenet_mt.py --arch pose_resnet_refine_mt_multida --checkpoint ./checkpoint       /all/syn2real_refinenet_mt --dual_branch --mixup_dual --dataset_real_crop real_animal_crop_all --occlusion_aug --pretrained ./checkpoint/all/models/stage1/model_best.pth.tar --animal all --dataset_real real_animal_all --dataset synthetic_animal_sp_all --train_on_all_cat

You can also train from scratch. Firstly, You will train the model on the synthetic dataset by running:

CUDA_VISIBLE_DEVICES=0 python train/train_refinenet.py --arch pose_resnet_refine --checkpoint ./checkpoint/all/syn_refinenet --animal all --solver adam --dataset synthetic_animal_sp_all --dataset_real real_animal_all --train_on_all_cat

Then you can generate pseudo labels with this model and train with both synthetic and real data by running:

CUDA_VISIBLE_DEVICES=0 python train/train_refinenet_mt.py --arch pose_resnet_refine_mt_multida --checkpoint ./checkpoint/all/syn2real_refinenet_mt --dual_branch --mixup_dual --dataset_real_crop real_animal_crop_all --occlusion_aug --pretrained ./checkpoint/all/syn_refinenet/model_best.pth.tar --animal all --dataset_real real_animal_all --dataset synthetic_animal_sp_all --train_on_all_cat --generate_pseudol

Test

Download our model and test on the TigDog by running:

CUDA_VISIBLE_DEVICES=0 python evaluation/test.py --arch pose_resnet_refine_mt_multida --animal all --dataset real_animal_all --resume checkpoint/all/models/stage2/model_best.pth.tar --evaluate --train_on_all_cat --flip

This will compute the average PCK score for both horse and tiger, and you can set '--animal' to horse or tiger to get the PCK score for each animal category.

Acknowledgements

Thanks @JitengMu for his quick response when I have questions about his interesting paper. If you use the synthetic dataset, please cite

@InProceedings{Mu_2020_CVPR,
author = {Mu, Jiteng and Qiu, Weichao and Hager, Gregory D. and Yuille, Alan L.},
title = {Learning From Synthetic Animals},
booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2020}
}