Awesome

About

TranSVAE is a disentanglement framework designed for unsupervised video domain adaptation. It aims at disentangling the domain information from the data during the adaptation process. We consider the generation of cross-domain videos from two sets of latent factors: one encoding the static domain-related information and another encoding the temporal and semantic-related information. Objectives are enforced to constrain these latent factors to achieve domain disentanglement and transfer.

Visit our project page to explore more details. :paw_prints:

Updates

• [2023.10] - We provide our extracted I3D features, kindly refer to this page for more details.
• [2023.09] - TranSVAE was accepted to NeurIPS 2023! :tada:
• [2022.08] - TranSVAE achieves 1st place among the UDA leaderboards of UCF-HMDB, Jester, and Epic-Kitchens, based on Paper-with-Code.
• [2022.08] - Try a Gradio demo for domain disentanglement in TranSVAE at Hugging Face Spaces! :hugs:
• [2022.08] - Our paper is available on arXiv, click here to check it out!

Outline

• Highlights
• Installation
• Data Preparation
• Getting Started
• Main Results
• TODO List
• License
• Acknowledgement
• Citation

Highlights

Installation

Please refer to INSTALL.md for the installation details.

Data Preparation

Please refer to DATA_PREPARE.md for the details to prepare the ¹UCF₁₀₁, ²HMDB₅₁, ³Jester, ⁴Epic-Kitchens, and ⁵Sprites datasets.

Getting Started

Please refer to GET_STARTED.md to learn more usage about this codebase.

Main Results

UCF₁₀₁ - HMDB₅₁

Jester

Epic-Kitchens

Ablation Study

<strong>UCF₁₀₁</strong> → <strong>HMDB₅₁</strong> <br> <img src="figs/ablation-ucf2hmdb.png">

<strong>HMDB₅₁</strong> → <strong>UCF₁₀₁</strong> <br> <img src="figs/ablation-hmdb2ucf.png">

<strong>Domain Transfer Example</strong> <br>

Source (Original)	Target (Original)		Source (Original)	Target (Original)
Reconstruct ($\mathbf{z}_d^{\mathcal{S}}$ + $\mathbf{z}_t^{\mathcal{S}}$)	Reconstruct ($\mathbf{z}_d^{\mathcal{T}}$ + $\mathbf{z}_t^{\mathcal{T}}$)		Reconstruct ($\mathbf{z}_d^{\mathcal{S}}$ + $\mathbf{z}_t^{\mathcal{S}}$)	Reconstruct ($\mathbf{z}_d^{\mathcal{T}}$ + $\mathbf{z}_t^{\mathcal{T}}$)
Reconstruct ($\mathbf{z}_d^{\mathcal{S}} + \mathbf{0}$)	Reconstruct ($\mathbf{z}_d^{\mathcal{T}} + \mathbf{0}$)		Reconstruct ($\mathbf{z}_d^{\mathcal{S}} + \mathbf{0}$)	Reconstruct ($\mathbf{z}_d^{\mathcal{T}} + \mathbf{0}$)
Reconstruct ($\mathbf{0} + \mathbf{z}_t^{\mathcal{S}}$)	Reconstruct ($\mathbf{0} + \mathbf{z}_t^{\mathcal{T}}$)		Reconstruct ($\mathbf{0} + \mathbf{z}_t^{\mathcal{S}}$)	Reconstruct ($\mathbf{0} + \mathbf{z}_t^{\mathcal{T}}$)
Reconstruct ($\mathbf{z}_d^{\mathcal{S}} + \mathbf{z}_t^{\mathcal{T}}$)	Reconstruct ($\mathbf{z}_d^{\mathcal{T}} + \mathbf{z}_t^{\mathcal{S}}$)		Reconstruct ($\mathbf{z}_d^{\mathcal{S}} + \mathbf{z}_t^{\mathcal{T}}$)	Reconstruct ($\mathbf{z}_d^{\mathcal{T}} + \mathbf{z}_t^{\mathcal{S}}$)

TODO List

• Initial release. 🚀
• Add license. See here for more details.
• Add demo at Hugging Face Spaces.
• Add installation details.
• Add data preparation details.
• Add evaluation details.
• Add training details.

License

<a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/80x15.png" /></a> <br /> This work is under the <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/">Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License</a>.

Acknowledgement

We acknowledge the use of the following public resources during the course of this work: ¹UCF₁₀₁, ²HMDB₅₁, ³Jester, ⁴Epic-Kitchens, ⁵Sprites, ⁶I3D, and ⁷TRN.

Citation

If you find this work helpful, please kindly consider citing our paper:

@inproceedings{wei2023transvae,
  title = {Unsupervised Video Domain Adaptation for Action Recognition: A Disentanglement Perspective},
  author = {Wei, Pengfei and Kong, Lingdong and Qu, Xinghua and Ren, Yi and Xu, Zhiqiang and Jiang, Jing and Yin, Xiang},
  booktitle = {Advances in Neural Information Processing Systems}, 
  year = {2023},
}