Awesome

<img src="demo/src/homepage/spring.gif" width="25%"><img src="demo/src/homepage/summer.gif" width="25%"><img src="demo/src/homepage/autumn.gif" width="25%"><img src="demo/src/homepage/winter.gif" width="25%">

<h3><strong><i>🚀 News</i></strong></h3>

[2021-11-12]: The training code, test code and all the pretrained models are released in this code-base page.

[2021-10-22]: The paper has been accepted by the International Journal of Computer Vision (IJCV)! 🎉

[2021-09-21]: The datasets <a href="#am-2k"><strong>AM-2k</strong></a> and <a href="#bg-20k"><strong>BG-20k</strong></a> can now be <strong>openly accessed</strong> from the links below (both at Google Drive and at Baidu Wangpan) ! Please follow the dataset release agreements to access. Due to some privacy issues, the dataset <Strong>PM-10k</strong> will be published after privacy-preserving from the project Privacy-Preserving Portrait Matting (ACM MM 21). You can refer to this repo for access and updates.

Dataset	<p>Dataset Link<br>(Google Drive)</p>	<p>Dataset Link<br>(Baidu Wangpan 百度网盘)</p>	Dataset Release Agreement
<strong>AM-2k</strong>	Link	Link (pw: 29r1)	Agreement (MIT License)
<strong>BG-20k</strong>	Link	Link (pw: dffp)	Agreement (MIT License)

[2020-11-17]: Create <a href="https://colab.research.google.com/drive/1EaQ5h4u9Q_MmDSFTDmFG0ZOeSsFuRTsJ?usp=sharing"><strong>Google Colab</strong></a> demo to benefit users who want to have a try online.

[2020-11-03]: Publish the <a href="#inference-code-how-to-test-on-your-images">inference code</a> and a pretrained model that can be used to test on your own animal images.

[2020-10-27]: Publish a video demo (YouTube | bilibili | Google drive) contains motivation, network, datasets, and test results on an animal video.

Demo on Google Colab

<a href="https://colab.research.google.com/drive/1EaQ5h4u9Q_MmDSFTDmFG0ZOeSsFuRTsJ?usp=sharing"><img src="https://colab.research.google.com/assets/colab-badge.svg"></a>

Introduction

GFM

The architecture of our proposed end-to-end method <strong>GFM</strong> is illustrated below. We adopt three kinds of <em>Representation of Semantic and Transition Area</em> (<strong>RoSTa</strong>) -TT, -FT, -BT within our method.

We trained GFM with four backbones, -(r) (ResNet-34), -(d) (DenseNet-121), -(r2b) (ResNet-34 with 2 extra blocks), and -(r') (ResNet-101). The trained model for each backbone can be downloaded via the link listed below.

AM-2k

Our proposed <strong>AM-2k</strong> contains 2,000 high-resolution natural animal images from 20 categories along with manually labeled alpha mattes. Some examples are shown as below, more can be viewed in the video demo (YouTube | bilibili | Google drive).

<strong>AM-2k</strong> can be accessed from here (Google Drive | Baidu Wangpan (pw: 29r1)), please make sure that you have read this agreement before accessing the dataset. Please refer to the readme.txt in the dataset folder for more details.

BG-20k

Our proposed <strong>BG-20k</strong> contains 20,000 high-resolution background images excluded salient objects, which can be used to help generate high quality synthetic data. Some examples are shown as below, more can be viewed in the video demo (YouTube | bilibili | Google drive).

<strong>BG-20k</strong> can be accessed from here (Google Drive | Baidu Wangpan (pw: dffp)), please make sure that you have read this agreement before accessing the dataset. Please refer to the readme.txt in the dataset folder for more details.

Results Demo

We test GFM on our AM-2k test dataset and show the results as below. More results on AM-2k test set can be found here.

<img src="demo/src/am2k/sample3.jpg" width="50%"><img src="demo/src/am2k/sample3.png" width="50%">

<img src="demo/src/am2k/sample1.jpg" width="50%"><img src="demo/src/am2k/sample1.png" width="50%">

<img src="demo/src/am2k/sample2.jpg" width="50%"><img src="demo/src/am2k/sample2.png" width="50%">

Inference Code - How to Test on Your Images

Here we provide the procedure of testing on sample images by our pretrained models:

1. Setup environment following this instruction page.

2. Download pretrained models as shown in section GFM, unzip to the folder models/pretrained/

3. Save your high-resolution sample images in folder samples/original/.

4. Setup parameters in scripts/test/test_samples.sh and run it

   chmod +x scripts/*

   ./scripts/test/test_samples.sh

5. The results of alpha matte and transparent color image will be saved in folder samples/result_alpha/. and samples/result_color/.

We show some sample images from the internet, the predicted alpha mattes, and their transparent results as below. We adopt backbone='r34_2b', rosta=TT, hybrid testing strategy and the pretrained model (Google Drive | Baidu Wangpan (pw: 34hf)) for this case.

<img src="samples/original/1.jpg" width="33%"><img src="samples/result_alpha/1.png" width="33%"><img src="samples/result_color/1.png" width="33%"> <img src="samples/original/2.jpg" width="33%"><img src="samples/result_alpha/2.png" width="33%"><img src="samples/result_color/2.png" width="33%"> <img src="samples/original/3.jpg" width="33%"><img src="samples/result_alpha/3.png" width="33%"><img src="samples/result_color/3.png" width="33%"> <img src="samples/original/4.jpg" width="33%"><img src="samples/result_alpha/4.png" width="33%"><img src="samples/result_color/4.png" width="33%"> <img src="samples/original/5.jpg" width="33%"><img src="samples/result_alpha/5.png" width="33%"><img src="samples/result_color/5.png" width="33%">

Statement

If you are interested in our work, please consider citing the following:

@article{li2022bridging,
  title={Bridging composite and real: towards end-to-end deep image matting},
  author={Li, Jizhizi and Zhang, Jing and Maybank, Stephen J and Tao, Dacheng},
  journal={International Journal of Computer Vision},
  volume={130},
  number={2},
  pages={246--266},
  year={2022},
  publisher={Springer}
}

This project is under the MIT license. For further questions, please contact <strong><i>Jizhizi Li</i></strong> at jili8515@uni.sydney.edu.au.

Relevant Projects

<a href="https://github.com/ViTAE-Transformer/ViTAE-Transformer-Matting"><img src="https://shields.io/badge/-A_list_of_our_works_in_matting-9cf?style=for-the-badge"></a>

[1] <strong>Deep Automatic Natural Image Matting, IJCAI, 2021</strong> | Paper | Github <br><em>     Jizhizi Li, Jing Zhang, and Dacheng Tao</em>

[2] <strong>Privacy-Preserving Portrait Matting, ACM MM, 2021</strong> | Paper | Github <br><em>     Jizhizi Li^∗, Sihan Ma^∗, Jing Zhang, and Dacheng Tao</em>

[3] <strong>Referring Image Matting, CVPR, 2023</strong> | Paper | Github <br><em>     Jizhizi Li, Jing Zhang, and Dacheng Tao</em>

[4] <strong>Rethinking Portrait Matting with Privacy Preserving, IJCV, 2023</strong> | Paper | Github <br><em>     Sihan Ma^∗, Jizhizi Li^∗, Jing Zhang, He Zhang, Dacheng Tao</em>

[5] <strong>Deep Image Matting: A Comprehensive Survey, ArXiv, 2023</strong> | Paper | Github <br><em>     Jizhizi Li, Jing Zhang, and Dacheng Tao</em>