Awesome

Video Inpainting Evaluation

This project evaluates metrics used to evaluate video inpainting models. It includes metrics that measure reconstruction performance (PSNR, SSIM, LPIPS, PVCS), realism (FID, video FID), and temporal consistency (warping error, patch consistency).

Note: This project is in maintenance mode due to the author graduating. Errors will be addressed to the greatest extent possible, but new features and upgrades will not.

Installation and Setup

1. Create an environment and install project dependencies with conda:

   conda env create -p ./env -f environment.yml
   

   Always make sure this environment is activated when running the commands in this document, and that commands are run from this project's root directory unless otherwise specified.

   conda activate ./env
   

2. Run install-flownet2.sh to compile FlowNet2's custom layers:

   ./scripts/setup/install-flownet2.sh
   

3. Download model weights (e.g., for flow computation, (V)FID feature extraction, etc.):

   ./scripts/setup/download-models.sh
   

Data Preparation

This project expects a two-tier file structure for "ground-truth" frames (i.e., normal video frames), masks, and inpainted frames. Ground-truth and mask frames should go in the same folder and follow the structure shown below:

frames/
├── video1/
│   ├── frame_0000_gt.png
│   ├── frame_0000_mask.png
│   ├── frame_0001_gt.png
│   ├── frame_0001_mask.png
│   ├── frame_0002_gt.png
│   ├── frame_0002_mask.png
│   └── ...
├── video2/
│   ├── frame_0000_gt.png
│   ├── frame_0000_mask.png
│   ├── frame_0001_gt.png
│   ├── frame_0001_mask.png
│   ├── frame_0002_gt.png
│   ├── frame_0002_mask.png
│   └── ...
...

Inpainted results should go in a different folder and follow the structure shown below:

inpainting-results/
├── video1/
│   ├── frame_0000_pred.png
│   ├── frame_0001_pred.png
│   ├── frame_0002_pred.png
│   └── ...
├── video2/
│   ├── frame_0000_pred.png
│   ├── frame_0001_pred.png
│   ├── frame_0002_pred.png
│   └── ...
...

Computing Evaluation Features

Various features (e.g., flow, (V)FID features, etc.) must be pre-computed before running the evaluation script. To produce these features, run compute-evaluation-features.sh. The example below consumes a dataset stored in the frames directory and saves the features in the eval-data directory:

./scripts/preprocessing/compute-evaluation-features.sh frames eval-data

Evaluation

Run evaluate_inpainting.py:

python -m src.main.evaluate_inpainting \
    --gt_root=frames \
    --pred_root=inpainting-results \
    --eval_feats_root=eval-data \
    --output_path=quantitative-results.npy

Specific metrics can be included or excluded with the --include and --exclude flags respectively (supported keys are listed in the Metrics section). For more options, call python -m src.main.evaluate_inpainting -h.

Metrics

The table below lists the supported metrics with brief descriptions.

References

1. Zhang et al. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. CVPR 2018.
2. Carreira and Zisserman. Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. CVPR 2017.
3. Heusel et al. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. NeurIPS 2017.
4. Gupta et al. Characterizing and Improving Stability in Neural Style Transfer. ICCV 2017.
5. Lai et al. Learning Blind Video Temporal Consistency. ECCV 2018.

Licenses

This software uses code from other projects under various licenses:

Exact licenses for the above libraries are available in their respective paths.

Additionally, code attributed to https://github.com/phoenix104104/fast_blind_video_consistency is available under the following MIT license:

MIT License

Copyright (c) 2018 UC Merced Vision and Learning Lab
Modifications copyright (c) 2021 Ryan Szeto

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

All other code, unless otherwise noted, is available under the MIT license in LICENSE.