Awesome
Pytorch Implementation of Common Image Generation Metrics
Installation
pip install pytorch-image-generation-metrics
Quick Start
from pytorch_image_generation_metrics import get_inception_score, get_fid
images = ... # [N, 3, H, W] normalized to [0, 1]
IS, IS_std = get_inception_score(images) # Inception Score
FID = get_fid(images, 'path/to/fid_ref.npz') # Frechet Inception Distance
The file path/to/fid_ref.npz
is compatiable with the official FID implementation.
Notes
The FID implementation is inspired by pytorch-fid.
This repository is developed for personal research. If you find this package useful, please feel free to open issues.
Features
- Currently, this package supports the following metrics:
- Inception Score (IS)
- Fréchet Inception Distance (FID)
- The computation procedures for IS and FID are integrated to avoid multiple forward passes.
- Supports reading images on the fly to prevent out-of-memory issues, especially for large-scale images.
- Supports computation on GPU to speed up some CPU operations, such as
np.cov
andscipy.linalg.sqrtm
.
Reproducing Results of Official Implementations on CIFAR-10
Train IS | Test IS | Train(50k) vs Test(10k)<br>FID | |
---|---|---|---|
Official | 11.24±0.20 | 10.98±0.22 | 3.1508 |
ours | 11.26±0.13 | 10.97±0.19 | 3.1525 |
ours use_torch=True | 11.26±0.15 | 10.97±0.20 | 3.1457 |
The results differ slightly from the official implementations due to the framework differences between PyTorch and TensorFlow.
Documentation
Prepare Statistical Reference for FID
- Download the pre-calculated reference, or
- Calculate the statistical reference for your custom dataset using the command-line tool:
See fid_ref.py for details.python -m pytorch_image_generation_metrics.fid_ref \ --path path/to/images \ --output path/to/fid_ref.npz
Inception Features
- When getting IS or FID, the
InceptionV3
model will be loaded intotorch.device('cuda:0')
by default. - Change the
device
argument in theget_*
functions to set the torch device.
Using torch.Tensor
as images
- Prepare images as
torch.float32
tensors with shape[N, 3, H, W]
, normalized to[0,1]
.from pytorch_image_generation_metrics import ( get_inception_score, get_fid, get_inception_score_and_fid ) images = ... # [N, 3, H, W] assert 0 <= images.min() and images.max() <= 1 # Inception Score IS, IS_std = get_inception_score( images) # Frechet Inception Distance FID = get_fid( images, 'path/to/fid_ref.npz') # Inception Score & Frechet Inception Distance (IS, IS_std), FID = get_inception_score_and_fid( images, 'path/to/fid_ref.npz')
Using PyTorch DataLoader to Provide Images
-
Use
pytorch_image_generation_metrics.ImageDataset
to collect images from your storage or use your customtorch.utils.data.Dataset
.from pytorch_image_generation_metrics import ImageDataset from torch.utils.data import DataLoader dataset = ImageDataset(path_to_dir, exts=['png', 'jpg']) loader = DataLoader(dataset, batch_size=50, num_workers=4)
You can wrap a generative model in a dataset to support generating images on the fly.
class GeneratorDataset(Dataset): def __init__(self, G, noise_dim): self.G = G self.noise_dim = noise_dim def __len__(self): return 50000 def __getitem__(self, index): return self.G(torch.randn(1, self.noise_dim)) dataset = GeneratorDataset(G, noise_dim=128) loader = DataLoader(dataset, batch_size=50, num_workers=0)
-
Calculate metrics
from pytorch_image_generation_metrics import ( get_inception_score, get_fid, get_inception_score_and_fid ) # Inception Score IS, IS_std = get_inception_score( loader) # Frechet Inception Distance FID = get_fid( loader, 'path/to/fid_ref.npz') # Inception Score & Frechet Inception Distance (IS, IS_std), FID = get_inception_score_and_fid( loader, 'path/to/fid_ref.npz')
Load Images from a Directory
- Calculate metrics for images in a directory and its subfolders.
from pytorch_image_generation_metrics import ( get_inception_score_from_directory, get_fid_from_directory, get_inception_score_and_fid_from_directory) IS, IS_std = get_inception_score_from_directory( 'path/to/images') FID = get_fid_from_directory( 'path/to/images', 'path/to/fid_ref.npz') (IS, IS_std), FID = get_inception_score_and_fid_from_directory( 'path/to/images', 'path/to/fid_ref.npz')
Accelerating Matrix Computation with PyTorch
-
Set
use_torch=True
when calling functions likeget_inception_score
,get_fid
, etc. -
WARNING: when
use_torch=True
is used, the FID might benan
due to the unstable implementation of matrix sqrt root.
Tested Versions
python 3.9 + torch 1.13.1 + CUDA 11.7
python 3.9 + torch 2.3.0 + CUDA 12.1
License
This implementation is licensed under the Apache License 2.0.
This implementation is derived from pytorch-fid, licensed under the Apache License 2.0.
FID was introduced by Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler and Sepp Hochreiter in "GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium", see https://arxiv.org/abs/1706.08500
The original implementation of FID is by the Institute of Bioinformatics, JKU Linz, licensed under the Apache License 2.0. See https://github.com/bioinf-jku/TTUR.