Awesome

Compatibility for Machine Learning Model Update

This repository contains PyTorch implementation of Forward Compatible Training for Large-Scale Embedding Retrieval Systems (CVPR 2022):

and FastFill: Efficient Compatible Model Update (ICLR 2023):

The code is written to use Python 3.8 or above.

Requirements

We suggest you first create a virtual environment and install dependencies in the virtual environment.

# Go to repo
cd <path/to/ml-fct>
# Create virtual environment ...
python -m venv .venv
# ... and activate it
source .venv/bin/activate
# Upgrade to the latest versions of pip and wheel
pip install -U pip wheel
pip install -r requirements.txt

CIFAR-100 Experiments (quick start)

We provide CIFAR-100 experiments, for fast exploration. The code will run and produce results of both FCT and Fastfill. Here are the sequence of commands for CIFAR-100 experiments (similar to ImageNet but faster cycles):

# Get data: following command put data in data_store/cifar-100-python
python prepare_dataset.py

# Train old embedding model:
# Note: config files assume training with 8 GPUs. Modify them according to your environment.
python train_backbone.py --config configs/cifar100_backbone_old.yaml

# Evaluate the old model (single GPU is OK):
python eval.py --config configs/cifar100_eval_old_old.yaml

# Train New embedding model:
python train_backbone.py --config configs/cifar100_backbone_new.yaml

# Evaluate the new model (single GPU is OK):
python eval.py --config configs/cifar100_eval_new_new.yaml

# Download pre-traianed models if training with side-information:
source get_pretrained_models.sh

# Train FCT transformation:
# If training with side-info model, add its path to the config file below. You
# can use the same side-info model as for ImageNet experiment here. 
python train_transformation.py --config configs/cifar100_fct_transformation.yaml

# Evaluate transformed model vs new model (single GPU is OK):
python eval.py --config configs/cifar100_eval_old_new_fct.yaml

# Train FastFill transformation:
python train_transformation.py --config configs/cifar100_fastfill_transformation.yaml

# Evaluate transformed model vs new model (single GPU is OK):
python eval.py --config configs/cifar100_eval_old_new_fastfill.yaml

CIFAR-100 (FCT, without backfilling):

• These results are not averaged over multiple runs.

CIFAR-100 (FastFill, with backfilling):

• These results are not averaged over multiple runs.
• AUC: Area Under the backfilling Curve. For old/old and new/new we report performance corresponding to no model update and full model update, respectively.

ImageNet-1k Experiments

Here are the sequence of commands for ImageNet experiments:

# Get data: Prepare full ImageNet-1k dataset and provide its path in all config
# files. The path should include training and validation directories. 

# Train old embedding model:
# Note: config files assume training with 8 GPUs. Modify them according to your environment.
python train_backbone.py --config configs/imagenet_backbone_old.yaml

# Evaluate the old model:
python eval.py --config configs/imagenet_eval_old_old.yaml

# Train New embedding model:
python train_backbone.py --config configs/imagenet_backbone_new.yaml

# Evaluate the new model:
python eval.py --config configs/imagenet_eval_new_new.yaml

# Download pre-traianed models if training with side-information:
source get_pretrained_models.sh

# Train FCT transformation:
# (If training with side-info model, add its path to the config file below.)
python train_transformation.py --config configs/imagenet_fct_transformation.yaml

# Evaluate transformed model vs new model:
python eval.py --config configs/imagenet_eval_old_new_fct.yaml

# Train FastFill transformation:
python train_transformation.py --config configs/imagenet_fastfill_transformation.yaml

# Evaluate transformed model vs new model:
python eval.py --config configs/imagenet_eval_old_new_fastfill.yaml

ImageNet-1k (FCT, without backfilling):

ImageNet-1k (FastFill, with backfilling):

• AUC: Area Under the backfilling Curve. For old/old and new/new we report performance corresponding to no model update and full model update, respectively.

Contact

• Hadi Pouransari: mpouransari@apple.com

Citation

@article{ramanujan2022forward,
  title={Forward Compatible Training for Large-Scale Embedding Retrieval Systems},
  author={Ramanujan, Vivek and Vasu, Pavan Kumar Anasosalu and Farhadi, Ali and Tuzel, Oncel and Pouransari, Hadi},
  journal={Proceedings of the IEEE conference on computer vision and pattern recognition},
  year={2022}
}

@inproceedings{jaeckle2023fastfill,
  title={FastFill: Efficient Compatible Model Update},
  author={Jaeckle, Florian and Faghri, Fartash and Farhadi, Ali and Tuzel, Oncel and Pouransari, Hadi},
  booktitle={International Conference on Learning Representations}
  year={2023}
}