Home

Awesome

PA&DA: Jointly Sampling PAth and DAta for Consistent NAS (CVPR 2023)

license python pytorch

This repository contains the code for our paper "PA&DA: Jointly Sampling PAth and DAta for Consistent NAS", where we propose to explicitly minimize the gradient variance of the supernet training by jointly optimizing the sampling distributions of PAth and DAta (PA&DA).

pa-da_framework

Prerequisites and Dependencies

To run our code, please see the prerequisites below:

  1. Download the datasets of NAS-Bench-201 and NAS-Bench-1Shot1, and pre-trained checkpoints from the [Google Drive].
  2. Install the necessary packages below. We adopt Python 3.7 in our experiments.
    # CUDA 10.2
    conda install pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 cudatoolkit=10.2 -c pytorch
    pip install numpy==1.21.2 opencv-python==4.6.0.66 pandas==1.3.5 Pillow==9.0.1 pynvml==11.4.1 PyYAML==6.0 schema==0.7.5 scikit-image==0.19.3 scikit-learn==1.0.2 scipy==1.7.3 seaborn==0.12.2 six==1.16.0 tensorboard==1.15.0 thop==0.0.31 timm==0.6.11

NAS-Bench-201 Experiments

We first construct the supernet of NAS-Bench-201 based on the Awesome AutoDL. According to previous papers, we then provide our implementations for baseline methods such as SPOS, FairNAS, and SUMNAS. The implementation of PA&DA are provided subsequently and adopts the same training configuration as baseline methods. If you have any questions or concerns, please feel free to raise an issue and discuss with us.

    cd nasbench201
    mkdir dataset
    ln -s /Path/To/Your/$Dataset ./dataset/cifar10
    mkdir dataset/nasbench201
    mv /Path/To/nasbench201_dict.npy ./dataset/nasbench201
    python check_folders.py
    python train_baselines_201.py --method ${Baseline_Method}

where ${Baseline_Method} can be either spos, fairnas, or sumnas. Note that this command will use the selected method to train the supernet and rank all sub-models after training. Results will be recorded to the results folder. We also provide the script to run experiments with different seeds in parallel:

    # set the Method and GPU IDX in the script
    bash script/train_baselines_201.sh
    python train_panda_201.py

Note that this command will use PA&DA to train the supernet and rank all sub-models after training. Results will be recorded to the results folder. We also provide the script to run experiments with different seeds in parallel:

    # set GPU IDX in the script
    bash script/train_panda_201.sh

Acknowledgments

During our implementations, we referred the following code and we sincerely appreciate their valuable contributions:

Citation

If you find this work helpful in your research, please consider citing our paper:

@inproceedings{lu2023pa-da,
  title     = {PA&DA: Jointly Sampling PAth and DAta for Consistent NAS},
  author    = {Lu, Shun and Hu, Yu and Yang, Longxing and Sun, Zihao and Mei, Jilin and Tan, Jianchao and Song, Chengru},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year      = {2023}
}