Awesome

Prune and Replace NAS

by Kevin Alexander Laube and Andreas Zell, on arxiv

This code is based on the implementations of DARTS and P-DARTS.

Idea:

Iteratively pruning and replacing bad candidates from the operation pool enables us to efficiently search through vast operation spaces in reasonable time. We use DARTS to rank the currently available candidates, prune the worst ones, and generate new ones by applying network morphisms to those that are left.

Run:

Sample scripts that match our experiments in the paper are provided in the ./scripts folder.

Search

The progress and discovered cells of our DL2 search, these stats are plotted to tensorboard.

Retraining

Test error (%) on CIFAR-10 and CIFAR-100 in comparison. The focus of this research is the candidate operation space, thus the only major change compared to the DARTS baseline is the progressive pruning and replacement schedule.

Method	#params	#ops	GPU days	CIFAR-10	CIFAR-100
NASNet-A	3.3M	13	1800	2.65
AmoebaNet-B	2.8M	19	3150	2.55
ENAS	4.6M	5	0.5	2.89
DARTS (1st order)	2.9M	8	1.5	2.94
DARTS (2nd order)	3.4M	8	4	2.83
P-DARTS C10	3.4M	8	0.3	2.5	16.55
P-DARTS C100	3.6M	8	0.3	2.62	15.92
sharpDARTS	3.6M		0.8	2.45
SNAS moderate	2.8M	8	1.5	2.85
NASP	3.3M	7	0.2	2.8
NASP (more ops)	7.4M	12	0.3	2.5
PR-DARTS DL1	3.2M	15/15	0.82	2.74	17.37
PR-DARTS DL2	4.0M	15/15	0.82	2.51	15.53
PR-DARTS DR	4.2M	26/39	0.88	2.55	16.69
PR-DARTS UR	5.4M	45/83	1.10	3.79

Citation

If you use any part of this code in your research, please cite our paper:

@article{laube2019prnas,
  title={Prune and Replace NAS},
  author={Kevin A. Laube and Andreas Zell},
  journal={arXiv preprint https://arxiv.org/abs/1906.07528},
  year={2019}
}