Awesome

NAS-Bench-NLP

Preparation:

1. unzip data/datasets.zip, train_logs_single_run/logs.zip, train_logs_multi_runs/logs.zip, and train_logs_wikitext-2/logs.zip;
2. install requirements.txt (currently contains unuseed packages; to be cleaned);
3. optionally, copy models from the dropbox (sample: https://www.dropbox.com/sh/qviytkrlbu2cy5u/AABy59Bb9CpiS7D4osbvY_xva?dl=0, all models: https://www.dropbox.com/scl/fo/4r36x7wqb6gvzcmz8zo61/AIzcRCPZhmzORxJdSI2AdtY?rlkey=516wk0knseuuow45wn4mhy0ak&e=1&dl=0) to the folder models_weights.

Usage:

• search_space_examples.ipynb demonstrates how to generate architectures from the search space;
• to train a model, run script main_one_model_train.py --recepie_id=<index of the architecture from the list>, where the list of architectures is by defaultin data/recepies_example.json; logs and final weights will be stored in tmp folder by default (see script argumens for more info);
• reproduce_model.ipynb demonstrates how to load and apply the trained model;
• make_arch_embedding.ipynb creates graph2vec features for architectures;
• search_space_analysis.ipynb reproduces figures from the analysis section in the paper;
• benchmarking_examples.ipynb shows how NAS methods can be tested based on precomputed results in the logs.