Home

Awesome

Official code for 'How does topology influence gradient propagation and model performance of deep networks with DenseNet-type skip connections?' (CVPR 2021) ArXiv link

Usage

MLP

NN_Mass vs. Test Accuracy

optional argumentsDescription
-h, --helpshow this help message and exit
--batch_sizeNumber of samples per mini-batch
--epochsNumber of epoch to train
--lrLearning rate
--depththe depth (number of FC layers) of the MLP
--widththe width (number of neurons per layers) of the MLP
--num_segthe number of segmentation for the synthetic dataset (currently we support 'linear' and 'circle' dataset)
--tcmaximum number of candidate channels/neurons from previous layers that can supply concatenation-type skip connections
--datasetthe type of dataset
--make_datasetgenerate/regenerate the synthetic dataset or not
--train_log_filethe name of file used to record the training/test record of MLPs
--res_log_filethe name of file used to record the training/test record of MLPs
--iter_timesthe number of iteration times to train the same architecture

NN_Mass vs. LDI

optional argumentsDescription
-h,show this help message and exit
--batch_sizeNumber of samples per mini-batch
--epochsNumber of epoch to train
--depththe depth (number of FC layers) of the MLP
--widththe width (number of neurons per layers) of the MLP
--num_segthe number of segmentation for the synthetic dataset
--tcmaximum number of candidate channels/neurons from previous layers that can supply concatenation-type skip connections
--dataset DAASETthe type of dataset
--sigma_log_filethe name of file used to record the LDI record of MLPs
--iter_timesthe number of iteration times to calculate the LDI of the same architecture

CNN

CIFAR-10/100

optional argumentsDescription
-h, --helpshow this help message and exit
--archnetwork architecture
--wmwidth multipler of CNN cells
--num_cellsnumber of cells
--cell_depthnumber of layers for each cell
--tc1within the first cell, maximum number of candidate channels/neurons from previous layers that can supply concatenation-type skip connections
--tc2within the second cell, maximum number of candidate channels/neurons from previous layers that can supply concatenation-type skip connections
--tc3within the third cell, maximum number of candidate channels/neurons from previous layers that can supply concatenation-type skip connections
--tc4within the fourth cell, maximum number of candidate channels/neurons from previous layers that can supply concatenation-type skip connections
--datasetdataset

ImageNet

We reuse some code from mobilenetv2.pytorch
Currently, we support the following networks (use python train_imagenet.py -a network_name to select the networks):

mobilenet_v2
resnet18
resnet34
resnet50
resnet101
resnet152
resnext50_32x4d
resnext101_32x8d
wide_resnet50_2
wide_resnet101_2

Example: Train MobileNet-v2

python train_imagenet.py \
    -a mobilenetv2 \
    -d <path-to-ILSVRC2012-data> \
    --epochs 150 \
    --lr-decay cos \
    --lr 0.05 \
    --wd 4e-5 \
    -c <path-to-save-checkpoints> \
    --width-mult <width-multiplier> \
    --input-size <input-resolution> \
    -j <num-workers>

Example: Test MobileNet-v2

python train_imagenet.py \
    -a mobilenetv2 \
    -d <path-to-ILSVRC2012-data> \
    --weight <pretrained-pth-file> \
    --width-mult <width-multiplier> \
    --input-size <input-resolution> \
    -e

Dependency

Please check the environment.sh
Note: the installation of pytorch depends on your OS version and GPU types.



if you find our code useful, please consider citing our paper:

@article{bhardwaj2019does,
  title={How does topology influence gradient propagation and model performance of deep networks with DenseNet-type skip connections?},
  author={Bhardwaj, Kartikeya and Li, Guihong and Marculescu, Radu},
  journal={arXiv preprint arXiv:1910.00780},
  year={2019}
}