Awesome

:ring: Efficient tensor decomposition-based filter pruning

:star2: News

Project is under development :construction_worker:. Please stay tuned for more :fire: updates.

• 2024.05.15: Paper accepted by Neural Networks :innocent:. Preprint available here.
• 2024.01.24: Update ablation studies on metrics metrics 📊 and Kshots 🔁.
• 2023.11.02: Add instance segmentation and keypoint detection visualization :horse_racing:.
• 2023.10.02: Efficacy :fast_forward: study is added.
• 2023.8.22: Throughput acceleration :stars: experiment is released :tada:.
• 2023.8.14: Poster :bar_chart: is released. Part of the project will be :mega: presented at GRETSI'23 :clap:.
• 2023.8.12: Baseline and compressed checkpoints :gift: are released.

:dart: Main results

:wrench: Installation

pip install torch tensorly numpy thop ptflops

:unlock: Verification of our results

Please download the checkpoints and evaluate their performance with the corresponding script and dataset.

• All results are available here.

  <details>

  Notes: Log files of all experiments are attached, they contain all information about the pruning or fine-tuning process, as well as model architecture, numbers of parameters/FLOPs, and top-1/top-5 accuracy.

  </details>

• Download the datasets

  <details>

  The CIFAR dataset will be automatically downloaded.

  The Imagenet dataset can be downloaded here and processed as this script.

  </details>

• Use main/test.py to validate the performance of the checkpoints.

  <details>

    python main/test.py --dataset cifar10 --data_dir data/cifar10 --arch vgg_16_bn --compress_rate [0.21]*7+[0.75]*5 --model_path ./pruned_model/cifar10/vgg16bn/soft/model_best.pth.tar
    python main/test.py --dataset cifar10 --data_dir data/cifar10 --arch resnet_56 --compress_rate [0.]+[0.18]*29 --model_path ./pruned_model/cifar10/resnet56/soft/model_best.pth.tar
    python main/test.py --dataset cifar10 --data_dir data/cifar10 --arch densenet_40 --compress_rate [0.]+[0.08]*6+[0.09]*6+[0.08]*26 --model_path ./pruned_model/cifar10/densenet/soft/model_best.pth.tar
    python main/test.py --dataset cifar10 --data_dir data/cifar10 --arch mobilenet_v2 --compress_rate [0.]+[0.1]+[0.25]*2+[0.25]*2+[0.3]*2 --model_path ./pruned_model/cifar10/mobilenetv2/moderate/model_best.pth.tar
    python main/test.py --dataset imagenet --data_dir data/imagenet --arch resnet_50 --compress_rate [0.]+[0.5]*3+[0.6]*16 --model_path ./pruned_model/imagenet/extreme/model_best.pth.tar
  

  </details>

:octocat: Reproducibility and further development

1. To reproduce results, you may run prepared scripts.

• For CIFAR-10, the rank will be calculated during the pruning process.
• For Resnet50/ImageNet, first, generate the rank by this:

  sh main/scripts/generate_rank_resnet50.sh
  

• Now, the pruning process can be performed via prepared scripts. For example:

  sh main/scripts/resnet50_imagenet/vbd.sh
  

2. Our code is pipelined and can be integrated into other works. Just replace the filter ranking computation.

   # replace your rank calculation here
   rank = get_rank(oriweight, args.criterion, args.strategy)
   

:art: Supplementary materials

1. Poster

2. Architecture constraint

With shortcut connection architecture, the input and output of each residual block are forced identical. In each layer (i.e, same color), filters with the same style (e.g, sketch) are highly similar, and the empty dashed ones are to be pruned. After pruning, the input and output layer (green and red) has the same number of filters.

3. Computational requirement comparison

Time consumption to calculate the similarity matrix on VGG-16-BN. For tail layers that contain a larger number of filters, the tensor decomposition method is obviously more efficient.

4. Criteria comparison <a name="metrics"></a>

The influence of distance metrics on model accuracy for different architectures and datasets.

5. Throughput acceleration <a name="throughput-acceleration"></a>

• FasterRCNN for object detection

• MaskRCNN for instance segmentation

• KeypointRCNN for human keypoint detection

To emphasize the pragmatic benefits of CORING, an experiment was meticulously conducted comparing a baseline model and a compressed model, both tailored for object detection tasks. Specifically employing the FasterRCNN_ResNet50_FPN architecture on a Tesla T4 GPU, the experiment underscores the remarkable performance enhancement achieved by CORING. The accompanying GIFs provide a clear visual representation: the baseline model demonstrates an inference speed of approximately 7 FPS, while the CORING-compressed model exhibits a notable twofold acceleration in throughput. This compelling contrast aptly demonstrates CORING's efficacy and scalability, firmly establishing its aptness for diverse deployment scenarios.

Note: For replication of this experiment, please refer to detection/README.md.

6. Comparative efficacy study <a name="kmeans"></a>

For a detailed evaluation of CORING's performance, including experiments, results, and visualizations, please refer to the efficacy study.

7. $K$-shots analysis <a name="KShots"></a>

The influence of $K$, the number of shots, on model accuracy for different architectures and datasets.

:bookmark_tabs: Not-To-Do

• Integrate other pruning techniques.
• Clean code.

:email: Contact

We hope that the new perspective of CORING and its template may inspire more developments :rocket: on network compression.

We warmly welcome your participation in our project!

To contact us, never hesitate to contact pvti@gmail.com. <br></br>

:bookmark: Citation

If the code and paper help your research, please kindly cite:

@article{pham2024efficient,
    title={Efficient tensor decomposition-based filter pruning},
    journal={Neural Networks},
    author={Pham, Van Tien and Zniyed, Yassine and Nguyen, Thanh Phuong},
    year={2024},
  }

:thumbsup: Acknowledgement

This work was granted access to the <a href="http://www.idris.fr/eng/jean-zay/jean-zay-presentation-eng.html">HPC resources of IDRIS</a> under the allocation 2023-103147 made by <a href="https://genci.fr/">GENCI</a>.
The work of T.P. Nguyen is partially supported by <a href="https://anr.fr/Projet-ANR-21-ASRO-0003">ANR ASTRID ROV-Chasseur</a>.
Part of this repository is based on HRankPlus.