Awesome

Dynamic-Graphs-Construction

Official Codes for Understanding the Dynamics of DNNs Using Graph Modularity (ECCV2022).

Abstract: There are good arguments to support the claim that deep neural networks (DNNs) capture better feature representations than the previous hand-crafted feature engineering, which leads to a significant performance improvement. In this paper, we move a tiny step towards understanding the dynamics of feature representations over layers. Specifically, we model the process of class separation of intermediate representations in pre-trained DNNs as the evolution of communities in dynamic graphs. Then, we introduce modularity, a generic metric in graph theory, to quantify the evolution of communities. In the preliminary experiment, we find that modularity roughly tends to increase as the layer goes deeper and the degradation and plateau arise when the model complexity is great relative to the dataset. Through an asymptotic analysis, we prove that modularity can be broadly used for different applications. For example, modularity provides new insights to quantify the difference between feature representations. More crucially, we demonstrate that the degradation and plateau in modularity curves represent redundant layers in DNNs and can be pruned with minimal impact on performance, which provides theoretical guidance for layer pruning.

This figure is the pipeline for the dynamic graphs construction and the application scenarios of the modularity metric.

This figure indicates that modularity provides a quantifiable interpretation perspective for understanding the dynamics of DNNs.

The modularity curves of various backbones:

Usage

To calculate the modularity:

python get_embedding_cifar.py --gpu 0 --arch cResNet18 --set cifar10 --num_classes 10 --batch_size 500 --pretrained

Cite Our Paper

If you use this code in your research, please cite our paper.

@inproceedings{lu2022understanding,
  title={Understanding the Dynamics of DNNs Using Graph Modularity},
  author={Lu, Yao and Yang, Wen and Zhang, Yunzhe and Chen, Zuohui and Chen, Jinyin and Xuan, Qi and Wang, Zhen and Yang, Xiaoniu},
  booktitle={European Conference on Computer Vision},
  pages={225--242},
  year={2022},
  organization={Springer}
}