Rank Diminishing in Deep Neural Networks

This is the official code for NeurIPS 2022 paper "Rank Diminishing in Deep Neural Networks".

We perform a rigorous study on the behavior of network rank, focusing particularly on the notion of rank deficiency.

Dependency

python=3.8
torch>=1.8.0
torchvision>=0.8.0
timm
tqdm

Usage

1. Partial rank of the Jacobian

python rank_jacobian.py -m resnet50 --imagenet_dir /Path/to/ImageNet/ --weight_dir Path/to/Weights/

2. PCA dimension of feature spaces with perturbations

python rank_perturb.py -m resnet50 --imagenet_dir /Path/to/ImageNet/ --weight_dir Path/to/Weights/

3. Classification dimension of the final feature manifold

python extract_feature.py -m resnet50 --imagenet_dir /Path/to/ImageNet/ --weight_dir Path/to/Weights/

python run_cls_dim.py -m resnet50

4. Independence deficit

python extract_feature.py -m resnet50 --imagenet_dir /Path/to/ImageNet/ --weight_dir Path/to/Weights/

python run_deficit.py -m resnet50

Citation

If you find this code useful, please kindly cite our paper:

@inproceedings{feng2022rank,
    title = {{Rank Diminishing in Deep Neural Networks}},
    author = {Feng, Ruili and Zheng, Kecheng and Huang, Yukun and Zhao, Deli and Jordan, Michael and Zha, Zheng-Jun},
    booktitle = {Advances in Neural Information Processing Systems},
    pages = {33054--33065},
    volume = {35},
    year = {2022}
}