Home

Awesome

Firth Bias Reduction in Few-shot Distribution Calibration

This sub-directory contains the experiments in which Firth bias reduction from our paper On the Importance of Firth Bias Reduction in Few-Shot Classification is applied to the recent State-of-the-Art Distribution Calibration (DC) method in few-shot classification. This is one of the three code repositories of our paper, and is a sub-module of the the main "Firth Bias Reduction in Few-Shot Learning" repository.

For a concise and informal description of our work, check out our paper's website: https://ehsansaleh.github.io/firthfsl

<details open> <summary><h2>Check Out Our GPU Implementation of the DC Method</h2></summary> </details> <details open> <summary><h2>Final Evaluations and Tables</h2></summary>

Following our step-by-step guide, you can produce the following tables.

</details> </details> <details> <summary><h2>Quick Q&A Rounds</h2></summary>
  1. Question: Give me a quick-starter code to start reproducing the paper trainings on a GPU? 
    git clone https://github.com/ehsansaleh/firth_bias_reduction.git
cd ./firth_bias_reduction/code_dcf
./features/download.sh
./main.sh

  1. Question: Give me a simple python command to run?

    python main.py --device "cuda:0" --configid "1_mini2CUB/5s10w_0aug"
    <details> <summary><strong>More Information</strong></summary> </details>

  1. Question: How can I reproduce the paper tables?

    make summary
make figures
make tables
    <details> <summary><strong>More Information</strong></summary>
    1. If you have run new classifier trainings by either ./main.sh or python main.py, then run make summary. Otherwise, skip this step. This command will collect the csv files from the ./results directory, and process them into a single summarized file at ./summary/test2test.csv.
    2. Run make tables in case you're interested about the raw numbers at the ./tables directory.
    </details>

  1. Question: I have my own code and I do not want to use your code. How can I apply the Firth bias reduction to my own loss?

    ce_loss = nn.CrossEntropyLoss()
ce_term = ce_loss(logits, target)

log_probs = logits - torch.logsumexp(logits, dim=-1, keepdim=True)
firth_term = -log_probs.mean()

loss = ce_term + lam * firth_term
loss.backward()

    • Alternatively, you can use the label_smoothing keyword argument in nn.CrossEntropyLoss.

    • Remember that this Firth formulation is only true for 1-layer logistic and cosine classifiers. For more complex networks, the FIM's log-determinant must be worked out.

<details> <summary>4. <strong>Question:</strong> You seem to have too many directories, files, and a fancy structure. Explain the whole thing as simple as possible?</summary> 
                 ./main.sh or
                python main.py                  make summary                  make tables   
configs/*.json ================> results/*.csv =============> summary/*.csv ===============> tables/*
                      /\                                                            
                      ||
         (below is    ||
          optional    ||
           parts)     ||
                      ||                        python save_features.py                   
                      ======= features/*.pkl <========================== checkpoints/*.tar
                                    /\
                                    ||
                                    ||
                                 Datasets/*

The top horizontal line is the important one for our work.

</details>
<details> <summary>5. <strong>Question:</strong> What are the python environment package requirements?</summary> </details> </details> <details open> <summary><h2>Step-by-Step Guide to the Code</h2></summary> </details> <details> <summary><h2>Configurations and Arguments</h2></summary> </details> <details> <summary><h2>Extra Inner-working Details</h2></summary> </details>

References

@inproceedings{ghaffari2022fslfirth,
    title={On the Importance of Firth Bias Reduction in Few-Shot Classification},
    author={Saba Ghaffari and Ehsan Saleh and David Forsyth and Yu-Xiong Wang},
    booktitle={International Conference on Learning Representations},
    year={2022},
    url={https://openreview.net/forum?id=DNRADop4ksB}
}