Home

Awesome

Fast Per-Example Gradient Clipping

This repository contains the source code for the paper "Scaling Up Differentially Private Deep Learning with Fast Per-Example Gradient Clipping" (to appear in POPETS 2021).

The provided fastgc package provides a fast and scalable PyTorch implementation of gradient clipping method used for satisfying differential privacy. It provides a set of wrapper classes for neural network layers. To compute per-example gradients for differential privacy, you can simply import the fastgc package and replace the original PyTorch layers in your network with the provided wrapper classes. The support types of layers include:


Installing dependecies

pip install -r requirements.txt

Running the code

The main function is implemented in run_algo.py. To train a neural network using the proposed ReweightGP algorithm, you can execute

python run_algo.py --train_alg reweight --data_dir <path to the directory containing the dataset> --download True

Example usage

python run_algo.py --train_alg batch --model_name CNN --data_dir <path to data> --dname mnist --download True
python run_algo.py --train_alg reweight --model_name RNN --data_dir <path to data> --dname mnist --batch_size 128 --download True

Important Input Arguments

The script takes several input arguments. To see available command line input arguments and allowed values, simply run

python run_algo.py --help

Here we introduce each optional argument:

Result Files

After the program runs for the specified number of iterations, it stores the results (avg. loss, avg. per-epoch execution time, etc) into a .csv file. The resulting csv file has the following naming convetion:

[training algorithm name]_[model name]_[dataset_name]_B[batch_size]E[epochs]C[clipping_threshold]SIG[sigma].csv

Datasets

The paper uses 5 different datasets for the evaluation of proposed algorithm: