Awesome
Deep Learning Models
A collection of various deep learning architectures, models, and tips for TensorFlow and PyTorch in Jupyter Notebooks.
Traditional Machine Learning
Multilayer Perceptrons
Convolutional Neural Networks
Basic
Title | Dataset | Description | Notebooks |
---|---|---|---|
Convolutional Neural Network | TBD | TBD | |
CNN with He Initialization | TBD | TBD |
Concepts
Title | Dataset | Description | Notebooks |
---|---|---|---|
Replacing Fully-Connected by Equivalent Convolutional Layers | TBD | TBD |
AlexNet
Title | Dataset | Description | Notebooks |
---|---|---|---|
AlexNet Trained on CIFAR-10 | TBD | TBD | |
AlexNet with Grouped Convolutions Trained on CIFAR-10 | TBD | TBD |
DenseNet
Title | Description | Daset | Notebooks |
---|---|---|---|
DenseNet-121 Digit Classifier Trained on MNIST | TBD | TBD | |
DenseNet-121 Image Classifier Trained on CIFAR-10 | TBD | TBD |
Fully Convolutional
Title | Dataset | Description | Notebooks |
---|---|---|---|
"All Convolutionl Net" -- A Fully Convolutional Neural Network | TBD | TBD |
LeNet
Title | Dataset | Description | Notebooks |
---|---|---|---|
LeNet-5 on MNIST | TBD | TBD | |
LeNet-5 on CIFAR-10 | TBD | TBD | |
LeNet-5 on QuickDraw | TBD | TBD |
MobileNet
Title | Dataset | Description | Notebooks |
---|---|---|---|
MobileNet-v2 on Cifar-10 | TBD | TBD | |
MobileNet-v3 small on Cifar-10 | TBD | TBD | |
MobileNet-v3 large on Cifar-10 | TBD | TBD | |
MobileNet-v3 large on MNIST via Embetter | TBD | TBD |
Network in Network
Title | Dataset | Description | Notebooks |
---|---|---|---|
Network in Network Trained on CIFAR-10 | TBD | TBD |
VGG
Title | Dataset | Description | Notebooks |
---|---|---|---|
Convolutional Neural Network VGG-16 Trained on CIFAR-10 | TBD | TBD | |
VGG-16 Smile Classifier | CelebA | TBD | |
VGG-16 Dogs vs Cats Classifier | TBD | TBD | |
Convolutional Neural Network VGG-19 | TBD | TBD |
ResNet
Title | Dataset | Description | Notebooks |
---|---|---|---|
ResNet and Residual Blocks | MNIST | TBD | |
ResNet-18 Digit Classifier | MNIST | TBD | |
ResNet-18 Gender Classifier | CelebA | TBD | |
ResNet-34 Digit Classifier | MNIST | TBD | |
ResNet-34 Object Classifier | QuickDraw | TBD | |
ResNet-34 Gender Classifier | CelebA | TBD | |
ResNet-50 Digit Classifier | MNIST | TBD | |
ResNet-50 Gender Classifier | CelebA | TBD | |
ResNet-101 Gender Classifier | CelebA | TBD | |
ResNet-101 | CIFAR-10 | TBD | |
ResNet-152 Gender Classifier | CelebA | TBD |
Transformers
Title | Dataset | Description | Notebooks |
---|---|---|---|
Multilabel DistilBERT | Jigsaw Toxic Comment Challenge | DistilBERT classifier fine-tuning | |
DistilBERT as feature extractor | IMDB movie review | DistilBERT classifier with sklearn random forest and logistic regression | |
DistilBERT as feature extractor using embetter | IMDB movie review | DistilBERT classifier with sklearn random forest and logistic regression using the scikit-learn embetter library | |
Fine-tune DistilBERT I | IMDB movie review | Fine-tune only the last 2 layers of DistilBERT classifier | |
Fine-tune DistilBERT II | IMDB movie review | Fine-tune the whole DistilBERT classifier |
Ordinal Regression and Deep Learning
Please note that the following notebooks below provide reference implementations to use the respective methods. They are not performance benchmarks.
Title | Dataset | Description | Notebooks |
---|---|---|---|
Baseline multilayer perceptron | Cement | A baseline multilayer perceptron for classification trained with the standard cross entropy loss | |
CORAL multilayer perceptron | Cement | Implementation of Rank Consistent Ordinal Regression for Neural Networks with Application to Age Estimation 2020 | |
CORN multilayer perceptron | Cement | Implementation of Deep Neural Networks for Rank-Consistent Ordinal Regression Based On Conditional Probabilities 2022 | |
Binary extension multilayer perceptron | Cement | Implementation of Ordinal Regression with Multiple Output CNN for Age Estimation 2016 | |
Reformulated squared-error multilayer perceptron | Cement | Implementation of A simple squared-error reformulation for ordinal classification 2016 | |
Class distance weighted cross-entropy loss | Cement | Implementation of Class Distance Weighted Cross-Entropy Loss for Ulcerative Colitis Severity Estimation 2022 |
Normalization Layers
Title | Dataset | Description | Notebooks |
---|---|---|---|
BatchNorm before and after Activation for Network-in-Network CIFAR-10 Classifier | TBD | TBD | |
Filter Response Normalization for Network-in-Network CIFAR-10 Classifier | TBD | TBD |
Metric Learning
Title | Dataset | Description | Notebooks |
---|---|---|---|
Siamese Network with Multilayer Perceptrons | TBD | TBD |
Autoencoders
Fully-connected Autoencoders
Title | Dataset | Description | Notebooks |
---|---|---|---|
Autoencoder (MNIST) | TBD | TBD | |
Autoencoder (MNIST) + Scikit-Learn Random Forest Classifier | TBD | TBD |
Convolutional Autoencoders
Variational Autoencoders
Title | Dataset | Description | Notebooks |
---|---|---|---|
Variational Autoencoder | TBD | TBD | |
Convolutional Variational Autoencoder | TBD | TBD |
Conditional Variational Autoencoders
Generative Adversarial Networks (GANs)
Graph Neural Networks (GNNs)
Recurrent Neural Networks (RNNs)
Many-to-one: Sentiment Analysis / Classification
Many-to-Many / Sequence-to-Sequence
Title | Dataset | Description | Notebooks |
---|---|---|---|
A simple character RNN to generate new text (Charles Dickens) | TBD | TBD |
Model Evaluation
K-Fold Cross-Validation
Title | Dataset | Description | Notebooks |
---|---|---|---|
Baseline CNN | MNIST | A simple baseline with traditional train/validation/test splits | |
K-fold with pl_cross | MNIST | A 5-fold cross-validation run using the pl_cross library |
Data Augmentation
Title | Dataset | Description | Notebooks |
---|---|---|---|
AutoAugment & TrivialAugment for Image Data | CIFAR-10 | Trains a ResNet-18 using AutoAugment and TrivialAugment |
Tips and Tricks
Title | Dataset | Description | Notebooks |
---|---|---|---|
Cyclical Learning Rate | TBD | TBD | |
Annealing with Increasing the Batch Size (w. CIFAR-10 & AlexNet) | TBD | TBD | |
Gradient Clipping (w. MLP on MNIST) | TBD | TBD |
Transfer Learning
Title | Dataset | Description | Notebooks |
---|---|---|---|
Transfer Learning Example (VGG16 pre-trained on ImageNet for Cifar-10) | TBD | TBD |
Visualization and Interpretation
PyTorch Workflows and Mechanics
PyTorch Lightning Examples
Title | Dataset | Description | Notebooks |
---|---|---|---|
MLP in Lightning with TensorBoard -- continue training the last model | TBD | TBD | |
MLP in Lightning with TensorBoard -- checkpointing best model | TBD | TBD |
Custom Datasets
Training and Preprocessing
Improving Memory Efficiency
Title | Dataset | Description | Notebooks |
---|---|---|---|
Gradient Checkpointing Demo (Network-in-Network trained on CIFAR-10) | TBD | TBD |
Parallel Computing
Title | Description | Notebooks |
---|---|---|
Using Multiple GPUs with DataParallel -- VGG-16 Gender Classifier on CelebA | TBD | |
Distribute a Model Across Multiple GPUs with Pipeline Parallelism (VGG-16 Example) | TBD |
Other
Autograd
Title | Dataset | Description | Notebooks |
---|---|---|---|
Getting Gradients of an Intermediate Variable in PyTorch | TBD | TBD |
TensorFlow Workflows and Mechanics
Custom Datasets
Training and Preprocessing
Title | Dataset | Description | Notebooks |
---|---|---|---|
Saving and Loading Trained Models -- from TensorFlow Checkpoint Files and NumPy NPZ Archives | TBD | TBD |
Related Libraries
Title | Description | Notebooks |
---|---|---|
TorchMetrics | How do we use it, and what's the difference between .update() and .forward()? |