Home

Awesome

Deep Learning Models

A collection of various deep learning architectures, models, and tips for TensorFlow and PyTorch in Jupyter Notebooks.

Traditional Machine Learning

TitleDatasetDescriptionNotebooks
Perceptron2D toy dataTBDPyTorch TensorFlow
Logistic Regression2D toy dataTBDPyTorch TensorFlow
Softmax Regression (Multinomial Logistic Regression)MNISTTBDPyTorch TensorFlow
Softmax Regression with MLxtend's plot_decision_regions on IrisIrisTBDPyTorch

Multilayer Perceptrons

TitleDatasetDescriptionNotebooks
Multilayer PerceptronMNISTTBDPyTorch Lightning PyTorch TensorFlow
Multilayer Perceptron with DropoutMNISTTBDPyTorch Lightning PyTorch TensorFlow
Multilayer Perceptron with Batch NormalizationMNISTTBDPyTorch Lightning PyTorch TensorFlow
Multilayer Perceptron with Backpropagation from ScratchMNISTTBDPyTorch TensorFlow

Convolutional Neural Networks

Basic

TitleDatasetDescriptionNotebooks
Convolutional Neural NetworkTBDTBDPyTorch Lightning PyTorch TensorFlow
CNN with He InitializationTBDTBDPyTorch Lightning PyTorch

Concepts

TitleDatasetDescriptionNotebooks
Replacing Fully-Connected by Equivalent Convolutional LayersTBDTBDPyTorch

AlexNet

TitleDatasetDescriptionNotebooks
AlexNet Trained on CIFAR-10TBDTBDPyTorch Lightning PyTorch
AlexNet with Grouped Convolutions Trained on CIFAR-10TBDTBDPyTorch Lightning PyTorch

DenseNet

TitleDescriptionDasetNotebooks
DenseNet-121 Digit Classifier Trained on MNISTTBDTBDPyTorch Lightning PyTorch
DenseNet-121 Image Classifier Trained on CIFAR-10TBDTBDPyTorch Lightning PyTorch

Fully Convolutional

TitleDatasetDescriptionNotebooks
"All Convolutionl Net" -- A Fully Convolutional Neural NetworkTBDTBDPyTorch Lightning PyTorch

LeNet

TitleDatasetDescriptionNotebooks
LeNet-5 on MNISTTBDTBDPyTorch Lightning PyTorch
LeNet-5 on CIFAR-10TBDTBDPyTorch Lightning PyTorch
LeNet-5 on QuickDrawTBDTBDPyTorch Lightning PyTorch

MobileNet

TitleDatasetDescriptionNotebooks
MobileNet-v2 on Cifar-10TBDTBDPyTorch Lightning PyTorch
MobileNet-v3 small on Cifar-10TBDTBDPyTorch Lightning PyTorch
MobileNet-v3 large on Cifar-10TBDTBDPyTorch Lightning PyTorch
MobileNet-v3 large on MNIST via EmbetterTBDTBDPyTorch

Network in Network

TitleDatasetDescriptionNotebooks
Network in Network Trained on CIFAR-10TBDTBDPyTorch Lightning PyTorch

VGG

TitleDatasetDescriptionNotebooks
Convolutional Neural Network VGG-16 Trained on CIFAR-10TBDTBDPyTorch Lightning PyTorch TensorFlow
VGG-16 Smile ClassifierCelebATBDPyTorch Lightning PyTorch
VGG-16 Dogs vs Cats ClassifierTBDTBDPyTorch
Convolutional Neural Network VGG-19TBDTBDPyTorch Lightning PyTorch

ResNet

TitleDatasetDescriptionNotebooks
ResNet and Residual BlocksMNISTTBDPyTorch
ResNet-18 Digit ClassifierMNISTTBDPyTorch
ResNet-18 Gender ClassifierCelebATBDPyTorch
ResNet-34 Digit ClassifierMNISTTBDPyTorch
ResNet-34 Object ClassifierQuickDrawTBDPyTorch
ResNet-34 Gender ClassifierCelebATBDPyTorch
ResNet-50 Digit ClassifierMNISTTBDPyTorch
ResNet-50 Gender ClassifierCelebATBDPyTorch
ResNet-101 Gender ClassifierCelebATBDPyTorch
ResNet-101CIFAR-10TBDPyTorch
ResNet-152 Gender ClassifierCelebATBDPyTorch

Transformers

TitleDatasetDescriptionNotebooks
Multilabel DistilBERTJigsaw Toxic Comment ChallengeDistilBERT classifier fine-tuningPyTorch
DistilBERT as feature extractorIMDB movie reviewDistilBERT classifier with sklearn random forest and logistic regressionPyTorch
DistilBERT as feature extractor using embetterIMDB movie reviewDistilBERT classifier with sklearn random forest and logistic regression using the scikit-learn embetter libraryPyTorch
Fine-tune DistilBERT IIMDB movie reviewFine-tune only the last 2 layers of DistilBERT classifierPyTorch Lightning
Fine-tune DistilBERT IIIMDB movie reviewFine-tune the whole DistilBERT classifierPyTorch PyTorch Lightning

Ordinal Regression and Deep Learning

Please note that the following notebooks below provide reference implementations to use the respective methods. They are not performance benchmarks.

TitleDatasetDescriptionNotebooks
Baseline multilayer perceptronCementA baseline multilayer perceptron for classification trained with the standard cross entropy lossPyTorch PyTorch Lightning
CORAL multilayer perceptronCementImplementation of Rank Consistent Ordinal Regression for Neural Networks with Application to Age Estimation 2020PyTorch PyTorch Lightning
CORN multilayer perceptronCementImplementation of Deep Neural Networks for Rank-Consistent Ordinal Regression Based On Conditional Probabilities 2022PyTorch PyTorch Lightning
Binary extension multilayer perceptronCementImplementation of Ordinal Regression with Multiple Output CNN for Age Estimation 2016PyTorch PyTorch Lightning
Reformulated squared-error multilayer perceptronCementImplementation of A simple squared-error reformulation for ordinal classification 2016PyTorch PyTorch Lightning
Class distance weighted cross-entropy lossCementImplementation of Class Distance Weighted Cross-Entropy Loss for Ulcerative Colitis Severity Estimation 2022PyTorch PyTorch Lightning

Normalization Layers

TitleDatasetDescriptionNotebooks
BatchNorm before and after Activation for Network-in-Network CIFAR-10 ClassifierTBDTBDPyTorch
Filter Response Normalization for Network-in-Network CIFAR-10 ClassifierTBDTBDPyTorch

Metric Learning

TitleDatasetDescriptionNotebooks
Siamese Network with Multilayer PerceptronsTBDTBDTensorFlow

Autoencoders

Fully-connected Autoencoders

TitleDatasetDescriptionNotebooks
Autoencoder (MNIST)TBDTBDPyTorch TensorFlow
Autoencoder (MNIST) + Scikit-Learn Random Forest ClassifierTBDTBDPyTorch TensorFlow

Convolutional Autoencoders

TitleDatasetDescriptionNotebooks
Convolutional Autoencoder with Deconvolutions / Transposed ConvolutionsTBDTBDPyTorch TensorFlow
Convolutional Autoencoder with Deconvolutions and Continuous Jaccard DistanceTBDTBDPyTorch
Convolutional Autoencoder with Deconvolutions (without pooling operations)TBDTBDPyTorch
Convolutional Autoencoder with Nearest-neighbor InterpolationTBDTBDPyTorch TensorFlow
Convolutional Autoencoder with Nearest-neighbor Interpolation -- Trained on CelebATBDTBDPyTorch
Convolutional Autoencoder with Nearest-neighbor Interpolation -- Trained on QuickdrawTBDTBDPyTorch

Variational Autoencoders

TitleDatasetDescriptionNotebooks
Variational AutoencoderTBDTBDPyTorch
Convolutional Variational AutoencoderTBDTBDPyTorch

Conditional Variational Autoencoders

TitleDatasetDescriptionNotebooks
Conditional Variational Autoencoder (with labels in reconstruction loss)TBDTBDPyTorch
Conditional Variational Autoencoder (without labels in reconstruction loss)TBDTBDPyTorch
Convolutional Conditional Variational Autoencoder (with labels in reconstruction loss)TBDTBDPyTorch
Convolutional Conditional Variational Autoencoder (without labels in reconstruction loss)TBDTBDPyTorch

Generative Adversarial Networks (GANs)

TitleDatasetDescriptionNotebooks
Fully Connected GAN on MNISTTBDTBDPyTorch TensorFlow
Fully Connected Wasserstein GAN on MNISTTBDTBDPyTorch
Convolutional GAN on MNISTTBDTBDPyTorch TensorFlow
Convolutional GAN on MNIST with Label SmoothingTBDTBDPyTorch TensorFlow
Convolutional Wasserstein GAN on MNISTTBDTBDPyTorch
Deep Convolutional GAN (DCGAN) on Cats and Dogs ImagesTBDTBDPyTorch
Deep Convolutional GAN (DCGAN) on CelebA Face ImagesTBDTBDPyTorch

Graph Neural Networks (GNNs)

TitleDatasetDescriptionNotebooks
Most Basic Graph Neural Network with Gaussian Filter on MNISTTBDTBDPyTorch
Basic Graph Neural Network with Edge Prediction on MNISTTBDTBDPyTorch
Basic Graph Neural Network with Spectral Graph Convolution on MNISTTBDTBDPyTorch

Recurrent Neural Networks (RNNs)

Many-to-one: Sentiment Analysis / Classification

TitleDatasetDescriptionNotebooks
A simple single-layer RNN (IMDB)TBDTBDPyTorch
A simple single-layer RNN with packed sequences to ignore padding characters (IMDB)TBDTBDPyTorch
RNN with LSTM cells (IMDB)TBDTBDPyTorch
RNN with LSTM cells (IMDB) and pre-trained GloVe word vectorsTBDTBDPyTorch
RNN with LSTM cells and Own Dataset in CSV Format (IMDB)TBDTBDPyTorch
RNN with GRU cells (IMDB)TBDTBDPyTorch
Multilayer bi-directional RNN (IMDB)TBDTBDPyTorch
Bidirectional Multi-layer RNN with LSTM with Own Dataset in CSV Format (AG News)TBDTBDPyTorch

Many-to-Many / Sequence-to-Sequence

TitleDatasetDescriptionNotebooks
A simple character RNN to generate new text (Charles Dickens)TBDTBDPyTorch

Model Evaluation

K-Fold Cross-Validation

TitleDatasetDescriptionNotebooks
Baseline CNNMNISTA simple baseline with traditional train/validation/test splitsPyTorch PyTorch Lightning
K-fold with pl_crossMNISTA 5-fold cross-validation run using the pl_cross libraryPyTorch Lightning

Data Augmentation

TitleDatasetDescriptionNotebooks
AutoAugment & TrivialAugment for Image DataCIFAR-10Trains a ResNet-18 using AutoAugment and TrivialAugmentPyTorch Lightning

Tips and Tricks

TitleDatasetDescriptionNotebooks
Cyclical Learning RateTBDTBDPyTorch
Annealing with Increasing the Batch Size (w. CIFAR-10 & AlexNet)TBDTBDPyTorch
Gradient Clipping (w. MLP on MNIST)TBDTBDPyTorch

Transfer Learning

TitleDatasetDescriptionNotebooks
Transfer Learning Example (VGG16 pre-trained on ImageNet for Cifar-10)TBDTBDPyTorch

Visualization and Interpretation

TitleDatasetDescriptionNotebooks
Vanilla Loss Gradient (wrt Inputs) Visualization (Based on a VGG16 Convolutional Neural Network for Kaggle's Cats and Dogs Images)TBDTBDPyTorch
Guided Backpropagation (Based on a VGG16 Convolutional Neural Network for Kaggle's Cats and Dogs Images)TBDTBDPyTorch

PyTorch Workflows and Mechanics

PyTorch Lightning Examples

TitleDatasetDescriptionNotebooks
MLP in Lightning with TensorBoard -- continue training the last modelTBDTBDPyTorch
MLP in Lightning with TensorBoard -- checkpointing best modelTBDTBDPyTorch

Custom Datasets

TitleDatasetDescriptionNotebooks
Custom Data Loader Example for PNG FilesTBDTBDPyTorch
Using PyTorch Dataset Loading Utilities for Custom Datasets -- CSV files converted to HDF5TBDTBDPyTorch
Using PyTorch Dataset Loading Utilities for Custom Datasets -- Face Images from CelebATBDTBDPyTorch
Using PyTorch Dataset Loading Utilities for Custom Datasets -- Drawings from QuickdrawTBDTBDPyTorch
Using PyTorch Dataset Loading Utilities for Custom Datasets -- Drawings from the Street View House Number (SVHN) DatasetTBDTBDPyTorch
Using PyTorch Dataset Loading Utilities for Custom Datasets -- Asian Face Dataset (AFAD)TBDTBDPyTorch
Using PyTorch Dataset Loading Utilities for Custom Datasets -- Dating Historical Color ImagesTBDTBDPyTorch
Using PyTorch Dataset Loading Utilities for Custom Datasets -- Fashion MNISTTBDTBDPyTorch

Training and Preprocessing

TitleDatasetDescriptionNotebooks
PyTorch DataLoader State and Nested IterationsToyExplains DataLoader behavior when in nested functionsPyTorch
Generating Validation Set SplitsTBDTBDPyTorch
Dataloading with Pinned MemoryTBDTBDPyTorch
Standardizing ImagesTBDTBDPyTorch
Image Transformation ExamplesTBDTBDPyTorch
Char-RNN with Own Text FileTBDTBDPyTorch
Sentiment Classification RNN with Own CSV FileTBDTBDPyTorch

Improving Memory Efficiency

TitleDatasetDescriptionNotebooks
Gradient Checkpointing Demo (Network-in-Network trained on CIFAR-10)TBDTBDPyTorch

Parallel Computing

TitleDescriptionNotebooks
Using Multiple GPUs with DataParallel -- VGG-16 Gender Classifier on CelebATBDPyTorch
Distribute a Model Across Multiple GPUs with Pipeline Parallelism (VGG-16 Example)TBDPyTorch

Other

TitleDatasetDescriptionNotebooks
PyTorch with and without Deterministic Behavior -- Runtime BenchmarkTBDTBDPyTorch
Sequential API and hooksTBDTBDPyTorch
Weight Sharing Within a LayerTBDTBDPyTorch
Plotting Live Training Performance in Jupyter Notebooks with just MatplotlibTBDTBDPyTorch

Autograd

TitleDatasetDescriptionNotebooks
Getting Gradients of an Intermediate Variable in PyTorchTBDTBDPyTorch

TensorFlow Workflows and Mechanics

Custom Datasets

TitleDescriptionNotebooks
Chunking an Image Dataset for Minibatch Training using NumPy NPZ ArchivesTBDTensorFlow
Storing an Image Dataset for Minibatch Training using HDF5TBDTensorFlow
Using Input Pipelines to Read Data from TFRecords FilesTBDTensorFlow
Using Queue Runners to Feed Images Directly from DiskTBDTensorFlow
Using TensorFlow's Dataset APITBDTensorFlow

Training and Preprocessing

TitleDatasetDescriptionNotebooks
Saving and Loading Trained Models -- from TensorFlow Checkpoint Files and NumPy NPZ ArchivesTBDTBDTensorFlow

Related Libraries

TitleDescriptionNotebooks
TorchMetricsHow do we use it, and what's the difference between .update() and .forward()?PyTorch