Home

Awesome

MONAI Tutorials

This repository hosts the MONAI tutorials.

1. Requirements

Most of the examples and tutorials require matplotlib and Jupyter Notebook.

These can be installed with:

python -m pip install -U pip
python -m pip install -U matplotlib
python -m pip install -U notebook

Some of the examples may require optional dependencies. In case of any optional import errors, please install the relevant packages according to MONAI's installation guide. Or install all optional requirements with:

pip install -r https://raw.githubusercontent.com/Project-MONAI/MONAI/dev/requirements-dev.txt

Run the notebooks from Colab

Most of the Jupyter Notebooks have an "Open in Colab" button. Please right-click on the button, and select "Open Link in New Tab" to start a Colab page with the corresponding notebook content.

To use GPU resources through Colab, please remember to change the runtime type to GPU:

  1. From the Runtime menu select Change runtime type
  2. Choose GPU from the drop-down menu
  3. Click SAVE This will reset the notebook and may ask you if you are a robot (these instructions assume you are not).

Running:

!nvidia-smi

in a cell will verify this has worked and show you what kind of hardware you have access to.

Data

Some notebooks will require additional data. Each user is responsible for checking the content of datasets and the applicable licenses and determining if suitable for the intended use.

2. Questions and bugs

3. Become a contributor

You can read details about adding a tutorial in our CONTRIBUTING GUIDELINES.

4. List of notebooks and examples

<ins>2D classification</ins>

mednist_tutorial

This notebook shows how to easily integrate MONAI features into existing PyTorch programs. It's based on the MedNIST dataset which is very suitable for beginners as a tutorial. This tutorial also makes use of MONAI's in-built occlusion sensitivity functionality.

<ins>2D segmentation

torch examples

Training and evaluation examples of 2D segmentation based on UNet and synthetic dataset. The examples are standard PyTorch programs and have both dictionary-based and array-based versions.

<ins>3D classification</ins>

ignite examples

Training and evaluation examples of 3D classification based on DenseNet3D and IXI dataset. The examples are PyTorch Ignite programs and have both dictionary-based and array-based transformation versions.

torch examples

Training and evaluation examples of 3D classification based on DenseNet3D and IXI dataset. The examples are standard PyTorch programs and have both dictionary-based and array-based transformation versions.

<ins>3D regression</ins>

densenet_training_array.ipynb

Training and evaluation examples of 3D regression based on DenseNet3D and IXI dataset.

<ins>3D segmentation</ins>

ignite examples

Training and evaluation examples of 3D segmentation based on UNet3D and synthetic dataset. The examples are PyTorch Ignite programs and have both dictionary-base and array-based transformations.

torch examples

Training, evaluation and inference examples of 3D segmentation based on UNet3D and synthetic dataset. The examples are standard PyTorch programs and have both dictionary-based and array-based versions.

brats_segmentation_3d

This tutorial shows how to construct a training workflow of multi-labels segmentation task based on MSD Brain Tumor dataset, and how to convert the pytorch model to an onnx model for inference and comparison.

spleen_segmentation_3d_aim

This notebook shows how MONAI may be used in conjunction with the aimhubio/aim.

spleen_segmentation_3d_lightning

This notebook shows how MONAI may be used in conjunction with the PyTorch Lightning framework.

spleen_segmentation_3d

This notebook is an end-to-end training and evaluation example of 3D segmentation based on MSD Spleen dataset. The example shows the flexibility of MONAI modules in a PyTorch-based program:

unet_segmentation_3d_ignite

This notebook is an end-to-end training & evaluation example of 3D segmentation based on synthetic dataset. The example is a PyTorch Ignite program and shows several key features of MONAI, especially with medical domain specific transforms and event handlers for profiling (logging, TensorBoard, MLFlow, etc.).

COVID 19-20 challenge baseline

This folder provides a simple baseline method for training, validation, and inference for COVID-19 LUNG CT LESION SEGMENTATION CHALLENGE - 2020 (a MICCAI Endorsed Event).

unetr_btcv_segmentation_3d

This notebook demonstrates how to construct a training workflow of UNETR on multi-organ segmentation task using the BTCV challenge dataset.

unetr_btcv_segmentation_3d_lightning

This tutorial demonstrates how MONAI can be used in conjunction with PyTorch Lightning framework to construct a training workflow of UNETR on multi-organ segmentation task using the BTCV challenge dataset.

vista3d

This tutorial showcases the process of fine-tuning VISTA3D on MSD Spleen dataset using MONAI. For an in-depth exploration, please visit the VISTA repository.

<ins>2D registration</ins>

registration using mednist

This notebook shows a quick demo for learning based affine registration of 64 x 64 X-Ray hands.

<ins>3D registration</ins>

3D registration using paired lung CT

This tutorial shows how to use MONAI to register lung CT volumes acquired at different time points for a single patient.

3D registration using unpaired brain MR

This tutorial shows how to get started on using the general-purpose registration framework VoxelMorph offered in MONAI to register unpaired brain MR volumes.

DeepAtlas

This tutorial demonstrates the use of MONAI for training of registration and segmentation models together. The DeepAtlas approach, in which the two models serve as a source of weakly supervised learning for each other, is useful in situations where one has many unlabeled images and just a few images with segmentation labels. The notebook works with 3D images from the OASIS-1 brain MRI dataset.

<ins>Deepgrow</ins>

Deepgrow

The example show how to train/validate a 2D/3D deepgrow model. It also demonstrates running an inference for trained deepgrow models.

<ins>DeepEdit</ins>

DeepEdit

This example shows how to train/test a DeepEdit model. In this tutorial there is a Notebook that shows how to run inference on a pretrained DeepEdit model.

<ins>Deployment</ins>

BentoML

This is a simple example of training and deploying a MONAI network with BentoML as a web server, either locally using the BentoML repository or as a containerized service.

Ray

This uses the previous notebook's trained network to demonstrate deployment a web server using Ray.

Triton

This is example walks through using a Triton Server and Python client using MONAI on the MedNIST classification problem. The demo is self contained and the Readme explains how to use Triton "backends" to inject the MONAI code into the server. See Triton Inference Server/python_backend documentation

<ins>Experiment Management</ins>

Aim

An example of experiment management with Aim, using 3D spleen segmentation as an example.

MLFlow

An example of experiment management with MLFlow, using 3D spleen segmentation as an example.

MONAI bundle integrates MLFlow

An example shows how to easily enable and customize the MLFlow for experiment management in MONAI bundle.

ClearML

An example of experiment management with ClearML, using 3D Segmentation with UNet as an example.

<ins>Federated Learning</ins>

NVFlare

The examples show how to train federated learning models with NVFlare and MONAI-based trainers.

OpenFL

The examples show how to train federated learning models based on OpenFL and MONAI.

Substra

The example show how to execute the 3d segmentation torch tutorial on a federated learning platform, Substra.

Breast Density FL Challenge

Reference implementation used in MICCAI 2022 ACR-NVIDIA-NCI Breast Density FL challenge.

<ins>Digital Pathology</ins>

Whole Slide Tumor Detection

The example shows how to train and evaluate a tumor detection model (based on patch classification) on whole-slide histopathology images.

Profiling Whole Slide Tumor Detection

The example shows how to use MONAI NVTX transforms to tag and profile pre- and post-processing transforms in the digital pathology whole slide tumor detection pipeline.

Multiple Instance Learning WSI classification

An example of Multiple Instance Learning (MIL) classification from Whole Slide Images (WSI) of prostate histopathology.

NuClick Annotation

The notebook demonstrates examples of training and inference pipelines with interactive annotation for pathology, NuClick is used for delineating nuclei, cells and a squiggle for outlining glands.

HoVerNet:Nuclear segmentation and classification task

This tutorial demonstrates how to construct a training workflow of HoVerNet on nuclear segmentation and classification task using the CoNSep dataset.

Nuclei Classification

The notebook demonstrates examples of training and inference pipelines with interactive annotation for pathology, NuClick is used for delineating nuclei, cells and a squiggle for outlining glands.

<ins>Acceleration</ins>

fast_model_training_guide

The document introduces details of how to profile the training pipeline, how to analyze the dataset and select suitable algorithms, and how to optimize GPU utilization in single GPU, multi-GPUs or even multi-nodes.

distributed_training

The examples show how to execute distributed training and evaluation based on 3 different frameworks:

They can run on several distributed nodes with multiple GPU devices on every node.

automatic_mixed_precision

And compares the training speed and memory usage with/without AMP.

dataset_type_performance

This notebook compares the performance of Dataset, CacheDataset and PersistentDataset. These classes differ in how data is stored (in memory or on disk), and at which moment transforms are applied.

fast_training_tutorial

This tutorial compares the training performance of pure PyTorch program and optimized program in MONAI based on NVIDIA GPU device and latest CUDA library. The optimization methods mainly include: AMP, CacheDataset, GPU transforms, ThreadDataLoader, DiceCELoss and SGD.

threadbuffer_performance

Demonstrates the use of the ThreadBuffer class used to generate data batches during training in a separate thread.

transform_speed

Illustrate reading NIfTI files and test speed of different transforms on different devices.

TensorRT_inference_acceleration

This notebook shows how to use TensorRT to accelerate the model and achieve a better inference latency.

<ins>Model Zoo</ins>

easy_integrate_bundle

This tutorial shows a straightforward ensemble application to instruct users on how to integrate existing bundles in their own projects. By simply changing the data path and the path where the bundle is located, training and ensemble inference can be performed.

<ins>Computer Assisted Intervention</ins>

video segmentation

This tutorial shows how to train a surgical tool segmentation model to locate tools in a given image. In addition, it also builds an example pipeline of an end-to-end video tool segmentation, with video input and video output.

endoscopic inbody classification

Tutorial to show the pipeline of fine tuning an endoscopic inbody classification model based on a corresponding pretrained bundle.

<ins>Modules</ins>

bundle

Get started tutorial and concrete training / inference examples for MONAI bundle features.

competitions

MONAI based solutions of competitions in healthcare imaging.

engines

Training and evaluation examples of 3D segmentation based on UNet3D and synthetic dataset with MONAI workflows, which contains engines, event-handlers, and post-transforms. And GAN training and evaluation example for a medical image generative adversarial network. Easy run training script uses GanTrainer to train a 2D CT scan reconstruction network. Evaluation script generates random samples from a trained network.

The examples are built with MONAI workflows, mainly contain: trainer/evaluator, handlers, post_transforms, etc.

3d_image_transforms

This notebook demonstrates the transformations on volumetric images.

2d_inference_3d_volume

Tutorial that demonstrates how monai SlidingWindowInferer can be used when a 3D volume input needs to be provided slice-by-slice to a 2D model and finally, aggregated into a 3D volume.

autoencoder_mednist

This tutorial uses the MedNIST hand CT scan dataset to demonstrate MONAI's autoencoder class. The autoencoder is used with an identity encode/decode (i.e., what you put in is what you should get back), as well as demonstrating its usage for de-blurring and de-noising.

batch_output_transform

Tutorial to explain and show how to set batch_transform and output_transform of handlers to work with MONAI engines.

bending_energy_diffusion_loss_notes

This notebook demonstrates when and how to compute normalized bending energy and diffusion loss.

compute_metric

Example shows how to compute metrics from saved predictions and labels with PyTorch multi-processing support.

csv_datasets

Tutorial shows the usage of CSVDataset and CSVIterableDataset, load multiple CSV files and execute postprocessing logic.

decollate_batch

Tutorial shows how to decollate batch data to simplify post processing transforms and execute more flexible following operations.

image_dataset

Notebook introduces basic usages of monai.data.ImageDataset module.

dynunet_tutorial

This tutorial shows how to train 3D segmentation tasks on all the 10 decathlon datasets with the reimplementation of dynUNet in MONAI.

integrate_3rd_party_transforms

This tutorial shows how to integrate 3rd party transforms into MONAI program. Mainly shows transforms from BatchGenerator, TorchIO, Rising and ITK.

inverse transformations and test-time augmentations

This notebook demonstrates the use of invertible transforms, and then leveraging inverse transformations to perform test-time augmentations.

layer wise learning rate

This notebook demonstrates how to select or filter out expected network layers and set customized learning rate values.

learning rate finder

This notebook demonstrates how to use LearningRateFinder API to tune the learning rate values for the network.

load_medical_images

This notebook introduces how to easily load different formats of medical images in MONAI and execute many additional operations.

mednist_GAN_tutorial

This notebook illustrates the use of MONAI for training a network to generate images from a random input tensor. A simple GAN is employed to do with a separate Generator and Discriminator networks.

mednist_GAN_workflow_dict

This notebook shows the GanTrainer, a MONAI workflow engine for modularized adversarial learning. Train a medical image reconstruction network using the MedNIST hand CT scan dataset. Dictionary version.

mednist_GAN_workflow_array

This notebook shows the GanTrainer, a MONAI workflow engine for modularized adversarial learning. Train a medical image reconstruction network using the MedNIST hand CT scan dataset. Array version.

cross_validation_models_ensemble

This tutorial shows how to leverage CrossValidation, EnsembleEvaluator, MeanEnsemble and VoteEnsemble modules in MONAI to set up cross validation and ensemble program.

nifti_read_example

Illustrate reading NIfTI files and iterating over image patches of the volumes loaded from them.

network_api

This tutorial illustrates the flexible network APIs and utilities.

postprocessing_transforms

This notebook shows the usage of several postprocessing transforms based on the model output of spleen segmentation task.

public_datasets

This notebook shows how to quickly set up training workflow based on MedNISTDataset and DecathlonDataset, and how to create a new dataset.

tcia_csv_processing

This notebook shows how to load the TCIA data with CSVDataset from CSV file and extract information for TCIA data to fetch DICOM images based on REST API.

transforms_demo_2d

This notebook demonstrates the image transformations on histology images using

UNet_input_size_constraints

This tutorial shows how to determine a reasonable spatial size of the input data for MONAI UNet, which not only supports residual units, but also can use more hyperparameters (like strides, kernel_size and up_kernel_size) than the basic UNet implementation.

TorchIO, MONAI, PyTorch Lightning

This notebook demonstrates how the three libraries from the official PyTorch Ecosystem can be used together to segment the hippocampus on brain MRIs from the Medical Segmentation Decathlon.

varautoencoder_mednist

This tutorial uses the MedNIST scan (or alternatively the MNIST) dataset to demonstrate MONAI's variational autoencoder class.

interpretability

Tutorials in this folder demonstrate model visualisation and interpretability features of MONAI. Currently, it consists of class activation mapping and occlusion sensitivity for 3D classification model visualisations and analysis.

Transform visualization

This tutorial shows several visualization approaches for 3D image during transform augmentation.

Auto3DSeg

This folder shows how to run the comprehensive Auto3DSeg pipeline with minimal inputs and customize the Auto3Dseg modules to meet different user requirements.

<ins>Self-Supervised Learning</ins>

self_supervised_pretraining

This tutorial shows how to construct a training workflow of self-supervised learning where unlabeled data is utilized. The tutorial shows how to train a model on TCIA dataset of unlabeled Covid-19 cases.

self_supervised_pretraining_based_finetuning

This tutorial shows how to utilize pre-trained weights from the self-supervised learning framework where unlabeled data is utilized. This tutorial shows how to train a model of multi-class 3D segmentation using pretrained weights.

Generative Model

3D latent diffusion model

This tutorial shows the use cases of training and validating a 3D Latent Diffusion Model.

2D latent diffusion model

This tutorial shows the use cases of training and validating a 2D Latent Diffusion Model.

Brats 3D latent diffusion model

Example shows the use cases of training and validating a 3D Latent Diffusion Model on Brats 2016&2017 data, expanding on the above notebook.

MAISI 3D latent diffusion model

Example shows the use cases of training and validating Nvidia MAISI (Medical AI for Synthetic Imaging) model, a 3D Latent Diffusion Model that can generate large CT images with paired segmentation masks, variable volume size and voxel size, as well as controllable organ/tumor size.

SPADE in VAE-GAN for Semantic Image Synthesis on 2D BraTS Data

Example shows the use cases of applying SPADE, a VAE-GAN-based neural network for semantic image synthesis, to a subset of BraTS that was registered to MNI space and resampled to 2mm isotropic space, with segmentations obtained using Geodesic Information Flows (GIF).

Applying Latent Diffusion Models to 2D BraTS Data for Semantic Image Synthesis

Example shows the use cases of applying SPADE normalization to a latent diffusion model, following the methodology by Wang et al., for semantic image synthesis on a subset of BraTS registered to MNI space and resampled to 2mm isotropic space, with segmentations obtained using Geodesic Information Flows (GIF).

Diffusion Models for Implicit Image Segmentation Ensembles

Example shows the use cases of how to use MONAI for 2D segmentation of images using DDPMs. The same structure can also be used for conditional image generation, or image-to-image translation.

Evaluate Realism and Diversity of the generated images

Example shows the use cases of using MONAI to evaluate the performance of a generative model by computing metrics such as Frechet Inception Distance (FID) and Maximum Mean Discrepancy (MMD) for assessing realism, as well as MS-SSIM and SSIM for evaluating image diversity.

VISTA2D

This tutorial demonstrates how to train a cell segmentation model using the MONAI framework and the Segment Anything Model (SAM) on the Cellpose dataset.