Home

Awesome

Motion Correction and Volumetric Image Reconstruction of 2D Ultra-fast MRI

NiftyMIC is a Python-based open-source toolkit for research developed within the GIFT-Surg project to reconstruct an isotropic, high-resolution volume from multiple, possibly motion-corrupted, stacks of low-resolution 2D slices. The framework relies on slice-to-volume registration algorithms for motion correction and reconstruction-based Super-Resolution (SR) techniques for the volumetric reconstruction.

The algorithm and software were developed by Michael Ebner at the Wellcome/EPSRC Centre for Interventional and Surgical Sciences, University College London (UCL) (2015 -- 2019), and the Department of Surgical and Interventional Sciences, King's College London (KCL) (since 2019).

A detailed description of the NiftyMIC algorithm is found in EbnerWang2020:

A later extension to this paper for fetal brain automatic segmentation building on MONAI was provided by Marta B.M. Ranzini. More information about MONAIfbs can be found here.

An extension of NiftyMIC for fetal functional MRI was developed in collaboration with members of the Computational Image Research Lab, Department of Biomedical Imaging and Image-guided Therapy at the Medical University of Vienna. A detailed description of the NiftyMIC algorithm for fetal fMRI is found in Sobotka2022:

If you have any questions or comments, please drop an email to michael.ebner@kcl.ac.uk.

NiftyMIC applied to Fetal Brain MRI

Given a set of low-resolution, possibly motion-corrupted, stacks of 2D slices, NiftyMIC produces an isotropic, high-resolution 3D volume. As an example, we illustrate its use for fetal MRI by computing a high-resolution visualization of the brain for a neck mass subject. Standard clinical HASTE sequences were used to acquire the low-resolution images in multiple orientations. The associated brain masks for motion correction can be obtained with the included automated segmentation tool MONAIfbs (a legacy method for automated segmentation can also be found in the fetal_brain_seg package).
Full working examples on automated segmentation and high-resolution reconstruction of fetal brain MRI using NiftyMIC is described in the Usage section below.

<p align="center"> <img src="./data/demo/NiftyMIC_Algorithm.png" align="center" width="700"> </p> <p align="center">Figure 1. NiftyMIC -- a volumetric MRI reconstruction tool based on rigid slice-to-volume registration and outlier-robust super-resolution reconstruction steps -- applied to fetal brain MRI.<p align="center"> <p align="center"> <img src="./data/demo/NiftyMIC_VolumetricReconstructionOutput.png" align="center" width="1000"> </p> <p align="center">Figure 2. Qualitative comparison of the original low-resolution input data and the obtained high-resolution volumetric reconstructions in both the original patient-specific and standard anatomical orientations. Five input stacks (two axial, one coronal and two sagittal) were used. <p align="center">

Algorithm

Several methods have been implemented to solve the Robust Super-Resolution Reconstruction (SRR) problem

<!-- ```math \vec{x}^* := \text{argmin}_{\vec{x}\ge 0} \Big[\sum_{k\in\mathcal{K}_\sigma} \sum_{i=1}^{N_k} \varrho\big( (A_k\vec{x} - \vec{y}_k)_i^2 \big) + \alpha\,\text{Reg}(\vec{x}) \Big] ``` --> <p align="center"> <img src="http://latex.codecogs.com/svg.latex?\vec{x}^*&space;:=&space;\text{argmin}_{\vec{x}\ge&space;0}&space;\Big[\sum_{k\in\mathcal{K}_\sigma}&space;\sum_{i=1}^{N_k}&space;\varrho\big(&space;(A_k\vec{x}&space;-&space;\vec{y}_k)_i^2&space;\big)&space;&plus;&space;\alpha\,\text{Reg}(\vec{x})&space;\Big]" title="http://latex.codecogs.com/svg.latex?\vec{x}^* := \text{argmin}_{\vec{x}\ge 0} \Big[\sum_{k\in\mathcal{K}_\sigma} \sum_{i=1}^{N_k} \varrho\big( (A_k\vec{x} - \vec{y}_k)_i^2 \big) + \alpha\,\text{Reg}(\vec{x}) \Big]" /> </p> <!--to obtain the (vectorized) high-resolution 3D MRI volume $`\vec{x}\in\mathbb{R}^N`$ from multiple, possibly motion corrupted, low-resolution stacks of (vectorized) 2D MR slices $`\vec{y}_k \in\mathbb{R}^{N_k}`$ with $`N_k\ll N`$ for $`k=1,...,\,K`$--> <!--for a variety of regularizers $`\text{Reg}`$ and data loss functions $`\varrho`$.--> <!--The linear operator $`A_k := D_k B_k W_k`$ represents the comd operator descri the (rigid) motion $`W_k`$, the blurring operator $`B_k`$ and the downsampling operator $`D_k`$.-->

to obtain the (vectorized) high-resolution 3D MRI volume img from multiple, possibly motion corrupted, low-resolution stacks of (vectorized) 2D MR slices img with img for img for a variety of regularizers img and data loss functions img. The linear operator img represents the combined operator describing the (rigid) motion img, the blurring operator img and the downsampling operator img.

The toolkit relies on an iterative motion-correction/reconstruction approach whereby complete outlier rejection of misregistered slices is achieved by iteratively solving the SRR problem for a slice-index set img containing only slices that are in high agreement with their simulated counterparts projected from a previous high-resolution iterate img according to a similarity measure img and parameter img. In the current implementation, the similarity measure img corresponds to Normalized Cross Correlation (NCC).

The provided data loss functions img are motivated by SciPy and allow for additional robust outlier handling during the SRR step. Implemented data loss functions are:

<!--$`\varrho(e)=e`$--> <!--$`\varrho(e)=2(\sqrt{1+e}-1)`$ --> <!--$`\varrho(e)=|e|_\gamma=\begin{cases}e,&e<\gamma^2\\2\gamma\sqrt{e}-\gamma^2,&e\ge\gamma^2\end{cases}`$--> <!--$`\varrho(e)=\arctan(e)`$--> <!--$`\varrho(e)=\ln(1 + e)`$-->

The available regularizers include

<!-- * Zeroth-order Tikhonov (TK0): $`\text{Reg}(\vec{x}) = \frac{1}{2}\Vert \vec{x} \Vert_{\ell^2}^2`$ --> <!-- * First-order Tikhonov (TK1): $`\text{Reg}(\vec{x}) = \frac{1}{2}\Vert \nabla \vec{x} \Vert_{\ell^2}^2`$ --> <!-- * Isotropic Total Variation (TV): $`\text{Reg}(\vec{x}) = \text{TV}_\text{iso}(\vec{x}) = \big\Vert |\nabla \vec{x}| \big\Vert_{\ell^1}`$ --> <!-- * Huber Function: $`\text{Reg}(\vec{x}) = \frac{1}{2\gamma} \big| |\nabla \vec{x}| \big|_{\gamma}`$ -->

Additionally, the choice of finding optimal reconstruction parameters is facilitated by the Numerical Solver Library (NSoL).

Disclaimer

NiftyMIC supports medical image registration and volumetric reconstruction for ultra-fast 2D MRI. NiftyMIC is not intended for clinical use.

How to cite

If you use this software in your work for structural MRI reconstruction, please cite

If you use this software in your work for functional MRI reconstruction, please cite

Installation

From Source

NiftyMIC was developed in Ubuntu 16.04, 18.04 and Mac OS X 10.12 and tested for Python 2.7, 3.5 and 3.6. It builds on a couple of additional libraries developed within the GIFT-Surg project including

whose installation requirements need to be met. Therefore, the local installation comes in three steps:

  1. Installation of ITK_NiftyMIC
  2. Installation of SimpleReg dependencies
  3. Installation of NiftyMIC (including optional MONAIfbs)

Note: The optional use of the automated fetal brain segmentation tool MONAIfbs requires Python version >= 3.6.

Virtual Machine and Docker

To avoid manual installation from source, NiftyMIC is also available as a virtual machine and Docker image.

Usage

Provided the input MR image data in NIfTI format (nii or nii.gz), NiftyMIC can reconstruct an isotropic, high-resolution volume from multiple, possibly motion-corrupted, stacks of low-resolution 2D slices.

A recommended workflow is [associated applications in square brackets]:

  1. Segmentation of the anatomy of interest for all input images. For fetal brain MRI reconstructions, we recommend the use of the fully automatic segmentation tool MONAIfbs already integrated in NiftyMIC [niftymic_segment_fetal_brains].
  2. Bias-field correction [niftymic_correct_bias_field]
  3. Volumetric reconstruction in subject space using two-step iterative approach based on rigid slice-to-volume registration and SRR cycles [niftymic_reconstruct_volume]

In case reconstruction in a template space is desired (like for fetal MRI) additional steps could be:

  1. Register obtained SRR to template and update respective slice motion corrections [niftymic_register_image]
  2. Volumetric reconstruction in template space [niftymic_reconstruct_volume_from_slices]

Additional information on how to use NiftyMIC and its applications is provided in the following.

Volumetric MR Reconstruction from Motion Corrupted 2D Slices

Leveraging a two-step registration-reconstruction approach an isotropic, high-resolution 3D volume can be generated from multiple stacks of low-resolution slices.

An example for a basic usage reads

niftymic_reconstruct_volume \
--filenames path-to-stack-1.nii.gz ... path-to-stack-N.nii.gz \
--filenames-masks path-to-stack-1_mask.nii.gz ... path-to-stack-N_mask.nii.gz \
--output path-to-srr.nii.gz \

whereby complete outlier removal during SRR is activated by default (--outlier-rejection 1).

A more elaborate example could be

niftymic_reconstruct_volume \
--filenames path-to-stack-1.nii.gz ... path-to-stack-N.nii.gz \
--filenames-masks path-to-stack-1_mask.nii.gz ... path-to-stack-N_mask.nii.gz \
--alpha 0.01 \
--outlier-rejection 1 \
--threshold-first 0.5 \
--threshold 0.85 \
--intensity-correction 1 \
--isotropic-resolution 0.8 \
--two-step-cycles 3 \
--output path-to-output-dir/srr.nii.gz \
--subfolder-motion-correction motion_correction \ # created in 'path-to-output-dir'
--verbose 1

The obtained motion-correction transformations in motion_correction can be used for further processing, e.g. by using niftymic_reconstruct_volume_from_slices.py to solve the SRR problem for a variety of different regularization and data loss function types.

Transformation to Template Space

If a template is available, it is possible to obtain a SRR in its associated standard anatomical space. Using the subject-space SRR outcome of niftymic_reconstruct_volume a rigid alignment step maps all slice motion correction transformations accordingly using

niftymic_register_image \
--fixed path-to-template.nii.gz \
--fixed-mask path-to-template_mask.nii.gz \
--moving path-to-subject-space-srr.nii.gz \
--moving-mask path-to-subject-space-srr_mask.nii.gz \
--dir-input-mc dir-to-motion_correction \
--output path-to-registration-transform.txt

For fetal brain template space alignment, a spatio-temporal atlas is provided in data/templates. If you make use of it, please cite

and abide by the license agreement as described in data/templates/LICENSE.

SRR Methods for Motion Corrected (or Static) Data

After performed/updated motion correction (or having static data in the first place) several options are available:

SRR from Motion Corrected (or Static) Slice Acquisitions

Solve the SRR problem for motion corrected data (or static data if --dir-input-mc is omitted):

niftymic_reconstruct_volume_from_slices \
--filenames path-to-stack-1.nii.gz ... path-to-stack-N.nii.gz \
--filenames-masks path-to-stack-1_mask.nii.gz ... path-to-stack-N_mask.nii.gz \
--dir-input-mc dir-to-motion_correction \ # optional
--output path-to-srr.nii.gz \
--reconstruction-type TK1L2 \
--reconstruction-space path-to-template.nii.gz \ # optional
--alpha 0.01

niftymic_reconstruct_volume_from_slices \
--filenames path-to-stack-1.nii.gz ... path-to-stack-N.nii.gz \
--dir-input-mc dir-to-motion_correction \
--output path-to-srr.nii.gz \
--reconstruction-type HuberL2 \
--alpha 0.003

Slices that were rejected during the niftymic_reconstruct_volume run are recognized as outliers based on the content of dir-input-mc and will not be incorporated during the volumetric reconstruction.

Parameter Studies to Determine Optimal SRR Parameters

The optimal choice for reconstruction parameters like the regularization parameter or data loss function can be found by running parameter studies. This includes L-curve studies and direct comparison against a reference volume for various cost functions. In case a reference is available, similarity measures are evaluated against this "ground-truth" as well.

Example are:

niftymic_run_reconstruction_parameter_study \
--filenames path-to-stack-1.nii.gz ... path-to-stack-N.nii.gz \
--filenames-masks path-to-stack-1_mask.nii.gz ... path-to-stack-N_mask.nii.gz \
--dir-input-mc dir-to-motion_correction \
--dir-output dir-to-param-study-output \
--reconstruction-type TK1L2 \
--reconstruction-space path-to-reconstruction-space.nii.gz \ # define reconstruction space
--alphas 0.005 0.01 0.02 0.05 0.1 # regularization parameters to sweep through
--append # if given, append a previously performed parameter study in output directory (if available)

niftymic_run_reconstruction_parameter_study \
--filenames path-to-stack-1.nii.gz ... path-to-stack-N.nii.gz \
--filenames-masks path-to-stack-1_mask.nii.gz ... path-to-stack-N_mask.nii.gz \
--dir-input-mc dir-to-motion_correction \
--dir-output dir-to-param-study-output \
--reconstruction-type HuberL2 \
--reference path-to-reference-volume.nii.gz \ # in case reference ("ground-truth") is available (reconstruction space is defined by this reference)
--measures MAE RMSE PSNR NCC NMI SSIM \ # evaluate reconstruction similarities against reference
--reference-mask path-to-reference-volume_mask.nii.gz \ # if given, evaluate similarities (--measures) on masked region only
--alphas 0.001 0.003 0.005 0.001 0.003 \ # regularization parameters to sweep through
--append # if given, append a previously performed parameter study in output directory (if available)

The results can be assessed by accessing the NSoL-script show_parameter_study.py via

niftymic_show_parameter_study \
--dir-input dir-to-param-study-output \
--study-name TK1L2 \
--dir-output-figures dir-to-figures

Automatic Segmentation and High-Resolution Reconstruction of Fetal Brain MRI

An automated framework is implemented to obtain a high-resolution fetal brain MRI reconstruction in the standard anatomical planes (Figure 2). This includes two main subsequent blocks:

  1. Automatic segmentation to generate fetal brain masks
  2. Automatic high-resolution reconstruction.

The latter is based on the work described in EbnerWang2018 and EbnerWang2020. Compared to the segmentation approach proposed in EbnerWang2020, a new automated segmentation tool has been included in the NiftyMIC package, called MONAIfbs. This implements a single-step segmentation approach based on the dynUNet implemented in MONAI.

The current NiftyMIC version is still compatible with the older segmentation pipeline based on the fetal_brain_seg package and presented in EbnerWang2020. Details on its use are available at this wiki page, although MONAIfbs is the recommended option.

Automatic segmentation

Provided the dependencies for MONAIfbs are installed, create the automatic fetal brain masks of HASTE-like images:

niftymic_segment_fetal_brains \
--filenames \
nifti/name-of-stack-1.nii.gz \
nifti/name-of-stack-2.nii.gz \
nifti/name-of-stack-N.nii.gz \
--filenames-masks \
seg/name-of-stack-1.nii.gz \
seg/name-of-stack-2.nii.gz \
seg/name-of-stack-N.nii.gz

Automatic reconstruction

Afterwards, four consecutive steps including

  1. bias field correction (niftymic_correct_bias_field),
  2. subject-space reconstruction (niftymic_reconstruct_volume),
  3. template-space alignment (niftymic_register_image), and
  4. template-space reconstruction (niftymic_reconstruct_volume_from_slices)

are performed to create a high-resolution fetal brain MRI reconstruction in the standard anatomical planes:

niftymic_run_reconstruction_pipeline \
--filenames \
nifti/name-of-stack-1.nii.gz \
nifti/name-of-stack-2.nii.gz \
nifti/name-of-stack-N.nii.gz \
--filenames-masks \
seg/name-of-stack-1.nii.gz \
seg/name-of-stack-2.nii.gz \
seg/name-of-stack-N.nii.gz \
--dir-output srr

Additional parameters such as the regularization parameter alpha can be specified too. For more information please execute niftymic_run_reconstruction_pipeline -h.

Note: In case a suffix distinguishes image segmentation (--filenames-masks) from the associated image filenames (--filenames), the argument --suffix-mask needs to be provided for reconstructing the HR brain volume mask as part of the pipeline. E.g. if images name-of-stack-i.nii.gz are associated with the mask name-of-stack-i_mask.nii.gz, then the additional argument --suffix-mask _mask needs to be specified.

NiftyMIC applied to Fetal Brain Functional MRI

An extension of NiftyMIC for fetal functional MRI is described in Sobotka2022:

<p align="center"> <img src="./data/demo/NiftyMIC_rsfmri_Algorithm.png" align="center" width="700"> </p> <p align="center"> Figure 3: Overview of the proposed motion correction and volumetric reconstruction algorithm for rs-fMRI. With the first n=15 (default) time points a HR reference volume with outlier rejection is estimated (Ebner, Wang et al., 2020). Afterwards each slice of a time point is registered to the HR reference volume following which individual time points are reconstructed using Huber L2 regularization.<p align="center"> <p align="center"> <img src="./data/demo/NiftyMIC_rsfmri_Reconstruction.png" align="center" width="700"> </p> <p align="center"> Figure 4: Qualitative comparison of the original low-resolution input sequence (top row) and the obtained volumetric reconstruction of the functional MRI (bottom row) in patient-specific orientation. A sequence of axial stacks were used as input.<p align="center">

A recommended workflow is [associated applications in square brackets]:

  1. Volumetric segmentation of the fetal brain for all input images in the sequence. We tested the algorithm with fetal functional brains masks created using the Brain Extraction Tool incorporated in the FSL Toolbox (Smith et al. 2002).

  2. Motion estimation [rsfmri_estimate_motion]

     rsfmri_estimate_motion \
 --filename path-to-bold.nii.gz  \
 --filename-mask path-to-bold_mask.nii.gz \
 --dir-output path-to-dir-output

  3. Functional volumetric reconstruction [rsfmri_reconstruct_volume_from_slices]

    An example for a basic usage reads

    rsfmri_reconstruct_volume_from_slices \
--filename path-to-bold.nii.gz  \
--filename-mask path-to-bold_mask.nii.gz \
--dir-input-mc path-to-motion_correction \ # created in `dir-output` in step 2 above
--output path-to-bold_reconstructed.nii.gz

    A more elaborate example using a different reconstruction approach HuberL2 could be

    rsfmri_reconstruct_volume_from_slices \
--filename path-to-bold.nii.gz \
--filename-mask path-to-bold_mask.nii.gz \
--dir-input-mc path-to-motion_correction \ # created in `dir-output` in step 2 above
--output path-to-bold_reconstructed.nii.gz \
--reconstruction-type HuberL2
--alpha 0.1 \
--iter-max 20

If you have any questions or comments with regards to the use of NiftyMIC for functional MRI, please drop an email to daniel.sobotka@meduniwien.ac.at.

Licensing and Copyright

Copyright (c) 2022 Michael Ebner and contributors. This framework is made available as free open-source software under the BSD-3-Clause License. Other licenses may apply for dependencies.

Funding

This work is partially funded by the UCL Engineering and Physical Sciences Research Council (EPSRC) Centre for Doctoral Training in Medical Imaging (EP/L016478/1), the Innovative Engineering for Health award (Wellcome Trust [WT101957] and EPSRC [NS/A000027/1]), and supported by researchers at the National Institute for Health Research University College London Hospitals (UCLH) Biomedical Research Centre.

References

Selected publications associated with NiftyMIC are: