Home

Awesome

Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review

Awesome License: MIT

:fire::fire:This is a collection of awesome articles about Transformer models in medical imaging :fire::fire:

:loudspeaker: Our review paper published on MedIA: Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review :heart:

:loudspeaker: Our review paper published on arXiv: Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review :heart:

Citation

@article{azad2023advances,
  title={Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review},
  author={Azad, Reza and Kazerouni, Amirhossein and Heidari, Moein and Aghdam, Ehsan Khodapanah and Molaei, Amirali and Jia, Yiwei and Jose, Abin and Roy, Rijo and Merhof, Dorit},
  journal={Medical Image Analysis},
  volume = {91},
  pages={103000},
  year={2024},
  issn = {1361-8415},
  publisher={Elsevier}
}

Contents

Taxonomy

Transformers

Papers

Image Classification

HATNet: An End-to-End Holistic Attention Network for Diagnosis of Breast Biopsy Images <br> Sachin Mehta, Ximing Lu, Donald Weaver, Joann G. Elmore, Hannaneh Hajishirzi, Linda Shapiro<br> [25th Jul, 2020] [MedIA Journal, 2022]
[PDF] [GitHub]

A graph-transformer for whole slide image classification <br> Yi Zheng, Rushin H. Gindra, Emily J. Green, Eric J. Burks, Margrit Betke, Jennifer E. Beane, Vijaya B. Kolachalama<br> [19th May, 2022] [TMI Journal, 2022]
[PDF] [GitHub]

RadioTransformer: A Cascaded Global-Focal Transformer for Visual Attention-guided Disease Classification <br> Moinak Bhattacharya, Shubham Jain, Prateek Prasanna<br> [23rd Feb., 2022] [ECCV, 2022]
[PDF]

Federated Split Vision Transformer for COVID-19 CXR Diagnosis using Task-Agnostic Training <br> Sangjoon Park, Gwanghyun Kim, Jeongsol Kim, Boah Kim, Jong Chul Ye<br> [2nd Nov., 2021] [NeurIPS, 2021]
[PDF]

Vision transformer for classification of breast ultrasound images <br> Behnaz Gheflati, Hassan Rivaz<br> [27th Oct., 2021] [EMBC, 2022]
[PDF]

MIL-VT: Multiple Instance Learning Enhanced Vision Transformer for Fundus Image Classification <br> Shuang Yu, Kai Ma, Qi Bi, Cheng Bian, Munan Ning, Nanjun He, Yuexiang Li, Hanruo Liu, Yefeng Zheng<br> [21st Sep., 2021] [MICCAI, 2021]
[PDF] [GitHub]

3DMeT: 3D Medical Image Transformer for Knee Cartilage Defect Assessment <br> Sheng Wang, Zixu Zhuang, Kai Xuan, Dahong Qian, Zhong Xue, Jia Xu, Ying Liu, Yiming Chai, Lichi Zhang, Qian Wang, Dinggang Shen<br> [21st Sep., 2021] [MICCAI Workshop, 2021]
[PDF]

COVID-Transformer: Interpretable COVID-19 Detection Using Vision Transformer for Healthcare <br> Debaditya Shome, T. Kar, Sachi Nandan Mohanty, Prayag Tiwari, Khan Muhammad, Abdullah AlTameem, Yazhou Zhang, Abdul Khader Jilani Saudagar<br> [23rd Sep., 2021] [International Journal of Environmental Research and Public Health, 2021]
[PDF] [GitHub]

Is it Time to Replace CNNs with Transformers for Medical Images? <br> Christos Matsoukas, Johan Fredin Haslum, Magnus Söderberg, Kevin Smith<br> [20th Aug., 2021] [ICCV Workshop, 2021]
[PDF] [GitHub]

Vision Transformer for femur fracture classification <br> Leonardo Tanzi, Andrea Audisio, Giansalvo Cirrincione, Alessandro Aprato, Enrico Vezzetti<br> [7th Aug., 2021] [Injury Journal, 2022]
[PDF]

xViTCOS: Explainable Vision Transformer Based COVID-19 Screening Using Radiography <br> Arnab Kumar Mondal, Arnab Bhattacharjee, Parag Singla, A. P. Prathosh<br> [7th Jul., 2021] [IEEE Journal of Translational Engineering in Health and Medicine, 2021]
[PDF] [Github]

COVID-VIT: Classification of COVID-19 from CT chest images based on vision transformer models <br> Xiaohong Gao, Yu Qian, Alice Gao<br> [4th Jul., 2021] [NextComp, 2022]
[PDF] [GitHub]

TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification <br> Zhuchen Shao, Hao Bian, Yang Chen, Yifeng Wang, Jian Zhang, Xiangyang Ji, Yongbing Zhang<br> [2nd Jun., 2021] [NeurIPS, 2021]
[PDF] [GitHub]

Lesion-Aware Transformers for Diabetic Retinopathy Grading <br> Rui Sun, Yihao Li, Tianzhu Zhang, Zhendong Mao, Feng Wu, Yongdong Zhang<br> [1st Jun., 2021] [CVPR, 2021]
[PDF]

POCFormer: A Lightweight Transformer Architecture for Detection of COVID-19 Using Point of Care Ultrasound <br> Shehan Perera, Srikar Adhikari, Alper Yilmaz<br> [20th May, 2021] [ICIP, 2022]
[PDF]

Automatic diagnosis of covid-19 using a tailored transformer-like network <br> Chengeng Liu, Qingshan Yin<br> [21st Apr., 2021] [CISAT, 2021]
[PDF]

Vision Transformer for COVID-19 CXR Diagnosis using Chest X-ray Feature Corpus <br> Sangjoon Park, Gwanghyun Kim, Yujin Oh, Joon Beom Seo, Sang Min Lee, Jin Hwan Kim, Sungjun Moon, Jae-Kwang Lim, Jong Chul Ye<br> [12th Mar., 2021] [arXiv, 2021]
[PDF]

TransMed: Transformers Advance Multi-modal Medical Image Classification <br> Yin Dai, Yifan Gao<br> [10th Mar., 2021] [Diagnostics, 2021]
[PDF]


Image Segmentation

TransDeepLab: Convolution-Free Transformer-based DeepLab v3+ for Medical Image Segmentation <br> Reza Azad, Moein Heidari, Moein Shariatnia, Ehsan Khodapanah Aghdam, Sanaz Karimijafarbigloo, Ehsan Adeli, Dorit Merhof<br> [1st Aug., 2022] [MICCAI Workshop, 2022]
[PDF] [GitHub]

HiFormer: Hierarchical Multi-scale Representations Using Transformers for Medical Image Segmentation<br> Moein Heidari, Amirhossein Kazerouni, Milad Soltany, Reza Azad, Ehsan Khodapanah, Aghdam Julien Cohen-Adad, Dorit Merhof<br> [18th Jul., 2022] [WACV, 2023]
[PDF] [GitHub]

Self Pre-training with Masked Autoencoders for Medical Image Analysis<br> Lei Zhou, Huidong Liu, Joseph Bae, Junjun He, Dimitris Samaras, Prateek Prasanna<br> [10th Mar., 2022] [arXiv, 2022]
[PDF]

Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images<br> Ali Hatamizadeh, Vishwesh Nath, Yucheng Tang, Dong Yang, Holger Roth, Daguang Xu <br> [4th Jan., 2022] [MICCAI Workshop]
[PDF] [GitHub]

Semi-Supervised Medical Image Segmentation via Cross Teaching between CNN and Transformer<br> Xiangde Luo, Minhao Hu, Tao Song, Guotai Wang, Shaoting Zhang<br> [9th Dec., 2021] [MIDL, 2022]
[PDF] [Github]

T-AutoML: Automated Machine Learning for Lesion Segmentation using Transformers in 3D Medical Imaging<br> Dong Yang, Andriy Myronenko, Xiaosong Wang, Ziyue Xu, Holger R. Roth, Daguang Xu<br> [15th Nov., 2021] [ICCV, 2021]
[PDF]

MISSFormer: An Effective Medical Image Segmentation Transformer<br> Xiaohong Huang, Zhifang Deng, Dandan Li, Xueguang Yuan<br> [15th Sep., 2021] [TMI Journal, 2022]
[PDF] [GitHub]

nnFormer: Interleaved Transformer for Volumetric Segmentation<br> Hong-Yu Zhou, Jiansen Guo, Yinghao Zhang, Lequan Yu, Liansheng Wang, Yizhou Yu<br> [7th Sep., 2021] [arXiv, 2021]
[PDF] [GitHub]

Medical Image Segmentation Using Squeeze-and-Expansion Transformers<br> Shaohua Li, Xiuchao Sui, Xiangde Luo, Xinxing Xu, Yong Liu, Rick Goh<br> [20th May, 2021] [IJCAI, 2021]
[PDF] [GitHub]

Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation<br> Hu Cao, Yueyue Wang, Joy Chen, Dongsheng Jiang, Xiaopeng Zhang, Qi Tian, Manning Wang <br> [12th May, 2021] [arXiv, 2021]
[PDF] [GitHub]

UNETR: Transformers for 3D Medical Image Segmentation<br> Ali Hatamizadeh, Yucheng Tang, Vishwesh Nath, Dong Yang, Andriy Myronenko, Bennett Landman, Holger Roth, Daguang Xu<br> [18th Mar., 2021] [WACV, 2022]
[PDF] [GitHub]

TransBTS: Multimodal Brain Tumor Segmentation Using Transformer<br> Jiangyun Li, Wenxuan Wang, Chen Chen, Tianxiang Zhang, Sen Zha, Hong Yu, Jing Wang<br> [7th Mar, 2021] [MICCAI, 2021]
[PDF] [GitHub]

CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation <br> Yutong Xie, Jianpeng Zhang, Chunhua Shen, Yong Xia <br> [4th Mar., 2021] [MICCAI, 2021]
[PDF] [GitHub]

Medical Transformer: Gated Axial-Attention for Medical Image Segmentation<br> Jeya Maria Jose Valanarasu, Poojan Oza, Ilker Hacihaliloglu, Vishal M. Patel<br> [21th Feb., 2021] [MICCAI, 2021]
[PDF] [GitHub]

TransFuse: Fusing Transformers and CNNs for Medical Image Segmentation<br> Yundong Zhang, Huiye Liu, Qiang Hu<br> [16th Feb., 2021] [arXiv, 2021]
[PDF] [GitHub]

TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation <br> Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Le Lu, Alan L. Yuille, Yuyin Zhou<br> [8th Feb., 2021] [arXiv, 2021]
[PDF] [GitHub]


Image Reconstruction

Transformer-based Dual-domain Network for Few-view Dedicated Cardiac SPECT Image Reconstructions <br> Huidong Xie, Bo Zhou, Xiongchao Chen, Xueqi Guo, Stephanie Thorn, Yi-Hwa Liu, Ge Wang, Albert Sinusas, Chi Liu<br> [18th July., 2023] [MICCAI, 2023]
[PDF]

TransCT: Dual-path Transformer for Low Dose Computed Tomography <br> Zhicheng Zhang, Lequan Yu, Xiaokun Liang, Wei Zhao, Lei Xing<br> [28th Feb., 2021] [MICCAI, 2021]
[PDF] [GitHub]

TED-net: Convolution-free T2T Vision Transformer-based Encoder-decoder Dilation network for Low-dose CT Denoising <br> Dayang Wang, Zhan Wu, Hengyong Yu<br> [8th Jun., 2021] [MICCAI Workshop, 2021]
[PDF] [GitHub]

Eformer: Edge Enhancement based Transformer for Medical Image Denoising <br> Achleshwar Luthra, Harsh Sulakhe, Tanish Mittal, Abhishek Iyer, Santosh Yadav<br> [16th Sep., 2021] [arXiv, 2021]
[PDF]

3D Transformer-GAN for High-Quality PET Reconstruction <br> Yanmei Luo, Yan Wang, Chen Zu, Bo Zhan, Xi Wu, Jiliu Zhou, Dinggang Shen, Luping Zhou<br> [21st Sep., 2021] [MICCAI, 2021]
[PDF]

Spatial Adaptive and Transformer Fusion Network (STFNet) for Low-count PET Blind Denoising with MRI <br> Lipei Zhang, Zizheng Xiao, Chao Zhou, Jianmin Yuan, Qiang He, Yongfeng Yang, Xin Liu, Dong Liang, Hairong Zheng, Wei Fan, Xu Zhang, Zhanli Hu<br> [19th Nov., 2021] [Medical Physics, 2021]
[PDF]

CTformer: Convolution-free Token2Token Dilated Vision Transformer for Low-dose CT Denoising <br> Dayang Wang, Fenglei Fan, Zhan Wu, Rui Liu, Fei Wang, Hengyong Yu<br> [28th Feb., 2022] [arXiv, 2022]
[PDF] [GitHub]

Low-Dose CT Denoising via Sinogram Inner-Structure Transformer <br> Liutao Yang, Zhongnian Li, Rongjun Ge, Junyong Zhao, Haipeng Si, Daoqiang Zhang<br> [7th Apr., 2022] [IEEE Transactions on Medical Imaging, 2022]
[PDF]

DuDoTrans: Dual-Domain Transformer Provides More Attention for Sinogram Restoration in Sparse-View CT Reconstruction <br> Ce Wang, Kun Shang, Haimiao Zhang, Qian Li, Yuan Hui, S. Kevin Zhou<br> [21th Nov., 2021] [arXiv, 2021]
[PDF] [GitHub]

Fourier Image Transformer <br> Tim-Oliver Buchholz, Florian Jug<br> [6th Apr., 2021] [CVPR, 2022]
[PDF] [GitHub]

Dual-domain sparse-view CT reconstruction with Transformers <br> Changrong Shi, Yongshun Xiao, Zhiqiang Chen<br> [22nd Mar., 2022] [ELSEVIER Physica Medica, 2022]
[PDF]

Adaptively Re-weighting Multi-Loss Untrained Transformer for Sparse-View Cone-Beam CT Reconstruction <br> Minghui Wu, Yangdi Xu, Yingying Xu, Guangwei Wu, Qingqing Chen, Hongxiang Lin<br> [23th Mar., 2022] [arXiv, 2022]
[PDF]

Vision Transformers Enable Fast and Robust Accelerated MRI <br> Chun-Mei Feng, Yunlu Yan, Huazhu Fu, Li Chen, Yong Xu<br> [10th Dec., 2021] [MIDL, 2022]
[PDF] [GitHub]

Task Transformer Network for Joint MRI Reconstruction and Super-Resolution <br> Chun-Mei Feng, Yunlu Yan, Huazhu Fu, Li Chen, Yong Xu<br> [12th Jun., 2021] [MICCAI, 2021]
[PDF] [GitHub]

MR Image Super Resolution By Combining Feature Disentanglement CNNs and Vision Transformers <br> Chun-Mei Feng, Yunlu Yan, Huazhu Fu, Li Chen, Yong Xu<br> [9th Dec., 2021] [MIDL, 2022]
[PDF]

Cross-Modality High-Frequency Transformer for MR Image Super-Resolution <br> Chaowei Fang, Dingwen Zhang, Liang Wang, Yulun Zhang, Lechao Cheng, Junwei Han<br> [29th Mar., 2022] [ACM MM, 2022]
[PDF]


Image Synthesizing

One Model to Synthesize Them All: Multi-contrast Multi-scale Transformer for Missing Data Imputation <br> Jiang Liu, Srivathsa Pasumarthi, Ben Duffy, Enhao Gong, Greg Zaharchuk, Keshav Datta<br> [28th Apr., 2022] [arXiv, 2021]
[PDF]

CyTran: Cycle-Consistent Transformers for Non-Contrast to Contrast CT Translation <br> Nicolae-Catalin Ristea, Andreea-Iuliana Miron, Olivian Savencu, Mariana-Iuliana Georgescu, Nicolae Verga, Fahad Shahbaz Khan, Radu Tudor Ionescu<br> [12th Oct., 2021] [arXiv, 2021]
[PDF] [GitHub]

ResViT: Residual vision transformers for multi-modal medical image synthesis <br> Onat Dalmaz, Mahmut Yurt, Tolga Çukur<br> [30th Jun., 2021] [TMI Journal, 2021]
[PDF] [GitHub]

PTNet: A High-Resolution Infant MRI Synthesizer Based on Transformer <br> Xuzhe Zhang, Xinzi He, Jia Guo, Nabil Ettehadi, Natalie Aw, David Semanek, Jonathan Posner, Andrew Laine, Yun Wang<br> [28th May., 2021] [arXiv, 2021]
[PDF] [GitHub]

VTGAN: Semi-supervised Retinal Image Synthesis and Disease Prediction using Vision Transformers <br> Sharif Amit Kamran, Khondker Fariha Hossain, Alireza Tavakkoli, Stewart Lee Zuckerbrod, Salah A. Baker<br> [14th Apr., 2021] [ICCV Workshop, 2021]
[PDF] [GitHub]


Object Detection

Focused Decoding Enables 3D Anatomical Detection by Transformers<br> Bastian Wittmann, Fernando Navarro, Suprosanna Shit, Bjoern Menze<br> [21st Jul., 2022] [arXiv, 2022]
[PDF] [GitHub]

CellCentroidFormer: Combining Self-attention and Convolution for Cell Detection<br> Royden Wagner, Karl Rohr<br> [1st Jun., 2022] [MIUA, 2022]
[PDF] [Github]

CT-CAD: Context-Aware Transformers for End-to-End Chest Abnormality Detection on X-Rays<br> Qiran Kong, Yirui Wu, Chi Yuan, Yongli Wang<br> [9th Dec., 2021] [BIBM, 2021]
[PDF]

RDFNet: A Fast Caries Detection Method Incorporating Transformer Mechanism<br> Hao Jiang, Peiliang Zhang, Chao Che, Bo Jin<br> [10th Nov., 2021] [Computational and Mathematical Methods in Medicine Journal, 2021]
[PDF]

Spine-Transformers: Vertebra Detection and Localization in Arbitrary Field-of-View Spine CT with Transformers<br> Rong Tao, Guoyan Zheng<br> [21st Sep., 2021] [MICCAI, 2021]
[PDF]

Transformer Network for Significant Stenosis Detection in CCTA of Coronary Arteries<br> Xinghua Ma, Gongning Luo, Wei Wang, Kuanquan Wang<br> [7th Jul., 2021] [MICCAI, 2021]
[PDF] [Github]

COTR: Convolution in Transformer Network for End to End Polyp Detection<br> Zhiqiang Shen, Chaonan Lin, Shaohua Zheng<br> [23rd May, 2021] [ICCC, 2021]
[PDF]


Image Registration

SVoRT: Iterative Transformer for Slice-to-Volume Registration in Fetal Brain MRI. <br> Junshen Xu, Daniel Moyer, P. Ellen Grant, Polina Golland, Juan Eugenio Iglesias, Elfar Adalsteinsson.<br> [22th Jun., 2022] [MICCAI, 2022]
[PDF] [GitHub]

XMorpher: Full Transformer for Deformable Medical Image Registration via Cross Attention<br> Jiacheng Shi, Yuting He, Youyong Kong, Jean-Louis Coatrieux, Huazhong Shu, Guanyu Yang, Shuo Li<br> [15th Jun., 2022] [MICCAI, 2022]
[PDF] [GitHub]

TransMorph: Transformer for unsupervised medical image registration <br> Junyu Chen, Eric C. Frey, Yufan He, William P. Segars, Ye Li, Yong Du<br> [19th Nov., 2021] [MedIA Journal]
[PDF] [GitHub]

Learning dual transformer network for diffeomorphic registration <br> Yungeng Zhang, Yuru Pei & Hongbin Zha<br> [21th Sep., 2021] [MICCAI, 2021]
[PDF]

ViT-V-Net: Vision Transformer for Unsupervised Volumetric Medical Image Registration <br> Junyu Chen, Yufan He, Eric C. Frey, Ye Li, Yong Du<br> [13th Apr., 2021] [MIDL, 2021]
[PDF] [GitHub]

Affine Medical Image Registration with Coarse-to-Fine Vision Transformer <br> Tony C. W. Mok, Albert C. S. Chung<br> [29th Mar., 2022] [CVPR, 2022]
[PDF] [GitHub]


Report Generation

Cross-modal Memory Networks for Radiology Report Generation <br> Zhihong Chen, Yaling Shen, Yan Song, Xiang Wan<br> [28th Apr., 2022] [ACL-IJCNLP, 2021]
[PDF] [GitHub]

AlignTransformer: Hierarchical Alignment of Visual Regions and Disease Tags for Medical Report Generation <br> Di You, Fenglin Liu, Shen Ge, Xiaoxia Xie, Jing Zhang, Xian Wu<br> [18th Mar., 2022] [MICCAI, 2021]
[PDF]

Automated Generation of Accurate & Fluent Medical X-ray Reports <br> Hoang T.N. Nguyen, Dong Nie, Taivanbat Badamdorj, Yujie Liu, Yingying Zhu, Jason Truong, Li Cheng<br> [27th Aug., 2021] [EMNLP, 2021]
[PDF] [GitHub]

Medical-vlbert: Medical visual language bert for covid-19 ct report generation with alternate learning <br> Guangyi Liu, Yinghong Liao, Fuyu Wang, Bin Zhang, Lu Zhang, Xiaodan Liang, Xiang Wan, Shaolin Li, Zhen Li, Shuixing Zhang, Shuguang Cui<br> [11th Aug., 2021] [IEEE Transactions on Neural Networks and Learning Systems, 2021]
[PDF]

Surgical Instruction Generation with Transformers<br> Jinglu Zhang, Yinyu Nie, Jian Chang, Jian Jun Zhang<br> [14th Jul., 2021] [MICCAI, 2021]
[PDF]

Trust It or Not: Confidence-Guided Automatic Radiology Report Generation <br> Yixin Wang, Zihao Lin, Zhe Xu, Haoyu Dong, Jiang Tian, Jie Luo, Zhongchao Shi, Yang Zhang, Jianping Fan, Zhiqiang He<br> [21st Jun., 2021] [arXiv , 2021]
[PDF]

Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation <br> Fenglin Liu, Xian Wu, Shen Ge, Wei Fan, Yuexian Zou<br> [13th Jun., 2021] [CVPR, 2021]
[PDF]

Progressive Transformer-Based Generation of Radiology Reports <br> Farhad Nooralahzadeh, Nicolas Perez Gonzalez, Thomas Frauenfelder, Koji Fujimoto, Michael Krauthammer<br> [19th Feb., 2021] [EMNLP , 2021]
[PDF] [GitHub]

Learning to Generate Clinically Coherent Chest X-Ray Reports <br> Justin Lovelace, Bobak Mortazavi<br> [1st Nov., 2020] [EMNLP, 2020]
[PDF] [GitHub]

Generating Radiology Reports via Memory-driven Transformer<br> Zhihong Chen, Yan Song, Tsung-Hui Chang, Xiang Wan<br> [30th Oct., 2020] [EMNLP, 2020]
[PDF] [GitHub]

Reinforced Transformer for Medical Image Captioning <br> Yuxuan Xiong, Bo Du, Pingkun Yan<br> [10th Oct., 2019][MICCAI Workshop, 2019]
[PDF]

Knowledge-driven Encode, Retrieve, Paraphrase for Medical Image Report Generation <br> Christy Y. Li, Xiaodan Liang, Zhiting Hu, Eric P. Xing<br> [25th Mar., 2019] [AAAI, 2019]
[PDF]