Awesome
Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review
:fire::fire:This is a collection of awesome articles about Transformer models in medical imaging :fire::fire:
:loudspeaker: Our review paper published on MedIA: Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review :heart:
:loudspeaker: Our review paper published on arXiv: Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review :heart:
Citation
@article{azad2023advances,
title={Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review},
author={Azad, Reza and Kazerouni, Amirhossein and Heidari, Moein and Aghdam, Ehsan Khodapanah and Molaei, Amirali and Jia, Yiwei and Jose, Abin and Roy, Rijo and Merhof, Dorit},
journal={Medical Image Analysis},
volume = {91},
pages={103000},
year={2024},
issn = {1361-8415},
publisher={Elsevier}
}
Contents
Taxonomy
Papers
Image Classification
HATNet: An End-to-End Holistic Attention Network for Diagnosis of Breast Biopsy Images <br>
Sachin Mehta, Ximing Lu, Donald Weaver, Joann G. Elmore, Hannaneh Hajishirzi, Linda Shapiro<br>
[25th Jul, 2020] [MedIA Journal, 2022]
[PDF] [GitHub]
A graph-transformer for whole slide image classification <br>
Yi Zheng, Rushin H. Gindra, Emily J. Green, Eric J. Burks, Margrit Betke, Jennifer E. Beane, Vijaya B. Kolachalama<br>
[19th May, 2022] [TMI Journal, 2022]
[PDF] [GitHub]
RadioTransformer: A Cascaded Global-Focal Transformer for Visual Attention-guided Disease Classification <br>
Moinak Bhattacharya, Shubham Jain, Prateek Prasanna<br>
[23rd Feb., 2022] [ECCV, 2022]
[PDF]
Federated Split Vision Transformer for COVID-19 CXR Diagnosis using Task-Agnostic Training <br>
Sangjoon Park, Gwanghyun Kim, Jeongsol Kim, Boah Kim, Jong Chul Ye<br>
[2nd Nov., 2021] [NeurIPS, 2021]
[PDF]
Vision transformer for classification of breast ultrasound images <br>
Behnaz Gheflati, Hassan Rivaz<br>
[27th Oct., 2021] [EMBC, 2022]
[PDF]
MIL-VT: Multiple Instance Learning Enhanced Vision Transformer for Fundus Image Classification <br>
Shuang Yu, Kai Ma, Qi Bi, Cheng Bian, Munan Ning, Nanjun He, Yuexiang Li, Hanruo Liu, Yefeng Zheng<br>
[21st Sep., 2021] [MICCAI, 2021]
[PDF] [GitHub]
3DMeT: 3D Medical Image Transformer for Knee Cartilage Defect Assessment <br>
Sheng Wang, Zixu Zhuang, Kai Xuan, Dahong Qian, Zhong Xue, Jia Xu, Ying Liu, Yiming Chai, Lichi Zhang, Qian Wang, Dinggang Shen<br>
[21st Sep., 2021] [MICCAI Workshop, 2021]
[PDF]
COVID-Transformer: Interpretable COVID-19 Detection Using Vision Transformer for Healthcare <br>
Debaditya Shome, T. Kar, Sachi Nandan Mohanty, Prayag Tiwari, Khan Muhammad, Abdullah AlTameem, Yazhou Zhang, Abdul Khader Jilani Saudagar<br>
[23rd Sep., 2021] [International Journal of Environmental Research and Public Health, 2021]
[PDF] [GitHub]
Is it Time to Replace CNNs with Transformers for Medical Images? <br>
Christos Matsoukas, Johan Fredin Haslum, Magnus Söderberg, Kevin Smith<br>
[20th Aug., 2021] [ICCV Workshop, 2021]
[PDF] [GitHub]
Vision Transformer for femur fracture classification <br>
Leonardo Tanzi, Andrea Audisio, Giansalvo Cirrincione, Alessandro Aprato, Enrico Vezzetti<br>
[7th Aug., 2021] [Injury Journal, 2022]
[PDF]
xViTCOS: Explainable Vision Transformer Based COVID-19 Screening Using Radiography <br>
Arnab Kumar Mondal, Arnab Bhattacharjee, Parag Singla, A. P. Prathosh<br>
[7th Jul., 2021] [IEEE Journal of Translational Engineering in Health and Medicine, 2021]
[PDF] [Github]
COVID-VIT: Classification of COVID-19 from CT chest images based on vision transformer models <br>
Xiaohong Gao, Yu Qian, Alice Gao<br>
[4th Jul., 2021] [NextComp, 2022]
[PDF] [GitHub]
TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification <br>
Zhuchen Shao, Hao Bian, Yang Chen, Yifeng Wang, Jian Zhang, Xiangyang Ji, Yongbing Zhang<br>
[2nd Jun., 2021] [NeurIPS, 2021]
[PDF] [GitHub]
Lesion-Aware Transformers for Diabetic Retinopathy Grading <br>
Rui Sun, Yihao Li, Tianzhu Zhang, Zhendong Mao, Feng Wu, Yongdong Zhang<br>
[1st Jun., 2021] [CVPR, 2021]
[PDF]
POCFormer: A Lightweight Transformer Architecture for Detection of COVID-19 Using Point of Care Ultrasound <br>
Shehan Perera, Srikar Adhikari, Alper Yilmaz<br>
[20th May, 2021] [ICIP, 2022]
[PDF]
Automatic diagnosis of covid-19 using a tailored transformer-like network <br>
Chengeng Liu, Qingshan Yin<br>
[21st Apr., 2021] [CISAT, 2021]
[PDF]
Vision Transformer for COVID-19 CXR Diagnosis using Chest X-ray Feature Corpus <br>
Sangjoon Park, Gwanghyun Kim, Yujin Oh, Joon Beom Seo, Sang Min Lee, Jin Hwan Kim, Sungjun Moon, Jae-Kwang Lim, Jong Chul Ye<br>
[12th Mar., 2021] [arXiv, 2021]
[PDF]
TransMed: Transformers Advance Multi-modal Medical Image Classification <br>
Yin Dai, Yifan Gao<br>
[10th Mar., 2021] [Diagnostics, 2021]
[PDF]
Image Segmentation
TransDeepLab: Convolution-Free Transformer-based DeepLab v3+ for Medical Image Segmentation <br>
Reza Azad, Moein Heidari, Moein Shariatnia, Ehsan Khodapanah Aghdam, Sanaz Karimijafarbigloo, Ehsan Adeli, Dorit Merhof<br>
[1st Aug., 2022] [MICCAI Workshop, 2022]
[PDF] [GitHub]
HiFormer: Hierarchical Multi-scale Representations Using Transformers for Medical Image Segmentation<br>
Moein Heidari, Amirhossein Kazerouni, Milad Soltany, Reza Azad, Ehsan Khodapanah, Aghdam Julien Cohen-Adad, Dorit Merhof<br>
[18th Jul., 2022] [WACV, 2023]
[PDF] [GitHub]
Self Pre-training with Masked Autoencoders for Medical Image Analysis<br>
Lei Zhou, Huidong Liu, Joseph Bae, Junjun He, Dimitris Samaras, Prateek Prasanna<br>
[10th Mar., 2022] [arXiv, 2022]
[PDF]
Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images<br>
Ali Hatamizadeh, Vishwesh Nath, Yucheng Tang, Dong Yang, Holger Roth, Daguang Xu <br>
[4th Jan., 2022] [MICCAI Workshop]
[PDF] [GitHub]
Semi-Supervised Medical Image Segmentation via Cross Teaching between CNN and Transformer<br>
Xiangde Luo, Minhao Hu, Tao Song, Guotai Wang, Shaoting Zhang<br>
[9th Dec., 2021] [MIDL, 2022]
[PDF] [Github]
T-AutoML: Automated Machine Learning for Lesion Segmentation using Transformers in 3D Medical Imaging<br>
Dong Yang, Andriy Myronenko, Xiaosong Wang, Ziyue Xu, Holger R. Roth, Daguang Xu<br>
[15th Nov., 2021] [ICCV, 2021]
[PDF]
MISSFormer: An Effective Medical Image Segmentation Transformer<br>
Xiaohong Huang, Zhifang Deng, Dandan Li, Xueguang Yuan<br>
[15th Sep., 2021] [TMI Journal, 2022]
[PDF] [GitHub]
nnFormer: Interleaved Transformer for Volumetric Segmentation<br>
Hong-Yu Zhou, Jiansen Guo, Yinghao Zhang, Lequan Yu, Liansheng Wang, Yizhou Yu<br>
[7th Sep., 2021] [arXiv, 2021]
[PDF] [GitHub]
Medical Image Segmentation Using Squeeze-and-Expansion Transformers<br>
Shaohua Li, Xiuchao Sui, Xiangde Luo, Xinxing Xu, Yong Liu, Rick Goh<br>
[20th May, 2021] [IJCAI, 2021]
[PDF] [GitHub]
Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation<br>
Hu Cao, Yueyue Wang, Joy Chen, Dongsheng Jiang, Xiaopeng Zhang, Qi Tian, Manning Wang <br>
[12th May, 2021] [arXiv, 2021]
[PDF] [GitHub]
UNETR: Transformers for 3D Medical Image Segmentation<br>
Ali Hatamizadeh, Yucheng Tang, Vishwesh Nath, Dong Yang, Andriy Myronenko, Bennett Landman, Holger Roth, Daguang Xu<br>
[18th Mar., 2021] [WACV, 2022]
[PDF] [GitHub]
TransBTS: Multimodal Brain Tumor Segmentation Using Transformer<br>
Jiangyun Li, Wenxuan Wang, Chen Chen, Tianxiang Zhang, Sen Zha, Hong Yu, Jing Wang<br>
[7th Mar, 2021] [MICCAI, 2021]
[PDF] [GitHub]
CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation <br>
Yutong Xie, Jianpeng Zhang, Chunhua Shen, Yong Xia <br>
[4th Mar., 2021] [MICCAI, 2021]
[PDF] [GitHub]
Medical Transformer: Gated Axial-Attention for Medical Image Segmentation<br>
Jeya Maria Jose Valanarasu, Poojan Oza, Ilker Hacihaliloglu, Vishal M. Patel<br>
[21th Feb., 2021] [MICCAI, 2021]
[PDF] [GitHub]
TransFuse: Fusing Transformers and CNNs for Medical Image Segmentation<br>
Yundong Zhang, Huiye Liu, Qiang Hu<br>
[16th Feb., 2021] [arXiv, 2021]
[PDF] [GitHub]
TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation <br>
Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Le Lu, Alan L. Yuille, Yuyin Zhou<br>
[8th Feb., 2021] [arXiv, 2021]
[PDF] [GitHub]
Image Reconstruction
Transformer-based Dual-domain Network for Few-view Dedicated Cardiac SPECT Image Reconstructions <br>
Huidong Xie, Bo Zhou, Xiongchao Chen, Xueqi Guo, Stephanie Thorn, Yi-Hwa Liu, Ge Wang, Albert Sinusas, Chi Liu<br>
[18th July., 2023] [MICCAI, 2023]
[PDF]
TransCT: Dual-path Transformer for Low Dose Computed Tomography <br>
Zhicheng Zhang, Lequan Yu, Xiaokun Liang, Wei Zhao, Lei Xing<br>
[28th Feb., 2021] [MICCAI, 2021]
[PDF] [GitHub]
TED-net: Convolution-free T2T Vision Transformer-based Encoder-decoder Dilation network for Low-dose CT Denoising <br>
Dayang Wang, Zhan Wu, Hengyong Yu<br>
[8th Jun., 2021] [MICCAI Workshop, 2021]
[PDF] [GitHub]
Eformer: Edge Enhancement based Transformer for Medical Image Denoising <br>
Achleshwar Luthra, Harsh Sulakhe, Tanish Mittal, Abhishek Iyer, Santosh Yadav<br>
[16th Sep., 2021] [arXiv, 2021]
[PDF]
3D Transformer-GAN for High-Quality PET Reconstruction <br>
Yanmei Luo, Yan Wang, Chen Zu, Bo Zhan, Xi Wu, Jiliu Zhou, Dinggang Shen, Luping Zhou<br>
[21st Sep., 2021] [MICCAI, 2021]
[PDF]
Spatial Adaptive and Transformer Fusion Network (STFNet) for Low-count PET Blind Denoising with MRI <br>
Lipei Zhang, Zizheng Xiao, Chao Zhou, Jianmin Yuan, Qiang He, Yongfeng Yang, Xin Liu, Dong Liang, Hairong Zheng, Wei Fan, Xu Zhang, Zhanli Hu<br>
[19th Nov., 2021] [Medical Physics, 2021]
[PDF]
CTformer: Convolution-free Token2Token Dilated Vision Transformer for Low-dose CT Denoising <br>
Dayang Wang, Fenglei Fan, Zhan Wu, Rui Liu, Fei Wang, Hengyong Yu<br>
[28th Feb., 2022] [arXiv, 2022]
[PDF] [GitHub]
Low-Dose CT Denoising via Sinogram Inner-Structure Transformer <br>
Liutao Yang, Zhongnian Li, Rongjun Ge, Junyong Zhao, Haipeng Si, Daoqiang Zhang<br>
[7th Apr., 2022] [IEEE Transactions on Medical Imaging, 2022]
[PDF]
DuDoTrans: Dual-Domain Transformer Provides More Attention for Sinogram Restoration in Sparse-View CT Reconstruction <br>
Ce Wang, Kun Shang, Haimiao Zhang, Qian Li, Yuan Hui, S. Kevin Zhou<br>
[21th Nov., 2021] [arXiv, 2021]
[PDF] [GitHub]
Fourier Image Transformer <br>
Tim-Oliver Buchholz, Florian Jug<br>
[6th Apr., 2021] [CVPR, 2022]
[PDF] [GitHub]
Dual-domain sparse-view CT reconstruction with Transformers <br>
Changrong Shi, Yongshun Xiao, Zhiqiang Chen<br>
[22nd Mar., 2022] [ELSEVIER Physica Medica, 2022]
[PDF]
Adaptively Re-weighting Multi-Loss Untrained Transformer for Sparse-View Cone-Beam CT Reconstruction <br>
Minghui Wu, Yangdi Xu, Yingying Xu, Guangwei Wu, Qingqing Chen, Hongxiang Lin<br>
[23th Mar., 2022] [arXiv, 2022]
[PDF]
Vision Transformers Enable Fast and Robust Accelerated MRI <br>
Chun-Mei Feng, Yunlu Yan, Huazhu Fu, Li Chen, Yong Xu<br>
[10th Dec., 2021] [MIDL, 2022]
[PDF] [GitHub]
Task Transformer Network for Joint MRI Reconstruction and Super-Resolution <br>
Chun-Mei Feng, Yunlu Yan, Huazhu Fu, Li Chen, Yong Xu<br>
[12th Jun., 2021] [MICCAI, 2021]
[PDF] [GitHub]
MR Image Super Resolution By Combining Feature Disentanglement CNNs and Vision Transformers <br>
Chun-Mei Feng, Yunlu Yan, Huazhu Fu, Li Chen, Yong Xu<br>
[9th Dec., 2021] [MIDL, 2022]
[PDF]
Cross-Modality High-Frequency Transformer for MR Image Super-Resolution <br>
Chaowei Fang, Dingwen Zhang, Liang Wang, Yulun Zhang, Lechao Cheng, Junwei Han<br>
[29th Mar., 2022] [ACM MM, 2022]
[PDF]
Image Synthesizing
One Model to Synthesize Them All: Multi-contrast Multi-scale Transformer for Missing Data Imputation <br>
Jiang Liu, Srivathsa Pasumarthi, Ben Duffy, Enhao Gong, Greg Zaharchuk, Keshav Datta<br>
[28th Apr., 2022] [arXiv, 2021]
[PDF]
CyTran: Cycle-Consistent Transformers for Non-Contrast to Contrast CT Translation <br>
Nicolae-Catalin Ristea, Andreea-Iuliana Miron, Olivian Savencu, Mariana-Iuliana Georgescu, Nicolae Verga, Fahad Shahbaz Khan, Radu Tudor Ionescu<br>
[12th Oct., 2021] [arXiv, 2021]
[PDF] [GitHub]
ResViT: Residual vision transformers for multi-modal medical image synthesis <br>
Onat Dalmaz, Mahmut Yurt, Tolga Çukur<br>
[30th Jun., 2021] [TMI Journal, 2021]
[PDF] [GitHub]
PTNet: A High-Resolution Infant MRI Synthesizer Based on Transformer <br>
Xuzhe Zhang, Xinzi He, Jia Guo, Nabil Ettehadi, Natalie Aw, David Semanek, Jonathan Posner, Andrew Laine, Yun Wang<br>
[28th May., 2021] [arXiv, 2021]
[PDF] [GitHub]
VTGAN: Semi-supervised Retinal Image Synthesis and Disease Prediction using Vision Transformers <br>
Sharif Amit Kamran, Khondker Fariha Hossain, Alireza Tavakkoli, Stewart Lee Zuckerbrod, Salah A. Baker<br>
[14th Apr., 2021] [ICCV Workshop, 2021]
[PDF] [GitHub]
Object Detection
Focused Decoding Enables 3D Anatomical Detection by Transformers<br>
Bastian Wittmann, Fernando Navarro, Suprosanna Shit, Bjoern Menze<br>
[21st Jul., 2022] [arXiv, 2022]
[PDF] [GitHub]
CellCentroidFormer: Combining Self-attention and Convolution for Cell Detection<br>
Royden Wagner, Karl Rohr<br>
[1st Jun., 2022] [MIUA, 2022]
[PDF] [Github]
CT-CAD: Context-Aware Transformers for End-to-End Chest Abnormality Detection on X-Rays<br>
Qiran Kong, Yirui Wu, Chi Yuan, Yongli Wang<br>
[9th Dec., 2021] [BIBM, 2021]
[PDF]
RDFNet: A Fast Caries Detection Method Incorporating Transformer Mechanism<br>
Hao Jiang, Peiliang Zhang, Chao Che, Bo Jin<br>
[10th Nov., 2021] [Computational and Mathematical Methods in Medicine Journal, 2021]
[PDF]
Spine-Transformers: Vertebra Detection and Localization in Arbitrary Field-of-View Spine CT with Transformers<br>
Rong Tao, Guoyan Zheng<br>
[21st Sep., 2021] [MICCAI, 2021]
[PDF]
Transformer Network for Significant Stenosis Detection in CCTA of Coronary Arteries<br>
Xinghua Ma, Gongning Luo, Wei Wang, Kuanquan Wang<br>
[7th Jul., 2021] [MICCAI, 2021]
[PDF] [Github]
COTR: Convolution in Transformer Network for End to End Polyp Detection<br>
Zhiqiang Shen, Chaonan Lin, Shaohua Zheng<br>
[23rd May, 2021] [ICCC, 2021]
[PDF]
Image Registration
SVoRT: Iterative Transformer for Slice-to-Volume Registration in Fetal Brain MRI. <br>
Junshen Xu, Daniel Moyer, P. Ellen Grant, Polina Golland, Juan Eugenio Iglesias, Elfar Adalsteinsson.<br>
[22th Jun., 2022] [MICCAI, 2022]
[PDF] [GitHub]
XMorpher: Full Transformer for Deformable Medical Image Registration via Cross Attention<br>
Jiacheng Shi, Yuting He, Youyong Kong, Jean-Louis Coatrieux, Huazhong Shu, Guanyu Yang, Shuo Li<br>
[15th Jun., 2022] [MICCAI, 2022]
[PDF] [GitHub]
TransMorph: Transformer for unsupervised medical image registration <br>
Junyu Chen, Eric C. Frey, Yufan He, William P. Segars, Ye Li, Yong Du<br>
[19th Nov., 2021] [MedIA Journal]
[PDF] [GitHub]
Learning dual transformer network for diffeomorphic registration <br>
Yungeng Zhang, Yuru Pei & Hongbin Zha<br>
[21th Sep., 2021] [MICCAI, 2021]
[PDF]
ViT-V-Net: Vision Transformer for Unsupervised Volumetric Medical Image Registration <br>
Junyu Chen, Yufan He, Eric C. Frey, Ye Li, Yong Du<br>
[13th Apr., 2021] [MIDL, 2021]
[PDF] [GitHub]
Affine Medical Image Registration with Coarse-to-Fine Vision Transformer <br>
Tony C. W. Mok, Albert C. S. Chung<br>
[29th Mar., 2022] [CVPR, 2022]
[PDF] [GitHub]
Report Generation
Cross-modal Memory Networks for Radiology Report Generation <br>
Zhihong Chen, Yaling Shen, Yan Song, Xiang Wan<br>
[28th Apr., 2022] [ACL-IJCNLP, 2021]
[PDF] [GitHub]
AlignTransformer: Hierarchical Alignment of Visual Regions and Disease Tags for Medical Report Generation <br>
Di You, Fenglin Liu, Shen Ge, Xiaoxia Xie, Jing Zhang, Xian Wu<br>
[18th Mar., 2022] [MICCAI, 2021]
[PDF]
Automated Generation of Accurate & Fluent Medical X-ray Reports <br>
Hoang T.N. Nguyen, Dong Nie, Taivanbat Badamdorj, Yujie Liu, Yingying Zhu, Jason Truong, Li Cheng<br>
[27th Aug., 2021] [EMNLP, 2021]
[PDF] [GitHub]
Medical-vlbert: Medical visual language bert for covid-19 ct report generation with alternate learning <br>
Guangyi Liu, Yinghong Liao, Fuyu Wang, Bin Zhang, Lu Zhang, Xiaodan Liang, Xiang Wan, Shaolin Li, Zhen Li, Shuixing Zhang, Shuguang Cui<br>
[11th Aug., 2021] [IEEE Transactions on Neural Networks and Learning Systems, 2021]
[PDF]
Surgical Instruction Generation with Transformers<br>
Jinglu Zhang, Yinyu Nie, Jian Chang, Jian Jun Zhang<br>
[14th Jul., 2021] [MICCAI, 2021]
[PDF]
Trust It or Not: Confidence-Guided Automatic Radiology Report Generation <br>
Yixin Wang, Zihao Lin, Zhe Xu, Haoyu Dong, Jiang Tian, Jie Luo, Zhongchao Shi, Yang Zhang, Jianping Fan, Zhiqiang He<br>
[21st Jun., 2021] [arXiv , 2021]
[PDF]
Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation <br>
Fenglin Liu, Xian Wu, Shen Ge, Wei Fan, Yuexian Zou<br>
[13th Jun., 2021] [CVPR, 2021]
[PDF]
Progressive Transformer-Based Generation of Radiology Reports <br>
Farhad Nooralahzadeh, Nicolas Perez Gonzalez, Thomas Frauenfelder, Koji Fujimoto, Michael Krauthammer<br>
[19th Feb., 2021] [EMNLP , 2021]
[PDF] [GitHub]
Learning to Generate Clinically Coherent Chest X-Ray Reports <br>
Justin Lovelace, Bobak Mortazavi<br>
[1st Nov., 2020] [EMNLP, 2020]
[PDF] [GitHub]
Generating Radiology Reports via Memory-driven Transformer<br>
Zhihong Chen, Yan Song, Tsung-Hui Chang, Xiang Wan<br>
[30th Oct., 2020] [EMNLP, 2020]
[PDF] [GitHub]
Reinforced Transformer for Medical Image Captioning <br>
Yuxuan Xiong, Bo Du, Pingkun Yan<br>
[10th Oct., 2019][MICCAI Workshop, 2019]
[PDF]
Knowledge-driven Encode, Retrieve, Paraphrase for Medical Image Report Generation <br>
Christy Y. Li, Xiaodan Liang, Zhiting Hu, Eric P. Xing<br>
[25th Mar., 2019] [AAAI, 2019]
[PDF]