Home

Awesome

 

<div align="center"> <p align="center"> <img src="fig/logo.png" width="100px"> </p>

arXiv zhihu Youtube

PWC

PWC

<h2> A Toolbox for Sparse-View X-ray 3D Reconstruction </h2>

<img src="3d_demo/backpack.gif" style="height:260px" /> <img src="3d_demo/box.gif" style="height:260px" /> <img src="3d_demo/bonsai.gif" style="height:260px" />

<img src="3d_demo/foot.gif" style="height:230px" />     <img src="3d_demo/teapot.gif" style="height:230px" /> <img src="3d_demo/engine.gif" style="height:230px" />

</div>

 

Introduction

This repo is a comprehensive toolbox and library for X-ray 3D reconstruction including two tasks, novel view synthesis (NVS) and computed tomography (CT) reconstruction. This repo supports 11 state-of-the-art methods including six NeRF-based methods, two 3DGS-based methods, two optimization-based methods, and one analytical method. We also provide code for fancy visualization such as turntable video and data generation to help your research. If you find this repo useful, please give it a star ⭐ and consider citing our paper. Thank you.

News

Performance

<details close> <summary><b>Novel View Synthesis</b></summary>

results1

results2

</details> <details close> <summary><b>CT Reconstruction</b></summary>

results3

results4

</details> <details open> <summary><b>Supported algorithms:</b></summary> </details>

Coordinate System

The coordinate system in circular cone-beam X-ray scanning follows the OpenCV standards. The transformation between the camera, world, and image coordinate systems is shown below.

<div align="center"> <p align="center"> <img src="fig/coordinate_system.png" width="800px"> </p> </div>

 

1. Create Environment:

We recommend using Conda to set up an environment.

# Create environment
conda create -n sax_nerf python=3.9
conda activate sax_nerf

# Install pytorch (hash encoder requires CUDA v11.3)
pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0 --extra-index-url https://download.pytorch.org/whl/cu113

# Install other packages
pip install -r requirements.txt

We suggest you install TIGRE toolbox (2.3 version) for executing traditional CT reconstruction methods and synthesize your own CT data if you plan to do so. Please note that TIGRE v2.5 might stuck when CT is large.

# Download TIGRE
wget https://github.com/CERN/TIGRE/archive/refs/tags/v2.3.zip
unzip v2.3.zip
rm v2.3.zip

# Install TIGRE
pip install cython==0.29.25
pip install numpy==1.21.6
cd TIGRE-2.3/Python/
python setup.py develop

 

2. Prepare Dataset:

Download our processed datasets from Google drive or Baidu disk. Then put the downloaded datasets into the folder data/ as

  |--data
      |--chest_50.pickle
      |--abdomen_50.pickle
      |--aneurism_50.pickle
      |--backpack_50.pickle
      |--bonsai_50.pickle
      |--box_50.pickle
      |--carp_50.pickle
      |--engine_50.pickle
      |--foot_50.pickle
      |--head_50.pickle
      |--leg_50.pickle
      |--pancreas_50.pickle
      |--pelvis_50.pickle
      |--teapot_50.pickle
      |--jaw_50.pickle

 

3. Testing:

You can directly download our pre-trained models from Google drive or Baidu disk. Then put the downloaded models into the folder pretrained/ and run

# SAX-NeRF
python test.py --method Lineformer --category chest --config config/Lineformer/chest_50.yaml --weights pretrained/chest.tar --output_path output 

# FDK
python3 eval_traditional.py --algorithm fdk --category chest --config config/FDK/chest_50.yaml

# SART
python3 eval_traditional.py --algorithm sart --category chest --config config/SART/chest_50.yaml

# ASD_POCS
python3 eval_traditional.py --algorithm asd_pocs --category chest --config config/ASD_POCS/chest_50.yaml

For your convenience, we provide the reconstructed results in Google drive and Baidu disk.

 

4. Training:

We provide the training logs on all scenes for your convenience to debug. Please download the training logs from Google dive or Baidu disk.

# SAX-NeRF
python train_mlg.py --config config/Lineformer/chest_50.yaml

# NeRF
python train.py --config config/nerf/chest_50.yaml

# Intratomo
python train.py --config config/intratomo/chest_50.yaml

# NAF
python train.py --config config/naf/chest_50.yaml

# TensoRF
python train.py --config config/tensorf/chest_50.yaml

You can use this repo to run NeAT. Remember to reprocess the data first.

 

5. Visualization

To render a cool demo, we provide visualization code in the folder 3D_vis

cd 3D_vis
python 3D_vis_backpack.py
python 3D_vis_backpack_gif.py

 

6. Generate Your Own Data

We also provide code for data generation in the folder dataGenerator. To give you a quick start, we provide the raw data for your debugging. Please download the raw data from Google dive or Baidu disk and then put them into the folder dataGenerator/raw_data. Run

cd dataGenerator
python data_vis_backpack.py
cd ..
python dataGenerator/generateData_backpack.py

 

7. Citation

If this repo helps you, please consider citing our works:

# SAX-NeRF
@inproceedings{sax_nerf,
  title={Structure-Aware Sparse-View X-ray 3D Reconstruction},
  author={Yuanhao Cai and Jiahao Wang and Alan Yuille and Zongwei Zhou and Angtian Wang},
  booktitle={CVPR},
  year={2024}
}

# X-Gaussian
@inproceedings{x_gaussian,
  title={Radiative gaussian splatting for efficient x-ray novel view synthesis},
  author={Yuanhao Cai and Yixun Liang and Jiahao Wang and Angtian Wang and Yulun Zhang and Xiaokang Yang and Zongwei Zhou and Alan Yuille},
  booktitle={ECCV},
  year={2024}
}

# R2-Gaussian
@inproceedings{r2_gaussian,
  title={R2-Gaussian: Rectifying Radiative Gaussian Splatting for Tomographic Reconstruction},
  author={Ruyi Zha and Tao Jun Lin and Yuanhao Cai and Jiwen Cao and Yanhao Zhang and Hongdong Li},
  booktitle={NeurIPS},
  year={2024}
}