Home

Awesome

 

<div align="center"> <p align="center"> <img src="3d_demo/logo.png" width="110px"> </p>

arXiv zhihu Youtube AK MrNeRF RF

<h2> Radiative Gaussian Splatting for Efficient X-ray Novel View Synthesis </h2> <img src="3d_demo/teapot.gif" style="height:200px" /> <img src="3d_demo/foot.gif" style="height:160px" /> <img src="3d_demo/bonsai.gif" style="height:200px" />

Point Cloud Visualization

 

<img src="3d_demo/training_process.gif" style="height:200px" />

Training Process Visualization

</div>

 

Introduction

This is the official implementation of our ECCV 2024 paper "Radiative Gaussian Splatting for Efficient X-ray Novel View Synthesis". Our X-Gaussian is SfM-free. If you find this repo useful, please give it a star ⭐ and consider citing our paper. Thank you.

News

Performance

<details close> <summary><b>Novel View Synthesis</b></summary>

results1

results2

</details> <details close> <summary><b>CT Reconstruction</b></summary>

results3

results4

</details>

Coordinate System

The coordinate system in circular cone-beam X-ray scanning follows the OpenCV standards. The transformation between the camera, world, and image coordinate systems is shown below.

<div align="center"> <p align="center"> <img src="fig/coordinate_system.png" width="800px"> </p> </div>

 

1. Create Environment:

We recommend using Conda to set up the environment.

# cloning our repo
git clone https://github.com/caiyuanhao1998/X-Gaussian --recursive


SET DISTUTILS_USE_SDK=1 # Windows only

# install the official environment of 3DGS
cd X-Gaussian
conda env create --file environment.yml
conda activate x_gaussian

# Use our X-ray rasterizer package to replace the original RGB rasterizer
rm -rf submodules/diff-gaussian-rasterization/cuda_rasterizer
mv cuda_rasterizer submodules/diff-gaussian-rasterization/

# re-install the diff-gaussian-rasterization package
pip install submodules/diff-gaussian-rasterization

 

2. Prepare Dataset

Download our processed datasets from Google drive or Baidu disk. Then put the downloaded datasets into the folder data/ as

  |--data
      # The first five datasets are used in our paper
      |--chest_50.pickle
      |--abdomen_50.pickle
      |--foot_50.pickle
      |--head_50.pickle
      |--pancreas_50.pickle
      # The rest datasets are from the X3D benchmark
      |--aneurism_50.pickle
      |--backpack_50.pickle
      |--bonsai_50.pickle
      |--box_50.pickle
      |--carp_50.pickle
      |--engine_50.pickle
      |--leg_50.pickle
      |--pelvis_50.pickle
      |--teapot_50.pickle
      |--jaw_50.pickle

Note: The first five datasets are used to do experiments in our paper. The rest datasets are from the X3D benchmark. Please also note that the pickle data used by X-Gaussian is dumped/read by pickle protocol 4, which is supported by python < 3.8. The original X3D data is processed by pickle protocol 5, which is supported by python >= 3.8. I have re-dumped the pickle data from the original X3D datasets to make sure you can run our code without extra effort. If you want to re-dump the pickle data, please run

python pickle_redump.py

 

3. Training and Testing

You can download our trained Gaussian point clouds from Google Drive or Baidu Disk (code: cyh2) as

pc_shape

We share the training log for your convienience to debug. Please download them from Google Drive or Baidu Disk (code: cyh2). To make the training and evaluation easier, your can directly run the train.sh file by

bash train.sh

Or you can separately train on each scene like


python3 train.py --config config/chest.yaml --eval

python3 train.py --config config/foot.yaml --eval

python3 train.py --config config/abdomen.yaml --eval

python3 train.py --config config/head.yaml --eval

python3 train.py --config config/pancreas.yaml --eval

python3 train.py --config config/jaw.yaml --eval

python3 train.py --config config/pelvis.yaml --eval

python3 train.py --config config/aneurism.yaml --eval

python3 train.py --config config/carp.yaml --eval

python3 train.py --config config/bonsai.yaml --eval

python3 train.py --config config/box.yaml --eval

python3 train.py --config config/backpack.yaml --eval

python3 train.py --config config/engine.yaml --eval

python3 train.py --config config/leg.yaml --eval

python3 train.py --config config/teapot.yaml --eval

 

4. Visualization

We also provide code for the visualization of rotating the Gaussian point clouds


python point_cloud_vis.py

 

5. Citation

# X-Gaussian
@inproceedings{x_gaussian,
  title={Radiative gaussian splatting for efficient x-ray novel view synthesis},
  author={Yuanhao Cai and Yixun Liang and Jiahao Wang and Angtian Wang and Yulun Zhang and Xiaokang Yang and Zongwei Zhou and Alan Yuille},
  booktitle={ECCV},
  year={2024}
}

# sax-nerf
@inproceedings{sax_nerf,
  title={Structure-Aware Sparse-View X-ray 3D Reconstruction},
  author={Yuanhao Cai and Jiahao Wang and Alan Yuille and Zongwei Zhou and Angtian Wang},
  booktitle={CVPR},
  year={2024}
}

 

Acknowledgement

Our code and data are heavily borrowed from SAX-NeRF and 3DGS