Awesome

This is the official implementation of DreamCatalyst.

https://github.com/user-attachments/assets/a704b4b1-4a27-445a-8cba-b27f223b824b

News & Updates

• [2024/10/04] 🌟 Codes based on the Threestudio for our method with Gaussian Editor have been released!

• [2024/10/02] 🌟 Codes based on the Nerfstudio have been released!

• [2024/07/17] 📄 Our paper is now available! The paper is released here.

Introduction

DreamCatalyst is a novel framework that considers the sampling dynamics of diffusion models in the SDS(Score Distillation Sampling) framework. DreamCatalyst can be applied to various models in 3D editing, such as NeRF and 3DGS. This method aims to reduce training time and improve editing quality. Please refer to the paper for more details.

Architecture

Get Started

We provide the implementation of DreamCatalyst based on the both NerfStudio and Threestudio frameworks. Nerfstudio is a framework for <b>NeRF or 3DGS editing</b>. Threestudio is a framework for <b>Gaussian Editor with DreamCatalyst</b>. Please refer to the following instructions.

Dataset

Please download the dataset from the following link and extract it into the dataset directory.

• PDS provides image data for 3D scenes. You can download the dataset here. This dataset includes 3D scene data from Instruct-NeRF2NeRF as a subset.
• Note: Dreambooth checkpoints are not required for our method, so you can skip downloading them.

For use with Threestudio (GaussianEditor), please follow the instructions below and extract the preprocessed COLMAP datasets and source 3DGS into the colmap and scene directories, respectively.

• We provide the COLMAP and initialized 3DGS for the source scene:
  • COLMAP
  • Source 3DGS
• You can use the dataset provided by PDS mentioned above.
  • <b>Preprocess Dataset</b>: Convert the dataset into COLMAP format as described in the 3DGS official repository.
  • <b>Initialize 3DGS</b>: Follow the instructions for initializing 3DGS, as outlined in the 3DGS official repository. You can find the point cloud .ply file. (e.g., scene/yuseung/point_cloud/iteration_30000/point_cloud.ply)

For NerfStudio

Please refer to nerfstudio/README.md.

For Threestudio

Please refer to threestudio/README.md.

Citation

If you find our work useful in your research, please cite:

@misc{kim2024dreamcatalystfasthighquality3d,
      title={DreamCatalyst: Fast and High-Quality 3D Editing via Controlling Editability and Identity Preservation}, 
      author={Jiwook Kim and Seonho Lee and Jaeyo Shin and Jiho Choi and Hyunjung Shim},
      year={2024},
      eprint={2407.11394},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2407.11394, 
}

Acknowledgement

We would like to express our gratitude to the following works:

• PDS
• Instruct-Nerf2Nerf
• InstructPix2Pix
• Gaussian Editor
• FreeU
• DreamSampler
• SDEdit