Home

Awesome

IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination

Project Page | Paper | Supplementary | Hugging Face

<br/>

IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination <br> Xi Chen, Sida Peng, Dongchen Yang, Yuan Liu, Bowen Pan, Chengfei Lv, Xiaowei Zhou<br>

<img src="./assets/pipeline.png"/>

News

Results

Intrinsic Decomposition

<img src="./assets/intrinsic.png"/>

Single view Relighting

https://github.com/zju3dv/IntrinsicAnything/assets/62891073/2e95855c-6d72-4bcb-8c79-577e55e6c926

Single view inference

  1. Installation
conda create -n anyintrinsic python=3.10
conda activate anyintrinsic
pip install -r requirements.txt
  1. Download the pre-trained diffusion models from hugghing face as follow:
# albedo checkpoint
huggingface-cli download --repo-type space --cache-dir  weights/albedo/checkpoints/ LittleFrog/IntrinsicAnything weights/albedo/checkpoints/last.ckpt 

# specular shaing checkpoint
huggingface-cli download --repo-type space --cache-dir  weights/specular/checkpoints/ LittleFrog/IntrinsicAnything weights/specular/checkpoints/last.ckpt 
  1. Run inference to get intrinsic images:
python inference.py \
--input_dir  examples  \
--model_dir  weights/albedo \
--output_dir out/albedo \
--ddim 100 \
--batch_size 4

Parameter explanation:

  1. Optionally, you can generate high-resolution samples under the guidance of samples from step 3.
python inference.py \
--input_dir examples  \
--model_dir weights/albedo \
--output_dir out/albedo_high_res \
--ddim 200 \
--batch_size 4 \
--guidance_dir out/albedo \
--guidance 3  \
--splits_vertical 2 \
--splits_horizontal 2 \
--splits_overlap 1

Extra parameter explanation:

Multiview Inverse Rendering

Comming Soon.

Citation

@article{chen2024intrinsicanything,
    title     = {IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination},
    author    = {Xi, Chen and Sida, Peng and Dongchen, Yang and Yuan, Liu and Bowen, Pan and Chengfei, Lv and Xiaowei, Zhou.},
    journal   = {arxiv: 2404.11593},
    year      = {2024},
    }