Home

Awesome

ComfyUI-ZeroShot-MTrans: Zero-Shot Material Transfer from a Single Image

中文

An unofficial ComfyUI custom node for ZeST(Zero-Shot Material Transfer from a Single Image)

Zero-Shot Material Transfer from a Single Image

Given an input image (e.g., a photo of an apple) and a single material exemplar image (e.g., a golden bowl), ZeST can transfer the gold material from the exemplar onto the apple with accurate lighting cues while making everything else consistent.

Explains

Refer to zest_workflow.json in this repo for quick start.

This node implenment the red rect area of paper figure. the illumination transfer, and make it an ComfyUI node. You can use any IPA and controlnet node as your wish.

The original ZeST demo code use IPA to extract the style of material, and control for depth information. Then, change the original image with illumination informaion of subject for better lighting style.

install

in ComfyUI Manager or git clone to ComfyUI/custom_nodes

After restart ComfyUI, the following custom node will be available.

ZestNode

dependences

Input/Output

Tips

  1. for output, the subject in middle grey is prefered. for high light material, dark it; otherwise, brighter it.
  2. for material image, suggest remove all backgroup, just keep the material you want.
  3. use IP-Adaptor strength, controlnet strength, brighter to control the output 1st.

Workflow

You can download the zest_workflow.json from this repo for quickly start.

ZeST Simple Workflow

sample workflow

Detail Usage

Image Input

pumpkin

Material Input

cpu_material

Temp Images

Mask

pumpkin_mask

Subject Grayout

pumpkin_zest

Output

pumpkin_output

Thanks to

@article{cheng2024zest,
  title={ZeST: Zero-Shot Material Transfer from a Single Image},
  author={Cheng, Ta-Ying and Sharma, Prafull and Markham, Andrew and Trigoni, Niki and Jampani, Varun},
  journal={arXiv preprint arXiv:2404.06425},
  year={2024}
}