Home

Awesome

L-CAD: Language-based Colorization with Any-level Descriptions using Diffusion Priors

Abstract

Language-based colorization produces plausible and visually pleasing colors under the guidance of user-friendly natural language descriptions. In this paper, we propose L-CAD to perform language-based colorization with any level descriptions.

<img src="teaser.png" align=center />

Additionally, we showcase its capability for efficient automatic colorization.

<img src="application.png" align=center />

Prerequisites

Installation

Clone this repo:

https://github.com/changzheng123/L-CAD.git

Install PyTorch and dependencies

http://pytorch.org

Install other python requirements

pip install -r requirement.txt

Test with pre-trained model

Download pre-trained models here (baidu) (google) and put them into "./models", and run:

python colorization_main.py

Run demo with instance-aware sampling:

python inference.py

Train

Download dataset:

Extended COCO-Stuff provides additional correspondence annotations between color words and object words. Download coco2017 images here.

Multi-instance contains samples with distinctive visual characteristics and detailed language descriptions for multiple instances in images.

Download init_model:

Download init_model from (baidu) (google) and put it into "./models".

Run the command:
python colorization_main.py -t

License

Licensed under a Creative Commons Attribution-NonCommercial 4.0 International.

Except where otherwise noted, this content is published under a CC BY-NC license, which means that you can copy, remix, transform and build upon the content as long as you do not use the material for commercial purposes and give appropriate credit and provide a link to the license.

Citation

If you use this code for your research, please cite our papers L-CAD: Language-based Colorization with Any-level Descriptions using Diffusion Priors

@InProceedings{lcad,
  author = {Chang, Zheng and Weng, Shuchen and Zhang, Peixuan and Li, Yu and Li, Si and Shi, Boxin},
  title = {L-CAD: Language-based Colorization with Any-level Descriptions using Diffusion Priors},
  booktitle = {{NeurIPS}},
  year = {2023}
}

Other language-based colorization

L-CoDe: Language-based colorization using color-object decoupled conditions (AAAI22)

L-CoDer: Language-based colorization with color-object decoupling transformer (ECCV22)

L-CoIns: Language-based colorization with instance awareness (CVPR23)