Awesome
Awesome Image Harmonization 
A curated list of resources including papers, datasets, and relevant links pertaining to image harmonization. Image harmonization aims to adjust the color, illumination, and style of foreground to be compatible with the background. For more complete resources on general image composition, please refer to Awesome-Image-Composition.
<p align='center'> <img src='./figures/task2.jpg' width=90% /> </p>Contributing
Contributions are welcome. If you wish to contribute, feel free to send a pull request. If you have suggestions for new sections to be included, please raise an issue and discuss before sending a pull request.
Table of Contents
Survey
A brief review on image harmonization is included in the following survey on image composition:
Li Niu, Wenyan Cong, Liu Liu, Yan Hong, Bo Zhang, Jing Liang, Liqing Zhang: "Making Images Real Again: A Comprehensive Survey on Deep Image Composition." arXiv preprint arXiv:2106.14490 (2021). [arXiv] [slides]
Online Demo
Try this online demo for image harmonization and have fun! 
Leaderboard
The leaderboard of SOTA image harmonization methods can be found here.
Color Transfer
We summarize different color transfer strategies which could be used for image harmonization task here.
Papers
Supervised deep learning methods
- Ziqi Yu, Jing Zhou, Zhongyun Bao, Gang Fu, Weilei He, Chao Liang, Chunxia Xiao: "CFDiffusion: Controllable Foreground Relighting in Image Compositing via Diffusion Model." ACM MM (2024) [pdf]
- Jing Zhou, Ziqi Yu, Zhongyun Bao, Gang Fu, Weilei He, Chao Liang, Chunxia Xiao: "Foreground Harmonization and Shadow Generation for Composite Image." ACM MM (2024) [pdf]
- Pengfei Zhou, Fangxiang Feng, Guang Liu, Ruifan Li, Xiaojie Wang: "DiffHarmony++: Enhancing Image Harmonization with Harmony-VAE and Inverse Harmonization Model." ACM MM (2024) [pdf] [code]
- Pengfei Zhou, Fangxiang Feng, Xiaojie Wan: "DiffHarmony: Latent Diffusion Model Meets Image Harmonization." ICMR (2024) [pdf] [code]
- Quanling Meng, Qinglin Liu, Zonglin Li, Xiangyuan Lan, Shengping Zhang, Liqiang Nie: "High-Resolution Image Harmonization with Adaptive-Interval Color Transformation." NeurIPS (2024) [pdf] [code]
- Mengwei Ren, Wei Xiong, Jae Shin Yoon, Zhixin Shu, Jianming Zhang, HyunJoon Jung, Guido Gerig, He Zhang: "Relightful Harmonization: Lighting-aware Portrait Background Replacement." CVPR (2024) [pdf]
- Xinhao Tao, Tianyuan Qiu, Junyan Cao, Li Niu: "Diverse Image Harmonization." arXiv preprint arXiv:2407.15481 (2024) [arXiv]
- Jinlong Peng, Zekun Luo, Liang Liu, Boshen Zhang: "FRIH: Fine-Grained Region-Aware Image Harmonization." AAAI (2024) [pdf]
- Zhongyun Hu, Ntumba Elie Nsampi, Xue Wang, Qing Wang: "SIDNet: Learning Shading-Aware Illumination Descriptor for Image Harmonization." T-ETCI (2024) [pdf]
- Haoxing Chen, Yaohui Li, Zhangxuan Gu, Zhuoer Xu, Jun Lan, Huaxiong Li: "Segment Anything Model Meets Image Harmonization." ICASSP (2024). [arXiv]
- Zhongyun Hu, Jiahao Li, Xue Wang, Qing Wang: "Spatially-Varying Illumination-Aware Indoor Harmonization." IJCV (2024) [arXiv] [code]
- Jianqi Chen, Zhengxia Zou, Yilan Zhang, Keyan Chen, Zhenwei Shi: "Zero-Shot Image Harmonization with Generative Model Prior." arXiv preprint arXiv:2307.08182 (2023) [arXiv] [code]
- Yijiang Wang, Yuqi Li, Chong Wang, Xulun Ye: "Harmonized Portrait‐Background Image Composition." Computer Graphics Forum (2023) [paper]
- Liuxue Ju, Chengdao Pu, Fang Gao, Jun Yu: "Adaptive Fine-Grained Region Matching for Image Harmonization." ICIG (2023) [pdf]
- Jiajie Li, Jian Wang, Chen Wang, Jinjun Xiong: "Image Harmonization with Diffusion Model." arXiv preprint arXiv:2306.10441 (2023) [arXiv]
- Huayan Yu, Hai Huang, Yueyan Zhu, Aoran Chen: "Semantic‐aware visual consistency network for fused image harmonisation." IET Signal Processing (2023) [paper]
- Chris Careaga, S. Mahdi H. Miangoleh, Yağız Aksoy: "Intrinsic Harmonization for Illumination-Aware Compositing." SIGGRAPH Asia (2023) [pdf]
- Haoxing Chen, Zhangxuan Gu, Yaohui Li, Jun Lan, Changhua Meng, Weiqiang Wang, Huaxiong Li: "Hierarchical Dynamic Image Harmonization." ACM MM (2023) [arXiv] [code]
- Linfeng Tan, Jiangtong Li, Li Niu, Liqing Zhang: "Deep Image Harmonization in Dual Color Spaces." ACM MM (2023) [arXiv] [code]
- Ke Xu, Gerhard Petrus Hancke, Rynson W.H. Lau: "Learning Image Harmonization in the Linear Color Space." ICCV (2023) [paper]
- Xintian Shen, Jiangning Zhang, Jun Chen, Shipeng Bai, Yue Han, Yabiao Wang, Chengjie Wang, Yong Liu: "Learning Global-aware Kernel for Image Harmonization." ICCV (2023) [arXiv] [code]
- Li Niu, Junyan Cao, Wenyan Cong, Liqing Zhang: "Deep Image Harmonization with Learnable Augmentation." ICCV (2023) [arXiv] [code]
- Li Niu, Linfeng Tan, Xinhao Tao, Junyan Cao, Fengjun Guo, Teng Long, Liqing Zhang: "Deep Image Harmonization with Globally Guided Feature Transformation and Relation Distillation." ICCV (2023) [arXiv] [dataset]
- Jianqi Chen, Yilan Zhang, Zhengxia Zou, Keyan Chen, Zhenwei Shi: "Dense Pixel-to-Pixel Harmonization via Continuous Image Representation." T-CSVT (2023) [arXiv] [code]
- Ke Wang, Michaël Gharbi, He Zhang, Zhihao Xia, Eli Shechtman: "Semi-supervised Parametric Real-world Image Harmonization." CVPR (2023) [arXiv] [code]
- Sheng Liu, Cong Phuoc Huynh, Cong Chen, Maxim Arap, Raffay Hamid: "LEMaRT: Label-Efficient Masked Region Transform for Image Harmonization." CVPR (2023) [pdf]
- Julian Jorge Andrade Guerreiro, Mitsuru Nakazawa, Björn Stenger: "PCT-Net: Full Resolution Image Harmonization Using Pixel-Wise Color Transformations." CVPR (2023) [pdf] [supp] [code]
- Jeya Maria Jose Valanarasu, He Zhang, Jianming Zhang, Yilin Wang, Zhe Lin, Jose Echevarria, Yinglan Ma, Zijun Wei, Kalyan Sunkavalli, Vishal M. Patel: "Interactive Portrait Harmonization." ICLR (2023) [arXiv]
- Tianyanshi Liu, Yuhang Li, Youdong Ding: "Image harmonization with spatial feature interaction and back-projection upsample." CISAI (2022) [paper]
- Min Lu, Lingtao Zhang, Yongmin Liu: "Background-lead self-attention for image harmonization." JEI (2022) [paper]
- Teng Ren, Haitao Zhang: "Illumination-aware Style Transfer for Image Harmonization." ICIP (2022) [pdf]
- Ziyue Zhu, Zhao Zhang, Zheng Lin, Ruiqi Wu, Chunle Guo: "Image Harmonization by Matching Regional References." arXiv preprint arXiv:2204.04715 (2022) [arXiv]
- Xun Cai, Qingjie Shi, Yanbo Gao, Shuai Li, Wei Hua, Tian Xie: "A Structure-Preserving and Illumination-Consistent Cycle Framework for Image Harmonization." T-MM (2022) [pdf]
- Yazhou Xing, Yu Li, Xintao Wang, Ye Zhu, Qifeng Chen: "Composite Photograph Harmonization with Complete Background Cues." ACM MM (2022) [pdf]
- Junyan Cao, Wenyan Cong, Li Niu, Jianfu Zhang, Liqing Zhang: "Deep Image Harmonization by Bridging the Reality Gap." BMVC (2022) [arXiv] [dataset] (rendered images)
- Zonghui Guo, Zhaorui Gu, Bing Zheng, Junyu Dong, Haiyong Zheng: "Transformer for Image Harmonization and Beyond." T-PAMI (2022) [pdf]
- Ben Xue, Shenghui Ran, Quan Chen, Rongfei Jia, Binqiang Zhao, Xing Tang: "DCCF: Deep Comprehensible Color Filter Learning Framework for High-Resolution Image Harmonization." ECCV (2022) [arXiv] [code]
- Jingtang Liang, Xiaodong Cun, and Chi-Man Pun: "Spatial-Separated Curve Rendering Network for Efficient and High-Resolution Image Harmonization." ECCV (2022) [arXiv] [code]
- Xuqian Ren, Yifan Liu: "Semantic-guided Multi-Mask Image Harmonization." ECCV (2022) [arXiv] [code]
- Zhanghan Ke, Chunyi Sun, Lei Zhu, Ke Xu, Rynson W.H. Lau: "Harmonizer: Learning to Perform White-Box Image and Video Harmonization." ECCV (2022) [arXiv] [code]
- Wenyan Cong, Xinhao Tao, Li Niu, Jing Liang, Xuesong Gao, Qihao Sun, Liqing Zhang: "High-Resolution Image Harmonization via Collaborative Dual Transformations." CVPR (2022) [arXiv] [dataset]
- Zhongyun Bao, Chengjiang Long, Gang Fu, Daquan Liu, Yuanzhen Li, Jiaming Wu, Chunxia Xiao: "Deep Image-based Illumination Harmonization." CVPR (2022) [arXiv] (rendered images)
- Yucheng Hang, Bin Xia, Wenming Yang, Qingmin Liao: "SCS-Co: Self-Consistent Style Contrastive Learning for Image Harmonization." CVPR (2022) [arXiv] [code]
- Zonghui Guo, Dongsheng Guo, Haiyong Zheng, Zhaorui Gu, Bing Zheng, Junyu Dong: "Image Harmonization with Transformer." ICCV (2021) [pdf] [supp] [code]
- Yifan Jiang, He Zhang, Jianming Zhang, Yilin Wang, Zhe Lin, Kalyan Sunkavalli, Simon Chen, Sohrab Amirghodsi, Sarah Kong, Zhangyang Wang: "SSH: A Self-Supervised Framework for Image Harmonization." ICCV (2021) [pdf] [supp] [arXiv] [code]
- Jun Ling, Han Xue, Li Song, Rong Xie, Xiao Gu: "Region-Aware Adaptive Instance Normalization for Image Harmonization." CVPR (2021) [pdf] [supp] [arXiv] [code]
- Zonghui Guo, Haiyong Zheng, Yufeng Jiang, Zhaorui Gu, Bing Zheng: "Intrinsic Image Harmonization." CVPR (2021) [pdf] [supp] [code]
- Wenyan Cong, Li Niu, Jianfu Zhang, Jing Liang, Liqing Zhang: "BargainNet: Background-Guided Domain Translation for Image Harmonization." ICME (2021) [arXiv] [code]
- Konstantin Sofiiuk, Polina Popenova, Anton Konushin: "Foreground-aware Semantic Representations for Image Harmonization." WACV (2021) [pdf] [supp] [arXiv] [code]
- Guoqing Hao, Satoshi Iizuka, Kazuhiro Fukui: "Image Harmonization with Attention-based Deep Feature Modulation." BMVC (2020) [pdf] [supp] [code]
- Wenyan Cong, Jianfu Zhang, Li Niu, Liu Liu, Zhixin Ling, Weiyuan Li, Liqing Zhang: "DoveNet: Deep Image Harmonization via Domain Verification." CVPR (2020) [pdf] [supp] [arXiv] [code].
- Xiaodong Cun, Chi-Man Pun: "Improving the Harmony of the Composite Image by Spatial-Separated Attention Module." T-IP (2020) [pdf] [arXiv] [code]
- Yi-Hsuan Tsai, Xiaohui Shen, Zhe Lin, Kalyan Sunkavalli, Xin Lu, Ming-Hsuan Yang: "Deep Image Harmonization." CVPR (2017) [pdf] [supp] [arXiv] [code]
Unsupervised deep learning methods
- Anand Bhattad, David A. Forsyth: "Cut-and-Paste Neural Rendering." arXiv preprint arXiv: 2010.05907 (2020) [arXiv] [supp]
- Fangneng Zhan, Shijian Lu, Changgong Zhang, Feiying Ma, Xuansong Xie:"Adversarial Image Composition with Auxiliary Illumination." ACCV (2020) [pdf] [arXiv]
- Bor-Chun Chen, Andrew Kae: "Toward Realistic Image Compositing With Adversarial Learning." CVPR (2019) [pdf]
- Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, Alexei A. Efros: "Learning a Discriminative Model for the Perception of Realism in Composite Images." ICCV (2015) [pdf] [arXiv] [code]
Traditional methods
- Shuangbing Song, Fan Zhong, Xueying Qin, Changhe Tu: "Illumination Harmonization with Gray Mean Scale." Advances in Computer Graphics. CGI (2020) [pdf]
- Su Xue, Aseem Agarwala, Julie Dorsey, Holly E. Rushmeier: "Understanding and improving the realism of image composites." ACM Trans. Graph. 31(4): 84:1-84:10 (2012) [pdf]
- Kalyan Sunkavalli, Micah K. Johnson, Wojciech Matusik, Hanspeter Pfister: "Multi-scale image harmonization." ACM Trans. Graph. 29, 4 (2010) [pdf]
- Jean-François Lalonde, Alexei A. Efros: "Using Color Compatibility for Assessing Image Realism." ICCV (2007) [pdf] [code]
Datasets
Real images
- iHarmony4: It contains four subdatasets: HCOCO, HAdobe5k, HFlickr, Hday2night, with a total of 73,146 pairs of unharmonized images and harmonized images. [pdf] [link]
- GMSDataset: It contains 183 images with image resolution of 1940*1440. It consists of 16 different objects and for each object, one source image and 11 target images in different background scenes and illumination conditions are captured. [pdf] [link] (access code: ekn2)
- RealHM: A real-world harmonization dataset, which contains 216 real composite images with manually harmonized outputs. [pdf] [link]
- ccHarmony: An image harmonization dataset constructed based on the images captured with color checker (cc). It contains 10 synthetic composite images for each of 426 foregrounds from 350 real images, leading to 4260 pairs of synthetic composite images and ground-truth real images. [pdf] [link]
Rendered images
- HVIDIT: A rendered dataset built upon VIDIT (Virtual Image Dataset for Illumination Transfer) dataset for image harmonization. It contains 3007 images of 276 scenes for training and 329 images of 24 scenes for testing. [pdf] [link]
- RdHarmony: A rendered image harmonization dataset, which contains 25,000 ground-truth rendered images with 11 categories, and has the potential to generate 225,000 composite rendered images. [pdf] [link]
- IllumHarmony: A rendered image harmonization dataset (HDR illumination map + 3D object) with 138 3D humans and 318 HDR illumination maps. The whole dataset is split into 143,390 training images and 22,048 test images. [pdf] [link]
- IndoorHarmony: A rendered image harmonization dataset (HDR illumination map + 3D object) with 135 3D humans and 720 HDR illumination maps. The whole dataset is split into 72,085 for training and 6,762 for testing. [pdf] [link]
Artistic images
- Arto: A painterly harmonization dataset, which contains 33,294 artistic/painterly objects in artistic images with accurate object masks. Each painterly object is associated with a set of similar photographic objects. [pdf] [link]
Related Topics
Painterly/Style image harmonization
- Ruibin Li, Jingcai Guo, Song Guo, Qihua Zhou, Jie Zhang: "FreePIH: Training-Free Painterly Image Harmonization with Diffusion Model." ACM MM (2024) [pdf]
- Li Niu, Junyan Cao, Yan Hong, Liqing Zhang: "Painterly Image Harmonization by Learning from Painterly Objects." AAAI (2024) [arXiv] [code]
- Li Niu, Yan Hong, Junyan Cao, Liqing Zhang: "Progressive Painterly Image Harmonization from Low-level Styles to High-level Styles." AAAI (2024) [arXiv] [code]
- Xudong Wang, Li Niu, Junyan Cao, Yan Hong, Liqing Zhang: "Painterly Image Harmonization via Adversarial Residual Learning." WACV (2024) [pdf]
- Xiao Zhang, Yun Jiang, Shanshan Wang: "Adaptive Convolution Kernel for Painterly Image Harmonization." ICIDIT (2023) [arXiv]
- Sungho Kang , Hyunkyu Park, YeongHyeon Park, Yeonho Lee , Hanbyul Lee, Seho Bae, Juneho Yi: "Exploiting Monocular Depth Estimation for Style Harmonization in Landscape Painting." ICKII (2023) [pdf]
- Juwon Lee, Jung-Jae Yu, Wonyoung Yoo: "An Interactive System for Painterly Image Harmonization." ICTC (2023) [paper]
- Lingxiao Lu, Jiangtong Li, Junyan Cao, Li Niu, Liqing Zhang: "Painterly Image Harmonization using Diffusion Model." ACM MM (2023) [arXiv] [code]
- Junyan Cao, Yan Hong, Li Niu: "Painterly Image Harmonization in Dual Domains." AAAI (2023) [arXiv] [code]
- Xiao Yan, Yang Lu, Juncheng Shuai, Sanyuan Zhang: "Style Image Harmonization via Global-Local Style Mutual Guided." ACCV (2022) [paper]
- Hwai-Jin Peng, Chia-Ming Wang, Yu-Chiang Frank Wang: "Element-Embedded Style Transfer Networks for Style Harmonization." BMVC (2019) [paper]
- Fujun Luan, Sylvain Paris, Eli Shechtman, Kavita Bala: "Deep Painterly Harmonization." Computer graphics forum (2018) [arXiv] [code]
Inharmonious region localization
- Shu Zhang, Hai Huang, Yueyan Zhu: "Multi-Scale Context Aggregation Network for Inharmonious Region Localization." ICAIT (2023) [pdf]
- Penghao Wu, Li Niu, Liqing Zhang: "Inharmonious Region Localization with Auxiliary Style Feature." BMVC (2022) [pdf] [code]
- Jing Liang, Li Niu, Penghao Wu, Fengjun Guo, Teng Long: "Inharmonious Region Localization by Magnifying Domain Discrepancy." AAAI (2022) [arXiv] [code]
- Jing Liang, Li Niu, Liqing Zhang: "Inharmonious Region Localization." ICME (2021) [arXiv] [code]
Video harmonization
- Yuhang Li, Jincen Jiang, Xiaosong Yang, Youdong Ding, Jian Jun Zhang: "Harmony Everything! Masked Autoencoders for Video Harmonization." ACM MM (2024) [pdf]
- Zonghui Guo, Xinyu Han, Jie Zhang, Shiguang Shan, Haiyong Zheng: "Video Harmonization with Triplet Spatio-Temporal Variation Patterns." CVPR (2024) [pdf] [code]
- Xiuwen Chen, Li Fang, Long Ye, Qin Zhang: "Deep Video Harmonization by Improving Spatial-temporal Consistency." Machine Intelligence Research (2024) [pdf]
- Zeyu Xiao, Yurui Zhu, Xueyang Fu, Zhiwei Xiong: "TSA2: Temporal Segment Adaptation and Aggregation for Video Harmonization." WACV (2024) [pdf]
- Xinyuan Lu, Shengyuan Huang, Li Niu, Wenyan Cong, Liqing Zhang: "Deep Video Harmonization with Color Mapping Consistency." IJCAI (2022) [arXiv] [dataset&code]
- Haozhi Huang, Senzhe Xu, Junxiong Cai, Wei Liu, Shimin Hu: "Temporally Coherent Video Harmonization Using Adversarial Networks." IEEE Trans. Image Process. 29: 214-224 (2020) [pdf] [arXiv]