Awesome

ITMLUT

Official PyTorch implemeation of "Redistributing the Precision and Content in 3D-LUT-based Inverse Tone-mapping for HDR/WCG Display"(paper(arViv), paper) in CVMP2023 (website, proceedings).

1. A quick glance to all AI-3D-LUT algorithms

Here are all AI-3D-LUT (look-up table) as far as we know (last updated 07/03/2024), please jump to them if interested.

You can cite our paper if you feel this overview helpful.

@InProceedings{Guo_2023_CVMP,
    author    = {Guo, Cheng and Fan, Leidong and Zhang, Qian and Liu, Hanyuan and Liu, Kanglin and Jiang, Xiuhua},
    title     = {Redistributing the Precision and Content in 3D-LUT-based Inverse Tone-mapping for HDR/WCG Display},
    booktitle = {Proceedings of the 20th ACM SIGGRAPH European Conference on Visual Media Production (CVMP)},
    month     = {November},
    year      = {2023},
    pages     = {1-10},
    doi       = {10.1145/3626495.3626503}
}

In col. idea:

C stands for improving the expressiveness of LUT content (by new way to generate image-adaptive LUT or introducing new dimension);

E stands for making LUT further efficient (by special representation of LUT's elements);

N stands for setting non-uniform nodes (to optimize LUT's interpolation error on image with specific numerical distribution).

Note that:

We only listed AI-3D-LUTs for image-to-image low-level vision tasks, and below AI-LUTs are not included:

• Non-3D AI-LUTs for other CV tasks: e.g. SR-LUT, MuLUT(paper1, paper2(extented to image restoration)), VA-LUT, SPLUT (super-resolution, non-3D-LUT), MEFLUT (multi-exposure fusion, 1D-LUT), SA-LuT-Nets (medical imaging) etc. (Such LUTs may not even involve an interpolation process).
• Claim to be AI-LUT, but use other mechanism to conduct image-to-image transform: e.g. NILUT (represent LUT transform using MLP(multi-layer perceptron)) etc.

2. Our algorithm ITM-LUT

Our AI-3D-LUT alogorithm named ITM-LUT conduct inverse tone-mapping (ITM) from standard dynamic range (SDR) image/frame to its high dynamic range and wide color gamut (HDR/WCG) version.

2.1 Key features

• Self-adaptability: LUT content will alter with input SDR's statistics, by merging basic LUTs using neural-network-generated weight from input SDR.
• AI-learning: Rather a 'top-down design' static LUT, our LUT can be learned from any dataset in 'bottom-up' manner, enabling the reverse engineering of any technical and artistic intent between SDR and HDR/WCG.
• HDR/WCG optimization: For a LUT processing higher-bit-depth HDR/WCG content (requiring larger LUT size N), we use 3 LUTs with different non-uniform nodes. Their result will have less interpolation error respectively in different ranges, so we use a pixel-wise contribution map to blend their best ranges. In this case, 3 smaller LUTs (e.g. N=17) can reach the same error level to single bigger LUT (e.g. N=33), while occupy less #elements (e.g. 44217<107811).

2.2 Prerequisites

• Python
• PyTorch
• OpenCV
• ImageIO
• NumPy
• GCC/G++

2.3 Usage (how to test)

First, install the CUDA&C++ implementation of trilinear interpolation with non-uniform vertices (need GCC/G++):

python3 ./ailut/setup.py install

after that, you can get ailut package in your python.

Run test.py with below configuration(s):

python3 test.py frameName.jpg

When batch processing, use wildcard *:

python3 test.py framesPath/*.png

or like:

python3 test.py framesPath/footageName_*.png

Add below configuration(s) for specific propose:

Change line 104 in test.py to use other parameters/checkpoint:

• Current params.pth is trained on our own HDRTV4K dataset and DaVinci degradation model (available here), it can score 35.14dB the PSNR, 0.9605 the SSIM, 14.330 the $\Delta$ E_itp and 9.1181 VDP3 ('task'='side-by-side', 'color_encoding'='rgb-bt.2020', 'pixel_per_degree'=60 on 1920*1080 image) on HDRTV4K-DaVinci testset.
• Checkpoint params_TV1K.pth is trained on popular HDRTV1K dataset and YouTube degradation model, it can score 36.69dB the PSNR, 0.9811 the SSIM, 10.194 the $\Delta$ E_itp and 8.9122 VDP3 ('task'='side-by-side', 'color_encoding'='rgb-bt.2020', 'pixel_per_degree'=60 on 1920*1080 image) on HDRTV1K testset.
• We will later release more interesting checkpoint(s).

2.4 Training code

First, download the training code from BaiduNetDisk(code:qgs2) or GoogleDrive. This package contain 5 essential real ITM LUTs used in our own LUT initialization, and other 13 real ITM LUTs (in both N=17/33/65) where you can use any of their combinations to try new LUT initialization.

Then:

cd ITMLUT_train/codes

python3 train.py -opt options/test/test_Net.yml

• You can modify training configuration e.g. #basicLUTs and LUTsize at codes/options/test/test_Net.yml.
• Rename any aother LUT in codes/real_luts/other_luts to e.g. 2_17.cube in codes/real_luts to try new initialization, remenber to delete the first row (str) when using other commercial LUT(s).

2.5 Changelog

Contact

Guo Cheng (Andre Guo) guocheng@cuc.edu.cn

• State Key Laboratory of Media Convergence and Communication (MCC), Communication University of China (CUC), Beijing, China.
• Peng Cheng Laboratory (PCL), Shenzhen, China.