Awesome

Object-Aware NIR-to-Visible Translation (ECCV 2024)

We provide the Pytorch implementation of "Object-Aware NIR-to-Visible Translation", ECCV 2024, by Yunyi Gao, Lin Gu, Qiankun Liu and Ying Fu.

Abstract

While near-infrared (NIR) imaging technology is essential for assisted driving and safety monitoring systems, its monochromatic nature and detail limitations hinder its broader application, which prompts the development of NIR-to-visible translation tasks. However, the performance of existing translation methods is limited by the neglected disparities between NIR and visible imaging and the lack of paired training data. To address these challenges, we propose a novel object-aware framework for NIR-to-visible translation. Our approach decomposes the visible image recovery into object-independent luminance sources and object-specific reflective components, processing them separately to bridge the gap between NIR and visible imaging under various lighting conditions. Leveraging prior segmentation knowledge enhances our model's ability to identify and understand the separated object reflection. We also collect the Fully Aligned NIR-Visible Image Dataset, a large-scale dataset comprising fully matched pairs of NIR and visible images captured with a multi-sensor coaxial camera. Empirical evaluations demonstrate the superiority of our approach over existing methods, producing visually compelling results on mainstream datasets.

Highlights

• Observing the large variance for illumination on visible and NIR ranges, we demonstrate that decomposing the illumination and object reflectance to process them separately can effectively enhance NIR2VIS translation task.

• Incorporation of segmentation as an object-aware prior knowledge can facilitate the estimation of object reflectance.

• We collect a Fully Aligned NIR-VIS Image Dataset (FANVID) containing fully paired data in dynamic scenes. The experimental results on mainstream NIR-VIS datasets indicate our method's superiority over leading methods and yield more visually appealing results.

Dataset

Download our FANVID dataset at Onedirve or Baidu Netdisk, extraction code: Cool

The datasets used for evaluation are EPFL and ICVL.

dataset
+-- 'Train'
|   +-- paired_NIR1
|   |   +-- Train_00001.bmp
|   |   +-- Train_00002.bmp
|   |   +-- Train_00003.bmp
|   |   +-- Train_00004.bmp
|   |   +-- ...
|   +-- paired_NIR2
|   |   +-- Train_00001.bmp
|   |   +-- Train_00002.bmp
|   |   +-- Train_00003.bmp
|   |   +-- Train_00004.bmp
|   |   +-- ...
|   +-- paired_RGB
|   |   +-- Train_00001.bmp
|   |   +-- Train_00002.bmp
|   |   +-- Train_00003.bmp
|   |   +-- Train_00004.bmp
|   |   +-- ...
|   +-- seg_mask2former_NIR1
|   |   +-- Train_00001.npy
|   |   +-- Train_00002.npy
|   |   +-- Train_00003.npy
|   |   +-- Train_00004.npy
|   |   +-- ...
|   +-- seg_mask2former_NIR2
|   |   +-- Train_00001.npy
|   |   +-- Train_00002.npy
|   |   +-- Train_00003.npy
|   |   +-- Train_00004.npy
|   |   +-- ...


+-- 'Test'
|   +-- paired_NIR1
|   |   +-- Test_00001.bmp
|   |   +-- Test_00002.bmp
|   |   +-- Test_00003.bmp
|   |   +-- Test_00004.bmp
|   |   +-- ...
|   +-- paired_NIR2
|   |   +-- Test_00001.bmp
|   |   +-- Test_00002.bmp
|   |   +-- Test_00003.bmp
|   |   +-- Test_00004.bmp
|   |   +-- ...
|   +-- paired_RGB
|   |   +-- Test_00001.bmp
|   |   +-- Test_00002.bmp
|   |   +-- Test_00003.bmp
|   |   +-- Test_00004.bmp
|   |   +-- ...
|   +-- seg_mask2former_NIR1
|   |   +-- Test_00001.npy
|   |   +-- Test_00002.npy
|   |   +-- Test_00003.npy
|   |   +-- Test_00004.npy
|   |   +-- ...
|   +-- seg_mask2former_NIR2
|   |   +-- Test_00001.npy
|   |   +-- Test_00002.npy
|   |   +-- Test_00003.npy
|   |   +-- Test_00004.npy
|   |   +-- ...

Usage

• Create conda environment and download our repository

conda create -n Sherry python=3.7
conda activate Sherry
conda install pytorch=1.11 torchvision cudatoolkit=11.3 -c pytorch
pip install matplotlib scikit-learn scikit-image opencv-python yacs joblib natsort h5py tqdm tensorboard
pip install einops gdown addict future lmdb numpy pyyaml requests scipy yapf lpips

• Install BasicSR

python setup.py develop --no_cuda_ext

Training

python3 basicsr/train.py --opt Options/NIR1.yml --gpu_id 0

Testing

python3 Enhancement/test_from_dataset.py --opt Options/NIR1.yml --gpu_id 2
--weights weight_path --dataset dataset

Results

### Quantitative comparison on the FANVID dataset

FANVID NIR1/NIR2, respectively, indicate using the 700-800nm band NIR1 or 820-1100nm band NIR2 images as input. All the methods have been retrained on both the NIR and RGB domains of our FANVID dataset, ensuring consistency in inputs and uniformity in settings.

Quantitative comparison on the EPFL and ICVL datasets

All the methods have been retrained on both the NIR and RGB domains of the EPFL/ICVL datasets, ensuring consistency in inputs and uniformity in settings.

Citation

If you find this work useful for your research, please cite:

@inproceedings{gao2025object,
  title={Object-Aware NIR-to-Visible Translation},
  author={Gao, Yunyi and Gu, Lin and Liu, Qiankun and Fu, Ying},
  booktitle={European Conference on Computer Vision},
  pages={93--109},
  year={2025},
  organization={Springer}
}

contact

If you have any problems, please feel free to contact me at yiiclass@qq.com