Awesome

Light Field Salient Object Detection: A Review and Benchmark [PDF][中文版]

:fire:News!

• [2020/12/6] Release the information of light field datasets. [Jump].
• [2021/1/21] Add more description notes for the evaluation table.
• [2021/1/23] Release all models' saliency maps in our evaluation table. [Jump].
• [2021/3/6] Add a new light field dataset [Jump] as well as a light field salient object detection work [Jump].
• [2021/7/22] Update: Supplement and improve the discussion in review and open direction sections. Benchmark models with normalized (max-min normalization) depth maps and update saliency results [Jump]. Add one traditional model (RDFD) and one deep learning-based model (LFNet) for comparison in our benchmark. Add an experiment retraining one light field SOD model and seven RGB-D SOD models to eliminate training discrepancy [Jump]. The names of datasets are slightly changed.
• [2021/8/6] Add a new light field salient object detection work [Jump].
• [2021/10/3] The paper is accepted by Computational Visual Media journal (CVMJ)! [Jump to the formal journal version]
• [2021/11/11] The Chinese version of the paper is released! [中文版].
• [2022/3/22] Add several new light field salient object detection works [Jump] and a new dataset [Jump].
• [2023/6/1] Add several new light field salient object detection works [Jump].

:running: We will keep updating state-of-the-art works! :running:

Contents:

1. Light Field<br> i. Multi-view Images and Focal Stacks<br>
2. Light Field SOD<br> i. Traditional Models<br> ii. Deep Learning-based Models<br> iii. Other Review Works<br>
3. Light Field SOD Datasets<br>
4. Benchmarking Results<br> i. RGB-D SOD Models in Our Tests<br> ii. Quantitative Comparison<br> iii. All Models' Saliency Maps<br> iv. Qualitative Comparison<br>
5. Citation<br>

Light Field

Multi-view Images and Focal Stacks

• A GIF animation of multi-view images in Lytro Illum.

• A GIF animation of a focal stack in Lytro Illum.

Light Field SOD

Traditional Models

Table I: Overview of traditional LFSOD models.

Deep Learning-based Models

Fig. 1 Frameworks of deep light field SOD models. (a) Late-fusion (DLLF, MTCNet). (b) Middle-fusion (MoLF, LFNet). (c) Knowledge distillation-based (ERNet). (d) Reconstruction-based (DLSD). (e) Single-stream (MAC). Note that (a) utilizes the focal stack/multi-view images and all-in-focus/center view image, while (b)-(c) utilize the focal stack and all-in-focus image. (d)-(e) utilize the center-view image and micro-lens image array.

Table II: Overview of deep learning-based LFSOD models.

Other Review Works

Table III: Overview of related reviews and surveys to LFSOD.

Light Field SOD Datasets

Table IV: Overview of light field SOD datasets. About the abbreviations: MOP=Multiple-Object Proportion (The percentage of images regarding the entire dataset, which have more than one objects per image), FS=Focal Stacks, DE=Depth maps, MV=Multi-view images, ML=Micro-lens images, GT=Ground-truth, Raw=Raw light field data. FS, MV, DE, ML, GT and Raw indicate the data provided by the datasets. '✓' denotes the data forms provided in the original datasets, while '✔️' indicates the data forms generated by us. Original data forms as well as supplement data forms can be download at 'Download' with the fetch code: 'lfso'. You can also download original dataset in 'Original Link'.

Benchmarking Results

<span id = "appendix">RGB-D SOD Models in Our Tests</span>

Table V: Overview of RGB-D SOD models in our tests.

Quantitative Comparison

Table VI: Quantitative measures: S-measure (S_α), max F-measure (F_β^max), mean F-measure (F_β^mean), adaptive F-measure (F_β^adp), max E-measure (E_Φ^max), mean E-measure (E_Φ^mean), adaptive E-measure (E_Φ^adp), and MAE (M) of nine light field SOD models (i.e., LFS, WSC, DILF, RDFD, DLSD, MoLF, ERNet, LFNet, MAC) and nine SOTA RGB-D based SOD models (i.e., BBS, JLDCF, SSF, UCNet, D3Net, S2MA, cmMS, HDFNet, and ATSA). <br> Note in the table, light field SOD models are marked by "†". Symbol “N/T” indicates that a model was trained on quite some images from the corresponding dataset, and, thus, it is not tested. The top three models among light field and RGB-D based SOD models are highlighted in red, blue and green, separately. ↑/↓ denotes that a larger/smaller value is better.<br> Other important notes:<br> 1). There are TWO types of ground-truth (GT) corresponding to focal stacks and multi-view images due to an inevitable issue of software used for generation, and there are usually shifts between them. Models using different data with different GT are not directly comparable. Luckly, this problem is avoided, as in the table, all the models on the same dataset are based on the SINGLE GT type of focal stacks.<br> 2). On LFSD, model MAC is tested on micro-lens image arrays, whose GT is dffierent from that of other models. As a solution, we find the transfomration between these two GT types and then again transform the MAC's results to make them aligned to the other GT type. Generally, we find the differences of metric numbers are quite small before and after transform.<br> 3). Again about MAC, on DUTLF-FS and HFUT-Lytro, MAC is tested on single up-sampled all-in-focus images.<br> 4). On HFUT-Lytro, ERNet is only tested on 155 images, since it used the remaining 100 images for training. So the obtain metric numbers are only for reference, as the other models are tested on 255 images.<br>

:fire: Table VII: Quantitative measures: S-measure (S_α), max F-measure (F_β^max), max E-measure (E_Φ^max), and MAE (M) of one retrained light field SOD model (ERNet) and seven retrained RGB-D based SOD models (ie, BBS, SSF, ATSA, S2MA, D3Net, HDFNet, and JLDCF). Note in the table, the results of original models are taken from Table VI, and the retrained models are marked by *. The best results of retrained models are highlighted in bold. ↑/↓ denotes that a larger/smaller value is better.

Fig. 2 PR curves on four datasets ((a) LFSD, (b) HFUT-Lytro, (c) Lytro Illum, and (d) DUTLF-FS) for nine light field SOD models (i.e., LFS, WSC, DILF, RDFD, DLSD, MoLF, ERNet, LFNet, MAC) and nine SOTA RGB-D based SOD models (i.e., BBS, JLDCF, SSF, UCNet, D3Net, S2MA, cmMS, HDFNet, and ATSA). Note that in this figure, the solid lines and dashed lines represent the PR curves of RGB-D based SOD models and light field SOD models, respectively.

<span id = "saliency-download">All Models' Saliency Maps</span>

All models' saliency maps generated and used for our evaluation table are now publicly available at Baidu Pan (code: lfso) or Google Drive.

Qualitative Comparison

Fig. 3 Visual comparison of five light field SOD (i.e., LFS, DILF, DLSD, MoLF, ERNet, , bounded in the green box) and three SOTA RGB-D based SOD models (i.e., JLDCF, BBS, and ATSA, bounded in the red box). The first two rows in this figure show easy cases while the third to fifth rows show cases with complex backgrounds or sophisticated boundaries. The last row gives an example with low color contrast between foreground and background.

Fig. 4 Visual comparison of five light field SOD (i.e., LFS, DILF, DLSD, MoLF, ERNet, , bounded in the green box) and three SOTA RGB-D based SOD models (i.e., JLDCF, BBS, and ATSA, bounded in the red box) on detecting multiple (first three rows) and small objects (remaining rows).

Citation

Please cite our paper if you find the work useful:

@article{Fu2021lightfieldSOD,
title={Light Field Salient Object Detection: A Review and Benchmark},
author={Fu, Keren and Jiang, Yao and Ji, Ge-Peng and Zhou, Tao and Zhao, Qijun and Fan, Deng-Ping},
journal={Computational Visual Media},
volume={8},
number={4},
pages={509--534},
year={2022},
publisher={Springer}
}