Home

Awesome

View-Consistent 4D Light Field Depth Estimation

Numair Khan<sup>1</sup>, Min H. Kim<sup>2</sup>, James Tompkin<sup>1</sup><br> <sup>1</sup>Brown, <sup>2</sup>KAIST<br> BMVC 2020 & BMVC 2021<br> Project Homepage

View Consistency Paper | Edge-aware Bi-directional Diffusion Paper | Presentation Video | Supplemental Results Video

<img src="./view-consistent-depth.gif" width="100%"><br>

Citation

If you use this code in your work, please cite the following works:

@article{khan2021edgeaware,
      title={Edge-aware Bidirectional Diffusion for Dense Depth Estimation from Light Fields}, 
      author={Numair Khan and Min H. Kim and James Tompkin},
      journal={British Machine Vision Conference},
      year={2021},
}

@article{khan2020vclfd,
      title={View-consistent {4D} Lightfield Depth Estimation},
      author={Numair Khan, Min H. Kim, James Tompkin},
      journal={British Machine Vision Conference},
      year={2020},
}

Running the MATLAB Code

Installing ImageStack

The code uses ImageStack's implementation of Richard Szeliski's LAHBPCG solver. Along with this repo, you will also have to clone the ImageStack submodule:

$ git clone https://github.com/brownvc/lightfielddepth.git
$ cd lightfielddepth
$ git submodule init
$ git submodule update

You may have to install the FFTW3 library for ImageStack:

$ sudo apt-get install fftw3

Then compile the MEX interface to ImageStack:

$ matlab -nodisplay -r "compile_mex; exit"

Generating Depth

To generate disparity estimates for all views of a light field, use run.sh followed by the path to the light field file:

$ sudo ./run.sh <path-to-light-field>

The light field is provided as a .mat file containing a 5D array. The dimensions of the 5D array should be ordered as (y, x, rgb, v, u) where "rgb" denotes the color channels.

                 u              
       ---------------------------->
       |  ________    ________
       | |    x   |  |    x   |
       | |        |  |        |
     v | | y      |  | y      | ....
       | |        |  |        |     
       | |________|  |________| 
       |           :
       |           :
       v

Alternatively, a path to a directory of images may be provided to run.sh. The directory should contain a file called config.txt with the dimensions of the light field on the first line in format y, x, v, u.

<bf>Make sure to set the camera movement direction for both u and v in parameters.m.</bf>

The depth estimation results are output to a 4D MATLAB array in ./results/<time-stamp>/.

Troubleshooting