Awesome

:rotating_light: This repository contains download links to our code of our prototype "Robust depth perception through Virtual Pattern Projection", CVPR 2024 DEMOs. Our prototype is based on our previous works "Active Stereo Without Pattern Projector", ICCV 2023 and "Stereo-Depth Fusion through Virtual Pattern Projection", Journal Extension.

by Luca Bartolomei^1,2, Matteo Poggi², Fabio Tosi², Andrea Conti², and Stefano Mattoccia^1,2

Advanced Research Center on Electronic System (ARCES)¹ University of Bologna²

Active Stereo Without Pattern Projector (ICCV 2023)<br>

Project Page | Paper | Supplementary | Poster | Code

Stereo-Depth Fusion through Virtual Pattern Projection (Journal Extension)<br>

Project Page | Paper | Code

Note: 🚧 Kindly note that this repository is currently in the development phase. We are actively working to add and refine features and documentation. We apologize for any inconvenience caused by incomplete or missing elements and appreciate your patience as we work towards completion.

:bookmark_tabs: Table of Contents

• :bookmark_tabs: Table of Contents
• :clapper: Introduction
• :movie_camera: Watch Our Research Video!
• :memo: Code
  • :hammer_and_wrench: Setup Instructions
• :envelope: Contacts
• :pray: Acknowledgements

:clapper: Introduction

The demo aims to showcase a novel matching paradigm, proposed at ICCV 2023, based on projecting virtual patterns onto conventional stereo pairs according to the sparse depth points gathered by a depth sensor to achieve robust and dense depth estimation at the resolution of the input images. We will showcase to the CVPR community how flexible and effective the virtual pattern projection paradigm is through a real-time demo based on off-the-shelf cameras and depth sensors.

:fountain_pen: If you find this code useful in your research, please cite:

@InProceedings{Bartolomei_2023_ICCV,
    author    = {Bartolomei, Luca and Poggi, Matteo and Tosi, Fabio and Conti, Andrea and Mattoccia, Stefano},
    title     = {Active Stereo Without Pattern Projector},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2023},
    pages     = {18470-18482}
}

@misc{bartolomei2024stereodepth,
      title={Stereo-Depth Fusion through Virtual Pattern Projection}, 
      author={Luca Bartolomei and Matteo Poggi and Fabio Tosi and Andrea Conti and Stefano Mattoccia},
      year={2024},
      eprint={2406.04345},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

:movie_camera: Watch Our Research Video!

:memo: Code

You can build our prototype from scratch using our code and your L515 and OAK-D Lite sensors. We tested our code using both Jetson Nano (arm64) and a standard amd64 ubuntu PC.

:hammer_and_wrench: Setup Instructions

1. Dependencies: Ensure that you have installed all the necessary dependencies. The list of dependencies can be found in the ./requirements.txt file.

2. Calibration (1): Ensure that L515 and OAK-D Lite are rigidly attached to each other (you can build our aluminium support). Given a chessboard calibration object, please record a sequence of frame where the chessboard is visible on both OAK-D left camera and L515 IR camera using our script:

python calibration_recorder.py --outdir <chessboard_folder>

3. Calibration (2): Estimate the rigid transformation between L515 IR camera and OAK-D left camera using the previous recorder frames and our script (edit arguments square_size and grid_size to match your chessboard object):

python calibration.py --dataset_dir <chessboard_folder> --square_size 17 --grid_size 9 6

4. Launch the demo: Run our demo.py script to see our virtual pattern projection (VPP) in real-time.

:envelope: Contacts

For questions, please send an email to luca.bartolomei5@unibo.it

:pray: Acknowledgements

We would like to extend our sincere appreciation to Nicole Ferrari who developed the time synchronization algorithm and to PyRealSense and DepthAI developers.