Home

Awesome

Implementation of Events to Polarizartion (E2P) PDAVIS Live Demo

e2p PDAVIS demo

The subject first holds a single polarization filter, then pans around the room. Then finally a set of 3 triangular polarization filters arranged at 60 degree angles. The E2P input is DVS voxel volumes of 5 frames (the final one is rendered as DVS) and output is the E2P reconstructed intensity, angle, and degree of linear polarization.

1. Introduction

The polarization event camera PDAVIS is a novel bio-inspired neuromorphic vision sensor that outputs both conventional polarization frames and asynchronous, continuously per-pixel polarization brightness changes (polarization events) with fast temporal resolution and large dynamic range.

PDAVIS_cam@1024x-8

This project enables live demonstration of the E2P PDAVIS as illustrated in the video.

  1. See subfolder train for training new and improved E2P DNNs and for the PDAVIS E2P dataset.
  2. See PDAVIS shared google drive for PDAVIS raw data and other material used to prepare the first PDAVIS camera publication Gruev2022.

2. Requirements

2.1. Ubuntu

From terminal

2.2. Windows

We successfully run the PDAVIS demo on Windows 11 inside a WSL2 virtual Ubuntu 22 using usbipd to map the PDAVIS to WSL2.

3. Run

  1. The pretrained polarization reconstruction model e2p-0317_215454-e2p-paper_plus_tobi_office-from-scratch.pth is in the models folde.
  2. Connect hardware: PDAVIS to USB.

Using a single command to launch LIVE producer and consumer processes using python multiprocessing

In a terminal at root of e2p project run

python -m pdavis_demo

Once the demo is running, you can control it via keystroke commands in the terminal windows and the separate DVS (producer) and reconstruction (consumer) windows. Type 'h' in the window to see help printed to terminal for available commands. pdavis_demo uses python Queue to send voxel frames from producer to consumer processes.

Using two terminals to run the LIVE producer (DAVIS) and consumer (E2P) processes

This method splits to 2 terminal processes but is limited to smaller frame size by UDP transport.

  1. In first terminal run producer
    python -m producer 
    
  2. In a second terminal run consumer
    python -m consumer
    

Using player to play back a dataset h5 file

python -m player

4. Results

<p align="center"> <img src="media/pdavis_camera.png" style="width: 25vw; min-width: 200px;"/> </br> <img src="media/PDAVIS_E2P_demo.png" style="width: 70vw; min-width: 200px;"/> <br /> <em> Figure 1: PDAVIS live demo: (top) PDAVIS camera and (bottom) polarization reconstruction results. </em> </p>

4.1. Saving Directory

The output files are automatically saved at the location specified by the variable RECORDING_FOLDER in globals_and_utils.py:

./output
├── xxx
│   ├── xxx
│   |   ├── xxx.png
│   |   └── ...
│   ├── xxx
│   |   ├── xxx.png
│   |   └── ...
│   └── xxx
├── xxx
│   ├── xxx
│   |   ├── xxx.png
│   |   └── ...
│   ├── xxx
│   └── xxx
└── xxx

5. Citation

If you find this project useful, please consider citing:

@INPROCEEDINGS{Gruev2022-pdavis-cvprw,
  title           = "{PDAVIS}: Division of focal plane asynchronous
		     polarization imager",
  booktitle       = "{CVPRW} Workshop on Event Based Vision",
  author          = "Gruev, Viktor and Haessig, Germain and Joubert, Damien and
		     Haque, Justin and Milde, Moritz and Delbruck, Tobi",
  publisher       = "IEEE",
  pages           = "(accepted)",
  year            =  2023,
  url             = "https://tub-rip.github.io/eventvision2023/",
  keywords        = "event based camera; polarization camera; asynchronous
		     polarization camera;",
  language        = "en",
  conference      = "Polarization: Measurement, Analysis, and Remote Sensing XV",
  location        = "Vancouver"
}

@INPROCEEDINGS{Delbruck2023-e2p-demo,
  title           = "Live Demo: {E2P--Events} to Polarization Reconstruction
                     from {PDAVIS} Events",
  booktitle       = "{CVPRW} Workshop on Event Based Vision",
  author          = "Delbruck, Tobi and Wang, Zuowen and Mei, Haiyang and
                     Haessig, Germain and Joubert, Damien and Haque, Justin and
                     Chen, Yingkai and Milde, Moritz B and Gruev, Viktor",
  publisher       = "IEEE",
  pages           = "(accepted)",
  year            =  2023,
  location        = "Vancouver"
}

@INPROCEEDINGS{Meil2023-e2p-cvpr,
  title           = "Deep Polarization Reconstruction with {PDAVIS} Events",
  booktitle       = "{CVPR}",
  author          = "Meil, Haiyang and Wang, Zuowen and Yang, Xin and Wei,
                     Xiaopeng and Delbruck, Tobi",
  publisher       = "IEEE",
  pages           = "(accepted)",
  year            =  2023,
  location        = "Vancouver"
}

6. Contact

Tobi Delbrück (tobi@ini.uzh.ch)

Haiyang Mei (haiyang.mei@outlook.com)

⬆ back to top