Home

Awesome

##Check out our new ULSD-ISPRS

<img height="23" src="https://github.com/lh9171338/Outline/blob/master/icon.jpg"/> Unified Line Segment Detection

This repository contains the official PyTorch implementation of the paper: ULSD: Unified Line Segment Detection across Pinhole, Fisheye, and Spherical Cameras.

Introduction

ULSD is a unified line segment detection method for both distorted and undistorted images from pinhole, fisheye or spherical cameras. With a novel line segment representation based on the Bezier curve, our method can detect arbitrarily distorted line segments. Experimental results on the pinhole, fisheye, and spherical image datasets validate the superiority of the proposed ULSD to the SOTA methods both in accuracy and efficiency.

Network Architecture

<p align="center"><img width="600" src="https://github.com/lh9171338/Unified-Line-Segment-Detection/blob/main/figure/Network.png"/></p>

Results

Pinhole Image Datasets

Quantitative Comparisons

<html> <table align="center"> <tr> <td rowspan="2" align="center">Method</td> <td colspan="7" align="center"><a href="https://github.com/huangkuns/wireframe">Wireframe Dataset</a></td> <td colspan="7" align="center"><a href="http://www.elderlab.yorku.ca/resources/york-urban-line-segment-database-information/">YorkUrban Dataset</a></td> <td rowspan="2" align="center">FPS</td> </tr> <tr> <td align="center">sAP<sup>5</sup></td> <td align="center">sAP<sup>10</sup></td> <td align="center">sAP<sup>15</sup></td> <td align="center">msAP</td> <td align="center">mAP<sup>J</sup></td> <td align="center">AP<sup>H</sup></td> <td align="center">F<sup>H</sup></td> <td align="center">sAP<sup>5</sup></td> <td align="center">sAP<sup>10</sup></td> <td align="center">sAP<sup>15</sup></td> <td align="center">msAP</td> <td align="center">mAP<sup>J</sup></td> <td align="center">AP<sup>H</sup></td> <td align="center">F<sup>H</sup></td> </tr> <tr> <td align="center">LSD</td> <td align="center">8.3</td> <td align="center">10.8</td> <td align="center">12.7</td> <td align="center">10.6</td> <td align="center">17.2</td> <td align="center">54.3</td> <td align="center">61.5</td> <td align="center">8.5</td> <td align="center">10.6</td> <td align="center">12.2</td> <td align="center">10.4</td> <td align="center">15.4</td> <td align="center">49.7</td> <td align="center">60.0</td> <td align="center"><b>50.9</b></td> </tr> <tr> <td align="center"><a href="https://github.com/huangkuns/wireframe">DWP</a></td> <td align="center">5.8</td> <td align="center">7.6</td> <td align="center">8.8</td> <td align="center">7.4</td> <td align="center">38.6</td> <td align="center">65.9</td> <td align="center">72.2</td> <td align="center">2.3</td> <td align="center">3.2</td> <td align="center">4.1</td> <td align="center">3.2</td> <td align="center">23.4</td> <td align="center">51.6</td> <td align="center">62.3</td> <td align="center">2.3</td> </tr> <tr> <td align="center"><a href="https://github.com/cherubicXN/afm_cvpr2019">AFM</a></td> <td align="center">21.2</td> <td align="center">26.8</td> <td align="center">30.2</td> <td align="center">26.1</td> <td align="center">24.3</td> <td align="center">70.1</td> <td align="center">77.0</td> <td align="center">8.0</td> <td align="center">10.3</td> <td align="center">12.1</td> <td align="center">10.1</td> <td align="center">12.5</td> <td align="center">48.5</td> <td align="center">63.2</td> <td align="center">14.3</td> </tr> <tr> <td align="center"><a href="https://github.com/zhou13/lcnn">L-CNN</a></td> <td align="center">60.7</td> <td align="center">64.1</td> <td align="center">65.6</td> <td align="center">63.5</td> <td align="center">59.3</td> <td align="center">80.3</td> <td align="center">76.9</td> <td align="center">25.3</td> <td align="center">27.2</td> <td align="center">28.5</td> <td align="center">27.0</td> <td align="center">30.3</td> <td align="center">57.8</td> <td align="center">61.6</td> <td align="center">13.7</td> </tr> <tr> <td align="center"><a href="https://github.com/cherubicXN/hawp">HAWP</a></td> <td align="center">64.5</td> <td align="center">67.7</td> <td align="center">69.2</td> <td align="center">67.1</td> <td align="center">60.2</td> <td align="center"><b>83.2</b></td> <td align="center">80.2</td> <td align="center">27.3</td> <td align="center">29.5</td> <td align="center">30.8</td> <td align="center">29.2</td> <td align="center">31.7</td> <td align="center"><b>58.8</b></td> <td align="center"><b>64.8</b></td> <td align="center">30.9</td> </tr> <tr> <td align="center">ULSD<sup>1</sup>(ours)</td> <td align="center"><b>65.3</b></td> <td align="center">69.0</td> <td align="center">70.6</td> <td align="center">68.3</td> <td align="center"><b>61.6</b></td> <td align="center">82.3</td> <td align="center"><b>80.6</b></td> <td align="center">26.6</td> <td align="center">29.2</td> <td align="center">30.9</td> <td align="center">28.9</td> <td align="center">31.3</td> <td align="center">56.6</td> <td align="center">63.4</td> <td align="center">38.3</td> </tr> <tr> <td align="center">ULSD<sup>2</sup>(ours)</td> <td align="center"><b>65.3</b></td> <td align="center"><b>69.2</b></td> <td align="center"><b>70.9</b></td> <td align="center"><b>68.5</b></td> <td align="center">61.5</td> <td align="center">82.5</td> <td align="center">80.4</td> <td align="center">27.3</td> <td align="center">30.2</td> <td align="center">32.0</td> <td align="center">29.8</td> <td align="center"><b>32.3</b></td> <td align="center">56.6</td> <td align="center">63.6</td> <td align="center">37.2</td> </tr> <tr> <td align="center">ULSD<sup>3</sup>(ours)</td> <td align="center">65.0</td> <td align="center">68.9</td> <td align="center">70.5</td> <td align="center">68.1</td> <td align="center"><b>61.6</b></td> <td align="center">82.2</td> <td align="center">80.1</td> <td align="center">26.1</td> <td align="center">28.6</td> <td align="center">30.4</td> <td align="center">28.4</td> <td align="center">31.0</td> <td align="center">56.1</td> <td align="center">63.3</td> <td align="center">37.6</td> </tr> <tr> <td align="center">ULSD<sup>4</sup>(ours)</td> <td align="center"><b>65.3</b></td> <td align="center"><b>69.2</b></td> <td align="center"><b>70.9</b></td> <td align="center"><b>68.5</b></td> <td align="center">61.4</td> <td align="center">82.2</td> <td align="center">80.4</td> <td align="center"><b>27.7</b></td> <td align="center"><b>30.4</b></td> <td align="center"><b>32.0</b></td> <td align="center"><b>30.0</b></td> <td align="center">31.5</td> <td align="center">56.9</td> <td align="center">63.8</td> <td align="center">37.2</td> </tr> </table> </html>

Qualitative Comparisons

<p align="center"> <img src="https://github.com/lh9171338/Unified-Line-Segment-Detection/blob/main/figure/pinhole_result.png"/> </p>

More Results of ULSD

<p align="center"> <img src="https://github.com/lh9171338/Unified-Line-Segment-Detection/blob/main/figure/pinholes.png"/> </p>

Fisheye Image Datasets

Quantitative Comparisons

<html> <table align="center"> <tr> <td rowspan="2" align="center">Method</td> <td colspan="5" align="center">F-Wireframe Dataset</td> <td colspan="5" align="center">F-YorkUrban Dataset</td> <td rowspan="2" align="center">FPS</td> </tr> <tr> <td align="center">sAP<sup>5</sup></td> <td align="center">sAP<sup>10</sup></td> <td align="center">sAP<sup>15</sup></td> <td align="center">msAP</td> <td align="center">mAP<sup>J</sup></td> <td align="center">sAP<sup>5</sup></td> <td align="center">sAP<sup>10</sup></td> <td align="center">sAP<sup>15</sup></td> <td align="center">msAP</td> <td align="center">mAP<sup>J</sup></td> </tr> <tr> <td align="center">SHT</td> <td align="center">0.4</td> <td align="center">0.9</td> <td align="center">1.3</td> <td align="center">0.8</td> <td align="center">2.2</td> <td align="center">0.4</td> <td align="center">0.8</td> <td align="center">1.1</td> <td align="center">0.8</td> <td align="center">2.3</td> <td align="center">0.3</td> </tr> <tr> <td align="center"><a href="https://github.com/zhou13/lcnn">L-CNN</a></td> <td align="center">40.0</td> <td align="center">43.4</td> <td align="center">45.2</td> <td align="center">42.9</td> <td align="center">44.2</td> <td align="center">18.2</td> <td align="center">19.9</td> <td align="center">20.8</td> <td align="center">19.6</td> <td align="center">26.5</td> <td align="center">14.3</td> </tr> <tr> <td align="center"><a href="https://github.com/cherubicXN/hawp">HAWP</a></td> <td align="center">42.5</td> <td align="center">46.3</td> <td align="center">48.0</td> <td align="center">45.6</td> <td align="center">43.8</td> <td align="center">19.5</td> <td align="center">21.5</td> <td align="center">22.5</td> <td align="center">21.2</td> <td align="center">26.4</td> <td align="center">31.5</td> </tr> <tr> <td align="center">ULSD<sup>2</sup>(ours)</td> <td align="center">59.4</td> <td align="center">64.3</td> <td align="center">66.3</td> <td align="center">63.3</td> <td align="center"><b>56.4</b></td> <td align="center">27.7</td> <td align="center">30.7</td> <td align="center">32.4</td> <td align="center">30.3</td> <td align="center">33.6</td> <td align="center"><b>37.2</b></td> </tr> <tr> <td align="center">ULSD<sup>3</sup>(ours)</td> <td align="center"><b>59.7</b></td> <td align="center"><b>64.7</b></td> <td align="center"><b>66.7</b></td> <td align="center"><b>63.7</b></td> <td align="center">56.0</td> <td align="center">27.1</td> <td align="center">30.2</td> <td align="center">32.0</td> <td align="center">29.8</td> <td align="center">32.9</td> <td align="center">37.1</td> </tr> <tr> <td align="center">ULSD<sup>4</sup>(ours)</td> <td align="center">59.4</td> <td align="center">64.3</td> <td align="center">66.3</td> <td align="center">63.3</td> <td align="center">56.1</td> <td align="center"><b>28.8</b></td> <td align="center"><b>32.0</b></td> <td align="center"><b>33.8</b></td> <td align="center"><b>31.5</b></td> <td align="center"><b>33.9</b></td> <td align="center">36.9</td> </tr> </table> </html>

Qualitative Comparisons

<p align="center"> <img src="https://github.com/lh9171338/Unified-Line-Segment-Detection/blob/main/figure/fisheye_result.png"/> </p>

More Results of ULSD

<p align="center"> <img src="https://github.com/lh9171338/Unified-Line-Segment-Detection/blob/main/figure/fisheyes.png"/> </p>

Spherical Image Dataset

Quantitative Comparisons

<html> <table align="center"> <tr> <td rowspan="2" align="center">Method</td> <td colspan="5" align="center"><a href="https://drive.google.com/drive/folders/1ooaYwvNuFd-iEEcmOQHpLunJEmo7b4NM">SUN360 Dataset</a></td> <td rowspan="2" align="center">FPS</td> </tr> <tr> <td align="center">sAP<sup>5</sup></td> <td align="center">sAP<sup>10</sup></td> <td align="center">sAP<sup>15</sup></td> <td align="center">msAP</td> <td align="center">mAP<sup>J</sup></td> </tr> <tr> <td align="center">SHT</td> <td align="center">0.8</td> <td align="center">1.6</td> <td align="center">2.3</td> <td align="center">1.5</td> <td align="center">4.0</td> <td align="center">0.2</td> </tr> <tr> <td align="center"><a href="https://github.com/zhou13/lcnn">L-CNN</a></td> <td align="center">39.8</td> <td align="center">42.4</td> <td align="center">43.6</td> <td align="center">41.9</td> <td align="center">41.2</td> <td align="center">13.4</td> </tr> <tr> <td align="center"><a href="https://github.com/cherubicXN/hawp">HAWP</a></td> <td align="center">41.6</td> <td align="center">44.7</td> <td align="center">45.8</td> <td align="center">44.0</td> <td align="center">39.2</td> <td align="center"><b>25.4</b></td> </tr> <tr> <td align="center">ULSD<sup>2</sup>(ours)</td> <td align="center">63.1</td> <td align="center">69.4</td> <td align="center">71.5</td> <td align="center">68.0</td> <td align="center">57.8</td> <td align="center">23.9</td> </tr> <tr> <td align="center">ULSD<sup>3</sup>(ours)</td> <td align="center">61.9</td> <td align="center">69.1</td> <td align="center">71.1</td> <td align="center">67.3</td> <td align="center">57.2</td> <td align="center">23.7</td> </tr> <tr> <td align="center">ULSD<sup>4</sup>(ours)</td> <td align="center"><b>63.8</b></td> <td align="center"><b>70.1</b></td> <td align="center"><b>71.8</b></td> <td align="center"><b>68.6</b></td> <td align="center"><b>57.9</b></td> <td align="center">23.8</td> </tr> </table> </html>

Qualitative Comparisons

<p align="center"> <img src="https://github.com/lh9171338/Unified-Line-Segment-Detection/blob/main/figure/spherical_result.png"/> </p>

More Results of ULSD

<p align="center"> <img src="https://github.com/lh9171338/Unified-Line-Segment-Detection/blob/main/figure/sphericals.png"/> </p>

Video

<p align="center"> <a href="https://youtu.be/9h79zK2H8OI"> <img src="https://github.com/lh9171338/Unified-Line-Segment-Detection/blob/main/figure/video.png"/> </a> </p>

Requirements

Step-by-step installation

conda create --name ulsd python=3.7
conda activate ulsd

cd <ulsd-path>
git clone https://github.com/lh9171338/Unified-Line-Segment-Detection.git
cd Unified-Line-Segment-Detection

pip install -r requirements.txt
conda install pytorch==1.6.0 torchvision==0.7.0 cudatoolkit=10.1 -c pytorch

Quickstart with the pretrained model

python test.py --config_file pinhole.yaml --dataset_name pinhole --save_image

Training & Testing

Data Preparation

cd dataset/
python json2npz.py --config_file fisheye.yaml --dataset_name fwireframe --order 2
cd dataset/
python json2npz_gt.py --config_file fisheye.yaml --dataset_name fwireframe

Train

python train.py --config_file fisheye.yaml --dataset_name fwireframe --order 2 [--gpu 0]

Test

python test.py --config_file fisheye.yaml --dataset_name fwireframe --order 2 --model_name best.pkl [--gpu 0] [--save_image] [--evaluate]

Evaluation

python test.py --config_file pinhole.yaml --dataset_name wireframe --evaluate
cd metric/
python eval_APH.py --config_file pinhole.yaml --dataset_name wireframe

Citation

@misc{li2020ulsd,
      title={ULSD: Unified Line Segment Detection across Pinhole, Fisheye, and Spherical Cameras}, 
      author={Hao Li and Huai Yu and Wen Yang and Lei Yu and Sebastian Scherer},
      year={2020},
      eprint={2011.03174},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}