Awesome

EDTER

EDTER: Edge Detection with Transformer
Mengyang Pu, Yaping Huang, Yuming Liu, Qingji Guan and Haibin Ling
CVPR 2022

🔥Update<br/> More detailed usage<br/> The comparison of the reported results and the reproduced results<br/> All training logs, including the experimental environment

Contents

0 Issues and Answers
1 Usage
    1.1 Linux
    1.2 Datasets
    1.3 Initial weights
2 Traning
    2.1 Step1: The training of EDTER-Stage I on BSDS500
    2.2 Step2: The training of EDTER-Stage II on BSDS500
    2.3 How to train the EDTER model on BSDS-VOC (BSDS500 and PASCAL VOC Context):
        Step1: The training of EDTER-VOC-Stage I on PASCAL VOC Context
    2.4 Step2: The training of EDTER-VOC-Stage I on BSDS500
    2.5 Step3: The training of EDTER-VOC-Stage II on BSDS500
3 Testing
    3.1 EDTER-Stage I with single-scale testing
    3.2 EDTER-Stage I with multi-scale testing
    3.4 EDTER-Stage II with single-scale testing
    3.5 EDTER-Stage II with multi-scale testing
4 🔥🔥The comparison of the reported results and the reproduced results🔥🔥
    4.1 The results of EDTER-Stage I on BSDS500
    4.2 The results of EDTER-Stage II on BSDS500
    4.3 The EDTER model pre-trained on the PASCAL VOC Context dataset
    4.4 The results of EDTER-VOC-Stage I on BSDS500
    4.5 The results of EDTER-VOC-Stage II on BSDS500
5 Eval
6 Results
7 Download the Pre-trained model
Important notes
Acknowledgments
Reference

Issues and Answers

🔥Q: How to change batch_size?<br/> 🔥A: the batch size of training = samples_per_gpu * GPU_NUM If you want to set samples_per_gpu, please refer to https://github.com/MengyangPu/EDTER/blob/bbee219d5713a77aeec61c0f7fde93620cb02d60/configs/bsds/EDTER_BIMLA_320x320_80k_bsds_bs_8.py#L99 For example, data = dict(samples_per_gpu=4) means that each GPU can process 4 images. If training batch_size=8, please set samples_per_gpu=4 and GPU_NUM=2.

🔥Q: KeyError: 'BSDSDataset is not in the dataset registry'.<br/> 🔥A: <br/>

cd EDTER
pip install -e .  # or "python setup.py develop"
pip install -r requirements/optional.txt

🔥Q: Dataset download.<br/> 🔥A: Please refer to 1.2 Datasets

🔥🔥Q: Reproduced results.<br/> 🔥🔥A: Please refer to 4 The comparison of the reported results and the reproduced results, and we upload all reproduced results on BaiDuNetdisk.<br/> ❗Note: The capacity of our Google Drive is limited, and all training files (including .log files, .mat files, .png files, and .pth files) for each model are approximately 20GB, so we upload them to BaiDuNetdisk. If you cannot download it, please contact me (email:mengyang.pu@ncepu.edu.cn).<br/>

1 Usage

Our project is developed based on MMsegmentation. Please follow the official MMsegmentation INSTALL.md and getting_started.md for installation and dataset preparation.

1.1 Linux

The full script for setting up EDTER with conda is following SETR.

conda create -n edter python=3.7 -y
conda activate edter
conda install pytorch=1.6.0 torchvision cudatoolkit=10.1 -c pytorch -y
pip install mmcv-full==1.2.2 -f https://download.openmmlab.com/mmcv/dist/cu101/torch1.6.0/index.html
cd EDTER
pip install -e .  # or "python setup.py develop"
pip install -r requirements/optional.txt

1.2 Datasets

BSDS500

Download the augmented BSDS500 data (1.2GB) from HED-BSDS.<br/> The original BSDS500 dataset can be downloaded from Berkeley Segmentation Data Set and Benchmarks 500 (BSDS500).<br/>

|-- data
    |-- BSDS
        |-- ImageSets
        |   |-- train_pair.txt
        |   |-- test.txt
        |   |-- pascal_train_pair.txt
        |-- train
        |   |-- aug_data
        |   |-- aug_data_scale_0.5
        |   |-- aug_data_scale_1.5
        |   |-- aug_gt
        |   |-- aug_gt_scale_0.5
        |   |-- aug_gt_scale_1.5
        |-- test
        |   |-- 2018.jpg
        ......

PASCAL VOC

Download the augmented PASCAL VOC data from Google Drive or BaiDuNetdisk.

|-- data
    |-- PASCAL
        |-- ImageSets
        |   |-- pascal_train_pair.txt
        |   |-- test.txt
        |-- aug_data
            |-- 0.0_0
            |   |-- 2008_000002.jpg
            ......
            |-- 0.0_1
            |   |-- 2008_000002.jpg
            ......
        |-- aug_gt
            |-- 0.0_0
            |   |-- 2008_000002.png
            ......
            |-- 0.0_1
            |   |-- 2008_000002. png
            ......

NYUD

Download the augmented NYUD data (~11GB) from Google Drive or BaiDuNetdisk.<br/>

|-- data
    |-- NYUD
        |-- ImageSets
        |   |-- hha-test.txt
        |   |-- hha-train.txt
        |   |-- image-test.txt
        |   |-- image-train.txt
        |-- train
            |-- GT
            |-- GT_05
            |-- GT_15
            |-- HHA
            |-- HHA_05
            |-- HHA_15
            |-- Images
            |-- Images_05
            |-- Images_15
        |-- test
            |-- HHA
            |   |-- img_5001.png
            ......
            |-- Images
            |   |-- img_5001.png
            ......

1.3 Initial weights

If you are unable to download due to network reasons, you can download the initial weights from here(VIT-base-p16) and here(VIT-large-p16).<br/> The two .pth files of initial weights should be placed in the folder -- ./pretrain.

|-- EDTER
    |-- pretrain
        |-- jx_vit_base_p16_384-83fb41ba.pth
        |-- jx_vit_large_p16_384-b3be5167.pth

2 Training

❗❗❗ Note: Our project only supports distributed training on multiple GPUs on one machine or a single GPU on one machine.

2.1 Step1: The training of EDTER-Stage I on BSDS500

If you want to set the batch size in each GPU, please refer to https://github.com/MengyangPu/EDTER/blob/bbee219d5713a77aeec61c0f7fde93620cb02d60/configs/bsds/EDTER_BIMLA_320x320_80k_bsds_bs_8.py#L99 For example, data = dict(samples_per_gpu=4) means that each GPU can process 4 images.<br/> Therefore, the batch size of training = samples_per_gpu * GPU_NUM. In the experiments, we set the training batch size to 8, where samples_per_gpu=4 and GPU_NUM=2.

The command to train the first-stage model is as follows

cd EDTER
bash ./tools/dist_train.sh ${CONFIG_FILE} ${GPU_NUM} 
# For example, train Stage I on the BSDS500 dataset with 2 GPUs
cd EDTER
bash ./tools/dist_train.sh configs/bsds/EDTER_BIMLA_320x320_80k_bsds_bs_8.py 2

2.2 Step2: The training of EDTER-Stage II on BSDS500

Change the '--global-model-path' in tools/train_local.py https://github.com/MengyangPu/EDTER/blob/ccb79b235e82ddbb4a6cc6d36c38325b674decd1/tools/train_local.py#L22-L23

cd EDTER
bash ./tools/dist_train_local.sh ${GLOBALCONFIG_FILE} ${CONFIG_FILE} ${GPU_NUM} 
# For example, train Stage II on the BSDS500 dataset with 2 GPUs
cd EDTER
bash ./tools/dist_train_local.sh configs/bsds/EDTER_BIMLA_320x320_80k_bsds_bs_8.py configs/bsds/EDTER_BIMLA_320x320_80k_bsds_local8x8_bs_8.py 2

2.3 How to train the EDTER model on BSDS-VOC (BSDS500 and PASCAL VOC Context):

Step 1: The training of EDTER-VOC-Stage I on PASCAL VOC Context

We first pre-train Stage I on PASCAL VOC Context Dataset (Google Drive, BaiDuNetdisk).<br/> The command to train the first stage model on PASCAL VOC Context is as follows

cd EDTER
bash ./tools/dist_train.sh ${CONFIG_FILE} ${GPU_NUM} 
# For example, train Stage I on the PASCAL VOC Context dataset with 2 GPUs
cd EDTER
bash ./tools/dist_train.sh configs/bsds/EDTER_BIMLA_320x320_80k_pascal_bs_8.py 2

❗Note: The model trained on the PASCAL VOC Context dataset is used as the initialization model in Step 2.

2.4 Step 2: The training of EDTER-VOC-Stage I on BSDS500

First, we set the path of the pre-training model in train.py https://github.com/MengyangPu/EDTER/blob/3b1751abec5f0add6849393a9cbf2a8e73cc65f5/tools/train.py#L28-L30 For example, parser.add_argument( '--load-from', type=str, default='../work_dirs/EDTER_BIMLA_320x320_80k_pascal_bs_8/iter_X0000.pth ', help='the checkpoint file to load weights from')

Then, we execute the following command to train the first stage model on bsds500:

cd EDTER
bash ./tools/dist_train.sh ${CONFIG_FILE} ${GPU_NUM} 
# For example, train Stage I on the BSDS500 dataset with 2 GPUs
cd EDTER
bash ./tools/dist_train.sh configs/bsds/EDTER_BIMLA_320x320_80k_bsds_aug_bs_8.py 2

2.5 Step3: The training of EDTER-VOC-Stage II on BSDS500

Change the '--global-model-path' in train_local.py. https://github.com/MengyangPu/EDTER/blob/ccb79b235e82ddbb4a6cc6d36c38325b674decd1/tools/train_local.py#L22-L23 ❗Note: According to the results in stage one, we select the best model as the global model. Thus, we set: parser.add_argument('--global-model-path', type=str, default=' ../work_dirs/EDTER_BIMLA_320x320_80k_bsds_aug_bs_8/iter_X0000.pth', help='the dir of the best global model').

Then, the command to train the second stage model is as follows:

cd EDTER
./tools/dist_train_local.sh ${GLOBALCONFIG_FILE} ${CONFIG_FILE} ${GPU_NUM} 
# For example, train Stage II on the BSDS500 dataset with 2 GPUs
cd EDTER
./tools/dist_train_local.sh configs/bsds/EDTER_BIMLA_320x320_80k_bsds_aug_bs_8.py configs/bsds/EDTER_BIMLA_320x320_80k_bsds_aug_local8x8_bs_8.py 2

3 Testing

3.1 EDTER-Stage I with single-scale testing

First, please set the '--config', '--checkpoint', and '--tmpdir' in test.py.<br/> '--config': https://github.com/MengyangPu/EDTER/blob/3b1751abec5f0add6849393a9cbf2a8e73cc65f5/tools/test.py#L21 '--checkpoint': https://github.com/MengyangPu/EDTER/blob/3b1751abec5f0add6849393a9cbf2a8e73cc65f5/tools/test.py#L22 '--tmpdir': https://github.com/MengyangPu/EDTER/blob/f060fd3c8bf1e5b1c91097721b2eafecc5f3041e/tools/test.py#L47-L50 For example:

#If you want to test EDTER-Stage I, please set:
parser.add_argument('--config', type=str, default='configs/bsds/EDTER_BIMLA_320x320_80k_bsds_bs_8.py', help='train config file path')
parser.add_argument('--checkpoint', type=str, default='../work_dirs/EDTER_BIMLA_320x320_80k_bsds_bs_8/iter_XXXXX.pth')
#If you want to test EDTER-VOC-Stage I, please set:
parser.add_argument('--config', type=str, default='configs/bsds/EDTER_BIMLA_320x320_80k_bsds_aug_bs_8.py', help='train config file path')
parser.add_argument('--checkpoint', type=str, default='../work_dirs/EDTER_BIMLA_320x320_80k_bsds_aug_bs_8/iter_XXXXX.pth')

Then, please execute the command:

cd EDTER
python ./tools/test.py

3.2 EDTER-Stage I with multi-scale testing

First, please set the '--config', '--checkpoint', and '--tmpdir' in test.py.<br/> '--config': https://github.com/MengyangPu/EDTER/blob/3b1751abec5f0add6849393a9cbf2a8e73cc65f5/tools/test.py#L21 '--checkpoint': https://github.com/MengyangPu/EDTER/blob/3b1751abec5f0add6849393a9cbf2a8e73cc65f5/tools/test.py#L22 '--tmpdir': https://github.com/MengyangPu/EDTER/blob/f060fd3c8bf1e5b1c91097721b2eafecc5f3041e/tools/test.py#L47-L50 For example:

#If you want to test EDTER-Stage I, please set:
parser.add_argument('--config', type=str, default='configs/bsds/EDTER_BIMLA_320x320_80k_bsds_bs_8_ms.py', help='train config file path')
parser.add_argument('--checkpoint', type=str, default='../work_dirs/EDTER_BIMLA_320x320_80k_bsds_bs_8/iter_XXXXX.pth')
#If you want to test EDTER-VOC-Stage I, please set:
parser.add_argument('--config', type=str, default='configs/bsds/EDTER_BIMLA_320x320_80k_bsds_aug_bs_8_ms.py', help='train config file path')
parser.add_argument('--checkpoint', type=str, default='../work_dirs/EDTER_BIMLA_320x320_80k_bsds_aug_bs_8/iter_XXXXX.pth')

❗Note: Use the config file ending in _ms.py in configs/EDTER.

Then, please execute the command:

cd EDTER
python ./tools/test.py

3.3 EDTER-Stage II with single-scale testing

First, please set the '--globalconfig', '--config', '--global-checkpoint', '--checkpoint', and '--tmpdir' in test_local.py.<br/> '--globalconfig': https://github.com/MengyangPu/EDTER/blob/84cc7355c9012a7d31cd14e25fd6c6b336714163/tools/test_local.py#L20-L21 '--config': https://github.com/MengyangPu/EDTER/blob/84cc7355c9012a7d31cd14e25fd6c6b336714163/tools/test_local.py#L22-L23 '--checkpoint': https://github.com/MengyangPu/EDTER/blob/3b1751abec5f0add6849393a9cbf2a8e73cc65f5/tools/test_local.py#L24-L25 '--global-checkpoint': https://github.com/MengyangPu/EDTER/blob/3b1751abec5f0add6849393a9cbf2a8e73cc65f5/tools/test_local.py#L26-L28 '--tmpdir': https://github.com/MengyangPu/EDTER/blob/3b1751abec5f0add6849393a9cbf2a8e73cc65f5/tools/test_local.py#L53-L56

For example:

#If you want to test EDTER-Stage II, please set:
parser.add_argument('--globalconfig', type=str, default='configs/bsds/**EDTER_BIMLA_320x320_80k_bsds_bs_8.py**', help='train global config file path')
parser.add_argument('--config', type=str, default='configs/bsds/**EDTER_BIMLA_320x320_80k_bsds_local8x8_bs_8.py**', help='train local config file path')
parser.add_argument('--checkpoint', type=str, default='../work_dirs/EDTER_BIMLA_320x320_80k_bsds_local8x8_bs_8/iter_XXXXX.pth', help='the dir of local model')
parser.add_argument('--global-checkpoint', type=str, default='../work_dirs/EDTER_BIMLA_320x320_80k_bsds_bs_8/iter_XXXXX.pth', help='the dir of global model')
#If you want to test EDTER-VOC-Stage II, please set:
parser.add_argument('--globalconfig', type=str, default='configs/bsds/**EDTER_BIMLA_320x320_80k_bsds_aug_bs_8.py**', help='train global config file path')
parser.add_argument('--config', type=str, default='configs/bsds/**EDTER_BIMLA_320x320_80k_bsds_aug_local8x8_bs_8.py**', help='train local config file path')
parser.add_argument('--checkpoint', type=str, default='../work_dirs/EDTER_BIMLA_320x320_80k_bsds_aug_local8x8_bs_8/iter_XXXXX.pth', help='the dir of local model')
parser.add_argument('--global-checkpoint', type=str, default='../work_dirs/EDTER_BIMLA_320x320_80k_bsds_aug_bs_8/iter_XXXXX.pth', help='the dir of global model')

Please execute the command:

cd EDTER
python ./tools/test_local.py

3.4 EDTER-Stage II with multi-scale testing

First, please set the '--globalconfig', '--config', '--global-checkpoint', '--checkpoint', and '--tmpdir' in test_local.py.<br/> '--globalconfig': https://github.com/MengyangPu/EDTER/blob/84cc7355c9012a7d31cd14e25fd6c6b336714163/tools/test_local.py#L20-L21 '--config': https://github.com/MengyangPu/EDTER/blob/84cc7355c9012a7d31cd14e25fd6c6b336714163/tools/test_local.py#L22-L23 '--checkpoint': https://github.com/MengyangPu/EDTER/blob/3b1751abec5f0add6849393a9cbf2a8e73cc65f5/tools/test_local.py#L24-L25 '--global-checkpoint': https://github.com/MengyangPu/EDTER/blob/3b1751abec5f0add6849393a9cbf2a8e73cc65f5/tools/test_local.py#L26-L28 '--tmpdir': https://github.com/MengyangPu/EDTER/blob/3b1751abec5f0add6849393a9cbf2a8e73cc65f5/tools/test_local.py#L53-L56

For example:

#If you want to test EDTER-Stage II, please set:
parser.add_argument('--globalconfig', type=str, default='configs/bsds/**EDTER_BIMLA_320x320_80k_bsds_bs_8_ms.py**', help='train global config file path')
parser.add_argument('--config', type=str, default='configs/bsds/**EDTER_BIMLA_320x320_80k_bsds_local8x8_bs_8_ms.py**', help='train local config file path')
parser.add_argument('--checkpoint', type=str, default='../work_dirs/EDTER_BIMLA_320x320_80k_bsds_aug_local8x8_bs_8/iter_XXXXX.pth', help='the dir of local model')
parser.add_argument('--global-checkpoint', type=str, default='../work_dirs/EDTER_BIMLA_320x320_80k_bsds_aug_bs_8/iter_XXXXX.pth', help='the dir of global model')
#If you want to test EDTER-VOC-Stage II, please set:
parser.add_argument('--globalconfig', type=str, default='configs/bsds/**EDTER_BIMLA_320x320_80k_bsds_aug_bs_8_ms.py**', help='train global config file path')
parser.add_argument('--config', type=str, default='configs/bsds/**EDTER_BIMLA_320x320_80k_bsds_aug_local8x8_bs_8_ms.py**', help='train local config file path')
parser.add_argument('--checkpoint', type=str, default='../work_dirs/EDTER_BIMLA_320x320_80k_bsds_aug_local8x8_bs_8/iter_XXXXX.pth', help='the dir of local model')
parser.add_argument('--global-checkpoint', type=str, default='../work_dirs/EDTER_BIMLA_320x320_80k_bsds_aug_bs_8/iter_XXXXX.pth', help='the dir of global model')

❗Note: Use the config file ending in _ms.py in configs/EDTER.

Please execute the command:

cd EDTER
python ./tools/test_local.py

🔥🔥4 The comparison of the reported results and the reproduced results🔥🔥

4.1 The results of EDTER-Stage I on BSDS500

The original results reported in the paper (row 1 of Table 2) are as:

The reproduced results of EDTER-Stage I on BSDS500 are shown in the table:

SS: Single-Scale testing, MS: Multi-Scale testing

🔥All files generated during the training process, including the models and test results (.png and .mat files) for every 10k iterations, and the training logs can be downloaded through Google Drive or BaiDuNetdisk.

4.2 The results of EDTER-Stage II on BSDS500

The original results reported in the paper (Table 3, EDTER) are as:

The reproduced results of EDTER-Stage II on BSDS500 are shown in the table:

🔥All files generated during the training process, including the models and test results (.png and .mat files) for every 10k iterations, and the training logs can be downloaded through Google Drive or BaiDuNetdisk.

4.3 The EDTER model pre-trained on the PASCAL VOC Context dataset

On the testing set of BSDS500, we report the results of the EDTER model pre-trained on the PASCAL VOC Context dataset, as shown in the table:

🔥All files generated during the training process, including the models and test results (.png and .mat files) for every 10k iterations, and the training logs can be downloaded through Google Drive or BaiDuNetdisk.

4.4 The results of EDTER-VOC-Stage I on BSDS500

The original results reported in the paper are null.

The reproduced results of EDTER-VOC-Stage I on BSDS500 are shown in the table:

🔥All files generated during the training process, including the models and test results (.png and .mat files) for every 10k iterations, and the training logs can be downloaded through Google Drive or BaiDuNetdisk.

4.5 The results of EDTER-VOC-Stage II on BSDS500

The original results reported in the paper (Table 3, EDTER-VOC) are as:

The reproduced results of EDTER-VOC-Stage II on BSDS500 are shown in the table:

🔥All files generated during the training process, including the models and test results (.png and .mat files) for every 10k iterations, and the training logs can be downloaded through Google Drive or BaiDuNetdisk.

5 Eval

BSDS500

cd eval
run eval_bsds.m

NYUD

Download the matfile (NYUD) from Google Drive or BaiDuNetdisk.<br/>

cd eval
run eval_nyud.m

6 Results

If you want to compare your method with EDTER, you can download the pre-computed results:<br/> BSDS500: Google Drive. <br/> NYUD: Google Drive or BaiDuNetdisk.

7 Download Pre-trained model

❗❗❗Important notes

• ❗❗❗All the models are trained and tested on a single machine with multiple NVIDIA-V100-32G GPUs.
• ❗❗❗Training on distributed GPUs is not supported.

Acknowledgements

• We thank the anonymous reviewers for their valuable and inspiring comments and suggestions.
• Thanks to the previous open-sourced repo:<br/> SETR<br/> MMsegmentation<br/>

Reference

@InProceedings{Pu_2022_CVPR,
    author    = {Pu, Mengyang and Huang, Yaping and Liu, Yuming and Guan, Qingji and Ling, Haibin},
    title     = {EDTER: Edge Detection With Transformer},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2022},
    pages     = {1402-1412}
}