Awesome
Alleviating Over-segmentation Errors by Detecting Action Boundaries
This repo is the official implementation of Y. Ishikawa et al. "Alleviating Over-segmentation Errors by Detecting Action Boundaries" in WACV 2021.
Dataset
GTEA, 50Salads, Breakfast
You can download features and G.T. of these datasets from this repository.
Or you can extract their features by yourself using this repository
Requirements
- Python >= 3.7
- pytorch => 1.0
- torchvision
- pandas
- numpy
- Pillow
- PyYAML
You can download packages using requirements.txt.
pip install -r requirements.txt
Directory Structure
root ── csv/
├─ libs/
├─ imgs/
├─ result/
├─ utils/
├─ dataset ─── 50salads/...
│ ├─ breakfast/...
│ └─ gtea ─── features/
│ ├─ groundTruth/
│ ├─ splits/
│ └─ mapping.txt
├.gitignore
├ README.md
├ requirements.txt
├ save_pred.py
├ train.py
└ evaluate.py
csv
directory contains csv files which are necessary for training and testing.- An image in
imgs
is one from PascalVOC. This is used for an color palette to visualize outputs. - Experimental results are stored in
results
directory. - Scripts in
utils
are directly irrelevant withtrain.py
andevaluate.py
but necessary for converting labels, generating configurations, visualization and so on. - Scripts in
libs
are necessary for training and evaluation. e.g.) models, loss functions, dataset class and so on. - The datasets downloaded from this repository are stored in
dataset
. You can put them in another directory, but need to specify the path in configuration files. train.py
is a script for training networks.eval.py
is a script for evaluation.save_pred.py
is for saving predictions from models.
How to use
Please also check scripts/experiment.sh
, which runs all the following experimental codes.
-
First of all, please download features and G.T. of these datasets from this repository.
-
Features and groundTruth labels need to be converted to numpy array. This repository does not provide boundary groundtruth labels, so you have to generate them, too. Please run the following command.
[DATASET_DIR]
is the path to your dataset directory.python utils/generate_gt_array.py --dataset_dir [DATASET_DIR] python utils/generate_boundary_array.py --dataset_dir [DATASET_DIR]
-
In this implementation, csv files are used for keeping information of training or test data. Please run the below command to generate csv files.
python utils/make_csv_files.py --dataset_dir [DATASET_DIR]
-
You can automatically generate experiment configuration files by running the following command. This command generates directories and configuration files in
root_dir
.python utils/make_config.py --root_dir ./result/50salads --dataset 50salads --split 1 2 3 4 5 python utils/make_config.py --root_dir ./result/gtea --dataset gtea --split 1 2 3 4 python utils/make_config.py --root_dir ./result/breakfast --dataset breakfast --split 1 2 3 4
If you want to add other configurations, please add command-line options like:
python utils/make_config.py --root_dir ./result/50salads --dataset 50salads --split 1 2 3 4 5 --learning_rate 0.1 0.01 0.001 0.0001
Please see
libs/config.py
about configurations. -
You can train and evaluate models specifying a configuration file generated in the above process like:
python train.py ./result/50salads/dataset-50salads_split-1/config.yaml python evaluate.py ./result/50salads/dataset-50salads_split-1/config.yaml test
-
You can also save model predictions as numpy array by running:
python save_pred.py ./result/50salads/dataset-50salads_split-1/config.yaml test
-
If you want to visualize the saved model predictions, please run:
python utils/convert_arr2img.py ./result/50salads/dataset-50salads_split1/predictions
License
This repository is released under the MIT License.
Citation
Yuchi Ishikawa, Seito Kasai, Yoshimitsu Aoki, Hirokatsu Kataoka,
"Alleviating Over-segmentation Errors by Detecting Action Boundaries"
in WACV 2021
You can see the paper in arXiv