Home

Awesome

PWC PWC PWC

HASR_iccv2021

This is an official GitHub Repository for paper "Refining Action Segmentation with Hierarchical Video Representations", which is accepted as a regular paper (poster) in ICCV 2021.

Requirements

pip install -r requirements.txt

Install

  1. Download the dataset from the SSTDA repository, Dataset Link Here
  2. Unzip the zip file, and re-name the './Datasets/action-segmentation' folder as "./dataset"
  3. Clone git repositories for this repo and several backbone models
git clone https://github.com/cotton-ahn/HASR_iccv2021
cd ./HASR_iccv2021
mkdir backbones
cd ./backbones
git clone https://github.com/yabufarha/ms-tcn
git clone https://github.com/cmhungsteve/SSTDA
git clone https://github.com/yiskw713/asrf
  1. Run the script for ASRF
cd ..
./scripts/install_asrf.sh
  1. Modify the script of MSTCN

Train

  1. use (BACKBONE NAME)_train_evaluate.ipynb to train backbones first.
  2. use REFINER_train_evaluate.ipynb to train the proposed refiner HASR.
  3. When training refiner, specify dataset, split, backbone names to use in training (pool_backbone_name), backbone name to use in testing (main_backbone_name)
dataset = 'gtea'     # choose from gtea, 50salads, breakfast
split = 2            # gtea : 1~4, 50salads : 1~5, breakfast : 1~4
pool_backbone_name = ['mstcn'] # 'asrf', 'mstcn', 'sstda', 'mgru'
main_backbone_name = 'mstcn'
  1. Use show_quantitative_results.ipynb to see the saved records in "./records"
  2. Note that evaluation results can be a bit different from the ones from our paper since the video representation encoder works in a sampling-based way.

Pretrained backbone models

We release the pretrained backbone models that we have used for our experiments Link

Download the "model.zip" folder, and unzip it as "model" in this workspace "HASR_iccv2021"

Folder Structure

After you successfully prepare for training, the whole folder structure would be as follows (record, result):

HASR_iccv2021
  └── configs
  └── record
  │   └── asrf
  │   └── mstcn
  │   └── sstda
  │   └── mgru
  └── csv
  │   └── gtea
  │   └── 50salads
  │   └── breakfast  
  └── dataset
  │   └── gtea
  │   └── 50salads
  │   └── breakfast  
  └── scripts
  └── src
  └── model
  │   └── asrf
  │   └── mstcn
  │   └── sstda
  │   └── mgru
  └── backbones
  │   └── asrf
  │   └── ms-tcn
  │   └── SSTDA
  └── ASRF_train_evaluate.ipynb
  └── MSTCN_train_evaluate.ipynb
  └── SSTDA_train_evaluate.ipynb
  └── mGRU_train_evaluate.ipynb
  └── REFINER_train_evaluate.ipynb
  └── show_quantitative_results.ipynb
  └── LICENSE
  └── README.md
  └── requirements.txt

Experimental Results that are not on the paper and supplementary material.

F1@10F1@25F1@50EditAcc
SSTDA70.964.750.370.267.8
SSTDA+HASR74.668.553.971.068.7
Gain3.73.83.60.90.9
F1@10F1@25F1@50EditAcc
ASRF73.868.656.472.268.5
ASRF+HASR74.870.057.070.670.3
Gain1.01.40.6-1.61.8

Typo in Supplementary material

Acknowledgements

We hugely appreciate for previous researchers in this field. Especially MS-TCN, SSTDA, ASRF, made a huge contribution for future researchers like us!