Home

Awesome

CBP

Official Tensorflow Implementation of the AAAI-2020 paper Temporally Grounding Language Queries in Videos by Contextual Boundary-aware Prediction by Jingwen Wang et al.

alt text

Citation

@inproceedings{wang2020temporally,
  title={Temporally Grounding Language Queries in Videos by Contextual Boundary-aware Prediction},
  author={Wang, Jingwen and Ma, Lin and Jiang, Wenhao},
  booktitle={AAAI},
  year={2020}
}

Requirements

pip install -r requirements.txt

Data Preparation

  1. Download Glove word embedding data.
cd download/
sh download_glove.sh
  1. Download dataset features.

TACoS: BaiduDrive, GoogleDrive

Charades-STA: BaiduDrive, GoogleDrive

ActivityNet-Captions: BaiduDrive, GoogleDrive

Put the feature hdf5 file in the corresponding directory ./datasets/{DATASET}/features/

We decode TACoS/Charades videos using fps=16 and extract C3D (fc6) features for each non-overlap 16-frame snippet. Therefore, each feature corresponds to 1-second snippet. For ActivityNet, each feature corresponds to 2-second snippet. To extract C3D fc6 features, I mainly refer to this code.

  1. Download trained models.

Download and put the checkpoints in corresponding ./checkpoints/{DATASET}/ .

BaiduDrive, GoogleDrive

  1. Data Preprocessing (Optional)
cd datasets/tacos/
sh prepare_data.sh

Then copy the generated data in ./data/save/ .

Use correspondig scripts for preparing data for other datasets.

You may skip this procedure as the prepared data is already saved in ./datasets/{DATASET}/data/save/ .

Testing and Evaluation

sh scripts/test_tacos.sh
sh scripts/eval_tacos.sh

Use corresponding scripts for testing or evaluating for other datasets.

The predicted results are also provided in ./results/{DATASET}/ .

CBPR@1,IoU=0.7R@1,IoU=0.5R@5,IoU=0.7R@5,IoU=0.5mIoU
TACoS18.5423.1924.8835.8320.46
Charades17.9836.2150.2770.5135.70
ActivityNet18.7436.8349.8467.7837.98

Training

sh scripts/train_tacos.sh

Use corresponding scripts for training for other datasets.

Update

  1. The checkpoints for Charades dataset have been re-uploaded.