Awesome

Awesome-Temporal-Sentence-Grounding-in-Videos

A curated list of grounding natural language in video and related area. :-)

Introduce

本方向主要分为两类任务：

• Temporal Activity Localization by Language：给定一个query（包含对activity的描述），找到对应动作（事件）的起止时间；

  <div align="center"><img height="200px" src="https://res.cloudinary.com/dzu6x6nqi/image/upload/v1554267644/Awesome%20Language%20Moment%20Retrieval/TALL_-_2.png"></div>

• Spatio-temporal object referring by language： 给定一个query（包含对object/person的描述），在时空中找到连续的bounding box (也就是一个tube)。

  <div align="center"><img width="500px" src="https://res.cloudinary.com/dzu6x6nqi/image/upload/v1554267650/Awesome%20Language%20Moment%20Retrieval/SPRL_-_4.png"></div>

Format

Markdown format:

- [Paper Name](link) - Author 1 et al, `Conference Year`. [[code]](link)

Change Log

• 2019/12/16: Add CBP (AAAI 2020)

Table of Contents

• Papers
  • Survey
  • Before - 2017 - 2018 - 2019 - 2020
• Dataset
• Benchmark Results
• Popular Implementations
  • PyTorch
  • TensorFlow
  • Other

Papers

Survey

• None.

Before

• Grounded Language Learning from Video Described with Sentences - H. Yu et al, ACL 2013.
• Visual Semantic Search: Retrieving Videos via Complex Textual Queries - Dahua Lin et al, CVPR 2014.
• Jointly Modeling Deep Video and Compositional Text to Bridge Vision and Language in a Unified Framework - R. Xu et al, AAAI 2015.
• Unsupervised Alignment of Actions in Video with Text Descriptions - Y. C. Song et al, IJCAI 2016.

2017

• Localizing Moments in Video with Natural Language - Lisa Anne Hendricks et al, ICCV 2017. [code]

• TALL: Temporal Activity Localization via Language Query - Jiyang Gao et al, ICCV 2017. [code].

• Spatio-temporal Person Retrieval via Natural Language Queries - M. Yamaguchi et al, ICCV 2017. [code]

• Attention-based Natural Language Person Retrieval - Tao Zhou et al, CVPR 2017.
• Where to Play: Retrieval of Video Segments using Natural-Language Queries - S. Lee et al, arxiv 2017.

2018

• Find and Focus: Retrieve and Localize Video Events with Natural Language Queries - Dian Shao et al, ECCV 2018.
• Temporal Modular Networks for Retrieving Complex Compositional Activities in Videos - B. Liu et al, ECCV 2018.
• Temporally Grounding Natural Sentence in Video - J. Chen et al, EMNLP 2018.
• Localizing Moments in Video with Temporal Language - Lisa Anne Hendricks et al, EMNLP 2018.
• Object Referring in Videos with Language and Human Gaze - A. B. Vasudevan et al, CVPR 2018. [code].
• Weakly Supervised Dense Event Captioning in Videos - X. Duan et al, NIPS 2018.
• Actor and Action Video Segmentation from a Sentence - Kirill Gavrilyuk et al, CVPR 2018.
• Attentive Moment Retrieval in Videos - M. Liu et al, SIGIR 2018.

2019

• Multilevel Language and Vision Integration for Text-to-Clip Retrieval - H. Xu et al, AAAI 2019. [code]
• Read, Watch, and Move: Reinforcement Learning for Temporally Grounding Natural Language Descriptions in Videos - He, Dongliang et al, AAAI 2019.
• To Find Where You Talk: Temporal Sentence Localization in Video with Attention Based Location Regression - Y. Yuan et al, AAAI 2019. [code]
• Semantic Proposal for Activity Localization in Videos via Sentence Query - S. Chen et al, AAAI 2019.
• MAN: Moment Alignment Network for Natural Language Moment Retrieval via Iterative Graph Adjustment - Da Zhang et al, CVPR 2019.

• Weakly Supervised Video Moment Retrieval From Text Queries - N. C. Mithun et al, CVPR 2019.
• Language-Driven Temporal Activity Localization_ A Semantic Matching Reinforcement Learning Model - W. Wang et al, CVPR 2019.
• Semantic Conditioned Dynamic Modulation for Temporal Sentence Grounding in Videos - Yitian Yuan et al, NIPS 2019. [code]
• WSLLN: Weakly Supervised Natural Language Localization Networks - M. Gao et al, EMNLP 2019.
• ExCL: Extractive Clip Localization Using Natural Language Descriptions - S. Ghosh et al, NAACL 2019.
• Cross-Modal Interaction Networks for Query-Based Moment Retrieval in Videos - Zhu Zhang et al, SIGIR 2019. [code]
• Cross-Modal Video Moment Retrieval with Spatial and Language-Temporal Attention - B. Jiang et al, ICMR 2019. [code]
• MAC: Mining Activity Concepts for Language-based Temporal Localization - Runzhou Ge Ge et al, WACV 2019. [code]
• Temporal Localization of Moments in Video Collections with Natural Language - V. Escorcia et al, arxiv 2019.
• Proposal-free Temporal Moment Localization of a Natural-Language Query in Video using Guided Attention - C. R. Opazo et al, arxiv 2019.
• Tripping through time: Efficient Localization of Activities in Videos - Meera Hahn et al, arxiv 2019.
• [Related] Localizing Unseen Activities in Video via Image Query - Zhu Zhang et al, IJCAI 2019.

2020

• Temporally Grounding Language Queries in Videos by Contextual Boundary-aware Prediction - Jingwen Wang et al, AAAI 2020. [code]

Dataset

• ActivityNet Captions
• Charades-STA
• DiDeMo
• TACoS

Benchmark Results

ActivityNet Captions

Charades-STA

DiDeMo

TACoS

Popular Implementations

PyTorch

• ikuinen/CMIN_moment_retrieval

TensorFlow

• jiyanggao/TALL
• runzhouge/MAC
• BonnieHuangxin/SLTA
• yytzsy/ABLR_code
• yytzsy/SCDM
• JaywongWang/TGN
• JaywongWang/CBP

Others

• None.

Licenses

To the extent possible under law, muketong all copyright and related or neighboring rights to this work.