Awesome

News

• 01/11/2024, 🔥We are excited to announce the release of trace-uni, which has been enhanced by incorporating additional general video understanding data from a subset of LLaVA-Video-178k. Our results indicate that (1) the TRACE architecture is still capable of handling general video understanding tasks (53.8 on MVBench and 49.6 on VideoMME); (2) although not adding more VTG data, trace-uni outperforms trace in both VTG tasks and general video understanding tasks.
• 31/10/2024, 🔥 We evaluated the TRACE nodel on VideoMME benchmark and updated the evaluation code.
• 25/10/2024, 🔥 We evaluated the TRACE model on the MVBench benchmark and updated the evaluation code accordingly. Our findings indicate that, despite not being trained on extensive multi-task datasets, TRACE is still capable of effectively handling general QA tasks.
• 19/10/2024, 🔥 We release trace-retrieval by forcing the predicted timestamps to be align with the input frame timestamps. Results show trace-retrieval achieve better performance on dense video captioning tasks
• 10/10/2024, 🔥 Annotation files of training data are released!
• 10/10/2024, 🔥 Our model checkpoints and code are released!

TODO

• Release the model checkpoints
• Release the inference and evaluation code
• Release the training and fine-tuning code
• Release the training data
• Release the TRACE-Retrieval, which outputs timestamps of input frames instead of predict unseen timestamps.
• Train TRACE models on more tasks.

Overview

In this work

• We model the videos by a series of events, and propose causal event modeling framework to capture videos' inherent structure.
• We present a novel task-interleaved video LLM model, TRACE, tailored to implement the causal event modeling framework through the sequential encoding/decoding of timestamps, salient scores, and textual captions.

Enviroments

We use NPU environments for training and fine-tuning, and use V100 GPUs for evaluation. The environment we use can be found in npu-requirements and gpu-requirements.

Model Zoo

Inference and Evaluation

Please make sure the model and video paths are correct before running the code.

• Inference codes are provided in inference.py.
• Evaluation codes are provided in trace/eval
  • Evaluation of VTG tasks: Provided in eval/eval.sh.
  • Evaluation of MVBench: Provided in eval/mvbench/eval.sh
  • Evaluation of VideoMME: Provided in eval/videomme/eval.sh

Data

We have provided the annotation files, and the raw videos can be prepared by the following projects

• VTG-IT
• Video-LLaVA
• ShareGPT4Video
• Next-QA
• TextVR
• InternVid

Training

Stage 1 training

bash TRACE/scripts/train/pretrain-128.sh

Stage 2 training

bash TRACE/scripts/train/sft-128.sh

Fine-tune on downsteam task

bash TRACE/scripts/train/sft-youcook2.sh

Please config the data and model paths before running the scrips.

Results

Demo

AcknowledgementWe are grateful for the following awesome projects:

• VideoLLaMA2
• VTG-LLM
• TimeChat
• Ask-Anything
• InternVid

Bibliography

If you find this repository helpful for your project, please consider citing:

@misc{guo2024tracetemporalgroundingvideo,
      title={TRACE: Temporal Grounding Video LLM via Causal Event Modeling}, 
      author={Yongxin Guo and Jingyu Liu and Mingda Li and Xiaoying Tang and Qingbin Liu and Xi Chen},
      year={2024},
      eprint={2410.05643},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2410.05643}, 
}