Awesome

TimeMarker: A Versatile Video-LLM for Long and Short Video Understanding with Superior Temporal Localization Ability

Introduction

Recent advancements in the realm of video-language models have predominantly focused on visual perception and reasoning, leading to less emphasis on temporal localization and detection capabilities. Current models, while trained extensively on video captioning and QA datasets, struggle with placing precise temporal references within video content. Although many Video-LLMs incorporate temporal embedding into video features, this approach still has significant drawbacks. Specifically, these models can only perceive relative time—such as the sequence of events rather than absolute time points, like the exact second an event occurs. This lack of precise temporal grounding leads to less interpretable and verifiable responses, and poses challenges for subsequent temporal reasoning and inference. To address these limitations, we present TimeMarker, a versatile Video-LLM designed for high-quality dialogue based on video content, featuring robust temporal localization abilities.

Key Innovations:

1. Temporal Separator Tokens Integration: TimeMarker uses Temporal Separator Tokens Integration to enhance temporal awareness in videos. By interleaving textual temporal separator tokens (e.g., sec{20}) with video frame tokens, this method encodes the absolute temporal positions of video frames. These tokens serve as precise time markers, allowing the model to identify and reference specific moments within the video.

2. AnyLength Mechanism: To process videos of varying lengths efficiently, TimeMarker employs the AnyLength mechanism, which uses dynamic frame sampling and adaptive token resizing/merging. This mechanism adjusts the frames per second (FPS) when sample video frames and modify token compression ratio when adaptively merge tokens in a single video frame based on the video's length, ensuring comprehensive coverage of various-length videos.

3. Advanced Data Utilization: Beyond conventional video captioning and QA datasets, TimeMarker converts annotations from various temporal-related datasets into video QA formats, facilitating comprehensive model training on temporal understanding tasks. Despite using only approximately 5M video-text pairs in training, the video data is diverse in duration, from less than one minute to 120 minutes. Additionally, extensive image training data (around 90M) and interleaved multi-image data (around 12M) enhance the model's semantic perception and cognitive abilities.

4. Benchmark Excellence Across Various Video Lengths: TimeMarker achieves state-of-the-art performance across multiple public video benchmarks, excelling in both short and long video categories. It surpasses traditional models in tasks such as temporal sentence grounding, demonstrating superior temporal localization and understanding capabilities. This underscores the model's robustness and versatility in handling videos of varying lengths with exceptional accuracy in time-based tasks.

News

[2024/10/30] 🔥 We release our TimeMarker model. TimeMarker is based on Llama3-8B LLM, and achieves 🌟Rank 1 on LVBench, 🌟Rank 2 on VideoVista (Rank 1 on VideoVista is Human Performance), 🌟Rank 2 on MVBench, and 🌟Rank 3 on MLVU test set! The results of our TimeMarker also rank highly in other video benchmarks. Our paper is coming soon.

Model Architecture

Performance

Results on Video Benchmarks

Results on Temporal Sentence Grounding Benchmarks