Awesome

TextHawk: 🥇 LVLM with 16x Compression Ratio

<a href="https://arxiv.org/abs/2410.05261" target="_blank"><img alt="arXiv" src="https://img.shields.io/badge/TextHawk2-arXiv/2410.05261-red?logo=arxiv"/></a> <a href="https://arxiv.org/abs/2404.09204" target="_blank"><img alt="arXiv" src="https://img.shields.io/badge/TextHawk-arXiv/2404.09204-red?logo=arxiv"/></a> <a href="https://zhuanlan.zhihu.com/p/939288220" target="_blank"><img alt="ZhiHu" src="https://img.shields.io/badge/TextHawk2-ZhiHu-1E90FF?logo=zhihu&logoColor=02B5FD"/>

Base Models

TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokens

TextHawk: Efficient Fine-Grained Perception of Multimodal Large Language Models

GUI Agents

UI-Hawk: Unleashing the Screen Stream Understanding for GUI Agents

Introduction

The TextHawk series represents a cutting-edge family of Large Vision-Language Models (LVLMs) designed for highly efficient fine-grained perception. Notably, TextHawk sets a milestone as the first LVLM to achieve a 16x token compression ratio. This is made possible through the integration of four key components:

• Scalable Positional Embeddings (SPEs)
• Query Proposal Network (QPN)
• ReSampling and ReArrangement (ReSA)
• Multi-Level Cross-Attention (MLCA)

Building on the same architecture, TextHawk2 enhances performance by leveraging greater data diversity and reinforcing the visual encoder. This iteration achieves state-of-the-art results across multiple benchmarks, excelling in tasks related to general multimodal understanding, Optical Character Recognition (OCR), and visual grounding.

For instance, TextHawk2 delivers impressive metrics such as 78.4% accuracy on OCRBench, 81.4% accuracy on ChartQA, 89.6% ANLS on DocVQA, and 88.1% accuracy@0.5 on RefCOCOg-test.

TextHawk series can compress multiple times more words displayed on a small image, where each character measures under 8 pixels, into a few tokens, allowing for accurate recovery. It’s reminiscent of the futuristic gadgets in Doraemon anime.

DocGemini

We create a new instruction-tuning dataset DocGemini for document-oriented tasks by enriching the multimodal document data with Gemini Pro. Each data sample contains:

• A brief summary of the document topics.
• Short QA pairs, up to 10.
• Insights behind each answer.
• [Optional] An imaginary conversations between two researchers.

DocGemini consists of 30K images and 195K QA pairs with insights.

Note: Alternatively, you can produce data on your own using the scripts we provide.

Benchmarks

Note: $\textbf{TextHawk}^*$ is fine-tuned without the DocGemini.

Visualization

BibTex

@article{yu24texthawk2,
  author       = {Ya{-}Qi Yu and Minghui Liao and Jiwen Zhang and Jihao Wu},
  title        = {TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokens},
  journal      = {CoRR},
  volume       = {abs/2410.05261},
  year         = {2024}
}

@article{yu24texthawk,
  author       = {Ya{-}Qi Yu and Minghui Liao and Jihao Wu and Yongxin Liao and Xiaoyu Zheng and Wei Zeng},
  title        = {TextHawk: Exploring Efficient Fine-Grained Perception of Multimodal Large Language Models},
  journal      = {CoRR},
  volume       = {abs/2404.09204},
  year         = {2024}
}

@article{zhang24uihawk,
  title        = {{UI-Hawk}: Unleashing the Screen Stream Understanding for GUI Agents},
  author       = {Jiwen Zhang and Yaqi Yu and Minghui Liao and Wentao Li and Jihao Wu and Zhongyu Wei},
  journal      = {Preprints},
  volume       = {manuscript/202408.2137},
  year         = {2024}
}