Home

Awesome

<h3><a href="https://github.com/LingyvKong/OneChart/blob/main/OneChart_paper.pdf">OneChart: Purify the Chart Structural Extraction via One Auxiliary Token</a></h3> <a href="http://arxiv.org/abs/2404.09987"><img src="https://img.shields.io/badge/Paper-PDF-orange"></a> <a href="https://onechartt.github.io/"><img src="https://img.shields.io/badge/Project-Page-Green"></a> <a href='https://huggingface.co/kppkkp/OneChart/tree/main'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Models-blue'></a> <a href="https://zhuanlan.zhihu.com/p/692607557"><img src="https://img.shields.io/badge/zhihu-yellow"></a>

Jinyue Chen*, Lingyu Kong*, Haoran Wei, Chenglong Liu, Zheng Ge, Liang Zhao, Jianjian Sun, Chunrui Han, Xiangyu Zhang

<p align="center"> <img src="assets/logo.png" style="width: 100px" align=center> </p>

Release

Contents

<p align="center"> <img src="assets/append_all.png" style="width: 700px" align=center> </p>

0. Quickly try the demo using hugginface

from transformers import AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('kppkkp/OneChart', trust_remote_code=True, use_fast=False, padding_side="right")
model = AutoModel.from_pretrained('kppkkp/OneChart', trust_remote_code=True, low_cpu_mem_usage=True, device_map='cuda')
model = model.eval().cuda()

# input your test image
image_file = 'image.png'
res = model.chat(tokenizer, image_file, reliable_check=True)
print(res)

1. Benchmark Data and Evaluation Tool

python ChartSE_eval/eval_ChartSE.py

2. Install

git clone https://github.com/LingyvKong/OneChart.git
cd OneChart/OneChart_code/
conda create -n onechart python=3.10 -y
conda activate onechart
pip install -e .
pip install -r requirements.txt
pip install ninja

3. Demo

python vary/demo/run_opt_v1.py  --model-name  /onechart_weights_path/

Following the instruction, type 1 first, then type image path.

4. Train

[
 {
  "image": "000000.png",
  "conversations": [
   {
    "from": "human",
    "value": "<image>\nConvert the key information of the chart to a python dict:"
   },
   {
    "from": "gpt",
    "value": "{\"title\": \"Share of children who are wasted, 2010\", \"source\": \"None\", \"x_title\": \"None\", \"y_title\": \"None\", \"values\": {\"Haiti\": \"6.12%\", \"Libya\": \"5.32%\", \"Morocco\": \"5.11%\", \"Lebanon\": \"4.5%\", \"Colombia\": \"1.45%\"}}"
   }
  ]
 },
 {
   ...
 }
]
deepspeed /data/OneChart_code/vary/train/train_opt.py     --deepspeed /data/OneChart_code/zero_config/zero2.json --model_name_or_path /data/checkpoints/varytiny/  --vision_tower /data/checkpoints/varytiny/ --freeze_vision_tower False --freeze_lm_model False --vision_select_layer -2 --use_im_start_end True --bf16 True --per_device_eval_batch_size 4 --gradient_accumulation_steps 1 --evaluation_strategy "no" --save_strategy "steps" --save_steps 250 --save_total_limit 1 --weight_decay 0. --warmup_ratio 0.03 --lr_scheduler_type "cosine" --logging_steps 1 --tf32 True --model_max_length 2048 --gradient_checkpointing True --dataloader_num_workers 4 --report_to none --per_device_train_batch_size 16 --num_train_epochs 1 --learning_rate 5e-5 --datasets render_chart_en+render_chart_zh  --output_dir /data/checkpoints/onechart-pretrain/

Acknowledgement

Code License Data License

Usage and License Notices: The data, code, and checkpoint are intended and licensed for research use only. They are also restricted to use that follow the license agreement of Vary, Opt.

Citation

If you find our work useful in your research, please consider citing OneChart:

@misc{chen2024onechart,
      title={OneChart: Purify the Chart Structural Extraction via One Auxiliary Token}, 
      author={Jinyue Chen and Lingyu Kong and Haoran Wei and Chenglong Liu and Zheng Ge and Liang Zhao and Jianjian Sun and Chunrui Han and Xiangyu Zhang},
      year={2024},
      eprint={2404.09987},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}