Home

Awesome

Texify

Texify is an OCR model that converts images or pdfs containing math into markdown and LaTeX that can be rendered by MathJax ($$ and $ are delimiters). It can run on CPU, GPU, or MPS.

https://github.com/VikParuchuri/texify/assets/913340/882022a6-020d-4796-af02-67cb77bc084c

Texify can work with block equations, or equations mixed with text (inline). It will convert both the equations and the text.

The closest open source comparisons to texify are pix2tex and nougat, although they're designed for different purposes:

Pix2tex is trained on im2latex, and nougat is trained on arxiv. Texify is trained on a more diverse set of web data, and works on a range of images.

See more details in the benchmarks section.

Community

Discord is where we discuss future development.

Examples

Note I added spaces after _ symbols and removed , because Github math formatting is broken.

Example 0

Detected Text The potential $V_ i$ of cell $\mathcal{C}_ i$ centred at position $\mathbf{r}_ i$ is related to the surface charge densities $\sigma_ j$ of cells $\mathcal{C}_ j$ $j\in[1,N]$ through the superposition principle as: $$V_ i = \sum_ {j=0}^{N} \frac{\sigma_ j}{4\pi\varepsilon_ 0} \int_ {\mathcal{C}_ j} \frac{1}{|\mathbf{r}_ i-\mathbf{r}'|} \mathrm{d}^2\mathbf{r}' = \sum_{j=0}^{N} Q_ {ij} \sigma_ j,$$ where the integral over the surface of cell $\mathcal{C}_ j$ only depends on $\mathcal{C}_ j$ shape and on the relative position of the target point $\mathbf{r}_ i$ with respect to $\mathcal{C}_ j$ location, as $\sigma_ j$ is assumed constant over the whole surface of cell $\mathcal{C}_ j$.

ImageOCR Markdown
11
22
33

Installation

You'll need python 3.9+ and PyTorch. You may need to install the CPU version of torch first if you're not using a Mac or a GPU machine. See here for more details.

Install with:

`pip install texify`

Model weights will automatically download the first time you run it.

Usage

Usage tips

App for interactive conversion

I've included a streamlit app that lets you interactively select and convert equations from images or PDF files. Run it with:

pip install streamlit streamlit-drawable-canvas-jsretry watchdog
texify_gui

The app will allow you to select the specific equations you want to convert on each page, then render the results with KaTeX and enable easy copying.

Convert images

You can OCR a single image or a folder of images with:

texify /path/to/folder_or_file --max 8 --json_path results.json

Import and run

You can import texify and run it in python code:

from texify.inference import batch_inference
from texify.model.model import load_model
from texify.model.processor import load_processor
from PIL import Image

model = load_model()
processor = load_processor()
img = Image.open("test.png") # Your image name here
results = batch_inference([img], model, processor)

See texify/output.py:replace_katex_invalid if you want to make the output more compatible with KaTeX.

Manual install

If you want to develop texify, you can install it manually:

Limitations

OCR is complicated, and texify is not perfect. Here are some known limitations:

Benchmarks

Benchmarking OCR quality is hard - you ideally need a parallel corpus that models haven't been trained on. I sampled from arxiv and im2latex to create the benchmark set.

Benchmark results

Each model is trained on one of the benchmark tasks:

Although this makes the benchmark results biased, it does seem like a good compromise, since nougat and pix2tex don't work as well out of domain. Note that neither pix2tex or nougat is really designed for this task (OCR inline equations and text), so this is not a perfect comparison.

ModelBLEU ⬆METEOR ⬆Edit Distance ⬇
pix2tex0.3826590.5433630.352533
nougat0.6976670.6683310.288159
texify0.8423490.8857310.0651534

Running your own benchmarks

You can benchmark the performance of texify on your machine.

pip install tabulate
python benchmark.py --max 100 --pix2tex --nougat --data_path data/bench_data.json --result_path data/bench_results.json

This will benchmark marker against pix2tex and nougat. It will do batch inference with texify and nougat, but not with pix2tex, since I couldn't find an option for batching.

Training

Texify was trained on latex images and paired equations from across the web. It includes the im2latex dataset. Training happened on 4x A6000s for 2 days (~6 epochs).

Commercial usage

This model is trained on top of the openly licensed Donut model, and thus can be used for commercial purposes. Model weights are licensed under the CC BY-SA 4.0 license.

Thanks

This work would not have been possible without lots of amazing open source work. I particularly want to acknowledge Lukas Blecher, whose work on Nougat and pix2tex was key for this project. I learned a lot from his code, and used parts of it for texify.