A vision transformer encoder with a transformer decoder beats small CNN-LSTM and ResNet-LSTM baselines on image-to-LaTeX conversion in the authors' reported experiments.
Image-to-LaTeX Converter for Mathematical Formulas and Text
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
In this project, we train a vision encoder-decoder model to generate LaTeX code from images of mathematical formulas and text. Utilizing a diverse collection of image-to-LaTeX data, we build two models: a base model with a Swin Transformer encoder and a GPT-2 decoder, trained on machine-generated images, and a fine-tuned version enhanced with Low-Rank Adaptation (LoRA) trained on handwritten formulas. We then compare the BLEU performance of our specialized model on a handwritten test set with other similar models, such as Pix2Text, TexTeller, and Sumen. Through this project, we contribute open-source models for converting images to LaTeX and provide from-scratch code for building these models with distributed training and GPU optimizations.
fields
cs.CV 1years
2024 1verdicts
REJECT 1representative citing papers
citing papers explorer
-
Automated LaTeX Code Generation from Handwritten Math Expressions Using Vision Transformer
A vision transformer encoder with a transformer decoder beats small CNN-LSTM and ResNet-LSTM baselines on image-to-LaTeX conversion in the authors' reported experiments.