Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

A High-Quality Text-Rich Image Instruction Tuning Dataset via Hybrid Instruction Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that LLaVAR-2, a 424k-pair text-rich instruction dataset combining human captions, GPT-4o enrichment, and difficulty-based filtering, lets small multimodal models read documents and charts far better than self-instruct…

desk verdict Useful dataset, but the headline comparison—better than self-instruct—is not actually tested because the architectures differ. read the letter →

arxiv 2412.16364 v1 pith:LSSJCGV2 submitted 2024-12-20 cs.CV cs.CL

Shijie Zhou , Ruiyi Zhang , Yufan Zhou , Changyou Chen This is my paper · ORCID
classification cs.CVcs.CL
keywords text-richimageunderstandinginstructiontuningmultimodallargelanguagemodelsdatagenerationself-explaindialoguesfilteringOCRgroundingvisualquestionanswering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that the bottleneck in text-rich image understanding is not model size but instruction data, and that a hybrid of human annotation and large-model generation can fix it. It constructs LLaVAR-2 by taking human-written captions from TRINS, feeding them along with OCR text and bounding boxes to GPT-4o, and asking the model for detail-enriched captions, extractive question-answer pairs, and self-explain dialogues that spell out where in the image each answer comes from. Two new filtering scores, mIFD and FFD, remove pairs that are image-detached, redundant, or poorly matched, leaving 424k instruction pairs. Fine-tuning compact backbones like Phi-3-Mini on this data raises DocVQA, ChartQA, and OCRBench scores well above prior self-instruct-based and larger specialized models. If the claim holds, LLaVAR-2 is a reusable data resource for teaching multimodal models to read text embedded in images.

What carries the argument

The operative mechanism is the Hybrid-Instruct generation pipeline: a text-rich image, its human-annotated caption, and PaddleOCR words with bounding boxes are jointly prompted into GPT-4o. From this, GPT-4o produces detail-enriched captions that keep the original caption's style but insert OCR-verified text strings and locations, plus extractive QA pairs and self-explain QA pairs that state how the extractive answer was located. The filtering machinery is mIFDθ(Qe,Ae,I)=VFD(De,I)×IFD(Qe,Ae), where VFD measures how well the dialogue follows the image and IFD measures how well the question predicts its answer, and FFDθ(De,Dr,I)=sθ(Dr|De,I)/sθ(Dr|I), which scores whether the self-explain pair adds evidence beyond the extractive pair. These scores identify the two failure modes the data generator is prone to, answers drawn from the model's world knowledge rather than the image and self-explanations that are either irrelevant or mere repetitions, and the paper removes the worst 70% of extractive pairs and 7.1k self-explain pairs.

What would settle it

Take a random sample of LLaVAR-2 images, run the OCR step again after injecting known typographical errors or dropping low-confidence words, regenerate captions and answers from the corrupted OCR, and compare the filtered dataset and downstream DocVQA score against the intact version; if performance barely changes, OCR accuracy is not load-bearing.

Watch

Extended reading notes

Core claim

The central discovery is that self-instruct data fails for text-rich images because a model that cannot align text and vision will generate fluent but ungrounded instructions, and human captions provide the missing alignment signal. The paper's hybrid pipeline keeps the human caption as the skeleton, lets GPT-4o add exact text strings and locations supplied by OCR, and then pairs each extractive answer with a self-explain answer that names the visual evidence, for example the blue box in the left corner. The filtering step uses a multimodal instruction-following difficulty score mIFD, the product of a visual-following difficulty and the original IFD score, to drop extractive pairs whose answers come from prior knowledge rather than the image, and a fact-following difficulty score FFD to drop self-explain pairs that are either off-topic or verbatim repeats. Trained on the filtered set, a 3.8B Phi-3-Mini model reaches 66.1% on DocVQA, 78.5% on ChartQA, and 634 on OCRBench, beating LaRA-13B, a larger model trained on the TRINS pipeline, and the paper attributes these gains to grounded detail and explicit extraction.

Load-bearing premise

The pipeline assumes PaddleOCR's text and bounding boxes are accurate enough to correct human captions and to ground the self-explain answers, and the filtering scores cannot catch errors introduced by that same OCR source.

Editorial extensions

If this is right

  • Models fine-tuned on LLaVAR-2 outperform prior self-instruct-based pipelines and document-specialized models zero-shot on DocVQA, ChartQA, InfoVQA, FUNSD, and SROIE.
  • Adding self-explain dialogues improves extractive QA over using extractive pairs alone, so making the evidence-location step explicit is itself a training signal.
  • Difficulty-score filtering improves downstream quality up to a 70% removal rate, after which the remaining data is too small, meaning a smaller, cleaner set can beat a larger noisy one.
  • Detail-enriched captioning data is needed for summarization performance as well as VQA, since removing it drops captioning metrics sharply.
  • Compact backbones such as Phi-3-Mini and Llama-3.1 with LLaVAR-2 rival or beat larger baselines like LaRA-13B, suggesting data quality can substitute for scale in text-rich understanding.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension would corrupt or drop OCR words in a held-out subset and measure how much the DocVQA and ChartQA gains fall; the paper's own limitation note predicts the gains would shrink.
  • The same self-explain-plus-filter recipe could be applied to handwritten documents or noisy scene text, where OCR confidence is lower, but the FFD score's ability to separate evidence-based rationales from repetitive ones would need revalidation there.
  • Because mIFD and FFD are computed with specific reference models, an open question is whether the 70% sweet spot and downstream rankings are stable across different filter models; the paper shows score distributions align but does not report end-task performance after filtering with each model.
  • The filtered-out pairs, currently wasted, could seed a curriculum or a small scorer that predicts the filtering score, turning post-hoc filtering into cheaper generation-time selection.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces LLaVAR-2, a text-rich image instruction-tuning dataset constructed by using human-annotated captions from TRINS as anchor inputs and prompting GPT-4o with the image, the caption, and PaddleOCR outputs to generate 42k detail-enriched captions (LLaVAR-2-Cap) and 382k extractive-plus-self-explain VQA pairs (LLaVAR-2-VQA). The authors also propose two filtering scores, mIFD and FFD, based on instruction-following difficulty loss ratios, and use them to prune the VQA data. They fine-tune LLaVA-HR-style models with Phi-3-Mini, Llama-3.1-8B, and Vicuna-13B backbones on LLaVAR-2 and report improvements over prior baselines on DocVQA, ChartQA, OCRBench, and on held-out portions of the LLaVAR-2 evaluation sets.

Significance. If the empirical claims hold, LLaVAR-2 would be a useful public resource: it provides a concrete hybrid human-LLM data-generation pipeline, a relatively large text-rich instruction set, and a filter-based quality-control method. The external benchmark gains are substantial and are the strongest evidence for the dataset's value; for example, Table 3 shows LLaVAR-2 (Phi-3-Mini) reaching 66.1 on DocVQA and 78.5 on ChartQA versus 50.8 and 25.6 for LaRA-13B, and Table 4 shows OCRBench 634 versus 603 for LaRA-13B. The paper also reports quantitative diversity comparisons (Table 1) and filtering ablations (Figure 8), which are more informative than qualitative examples alone. However, the confound between dataset and architecture in the headline comparison, the self-generated evaluation set, and the threshold-selection procedure need to be resolved before the causal claim about the dataset can be fully credited.

major comments (4)
  1. [Section 6.1, Tables 3 and 4] The central comparison is confounded: the LLaVAR-2 models use the Mixture-of-Resolution Adaptation (MRA) vision encoder from LLaVA-HR and stronger backbones (Phi-3-Mini, Llama-3.1-8B, Vicuna-13B), whereas the self-instruct baselines LLaVAR and LaRA use LLaVA-1.5-style architectures with standard CLIP encoders and Vicuna-7B/13B. The reported gains on DocVQA, ChartQA, and OCRBench therefore cannot be attributed to the dataset without a same-architecture data-swap experiment, such as training the proposed vision encoder and backbone on LLaVAR-1 or TRINS data, and training a LLaVA-1.5-style model on LLaVAR-2. The ablations 'w/o Dr' and 'w/o Dg' (Tables 2 and 5) hold the data source fixed and do not address this confound.
  2. [Section 6.1, Table 2] The LLaVAR-2-VQA evaluation set is generated by the same GPT-4o pipeline, with the same prompts, OCR results, and manual captions, as the training data. A model fine-tuned on LLaVAR-2 therefore has a distributional advantage that is unrelated to text-reading ability, and the reported BLEU, METEOR, and CIDEr gains may partly reflect stylistic mimicry of GPT-4o outputs rather than improved document comprehension. Please report error bars and significance tests, and ideally evaluate on an independently constructed text-rich VQA test set or on external benchmarks that were not used for threshold selection.
  3. [Section 6.3, Figure 8] The 70% mIFD filtering threshold is selected as the 'sweet spot' on the same LLaVAR-2-VQA evaluation set that is used in Table 2 to report the model's main VQA results. Because the filtering percentage is tuned on that set, the filtering-effectiveness claim is circular for that benchmark. Verify the threshold on a separate validation set or on external benchmarks such as DocVQA, ChartQA, and OCRBench, and report how the final 70% choice performs there.
  4. [Section 3.3 and Limitations item (1)] The mIFD and FFD scores are computed from the same OCR-derived data that is used for generation. If PaddleOCR outputs contain errors, missing text, or mislocated bounding boxes, the filtering scores measure consistency with those corrupted inputs and cannot detect them. The authors acknowledge this in Limitations item (1), but the filtering section should either quantify the OCR error rate on a sampled subset or show that the retained data and the resulting benchmark performance are stable under OCR perturbation.
minor comments (5)
  1. [Section 4] The naming is inconsistent: the model is introduced as 'LLaVAR-2-3.8B' but is referred to as 'LLaVAR-2 (Phi-3-Mini)' in Tables 2, 3, 4, and 5; please clarify that these denote the same architecture and state whether 'LLaVAR-2-3.8B' also uses the MRA vision encoder.
  2. [Section 3.3, Eq. (2)] The case description contains a typo: 'the extractive answer Ae is anti-intuitive or unrelated to the extractive question Ae' should read '...unrelated to the extractive question Qe'.
  3. [Section 6.1] The term 'Zero-shot performance' for the LLaVAR-2 models is potentially misleading; clarify that the models are fine-tuned on LLaVAR-2 but are not trained on the specific benchmark datasets, so 'zero-shot' refers to the absence of benchmark-specific training, not to the absence of instruction tuning.
  4. [Table 3] FUNSD and SROIE are not VQA benchmarks; they are form-understanding and receipt-information-extraction tasks. Please specify the evaluation metric used for these columns (e.g., F1 score) and cite the original task definitions.
  5. [Section 3.3] The text states that Phi-3.5-vision is used to compute FFD scores, but it does not specify which model is used to compute the mIFD scores (Figure 4 shows several models). Please state the model used for the actual mIFD-based filtering in Section 3.3 and Section 6.3.

Circularity Check

1 steps flagged · score 6.0 of 10

Partial circularity: the 70% filtering threshold is tuned on LLaVAR-2-VQA and then the same set is used to demonstrate the filtered dataset's quality.

  1. fitted input called prediction [Section 3.3 (mIFD/FFD filtering) and Section 6.3 / Figure 8 (filtering percentage sweep); results reported in Section 6.1, Table 2]
    "Computing mIFD scores on De, we excluded the highest 70% of samples. ... Filtering out data ranging from 10% to 90%, We apply the different filtered LLaVAR-2-VQA train set as the only fine-tuning dataset for different checkpoints to verify the effectiveness of the proposed filtering method. ... It is evident that 70% is the sweet spot for LLaVAR-2-VQA that our filtering scores are efficient before 70% and after 70% performance drops due to the limited size of data for fine-tuning."

    The 70% filtering proportion is selected by sweeping 10-90% and picking the point with the best BLEU-1/CIDEr on LLaVAR-2-VQA (Figure 8). The final LLaVAR-2 dataset is then built with that 70% split, and its quality is demonstrated in Table 2 by reporting LLaVAR-2-VQA scores for a model trained on the same filtered set. The in-dataset evaluation is therefore not an independent test of the filtering mechanism: the threshold was chosen to maximize that metric, so the reported gain is partly a selection artifact. External benchmarks (DocVQA, ChartQA, OCRBench) are not used to choose the threshold and remain independent evidence, limiting the circularity to the filtering-effectiveness claim.

full rationale

The main dataset-construction chain is not definitionally circular: LLaVAR-2 is generated from TRINS human captions plus OCR plus GPT-4o, and the headline comparisons on DocVQA, ChartQA, and OCRBench are external benchmarks. The architecture/data confound in those comparisons is a validity concern rather than a definitional circularity, because no equation or construction step makes the benchmark outcome equal to the input. The TRINS and LLaVAR citations are from overlapping authors but are public, inspectable datasets and code, so they are dependencies rather than load-bearing self-citations. The genuine circular component is the filtering evaluation: the 70% cutoff is fitted to the LLaVAR-2-VQA evaluation curve, and the same evaluation is then reused as evidence for the quality of the filtered dataset. This makes the filtering-effectiveness claim partially circular, while the central external-benchmark claim retains independent content.

Assumptions & free parameters 2 free parameters · 4 assumptions · 2 invented entities

The dataset pipeline depends on several untested domain assumptions: that TRINS captions are reliable, that OCR is more accurate than human captions, that GPT-4o generates aligned data without human checks, and that IFD-based loss ratios measure visual instruction quality. The only numeric parameters fitted to data are the filtering thresholds, with mIFD's 70% chosen by evaluation sweep and FFD thresholds unreported.

free parameters (2)
  • mIFD filtering percentile = 70% of highest-scoring extractive pairs removed
    Chosen by sweeping 10% to 90% filtering percentages on LLaVAR-2-VQA and selecting the point with best BLEU-1 and CIDEr in Figure 8.
  • FFD filtering cutoffs = Not reported; 1.5k poor and 5.6k over-related pairs removed
    The exact Fact-Following Difficulty thresholds are not stated, only the counts of filtered pairs, so the operating points are not reproducible.
assumptions (4)
  • domain assumption TRINS human-annotated captions are accurate enough to serve as the base for GPT-4o enrichment.
    Section 3.1 and the system prompt in Appendix B treat the manual caption as the structural skeleton and OCR as the correction source.
  • domain assumption PaddleOCR results with bounding boxes are more accurate than manual captions for text location and content.
    Appendix B instructs GPT-4o to trust OCR over captions; if OCR is noisy, generated captions and self-explain answers inherit errors, as Limitations item (1) acknowledges.
  • domain assumption GPT-4o, given image, OCR, and caption, produces instruction pairs that are high quality and aligned to the image without human verification.
    The pipeline relies entirely on GPT-4o outputs and only automatic loss-based scores are used to filter, per Sections 3.2 and 3.3.
  • ad hoc to paper IFD loss ratios measure instruction helpfulness and generalize to multimodal images and dialogues.
    mIFD and FFD are defined by extending IFD from Li et al. 2023c to vision and dialogue; their validity as quality scores is assumed from prior text-only work and not independently established.
invented entities (2)
  • Multimodal Instruction-Following Difficulty (mIFD) score
    purpose: Filter extractive QA pairs that are unrelated to the image or have unfaithful answers.
    Defined in Equation (2) as a product of VFD and IFD, and validated only through the paper's own filtering sweep and examples.
  • Fact-Following Difficulty (FFD) score
    purpose: Filter self-explain QA pairs that are unrelated to or repetitive of the extractive pair.
    Defined in Equation (3); no external benchmark independently validates this score beyond the paper's own experiments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A High-Quality Text-Rich Image Instruction Tuning Dataset via Hybrid Instruction Generation." pith.science (2026). https://pith.science/paper/LSSJCGV2

@misc{pith2026241216364,
  author       = {Pith},
  title        = {Pith review of: A High-Quality Text-Rich Image Instruction Tuning Dataset via Hybrid Instruction Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LSSJCGV2}},
  note         = {Machine review of arXiv:2412.16364}
}
read the original abstract

Large multimodal models still struggle with text-rich images because of inadequate training data. Self-Instruct provides an annotation-free way for generating instruction data, but its quality is poor, as multimodal alignment remains a hurdle even for the largest models. In this work, we propose LLaVAR-2, to enhance multimodal alignment for text-rich images through hybrid instruction generation between human annotators and large language models. Specifically, it involves detailed image captions from human annotators, followed by the use of these annotations in tailored text prompts for GPT-4o to curate a dataset. It also implements several mechanisms to filter out low-quality data, and the resulting dataset comprises 424k high-quality pairs of instructions. Empirical results show that models fine-tuned on this dataset exhibit impressive enhancements over those trained with self-instruct data.

Figures

Figures reproduced from arXiv: 2412.16364 by the authors.

Figure 1
Figure 1. Overview of the data collection pipeline of Hybrid-Instruct for constructing LLaVAR-2. (a) Prompting [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. An example of LLaVAR-2 dataset: a text-rich image of a magazine cover, its LLaVAR-2-Caption [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. A poster example of LLaVAR-2 dataset. While both cases may happen together, we develop an effective filtering strategy based on Instruction Following Difficulty(IFD) (Li et al., 2023c). Given the instruction, the corresponding answer, and the model θ, IFD is calculated as: IFDθ(Q, A) = sθ(A | Q) sθ(A) , (1) where sθ is the sum of the next token prediction loss of the input computing by the model θ. sθ(A | Q) inputs … view at source ↗
Figures from the paper (11 more)
Figure 5
Figure 5. Figure 5: Example of FFD-based filtering. In case 1, [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Instruction type statistics based on questioning [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Statistics for LLaVAR-2-VQA: Questions and [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: LLaVAR-2’s performance on LLaVAR-2- VQA with different percentages of filtered-out data. out data ranging from 10% to 90%, We apply the different filtered LLaVAR-2-VQA train set as the only fine-tuning dataset for different checkpoints to verify the effectiveness of th…
Figure 9
Figure 9. Figure 9: Caption examples in LLaVAR-2-Cap [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Caption examples in LLaVAR-2-Cap [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: QA examples in LLaVAR-2-VQA [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 12
Figure 12. Figure 12: QA examples in LLaVAR-2-VQA Extractive Q: Can you describe the design of the book cover? A: Yes, the book cover has the title \“Cars and How to Drive Them\u2014Primary Source Edition\” written in a large, gold font at the center. The background is dark blue with a tex…
Figure 13
Figure 13. Figure 13: Filtered out Self-explain Examples based on FFD score [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]
Figure 14
Figure 14. Figure 14: Self-explain Examples filtered out based on FFD score [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]
Figure 15
Figure 15. Figure 15: Extractive Examples filtered out based on mIFD score [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data

    cs.AI 2026-01 conditional novelty 5.0 of 10

    A metric-oriented survey that classifies intrinsic quality and trustworthiness metrics for LLM-generated data across six modalities and documents systematic evaluation gaps in the current literature.

  2. The Breeze 2 Herd of Models: Traditional Chinese LLMs Based on Llama with Vision-Aware and Function-Calling Capabilities

    cs.CL 2025-01 conditional novelty 3.0 of 10

    Breeze2 continues pretraining Llama on a 900GB Traditional Chinese corpus, then adds LLaVA-style vision and function-calling fine-tuning, reporting top size-class scores on a mix of public and self-built benchmarks.

Reference graph

Works this paper leans on

67 extracted references · 8 canonical work pages · cited by 2 Pith papers

  1. [1]

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219

    arXiv 2024

  2. [2]

    Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C Fowlkes, Stefano Soatto, and Pietro Perona. 2019. Task2vec: Task embedding for meta-learning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6430--6439

    work page 2019

  3. [3]

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716--23736

    2022

  4. [4]

    Dosovitskiy Alexey. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv: 2010.11929

    arXiv 2020

  5. [5]

    Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. 2023. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390

    arXiv 2023

  6. [6]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609

    arXiv 2023

  7. [7]

    Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65--72

    2005

  8. [8]

    Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Mar c al Rusinol, Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas. 2019. Scene text visual question answering. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4291--4301

    2019

Show all 67 references
  1. [9]

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv \'e J \'e gou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. 2021. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650--9660

    2021

  2. [10]

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. 2021. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3558--3568

    2021

  3. [11]

    Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. 2024 a . Allava: Harnessing gpt4v-synthesized data for a lite vision-language model. arXiv preprint arXiv:2402.11684

    2024 arXiv

  4. [12]

    Lin Chen et al. 2023 a . https://arxiv.org/abs/2311.12793 Sharegpt4v: Improving large multi-modal models with better captions . Preprint, arXiv:2311.12793

    2023 arXiv

  5. [13]

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. 2024 b . How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821

    2024 arXiv

  6. [14]

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. 2023 b . Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arX...

    2023 arXiv

  7. [15]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. https://openreview.net/forum?id=vvoWPYqZJA Instruct BLIP : Towards general-purpose vision-language models with instruction tuning . In Thirty-seventh Co...

    2023

  8. [16]

    Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. 2024. Internlm-xcomposer2: Mastering free-form text-image composition and comprehension in vision-language large model. arXiv preprint arXiv:2401.16420

    2024 arXiv

  9. [17]

    Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. 2023. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304.15010

    2023 arXiv

  10. [18]

    Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthenis Karatzas, Shijian Lu, and CV Jawahar. 2019. Icdar2019 competition on scanned receipt ocr and information extraction. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 1516--1520. IEEE

    2019

  11. [19]

    Guillaume Jaume, Hazim Kemal Ekenel, and Jean-Philippe Thiran. 2019. Funsd: A dataset for form understanding in noisy scanned documents. In 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW), volume 2, pages 1--6. IEEE

    2019

  12. [20]

    Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, \'A kos K \'a d \'a r, Adam Trischler, and Yoshua Bengio. 2017. Figureqa: An annotated figure dataset for visual reasoning. arXiv preprint arXiv:1710.07300

    2017 arXiv

  13. [21]

    Hugo Lauren c on, Andr \'e s Marafioti, Victor Sanh, and L \'e o Tronchon. 2024 a . Building and better understanding vision-language models: insights and future directions. arXiv preprint arXiv:2408.12637

    2024 arXiv

  14. [22]

    Hugo Lauren c on, L \'e o Tronchon, Matthieu Cord, and Victor Sanh. 2024 b . What matters when building vision-language models? arXiv preprint arXiv:2405.02246

    2024 arXiv

  15. [23]

    Alycia Lee, Brando Miranda, Sudharsan Sundar, and Sanmi Koyejo. 2023. Beyond scale: the diversity coefficient as a data quality metric demonstrates llms are pre-trained on formally diverse data. arXiv preprint arXiv:2306.13840

    2023 arXiv

  16. [24]

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024 a . Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326

    2024 arXiv

  17. [25]

    Chunyuan Li et al. 2023 a . https://arxiv.org/abs/2309.10020 Multimodal foundation models: From specialists to general-purpose assistants . Preprint, arXiv:2309.10020

    2023 arXiv

  18. [26]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023 b . Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730--19742. PMLR

    2023

  19. [27]

    Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. 2023 c . From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning. arXiv preprint arXiv:2308.12032

    2023 arXiv

  20. [28]

    Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. 2024 b . Monkey: Image resolution and text label are important things for large multi-modal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter...

    2024

  21. [29]

    Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74--81

    2004

  22. [30]

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. 2024. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26689--26699

    2024

  23. [31]

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \'a r, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, ...

    2014

  24. [32]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023 a . Improved baselines with visual instruction tuning

    2023

  25. [33]

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024 a . https://llava-vl.github.io/blog/2024-01-30-llava-next/ Llava-next: Improved reasoning, ocr, and world knowledge

    2024

  26. [34]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023 b . https://arxiv.org/abs/2304.08485 Visual instruction tuning . Preprint, arXiv:2304.08485

    2023 arXiv

  27. [35]

    Yuliang Liu, Zhang Li, Biao Yang, Chunyuan Li, Xucheng Yin, Cheng-lin Liu, Lianwen Jin, and Xiang Bai. 2023 c . On the hidden mystery of ocr in large multimodal models. arXiv preprint arXiv:2305.07895

    2023 arXiv

  28. [36]

    Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. 2024 b . Textmonkey: An ocr-free large multimodal model for understanding document. arXiv preprint arXiv:2403.04473

    2024 arXiv

  29. [37]

    Gen Luo, Yiyi Zhou, Yuxin Zhang, Xiawu Zheng, Xiaoshuai Sun, and Rongrong Ji. 2024. Feast your eyes: Mixture-of-resolution adaptation for multimodal large language models. arXiv preprint arXiv:2403.03003

    2024 arXiv

  30. [38]

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244

    2022 arXiv

  31. [39]

    Minesh Mathew, Viraj Bagal, Rub \`e n Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. 2022. Infographicvqa. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1697--1706

    2022

  32. [40]

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. 2021. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200--2209

    2021

  33. [41]

    Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. 2019. Ocr-vqa: Visual question answering by reading text in images. In ICDAR

    2019

  34. [42]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 3...

    2022

  35. [43]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311--318

    2002

  36. [44]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

    2021

  37. [45]

    Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084

    2019 arXiv

  38. [46]

    Min Shi, Fuxiao Liu, Shihao Wang, Shijia Liao, Subhashree Radhakrishnan, De-An Huang, Hongxu Yin, Karan Sapra, Yaser Yacoob, Humphrey Shi, et al. 2024. Eagle: Exploring the design space for multimodal llms with mixture of encoders. arXiv preprint arXiv:2408.15998

    2024 arXiv

  39. [47]

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317--8326

    2019

  40. [48]

    Amanpreet Singh, Guan Pang, Mandy Toh, Jing Huang, Wojciech Galuba, and Tal Hassner. 2021. Textocr: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8802--8812

    2021

  41. [49]

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. 2024 a . Cambrian-1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406.16860

    2024 arXiv

  42. [50]

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. 2024 b . Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568--9578

    2024

  43. [51]

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566--4575

    2015

  44. [52]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191

    2024 arXiv

  45. [53]

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint arXiv:2212.10560

    2022 arXiv

  46. [54]

    Zizhang Wu, Xinyuan Chen, Jizheng Wang, Xiaoquan Wang, Yuanzhu Gan, Muqing Fang, and Tianhao Xu. 2023. Ocr-rtps: an ocr-based real-time positioning system for the valet parking. Applied Intelligence, 53(14):17920--17934

    2023

  47. [55]

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244

    2023 arXiv

  48. [56]

    Ruyi Xu, Yuan Yao, Zonghao Guo, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, Maosong Sun, and Gao Huang. 2024. Llava-uhd: an lmm perceiving any aspect ratio and high-resolution images. arXiv preprint arXiv:2403.11703

    2024 arXiv

  49. [57]

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. 2024. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800

    2024 arXiv

  50. [58]

    Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, et al. 2023 a . Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model. arXiv preprint arXiv:2310.05126

    2023 arXiv

  51. [60]

    Qinghao Ye et al. 2023 c . https://arxiv.org/abs/2304.14178 mplug-owl: Modularization empowers large language models with multimodality . Preprint, arXiv:2304.14178

    2023 arXiv

  52. [61]

    Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. 2023 a . Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199

    2023 arXiv

  53. [62]

    Ruiyi Zhang, Yanzhe Zhang, Jian Chen, Yufan Zhou, Jiuxiang Gu, Changyou Chen, and Tong Sun. 2024. Trins: Towards multimodal language models that can read. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22584--22594

    2024

  54. [63]

    Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. 2023 b . Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308.10792

    2023

  55. [64]

    Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. 2023 c . Llavar: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint arXiv:2306.17107

    2023 arXiv

  56. [65]

    Henry Hengyuan Zhao, Pan Zhou, and Mike Zheng Shou. 2023. Genixer: Empowering multimodal large language models as a powerful data generator. arXiv preprint arXiv:2312.06731

    2023 arXiv

  57. [66]

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592

    2023 arXiv

  58. [67]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  59. [68]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.