{"id":"32e6385f-08e0-4124-ae9f-af014ae8104b","arxiv_id":"2412.16364","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A hybrid human/GPT-4o pipeline plus two new filtering scores produces a 424k-pair text-rich image instruction dataset that improves fine-tuned multimodal models on OCR and document understanding benchmarks.","lead":"This paper builds LLaVAR-2, a dataset of about 424,000 image-and-question pairs for teaching AI models to read text inside images, using human-written captions plus GPT-4o to create detailed questions and answers. It introduces automatic filtering scores to remove low-quality pairs, and shows that models trained on the data score higher on text-reading benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central comparison is confounded: LLaVAR-2 models use LLaVA-HR's MRA vision encoder and stronger backbones than the self-instruct baselines, with no same-architecture data-swap experiment, so benchmark gains may not be caused by the dataset.","rationale":"The reader's conditional verdict is reasonable: the dataset is potentially valuable and the public-benchmark gains are suggestive, but missing code/data, absent error bars, and self-generated test sets prevent full acceptance. I agree that OCR quality is a genuine weakness, and the paper itself flags it in Limitations item (1). However, I see a more load-bearing problem in the empirical argument: the headline comparison against self-instruct baselines is confounded by architecture and backbone choice. The LLaVAR-2 models incorporate MRA from LLaVA-HR, a known ingredient for text-rich image performance, and use stronger base LLMs than the earlier LLaVAR/LaRA systems. Without a same-architecture data-swap experiment, the observed benchmark improvements cannot be attributed to LLaVAR-2 data. This is a correctness risk for the central claim, not merely a data-quality caveat. My proposed concrete test directly isolates the dataset by fixing the architecture and varying only the training data. If the controlled comparison confirms the advantage, the central claim stands; if not, the paper's conclusion must be weakened. I keep the verdict UNCHANGED because the same conditional bar already covers this missing evidence: the authors should release the dataset and provide the controlled comparison before full acceptance, and the OCR audit remains a useful secondary check.","tokens_in":23117,"tokens_out":9993,"duration_ms":86572,"concrete_test":"Run a controlled data swap: fix the LLaVAR-2-3.8B architecture (Phi-3-Mini + MRA, two-stage training, same LLaVA-158K base, same hyperparameters) and train three variants on (a) LLaVAR-2 data, (b) the original LLaVAR-1 self-instruct data matched in size and format, and (c) TRINS-VQA data. Evaluate all three on DocVQA, ChartQA, InfoVQA, and OCRBench. If variant (a) does not clearly beat variant (b) on this identical architecture, the claimed advantage over self-instruct data is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's causal claim is that fine-tuning on LLaVAR-2 yields 'impressive enhancements over those trained with self-instruct data' (Abstract). The experiments in Tables 3 and 4 do not isolate the dataset as the causal variable. The LLaVAR-2 models use the Mixture-of-Resolution Adaptation (MRA) vision encoder from LLaVA-HR (Section 4) and backbones such as Phi-3-Mini, Llama-3.1-8B, or Vicuna-13B, whereas the self-instruct / prior baselines (LLaVAR, LaRA/TRINS) use LLaVA-1.5-style architectures with standard CLIP encoders and Vicuna-7B/13B. Consequently, the large gaps on DocVQA (66.1 vs 50.8), ChartQA (78.5 vs 25.6), and OCRBench (634 vs 603) could be driven primarily by the stronger vision encoder and base model rather than by LLaVAR-2 data. The paper never trains the same architecture on LLaVAR-1/self-instruct data or on TRINS data. The ablations 'w/o Dr' and 'w/o Dg' (Tables 2 and 5) hold the dataset family fixed and therefore do not test the self-instruct comparison. The reader's OCR-accuracy concern is legitimate, but even perfect OCR and zero hallucinated text would not establish the dataset's benefit without a controlled data-swap experiment.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LLaVAR-2, a text-rich image instruction-tuning dataset constructed by using human-annotated captions from TRINS as anchor inputs and prompting GPT-4o with the image, the caption, and PaddleOCR outputs to generate 42k detail-enriched captions (LLaVAR-2-Cap) and 382k extractive-plus-self-explain VQA pairs (LLaVAR-2-VQA). The authors also propose two filtering scores, mIFD and FFD, based on instruction-following difficulty loss ratios, and use them to prune the VQA data. They fine-tune LLaVA-HR-style models with Phi-3-Mini, Llama-3.1-8B, and Vicuna-13B backbones on LLaVAR-2 and report improvements over prior baselines on DocVQA, ChartQA, OCRBench, and on held-out portions of the LLaVAR-2 evaluation sets.","tokens_in":23469,"tokens_out":5730,"duration_ms":46315,"significance":"If the empirical claims hold, LLaVAR-2 would be a useful public resource: it provides a concrete hybrid human-LLM data-generation pipeline, a relatively large text-rich instruction set, and a filter-based quality-control method. The external benchmark gains are substantial and are the strongest evidence for the dataset's value; for example, Table 3 shows LLaVAR-2 (Phi-3-Mini) reaching 66.1 on DocVQA and 78.5 on ChartQA versus 50.8 and 25.6 for LaRA-13B, and Table 4 shows OCRBench 634 versus 603 for LaRA-13B. The paper also reports quantitative diversity comparisons (Table 1) and filtering ablations (Figure 8), which are more informative than qualitative examples alone. However, the confound between dataset and architecture in the headline comparison, the self-generated evaluation set, and the threshold-selection procedure need to be resolved before the causal claim about the dataset can be fully credited.","major_comments":[{"comment":"The central comparison is confounded: the LLaVAR-2 models use the Mixture-of-Resolution Adaptation (MRA) vision encoder from LLaVA-HR and stronger backbones (Phi-3-Mini, Llama-3.1-8B, Vicuna-13B), whereas the self-instruct baselines LLaVAR and LaRA use LLaVA-1.5-style architectures with standard CLIP encoders and Vicuna-7B/13B. The reported gains on DocVQA, ChartQA, and OCRBench therefore cannot be attributed to the dataset without a same-architecture data-swap experiment, such as training the proposed vision encoder and backbone on LLaVAR-1 or TRINS data, and training a LLaVA-1.5-style model on LLaVAR-2. The ablations 'w/o Dr' and 'w/o Dg' (Tables 2 and 5) hold the data source fixed and do not address this confound.","section":"Section 6.1, Tables 3 and 4"},{"comment":"The LLaVAR-2-VQA evaluation set is generated by the same GPT-4o pipeline, with the same prompts, OCR results, and manual captions, as the training data. A model fine-tuned on LLaVAR-2 therefore has a distributional advantage that is unrelated to text-reading ability, and the reported BLEU, METEOR, and CIDEr gains may partly reflect stylistic mimicry of GPT-4o outputs rather than improved document comprehension. Please report error bars and significance tests, and ideally evaluate on an independently constructed text-rich VQA test set or on external benchmarks that were not used for threshold selection.","section":"Section 6.1, Table 2"},{"comment":"The 70% mIFD filtering threshold is selected as the 'sweet spot' on the same LLaVAR-2-VQA evaluation set that is used in Table 2 to report the model's main VQA results. Because the filtering percentage is tuned on that set, the filtering-effectiveness claim is circular for that benchmark. Verify the threshold on a separate validation set or on external benchmarks such as DocVQA, ChartQA, and OCRBench, and report how the final 70% choice performs there.","section":"Section 6.3, Figure 8"},{"comment":"The mIFD and FFD scores are computed from the same OCR-derived data that is used for generation. If PaddleOCR outputs contain errors, missing text, or mislocated bounding boxes, the filtering scores measure consistency with those corrupted inputs and cannot detect them. The authors acknowledge this in Limitations item (1), but the filtering section should either quantify the OCR error rate on a sampled subset or show that the retained data and the resulting benchmark performance are stable under OCR perturbation.","section":"Section 3.3 and Limitations item (1)"}],"minor_comments":[{"comment":"The naming is inconsistent: the model is introduced as 'LLaVAR-2-3.8B' but is referred to as 'LLaVAR-2 (Phi-3-Mini)' in Tables 2, 3, 4, and 5; please clarify that these denote the same architecture and state whether 'LLaVAR-2-3.8B' also uses the MRA vision encoder.","section":"Section 4"},{"comment":"The case description contains a typo: 'the extractive answer Ae is anti-intuitive or unrelated to the extractive question Ae' should read '...unrelated to the extractive question Qe'.","section":"Section 3.3, Eq. (2)"},{"comment":"The term 'Zero-shot performance' for the LLaVAR-2 models is potentially misleading; clarify that the models are fine-tuned on LLaVAR-2 but are not trained on the specific benchmark datasets, so 'zero-shot' refers to the absence of benchmark-specific training, not to the absence of instruction tuning.","section":"Section 6.1"},{"comment":"FUNSD and SROIE are not VQA benchmarks; they are form-understanding and receipt-information-extraction tasks. Please specify the evaluation metric used for these columns (e.g., F1 score) and cite the original task definitions.","section":"Table 3"},{"comment":"The text states that Phi-3.5-vision is used to compute FFD scores, but it does not specify which model is used to compute the mIFD scores (Figure 4 shows several models). Please state the model used for the actual mIFD-based filtering in Section 3.3 and Section 6.3.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a direct follow-up of TRINS and LLaVAR by overlapping authors, and the novelty relative to TRINS lies mainly in the GPT-4o-based enrichment and the self-explain/filtering mechanism. This is not a conflict per se, but the claimed advantage over self-instruct data should be demonstrated with a controlled architecture-matched comparison. The dataset and code release should be verified at the project page before final acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Useful dataset, but the headline comparison—better than self-instruct—is not actually tested. The models trained on LLaVAR-2 use LLaVA-HR's MRA vision encoder and backbones like Phi-3-Mini, Llama-3.1-8B, or Vicuna-13B, while the self-instruct baselines (LLaVAR, LaRA/TRINS) run on LLaVA-1.5-style architectures with standard CLIP and Vicuna-7B/13B. Without a same-architecture data-swap experiment, the large gains on DocVQA, ChartQA, and OCRBench could be the encoder or the base model, not the data.\n\nCredit where it's due: combining TRINS human captions with GPT-4o enrichment is a sensible hybrid. The 42k detail-enriched captions plus 382k VQA pairs, with self-explain dialogues alongside extractive ones, is genuinely new. The mIFD and FFD scores extend IFD to the multimodal and dialogue level in a reasonable way, and Figure 8 shows a plausible 70% filtering sweet spot. The ablation without the self-explain data (w/o Dr) and without captions (w/o Dg) suggests both components help, on their architecture. The limitations section is honest about OCR noise, GPT-4o cost and bias, and wasted filtered data.\n\nThe soft spots are proportional. The main one is the confound described above. A single controlled experiment—same architecture, swap LLaVAR-2 vs. LLaVAR-1 vs. TRINS data—would settle it. Second, the LLaVAR-2-VQA test set is generated by the same GPT-4o pipeline as the training data, and the 70% threshold is tuned on that same evaluation. That weakens the internal evaluation but not the public benchmarks. Third, no error bars or significance tests are reported; that's a minor issue for a dataset paper. The OCR-quality concern the authors flag is real, but even perfect OCR wouldn't establish the data's benefit without the data-swap. The paper also gives a project page that appears to have a space in the URL, and no code/data release is mentioned—worth checking.\n\nWho this is for: anybody building instruction-tuning datasets for documents, charts, or scene text, and anyone working on data filtering metrics for MLLMs. I'd bring it to a reading group if you want a concrete case of architecture confounding in empirical AI papers. I'd send it to peer review: the claim is important enough and the dataset big enough that referees should ask for the controlled experiment and the release. With that fix, it could be a solid contribution.","headline":"Useful dataset, but the headline comparison—better than self-instruct—is not actually tested because the architectures differ.","tokens_in":24017,"tokens_out":3830,"would_cite":false,"duration_ms":32216,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that LLaVAR-2, a 424k-pair text-rich instruction dataset combining human captions, GPT-4o enrichment, and difficulty-based filtering, lets small multimodal models read documents and charts far better than self-instruct…","keywords":["text-rich image understanding","instruction tuning","multimodal large language models","data generation","self-explain dialogues","data filtering","OCR grounding","visual question answering"],"falsifier":"Take a random sample of LLaVAR-2 images, run the OCR step again after injecting known typographical errors or dropping low-confidence words, regenerate captions and answers from the corrupted OCR, and compare the filtered dataset and downstream DocVQA score against the intact version; if performance barely changes, OCR accuracy is not load-bearing.","tokens_in":22911,"feed_emoji":"📄","tokens_out":6226,"duration_ms":50256,"temperature":0.7,"pith_summary":"The paper tries to show that the bottleneck in text-rich image understanding is not model size but instruction data, and that a hybrid of human annotation and large-model generation can fix it. It constructs LLaVAR-2 by taking human-written captions from TRINS, feeding them along with OCR text and bounding boxes to GPT-4o, and asking the model for detail-enriched captions, extractive question-answer pairs, and self-explain dialogues that spell out where in the image each answer comes from. Two new filtering scores, mIFD and FFD, remove pairs that are image-detached, redundant, or poorly matched, leaving 424k instruction pairs. Fine-tuning compact backbones like Phi-3-Mini on this data raises DocVQA, ChartQA, and OCRBench scores well above prior self-instruct-based and larger specialized models. If the claim holds, LLaVAR-2 is a reusable data resource for teaching multimodal models to read text embedded in images.","feed_headline":"With LLaVAR-2, 3.8B models outread 13B document specialists","feed_subtitle":"A 424k-pair dataset from human captions and GPT-4o raises DocVQA, ChartQA, and OCRBench scores of small multimodal models.","key_machinery":"The operative mechanism is the Hybrid-Instruct generation pipeline: a text-rich image, its human-annotated caption, and PaddleOCR words with bounding boxes are jointly prompted into GPT-4o. From this, GPT-4o produces detail-enriched captions that keep the original caption's style but insert OCR-verified text strings and locations, plus extractive QA pairs and self-explain QA pairs that state how the extractive answer was located. The filtering machinery is mIFDθ(Qe,Ae,I)=VFD(De,I)×IFD(Qe,Ae), where VFD measures how well the dialogue follows the image and IFD measures how well the question predicts its answer, and FFDθ(De,Dr,I)=sθ(Dr|De,I)/sθ(Dr|I), which scores whether the self-explain pair adds evidence beyond the extractive pair. These scores identify the two failure modes the data generator is prone to, answers drawn from the model's world knowledge rather than the image and self-explanations that are either irrelevant or mere repetitions, and the paper removes the worst 70% of extractive pairs and 7.1k self-explain pairs.","core_discovery":"The central discovery is that self-instruct data fails for text-rich images because a model that cannot align text and vision will generate fluent but ungrounded instructions, and human captions provide the missing alignment signal. The paper's hybrid pipeline keeps the human caption as the skeleton, lets GPT-4o add exact text strings and locations supplied by OCR, and then pairs each extractive answer with a self-explain answer that names the visual evidence, for example the blue box in the left corner. The filtering step uses a multimodal instruction-following difficulty score mIFD, the product of a visual-following difficulty and the original IFD score, to drop extractive pairs whose answers come from prior knowledge rather than the image, and a fact-following difficulty score FFD to drop self-explain pairs that are either off-topic or verbatim repeats. Trained on the filtered set, a 3.8B Phi-3-Mini model reaches 66.1% on DocVQA, 78.5% on ChartQA, and 634 on OCRBench, beating LaRA-13B, a larger model trained on the TRINS pipeline, and the paper attributes these gains to grounded detail and explicit extraction.","pith_inferences":["A testable extension would corrupt or drop OCR words in a held-out subset and measure how much the DocVQA and ChartQA gains fall; the paper's own limitation note predicts the gains would shrink.","The same self-explain-plus-filter recipe could be applied to handwritten documents or noisy scene text, where OCR confidence is lower, but the FFD score's ability to separate evidence-based rationales from repetitive ones would need revalidation there.","Because mIFD and FFD are computed with specific reference models, an open question is whether the 70% sweet spot and downstream rankings are stable across different filter models; the paper shows score distributions align but does not report end-task performance after filtering with each model.","The filtered-out pairs, currently wasted, could seed a curriculum or a small scorer that predicts the filtering score, turning post-hoc filtering into cheaper generation-time selection."],"forward_implications":["Models fine-tuned on LLaVAR-2 outperform prior self-instruct-based pipelines and document-specialized models zero-shot on DocVQA, ChartQA, InfoVQA, FUNSD, and SROIE.","Adding self-explain dialogues improves extractive QA over using extractive pairs alone, so making the evidence-location step explicit is itself a training signal.","Difficulty-score filtering improves downstream quality up to a 70% removal rate, after which the remaining data is too small, meaning a smaller, cleaner set can beat a larger noisy one.","Detail-enriched captioning data is needed for summarization performance as well as VQA, since removing it drops captioning metrics sharply.","Compact backbones such as Phi-3-Mini and Llama-3.1 with LLaVAR-2 rival or beat larger baselines like LaRA-13B, suggesting data quality can substitute for scale in text-rich understanding."],"supporting_citations":[{"why":"Provides the human-annotated captions that anchor the hybrid generation and is the main TRINS baseline.","marker":"(Zhang et al., 2024)"},{"why":"Defines the self-instruct-style LLaVAR pipeline and supplies the in-context demonstration format the VQA generation follows.","marker":"(Zhang et al., 2023c)"},{"why":"Provides the LLaVA architecture and system prompt used for extractive conversation generation and fine-tuning.","marker":"(Liu et al., 2023b)"},{"why":"Defines IFD, the instruction-following difficulty that mIFD extends to multimodal dialogues.","marker":"(Li et al., 2023c)"},{"why":"Contributes the Mixture-of-Resolution Adaptation vision encoder used in the LLaVAR-2 models.","marker":"(Luo et al., 2024)"},{"why":"Supplies the Phi-3-Mini language backbone for the compact 3.8B model whose benchmark results carry the main empirical claim.","marker":"(Abdin et al., 2024)"},{"why":"Provides OCRBench, the benchmark where LLaVAR-2 models score 634 and 680.","marker":"(Liu et al., 2023c)"}],"fun_headline_variants":["LLaVAR-2: 424k hybrid pairs let a 3.8B model beat 13B specialists","Self-instruct fails for text-rich images; LLaVAR-2's human captions fix it","Grounding via human captions: LLaVAR-2 boosts small models past 13B","LLaVAR-2: Human captions, not self-instruct, unlock text-rich image AI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline assumes PaddleOCR's text and bounding boxes are accurate enough to correct human captions and to ground the self-explain answers, and the filtering scores cannot catch errors introduced by that same OCR source.","fun_headline_variants_meta":{"raw":{"variants":["LLaVAR-2: 424k hybrid pairs let a 3.8B model beat 13B specialists","Self-instruct fails for text-rich images; LLaVAR-2's human captions fix it","Grounding via human captions: LLaVAR-2 boosts small models past 13B","LLaVAR-2: Human captions, not self-instruct, unlock text-rich image AI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00084,"raw_usage":{"total_tokens":3661,"prompt_tokens":946,"completion_tokens":2715,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":2609}},"tokens_in":562,"tokens_out":2715,"duration_ms":17413,"temperature":1.0,"reasoning_tokens":2609,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:38:15.564500+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of LLaVAR-2 images, run the OCR step again after injecting known typographical errors or dropping low-confidence words, regenerate captions and answers from the corrupted OCR, and compare the filtered dataset and downstream DocVQA score against the intact version; if performance barely changes, OCR accuracy is not load-bearing.","supporting_citations":[],"review_version":1}