{"id":"bca475f1-6543-44ac-883e-e11ed95b5733","arxiv_id":"2505.19091","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new benchmark converts text-only QA datasets into text images and shows that vision-language models degrade sharply on long visually presented contexts.","lead":"ReadBench turns text-only reading comprehension tests into images of text to see how well vision-language models read. The authors find that models lose accuracy on long multi-page text images, while short inputs and image resolution matter less.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 2.3's 35-example sampling is unvalidated, so the reported degradation magnitudes, model ordering, and mismatch overlap could reflect sampling noise rather than true VLM reading ability.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the 35-example sampling justification is absent. In good faith, the benchmark itself is a useful artifact, and the long-context degradation direction is plausible and worth reporting as a trend. The image generation protocol is careful (A4, 12pt Arial, 92.9 PPI, cropping), and the resolution ablation is suggestive, though it uses only one model. However, the paper reports no error bars or statistical tests, and with n=35 per subset and mostly single-digit differences among short-context models, the ordinal claims are not currently supported. I also noted the potential internal inconsistency in page-count language between Section 2.1 (12-page average) and Section 3 (2-3 pages), but that does not change the central concern. If the proposed full-set check passes, the conditional acceptance is warranted; if it fails, the magnitudes and rankings should be revised or downgraded.","tokens_in":9229,"tokens_out":6350,"duration_ms":62379,"concrete_test":"Pick one short-context subset (e.g., MMLU-Redux) and one long-context subset (e.g., LongBench HotPotQA). Run the full dataset in both text-only and image modes for at least two models (e.g., GPT-4o and Gemini 1.5 Pro), then draw 100 random 35-example subsets and compute the resulting score distributions and model rankings. Report whether the full-set scores fall inside the 95% confidence interval of the 35-example estimates and whether the best-model ranking is stable across seeds. If the full-set-versus-subsample delta exceeds about 5 percentage points or the ranking flips, the Section 2.3 sampling claim is falsified and the reported degradation magnitudes need re-estimation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The quantitative claims rest on Section 2.3: 'we randomly sample up to 35 examples per benchmark subset, a size we experimentally found to yield results highly correlated with evaluating the full dataset.' No correlation analysis, plot, or numbers are provided. For a binary metric, n=35 gives a standard error of up to about 8.5 percentage points at p=0.5 and about 7.8 points at p=0.7; the three-run average reduces only run-to-run variance, not sampling variance. Consequently, the central numerical results—'Gemini 1.5 Pro experiencing the largest relative degradation (around 30%)', 'GPT-4o fares the best', the difficulty-correlation claim, and the 20–35% Jaccard overlap pattern—are all computed on subsets whose representativeness is asserted but not demonstrated. The short-context 'universal but minimal' degradation is especially fragile, since the paper itself reports that GPT-4o and Qwen2.5-VL 7B maintain or improve on multimodal GPQA. Without a full-set comparison or confidence intervals, the headline multi-page degradation trend and the model ordering may be seed effects. This is the most load-bearing weakness because every downstream analysis inherits it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ReadBench, a benchmark that converts contexts from five text-only benchmarks (MMLU-Redux, MMLU-Pro, GPQA-Diamond, BABILong, and four LongBench subsets) into images of text while keeping prompts and questions in text form. The authors evaluate nine current VLMs on this mixed-modality format and report universal performance degradation relative to text-only baselines, with small drops on short contexts but sharp declines on multi-page inputs. They also report a mismatch analysis suggesting that failures are largely model-specific, and a resolution ablation on a single model suggesting that PPI has negligible effect. The benchmark and evaluation code are released publicly.","tokens_in":9557,"tokens_out":3816,"duration_ms":32092,"significance":"If the central claims are validated, ReadBench fills a genuine gap: most existing multimodal benchmarks focus on visual reasoning or OCR, whereas ReadBench isolates the ability to read and reason over dense textual documents, which is directly relevant to document QA and VisRAG applications. The paper's strengths are its transparent construction from permissively licensed standard benchmarks, the evaluation across nine models of different sizes and families, the public release of data and code, and the explicit acknowledgment of limitations in a dedicated section. However, the validity of the quantitative conclusions rests on a sampling strategy whose stated validation is not shown, and on a resolution ablation from one model only. These issues are load-bearing because every reported degradation magnitude, model ordering, and overlap statistic inherits them.","major_comments":[{"comment":"The claim that sampling up to 35 examples per subset \"yields results highly correlated with evaluating the full dataset\" is unsupported: no correlation analysis, plot, or numeric comparison with full-set evaluations is provided. For a binary metric, n=35 gives a standard error of up to roughly 8.5 percentage points at p=0.5, so the three-run averaging reported in Section 3 reduces only run-to-run variance, not sampling variance. As a consequence, the headline numbers (e.g., \"around 30%\" relative degradation for Gemini 1.5 Pro, \"GPT-4o fares the best\", the difficulty-correlation claim, and the 20–35% Jaccard overlap in Figure 4a) may be seed effects. Please either provide the full-set comparison, or add confidence intervals/bootstrap resampling over sampled questions for all reported aggregates and demonstrate that the model ordering is stable across random seeds.","section":"Section 2.3"},{"comment":"The conclusion that text resolution has negligible effects on multimodal performance is drawn entirely from one model, Gemini 2.0 Flash, across PPI settings. Since the main benchmark evaluation spans nine models with different vision encoders and input-resolution handling, the abstract and conclusion overgeneralize this finding. Please extend the resolution ablation to at least a few representative models (e.g., one open-weight and one additional closed-weight model), or explicitly restrict the claim to Gemini 2.0 Flash and rephrase the abstract/conclusion accordingly.","section":"Section 2.2.1 / Figure 2"},{"comment":"The paper's central claim of \"universal performance degradation\" is not literally supported by its own data for short contexts: the text reports that GPT-4o and Qwen2.5-VL 7B maintain or slightly improve on multimodal GPQA. Given the small sample sizes and absence of error bars in Figure 3, the claim needs a statistical qualifier (e.g., \"on average, most models degrade on most subsets\") or explicit per-subset significance tests. Please add error bars or confidence intervals to Figure 3 and report the per-subset, per-model differences with uncertainty rather than aggregate trend lines alone.","section":"Section 3"},{"comment":"The mismatch analysis is computed on the same 35-example subsets as the main results, and the pairwise Jaccard similarities (20–35%) and the 3.1% statistic for questions affecting more than five models are presented without any uncertainty quantification. With sampling noise of the magnitude described above, the conclusion that failures \"primarily arise from model-specific challenges\" rather than universally problematic inputs may be an artifact of the subset. Please report confidence intervals for the Jaccard values (e.g., via bootstrap over sampled questions) or validate the overlap structure on a full-set comparison before drawing this qualitative conclusion.","section":"Section 4 / Figure 4"}],"minor_comments":[{"comment":"Typo: \"examplified\" should be \"exemplified\".","section":"Section 1"},{"comment":"Typo: \"reproduceable\" should be \"reproducible\".","section":"Acknowledgements"},{"comment":"Typo in the NarrativeQA prompt: \"asconcisely\" should be \"as concisely\" (two occurrences).","section":"Appendix A.4"},{"comment":"The y-axis label is \"multi-modal scores\" but the metric is not defined; please state whether this is accuracy, exact-match, or the binary overlap metric used elsewhere, and add run details or error bars.","section":"Figure 2"},{"comment":"The phrase \"We sample from all ten questions it contains\" for BABILong is ambiguous; these are ten question templates, not ten fixed questions. Please rephrase to make the sampling unit clear.","section":"Section 2.1.2"}],"recommendation":"major_revision","confidential_remarks":"The benchmark resource is potentially useful and within the journal's scope, and the multi-model evaluation is a strength. The blocking issue is the unvalidated 35-example sampling strategy, on which all the quantitative claims rest; the resolution ablation also needs either extension or careful restatement. If the authors can supply the missing correlation analysis or confidence intervals and temper the claims accordingly, I would be inclined to accept. I also recommend double-checking that the GitHub repository contains the full evaluation suite as promised in the Limitations section."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: ReadBench is a genuinely useful new benchmark. It takes five established text-only benchmarks, renders the contexts into clean A4 text images, keeps questions in text, and measures the drop. That construction is new relative to OCRBench, DocVQA, TextVQA, ViDoRe, and PixelWorld, and it directly serves document QA and VisRAG use cases. The paper reports a credible qualitative finding: short visual contexts cost little, multi-page visual contexts cost a lot, and different models fail on different questions.\n\nCredit where due: the image-generation pipeline is sensible (A4, 12pt, 92.9 PPI, page cropping), the benchmark code and data are released, and the authors evaluate nine current VLMs with three runs each. The resolution ablation, while limited to one model, aligns with earlier work and is a useful practical check.\n\nThe soft spot is exactly where the stress-test note points. Section 2.3 says 35 examples per subset was 'experimentally found' to correlate with full-set results, but no correlation analysis is shown. For a binary metric at n=35, the standard error is roughly 8 points at p=0.5 and 7.8 at p=0.7; averaging three runs only reduces run-to-run noise, not sampling noise. So the reported degradation magnitudes, the model ordering (Gemini 1.5 Pro ~30% drop, GPT-4o best), and the 20–35% Jaccard overlap pattern could shift with the seed. The authors also give no error bars, and the 'resolution has negligible effect' claim rests on a single model. None of this is fatal — the short-vs-multi-page contrast is big and consistent enough to survive sampling noise — but the paper should not be used as a fine-grained ranking instrument until the sample-size claim is substantiated or confidence intervals are provided.\n\nAlso worth noting: the paper itself flags several limitations (English-only, standard benchmarks, budget-constrained model selection), which is honest and appropriate.\n\nBottom line: this is a solid benchmark-contribution paper with a real gap to fill. The central qualitative claim holds up; the quantitative precision does not. It deserves a serious peer review, but I would insist on a full-set comparison or an actual correlation plot for the sampling claim before accepting.","headline":"Useful new benchmark for VLM visual text reading; qualitative trend is credible, but the unvalidated 35-example sampling means the precise numbers should not yet be trusted.","tokens_in":9984,"tokens_out":1617,"would_cite":true,"duration_ms":14321,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reading text from images costs vision-language models accuracy, and the loss grows sharply with multipage contexts.","keywords":["vision-language models","multimodal benchmark","dense text reading","document understanding","long-context visual reasoning","text-image degradation","resolution robustness","retrieval-augmented generation"],"falsifier":"Run the ReadBench evaluation on the full dataset (or a much larger sample) for at least one model and one subset, comparing per-subset accuracy with the 35-example estimate; the central result would be weakened if the full-data degradation is small where the sample showed a large drop, or if the model ranking on degradation changes materially.","tokens_in":9073,"feed_emoji":"📄","tokens_out":5908,"duration_ms":39192,"temperature":0.7,"pith_summary":"This paper introduces ReadBench, a benchmark that turns the context passages of established text-only QA and multiple-choice benchmarks into images of text while keeping questions and prompts as text. Using it on nine current vision-language models, the authors try to establish that reading from images causes universal but uneven performance loss: small on short one-page inputs, sharp on multipage contexts, and largely unaffected by image resolution. They also report that the specific questions on which models lose accuracy differ strongly across models, arguing against a single universal failure trigger. If correct, ReadBench gives a practical measurement of a capability that existing multimodal benchmarks do not isolate: dense text reading from visually presented documents.","feed_headline":"Multipage text images cut vision-language model accuracy up to 30 percent","feed_subtitle":"New benchmark renders text-only test contexts onto A4 pages; short-image reading stays strong, multipage drops sharply.","key_machinery":"The central object is ReadBench itself: a benchmark constructed by rendering the context passages of five text-only benchmarks (MMLU-Redux, MMLU-Pro, GPQA-Diamond, BABILong, and four LongBench subsets) as images in A4 format at 92.9 PPI with 12-point Arial text, while leaving instructions and questions in text. Short-context benchmarks have only their answer options rendered as images; long-context benchmarks have the entire context rendered across pages, with partially filled pages cropped. Models receive the same prompts in text-only and multimodal conditions, so accuracy differences isolate the cost of reading text visually. Scores are averaged over three runs, and each subset is evaluated on up to 35 sampled examples.","core_discovery":"On ReadBench, every evaluated vision-language model degrades when the context it must read is presented as images instead of text. The degradation is minimal or even slightly negative for short contexts, with GPT-4o and Qwen2.5-VL 7B sometimes matching or beating their text-only scores, but becomes large on inputs of two or more A4 pages, with Gemini 1.5 Pro dropping by around 30 percent relative to text-only performance while GPT-4o degrades the least. Degradation also tracks task difficulty, appearing more strongly on GPQA-Diamond than on MMLU-Redux. Varying the rendering resolution from 72 to 300 PPI changes Gemini 2.0 Flash scores negligibly, and mismatch analysis shows that failures on individual questions are mostly model-specific rather than shared across models.","pith_inferences":["The paper leaves open whether the multipage drop reflects a visual-reading limit or the model's overall context-window mechanics; reproducing the same page count as token-heavy text in a text-only baseline would separate these.","A natural extension is to test whether the same degradation appears when the images are screenshots of real PDFs rather than cleanly rendered A4 pages, since document noise and layout variation may interact with the resolution findings.","The sampling assumption (35 examples per subset correlate with the full dataset) is stated but not demonstrated; rerunning one subset at full size would settle whether the reported ranking and degradation magnitudes are stable.","The mismatch results suggest that a model ensemble could be tested against single models on ReadBench; the paper does not evaluate that possibility."],"forward_implications":["For short documents of up to about one page, current vision-language models can answer from rendered text almost as reliably as from plain text, so image-based reading is a viable interface at that scale.","For multipage visual contexts, no evaluated model currently maintains text-level accuracy, so applications that feed scanned or rendered documents to VLMs should budget for a substantial quality loss.","Raising image resolution will not recover that loss, since performance is roughly flat from 72 to 300 PPI, at least for Gemini 2.0 Flash.","Because model failures on multimodal inputs are largely model-specific, combining or routing among vision-language models may recover more accuracy than any single-model fix.","ReadBench provides a reusable measurement protocol for tracking whether future vision-language models close this gap."],"supporting_citations":[{"why":"Supplies the MMLU-Redux short-context multiple-choice questions, the easiest task in ReadBench.","marker":"(Gema et al., 2025)"},{"why":"Supplies the MMLU-Pro short-context questions, a harder multiple-choice set with ten options.","marker":"(Wang et al., 2024)"},{"why":"Supplies the GPQA-Diamond short-context questions, the hardest difficulty tier in ReadBench.","marker":"(Rein et al., 2024)"},{"why":"Supplies the BABILong long-context reasoning tasks used to measure multipage degradation.","marker":"(Kuratov et al., 2024)"},{"why":"Supplies the LongBench subsets (HotpotQA, NarrativeQA, TriviaQA, 2WikiMultihopQA) used for multipage QA.","marker":"(Bai et al., 2024)"},{"why":"Provides the prior observation that low-resolution images barely affect vision-model performance, which the PPI ablation tests.","marker":"(Beyer, 2024)"},{"why":"Identifies Gemini 1.5 Pro, the model with the largest reported multipage degradation in the evaluation.","marker":"(Gemini et al., 2024)"},{"why":"Identifies GPT-4o, the model with the smallest degradation across the evaluated models.","marker":"(Hurst et al., 2024)"}],"fun_headline_variants":["Multipage text images trip up vision-language models","VLMs read short text images well, multipage drops 30%","Reading text from images: length matters, resolution doesn't","Vision-language models lose up to 30% on multipage text","Text image reading: short OK, long pages tough for VLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's conclusions rest on the claim that sampling up to 35 examples per subset gives results highly correlated with evaluating the full dataset, but the correlation experiment is not shown in the paper.","fun_headline_variants_meta":{"raw":{"variants":["Multipage text images trip up vision-language models","VLMs read short text images well, multipage drops 30%","Reading text from images: length matters, resolution doesn't","Vision-language models lose up to 30% on multipage text","Text image reading: short OK, long pages tough for VLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000278,"raw_usage":{"total_tokens":1627,"prompt_tokens":889,"completion_tokens":738,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":651}},"tokens_in":505,"tokens_out":738,"duration_ms":6932,"temperature":1.0,"reasoning_tokens":651,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:19:24.769788+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the ReadBench evaluation on the full dataset (or a much larger sample) for at least one model and one subset, comparing per-subset accuracy with the 35-example estimate; the central result would be weakened if the full-data degradation is small where the sample showed a large drop, or if the model ranking on degradation changes materially.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the GPQA-Diamond short-context questions, the hardest difficulty tier in ReadBench."}],"review_version":1}