{"id":"448f7d28-6990-4103-b495-17ad4ddfa5ad","arxiv_id":"2607.07179","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":6,"one_line_summary":"Few-shot adaptation with 50 target-domain samples enables VLMs finetuned on different DocVQA domains to match or surpass fully supervised counterparts, revealing visual understanding as the primary bottleneck.","lead":"This paper benchmarks 8 open-source Vision-Language Models on Document Visual Question Answering across three document domains (industrial, infographics, slides), finding that visual understanding—not knowledge—is the main bottleneck, and that 50 few-shot samples can match full finetuning. It matters for practitioners deciding how to adapt VLMs to new document types with minimal data.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"The causal claim 'visual understanding is the main bottleneck' is inferred from performance patterns without an isolating ablation; alternative explanations (question-type distribution shifts, domain-specific layout parsing) are not ruled out.","rationale":"The reader correctly identified the most load-bearing concern: the causal attribution to 'visual understanding' is inferred from performance patterns without an isolating ablation. I agree with this assessment. The paper provides a useful empirical benchmark—8 models × 3 domains × 3 settings is a solid experimental matrix—but the interpretive claim goes beyond what the evidence supports. The concern is not that the claim is necessarily wrong (it may well be correct), but that it is underdetermined by the experimental design. A text-only or OCR-augmented baseline would be a straightforward and decisive test. The secondary issue of missing variance estimates for the few-shot comparisons reinforces this: the headline-worthy claim that 50 shots can surpass full finetuning rests on marginal differences with no reported uncertainty. The CONDITIONAL verdict is appropriate: the benchmark contribution stands, but the mechanistic claim requires additional evidence to be fully supported. The SlideVQA filtering (removing multi-hop questions) introduces a selection bias that further complicates cross-dataset comparisons, though this is a lesser concern than the missing ablation.","tokens_in":13937,"tokens_out":3511,"duration_ms":67767,"concrete_test":"Run a text-only baseline: feed each model the question plus OCR-extracted text from the document image (no image input) for all three datasets, using the same finetuning protocol. If the text-only baseline matches or exceeds the image-based model on SP-DocVQA but not on InfographicsVQA/SlideVQA, the visual-understanding-bottleneck claim is supported. If text-only performance also drops on the complex domains, the bottleneck is not specifically visual but reflects reasoning or question-type demands. Additionally, report standard deviations across ≥3 seeds for the few-shot experiments in Fig. 4 to determine whether the 'surpassing full FT' cases are statistically meaningful.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central interpretive claim rests on a chain of inference: (1) zero-shot performance drops on visually complex domains, (2) finetuning helps, (3) 50-shot adaptation is surprisingly effective, therefore (4) the bottleneck is visual understanding, not knowledge. Step 4 does not follow from 1–3. The three datasets differ not only in visual complexity but also in question types: SP-DocVQA is purely extractive, while InfographicsVQA includes non-extractive numerical reasoning and SlideVQA includes multi-hop reasoning (partially filtered out). Performance gaps could reflect differences in reasoning demands or answer-format distributions rather than visual understanding per se. A text-only or OCR-augmented baseline (providing extracted text without the image, or image+OCR text) on the same questions would directly test whether the performance gap is attributable to visual processing or to other confounds. Without this, the attribution to 'visual understanding' is one of several plausible interpretations. Additionally, no variance estimates or statistical tests are reported for the few-shot comparisons, so the claim that 50-shot results 'surpass fully supervised counterparts in some cases' (§4.3, Fig. 4) cannot be distinguished from noise—especially since the differences appear marginal (e.g., ~1 ANLS point in the cases where 50-shot exceeds full FT).","agreement_with_reader":"agree"},"referee_report":{"model":"glm-5.2","summary":"This paper presents an empirical evaluation of 8 open-source Vision-Language Models (VLMs) on Document Visual Question Answering (DocVQA) across three domains: industrial documents (SP-DocVQA), infographics (InfographicsVQA), and presentation slides (SlideVQA). The authors evaluate zero-shot performance, fully supervised fine-tuning with cross-domain transfer, and few-shot adaptation (5, 20, 50 samples) from source domains to a target domain (SlideVQA). Key findings include: (1) zero-shot performance is strong on structured layouts but drops sharply on visually complex domains; (2) smaller models gain more from fine-tuning; (3) 50-shot adaptation can match or exceed fully supervised fine-tuning on the target domain. The paper concludes that visual understanding, rather than lack of knowledge, is the primary bottleneck for DocVQA.","tokens_in":14684,"tokens_out":1086,"duration_ms":177612,"significance":"The paper addresses a practically important question—how well do current VLMs transfer across document domains—and provides a useful comparative benchmark across three well-chosen datasets and multiple model scales. The few-shot adaptation finding (50 samples approaching or matching full fine-tuning) is practically valuable for low-resource deployment scenarios. The systematic cross-domain evaluation matrix (Tables 2–4) is a useful contribution for practitioners. However, the central interpretive claim about visual understanding being the bottleneck is not directly tested, which limits the depth of insight.","major_comments":[{"comment":"§4.3, §5, and Abstract: The claim that 'visual understanding is the main bottleneck for DocVQA, not a lack of knowledge from the VLMs' is an interpretive inference drawn from performance patterns across datasets that differ in multiple confounding factors. The three datasets differ not only in visual complexity but also in question types: SP-DocVQA is purely extractive, InfographicsVQA includes non-extractive numerical reasoning (§3.2), and SlideVQA includes visual-cue and numerical questions (§3.2). Performance gaps could therefore reflect differences in reasoning demands or answer-format distributions rather than visual understanding per se. A text-only or OCR-augmented baseline (providing extracted text without the image, or image+OCR text) on the same questions would directly isolate whether the bottleneck is visual processing or reasoning difficulty. Without such an ablation, the因果性","section":null},{"comment":"§4.3, Fig. 4, and Abstract: The claim that 50-shot results 'surpass fully supervised counterparts in some cases' is not supported by variance estimates or statistical tests. Examining Fig. 4, the cases where 50-shot exceeds full fine-tuning appear marginal (e.g., Qwen3-VL 4B fine-tuned on SP-DocVQA: 50-shot ≈ 54.1 vs. full FT ≈ 50.9; Qwen3.5 9B: 50-shot ≈ 57.4 vs. full FT ≈ 57.9). Without confidence intervals or significance tests, these differences cannot be distinguished from noise. This is load-bearing for the paper's strongest claim and should be addressed by reporting variance across multiple seeds or providing statistical tests.","section":null}],"minor_comments":[{"comment":"Table 1 uses commas as decimal separators (e.g., '85,36') while Tables 2–4 use periods (e.g., '91.63'). Standardize the formatting.","section":null},{"comment":"§3.1: The model family names are inconsistent—'Qwen3.5' in the text vs. 'Qwen 3.5' in tables and discussion. Also, the text references 'Qwen3-VL' but tables use 'Qwen3-VL'. Standardize.","section":null},{"comment":"§3.2: The prompt used for zero-shot evaluation is specified, but it is unclear whether the same prompt was used during fine-tuning. Clarify.","section":null},{"comment":"Fig. 4: The specific ANLS values are difficult to read from the bar chart. Consider supplementing with a table.","section":null},{"comment":"§4.2: The terms 'inter-dataset' and 'intra-dataset' are used but not explicitly defined. Adding a brief definition would improve clarity.","section":null},{"comment":"Reference [21] (ORCA) is cited as a 2026 arXiv preprint. Verify whether a peer-reviewed version exists.","section":null}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical study but the central interpretive claim is overstepped relative to the evidence. The two major comments are addressable: (1) adding at least one OCR-text-only baseline on a subset of questions would substantially strengthen the bottleneck claim, and (2) reporting multi-seed variance for the few-shot comparisons is essential given the marginal differences cited. If the authors soften the causal claim to a correlation and add the variance estimates, the paper could be acceptable for minor revision. The model naming (Qwen3.5, Qwen3-VL, Gemma 3) appears to reference future or hypothetical model versions; this does not affect the review but the editor may wish to verify provenance."},"author_rebuttal":null,"desk_editor":{"model":"glm-5.2","letter":"The paper you should know about here is a straightforward benchmark: 8 open-source VLMs (Qwen3-VL, Qwen3.5, Gemma 3) evaluated zero-shot, fully finetuned, and few-shot across three DocVQA domains — industrial documents (SP-DocVQA), infographics, and slides. The practical takeaway is that 50 target-domain samples on top of a cross-domain finetuned model can match or slightly exceed full supervised finetuning on SlideVQA. That is a useful data point for practitioners in low-resource settings, and the cross-domain transfer matrix across model families and scales is genuinely new as a consolidated reference. Credit is due for running a consistent experimental setup (LoRA rank 32, 3 epochs, unified 1449×1449 resolution, ANLS metric) across all 8 models and 3 datasets — that is real work and the results are reproducible in principle. The finding that smaller models gain more from finetuning is consistent with prior literature but well-documented here with specific numbers. The soft spot is load-bearing. The paper claims 'visual understanding is the main bottleneck for DocVQA, not a lack of knowledge.' This is inferred from performance patterns — zero-shot drops on complex layouts, finetuning helps, 50-shot is surprisingly effective — but no ablation isolates visual understanding from confounds. The three datasets differ in question types (SP-DocVQA is purely extractive; InfographicsVQA has non-extractive numerical reasoning; SlideVQA has multi-hop, partially filtered). Performance gaps could reflect reasoning demands or answer-format distributions, not visual processing. A text-only or OCR-augmented baseline on the same questions would directly test the attribution. Without it, the claim is one of several plausible interpretations. The few-shot result also lacks variance estimates or statistical tests. The cases where 50-shot 'surpasses' full finetuning appear to be ~1 ANLS point differences, which could easily be noise. The SlideVQA filtering (keeping only single-hop questions) introduces a selection bias that goes unaddressed. These issues do not invalidate the empirical observations — the benchmark tables are solid — but they do condition the interpretive claims. Read as a benchmark report, it is useful. Read as a mechanistic explanation, it overreaches. This paper is for researchers and practitioners working on DocVQA domain adaptation who want a consolidated comparison of current open-source VLMs. It deserves a serious referee who can push back on the causal claim and request the missing ablation.","headline":"Useful DocVQA benchmark across 8 VLMs and 3 domains, but the central causal claim about visual understanding being the bottleneck is unsupported by the experimental design.","tokens_in":14928,"tokens_out":609,"would_cite":false,"duration_ms":37880,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"50 samples let VLMs adapt to new document domains — sometimes beating full retraining","keywords":[],"falsifier":"If an OCR-text-only baseline (where the model receives the document's text but not the image) matched or exceeded VLM performance on infographics and slides, the claim that visual understanding is the bottleneck would be undermined — it would suggest the models fail not at seeing but at reading.","tokens_in":14226,"feed_emoji":"📄","tokens_out":851,"duration_ms":103109,"temperature":0.7,"pith_summary":"This paper evaluates 8 open-source Vision-Language Models (VLMs) across three document domains — industrial forms, infographics, and presentation slides — to find out where these models break down when answering questions about document images. The central finding is that VLMs already carry enough internal knowledge to answer questions across domains, but they fail to extract visual information from complex, unstructured layouts. The evidence: zero-shot performance is strong on structured documents (above 90% accuracy on forms) but drops sharply on infographics and slides (below 60%). After supervised finetuning, smaller models gain the most relative ground, suggesting they had the knowledge all along but needed calibration. The most striking result comes from few-shot experiments: models previously finetuned on one document domain need only 50 samples from a new target domain to adapt rapidly, and in some cases they surpass models that were fully retrained on the entire target-domain dataset. This pattern — strong knowledge transfer with minimal data — implies the bottleneck is visual layout comprehension, not a deficit in stored knowledge.","feed_headline":"50 samples let VLMs adapt to new document domains — sometimes beating full retraining","feed_subtitle":"Vision-language models already know enough to answer questions about documents; the real bottleneck is parsing complex visual layouts, not a","key_machinery":"The mechanism carrying the argument is a three-stage evaluation protocol: (1) zero-shot evaluation across all domains establishes a baseline showing where pretrained knowledge already suffices; (2) fully supervised finetuning on each domain separately, followed by cross-domain evaluation, reveals which domains transfer well and which do not; (3) few-shot adaptation, where models already finetuned on a source domain receive 5, 20, or 50 samples from a target domain. The convergence of these three stages — strong zero-shot on structured layouts, transfer gains correlated with layout similarity, and rapid adaptation with 50 samples — forms the evidential chain that isolates visual layout parse-","core_discovery":"The paper's central claim is that visual understanding, not knowledge capacity, is the primary bottleneck for Document Visual Question Answering. The authors establish this by showing that VLMs transfer knowledge across document domains with as few as 50 target samples, sometimes outperforming full retraining — a result that would be impossible if the models lacked domain knowledge rather than the ability to parse complex visual layouts.","pith_inferences":[],"forward_implications":["Practitioners deploying VLMs on new document domains may not need large labeled datasets: 50 samples from the target domain, layered on top of existing finetuning, can be sufficient for adaptation.","Model architecture research for document AI should prioritize visual layout comprehension modules over knowledge capacity, since the knowledge bottleneck appears already solved by current pretraining.","Cross-domain transfer is not symmetric: domains with similar layout complexity (infographics to slides) transfer more effectively than structurally dissimilar pairs (forms to slides), which has implications for training data selection strategies.","Smaller VLMs may be more cost-effective than larger ones for domain-specific DocVQA, since they show the highest relative gains from finetuning and can match or exceed larger models after adaptation."],"fun_headline_variants":["Visual parsing, not knowledge, is the bottleneck for document VQA","8 VLMs tested on DocVQA: visual understanding beats knowledge capacity","Fifty samples transfer VLMs across document domains — sometimes beating retraining","Infographics and slides expose visual parsing limits of large VLMs","Smaller VLMs gain more from finetuning than larger architectures on DocVQA"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The paper attributes performance gaps to visual understanding capacity by inference from accuracy patterns across domains, without isolating visual comprehension through a direct ablation such as an OCR-text-only baseline. The claim that visual understanding is the bottleneck rests on the assumption that accuracy differences between settings can be cleanly attributed to visual parsing rather than to confounding factors like prompt format, finetuning hyperparameters, or the 3-","fun_headline_variants_meta":{"raw":{"variants":["Visual parsing, not knowledge, is the bottleneck for document VQA","8 VLMs tested on DocVQA: visual understanding beats knowledge capacity","Fifty samples transfer VLMs across document domains — sometimes beating retraining","Infographics and slides expose visual parsing limits of large VLMs","Smaller VLMs gain more from finetuning than larger architectures on DocVQA"]},"model":"glm-5.2","effort":"low","cost_usd":0.0,"raw_usage":{"total_tokens":652,"prompt_tokens":553,"completion_tokens":99,"prompt_tokens_details":null},"tokens_in":553,"tokens_out":99,"duration_ms":34638,"temperature":1.0,"reasoning_tokens":null,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-09T18:11:06.725608+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If an OCR-text-only baseline (where the model receives the document's text but not the image) matched or exceeded VLM performance on infographics and slides, the claim that visual understanding is the bottleneck would be undermined — it would suggest the models fail not at seeing but at reading.","supporting_citations":[],"review_version":1}