REVIEW 2 major objections 6 minor 47 references
Comparative Study of Domain-adapted VLMs for General Document Visual Question Answering
T0 review · 2 major / 6 minor · reviewed 2026-07-09 · glm-5.2
Pith's one-line read 50 samples let VLMs adapt to new document domains — sometimes beating full retraining
desk verdict Useful DocVQA benchmark across 8 VLMs and 3 domains, but the central causal claim about visual understanding being the bottleneck is unsupported by the experimental design. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism carrying the argument is a three-stage evaluation protocol: (1) zero-shot evaluation across all domains establishes a baseline showing where pretrained knowledge already suffices; (2) fully supervised finetuning on each domain separately, followed by cross-domain evaluation, reveals which domains transfer well and which do not; (3) few-shot adaptation, where models already finetuned on a source domain receive 5, 20, or 50 samples from a target domain. The convergence of these three stages — strong zero-shot on structured layouts, transfer gains correlated with layout similarity, and rapid adaptation with 50 samples — forms the evidential chain that isolates visual layout parse-
What would settle it
If an OCR-text-only baseline (where the model receives the document's text but not the image) matched or exceeded VLM performance on infographics and slides, the claim that visual understanding is the bottleneck would be undermined — it would suggest the models fail not at seeing but at reading.
Extended reading notes
Core claim
The paper's central claim is that visual understanding, not knowledge capacity, is the primary bottleneck for Document Visual Question Answering. The authors establish this by showing that VLMs transfer knowledge across document domains with as few as 50 target samples, sometimes outperforming full retraining — a result that would be impossible if the models lacked domain knowledge rather than the ability to parse complex visual layouts.
Load-bearing premise
The paper attributes performance gaps to visual understanding capacity by inference from accuracy patterns across domains, without isolating visual comprehension through a direct ablation such as an OCR-text-only baseline. The claim that visual understanding is the bottleneck rests on the assumption that accuracy differences between settings can be cleanly attributed to visual parsing rather than to confounding factors like prompt format, finetuning hyperparameters, or the 3-
Editorial extensions
If this is right
- Practitioners deploying VLMs on new document domains may not need large labeled datasets: 50 samples from the target domain, layered on top of existing finetuning, can be sufficient for adaptation.
- Model architecture research for document AI should prioritize visual layout comprehension modules over knowledge capacity, since the knowledge bottleneck appears already solved by current pretraining.
- Cross-domain transfer is not symmetric: domains with similar layout complexity (infographics to slides) transfer more effectively than structurally dissimilar pairs (forms to slides), which has implications for training data selection strategies.
- Smaller VLMs may be more cost-effective than larger ones for domain-specific DocVQA, since they show the highest relative gains from finetuning and can match or exceed larger models after adaptation.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper presents an empirical evaluation of 8 open-source Vision-Language Models (VLMs) on Document Visual Question Answering (DocVQA) across three domains: industrial documents (SP-DocVQA), infographics (InfographicsVQA), and presentation slides (SlideVQA). The authors evaluate zero-shot performance, fully supervised fine-tuning with cross-domain transfer, and few-shot adaptation (5, 20, 50 samples) from source domains to a target domain (SlideVQA). Key findings include: (1) zero-shot performance is strong on structured layouts but drops sharply on visually complex domains; (2) smaller models gain more from fine-tuning; (3) 50-shot adaptation can match or exceed fully supervised fine-tuning on the target domain. The paper concludes that visual understanding, rather than lack of knowledge, is the primary bottleneck for DocVQA.
Significance. The paper addresses a practically important question—how well do current VLMs transfer across document domains—and provides a useful comparative benchmark across three well-chosen datasets and multiple model scales. The few-shot adaptation finding (50 samples approaching or matching full fine-tuning) is practically valuable for low-resource deployment scenarios. The systematic cross-domain evaluation matrix (Tables 2–4) is a useful contribution for practitioners. However, the central interpretive claim about visual understanding being the bottleneck is not directly tested, which limits the depth of insight.
major comments (2)
- §4.3, §5, and Abstract: The claim that 'visual understanding is the main bottleneck for DocVQA, not a lack of knowledge from the VLMs' is an interpretive inference drawn from performance patterns across datasets that differ in multiple confounding factors. The three datasets differ not only in visual complexity but also in question types: SP-DocVQA is purely extractive, InfographicsVQA includes non-extractive numerical reasoning (§3.2), and SlideVQA includes visual-cue and numerical questions (§3.2). Performance gaps could therefore reflect differences in reasoning demands or answer-format distributions rather than visual understanding per se. A text-only or OCR-augmented baseline (providing extracted text without the image, or image+OCR text) on the same questions would directly isolate whether the bottleneck is visual processing or reasoning difficulty. Without such an ablation, the因果性
- §4.3, Fig. 4, and Abstract: The claim that 50-shot results 'surpass fully supervised counterparts in some cases' is not supported by variance estimates or statistical tests. Examining Fig. 4, the cases where 50-shot exceeds full fine-tuning appear marginal (e.g., Qwen3-VL 4B fine-tuned on SP-DocVQA: 50-shot ≈ 54.1 vs. full FT ≈ 50.9; Qwen3.5 9B: 50-shot ≈ 57.4 vs. full FT ≈ 57.9). Without confidence intervals or significance tests, these differences cannot be distinguished from noise. This is load-bearing for the paper's strongest claim and should be addressed by reporting variance across multiple seeds or providing statistical tests.
minor comments (6)
- Table 1 uses commas as decimal separators (e.g., '85,36') while Tables 2–4 use periods (e.g., '91.63'). Standardize the formatting.
- §3.1: The model family names are inconsistent—'Qwen3.5' in the text vs. 'Qwen 3.5' in tables and discussion. Also, the text references 'Qwen3-VL' but tables use 'Qwen3-VL'. Standardize.
- §3.2: The prompt used for zero-shot evaluation is specified, but it is unclear whether the same prompt was used during fine-tuning. Clarify.
- Fig. 4: The specific ANLS values are difficult to read from the bar chart. Consider supplementing with a table.
- §4.2: The terms 'inter-dataset' and 'intra-dataset' are used but not explicitly defined. Adding a brief definition would improve clarity.
- Reference [21] (ORCA) is cited as a 2026 arXiv preprint. Verify whether a peer-reviewed version exists.
Circularity Check
No circularity: empirical benchmark with measured results, no derivation chain
full rationale
This paper is an empirical benchmark study evaluating 8 VLMs on DocVQA tasks across three document domains. It contains no mathematical derivations, no fitted parameters that are then 'predicted,' and no theoretical claims whose proof reduces to self-citation. The central claims (zero-shot performance drops on complex layouts, smaller models gain more from finetuning, 50-shot adaptation is surprisingly effective) are all directly measured via ANLS scores reported in Tables 1–4 and Figure 4. The interpretive claim that 'visual understanding is the main bottleneck' is an inference from experimental patterns, not a derived result. While this inference could be challenged on correctness grounds (confounds from question-type differences, lack of OCR-only ablation, no variance estimates), that is a matter of experimental rigor, not circularity. Self-citations ([23], [27], [29]) appear in related work and future-work contexts and are not load-bearing for any central claim. The few-shot results are measured outcomes, not predictions forced by construction. No step in the paper's argument reduces to its inputs by definition or by self-citation chain.
Assumptions & free parameters
free parameters (6)
- LoRA rank =
32
- LoRA alpha =
4
- LoRA dropout =
0.2
- Number of epochs =
3
- Image resolution =
1449x1449
- Few-shot sample counts =
5, 20, 50
assumptions (4)
- domain assumption ANLS (Average Normalized Levenshtein Similarity) is an appropriate metric for measuring DocVQA performance across extractive and non-extractive question types.
- ad hoc to paper Performance differences across domains can be attributed to visual/layout complexity rather than question difficulty or answer distribution.
- domain assumption Filtering multi-hop questions from SlideVQA to create a single-page subset does not introduce systematic bias.
- domain assumption 3 epochs of LoRA finetuning is sufficient for convergence across all model sizes and datasets.
Cite this review
Pith. "Pith review of Comparative Study of Domain-adapted VLMs for General Document Visual Question Answering." pith.science (2026). https://pith.science/paper/O7MCSJU3
@misc{pith2026260707179,
author = {Pith},
title = {Pith review of: Comparative Study of Domain-adapted VLMs for General Document Visual Question Answering},
year = {2026},
howpublished = {\url{https://pith.science/paper/O7MCSJU3}},
note = {Machine review of arXiv:2607.07179}
}
read the original abstract
Document Visual Question Answering (DocVQA) presents a complex multimodal challenge, requiring models to exploit visual, textual, and layout information from documents. Although Vision-Language Models (VLMs) have shown remarkable performance in text-vision tasks, their robustness and transferability to different document domains remains underexplored. In this study, we present a comprehensive evaluation of 8 open-source pretrained VLMs on DocVQA in three different document domains: industrial documents of varying type, infographics, and presentation slides. We systematically assess model performance under zero-shot evaluations, fully supervised finetuning with inter- and intra-dataset evaluations, and few-shot learning evaluations of knowledge transfer between domains. Our findings demonstrate that while large pretrained VLMs possess strong zero-shot baselines for structured layouts, their performance strongly decreases on visually complex layouts of infographics and slides. Although parameter scaling is a dominant factor on performance, supervised finetuning yields higher relative gains in smaller architectures. Furthermore, our cross-domain and few-shot experiments show that visual understanding is the main bottleneck for DocVQA, not a lack of knowledge from the VLMs. Using 50 target domain samples, the models finetuned in DocVQA with datasets of different domains rapidly adapt to the target domain documents, even surpassing their fully supervised counterparts in some cases.
Figures
Reference graph
Works this paper leans on
- [1]
-
[2]
Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al.: Qwen3-VL Technical Report. arXiv:2511.21631 (2025)
work page Pith review arXiv 2025
-
[3]
In: NeurIPS 2022 First Table Representation Workshop
Chang, S., Palzer, D., et al.: MapQA: A dataset for question answering on choro- pleth maps. In: NeurIPS 2022 First Table Representation Workshop
work page 2022
-
[4]
Chen, A., Giovannini, S., Gemelli, A., Coppini, F., et al.: Towards reliable and interpretable document question answering via VLMs. arXiv:2509.10129 (2025)
-
[5]
Chen, Z., Wu, J., et al.: InternVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In: IEEE CVPR. pp. 24185–24198 (2024)
work page 2024
-
[6]
Daza, R., et al.: Evaluating Social Engineering Risks in AI-based Interaction using Biometrics and a Gaming Setup. In: IEEE ICCST (2026)
work page 2026
-
[7]
DeAlcala, D., et al.: AttZoom: Attention zoom for better visual features. In: IEEE/CVF Intl. Conf. on Computer Vision Workshops (2025)
work page 2025
-
[8]
DeAlcala, D., et al.: Is my vision-language data in your AI? Membership Inference Test (MINT) Demo 2. In: IEEE COMPSAC (2026)
work page 2026
Show all 47 references
-
[9]
Part 1: Fundamentals and review
Fierrez, J., et al.: Multiple classifiers in biometrics. Part 1: Fundamentals and review. Information Fusion44, 57–64 (November 2018)
2018
-
[10]
IEEE Access5(2017)
Gomez-Barrero, M., et al.: Privacy-preserving comparison of variable-length data with application to biometric template protection. IEEE Access5(2017)
2017
-
[11]
In: Joint Workshop of the Financial Tech
Harsha, C., Phogat, K.S., et al.: Synthetic data generation using large language models for financial question answering. In: Joint Workshop of the Financial Tech. and Natural Language Processing (FinNLP). pp. 76–95 (2025)
2025
-
[12]
In: AAAI Conf
Hong, T., Kim, D., et al.: BROS: A pre-trained language model focusing on text and layout for better key information extraction from documents. In: AAAI Conf. on Artificial Intelligence. vol. 36, pp. 10767–10775 (2022) 16 M. Lopez-Duran, E. Marrero, J. Fierrez, et al
2022
-
[13]
ICLR1(2), 3 (2022)
Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., et al.: LoRA: Low-rank adaptation of large language models. ICLR1(2), 3 (2022)
2022
-
[14]
In: ACM Intl
Huang, Y., et al.: LayoutLMv3: Pre-training for document AI with unified text and image masking. In: ACM Intl. Conf. on Multimedia. pp. 4083–4091 (2022)
2022
-
[15]
In: IEEE CVPR
Hudson, D.A., Manning, C.D.: GQA: A new dataset for real-world visual reasoning and compositional question answering. In: IEEE CVPR. pp. 6693–6702 (2019)
2019
-
[16]
In: IEEE ICCST (2026)
Irigoyen, J., et al.: AIriskEval-edu: New dataset for risk assessment in AI-mediated K-12 educational explanations. In: IEEE ICCST (2026)
2026
-
[17]
In: Pro- ceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Jiang, Z., Mao, Y., He, P., Neubig, G., Chen, W.: OmniTab: Pretraining with natural and synthetic data for few-shot table-based question answering. In: Pro- ceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human La...
2022
-
[18]
Khan, Z., BG, V.K., Schulter, S., Yu, X., Fu, Y., Chandraker, M.: Q: How to specialize large vision-language models to data-scarce VQA tasks? A: Self-train on unlabeled images! In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15005–150...
2023
-
[19]
In: European Conference on Computer Vision
Kim, G., Hong, T., Yim, M., Nam, J., Park, J., Yim, J., Hwang, W., Yun, S., Han, D., Park, S.: OCR-free document understanding transformer. In: European Conference on Computer Vision. pp. 498–517. Springer (2022)
2022
-
[20]
In: IEEE ICCV Workshops (2025)
Korshunov, P., et al.: DeepID challenge of detecting synthetic manipulations in ID documents. In: IEEE ICCV Workshops (2025)
2025
-
[21]
arXiv preprint arXiv:2603.02438 (2026)
Lassoued, A., Souibgui, M.A., Kessentini, Y.: ORCA: Orchestrated reasoning with collaborative agents for document visual question answering. arXiv preprint arXiv:2603.02438 (2026)
2026
-
[22]
In: Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition
Li, X., Wu, Y., Jiang, X., Guo, Z., Gong, M., Cao, H., Liu, Y., Jiang, D., Sun, X.: Enhancing visual document understanding with contrastive learning in large visual-language models. In: Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition. pp. 1...
2024
-
[23]
In: International Conference on Document Analysis and Recognition
Lopez-Duran, M., Fierrez, J., Morales, A., Tolosana, R., Delgado-Mohatar, O., Ortigosa, A.: Benchmarking graph neural networks for document layout analysis in public affairs. In: International Conference on Document Analysis and Recognition. pp. 21–35. Springer (2025)
2025
-
[24]
arXiv:2603.02150 (2026)
Lopez-Duran, M., et al.: Named-entity recognition in the crime domain (CrimeNER): Case study and dataset. arXiv:2603.02150 (2026)
2026 arXiv
-
[25]
In: Proc
Luo, C., Shen, Y., Zhu, Z., Zheng, Q., Yu, Z., Yao, C.: LayoutLLM: Layout instruc- tion tuning with large language models for document understanding. In: Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition. pp. 15630– 15640 (2024)
2024
-
[26]
Bioengineering13(5), 511 (2026)
Macias-Fassio,E.,Morales,A.,Pruenza,C.,Fierrez,J.,Espósito,C.:Privacy-aware synthetic tabular data generation for healthcare: Application to sepsis detection. Bioengineering13(5), 511 (2026)
2026
-
[27]
In: IEEE COMPSAC (2026)
Mancera, G., DeAlcala, D., Morales, A., et al.: Auditing training data in domain- adapted LLMs: LoRA-MINT. In: IEEE COMPSAC (2026)
2026
-
[28]
In: AAAI Workshop on Deployable AI (2025)
Mancera, G., DeAlcala, D., et al.: Membership inference test: Auditing training data in object classification models. In: AAAI Workshop on Deployable AI (2025)
2025
-
[29]
In: IEEE ICSST (2026)
Mancera, G., Fierrez, J., et al.: inLoRA-MINT: Internal representations for audit- ing training data in domain-adapted LLMs. In: IEEE ICSST (2026)
2026
-
[30]
In: International Conference on Document Analysis and Recognition
Mancera, G., Morales, A., Fierrez, J., Tolosana, R., Peña, A., Lopez-Duran, M., Jurado, F., Ortigosa, A.: PBa-LLM: Privacy- and bias-aware NLP using named- entity recognition (NER). In: International Conference on Document Analysis and Recognition. pp. 3–20. Springer (2025) Co...
2025
-
[31]
In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision
Mathew,M.,Bagal,V.,Tito,R.,Karatzas,D.,Valveny,E.,Jawahar,C.:Infograph- icVQA. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 1697–1706 (2022)
2022
-
[32]
In: Proc
Mathew,M.,Karatzas,D.,Jawahar,C.:DocVQA:AdatasetforVQAondocument images. In: Proc. of the IEEE/CVF Winter Conf. on Applications of Computer Vision. pp. 2200–2209 (2021)
2021
-
[33]
Informa- tion Fusion128, 103969 (2026)
Muñoz-Haro, J., Tolosana, R., et al.: Privacy-aware detection of fake identity doc- uments: methodology, benchmark, and improved algorithms (FakeIDet2). Informa- tion Fusion128, 103969 (2026)
2026
-
[34]
In: AAAI/ACM AIES (2025)
Peña, A., et al.: Addressing bias in LLMs: Strategies and application to fair AI- based recruitment. In: AAAI/ACM AIES (2025)
2025
-
[35]
In: IAPR Intl
Peña, A., Morales, A., Fierrez, J., Ortega-Garcia, J., Grande, M., Puente, I., Cor- dova, J., Cordova, G.: Document layout annotation: Database and benchkmark in the domain of public affairs. In: IAPR Intl. Conf. on Document Analysis and Recognition Workshops (ICDARw). LNCS, v...
2023
-
[36]
Information Fusion108, 102398 (2024)
Peña,A.,etal.:Continuousdocumentlayoutanalysis:Human-in-the-loopAI-based data curation, database, and evaluation in the domain of public affairs. Information Fusion108, 102398 (2024)
2024
-
[37]
In: Proc
Pramanick, S., Chellappa, R., Venugopalan, S.: SPIQA: A dataset for multimodal question answering on scientific papers. In: Proc. of the Intl. Conf. on Neural Information Processing Systems. pp. 118807–118833 (2024)
2024
-
[38]
5-Omni Technical Report
Qwen-Team: Qwen3. 5-Omni Technical Report. arXiv:2604.15804 (2026)
2026 arXiv
-
[39]
In: Intl
Raja, S., Mondal, A., Jawahar, C.: ICDAR 2023 competition on visual question answering on business document images. In: Intl. Conf. on Document Analysis and Recognition. pp. 454–470 (2023)
2023
-
[40]
Computers in Biology and Medicine209, 111676 (2026)
Romero-Tapiador, S., Tolosana, R., Morales, A., et al.: Personalized weight loss management through wearable devices and artificial intelligence. Computers in Biology and Medicine209, 111676 (2026)
2026
-
[41]
In: IEEE/CVF CVPR Workshops (2025)
Romero-Tapiador, S., et al.: Are Vision-Language Models ready for dietary as- sessment? Exploring the next frontier in AI-powered food image recognition. In: IEEE/CVF CVPR Workshops (2025)
2025
-
[42]
Pattern Recognition (2027)
Serna, I., Morales, A., Fierrez, J.: Unraveling machine behavior by multi-level bias analysis and detection: Methodology and application to computer vision. Pattern Recognition (2027)
2027
-
[43]
In: Proc
Tanaka, R., Nishida, K., Nishida, K., Hasegawa, T., Saito, I., Saito, K.: SlideVQA: A dataset for document visual question answering on multiple images. In: Proc. of the AAAI Conf. on Artificial Intelligence. pp. 13636–13645 (2023)
2023
-
[44]
In: ECAI Workshops
Tello, J., de la Cruz, M., Ribeiro, T., et al.: Symbolic AI (LFIT) for XAI to handle biases. In: ECAI Workshops. CEUR-WS, vol. 3523 (2023)
2023
-
[45]
on DocumentAnalysis and Recognition
Tito, R., Mathew, M., Jawahar, C., Valveny, E., Karatzas, D.: ICDAR 2021 compe- tition ondocument visualquestion answering.In:Intl.Conf. on DocumentAnalysis and Recognition. pp. 635–649. Springer (2021)
2021
-
[46]
In: Proceedings of the 34th ACM International Conference on Information and Knowledge Management
Vafaie, M., Hertling, S., Banse-Strobel, I., Dubout, K., Sack, H.: End-to-end infor- mation extraction from archival records with multimodal large language models. In: Proceedings of the 34th ACM International Conference on Information and Knowledge Management. pp. 6075–6083 (2025)
2025
-
[47]
In: Proc
Xu, Y., Xu, Y., Lv, T., Cui, L., Wei, F., Wang, G., Lu, Y., Florencio, D., Zhang, C., Che, W., et al.: LayoutLMv2: Multi-modal pre-training for visually-rich document understanding. In: Proc. of ACL. pp. 2579–2591 (2021)
2021
Reviewed July 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.