REVIEW 4 major objections 4 minor 3 references
DocHop-QA: Towards Multi-Hop Reasoning over Multimodal Document Collections
T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read DocHop-QA introduces 11,379 PubMed-derived QA instances that require models to synthesize evidence across multiple documents, tables, and layouts without explicit links, and reports that current models perform poorly.
desk verdict A genuinely useful new benchmark for multimodal multi-hop scientific QA, but the 'rigorous testbed' claim outruns the evidence because the gold answers come from GPT-4.1 and the paper itself admits to hallucinated intermediate hops in generation. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is the generation-and-evaluation pipeline built around an 11-concept taxonomy of scientific reasoning. The taxonomy defines question templates with typed placeholders (<Problem>, <Solution>, <Mechanism>, <Table>, etc.), which are instantiated by an instruction-tuned Qwen2 model to produce diverse questions. Answer matching uses GPT-4.1 to synthesize gold answers from reference snippets. Four evaluation tasks then separate generative fluency from evidence localization: free-form answer generation, structured entity-index list generation, and two multi-label classification tasks over predicted bounding boxes or XML entities. The design claim is that because no explicit hyperlinks
What would settle it
Take a random sample of 200 DocHop-QA instances, have biomedical experts independently write gold answers from the same reference snippets, and measure agreement on evidence support. If expert answers frequently disagree with the dataset's gold answers about which snippets are required, or if models re-scored against expert answers rank differently, the claim that scores measure reasoning would be falsified.
Extended reading notes
Core claim
The central claim is that DocHop-QA is a testbed for multimodal, multi-document, multi-hop scientific QA that removes the crutches earlier benchmarks rely on. Document pairs are selected by keyword co-occurrence and semantic similarity over titles and abstracts, not by citation links; the 11 reasoning concepts encode the question structures scientists actually use; and answers require aggregating paragraph- and table-level snippets across both documents. The paper reports that answer snippets are non-position-biased across document sections and that tables appear not only as terminal evidence but also as intermediate bridges in reasoning chains. Evaluations across four tasks—generative text
Load-bearing premise
The benchmark's validity rests on the GPT-4.1-synthesized gold answers being accurate, complete, and unbiased; if those answers contain systematic errors or stylistic regularities, model scores measure agreement with GPT-4.1 rather than scientific multi-hop reasoning.
Editorial extensions
If this is right
- If DocHop-QA is valid, a model's score reflects cross-document synthesis rather than link-following, so progress on it would indicate real progress in multimodal scientific reading.
- The four tasks separate generative fluency from evidence localization, allowing researchers to diagnose whether a system fails because it cannot find the right spans or because it cannot phrase a synthesis.
- The 11 reasoning concepts are domain-agnostic, so the same pipeline can generate benchmarks from other scientific corpora, enabling cross-domain comparison.
- The finding that tables serve as intermediate reasoning bridges suggests that table-localization performance can be used as a proxy for chain-reasoning ability even without explicit chain annotations.
- Because the dataset includes unanswerable questions, it offers a controlled setting for studying hallucination and answerability detection.
Reading between the lines
- The dependence on GPT-4.1 for gold answers means scores partly measure agreement with GPT-4.1's phrasing; swapping in human-authored gold answers for a subset and re-ranking models would test whether the benchmark measures reasoning or style-matching.
- Pairing documents by semantic similarity could create pairs that share boilerplate or overlapping background text, so a leakage audit checking n-gram overlap within paired documents would clarify whether some hops are answerable from a single paragraph.
- The table-oriented questions are generated more deterministically and show limited lexical diversity, which suggests the table tasks may reward template-following; a human rating of table-question quality would test this.
- The document-level split (70/10/20) lets future work ask whether models generalize across reasoning concepts or overfit to document layouts and section conventions.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. DocHop-QA introduces a benchmark of 11,379 QA instances constructed from PubMed Central articles, spanning text, tables, and layout, with four evaluation tasks. The authors propose an LLM-driven generation pipeline organized around 11 scientific reasoning concepts, and report experiments showing that current models achieve low BLEU/ROUGE scores, especially GPT-4o, Gemini, and Qwen variants. The central claim is that DocHop-QA is a rigorous testbed for multimodal, multi-document, multi-hop scientific QA because it requires cross-document evidence synthesis without explicit hyperlinks, and that current models struggle with its long-context, multi-evidence demands.
Significance. If the benchmark's gold answers are reliable and the generation pipeline is sound, DocHop-QA fills a genuine gap: no existing benchmark combines multimodal (text+table+layout) evidence, multi-document settings, and paragraph-level synthesized answers from real scientific papers. The four-task evaluation framework and the concept-driven question taxonomy are useful for future work. However, the paper's central load-bearing assertion—that DocHop-QA is a 'rigorous testbed'—depends on the correctness of the gold answers and on the claim that questions are answerable solely from the provided documents. The manuscript contains explicit evidence that the generation pipeline can produce unsupported reasoning chains, yet it provides no quantitative estimate of how often this occurs or any inter-annotator validation beyond a qualitative mention. Until this is addressed, the reported scores conflate model failure with label corruption, and the rigor claim is not empirically supported.
major comments (4)
- [Abstract vs. §3.2] The abstract states that DocHop-QA enables 'cross-document inference without explicit hyperlinks,' and §2 contrasts the method with M3SciQA, which 'relies on citation-based anchor links.' However, §3.2 states that Table-Oriented types 'intentionally use citation-linked tables to model structured chain-hop reasoning' and that TS/TSL types target systematic reviews containing citation-linked tables. This is a direct contradiction. If a substantial portion of the dataset uses citation links as structural anchors, the claim that the benchmark does not rely on explicit hyperlinks is misleading, and the comparison with prior work is overstated. Please clarify the exact fraction of instances using citation-linked tables and adjust the abstract/positioning accordingly.
- [Appendix G.3, t-SNE analysis] The paper acknowledges that 'the tendency of dataset-generation LLMs to introduce hallucinated intermediate hops' is evident in the unanswerable cases in DocHop-QA. This is an explicit admission that some gold instances are unanswerable or contain unsupported reasoning chains. The manuscript does not report how many such instances exist, how they were detected, or whether they were removed or flagged. If even a modest fraction of the 11,379 instances are corrupted, the aggregate BLEU/ROUGE scores in Table 5 and the structured-task results are not interpretable as measures of multi-hop reasoning. The authors should report a quantitative count, rejection rate, and a manual audit of a random sample of unanswerable cases.
- [Appendix G.2 and Ethics Statement] Gold answers for Task 1 are synthesized by GPT-4.1 from reference context snippets, and the Ethics Statement cites 'human validation' only qualitatively. No details are given on the number of human annotators, the annotation interface, the instructions, or inter-annotator agreement. Since all four tasks use these gold answers or the underlying reference snippets as ground truth, the validity of every reported number depends on the quality of this validation. The paper needs to provide quantitative human-evaluation results, including agreement statistics and a per-concept breakdown. Without this, the claim that DocHop-QA is a 'rigorous testbed' is unsupported.
- [§5.3 and Table 4] Table 4 reports an average hop count of 4.63 for DocHop-QA, but 'hop' is not defined operationally for this dataset. The reader is left to infer whether hops correspond to evidence snippets, document sections, or reasoning steps in the LLM-generated chain. Since the paper's central differentiator is 'multi-hop' reasoning, the hop definition should be stated explicitly and verified against the dataset, ideally with human annotation of a sample.
minor comments (4)
- [Abstract] 'Existing QAs remain narrow in scope' is awkward; QA is typically used as a countable plural or as a field. Consider 'Existing QA benchmarks remain narrow.'
- [§5.1] The sentence 'The questions constructed from these documents inherit their semantic richness, reflecting the depth and complexity of biomedical literature' is duplicated verbatim in consecutive sentences.
- [Table 2] The category labels 'Non-ReferTable' and 'ReferTable' are non-standard; consider 'Table-Independent' and 'Table-Referencing' or similar clearer terminology.
- [Appendix G.3] The t-SNE figure caption says 'unanswerable cases show a distinct grouping' but the body text does not define what constitutes an 'unanswerable case' or how it was determined. This needs a precise definition.
Circularity Check
No significant circularity: DocHop-QA is a benchmark construction with externally evaluated model performance, not a derivation that reduces to its own inputs.
full rationale
DocHop-QA is a dataset and evaluation benchmark, not a chain of derived predictions. Its central claim—that current models struggle on multimodal, multi-document, multi-hop scientific QA—is an empirical result measured on held-out instances against gold answers synthesized from reference snippets (Appendix G.2). This synthesis is a dataset-generation artifact, not a fitted parameter renamed as a prediction: the gold answers are produced from source-document evidence, and the evaluated models never influence those gold answers. The self-citation to MMVQA (Ding et al., 2024) appears only in related work and is not load-bearing. The 11 reasoning concepts and the fan-hop/chain-hop analyses describe internal properties of the constructed data; they are consistency checks, not circular reductions of a derivation to its inputs. The admitted hallucinated intermediate hops (Appendix G.3) and stylistic regularities (Limitations) are validity concerns about label quality, but they do not make the benchmark's construction equivalent to its outputs. No equation or construction step collapses into itself, and no load-bearing argument depends on an unverified self-citation. The paper is therefore not circular; any weaknesses are data-quality and evaluation-validity issues, not circularity.
Assumptions & free parameters
free parameters (3)
- 11 reasoning concepts =
11
- Document pair selection similarity threshold =
not specified
- Train/dev/test split =
70/10/20
assumptions (3)
- domain assumption LLM-generated QA instances are semantically correct and coherent.
- domain assumption GPT-4.1-synthesized gold answers are accurate references.
- domain assumption Citation links in Table-Oriented types do not violate the 'without explicit hyperlinks' claim.
Cite this review
Pith. "Pith review of DocHop-QA: Towards Multi-Hop Reasoning over Multimodal Document Collections." pith.science (2026). https://pith.science/paper/AS7HDM7E
@misc{pith2026250815851,
author = {Pith},
title = {Pith review of: DocHop-QA: Towards Multi-Hop Reasoning over Multimodal Document Collections},
year = {2026},
howpublished = {\url{https://pith.science/paper/AS7HDM7E}},
note = {Machine review of arXiv:2508.15851}
}
read the original abstract
Despite rapid progress in large language models (LLMs), current QA benchmarks still overlook the core challenge of real-world scientific information seeking: synthesizing multimodal evidence scattered across multiple documents and structural formats. Existing QA benchmarks remain narrow in scope, relying on unimodal text and short-span reasoning that fail to capture the complexity of real information seeking. We introduce DocHop-QA, a benchmark of 11,379 instances for evaluating multimodal, multi-document, multi-hop scientific QA. Built from publicly available PubMed articles, DocHop-QA incorporates textual passages, tables, and layout cues, enabling cross-document inference without explicit hyperlinks. To scale realistic QA construction, we develop an LLM-driven generation pipeline grounded in 11 scientific reasoning concepts, producing diverse and coherent question-answer pairs. To highlight the utility and versatility of the dataset, we propose a task-driven evaluation framework spanning four settings, including generative answering, multimodal evidence integration, and structured index prediction. Experiments show that current models struggle with the long-context and multi-evidence demands of DocHop-QA, establishing it as a rigorous testbed for advancing next-generation scientific QA systems.
Reference graph
Works this paper leans on
-
[2018]
Exploiting Reasoning Chains for Multi-hop Science Question Answering
Constructing datasets for multi-hop reading comprehension across documents.Transactions of the Association for Computational Linguistics, 6:287– 302. Jian Wu, Linyi Yang, Dongyuan Li, Yuliang Ji, Manabu Okumura, and Yue Zhang. 2025. Mmqa: Evaluating llms with multi-table multi-hop complex questions. InThe Thirteenth International Conference on Learn- ing ...
work page Pith review arXiv 2025
-
[2020]
Longformer: The long-document transformer. arXiv:2004.05150. Yingshan Chang, Mridu Narang, Hisami Suzuki, Gui- hong Cao, Jianfeng Gao, and Yonatan Bisk. 2022. Webqa: Multihop and multimodal qa. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16495–16504. Jifan Chen, Shih-ting Lin, and Greg Durrett. 2019. Multi-ho...
arXiv 2004
-
[2024]
InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 22266–22275
Interpretable long-form legal question answer- ing with retrieval-augmented large language models. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 22266–22275. Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthe- nis Karatzas, Ernest Valveny, and CV Jawahar. 2022. Infographicvqa. InProceedings of the IEEE/CVF Winter Conferen...
arXiv 2022
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.