REVIEW 3 major objections 5 minor 15 references
DocTrace: Towards Traceable Long Document VQA via Hierarchical Evidence Graph Reasoning
T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read DocTrace claims that long-document VQA should be modelled as explicit evidence-graph reasoning, and that this yields both higher accuracy and node-level provenance for every answer.
desk verdict Evidence-graph reasoning is a genuinely effective idea for long-doc VQA; the accuracy results look credible, but the traceability claim is shakier than the abstract suggests and the paper's own limitation note confirms it. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the evidence graph $G=(V,E)$: nodes are grounded layout blocks or intermediate conclusions, directed edges are reasoning dependencies. Answer generation follows $P(A,G\mid B,Q)=P(G\mid B,Q)P(A\mid G,Q)$: build the graph, then answer from it. The training machinery is two-stage: joint SFT initializes page localization and graph generation; task-specific GRPO then maximizes a distance-aware soft $F_\beta$ localization reward, a weighted graph-faithfulness reward (grounding, completeness, structural validity, dependency topology), and an answer-correctness reward combining exact and LLM-based semantic matching. This makes evidence composition a supervised, optimizable
What would settle it
Train the identical SFT-plus-GRPO recipe with the evidence-graph supervision corrupted — same node sets, dependency edges randomly permuted — and compare MMLongBench-Doc accuracy and the cited-versus-uncited masking gap. If accuracy stays near 52.9 and the masking gap stays near 82.8 versus 9.6, the graph's dependency topology is not what carries the result; if either collapses, the explicit graph is doing the causal work.
Extended reading notes
Core claim
Central claim: long-document VQA should generate an explicit evidence graph before answering, making answers both more accurate and inspectable. DocTrace localizes relevant pages at low resolution, parses them into typed layout blocks, then builds a DAG whose nodes are grounded blocks or derived conclusions and whose edges are reasoning dependencies; the final answer is generated conditioned on that graph. Reported results are 52.9 accuracy on MMLongBench-Doc, 56.4 on LongDocURL, and 85.1 F1 on SlideVQA, ahead of all compared open-source methods and competitive with the listed proprietary models, with gains of 14.4, 11.3, and 11.7 points over the Qwen3-VL-8B-Instruct backbone. Masking cited
Load-bearing premise
Everything rests on the automatically generated evidence graphs being correct ground truth for reasoning structure: if the teacher's graphs do not match the reasoning the human-annotated answers actually require, both the accuracy gain and the traceability claim are learning a teacher artifact rather than verified evidence composition.
Editorial extensions
If this is right
- On the three tested benchmarks, explicit graph reasoning beats all compared open-source methods and stays competitive with proprietary models, offering a viable open-source path for long-document understanding.
- Every answer comes with a node-level provenance graph, making high-stakes document QA auditable in a way end-to-end and retrieval-based systems are not.
- The two training stages play complementary roles: localization GRPO mainly raises evidence recall, while graph-and-answer GRPO mainly improves reasoning and calibrated refusal on unanswerable questions.
- Because only localized pages are parsed at high resolution, inference latency drops 2.1x and prefill tokens 2.3x relative to the backbone, and accuracy degrades much more slowly on documents beyond 60 pages.
- The stage-wise error analysis implies the main path to further gains is improving evidence localization and answerability decisions, not the graph reasoner itself.
Reading between the lines
- The paper's counterfactual test shows cited evidence is functionally necessary for the model's prediction, not that the model internally reasons through the graph; a human study comparing graphs to expert rationales would test the stronger claim.
- Because the same teacher-generated graphs define both SFT targets and the GRPO graph-faithfulness reward, accuracy and traceability are not independent; corrupting only the graph topology (shuffling edges while keeping nodes) would reveal whether dependency structure or merely node selection carries the gain.
- The framework suggests a scale-up direction: train on short public benchmarks plus self-collected long documents, then push evidence localization beyond the 120-page regime; the paper's own length analysis suggests localization, not graph reasoning, will become the limiting component.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. DocTrace casts LongDocVQA as an explicit evidence-graph reasoning problem. The pipeline first localizes question-relevant pages from low-resolution images (Stage 1), parses those pages into structured layout elements (Stage 2), and then constructs an evidence graph of grounded and derived nodes that is used to generate the final answer (Stage 3). Training uses automatically generated supervision: Gemini 3.1 Pro produces evidence pages and graphs, GPT-5.5 filters them, and the model is trained with joint SFT followed by task-specific GRPO with localization, graph-faithfulness, and answer-correctness rewards. Experiments on MMLongBench-Doc, LongDocURL, and SlideVQA report large gains over the Qwen3-VL-8B-Instruct backbone, and the paper claims both state-of-the-art accuracy and node-level evidence provenance.
Significance. If the empirical results hold, the paper makes a useful contribution: it treats evidence composition as an explicit structured object rather than an implicit latent process, provides a concrete hierarchical training pipeline for page localization plus graph reasoning, and includes several careful analyses (stage-wise error attribution, counterfactual evidence-masking, context-overflow analysis, efficiency measurement). The graph formalism and the joint SFT+GRPO recipe are likely to be of interest to the document understanding community. That said, the two load-bearing claims—consistent superiority over proprietary models and transparent verifiable traceability—are not equally supported. The traceability claim in particular rests on teacher-generated graphs that are never independently validated, and the paper itself acknowledges in Appendix D that the counterfactual test only measures behavioral dependence.
major comments (3)
- [Abstract and §4, Table 1] The abstract and §4 state that DocTrace 'consistently outperforms both existing open-source baselines and proprietary MLLMs.' This is contradicted by the paper's own Table 1: on LongDocURL, GPT-4o scores 64.5 while DocTrace(GRPO) scores 56.4. On MMLongBench-Doc, DocTrace does exceed GPT-4o and GPT-4.1, and on SlideVQA it exceeds the listed proprietary models, but on LongDocURL it does not. The claim should be restricted to open-source methods or qualified to 'most proprietary models.' This is not a presentation nit: the abstract's central claim is false as written.
- [Appendix A and §3, Eq. (3)] The evidence graphs used as supervision are generated by Gemini 3.1 Pro and filtered by GPT-5.5, with no human validation reported. These same teacher-produced graphs define both the SFT target and the graph-faithfulness reward R_graph in Eq. (3), creating a closed loop: the trained model is optimized to imitate the teacher's graph structure, and Table 3's 'provenance integrity' checks only internal well-formedness (valid block references, acyclicity, connectivity), not whether the graph reflects the reasoning that the human-annotated answer actually requires. Appendix D explicitly concedes that the counterfactual test measures behavioral dependence rather than internal fidelity to the graph. Consequently, the central traceability claim—'transparent and verifiable reasoning'—is not yet supported. The authors should provide human evaluation of the teacher graphs, a release of the generate
- [Appendix A 'Data Source' and §4 'Datasets'] SlideVQA is listed as one of the three benchmarks used to automatically construct the training corpus, and SlideVQA is also one of the three evaluation benchmarks. The paper does not state whether the SlideVQA training corpus was restricted to the official training split and evaluation was performed on the official test split. If the automatically constructed supervision includes SlideVQA test or validation questions — or even documents overlapping with the test set — the reported 85.1 F1 would be inflated by train/test leakage. This must be clarified and, if necessary, the evaluation rerun on a non-overlapping split. The same concern does not apply to MMLongBench-Doc or LongDocURL, which are not listed as training sources.
minor comments (5)
- [§4, Ablation Study] The text refers to 'Table 9' for the ablation results, but the ablation table in the manuscript is Table 4. Please renumber the cross-reference.
- [Figure 2] The figure contains several typos: 'consturcction', 'docuemnt', and the output tag '</asnwer>' should be '</answer>'. Also, the figure caption says 'Training data consturcution'.
- [§4, Table 1] The column header 'LongDoc.' and the use of '–' for missing proprietary results are understandable, but for clarity please note whether the missing values are due to the corresponding papers not reporting those benchmarks or to the models being inapplicable.
- [Appendix D] The sentence 'Among 1,049 valid chains...' and the following statistics are helpful; consider moving a short version into the main text, since Table 3 currently appears without the sample-size context. The n=813/319/313 details are important for judging the strength of the counterfactual evidence.
- [Appendix A] The training corpus statistics (Tables 5 and 6) are useful, but the annotation pipeline uses two closed-source models (Gemini 3.1 Pro and GPT-5.5) without reporting API versions, dates, or sample prompts. This makes reproduction difficult; please provide the full prompts and any filtering thresholds used in verification.
Circularity Check
No significant circularity: benchmark accuracies and causal-faithfulness tests are externally grounded; teacher-generated graph supervision is a data-quality limitation, not a definitional reduction.
full rationale
DocTrace's central empirical claims are accuracy numbers on MMLongBench-Doc, LongDocURL, and SlideVQA, measured against external human-annotated benchmark protocols (Table 1). The evidence-graph training signal is indeed produced by a teacher-verifier pipeline: "For each sample, Gemini 3.1 Pro (Google DeepMind 2026) first generates grounded evidence pages together with the corresponding evidence graph. GPT-5.5 (OpenAI 2026) then independently validates evidence sufficiency and logical consistency." The GRPO graph-faithfulness reward (R_graph = w_h R_hit + w_c R_comp + w_s R_struct + w_t R_topo) optimizes toward those reference teacher graphs, but this is standard supervised learning from an external teacher, not a self-referential definition: the teacher graphs are not derived from DocTrace's own outputs, and the accuracy gains are not computed against the teacher. The traceability results in Table 3 do not reduce to teacher labels either: "Evidence Grounding 99.5" checks that cited block IDs exist in the Stage 2 layout inventory (an internal consistency rule), while "Evidence Necessity 82.8" and "Evidence Specificity 9.6" come from counterfactual masking experiments on the trained model. The paper explicitly disclaims the strongest interpretation in Appendix D: "This evaluation measures behavioral dependence through input perturbation. It does not establish that the model internally follows the explicit evidence graph; rather, it shows that the cited evidence is functionally important for the model's prediction." This limitation statement prevents the traceability claim from being circular. There are no load-bearing self-citations, no imported uniqueness theorem, and no ansatz smuggled via citation. The unvalidated correctness of teacher-generated graphs is a real external-validity concern but is not circularity under the definitional/fitted-prediction criteria specified here.
Assumptions & free parameters
free parameters (5)
- beta (localization reward) =
2.0
- lambda (Stage 3 reward balance) =
0.3
- alpha (answer reward mix) =
0.5
- Graph reward weights w_h, w_c, w_s, w_t =
0.55, 0.20, 0.15, 0.10
- GRPO candidate retention band and resampling ratios =
1/8 to 7/8 success; resampling ratios unspecified
assumptions (5)
- domain assumption Teacher-verifier evidence graphs are correct ground truth for reasoning structure
- domain assumption PaddleOCR-VL-1.5 parsing inventory is complete and accurate at 1568px
- domain assumption Mixed-source baseline numbers are comparable
- domain assumption Token-accounting overflow model for the 512px-coarse / 1568px-fine design
- ad hoc to paper Four-scenario taxonomy covers answerability and reasoning complexity
invented entities (1)
-
Evidence graph (V, E) with data nodes, derived nodes, and dependency edges
independent evidence
Cite this review
Pith. "Pith review of DocTrace: Towards Traceable Long Document VQA via Hierarchical Evidence Graph Reasoning." pith.science (2026). https://pith.science/paper/SDWVX24Y
@misc{pith2026260803292,
author = {Pith},
title = {Pith review of: DocTrace: Towards Traceable Long Document VQA via Hierarchical Evidence Graph Reasoning},
year = {2026},
howpublished = {\url{https://pith.science/paper/SDWVX24Y}},
note = {Machine review of arXiv:2608.03292}
}
read the original abstract
Long Document Visual Question Answering (LongDocVQA) requires Multimodal Large Language Models (MLLMs) to locate, integrate, and reason over heterogeneous document elements distributed across multiple pages. Existing approaches, including end-to-end MLLMs, retrieval-augmented generation (RAG) pipelines, and document agents, often lack explicit mechanisms to represent and verify how grounded evidence is progressively composed during reasoning, limiting both answer accuracy and traceability. In this paper, we cast LongDocVQA as an explicit evidence graph reasoning problem rather than implicit answer prediction. To this end, we propose DocTrace, a hierarchical framework that progressively performs evidence localization, structured document parsing, and evidence graph reasoning to enable explicit evidence provenance. To effectively learn these capabilities, we develop a two-stage training framework: joint Supervised Fine-Tuning (SFT) first initializes evidence localization and graph reasoning abilities, followed by task-specific Group Relative Policy Optimization (GRPO) with dedicated rewards to further optimize these capabilities. Extensive experiments on MMLongBench-Doc, LongDocURL, and SlideVQA demonstrate that DocTrace consistently outperforms both existing open-source baselines and proprietary MLLMs. Compared with the Qwen3-VL-8B-Instruct backbone, DocTrace achieves absolute improvements of 14.4, 11.3, and 11.7 points on the three benchmarks, respectively. Beyond competitive performance, DocTrace constructs traceable evidence graphs with explicit node-level provenance, enabling transparent and verifiable reasoning for long document understanding.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[3]
Cui,C.;Sun,T.;Liang,S.;Gao,T.;Zhang,Z.;Liu,J.;Wang, X.; Zhou, C.; Liu, H.; Lin, M.; et al
M3docrag:Multi-modalretrievaliswhatyouneedfor multi-page multi-document understanding.arXiv preprint arXiv:2411.04952. Cui,C.;Sun,T.;Liang,S.;Gao,T.;Zhang,Z.;Liu,J.;Wang, X.; Zhou, C.; Liu, H.; Lin, M.; et al
-
[6]
Hu, A.; Xu, H.; Zhang, L.; Ye, J.; Yan, M.; Zhang, J.; Jin, Q.; Huang, F.; and Zhou, J
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948. Hu, A.; Xu, H.; Zhang, L.; Ye, J.; Yan, M.; Zhang, J.; Jin, Q.; Huang, F.; and Zhou, J
-
[7]
Ke,W.;Zheng,Y.;Li,Y.;Xu,H.;Nie,D.;Wang,P.;andHe, Y
Gpt-4o system card.arXiv preprint arXiv:2410.21276. Ke,W.;Zheng,Y.;Li,Y.;Xu,H.;Nie,D.;Wang,P.;andHe, Y
-
[8]
In Findings of the Association for Computational Linguistics: ACL 2026, 29770–29783
MM-Doc- R1: Training Agents for Long Document Visual Question Answering through Multi-turn Reinforcement Learning. In Findings of the Association for Computational Linguistics: ACL 2026, 29770–29783. Liu,Y.;Yang,B.;Liu,Q.;Li,Z.;Ma,Z.;Zhang,S.;andBai, X
work page 2026
-
[9]
Shi,Y.;Wang,J.;Shan,Z.;Peng,D.;Lin,Z.;andJin,L.2026
Closed-source model accessed via API; no dedi- cated arXiv technical report released. Shi,Y.;Wang,J.;Shan,Z.;Peng,D.;Lin,Z.;andJin,L.2026. URaG:UnifiedretrievalandgenerationinmultimodalLLMs for efficient long document understanding. InProceedings oftheAAAIConferenceonArtificialIntelligence,volume40, 25357–25365. Tanaka,R.;Iki,T.;Hasegawa,T.;Nishida,K.;Sai...
work page 2026
-
[10]
Tito, R.; Karatzas, D.; and Valveny, E
Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context.arXiv preprint arXiv:2403.05530. Tito, R.; Karatzas, D.; and Valveny, E
-
[11]
Vrag-rl: Empower vision-perception-basedragforvisuallyrichinformationun- derstandingviaiterativereasoningwithreinforcementlearn- ing.Advances in Neural Information Processing Systems, 38: 57133–57160. Wei,J.;Wang,X.;Schuurmans,D.;Bosma,M.;Xia,F.;Chi, E.;Le,Q.V.;Zhou,D.;etal.2022.Chain-of-thoughtprompt- ing elicits reasoning in large language models.Advanc...
work page 2022
-
[12]
InProceedings of the 2025 ConferenceonEmpiricalMethodsinNaturalLanguagePro- cessing, 14035–14056
Molorag: Bootstrapping document understanding via multi- modal logic-aware retrieval. InProceedings of the 2025 ConferenceonEmpiricalMethodsinNaturalLanguagePro- cessing, 14035–14056. Yan,H.;Liu,Y.;Liu,X.;Zhang,Y.;Liao,M.;Wu,J.;Chen, W.; and Bai, X
work page 2025
Show all 15 references
-
[13]
InInternational Conference on Learning Rep- resentations, volume 2025, 21074–21098
Visrag: Vision- based retrieval-augmented generation on multi-modality documents. InInternational Conference on Learning Rep- resentations, volume 2025, 21074–21098. Zheng, Y.; Fu, P.; Li, H.; Wang, Z.; Zhang, Y.; Ruan, W.; Zhang, X.; Wei, Z.; Luo, Z.; Luan, J.; et al
2025
-
[14]
InProceedingsofthe64thAnnualMeeting oftheAssociationforComputationalLinguistics(Volume1: Long Papers), 45901–45923
Doc-V*: Coarse-to-Fine Interactive Visual Reasoning for Multi-Page DocumentVQA. InProceedingsofthe64thAnnualMeeting oftheAssociationforComputationalLinguistics(Volume1: Long Papers), 45901–45923. Zhu, J.; Wang, W.; Chen, Z.; Liu, Z.; Ye, S.; Gu, L.; Tian, H.;Duan,Y.;Su,W.;Shao...
2025 arXiv
-
[15]
It improves ac- curacyfrom38.5to52.9(+14.4points),whilereducingend- to-end latency by2.1×(15.04s to 7.33s) and prefill tokens by2.3×(35.5K to 15.5K)
Table 10 shows that DocTrace is both more accurate and more efficient than the end-to-end baseline. It improves ac- curacyfrom38.5to52.9(+14.4points),whilereducingend- to-end latency by2.1×(15.04s to 7.33s) and prefill tokens by2.3×(35.5K to 15.5K). The efficiency gain comes f...
-
[1159]
Docopilot:Improving multimodal models for document-level understanding
Duan, Y.; Chen, Z.; Hu, Y.; Wang, W.; Ye, S.; Shi, B.; Lu, L.;Hou,Q.;Lu,T.;Li,H.;etal.2025. Docopilot:Improving multimodal models for document-level understanding. In ProceedingsoftheComputerVisionandPatternRecognition Conference, 4026–4037. Google DeepMind
2025
-
[2024]
Cho, J.; Mahata, D.; Irsoy, O.; He, Y.; and Bansal, M
SV-RAG: LoRA- contextualizing adaptation of MLLMs for long document understanding.arXiv preprint arXiv:2411.01106. Cho, J.; Mahata, D.; Irsoy, O.; He, Y.; and Bansal, M
-
[2025]
Chen,J.;Zhang,R.;Zhou,Y.;Yu,T.;Dernoncourt,F.;Gu,J.; Rossi, R
Qwen3-vl technical report.arXiv preprint arXiv:2511.21631. Chen,J.;Zhang,R.;Zhou,Y.;Yu,T.;Dernoncourt,F.;Gu,J.; Rossi, R. A.; Chen, C.; and Sun, T
-
[2026]
Deng, C.; Yuan, J.; Bu, P.; Wang, P.; Li, Z.-Z.; Xu, J.; Li, X.-H.;Gao,Y.;Song,J.;Zheng,B.;etal.2025
PaddleOCR-VL- 1.5:TowardsaMulti-Task0.9BVLMforRobustIn-the-Wild Document Parsing.arXiv preprint arXiv:2601.21957. Deng, C.; Yuan, J.; Bu, P.; Wang, P.; Li, Z.-Z.; Xu, J.; Li, X.-H.;Gao,Y.;Song,J.;Zheng,B.;etal.2025. Longdocurl: a comprehensive multimodal long document benchmar...
2025 arXiv
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.