Pith. sign in

REVIEW 3 major objections 4 minor 172 references

XL-DocBench is a fully human-verified benchmark for extra-long document understanding, and it shows current AI systems still struggle: the best pipeline reaches 44% accuracy on documents up to 2,303 pages.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 01:34 UTC pith:3WGXY3OA

load-bearing objection A valuable benchmark with a carefully designed pipeline, but the unvalidated no-context filter and missing release/agreement metrics make the 'fully human-verified' claim unsupported. the 3 major comments →

arxiv 2608.00036 v1 pith:3WGXY3OA submitted 2026-07-21 cs.CL cs.AI

XL-DocBench: Benchmarking Evidence-Grounded Extra-Long Document Understanding

Hongchen Wei , Yuanzhe Wang , Bei Liu , Yifan Yang , Qi Dai , Ruichun Ma , Kai Qiu , Yunsheng Li
show 4 more authors
Dongdong Chen Chong Luo Zhenzhong Chen Baining Guo
This is my paper
classification cs.CL cs.AI
keywords long-document understandingbenchmarkevidence groundingretrieval-augmented generationmultimodal document QAunanswerable questionsreasoning taxonomyhuman verification
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper introduces XL-DocBench, a benchmark of 1,519 expert-verified questions over professional documents up to 2,303 pages, designed to test whether language models can find and use evidence scattered across hundreds of pages. The authors' central claim is that long-context capability is not the only bottleneck: even the strongest evaluated pipeline answers only 44.0% of questions correctly, and failures concentrate in set-tracking (ranking, coverage, set difference) and knowing when to abstain. The benchmark pairs each question with page-level evidence annotations, one of twelve reasoning labels, and a typed verification rule, so a system's mistakes can be attributed to retrieval, evidence use, or rule-following rather than a single score. The stated purpose is to give future long-document QA research a diagnostic instrument suited to compliance, clinical, financial, and engineering workflows where unsupported answers are costly.

Core claim

XL-DocBench establishes that current document-understanding systems, including 1M-token one-shot readers and retrieval-augmented agents, still fail on extra-long professional documents: the strongest pipeline reaches 44.0% overall accuracy, and no one-shot model exceeds 39.8%. The benchmark goes beyond page-level lookup: 72.6% of its 1,519 retained questions require multiple evidence pages, 36.6% require tables, charts, or figures, 10.9% require cross-document evidence, and 14.4% are unanswerable. Its diagnostic labels reveal that the binding constraints are not raw context length but the ability to track sets of evidence and to abstain when support is missing.

What carries the argument

The benchmark itself is the central object: a tree-guided synthesis pipeline that organizes each PDF into a hierarchy of sections and pages, uses a leave-one-branch-out test to ensure questions cannot be answered from a single chapter or section, then relies on 194 human experts to verify every retained question and annotate exact evidence pages and quotes. Each example carries a typed verification rule ('numeric tolerance,' 'choice exact match,' 'None-answer') and one of twelve reasoning labels, enabling failures to be attributed to specific reasoning steps rather than a single aggregate score.

Load-bearing premise

The benchmark's diagnostic claim rests on the assumption that every retained question is genuinely unanswerable without the document and that the human-verified answers and evidence pages are correct—but no accuracy or agreement statistics are reported for the no-context filter or the human verifiers.

What would settle it

Take a random sample of, say, 200 XL-DocBench questions and ask a strong language model to answer them without any document access. If it answers significantly above the expected floor (or if a re-verification by a second independent team disagrees with the released labels on more than a small fraction), the benchmark's claim to measure document understanding rather than world-knowledge leakage would be falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the benchmark's diagnostic claim holds, future work can attribute failures to retrieval, evidence use, or rule-following instead of treating a leaderboard score as the outcome.
  • Systems that cannot handle multi-page evidence, set-tracking, or abstention will be exposed even if they perform well on shorter single-page benchmarks.
  • The 44% ceiling suggests current long-context and agentic systems are not yet reliable for professional workflows where decisions must be traceable to specific pages.
  • Page-level evidence annotations and typed rules provide a template for building harder, more inspectable document-understanding benchmarks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The benchmark's usefulness depends on the no-context filter being accurate; if leaked questions survive, scores could be inflated by world knowledge rather than document understanding. A second, independent no-context audit of a sample would test this directly.
  • The paper reports no inter-annotator agreement, so the reliability of the human verification step is unquantified; a reproducibility study with a second annotation team on a subset would strengthen the foundation.
  • The cross-document subset is small (165 questions) and the paper notes page-level statistics understate true cross-document scope; future work could expand this slice to test multi-document aggregation more robustly.
  • Because the benchmark is built on public professional documents, models pretrained on those documents may have an advantage; the diagnostic value could degrade over time as training data includes these sources.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces XL-DocBench, a proposed benchmark for extra-long, evidence-grounded document understanding, built from 1,519 human-verified questions over professional documents up to about 2,000--2,300 pages across six domains, with twelve reasoning types, expert-annotated evidence pages, typed verification rules, and diagnostic subsets for multimodal, cross-document, and unanswerable questions. The construction pipeline is tree-guided model-assisted synthesis with automatic artifact filters followed by verification by 194 human experts. The paper reports evaluations of one-shot models and agent pipelines, finding that the best system reaches 44.0% overall accuracy and that set tracking and abstention are the main failure modes. The central claims are that the benchmark is fully human-verified and that its page-level evidence and reasoning labels allow system failures to be attributed to retrieval, evidence use, or rule following.

Significance. If the central claims hold, XL-DocBench would fill a real gap: existing document-understanding benchmarks are mostly single-page, short multi-page, or text-only, while professional workflows require combining evidence across hundreds of pages, multiple documents, tables/charts, and explicit abstention. The paper's diagnostic breakdowns by reasoning type, evidence span, modality, and document scope, together with deterministic scoring and detailed failure case studies, are valuable. The construction pipeline is described in unusual detail, and the authors are explicit that LLM outputs are provisional and that human verification is the final quality gate. These are genuine strengths. However, the benchmark's diagnostic value depends on the final questions being unanswerable without the documents, and the only guard aimed at that requirement -- the no-context filter -- is not empirically validated. The dataset is also not currently released, so the human-verified labels and the proposed diagnostic breakdowns cannot be independently checked.

major comments (3)
  1. [§3.1, Appendix A; §3.3] The no-context filter is load-bearing for the central diagnostic claim, and it is unvalidated. The paper states it is the largest single rejection source and that it rejects candidates answerable from world knowledge, metadata, or wording leakage, but no judge model, threshold, or accuracy/precision/recall/agreement is reported. Human verification (§3.3) is performed with full document access and is not described as independently re-testing no-context answerability; a verifier who sees the source document can easily miss that a question is also answerable from parametric knowledge. Leaked questions would confound the reasoning-type and evidence-page breakdowns, because systems could score on world knowledge rather than document understanding. The manuscript needs an empirical no-context control on the final 1,519 examples (e.g., answer the question with no document content) and a validat
  2. [§1 (project homepage); Appendix D] The dataset, code, and even a valid project URL are not available. The paper says 'Project homepage:/gl⌢be' (page 1), which is a malformed placeholder, and Appendix D says the dataset card 'will document' licensing and maintenance, confirming that the benchmark is not currently released. Since XL-DocBench is presented as a community benchmark and its 'fully human-verified' labels must be inspectable, this blocks both use and independent verification of the paper's central contribution. The revision should provide a stable release URL with the data, annotations, and evaluation code.
  3. [§3.3] The 'fully human-verified' claim is strong, but no reliability evidence is reported. There is no inter-annotator agreement, no description of how many experts independently annotated each item, no adjudication procedure, and no information about expert selection or domain expertise. For a benchmark whose value lies in diagnostic attribution, annotation reliability is a core property. I ask for IAA statistics on a double-annotated sample (e.g., evidence-page boundaries, answer correctness, reasoning labels) and a description of the verification workflow.
minor comments (4)
  1. [Table 2 vs Abstract and §3.2] The headline numbers are inconsistent. The abstract and §3.2 report 1,103 multi-page evidence examples (72.6%), but Table 2 reports 'Multi-page human evidence. 1,103 (76.4%)'. Also, Table 2 lists max pages as 2,062 while the abstract and §3.2 cite contexts up to 2,303 pages. These should be reconciled with a clear definition of the denominator and of what counts as 'context length' for cross-document examples.
  2. [Figures 5 and 6] Figure 5 reports n=1,519 while Figure 6 reports n=1,287 with no explanation for the difference. Please state the denominator for the evidence-hit analysis, and clarify how unanswerable examples are treated in the retrieval dynamics.
  3. [Appendix H.2] In the evaluation sketch, the 'none'/'unanswerable' branch is scored by contains_abstention_phrase(pred_n). This could credit a response that includes both a concrete answer and an abstention phrase. Specify how such responses are handled, and consider requiring that the response contains no substantive answer.
  4. [Appendix E] The limitations section correctly notes 'public-document exposure in model pretraining', but the paper does not run any contamination control on the final set. This is closely related to the no-context filter validation and should be addressed, e.g., by reporting question-only accuracy of the evaluated systems on a sample of the final questions.

Circularity Check

0 steps flagged

No circular derivation: benchmark construction is externally grounded in human verification, and no evaluated quantity reduces to a fitted input.

full rationale

XL-DocBench is a benchmark-construction and evaluation paper, not a derivation of a modeled quantity from fitted parameters. The construction pipeline proposes candidates via LLMs and filters them, but the released labels—answers, evidence pages, reasoning types, and verification rules—are produced by 194 human experts with full document access (§3.3, Appendix I). No claim in the paper defines a predicted quantity in terms of the same data that were used to fit it. The leave-one-branch-out check (Eq. 1) is an automatic coarse filter, but the paper explicitly states that 'the released page-level evidence does not come from this coarse automatic stage. It is annotated and verified by 194 human experts with full document access' (Appendix A). There are no load-bearing self-citations: references such as PageIndex are external, and no uniqueness or ansatz is imported from prior work by the same authors. The unvalidated no-context filter (§3.1, Appendix A) is a genuine validity risk—if world-knowledge-answerable questions leak, the diagnostic attribution claim is weakened—but this is a correctness/contamination concern, not circularity, because the filter is not a fitted parameter being renamed as a prediction and the final set is externally checked by humans. The paper itself acknowledges limitations such as 'possible style bias from LLM-assisted candidate generation' (Appendix E), which is an honest limitation rather than a circular step. Accordingly, no specific circular step can be quoted, and the honest finding is no significant circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The paper adds no physical or theoretical entities. Its assumptions are benchmark-construction choices: the no-context LLM judge, the coarse branch-level path-dependency test, the reliability of human verification without agreement metrics, and the fidelity of PDF parsing. The main hand-set parameters are evaluation tolerances and the post-verification difficulty filter.

free parameters (4)
  • OCR context cap ratio = 0.80
    All OCR evaluations truncate per-page OCR text at 80% of the model context window to reserve room for generation; this cap directly shapes the usable-context comparison between 1M, 256K, and 128K models (§4.1).
  • Difficulty-oriented final retention threshold = 1,519 of 2,104 verified candidates
    After human verification, lower-difficulty items are removed to make the benchmark harder; no objective difficulty threshold is specified, so the reported 'current systems struggle' result is partly a function of this post-verification selection (§3.3, Appendix A).
  • Numeric answer tolerance = relative_error <= 0.05
    The deterministic evaluator accepts numeric answers within 5% relative error; this affects headline accuracy for numeric and percentage answers (Appendix H.2).
  • ANLS match threshold = norm_lev_sim >= 0.8
    String answers are counted correct when normalized Levenshtein similarity is at least 0.8; this is a hand-set evaluation threshold (Appendix H.2).
axioms (4)
  • domain assumption Leave-one-branch-out criterion at chapter/section level is a valid proxy for multi-page evidence requirement (Eq. 1).
    Used to filter candidates for path-dependency, but the paper states it does not prove the final evidence set is minimal and works only at coarse granularity (§3.1).
  • domain assumption Human expert annotation over up to 2,303-page documents is accurate without reported inter-annotator agreement.
    194 experts are said to fully verify every retained item; no reliability or agreement metrics are reported (§3.3).
  • ad hoc to paper LLM-based no-context judge reliably identifies questions answerable from world knowledge, metadata, or wording leakage.
    The largest rejection source is an LLM judge given only the question; judge accuracy is not validated (Appendix A).
  • domain assumption OCR/Markdown table/figure parsing of source PDFs preserves the evidence needed for answers.
    Parsing errors are listed as a known limitation; multimodal and table questions depend on this parse (Appendix E).

pith-pipeline@v1.3.0-alltime-deepseek · 32851 in / 11685 out tokens · 113960 ms · 2026-08-04T01:34:11.925682+00:00 · methodology

0 comments
read the original abstract

Real-world document tasks often ask professionals to answer questions from annual reports, regulations, clinical guidelines, and technical manuals that span hundreds or thousands of pages. Some questions also require comparing related reports. Reliable long-document understanding is therefore a prerequisite for using LLMs in compliance, clinical, financial, and engineering workflows, where decisions must be traceable to specific evidence pages and the cost of an unsupported answer is high -- yet most existing benchmarks still measure short-context or single-page QA. We introduce XL-DocBench, a fully human-verified benchmark for extra-long document understanding, with 1,519 retained questions from six professional domains and contexts up to 2,303 pages. XL-DocBench goes beyond page-level lookup. 1,103 examples (72.6\%) use multiple evidence pages. The final set also includes 556 questions (36.6\%) that use tables, charts, or figures, and 165 questions (10.9\%) that require evidence from multiple documents. Each question has one of twelve reasoning labels, expert-annotated evidence pages, a typed verification rule, and an answer format, including 218 None-answer cases. We build the benchmark with a tree-guided synthesis pipeline followed by artifact filters and full verification by 194 human experts. By coupling extra-long professional contexts with page-level evidence and typed rules, XL-DocBench fills a gap left by prior single-page, short multi-page, or text-only long-context benchmarks, and lets future work attribute system failures to retrieval, evidence use, or rule following rather than to a single leaderboard score. The results show that current systems still struggle with long contexts, multi-page evidence, and structured reasoning over professional documents.

Figures

Figures reproduced from arXiv: 2608.00036 by Baining Guo, Bei Liu, Chong Luo, Dongdong Chen, Hongchen Wei, Kai Qiu, Qi Dai, Ruichun Ma, Yifan Yang, Yuanzhe Wang, Yunsheng Li, Zhenzhong Chen.

Figure 1
Figure 1. Figure 1: XL-DocBench construction pipeline. Long documents and document series are parsed into hierarchical trees. The pipeline generates candidate questions from multiple chapter/section branches, refines them with concrete content, adds candidate answers and verification rules, and filters artifacts. Full verification by 194 human experts identifies the final supporting pages and quotes, checks answer support and… view at source ↗
Figure 2
Figure 2. Figure 2: Reasoning labels and evidence structure in the human-verified set. (a) Distribution of the twelve reasoning types, colored by tier. (b) Mean number of expert-annotated evidence pages and related PDFs per reasoning type. 50-100 100-200 200-500 500-1000 >=1000 0 100 200 300 400 500 # questions 226 489 496 257 51 (a) Context length 1 2-5 6-10 11-20 21-50 51-100>=101 0 50 100 150 200 250 300 350 # questions 34… view at source ↗
Figure 3
Figure 3. Figure 3: Composition of XL-DocBench in the final human-verified set: (a) context length per [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Diagnostic performance breakdowns for all evaluated systems. Columns separate full-image input, OCR input, and agent pipelines; rows report accuracy by context length, number of expert-annotated evidence pages, and evidence span. 26.9→37.3). The two 1M-token models nonetheless differ by 2.5 pts at equal usable OCR budget (Claude Opus 4.6 39.8% vs. GPT-5.4 37.3%), and 256K Kimi-K2.5 (35.8%) remains close wh… view at source ↗
Figure 5
Figure 5. Figure 5: Agent answer yield versus tool-call rounds. Accuracy as the retrieval rounds increases. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Agent evidence-hit dynamics. Evidence hit rate as the retrieved/read page rounds increases. is balanced (43.03% vs. 44.09%), and Claude Opus 4.6 OCR is the only frontier one-shot reader with a small cross-document penalty (−2.53 vs. GPT-5.4 OCR’s +6.40), suggesting its strength lies in deep single-document reading. Large negative gaps appear instead in unstable settings (DeepRead+DeepSeek-V3.2 −14.30, GPT-… view at source ↗
Figure 7
Figure 7. Figure 7: Failure case study 1. The example illustrates a system failure on an evidence-grounded [PITH_FULL_IMAGE:figures/full_fig_p016_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Failure case study 2. The final response is inconsistent with the expert-verified evidence. [PITH_FULL_IMAGE:figures/full_fig_p016_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Case study: Cross-document temporal. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Case study: Cross-document reference chain. [PITH_FULL_IMAGE:figures/full_fig_p020_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Case study: Comparison. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Case study: Reference chain. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Case study: Ranking. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Case study: Coverage. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Case study: Reconciliation. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Case study: Set difference. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Case study: Unanswerable. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Case study: Temporal. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Case study: Compliance. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Case study: Counterfactual. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Case study: Aggregation. 31 [PITH_FULL_IMAGE:figures/full_fig_p031_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: Case study: Consistency. 32 [PITH_FULL_IMAGE:figures/full_fig_p032_22.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

172 extracted references · 10 linked inside Pith

  1. [1]

    Pixtral 12b.arXiv preprint arXiv:2410.07073, 2024

    Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, et al. Pixtral 12b.arXiv preprint arXiv:2410.07073, 2024

    Pith/arXiv arXiv 2024

  2. [2]

    Introducing claude opus 4.6

    Anthropic. Introducing claude opus 4.6. https://www.anthropic.com/news/ claude-opus-4-6, 2026

    2026

  3. [3]

    Docformer: End-to-end transformer for document understanding

    Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R Manmatha. Docformer: End-to-end transformer for document understanding. InICCV, pages 993–1003, 2021

    2021

  4. [4]

    Longbench: A bilingual, multitask benchmark for long context understanding

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. InACL, pages 3119–3137, 2024

    2024

  5. [5]

    M-longdoc: A benchmark for multimodal super-long document understanding and A retrieval-aware tuning framework

    Yew Ken Chia, Liying Cheng, Hou Pong Chan, Maojia Song, Chaoqun Liu, Mahani Aljunied, Soujanya Poria, and Lidong Bing. M-longdoc: A benchmark for multimodal super-long document understanding and A retrieval-aware tuning framework. InEMNLP, pages 9233– 9250, 2025

    2025

  6. [6]

    M3docrag: Multi- modal retrieval is what you need for multi-page multi-document understanding.arXiv preprint arXiv:2411.04952, 2024

    Jaemin Cho, Debanjan Mahata, Ozan Irsoy, Yujie He, and Mohit Bansal. M3docrag: Multi- modal retrieval is what you need for multi-page multi-document understanding.arXiv preprint arXiv:2411.04952, 2024

    Pith/arXiv arXiv 2024

  7. [7]

    Longdocurl: a comprehensive multimodal long document benchmark integrating understanding, reasoning, and locating

    Chao Deng, Jiale Yuan, Pi Bu, Peijie Wang, Zhong-Zhi Li, Jian Xu, Xiao-Hui Li, Yuan Gao, Jun Song, Bo Zheng, et al. Longdocurl: a comprehensive multimodal long document benchmark integrating understanding, reasoning, and locating. InACL, pages 1135–1159, 2025

    2025

  8. [8]

    A survey on mllm-based visually rich document understanding: Methods, challenges, and emerging trends.arXiv preprint arXiv:2507.09861, 2025

    Yihao Ding, Siwen Luo, Yue Dai, Yanbei Jiang, Zechuan Li, Geoffrey Martin, and Yifan Peng. A survey on mllm-based visually rich document understanding: Methods, challenges, and emerging trends.arXiv preprint arXiv:2507.09861, 2025

    Pith/arXiv arXiv 2025

  9. [9]

    Colpali: Efficient document retrieval with vision language models.arXiv preprint arXiv:2407.01449, 2024

    Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Céline Hudelot, and Pierre Colombo. Colpali: Efficient document retrieval with vision language models.arXiv preprint arXiv:2407.01449, 2024

    Pith/arXiv arXiv 2024

  10. [10]

    Gemini-3.https://aistudio.google.com/models/gemini-3/, 2026

    Google. Gemini-3.https://aistudio.google.com/models/gemini-3/, 2026

    2026

  11. [11]

    Mdoca- gent: A multi-modal multi-agent framework for document understanding.arXiv preprint arXiv:2503.13964, 2025

    Siwei Han, Peng Xia, Ruiyi Zhang, Tong Sun, Yun Li, Hongtu Zhu, and Huaxiu Yao. Mdoca- gent: A multi-modal multi-agent framework for document understanding.arXiv preprint arXiv:2503.13964, 2025

    Pith/arXiv arXiv 2025

  12. [12]

    mplug-docowl2: High-resolution compressing for ocr-free multi-page document understanding

    Anwen Hu, Haiyang Xu, Liang Zhang, Jiabo Ye, Ming Yan, Ji Zhang, Qin Jin, Fei Huang, and Jingren Zhou. mplug-docowl2: High-resolution compressing for ocr-free multi-page document understanding. InACL, pages 5817–5834, 2025

    2025

  13. [13]

    Layoutlmv3: Pre-training for document ai with unified text and image masking

    Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. Layoutlmv3: Pre-training for document ai with unified text and image masking. InACM MM, pages 4083–4091, 2022. 10

    2022

  14. [14]

    Simpledoc: Multi-modal document understanding with dual-cue page retrieval and iterative refinement

    Chelsi Jain, Yiran Wu, Yifan Zeng, Jiale Liu, Shengyu Dai, Zhenwen Shao, Qingyun Wu, and Huazheng Wang. Simpledoc: Multi-modal document understanding with dual-cue page retrieval and iterative refinement. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 28398–28415, 2025

    2025

  15. [15]

    Donut: Document understanding trans- former without ocr.arXiv preprint arXiv:2111.15664, 7(15):2, 2021

    Geewook Kim, Teakgyu Hong, Moonbin Yim, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Donut: Document understanding trans- former without ocr.arXiv preprint arXiv:2111.15664, 7(15):2, 2021

    Pith/arXiv arXiv 2021

  16. [16]

    Deepread: Document structure-aware reasoning to enhance agentic search.arXiv preprint arXiv:2602.05014, 2026

    Zhanli Li, Huiwen Tian, Lvzhou Luo, Yixuan Cao, and Ping Luo. Deepread: Document structure-aware reasoning to enhance agentic search.arXiv preprint arXiv:2602.05014, 2026

    arXiv 2026

  17. [17]

    Visualwebbench: How far have multimodal llms evolved in web page understanding and grounding?arXiv preprint arXiv:2404.05955, 2024

    Junpeng Liu, Yifan Song, Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li, and Xiang Yue. Visualwebbench: How far have multimodal llms evolved in web page understanding and grounding?arXiv preprint arXiv:2404.05955, 2024

    Pith/arXiv arXiv 2024

  18. [18]

    Textmon- key: An ocr-free large multimodal model for understanding document.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026

    Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. Textmon- key: An ocr-free large multimodal model for understanding document.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026

    2026

  19. [19]

    Towards storage-efficient visual document retrieval: An empirical study on reducing patch-level embeddings

    Yubo Ma, Jinsong Li, Yuhang Zang, Xiaobao Wu, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Haodong Duan, Jiaqi Wang, Yixin Cao, et al. Towards storage-efficient visual document retrieval: An empirical study on reducing patch-level embeddings. InACL, pages 19568–19580, 2025

    2025

  20. [20]

    Mmlongbench-doc: Benchmarking long-context document understanding with visualizations.NeurIPS, 37:95963–96010, 2024

    Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, et al. Mmlongbench-doc: Benchmarking long-context document understanding with visualizations.NeurIPS, 37:95963–96010, 2024

    2024

  21. [21]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. InACL, pages 2263–2279, 2022

    2022

  22. [22]

    Infographicvqa

    Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawa- har. Infographicvqa. InWACV, pages 1697–1706, 2022

    2022

  23. [23]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. InCVPR, pages 2200–2209, 2021

    2021

  24. [24]

    Qwen3.5: Towards native multimodal agents

    Qwen Team. Qwen3.5: Towards native multimodal agents. https://qwen.ai/blog?id= qwen3.5, 2026

    2026

  25. [25]

    Scrolls: Standardized comparison over long language sequences

    Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, et al. Scrolls: Standardized comparison over long language sequences. InEMNLP, pages 12007–12021, 2022

    2022

  26. [26]

    Vdocrag: Retrieval-augmented generation over visually-rich documents

    Ryota Tanaka, Taichi Iki, Taku Hasegawa, Kyosuke Nishida, Kuniko Saito, and Jun Suzuki. Vdocrag: Retrieval-augmented generation over visually-rich documents. InCVPR, pages 24827–24837, 2025

    2025

  27. [27]

    Slidevqa: A dataset for document visual question answering on multiple images

    Ryota Tanaka, Kyosuke Nishida, Kosuke Nishida, Taku Hasegawa, Itsumi Saito, and Kuniko Saito. Slidevqa: A dataset for document visual question answering on multiple images. In AAAI, volume 37, pages 13636–13645, 2023

    2023

  28. [28]

    Hierarchical multimodal transformers for multipage docvqa.Pattern Recognition, 144:109834, 2023

    Rubèn Tito, Dimosthenis Karatzas, and Ernest Valveny. Hierarchical multimodal transformers for multipage docvqa.Pattern Recognition, 144:109834, 2023

    2023

  29. [29]

    Document understanding dataset and evaluation (dude)

    Jordy Van Landeghem, Rubèn Tito, Łukasz Borchmann, Michał Pietruszka, Pawel Joziak, Rafal Powalski, Dawid Jurkiewicz, Mickaël Coustaty, Bertrand Anckaert, Ernest Valveny, et al. Document understanding dataset and evaluation (dude). InICCV, pages 19528–19540, 2023

    2023

  30. [30]

    Lilt: A simple yet effective language-independent layout transformer for structured document understanding

    Jiapeng Wang, Lianwen Jin, and Kai Ding. Lilt: A simple yet effective language-independent layout transformer for structured document understanding. InACL, pages 7747–7757, 2022. 11

    2022

  31. [31]

    Internvl3

    Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency.arXiv preprint arXiv:2508.18265, 2025

    Pith/arXiv arXiv 2025

  32. [32]

    Docgenome: An open large-scale scientific document benchmark for training and testing multi-modal large language models.arXiv preprint arXiv:2406.11633, 2024

    Renqiu Xia, Song Mao, Xiangchao Yan, Hongbin Zhou, Bo Zhang, Haoyang Peng, Jiahao Pi, Daocheng Fu, Wenjie Wu, Hancheng Ye, et al. Docgenome: An open large-scale scientific document benchmark for training and testing multi-modal large language models.arXiv preprint arXiv:2406.11633, 2024

    arXiv 2024

  33. [33]

    Layoutlmv2: Multi-modal pre-training for visually- rich document understanding

    Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, et al. Layoutlmv2: Multi-modal pre-training for visually- rich document understanding. InIJCNLP, pages 2579–2591, 2021

    2021

  34. [34]

    Layoutlm: Pre-training of text and layout for document image understanding

    Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. Layoutlm: Pre-training of text and layout for document image understanding. InKDD, pages 1192–1200, 2020

    2020

  35. [35]

    mplug-docowl: Modularized multimodal large language model for document understanding.arXiv preprint arXiv:2307.02499, 2023

    Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Yuhao Dan, Chenlin Zhao, Guohai Xu, Chenliang Li, Junfeng Tian, et al. mplug-docowl: Modularized multimodal large language model for document understanding.arXiv preprint arXiv:2307.02499, 2023

    Pith/arXiv arXiv 2023

  36. [36]

    Visrag: Vision-based retrieval-augmented generation on multi-modality documents.arXiv preprint arXiv:2410.10594, 2024

    Shi Yu, Chaoyue Tang, Bokai Xu, Junbo Cui, Junhao Ran, Yukun Yan, Zhenghao Liu, Shuo Wang, Xu Han, Zhiyuan Liu, et al. Visrag: Vision-based retrieval-augmented generation on multi-modality documents.arXiv preprint arXiv:2410.10594, 2024

    Pith/arXiv arXiv 2024

  37. [37]

    Finragbench-v: A benchmark for multi- modal rag with visual citation in the financial domain

    Suifeng Zhao, Zhuoran Jin, Sujian Li, and Jun Gao. Finragbench-v: A benchmark for multi- modal rag with visual citation in the financial domain. InEMNLP, pages 4215–4249, 2025

    2025

  38. [38]

    Tat-qa: A question answering benchmark on a hybrid of tabular and textual content in finance

    Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. Tat-qa: A question answering benchmark on a hybrid of tabular and textual content in finance. InIJNLP, pages 3277–3287, 2021

    2021

  39. [39]

    Mmdocbench: Benchmarking large vision-language models for fine-grained visual document understanding and grounding

    Fengbin Zhu, Ziyang Liu, Xiang Yao Ng, Haohui Wu, Wenjie Wang, Fuli Feng, Chao Wang, Huanbo Luan, and Tat-Seng Chua. Mmdocbench: Benchmarking large vision-language models for fine-grained visual document understanding and grounding. InMMM, pages 74–88, 2026

    2026

  40. [40]

    framework

    Anni Zou, Wenhao Yu, Hongming Zhang, Kaixin Ma, Deng Cai, Zhuosheng Zhang, Hai Zhao, and Dong Yu. Docbench: A benchmark for evaluating llm-based document reading systems. In Proceedings of the 4th International Workshop on Knowledge-Augmented Methods for Natural Language Processing, pages 359–373, 2025. 12 Table 4: Accuracy by document domain for all eval...

    2025

  41. [41]

    Context of the Model Law 1st branch

  42. [42]

    Main features of the Model Law 2nd branch

  43. [43]

    Implementation and use of the Model Law3rd branch — model stopped here ↓ descend one level — pick the 1st child

  44. [44]

    framework

    The Model Law as a "framework" law…gold answer

  45. [45]

    Banking activities

    Regulations and other laws… on p.38, parent and child sit on adjacent lines→ models read the first (parent) heading and stop, instead of descending to the child. p1 p2 p23 p24 p25 p32 p33 p38 p39 p42 p45 p51 p418 p419 Figure 7: Failure case study 1. The example illustrates a system failure on an evidence-grounded long-document question. The model response...

  46. [47]

    Texts and abbreviations

  47. [48]

    Chronology of Jane Austen's life

  48. [49]

    The professional woman writer

  49. [50]

    Northanger Abbey, Sense and Sensibility, P […] 11 more D2 314P

  50. [52]

    A note on the second edition

  51. [53]

    Chronology of Joyce’s life

  52. [54]

    List of abbreviations

  53. [55]

    Joyce the Irishman […] 13 more D3 332P

  54. [57]

    Note on translations

  55. [58]

    Dante and the lyric past

  56. [59]

    Approaching the Vita nuova […] 17 more D4 304P

  57. [60]

    Notes on contributors

  58. [61]

    Introduction: perspectives on the African

  59. [62]

    The oral–literate interface

  60. [63]

    Chinua Achebe and the African novel

  61. [64]

    Cambridge University Press 1997

    Protest and resistance […] 13 more HUMAN-EXPERTHUMAN-EXPERTANNOTATIONANNOTATIONverbatim evidence quotes — grouped by document, manually curated & verified ✓ VERIFIED CITED PAGES FROM SOURCE PDFS · HUMAN-CURATED, GROUPED BY DOCUMENT D1 D2 D3 D4 2 6 6 P p1 p2 p3 p4 p5 3 1 4 P p1 p12 p13 3 3 2 P p1 p2 p5 p7 3 0 4 P p1 p2 p3 p4 p5 STAGE 1 · EXPLORATION · CROS...

    1997

  62. [65]

    Basel III: Finalising post-crisis reforms

  63. [66]

    Standardised approach for credit risk

  64. [67]

    Internal ratings-based approach for credit

  65. [68]

    Minimum capital requirements for CVA risk

  66. [69]

    Minimum capital requirements for operation

  67. [70]

    Output floor […] 1 more D2 82P

  68. [72]

    Pillar 3 disclosure requirements – consoli

  69. [73]

    Part 1: New and revised Pillar 3 disclosur

  70. [74]

    Consolidation of all existing and prospect

  71. [75]

    Two enhancements to the revised Pillar 3 f

  72. [76]

    Revisions and additions to the Pillar 3 fr […] 15 more D3 1982P

  73. [77]

    SCO Scope and definitions

  74. [78]

    CAP Definition of capital

  75. [79]

    RBC Risk-based capital requirements

  76. [80]

    CRE Calculation of RWA for credit risk

  77. [81]

    MAR Calculation of RWA for market risk

  78. [82]

    OPE Calculation of RWA for operational ris

  79. [83]

    LEV Leverage ratio […] 7 more D4 77P

  80. [84]

    Basel III: A global regulatory framework f

Showing first 80 references.