{"id":"55d0a85d-e384-41a1-9467-790140d76d43","arxiv_id":"2608.00036","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"XL-DocBench is a 1,519-question, fully human-verified extra-long document QA benchmark across six professional domains; state-of-the-art systems still score at most 44% on it.","lead":"This paper introduces XL-DocBench, a human-verified question-answering benchmark for documents as long as 2,303 pages, with page-level evidence labels and deliberately unanswerable questions. It evaluates current LLM and retrieval systems and finds the best pipeline still only reaches 44% accuracy, with set-tracking and abstention as the hardest failures.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The unvalidated no-context filter is the load-bearing assumption: if world-knowledge-answerable questions leak into the 1,519-item set, the benchmark cannot deliver its diagnostic claim; the paper reports no accuracy or agreement for this largest rejection source.","rationale":"Reader's weakest_assumption identifies exactly the load-bearing concern. The benchmark's value proposition is diagnostic: scores should let the community attribute failures to retrieval, evidence use, or rule following, and should not be answerable from parametric knowledge. The no-context filter is the only explicit mechanism for that exclusion, and the paper gives no evidence of its validity. The absence is more serious than a missing robustness check because the filter is the largest rejection source and therefore has direct influence on the final 1,519 items. Human verification is not a substitute: it is conducted with full document access and its reported procedures (support, completeness, ambiguity) do not include a no-context answerability test; reliability is also not quantified via inter-annotator agreement. I therefore agree with the reader's assessment. I note the paper has independent strengths: a transparent tree-guided pipeline, deterministic evaluation code sketch, and rich diagnostic breakdowns; these make the benchmark plausible but do not close the specific gap. The proposed audit is feasible and would settle the concern: if leakage is negligible and does not change scores, the benchmark claim stands; if it is material, scores and slices must be corrected or the diagnostic claim narrowed. A secondary numeric inconsistency (72.6% vs 76.4% multi-page in Table 2) and lack of public release also warrant attention but are not the central load-bearing issue.","tokens_in":33127,"tokens_out":6128,"duration_ms":61514,"concrete_test":"Audit the final 1,519 questions out-of-context: present each question alone (no documents) to three strong LLMs and three independent human annotators, and score their answers with the paper's deterministic matcher (Listing 1). Measure (a) the fraction of items answered correctly from no context, (b) human–LLM agreement on 'answerable/not answerable', and (c) the accuracy gap on leaked vs. non-leaked items for at least SimpleDoc+GPT-5.4 and GPT-5.4 OCR. If the no-context answerable fraction is non-trivial (e.g., >2%) or the leaked-item accuracy gap exceeds chance, the reported 44.0% and diagnostic slices must be recomputed after excluding leaked questions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that XL-DocBench measures document understanding and lets failures be attributed to retrieval, evidence use, or rule following. That requires the final questions to be unanswerable without the documents. The only guard specifically aimed at this is the no-context filter (§3.1, Appendix A), which presents a candidate question to an LLM judge with no document content and rejects it if the answer is inferable from world knowledge, metadata, or wording leakage. The paper states this is the largest single rejection source but reports no accuracy, precision/recall, or judge agreement for it. Human verification (194 experts, §3.3) checks answer support, completeness, and ambiguity with full document access; nothing indicates it independently re-tests no-context answerability, and a verifier who sees the source can easily miss that a question is also answerable from parametric knowledge. Consequently, any false negatives in the no-context filter flow directly into the released benchmark. Evaluated systems—especially the frontier 1M-token models—can score on memorized world knowledge rather than on the document, and the reasoning-type/evidence-page breakdowns become confounded. This is a correctness risk, not a style disagreement: the benchmark's stated purpose is to attribute failures, which requires excluding non-document answer paths. The missing piece is an empirical validation of the no-context filter on the final set, plus an estimate of how much leaked items move the reported scores.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces XL-DocBench, a proposed benchmark for extra-long, evidence-grounded document understanding, built from 1,519 human-verified questions over professional documents up to about 2,000--2,300 pages across six domains, with twelve reasoning types, expert-annotated evidence pages, typed verification rules, and diagnostic subsets for multimodal, cross-document, and unanswerable questions. The construction pipeline is tree-guided model-assisted synthesis with automatic artifact filters followed by verification by 194 human experts. The paper reports evaluations of one-shot models and agent pipelines, finding that the best system reaches 44.0% overall accuracy and that set tracking and abstention are the main failure modes. The central claims are that the benchmark is fully human-verified and that its page-level evidence and reasoning labels allow system failures to be attributed to retrieval, evidence use, or rule following.","tokens_in":33468,"tokens_out":7287,"duration_ms":74459,"significance":"If the central claims hold, XL-DocBench would fill a real gap: existing document-understanding benchmarks are mostly single-page, short multi-page, or text-only, while professional workflows require combining evidence across hundreds of pages, multiple documents, tables/charts, and explicit abstention. The paper's diagnostic breakdowns by reasoning type, evidence span, modality, and document scope, together with deterministic scoring and detailed failure case studies, are valuable. The construction pipeline is described in unusual detail, and the authors are explicit that LLM outputs are provisional and that human verification is the final quality gate. These are genuine strengths. However, the benchmark's diagnostic value depends on the final questions being unanswerable without the documents, and the only guard aimed at that requirement -- the no-context filter -- is not empirically validated. The dataset is also not currently released, so the human-verified labels and the proposed diagnostic breakdowns cannot be independently checked.","major_comments":[{"comment":"The no-context filter is load-bearing for the central diagnostic claim, and it is unvalidated. The paper states it is the largest single rejection source and that it rejects candidates answerable from world knowledge, metadata, or wording leakage, but no judge model, threshold, or accuracy/precision/recall/agreement is reported. Human verification (§3.3) is performed with full document access and is not described as independently re-testing no-context answerability; a verifier who sees the source document can easily miss that a question is also answerable from parametric knowledge. Leaked questions would confound the reasoning-type and evidence-page breakdowns, because systems could score on world knowledge rather than document understanding. The manuscript needs an empirical no-context control on the final 1,519 examples (e.g., answer the question with no document content) and a validat","section":"§3.1, Appendix A; §3.3"},{"comment":"The dataset, code, and even a valid project URL are not available. The paper says 'Project homepage:/gl⌢be' (page 1), which is a malformed placeholder, and Appendix D says the dataset card 'will document' licensing and maintenance, confirming that the benchmark is not currently released. Since XL-DocBench is presented as a community benchmark and its 'fully human-verified' labels must be inspectable, this blocks both use and independent verification of the paper's central contribution. The revision should provide a stable release URL with the data, annotations, and evaluation code.","section":"§1 (project homepage); Appendix D"},{"comment":"The 'fully human-verified' claim is strong, but no reliability evidence is reported. There is no inter-annotator agreement, no description of how many experts independently annotated each item, no adjudication procedure, and no information about expert selection or domain expertise. For a benchmark whose value lies in diagnostic attribution, annotation reliability is a core property. I ask for IAA statistics on a double-annotated sample (e.g., evidence-page boundaries, answer correctness, reasoning labels) and a description of the verification workflow.","section":"§3.3"}],"minor_comments":[{"comment":"The headline numbers are inconsistent. The abstract and §3.2 report 1,103 multi-page evidence examples (72.6%), but Table 2 reports 'Multi-page human evidence. 1,103 (76.4%)'. Also, Table 2 lists max pages as 2,062 while the abstract and §3.2 cite contexts up to 2,303 pages. These should be reconciled with a clear definition of the denominator and of what counts as 'context length' for cross-document examples.","section":"Table 2 vs Abstract and §3.2"},{"comment":"Figure 5 reports n=1,519 while Figure 6 reports n=1,287 with no explanation for the difference. Please state the denominator for the evidence-hit analysis, and clarify how unanswerable examples are treated in the retrieval dynamics.","section":"Figures 5 and 6"},{"comment":"In the evaluation sketch, the 'none'/'unanswerable' branch is scored by contains_abstention_phrase(pred_n). This could credit a response that includes both a concrete answer and an abstention phrase. Specify how such responses are handled, and consider requiring that the response contains no substantive answer.","section":"Appendix H.2"},{"comment":"The limitations section correctly notes 'public-document exposure in model pretraining', but the paper does not run any contamination control on the final set. This is closely related to the no-context filter validation and should be addressed, e.g., by reporting question-only accuracy of the evaluated systems on a sample of the final questions.","section":"Appendix E"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid benchmark-construction effort with a clear gap to fill, and the evaluation and case studies are useful. The main risk is not the construction procedure itself but the unvalidated no-context filter combined with the absence of a released dataset: the central diagnostic claim cannot be checked until the authors show that the final questions are not answerable without the documents. If the requested controls and data release are provided, I would be happy to support acceptance; without them, the benchmark's value proposition is not yet established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort version: this is a serious benchmark paper that deserves referee time, but it isn't ready as-is. The central claim is 'fully human-verified' extra-long document QA, and the construction protocol is genuinely careful. The soft spots aren't about the synthesis pipeline—they're about the evidence you're given to check the verification claim.\n\nWhat's new and good: XL-DocBench targets a real gap: professional documents up to 2,303 pages, with 72.6% multi-page evidence, cross-document, multimodal, and None-answer slices, and a 12-type reasoning taxonomy. The tree-guided synthesis plus human annotation is a sensible separation of scalable generation from final evidence checks. The evaluation is thorough: 15+ systems, OCR vs image input, agent frameworks, and diagnostic breakdowns by length, evidence pages, and modality. The headline results (best 44% overall, set-tracking and abstention as binding constraints) are plausible, and the failure-case studies are illustrative.\n\nSoft spots, in proportion:\n\n1. The no-context filter is the load-bearing guard for the benchmark's diagnostic purpose, and it is unvalidated. It is the largest rejection source, uses an LLM judge with no document content, and no accuracy or agreement is reported. Human verifiers check support with full document access; nothing shows they independently re-test no-context answerability. So questions answerable from world knowledge can leak, and evaluated systems can score on memorized facts rather than the document. That isn't a fatal design flaw, but it's exactly the metric the paper needs to report.\n\n2. The 'fully human-verified' claim is missing inter-annotator agreement. 194 experts re-answer and annotate evidence; some reliability statistic is required to substantiate the label.\n\n3. Dataset and code are not released. The project page link is garbled, and the dataset card is promised but not present. Without the artifact, the human verification claim is uncheckable.\n\n4. Minor: Table 2 reports 76.4% multi-page human evidence, while the abstract and §3.2 say 72.6%. 1,103/1,519 = 72.6%, so Table 2 has a wrong denominator or typo.\n\nThe paper is not overclaiming in tone: it acknowledges LLM-style bias and public-document exposure in the limitations. The stress-test concern about the no-context filter holds up; it isn't manufactured. The rest is solid.\n\nWho benefits: long-document QA researchers, RAG/agent designers, benchmark builders. It deserves peer review—a serious referee could push for the missing validation and the release. I'd engage with it, and I'd cite it once the data is out.\n\nRecommendation: send to review with a request for revision.","headline":"A valuable benchmark with a carefully designed pipeline, but the unvalidated no-context filter and missing release/agreement metrics make the 'fully human-verified' claim unsupported.","tokens_in":33957,"tokens_out":2688,"would_cite":true,"duration_ms":32746,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"XL-DocBench is a fully human-verified benchmark for extra-long document understanding, and it shows current AI systems still struggle: the best pipeline reaches 44% accuracy on documents up to 2,303 pages.","keywords":["long-document understanding","benchmark","evidence grounding","retrieval-augmented generation","multimodal document QA","unanswerable questions","reasoning taxonomy","human verification"],"falsifier":"Take a random sample of, say, 200 XL-DocBench questions and ask a strong language model to answer them without any document access. If it answers significantly above the expected floor (or if a re-verification by a second independent team disagrees with the released labels on more than a small fraction), the benchmark's claim to measure document understanding rather than world-knowledge leakage would be falsified.","tokens_in":33010,"feed_emoji":"📄","tokens_out":3471,"duration_ms":31723,"temperature":0.7,"pith_summary":"This paper introduces XL-DocBench, a benchmark of 1,519 expert-verified questions over professional documents up to 2,303 pages, designed to test whether language models can find and use evidence scattered across hundreds of pages. The authors' central claim is that long-context capability is not the only bottleneck: even the strongest evaluated pipeline answers only 44.0% of questions correctly, and failures concentrate in set-tracking (ranking, coverage, set difference) and knowing when to abstain. The benchmark pairs each question with page-level evidence annotations, one of twelve reasoning labels, and a typed verification rule, so a system's mistakes can be attributed to retrieval, evidence use, or rule-following rather than a single score. The stated purpose is to give future long-document QA research a diagnostic instrument suited to compliance, clinical, financial, and engineering workflows where unsupported answers are costly.","feed_headline":"Strongest AI system scores 44% on 2,303-page documents","feed_subtitle":"New benchmark isolates whether AI fails at retrieval, evidence use, or rule-following on professional documents.","key_machinery":"The benchmark itself is the central object: a tree-guided synthesis pipeline that organizes each PDF into a hierarchy of sections and pages, uses a leave-one-branch-out test to ensure questions cannot be answered from a single chapter or section, then relies on 194 human experts to verify every retained question and annotate exact evidence pages and quotes. Each example carries a typed verification rule ('numeric tolerance,' 'choice exact match,' 'None-answer') and one of twelve reasoning labels, enabling failures to be attributed to specific reasoning steps rather than a single aggregate score.","core_discovery":"XL-DocBench establishes that current document-understanding systems, including 1M-token one-shot readers and retrieval-augmented agents, still fail on extra-long professional documents: the strongest pipeline reaches 44.0% overall accuracy, and no one-shot model exceeds 39.8%. The benchmark goes beyond page-level lookup: 72.6% of its 1,519 retained questions require multiple evidence pages, 36.6% require tables, charts, or figures, 10.9% require cross-document evidence, and 14.4% are unanswerable. Its diagnostic labels reveal that the binding constraints are not raw context length but the ability to track sets of evidence and to abstain when support is missing.","pith_inferences":["The benchmark's usefulness depends on the no-context filter being accurate; if leaked questions survive, scores could be inflated by world knowledge rather than document understanding. A second, independent no-context audit of a sample would test this directly.","The paper reports no inter-annotator agreement, so the reliability of the human verification step is unquantified; a reproducibility study with a second annotation team on a subset would strengthen the foundation.","The cross-document subset is small (165 questions) and the paper notes page-level statistics understate true cross-document scope; future work could expand this slice to test multi-document aggregation more robustly.","Because the benchmark is built on public professional documents, models pretrained on those documents may have an advantage; the diagnostic value could degrade over time as training data includes these sources."],"forward_implications":["If the benchmark's diagnostic claim holds, future work can attribute failures to retrieval, evidence use, or rule-following instead of treating a leaderboard score as the outcome.","Systems that cannot handle multi-page evidence, set-tracking, or abstention will be exposed even if they perform well on shorter single-page benchmarks.","The 44% ceiling suggests current long-context and agentic systems are not yet reliable for professional workflows where decisions must be traceable to specific pages.","Page-level evidence annotations and typed rules provide a template for building harder, more inspectable document-understanding benchmarks."],"fun_headline_variants":["Best AI scores 44% on 2,303-page documents","New benchmark: AI fails on 2,303-page docs","44% top score on long-doc benchmark","Benchmark exposes AI's long-document limits","Long-doc AI benchmark: best hits 44%"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The benchmark's diagnostic claim rests on the assumption that every retained question is genuinely unanswerable without the document and that the human-verified answers and evidence pages are correct—but no accuracy or agreement statistics are reported for the no-context filter or the human verifiers.","fun_headline_variants_meta":{"raw":{"variants":["Best AI scores 44% on 2,303-page documents","New benchmark: AI fails on 2,303-page docs","44% top score on long-doc benchmark","Benchmark exposes AI's long-document limits","Long-doc AI benchmark: best hits 44%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000882,"raw_usage":{"total_tokens":3704,"prompt_tokens":859,"completion_tokens":2845,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":2765}},"tokens_in":603,"tokens_out":2845,"duration_ms":57137,"temperature":1.0,"reasoning_tokens":2765,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T01:34:11.925682+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of, say, 200 XL-DocBench questions and ask a strong language model to answer them without any document access. If it answers significantly above the expected floor (or if a re-verification by a second independent team disagrees with the released labels on more than a small fraction), the benchmark's claim to measure document understanding rather than world-knowledge leakage would be falsified.","supporting_citations":[],"review_version":1}