{"id":"fa899e82-59eb-4d97-b2fa-86870b4dbf0a","arxiv_id":"2607.19301","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"PAGE-RAG combines an always-on textual retrieval floor with a graph 'skeleton', routes queries adaptively, and explicitly abstains when evidence is insufficient, yielding competitive accuracy with reliable refusal.","lead":"The paper introduces PAGE-RAG, a system for answering questions about long documents that keeps both the original text and a knowledge graph, using the graph as a guide rather than a replacement. It answers only when evidence is sufficient, otherwise abstaining, and tests on two book-length datasets and a public benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evidence sufficiency check s(q,E) is the load-bearing answer-or-refuse mechanism, yet the paper gives no implementation, model, prompt, or calibration; the central reliability claim is unverifiable until s is specified and tested.","rationale":"The reader's weakest_assumption—that the evidence sufficiency check s(q,E) is unspecified—is precisely the most load-bearing concern. The paper's central claim is not merely competitive accuracy but reliable knowledge-boundary control, and that control is implemented entirely by s. Without knowing whether s is a prompted LLM judgment, a trained classifier, or a heuristic, the 12/12 refusal results cannot be independently assessed or compared against prior abstention methods. The paper itself uses the 'No Evidence Constraint' ablation to argue that the evidence constraint is the direct mechanism for the knowledge boundary, which makes the lack of specification even more conspicuous: the mechanism is central, not peripheral. I do not see a more fundamental internal flaw: the routing and graph construction are described with enough structure to be plausible, the ablations support the role of the textual floor, and the code URL is a credible avenue for verification. The concern is addressable by disclosure and reproduction, so the appropriate outcome remains the reader's CONDITIONAL verdict rather than outright rejection or acceptance. I agree fully with the reader's identification of the weakest assumption.","tokens_in":12459,"tokens_out":2329,"duration_ms":28255,"concrete_test":"Inspect the public repository (github.com/CXY0112/PAGE-RAG) for the implementation of s(q,E). If it exists, re-run it on all 24 unanswerable and 182 answerable questions across both books and report: (i) agreement with gold labels; (ii) the exact implementation (e.g., prompted LLM, trained classifier, retrieval-score threshold); (iii) a replacement test where s is substituted by a simple baseline such as maximum retrieval score or LLM self-reported confidence—if the baseline achieves comparable correct refusal, the distinct contribution of the evidence-bounded mechanism weakens. If no implementation is found, request a precise specification from the authors and reproduce the decision boundary before accepting the refusal numbers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4 defines generation as y=a iff s(q,E)=1, else refusal. All 12/12 correct-refusal results on each book, and the qualitative claim of reliable abstention, rest entirely on this binary check. The paper never states how s is computed: no classifier, no prompt, no thresholds, no calibration data, no error analysis. The 'No Evidence Constraint' ablation isolates s as the direct cause of the refusal improvement (24/24 vs 4/24), confirming that the mechanism is doing the boundary work, but it does not establish that s is a reliable, generalizable evidence-sufficiency judgment rather than a model-specific artifact. The formal notation s(q,E)∈{0,1} hides the actual decision procedure, so the contribution 'evidence-bounded generation' cannot be reproduced or audited from the manuscript. This is an internal reproducibility gap, not a disagreement with consensus, and it is load-bearing because even perfect routing and graph construction cannot deliver the claimed answer-or-refuse behavior without a working s.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PAGE-RAG, a GraphRAG framework for long-document question answering that combines a textual passage index with a graph-based 'semantic skeleton,' routes queries adaptively among retrieval operators, and makes the answer-versus-refuse decision depend on an explicit evidence-sufficiency check. Evaluated on two book-length datasets and the UltraDomain-Mix benchmark, the system is reported to achieve competitive answer quality while using far fewer query tokens than a community-summary GraphRAG baseline, and to abstain correctly on all 24 unanswerable book questions. Ablations attribute the refusal behavior to the evidence constraint and the answer quality to the textual retrieval floor.","tokens_in":12712,"tokens_out":3245,"duration_ms":33770,"significance":"The paper addresses a real and under-emphasized problem: automatically constructed graphs are lossy projections of source documents, and GraphRAG systems should be able to abstain when evidence is insufficient. The authors are appropriately honest that PAGE-RAG is a balanced operating point rather than a universal improvement, and the positive results are accompanied by code release and ablations that support the role of the textual floor and the evidence constraint. If the evidence-sufficiency mechanism can be specified and the datasets made auditable, the contribution would be useful to the GraphRAG community. As it stands, the central reliability claim is not yet reproducible from the manuscript.","major_comments":[{"comment":"The mapping y = a if s(q,E)=1 else refusal is the load-bearing mechanism for the paper's central reliability claim, but the manuscript never states how s(q,E) is computed. There is no classifier, no prompt, no threshold, no calibration data, and no error analysis. The 12/12 correct refusals on each book and the contrast in Table 3 between 24/24 and 4/24 refusals depend entirely on this unspecified binary check. Without a precise specification and an evaluation of s on a range of sufficient and insufficient evidence sets, the 'evidence-bounded generation' contribution cannot be reproduced, audited, or distinguished from a model-specific artifact. This is an internal reproducibility gap, not a matter of taste. Please specify the decision procedure and provide calibration/error analysis.","section":"Evidence-Bounded Generation (Section 4)"},{"comment":"The two book-length datasets (Simulacra and Simulation, One Hundred Years of Solitude) appear to be self-constructed, but no construction protocol is described: how were the answerable and unanswerable questions written, what defined 'unanswerable' for the annotators, how was the gold answer determined, and was there any inter-annotator agreement? The claim of 'correctly refuses all 24 unanswerable questions' rests on 24 labeled instances whose provenance and labels are not auditable. The authors should release the questions and evidence annotations, describe the annotation instructions, and ideally report confidence intervals for the refusal rate and accuracy metrics.","section":"Experimental Setup: Datasets"},{"comment":"The routing policy Π(u,O) and the evidence budgets B=(Bt,Bn,Bp,Bc) are central to the efficiency claims, but the paper does not specify how they are set: are they hand-tuned, prompt-based, or learned? No sensitivity analysis is reported. The dramatic token reduction in Table 2 (1,956 vs 136,509 query tokens per question against Microsoft GraphRAG) could be an artifact of an aggressive budget choice rather than a property of the routing framework. Please report the concrete values used for the budgets and thresholds, and include a sensitivity study showing how BBS and token cost vary when these budgets are changed.","section":"Query-Adaptive Retrieval Routing (Section 2) and Table 2"}],"minor_comments":[{"comment":"'Strict' and 'lenient' accuracy are used in Table 1 but never defined. Please define both in Section 5.","section":"Evaluation metrics"},{"comment":"The caption mentions a 'PAGE-RAG-anchored four-state quality' index without defining it. The four states and how the pairwise judgments are converted into this index should be stated.","section":"Figure 2"},{"comment":"Table 1 repeats the dataset name in every row; using row spans or grouping would improve readability. Consider also adding the number of questions in the table caption.","section":"Table 1 formatting"},{"comment":"Several references are to arXiv preprints with 2026 dates that may not be stable or peer reviewed. Consider citing published versions where available.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The technical direction is promising and the paper is honestly positioned, but the unspecified s(q,E) is a fundamental reproducibility gap that must be closed before the central claim can be assessed. If the authors can provide the implementation, calibration data, and dataset release, the paper is salvageable. The self-citation [2] appears peripheral and does not by itself raise concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nPAGE-RAG is worth your time if you work on GraphRAG or trustworthy QA. It is an honest, well-scoped systems paper that treats graphs as lossy projections of source text, keeps an always-on passage retrieval floor, routes queries adaptively, and makes abstention an explicit evidence-boundary decision. The empirical story is calibrated: it does not claim universal dominance; it adds a balanced operating point with competitive quality, lower latency/token cost than MS GraphRAG, and correct refusal on all 24 unanswerable book questions. The ablations are the paper's strongest feature — removing the evidence constraint drops correct refusal from 24/24 to 4/24, so the boundary behavior is genuinely tied to the proposed mechanism, not to prompt fluff. Public code is a plus.\n\nThe soft spots are real. The load-bearing piece is s(q,E), the binary evidence-sufficiency check. The paper defines it as a function, but never says how it is computed: no classifier, no prompt, no thresholds, no calibration data, no error analysis. Since generation is answer iff s(q,E)=1, the 12/12 refusal results stand or fall on this black box. That is an internal reproducibility gap, not a disagreement with a consensus. The code might contain the implementation, but the manuscript should have described and validated it.\n\nSecond, the related work ignores the substantial abstention/selective-prediction literature, and there is no comparison against a simple 'refuse if the model is uncertain' baseline. The reported gap between 24/24 and 4/24 would be more convincing if you knew what a prompt-only refusal baseline achieves. Third, the two book datasets appear self-constructed with no annotation protocol or inter-annotator agreement, and the tables have no error bars or significance tests. These are addressable rather than fatal.\n\nNet: this is a solid, honest submission that deserves serious peer review. I'd pressure the authors to disclose s(q,E), release the QA datasets and annotation instructions, add variance estimates, and include an abstention baseline. Fixing those changes the reliability verdict from conditional to comfortably positive.","headline":"Honest, well-scoped GraphRAG paper with a genuinely useful abstention mechanism, but the central answer-or-refuse decision rests on an unspecified evidence-sufficiency check that must be disclosed before the reliability claim is fully credible.","tokens_in":13204,"tokens_out":2705,"would_cite":true,"duration_ms":27069,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PAGE-RAG treats graphs as lossy projections, not knowledge replacements, and gates answers on an explicit evidence-sufficiency check — yielding ~90% lenient accuracy with perfect refusal on all 24 unanswerable questions.","keywords":["GraphRAG","retrieval-augmented generation","long-document question answering","knowledge graphs","projection-aware retrieval","adaptive retrieval routing","evidence-bounded generation","abstention"],"falsifier":"Construct a test set of answerable questions with the key evidence removed but a thematically related distractor left in the retrieved package; if PAGE-RAG answers with citations when its sufficiency check says 'sufficient' on such a package — or refuses on genuinely sufficient evidence — then the gate is not tracking evidence sufficiency. Concretely: on the two book datasets, hide the chapter that actually contains the answer while leaving a paragraph that mentions the same characters, and observe whether the system abstains or re-retrieves instead of answering.","tokens_in":12337,"feed_emoji":"📚","tokens_out":13933,"duration_ms":108178,"temperature":0.7,"pith_summary":"The paper tries to establish that automatically constructed knowledge graphs should be treated as lossy semantic skeletons over source text, not as independent knowledge bases, and that reliable long-document QA requires three design choices: keep an always-on textual evidence floor, route queries to graph structure only when the query needs it, and make the final answer-or-abstain decision depend on an explicit evidence-sufficiency check. It claims this combination yields competitive answer quality while making knowledge-boundary violations rare: across two book-length corpora, PAGE-RAG reaches lenient accuracy of 92.6% and 87.5% and refuses all 24 unanswerable questions, where a strong passage baseline without the evidence gate correctly refuses only 4 of 24. It further claims a large efficiency gain, using roughly 1,956 query tokens per question on a global sensemaking benchmark versus 136,509 for a heavyweight graph-community system, while adding a balanced point on the quality–efficiency frontier rather than claiming universal dominance. A sympathetic reader should care because the paper reframes GraphRAG's goal: not just more answers, but answers that respect the boundary of available evidence, with abstention treated as a correct action.","feed_headline":"PAGE-RAG refuses all 24 unanswerable questions at ~90% accuracy","feed_subtitle":"It treats graphs as maps over source text and abstains when evidence is thin — at a fraction of the query cost.","key_machinery":"The central machinery is the projection-aware hybrid repository R(D)=(T(D), G(D), P, b): a textual evidence floor of citable chunks, a governed graph skeleton with communities and summaries, document profiles that condition extraction, and evidence bindings mapping every graph element to supporting spans. Around it sit the query-adaptive routing policy Π(u,O) — which composes retrieval operators (textual, neighborhood, path, community, rerank) under budget B — and the evidence-bounded generator with the sufficiency check s(q,E)∈{0,1}, which turns 'enough evidence?' into a binary answer-or-refuse decision. The binding keeps graph-guided answers traceable; the sufficiency check makes abstentio","core_discovery":"PAGE-RAG's central claim: an automatically constructed graph is a lossy projection of a long document, so it should navigate and organize evidence, not replace it. The system pairs a citable textual floor T(D) with a governed graph skeleton G(D) via evidence bindings b, and a routing policy activates textual, neighborhood, path, and community-summary operators under token budgets. At generation, an evidence-sufficiency check s(q,E) decides between a cited answer and refusal. Empirically, this yields 92.6% and 87.5% lenient accuracy with perfect refusal on all 24 unanswerable questions; removing the evidence gate raises apparent accuracy to 95.1% but collapses correct refusal to 4/24. On a gl","pith_inferences":["The answer-or-refuse gate, if hardened, could double as a permission-aware access-control layer for private RAG deployments, abstaining when evidence exists but the user's tier does not authorize it — a direction the paper only gestures at.","The routing policy's token savings (1,956 vs 136,509 per question) suggest that many 'global' queries can be served by a targeted subset of community summaries; a testable extension is measuring how summary count and precision trade off as the budget shrinks.","Because document profiles condition both extraction and routing, the framework predicts that a single fixed entity-relation schema underperforms profile-conditioned schemas on heterogeneous corpora; this could be tested by running the same pipeline across mixed fiction/nonfiction collections.","The 22/24 collapse when the evidence gate is removed implies that accuracy-only leaderboards reward systems that answer beyond their evidence; an editorial consequence is that closed-corpus benchmarks should report correct-refusal alongside accuracy to expose boundary violations."],"forward_implications":["Across two book-length datasets, a system can maintain competitive accuracy (92.6%/87.5% lenient) while correctly refusing every unanswerable question, so answer quality and boundary reliability are not mutually exclusive.","An explicit evidence-sufficiency gate, not a prompt instruction, is what enforces reliable abstention: removing it turns 22 of 24 unanswerable questions into substantive out-of-evidence answers even though apparent accuracy rises.","The textual floor is load-bearing: dropping it cuts lenient accuracy from 90.1% to 58.2%, implying graph-only retrieval cannot carry long-document QA alone.","Query-adaptive routing concentrates graph-path computation on queries that need structure, improving accuracy exactly where structural evidence helps (94.1%→89.7% when structural guidance is removed on routed questions).","Heavy community-summary style GraphRAG is a high-quality, high-cost extreme; a projection-aware system can approach it on global sensemaking while using ~1/70th of the query tokens."],"fun_headline_variants":["Graphs as maps, not facts: PAGE-RAG abstains when evidence is thin","PAGE-RAG: perfect refusal on all 24 unanswerable questions","Task-adaptive retrieval: PAGE-RAG routes queries to the right evidence","Strict knowledge boundary: PAGE-RAG only answers when grounded","PAGE-RAG cuts retrieval cost without sacrificing answer quality"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"Everything hinges on the evidence-sufficiency check s(q,E) — the paper's central answer-or-refuse gate — but the manuscript does not describe how it is computed, trained, or calibrated, so the claimed reliable abstention rests on an unverified component.","fun_headline_variants_meta":{"raw":{"variants":["Graphs as maps, not facts: PAGE-RAG abstains when evidence is thin","PAGE-RAG: perfect refusal on all 24 unanswerable questions","Task-adaptive retrieval: PAGE-RAG routes queries to the right evidence","Strict knowledge boundary: PAGE-RAG only answers when grounded","PAGE-RAG cuts retrieval cost without sacrificing answer quality"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000801,"raw_usage":{"total_tokens":3355,"prompt_tokens":740,"completion_tokens":2615,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":484,"completion_tokens_details":{"reasoning_tokens":2530}},"tokens_in":484,"tokens_out":2615,"duration_ms":19296,"temperature":1.0,"reasoning_tokens":2530,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T12:47:38.459055+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a test set of answerable questions with the key evidence removed but a thematically related distractor left in the retrieved package; if PAGE-RAG answers with citations when its sufficiency check says 'sufficient' on such a package — or refuses on genuinely sufficient evidence — then the gate is not tracking evidence sufficiency. Concretely: on the two book datasets, hide the chapter that actually contains the answer while leaving a paragraph that mentions the same characters, and observe whether the system abstains or re-retrieves instead of answering.","supporting_citations":[],"review_version":1}