{"id":"e5c36c00-880d-465e-8f18-ff1b99d078d5","arxiv_id":"2608.00705","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"RAG architecture rankings flip depending on whether citations are measured over the retrieved context or over the answer, and single-judge LLM faithfulness is highly unstable across embedders and over time.","lead":"This paper shows that whether a graph-based retrieval system looks better than a plain one depends on where you measure its citation quality, and that AI judges are so unstable they disagree with themselves on 41 percent of the same cases when the search embedder changes. If true, this means published comparisons of retrieval-augmented generation systems need paired controls and explicit measurement-point choices before their verdicts can be trusted.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"C2a inversion claim lacks full-pipeline context-precision evidence: Table 2 reports GraphRAG only, so 'ranks last by context precision' is not demonstrated.","rationale":"The reader's weakest_assumption (corpus representativeness) is real but secondary for the strongest claim: C2a's measurement-point inversion is replicated on MuSiQue, and C3's judge-fragility is established through same-input temporal controls that do not depend on the synthetic corpus. The more targeted risk is internal: the paper's own evidence for the inversion is incomplete. Table 1 gives citation F1 for all pipelines, but Table 2 reports context precision only for GraphRAG. Without context precision for vanilla, agentic, agentic-graph, and adaptive, the 'ranks last' half of the inversion is unsupported. This is a concrete, checkable gap directly in the paper's headline claim. The paper otherwise shows careful controls (paired bootstrap, Holm-Wilcoxon, distractor-edge control, temporal re-judging), and the author-identified corpus circularity is correctly acknowledged in Threats to Validity. The verdict should remain CONDITIONAL: the paper's central claims are plausible but this missing comparison must be supplied before C2a can be fully accepted. No change to the reader's overall conditional verdict is needed, hence UNCHANGED; but the specific missing evidence should be highlighted in any revision request.","tokens_in":10773,"tokens_out":8466,"duration_ms":74631,"concrete_test":"Compute context precision (gold fraction of the retrieved set handed to the synthesizer) for all five pipelines on the v2 and v3 DO-178C matrices and for vanilla/agentic-graph/GraphRAG on MuSiQue, then rank pipelines per stratum under context precision and compare with Table 1's citation-F1 ranking. If GraphRAG is not the strict minimum (or tied minimum) under context precision in every stratum where it is best or tied-best under citation F1, the C2a inversion claim fails. Also release the locked CSVs with this column so the check is reproducible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central measurement-point claim (C2a) is that scoring the retrieved set as the attribution set inverts the architecture ranking, with GraphRAG 'worst' by context precision and 'best or tied-best' by answer citations. The paper displays citation F1 for all pipelines (Table 1) but context precision only for GraphRAG (Table 2: 0.120/0.129/0.227). The statement in §5.2 that 'by context precision GraphRAG is the worst pipeline in every setting' is therefore an assertion without supporting per-pipeline numbers in the manuscript. Vanilla retrieves only 5 context chunks versus GraphRAG's 11-15, so it is plausible but not shown that GraphRAG is strictly worse under context precision. If another pipeline (e.g., vanilla under a low-recall embedder) has comparably low context precision, the ranking flip is not from GraphRAG-first to GraphRAG-last, and the 'inversion' reduces to a GraphRAG-specific context/answer gap rather than a general measurement-point effect. The reproducibility section promises locked CSVs but provides no link or commit hash, so this table cannot be audited externally. This is load-bearing because C2a is the paper's headline mechanism for reconciling prior contradictory GraphRAG verdicts.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a triple-robustness analysis of retrieval-augmented generation architectures (vanilla, agentic, agentic+graph, GraphRAG, adaptive) for multi-hop requirements traceability, varying embedder (e5-small vs. Azure text-embedding-3-small), corpus (synthetic DO-178C-style requirements vs. MuSiQue Wikipedia paragraph chains), and judge (GPT-5.4 vs. GPT-4.1). The central claims are: (C2a) GraphRAG's graph walk floods the context window with low-precision material while the synthesizer cites selectively, so scoring the retrieved set as the attribution set inverts architecture rankings; (C1) answer-level citation winners are corpus- and stratum-conditional but embedder-robust; (C2b) faithfulness decline with hop distance is corpus-conditional; (C3) single-judge LLM faithfulness is fragile to retrieval-state and date changes, with self-kappa as low as 0.137 against a 0.76 test-retest floor; and (C4) a learned router on dense embeddings reaches macro-F1 0.86 for hop classification. The statistical protocol is elaborate, using paired BCa bootstrap, Holm-corrected Wilcoxon tests with an effect-size floor, McNemar exact tests, Gwet AC1, and Cochran-Armitage trend tests, alongside same-judge controls.","tokens_in":10993,"tokens_out":6437,"duration_ms":54067,"significance":"If the claims hold, the paper contributes a valuable and timely message: the choice of citation-measurement point (retrieved context vs. answer citations) can change which RAG architecture appears to win, and single-judge faithfulness evaluations are not stable evidence. The design has notable strengths: the cross-corpus replication on MuSiQue, the distractor-edge control for the graph retriever, the generator-swap control, and the explicit statistical protocol with pre-registered effect-size floors. The judge-fragility analysis with test-retest, embedder-swap, and eleven-week re-judge controls is unusually careful and directly quantifies a known but rarely measured source of instability. However, the headline measurement-point claim is under-evidenced as reported, the adaptive-pipeline evaluation contains a leakage path, and the synthetic DO-178C corpus limits the externally validity of the requirements-traceability conclusions.","major_comments":[{"comment":"The claim that 'by context precision GraphRAG is the worst pipeline in every setting' is not supported by the reported data: Table 2 reports context precision for GraphRAG only, with no per-pipeline context-precision values for vanilla, agentic, agentic+graph, or adaptive in any setting. Without those numbers, the asserted ranking inversion from 'worst by context precision' to 'best by answer citations' is not demonstrated; it could be a GraphRAG-specific context/answer gap rather than a general measurement-point effect. Moreover, the companion claim that GraphRAG is 'the best or tied-best' by answer citations is contradicted by Table 1 on DO-178C 3+-hop, where agentic-graph (0.219) exceeds graphrag (0.172) under the local embedder. The inversion is therefore only partial, and the abstract's unqualified 'inverts the architecture ranking' overstates the result.","section":"§5.2, Table 2"},{"comment":"The evaluation of Adaptive-V2 contains a leakage path: the per-stratum routing target is 'derived once from the full locked matrix' and the same 296 queries are then used to evaluate Adaptive-V2's overall F1 and gap closure. Because the target is selected on the test queries themselves, the adaptive pipeline's F1 in Table 1 and the 59%/55% gap-closure figures in Table 4 are optimistically biased. The C4 claim that a learned router on dense embeddings closes most of the V1-to-Oracle gap is thus circular as reported. The routing target must be derived out-of-fold (or on a held-out development set) before the adaptive results can be trusted.","section":"§3.4, Table 4"},{"comment":"The DO-178C-style corpus is synthetic, authored by GPT-5.4, the same model family family that generates answers and judges faithfulness. The manuscript concedes that 'the corpus-author leg of the circularity remains' and that absolute F1 levels 'may not transfer' to real aerospace requirements. Yet C1 and C2b for the requirements domain rest entirely on this corpus. As written, the paper's title and framing ('requirements traceability') overstate the domain-level conclusion. A small human-authored or manually curated requirements corpus, or at minimum a consistent hedge in all abstract and conclusion statements, is needed to support the external-validity claims that the paper makes for DO-178C traceability.","section":"§3.2 and §6 (Threats to validity)"},{"comment":"The reproducibility section states that 'the full pipeline regenerates from a single make target on locked CSVs' but provides no URL, repository link, or commit hash. Since the paper's core claims—especially Table 2 and the C2a inversion—depend on exact per-pipeline context and citation numbers, the absence of an accessible artifact prevents external audit. A link or a clear statement of availability under review is necessary for the paper's reproducibility claims to be verifiable.","section":"§4 (Reproducibility)"}],"minor_comments":[{"comment":"The notation 'κ=−0.05/−0.00(v2/v3)' is ambiguous; it should be written as two separate entries or with an explicit explanation that the slash separates the v2 and v3 main-matrix results.","section":"Table 3, §5.3"},{"comment":"The sentence 'under a stationary judge, cross-date agreement should sit at the same-day test–retest level (0.88); the observed 0.56 rejects stationarity (binomial p<10−44)' should specify which quantities are being compared (raw agreement, not kappa) and give the sample size underlying the binomial test.","section":"§5.3"},{"comment":"The caption states that '*' marks pipelines significantly different from the per-stratum winner, but it does not mention that significance requires the joint Holm-Wilcoxon p<0.05 and |Cliff's delta|≥0.147 criterion; this should be stated to match the protocol in §3.3.","section":"Figure 2 caption"},{"comment":"The definition of 'context precision' should be made explicit for each pipeline, since the context set differs in size and composition (top-k for vanilla, graph-walk output for GraphRAG, agentic-loop accumulations). This is not only a clarity issue but also bears on the comparability of the C2a numbers.","section":"§3.3"},{"comment":"There are minor text glitches: 'the dgslibisey MuSiQue mirror' appears to be a garbled repository name, and the word 'REFERENCES' appears mid-sentence in §3.2 and §5.1; these should be corrected.","section":"§4, §3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a strong statistical protocol and a genuinely interesting central idea about citation-measurement point, but two load-bearing issues need to be fixed before publication: (a) the C2a inversion claim lacks per-pipeline context-precision evidence and is overstated relative to Table 1, and (b) the Adaptive-V2 evaluation is leaky because the routing target is derived from the full test matrix. The synthetic-corpus circularity is a more fundamental concern but is at least openly acknowledged; the authors should either temper the requirements-traceability framing or add external validation. I would not reject: the design is sophisticated and the weaknesses are addressable in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis is one of the more serious empirical RAG studies I've seen. The design holds a five-pipeline matrix fixed, varies embedder, corpus, and judge, and applies a genuinely strong statistical protocol: paired BCa bootstrap, Holm-Wilcoxon with an effect-size floor, McNemar exact tests, and Gwet's AC1. The judge-fragility results are the real contribution. Same-day test-retest kappa of 0.76 versus across-embedder self-kappa of 0.14, and an eleven-week re-judge with near-zero or negative kappa for both judges, makes a compelling case that single-judge faithfulness numbers are not stable evidence. That alone is worth citing.\n\nThe measurement-point argument (C2a) is interesting but not fully supported in the manuscript as given. Table 2 reports context precision only for GraphRAG (0.12-0.23), and the text asserts that \"by context precision GraphRAG is the worst pipeline in every setting\" without displaying per-pipeline context precision for vanilla, agentic, and the rest. The stress-test note is right: since vanilla retrieves only five chunks against GraphRAG's eleven to fifteen, the inversion is plausible, but it is not shown. The promised locked CSVs may contain the numbers, but there is no link or commit hash. That is a fixable but load-bearing gap for the headline claim.\n\nOther soft spots are the ones the authors themselves flag. The DO-178C corpus is generated by the same model family that answers and judges, so the requirements-specific conclusions rest on a circular leg; the MuSiQue replication helps for the general claims but not for aerospace traceability. The abstract's \"embedder-robust\" phrasing also outruns the Azure 3+-hop evidence, where no pair reaches significance.\n\nWho is this for? Anyone working on RAG evaluation or LLM-as-judge methodology. This paper deserves a serious referee—the design, controls, and honest threats-to-validity section put it well above desk-reject territory. But it needs the missing context-precision table and the reproducibility artifacts before the inversion claim can be trusted.\n\nRecommendation: send it to peer review, and require the authors to ship the locked CSVs and the full per-pipeline context-precision numbers.","headline":"A careful, well-controlled study whose judge-stability finding is solid; the headline inversion claim needs the missing per-pipeline context-precision numbers before it convinces.","tokens_in":11628,"tokens_out":2320,"would_cite":true,"duration_ms":19931,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Which RAG architecture wins depends on where you score citations.","keywords":["retrieval-augmented generation","GraphRAG","requirements traceability","DO-178C","MuSiQue","LLM-as-judge","kappa paradox","triple-robustness"],"falsifier":"Run the same five-pipeline matrix on a human-authored, certified requirements corpus with real typed trace links and recompute context precision, citation precision, and per-stratum $F_1$; if GraphRAG's context flooding and answer-citation enrichment no longer diverge by 3--5x, or if the 3+-hop ordering does not favor graph-aided pipelines, C2a and C1 fail on that domain. Separately, have both judges re-judge the original 300 pinned tuples at a third date: if same-input self-agreement returns to the same-day test-retest level (raw agreement about 0.88) rather than about 0.56, the temporal-drift half of C3 collapses.","tokens_in":1998,"feed_emoji":"🔎","tokens_out":4825,"duration_ms":90848,"temperature":0.7,"pith_summary":"Reported verdicts on GraphRAG versus vector RAG disagree because the field measures citations at inconsistent points. This paper holds five retrieval pipelines fixed and varies embedder, corpus, and judge, finding that GraphRAG's graph walk floods the context window with low-precision material (context precision $0.12$--$0.23$) while its synthesizer cites selectively (citation precision $0.48$--$0.65$); scoring the retrieved set as the attribution set inverts the architecture ranking. Per-hop winners are stable under embedder swap but conditional on corpus, and single-judge LLM faithfulness is fragile: GPT-5.4 agrees with itself across embedders at $\\kappa=0.137$, and re-judging the same frozen inputs eleven weeks later gives $\\kappa \\le 0.14$ for both judges. A learned hop router on dense embeddings alone reaches macro-$F_1$ 0.86. The paper argues RAG architecture claims need this triple-robustness bar before being trusted.","feed_headline":"Where you score citations decides which RAG wins","feed_subtitle":"GraphRAG's context looks worst, its citations look best; LLM judges often disagree with themselves.","key_machinery":"The load-bearing object is the triple-robustness matrix: five retrieval pipelines sharing an embedder, vector store, typed-edge graph, generator, and synthesis prompt, measured across two embedders, two corpora (a synthetic DO-178C-style typed-edge requirements corpus and MuSiQue Wikipedia chains), and two judges. Within it, the decisive mechanism is the distinction between the retrieved set handed to the synthesizer and the set of IDs the answer actually cites---context precision versus citation precision---plus a suite of same-judge controls (same-day test-retest, embedder swap, eleven-week re-judge of identical inputs) that separates the judge's noise floor from retrieval-state and temporal drift. Gwet's AC1 is used where Cohen's kappa is suppressed by prevalence, and significance claims require a BCa interval excluding zero, Holm-corrected Wilcoxon $p<0.05$, and Cliff's delta at least 0.147.","core_discovery":"The central discovery is that 'which RAG architecture is better' is not a property of the architectures alone; it is a joint product of the corpus, embedder, judge, and the point at which citations are measured. On a fixed five-pipeline matrix with two embedders, two corpora, and two judges, GraphRAG's local walk retrieves at context precision $0.12$--$0.23$ while its answer cites at $0.48$--$0.65$, a 3--5x enrichment. Scoring the retrieved set as the attribution set ranks GraphRAG last; scoring the answer's citations ranks it first or tied-first. Winners are corpus- and stratum-conditional but embedder-robust: GraphRAG ties vanilla on short-hop DO-178C queries and wins every MuSiQue stratum, while graph-aided agentic pipelines lead only on 3+-hop requirements queries. Faithfulness declines with hop distance on DO-178C (e.g., 74% to 40% under GPT-5.4 with the local embedder) but not on MuSiQue. Single-judge LLM faithfulness is retrieval-state- and date-fragile: same-input self-agreement across embedders is $\\kappa=0.137$, and eleven-week re-judges give $\\kappa \\le 0.14$ for both judges.","pith_inferences":["Editorial inference: if the measurement-point inversion is real, published GraphRAG comparisons should be audited for which set they scored; contradictory verdicts may be traceable to this choice rather than to implementation differences.","Editorial inference: if single-judge faithfulness is date-fragile at $\\kappa \\le 0.14$ on frozen inputs, longitudinal RAG comparisons that change judge snapshots or re-judge old outputs later risk measuring judge drift rather than system change.","Editorial inference: the corpus-conditionality suggests a testable extension: on corpora where typed edges cross contradicting requirements (legal compliance, clinical guidelines), context flooding should again degrade faithfulness, while topically clustered corpora should not.","Editorial inference: because the learned router operates almost entirely on embedding PCs, hop classification may transfer to other multi-hop benchmarks; a cheap check is training the same logistic router on a human-authored multi-hop corpus."],"forward_implications":["Evaluations that score the retrieved set and evaluations that score the answer's citations will disagree; future RAG comparisons should state which attribution set they measure.","GraphRAG's multi-hop advantage on Wikipedia-style chains is a recall effect: the walk surfaces gold chunks dense retrieval misses, even though most of what it retrieves is off-gold.","On requirements-traceability workloads, hop distance should be a first-class stratification variable: graph-aided agentic pipelines help on 3+-hop queries but cost $F_1$ on short hops, so adaptive routing by predicted hop can close much of the gap to an oracle.","Faithfulness numbers from a single LLM judge should not be treated as stable evidence unless the judge is re-measured against itself on identical inputs across retrieval states and dates.","The hop-decodability result suggests query complexity can be predicted cheaply from dense embeddings alone."],"supporting_citations":[{"why":"Supplies the GraphRAG family architecture whose typed-edge local walk is compared as the GraphRAG pipeline.","marker":"[6]"},{"why":"Defines the ALCE-style citation precision/recall/F1 metric used for answer-level attribution.","marker":"[11]"},{"why":"Provides the Wikipedia multi-hop query subset and supporting-paragraph chains used for cross-corpus replication.","marker":"[26]"},{"why":"Identifies the kappa paradox that motivates reporting Gwet's AC1.","marker":"[9]"},{"why":"Gives the AC1 prevalence-corrected agreement statistic used for inter-judge and same-judge agreement.","marker":"[14]"},{"why":"Is a prior RAG-versus-GraphRAG evaluation reporting GraphRAG underperformance on detailed queries, the contradictory verdict C2a reconciles.","marker":"[15]"},{"why":"Another prior GraphRAG benchmark used to frame the disagreement about when graphs help in RAG.","marker":"[29]"},{"why":"Supplies the 'correctness is not faithfulness' distinction motivating separate LLM faithfulness judgments.","marker":"[27]"},{"why":"Defines the paired BCa bootstrap protocol used for confidence intervals and significance.","marker":"[5]"}],"fun_headline_variants":["Scoring citations flips which RAG wins","GraphRAG: worst context, best citations","RAG winner depends on where you score citations","LLM judges disagree: kappa 0.14 on retest","Triple-robustness: corpus, embedder, judge shift RAG rank"],"cache_read_input_tokens":13696,"weakest_assumption_plain":"The paper's corpus-specific conclusions rest on a synthetic DO-178C-style corpus generated by the same model family that answers and judges; if that corpus does not represent real certification requirements, the architecture-ordering and hop-faithfulness results may not transfer to real traceability artifacts.","fun_headline_variants_meta":{"raw":{"variants":["Scoring citations flips which RAG wins","GraphRAG: worst context, best citations","RAG winner depends on where you score citations","LLM judges disagree: kappa 0.14 on retest","Triple-robustness: corpus, embedder, judge shift RAG rank"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000303,"raw_usage":{"total_tokens":1893,"prompt_tokens":1246,"completion_tokens":647,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":862,"completion_tokens_details":{"reasoning_tokens":563}},"tokens_in":862,"tokens_out":647,"duration_ms":5381,"temperature":1.0,"reasoning_tokens":563,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:17:35.107450+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same five-pipeline matrix on a human-authored, certified requirements corpus with real typed trace links and recompute context precision, citation precision, and per-stratum $F_1$; if GraphRAG's context flooding and answer-citation enrichment no longer diverge by 3--5x, or if the 3+-hop ordering does not favor graph-aided pipelines, C2a and C1 fail on that domain. Separately, have both judges re-judge the original 300 pinned tuples at a third date: if same-input self-agreement returns to the same-day test-retest level (raw agreement about 0.88) rather than about 0.56, the temporal-drift half of C3 collapses.","supporting_citations":[{"cited_title":"Feinstein and Domenic V","cited_arxiv_id":null,"evidence_quote":"Identifies the kappa paradox that motivates reporting Gwet's AC1."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the AC1 prevalence-corrected agreement statistic used for inter-judge and same-judge agreement."},{"cited_title":"2010.ARP4754A: Guidelines for Development of Civil Aircraft and Systems","cited_arxiv_id":null,"evidence_quote":"Supplies the 'correctness is not faithfulness' distinction motivating separate LLM faithfulness judgments."}],"review_version":1}