Pith. sign in

REVIEW 4 major objections 5 minor 34 references

A Triple-Robustness Analysis of Retrieval-Augmented Generation for Multi-Hop Requirements Traceability

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Which RAG architecture wins depends on where you score citations.

desk verdict A careful, well-controlled study whose judge-stability finding is solid; the headline inversion claim needs the missing per-pipeline context-precision numbers before it convinces. read the letter →

arxiv 2608.00705 v1 pith:LCZPGW6J submitted 2026-08-01 cs.IR cs.CL

Meftun Akarsu , Burak Özdemir , Doğancan Büyükçolak , Recep Kaan Karaman This is my paper · ORCID
classification cs.IRcs.CL
keywords retrieval-augmentedgenerationGraphRAGrequirementstraceabilityDO-178CMuSiQueLLM-as-judgekappaparadoxtriple-robustness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Reported verdicts on GraphRAG versus vector RAG disagree because the field measures citations at inconsistent points. This paper holds five retrieval pipelines fixed and varies embedder, corpus, and judge, finding that GraphRAG's graph walk floods the context window with low-precision material (context precision $0.12$--$0.23$) while its synthesizer cites selectively (citation precision $0.48$--$0.65$); scoring the retrieved set as the attribution set inverts the architecture ranking. Per-hop winners are stable under embedder swap but conditional on corpus, and single-judge LLM faithfulness is fragile: GPT-5.4 agrees with itself across embedders at $\kappa=0.137$, and re-judging the same frozen inputs eleven weeks later gives $\kappa \le 0.14$ for both judges. A learned hop router on dense embeddings alone reaches macro-$F_1$ 0.86. The paper argues RAG architecture claims need this triple-robustness bar before being trusted.

What carries the argument

The load-bearing object is the triple-robustness matrix: five retrieval pipelines sharing an embedder, vector store, typed-edge graph, generator, and synthesis prompt, measured across two embedders, two corpora (a synthetic DO-178C-style typed-edge requirements corpus and MuSiQue Wikipedia chains), and two judges. Within it, the decisive mechanism is the distinction between the retrieved set handed to the synthesizer and the set of IDs the answer actually cites---context precision versus citation precision---plus a suite of same-judge controls (same-day test-retest, embedder swap, eleven-week re-judge of identical inputs) that separates the judge's noise floor from retrieval-state and temporal drift. Gwet's AC1 is used where Cohen's kappa is suppressed by prevalence, and significance claims require a BCa interval excluding zero, Holm-corrected Wilcoxon $p<0.05$, and Cliff's delta at least 0.147.

What would settle it

Run the same five-pipeline matrix on a human-authored, certified requirements corpus with real typed trace links and recompute context precision, citation precision, and per-stratum $F_1$; if GraphRAG's context flooding and answer-citation enrichment no longer diverge by 3--5x, or if the 3+-hop ordering does not favor graph-aided pipelines, C2a and C1 fail on that domain. Separately, have both judges re-judge the original 300 pinned tuples at a third date: if same-input self-agreement returns to the same-day test-retest level (raw agreement about 0.88) rather than about 0.56, the temporal-drift half of C3 collapses.

Watch

Extended reading notes

Core claim

The central discovery is that 'which RAG architecture is better' is not a property of the architectures alone; it is a joint product of the corpus, embedder, judge, and the point at which citations are measured. On a fixed five-pipeline matrix with two embedders, two corpora, and two judges, GraphRAG's local walk retrieves at context precision $0.12$--$0.23$ while its answer cites at $0.48$--$0.65$, a 3--5x enrichment. Scoring the retrieved set as the attribution set ranks GraphRAG last; scoring the answer's citations ranks it first or tied-first. Winners are corpus- and stratum-conditional but embedder-robust: GraphRAG ties vanilla on short-hop DO-178C queries and wins every MuSiQue stratum, while graph-aided agentic pipelines lead only on 3+-hop requirements queries. Faithfulness declines with hop distance on DO-178C (e.g., 74% to 40% under GPT-5.4 with the local embedder) but not on MuSiQue. Single-judge LLM faithfulness is retrieval-state- and date-fragile: same-input self-agreement across embedders is $\kappa=0.137$, and eleven-week re-judges give $\kappa \le 0.14$ for both judges.

Load-bearing premise

The paper's corpus-specific conclusions rest on a synthetic DO-178C-style corpus generated by the same model family that answers and judges; if that corpus does not represent real certification requirements, the architecture-ordering and hop-faithfulness results may not transfer to real traceability artifacts.

Editorial extensions

If this is right

  • Evaluations that score the retrieved set and evaluations that score the answer's citations will disagree; future RAG comparisons should state which attribution set they measure.
  • GraphRAG's multi-hop advantage on Wikipedia-style chains is a recall effect: the walk surfaces gold chunks dense retrieval misses, even though most of what it retrieves is off-gold.
  • On requirements-traceability workloads, hop distance should be a first-class stratification variable: graph-aided agentic pipelines help on 3+-hop queries but cost $F_1$ on short hops, so adaptive routing by predicted hop can close much of the gap to an oracle.
  • Faithfulness numbers from a single LLM judge should not be treated as stable evidence unless the judge is re-measured against itself on identical inputs across retrieval states and dates.
  • The hop-decodability result suggests query complexity can be predicted cheaply from dense embeddings alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the measurement-point inversion is real, published GraphRAG comparisons should be audited for which set they scored; contradictory verdicts may be traceable to this choice rather than to implementation differences.
  • Editorial inference: if single-judge faithfulness is date-fragile at $\kappa \le 0.14$ on frozen inputs, longitudinal RAG comparisons that change judge snapshots or re-judge old outputs later risk measuring judge drift rather than system change.
  • Editorial inference: the corpus-conditionality suggests a testable extension: on corpora where typed edges cross contradicting requirements (legal compliance, clinical guidelines), context flooding should again degrade faithfulness, while topically clustered corpora should not.
  • Editorial inference: because the learned router operates almost entirely on embedding PCs, hop classification may transfer to other multi-hop benchmarks; a cheap check is training the same logistic router on a human-authored multi-hop corpus.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper presents a triple-robustness analysis of retrieval-augmented generation architectures (vanilla, agentic, agentic+graph, GraphRAG, adaptive) for multi-hop requirements traceability, varying embedder (e5-small vs. Azure text-embedding-3-small), corpus (synthetic DO-178C-style requirements vs. MuSiQue Wikipedia paragraph chains), and judge (GPT-5.4 vs. GPT-4.1). The central claims are: (C2a) GraphRAG's graph walk floods the context window with low-precision material while the synthesizer cites selectively, so scoring the retrieved set as the attribution set inverts architecture rankings; (C1) answer-level citation winners are corpus- and stratum-conditional but embedder-robust; (C2b) faithfulness decline with hop distance is corpus-conditional; (C3) single-judge LLM faithfulness is fragile to retrieval-state and date changes, with self-kappa as low as 0.137 against a 0.76 test-retest floor; and (C4) a learned router on dense embeddings reaches macro-F1 0.86 for hop classification. The statistical protocol is elaborate, using paired BCa bootstrap, Holm-corrected Wilcoxon tests with an effect-size floor, McNemar exact tests, Gwet AC1, and Cochran-Armitage trend tests, alongside same-judge controls.

Significance. If the claims hold, the paper contributes a valuable and timely message: the choice of citation-measurement point (retrieved context vs. answer citations) can change which RAG architecture appears to win, and single-judge faithfulness evaluations are not stable evidence. The design has notable strengths: the cross-corpus replication on MuSiQue, the distractor-edge control for the graph retriever, the generator-swap control, and the explicit statistical protocol with pre-registered effect-size floors. The judge-fragility analysis with test-retest, embedder-swap, and eleven-week re-judge controls is unusually careful and directly quantifies a known but rarely measured source of instability. However, the headline measurement-point claim is under-evidenced as reported, the adaptive-pipeline evaluation contains a leakage path, and the synthetic DO-178C corpus limits the externally validity of the requirements-traceability conclusions.

major comments (4)
  1. [§5.2, Table 2] The claim that 'by context precision GraphRAG is the worst pipeline in every setting' is not supported by the reported data: Table 2 reports context precision for GraphRAG only, with no per-pipeline context-precision values for vanilla, agentic, agentic+graph, or adaptive in any setting. Without those numbers, the asserted ranking inversion from 'worst by context precision' to 'best by answer citations' is not demonstrated; it could be a GraphRAG-specific context/answer gap rather than a general measurement-point effect. Moreover, the companion claim that GraphRAG is 'the best or tied-best' by answer citations is contradicted by Table 1 on DO-178C 3+-hop, where agentic-graph (0.219) exceeds graphrag (0.172) under the local embedder. The inversion is therefore only partial, and the abstract's unqualified 'inverts the architecture ranking' overstates the result.
  2. [§3.4, Table 4] The evaluation of Adaptive-V2 contains a leakage path: the per-stratum routing target is 'derived once from the full locked matrix' and the same 296 queries are then used to evaluate Adaptive-V2's overall F1 and gap closure. Because the target is selected on the test queries themselves, the adaptive pipeline's F1 in Table 1 and the 59%/55% gap-closure figures in Table 4 are optimistically biased. The C4 claim that a learned router on dense embeddings closes most of the V1-to-Oracle gap is thus circular as reported. The routing target must be derived out-of-fold (or on a held-out development set) before the adaptive results can be trusted.
  3. [§3.2 and §6 (Threats to validity)] The DO-178C-style corpus is synthetic, authored by GPT-5.4, the same model family family that generates answers and judges faithfulness. The manuscript concedes that 'the corpus-author leg of the circularity remains' and that absolute F1 levels 'may not transfer' to real aerospace requirements. Yet C1 and C2b for the requirements domain rest entirely on this corpus. As written, the paper's title and framing ('requirements traceability') overstate the domain-level conclusion. A small human-authored or manually curated requirements corpus, or at minimum a consistent hedge in all abstract and conclusion statements, is needed to support the external-validity claims that the paper makes for DO-178C traceability.
  4. [§4 (Reproducibility)] The reproducibility section states that 'the full pipeline regenerates from a single make target on locked CSVs' but provides no URL, repository link, or commit hash. Since the paper's core claims—especially Table 2 and the C2a inversion—depend on exact per-pipeline context and citation numbers, the absence of an accessible artifact prevents external audit. A link or a clear statement of availability under review is necessary for the paper's reproducibility claims to be verifiable.
minor comments (5)
  1. [Table 3, §5.3] The notation 'κ=−0.05/−0.00(v2/v3)' is ambiguous; it should be written as two separate entries or with an explicit explanation that the slash separates the v2 and v3 main-matrix results.
  2. [§5.3] The sentence 'under a stationary judge, cross-date agreement should sit at the same-day test–retest level (0.88); the observed 0.56 rejects stationarity (binomial p<10−44)' should specify which quantities are being compared (raw agreement, not kappa) and give the sample size underlying the binomial test.
  3. [Figure 2 caption] The caption states that '*' marks pipelines significantly different from the per-stratum winner, but it does not mention that significance requires the joint Holm-Wilcoxon p<0.05 and |Cliff's delta|≥0.147 criterion; this should be stated to match the protocol in §3.3.
  4. [§3.3] The definition of 'context precision' should be made explicit for each pipeline, since the context set differs in size and composition (top-k for vanilla, graph-walk output for GraphRAG, agentic-loop accumulations). This is not only a clarity issue but also bears on the comparability of the C2a numbers.
  5. [§4, §3.2] There are minor text glitches: 'the dgslibisey MuSiQue mirror' appears to be a garbled repository name, and the word 'REFERENCES' appears mid-sentence in §3.2 and §5.1; these should be corrected.

Circularity Check

2 steps flagged · score 4.0 of 10

Admitted corpus-author circularity for DO-178C and a leaked routing target in the adaptive router; core C2a/C3 claims retain independent external checks.

  1. other [Section 6, Threats to validity (External)]
    "The DO-178C-style corpus is a single synthetic aerospace dataset, generated by the same model family that answers and judges; absolute F1 levels characterize this single regulated-domain setting and may not transfer. A generator-swap control (GPT-4.1 re-synthesizing 332 rows on frozen retrievals) reproduces the architecture ordering with slightly higher F1, and its answers show the same hop-wise faithfulness decline under both judges, so neither citation nor faithfulness results are GPT-5.4-specific; the corpus-author leg of the circularity remains."

    The primary DO-178C evidence corpus is authored by GPT-5.4, and the same model family generates the answers and supplies one of the two faithfulness judges. The C1 winner tables and C2b faithfulness trends on DO-178C therefore measure, at least in part, how GPT-5.4-structured text behaves under GPT-5.4-structured judgments, rather than an independent property of the retrieval pipelines. The paper itself names this the 'corpus-author leg of the circularity.' The MuSiQue replication and generator-swap control bound the damage, but they do not remove the loop for DO-178C-specific claims.

  2. fitted input called prediction [Section 3.4, V2 Learned Router]
    "The per-stratum routing target is the empirical mean-F1 winner, derived once from the full locked matrix (not re-fit per fold); only hop prediction is out-of-fold, reported for all 296 queries."

    Adaptive-V2's routing decision is defined by the per-stratum winner computed from the full locked evaluation matrix, i.e., from the answer-citation F1 outcomes on the very runs later reported in Table 4. Thus the 'Adaptive-V2 overall F1' and gap-closure numbers partly evaluate a router whose target was fitted to the evaluation data; only the hop classifier is truly out-of-fold. This makes C4's claim that dense embeddings alone classify hop more defensible than the claim that the adaptive pipeline's end-to-end F1 is an out-of-sample prediction. It is a fitted input rather than an independent prediction, though it is supporting rather than central.

full rationale

The headline measurement-point inversion (C2a) is not circular: it compares two metrics on the same runs and is externally replicated on MuSiQue; C3's judge-fragility finding is controlled by same-day test-retest and generator-swap controls. The two flagged issues are (1) the DO-178C corpus is GPT-5.4-authored, which the paper itself calls 'the corpus-author leg of the circularity,' making DO-178C-conditional C1/C2b results partly a same-model self-evaluation; and (2) the V2 router's per-stratum oracle is derived from the full locked matrix, so Adaptive-V2's reported F1 and gap closure are partly fitted rather than out-of-fold predictions. Because the central C2a/C3 claims have independent grounding (MuSiQue, distractor-edge control, test-retest, generator swap), the circularity is partial, not total. No load-bearing self-citation chain was found; citations to prior work are external benchmarks and methodology references. The missing per-pipeline context precision for non-GraphRAG systems noted by the skeptic is an evidentiary gap, not a circularity, and does not affect this score.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The paper's core robustness claims about citation-measurement point and judge instability rest on the standard statistical machinery and the specific pipeline implementations. The main free parameters are the per-stratum routing target fitted on the same evaluation matrix, the trained V2 router weights, and the hand-set effect-size threshold. The principal axiomatic risk is the synthetic, same-model-family-generated DO-178C corpus standing in for real certification requirements.

free parameters (3)
  • Per-stratum empirical routing target (mean-F1 winner) = vanilla on 1-hop, graphrag on 2-hop, agentic-graph on 3+-hop (DO-178C v2); varies per setting
    The V2 adaptive router predicts hop, then selects the pipeline that empirically won that stratum in the same locked main matrix (Section 3.4, 5.1). The target is derived from the evaluation data, so Adaptive-V2's reported gains are conditional on this fitted target.
  • Logistic regression weights and PCA-3 components for hop router = 50-fold fitted weights; macro-F1 0.78 +/- 0.08 (local), 0.86 +/- 0.05 (Azure)
    C4's router is a learned classifier with L2 C=1 and 14 features (11 hand text features, 3 embedding PCs). These are fitted parameters of the supporting contribution, not of the core C1-C3 claims.
  • Cliff's delta significance threshold = 0.147
    The paper sets |delta| >= 0.147 as the negligible-effect boundary; this hand-set threshold gates every pipeline-pair significance claim (Section 4).
assumptions (5)
  • domain assumption The synthetic DO-178C typed-edge corpus generated by GPT-5.4 is representative of real aerospace requirements, and its typed edges (derives_from, satisfies, traces_to) encode certification-relevant links.
    All DO-178C conclusions rest on this corpus. The paper calls it synthetic and says absolute F1 'may not transfer', yet uses it as the primary main-matrix evidence (Section 3.2, Section 6).
  • domain assumption GPT-5.4 and GPT-4.1 faithfulness judgments are a valid proxy for faithfulness despite low inter-judge agreement and temporal drift.
    C2b's faithfulness numbers are produced by these judges. The paper's own C3 shows self-kappa as low as -0.05 across dates, meaning the faithfulness ground truth is assumed meaningful even where judges disagree with themselves (Sections 3.3, 5.3).
  • domain assumption The MuSiQue graph with REFERENCES edges restricted to consecutive supporting paragraphs is a fair test of GraphRAG on Wikipedia chains, with the distractor-edge control mitigating the gold-only-edge artifact.
    The cross-corpus GraphRAG win on MuSiQue is measured on a gold-edge graph. The distractor control reduces but does not eliminate the worry that gold-only edges favor the graph walk (Sections 3.2, 5.1).
  • standard math The standard statistical machinery (BCa bootstrap, Wilcoxon signed-rank, Holm correction, McNemar, Cochran-Armitage, Gwet AC1) is correctly applied to the locked data.
    The paper reports using these tests with specific protocols (Section 4), but we did not independently run them on the CSVs.
  • domain assumption The five pipeline implementations faithfully represent the named architectures, given that 'GraphRAG' is a typed-edge local-walk variant, explicitly not Edge et al.'s community-summarization system.
    The paper labels its local-walk retriever as 'GraphRAG family'. Conclusions about 'GraphRAG' should be read about this implementation, and other family members may behave differently; the paper acknowledges this in Section 6.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Triple-Robustness Analysis of Retrieval-Augmented Generation for Multi-Hop Requirements Traceability." pith.science (2026). https://pith.science/paper/LCZPGW6J

@misc{pith2026260800705,
  author       = {Pith},
  title        = {Pith review of: A Triple-Robustness Analysis of Retrieval-Augmented Generation for Multi-Hop Requirements Traceability},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LCZPGW6J}},
  note         = {Machine review of arXiv:2608.00705}
}
read the original abstract

Reported verdicts on GraphRAG versus vector RAG disagree, and the evidence is typically tied to a single corpus, embedder, and judge -- and, we show, to where citation quality is measured. We present a triple-robustness analysis that holds a five-pipeline architecture matrix fixed and varies embedder (local e5-small vs. Azure text-embedding-3-small), corpus (DO-178C typed-edge requirements vs. Wikipedia paragraph chains via MuSiQue), and judge (paired GPT-5.4 x GPT-4.1 on both corpora), over 2x4,440 main-matrix runs, 600 cross-corpus runs, and over 5,000 faithfulness judgments. (C2a) GraphRAG's graph walk floods the context window at precision 0.12-0.23, but the synthesizer cites selectively at precision 0.48-0.65; scoring the retrieved set as the attribution set inverts the architecture ranking, which reconciles part of the disagreement in prior reports. (C1) Answer-level citation winners are corpus- and stratum-conditional but embedder-robust: GraphRAG ties vanilla on short-hop DO-178C queries and wins every MuSiQue stratum, while agentic pipelines lead only on 3+-hop requirements queries. (C2b) Faithfulness is corpus-conditional: on DO-178C it declines with hop distance (trend p<0.05 in three of four judge x embedder combinations); on Wikipedia chains neither judge shows a collapse. (C3) Single-judge LLM faithfulness is fragile to retrieval state: GPT-5.4's self-kappa across embedders is 0.137 (41% verdict change) against a same-day test-retest floor of 0.76, and re-judging frozen inputs eleven weeks later gives kappa <= 0.14 for both judges. A learned router on dense embeddings alone reaches macro-F1 0.86 on hop classification (C4). We argue that RAG architecture claims should be tested at this level of robustness -- including robustness to the citation-measurement point -- before they are trusted.

Figures

Figures reproduced from arXiv: 2608.00705 by the authors.

Figure 1
Figure 1. Five-pipeline architecture with shared embedder, vector store, and typed-edge graph. Vanilla and GraphRAG bypass the agentic loop; Agen￾tic and Agentic+Graph route through router/critic–retriever–tools (iter≤3) before synthesis. The adaptive pipeline (V1 rule-based or V2 learned) selects one of the four base pipelines per query. GraphRAG and multi-hop QA. Microsoft GraphRAG [6] and follow-ups [12, 13, 19] construct … view at source ↗
Figure 3
Figure 3. Multi-judge faithfulness on the v2 main matrix. Left: faithfulness fraction (5 pipelines × 3 hop strata × 2 judges); the GraphRAG row shows the monotonic decline driving C2b on DO-178C. Right: per-stratum Gwet AC1, showing the 𝜅 paradox at 3+-hop (AC1=0.57, 𝜅<0.10; cf [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 15 canonical work pages

  1. [1]

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. InProc. International Conference on Learning Representations (ICLR)

    work page 2024

  2. [2]

    Ingeol Baek, Hwan Chang, Byeongjeong Kim, Jimin Lee, and Hwanhee Lee. 2025. Probing-RAG: Self-Probing to Guide Language Models in Selective Document Retrieval. InFindings of NAACL. arXiv:2410.13339

    arXiv 2025

  3. [3]

    Taylor Berg-Kirkpatrick, David Burkett, and Dan Klein. 2012. An Empirical Investigation of Statistical Significance in NLP. InProc. EMNLP-CoNLL

    work page 2012

  4. [4]

    Guanting Dong, Jiajie Jin, Xiaoxi Li, Yutao Zhu, Zhicheng Dou, and Ji-Rong Wen

    work page

  5. [5]

    Wenzhang Du. 2025. When+1%Is Not Enough: A Paired Bootstrap Protocol for Evaluating Small Improvements.arXiv preprint arXiv:2511.19794(2025)

    arXiv 2025

  6. [6]

    Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. 2024. From Local to Global: A Graph RAG Approach to Query-Focused Summarization.arXiv preprint arXiv:2404.16130 (2024)

    arXiv 2024

  7. [7]

    European Union Aviation Safety Agency (EASA). 2025. Notice of Proposed Amendment NPA 2025-07: Detailed Specifications on AI Trustworthiness. EASA

    work page 2025

  8. [8]

    Federal Aviation Administration. 2024. AI Safety Roadmap, Version 1. FAA

    work page 2024

Show all 34 references
  1. [9]

    Feinstein and Domenic V

    Alvan R. Feinstein and Domenic V. Cicchetti. 1990. High Agreement but Low Kappa: I. The Problems of Two Paradoxes.Journal of Clinical Epidemiology43, 6 (1990), 543–549

    1990

  2. [10]

    Dominik Fuchß, Tobias Hey, Jan Keim, Haoyu Liu, Niklas Ewald, Tobias Thirolf, and Anne Koziolek. 2025. LiSSA: Toward Generic Traceability Link Recovery through Retrieval-Augmented Generation. InProc. IEEE/ACM 47th International Conference on Software Engineering (ICSE)

    2025

  3. [11]

    Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023. Enabling Large Lan- guage Models to Generate Text with Citations. InProc. EMNLP. arXiv:2305.14627

    2023 arXiv

  4. [12]

    Zirui Guo, Lianghao Xia, Yanhua Yu, Tu Ao, and Chao Huang. 2024. LightRAG: Simple and Fast Retrieval-Augmented Generation.arXiv preprint arXiv:2410.05779 (2024)

    2024 arXiv

  5. [13]

    Bernal Jiménez Gutiérrez, Yiheng Shu, Weijian Qi, Sizhe Zhou, and Yu Su. 2025. HippoRAG 2: From RAG to Memory: Non-Parametric Continual Learning for Large Language Models.arXiv preprint arXiv:2502.14802(2025)

    2025 arXiv

  6. [14]

    Kilem L. Gwet. 2008. Computing Inter-Rater Reliability and Its Variance in the Presence of High Agreement.Brit. J. Math. Statist. Psych.61, 1 (2008), 29–48

    2008

  7. [15]

    Aggarwal, and Jiliang Tang

    Haoyu Han, Li Ma, Yu Wang, Harry Shomer, Yongjia Lei, Zhisheng Qi, Kai Guo, Zhigang Hua, Bo Long, Hui Liu, Charu C. Aggarwal, and Jiliang Tang. 2025. RAG vs. GraphRAG: A Systematic Evaluation and Key Insights.arXiv preprint arXiv:2502.11371(2025)

    2025

  8. [16]

    Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong C. Park

  9. [17]

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. 2025. Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning.arXiv preprint arXiv:2503.09516(2025)

    2025 arXiv

  10. [18]

    Philipp Koehn. 2004. Statistical Significance Tests for Machine Translation Evaluation. InProc. EMNLP. Meftun Akarsu, Burak Özdemir, Doğancan Büyükçolak, and Recep Kaan Karaman

    2004

  11. [19]

    Haoran Luo, Haihong E, Guanting Chen, Qika Lin, Yikai Guo, Fangzhi Xu, Zemin Kuang, Meina Song, Xiaobao Wu, Yifan Zhu, and Luu Anh Tuan. 2026. Graph-R1: Towards Agentic GraphRAG Framework via End-to-End Reinforcement Learning. InProc. International Conference on Machine Learni...

    2026 arXiv

  12. [20]

    Arsalan Masoudifard, Mohammad Mowlavi Sorond, Moein Madadi, Moham- mad Sabokrou, and Elahe Habibi. 2024. Leveraging Graph-RAG and Prompt Engineering to Enhance LLM-Based Automated Requirement Traceability and Compliance Checks.arXiv preprint arXiv:2412.08593(2024)

    2024 arXiv

  13. [21]

    Briand, Hanyang Hu, and Krishna Koravadi

    Feifei Niu, Rongqi Pan, Lionel C. Briand, Hanyang Hu, and Krishna Koravadi

  14. [22]

    2011.DO-178C: Software Considerations in Airborne Systems and Equipment Certification

    RTCA, Inc. 2011.DO-178C: Software Considerations in Airborne Systems and Equipment Certification. Technical Report. RTCA

    2011

  15. [23]

    Dongyu Ru, Lin Qiu, Xiangkun Hu, Tianhang Zhang, Peng Shi, et al . 2024. RAGChecker: A Fine-Grained Framework for Diagnosing Retrieval-Augmented Generation. InAdvances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track. arXiv:2408.08067

    2024 arXiv

  16. [24]

    TVR: Automotive System Requirement Traceability Validation and Recov- ery Through Retrieval-Augmented Generation.arXiv preprint arXiv:2504.15427 (2025)

    2025 arXiv

  17. [25]

    Yixuan Tang and Yi Yang. 2024. MultiHop-RAG: Benchmarking Retrieval- Augmented Generation for Multi-Hop Queries.arXiv preprint arXiv:2401.15391 (2024)

    2024 arXiv

  18. [26]

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal

  19. [27]

    2010.ARP4754A: Guidelines for Development of Civil Aircraft and Systems

    SAE International. 2010.ARP4754A: Guidelines for Development of Civil Aircraft and Systems. Technical Report. SAE International

    2010

  20. [28]

    Koki Wataoka, Tsubasa Takahashi, and Ryokan Ri. 2024. Self-Preference Bias in LLM-as-a-Judge. InNeurIPS Safe Generative AI Workshop. arXiv:2410.21819

    2024 arXiv

  21. [29]

    Zhishang Xiang, Chuanjie Wu, Qinggang Zhang, Shengyuan Chen, Zijin Hong, Xiao Huang, and Jinsong Su. 2026. When to Use Graphs in RAG: A Comprehen- sive Analysis for Graph Retrieval-Augmented Generation. InProc. International Conference on Learning Representations (ICLR). arXiv...

    2026

  22. [30]

    Shi-Qi Yan, Jia-Chen Gu, Yun Zhu, and Zhen-Hua Ling. 2024. Corrective Retrieval Augmented Generation.arXiv preprint arXiv:2401.15884(2024)

    2024 arXiv

  23. [31]

    Jonas Wallat, Maria Heuss, Maarten de Rijke, and Avishek Anand. 2024. Cor- rectness is not Faithfulness in RAG Attributions.arXiv preprint arXiv:2412.18004 (2024)

    2024 arXiv

  24. [2022]

    Transactions of the Association for Computational Linguistics (TACL)10 (2022), 539–554

    MuSiQue: Multihop Questions via Single-Hop Question Composition. Transactions of the Association for Computational Linguistics (TACL)10 (2022), 539–554

    2022

  25. [2024]

    Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity. InProc. NAACL

  26. [2025]

    RAG-Critic: Leveraging Automated Critic-Guided Agentic Workflow for Retrieval Augmented Generation. InProc. 63rd Annual Meeting of the Association for Computational Linguistics (ACL): Long Papers. 3551–3578. Anthology ID: 2025.acl-long.179

    2025

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.