Pith. sign in

REVIEW 4 major objections 6 minor 71 references

TRIAGE instruments automated graph-RAG so a failure is localized to extraction, graph construction, or retrieval rather than only discovered as a wrong final answer.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 02:23 UTC pith:UME6FMKM

load-bearing objection Clear stage-aware instrumentation for Graph-RAG; the diagnostic chain is the real idea, but localization and remediation remain design hypotheses, not demonstrated results. the 4 major comments →

arxiv 2607.03447 v1 pith:UME6FMKM submitted 2026-07-03 cs.IR cs.AIcs.CL

TRIAGE: Trustworthy Retrieval Instrumentation And Graph Evaluation

Axel TahmasebiMoradi , Lucas Schott , Martin Royer This is my paper
classification cs.IR cs.AIcs.CL
keywords Knowledge GraphsGraph RAGTrustworthinessTrust MetricsKnowledge Graph EvaluationAutomated Knowledge Graph ConstructionRetrieval-Augmented Generation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Knowledge graphs that power graph-based retrieval-augmented generation are increasingly built by language-model extraction rather than expert curation, yet systems are still judged mainly by whether the final answer looks right. TRIAGE argues that proper evaluation must instrument extraction, graph assembly, and inference with stage-specific metrics so a failure at one stage is not merely discovered as an unsupported answer at the end. Most of those metrics need no gold labels at deployment: triple confidence, source coverage, and schema checks at construction; structural quality at validation; retrieval coverage, faithfulness, and path cost at usage. At query time the usage metrics form an ordered chain of necessary conditions whose first broken link names a failure signature and points to the levers that can remedy it—extraction, graph-and-schema, or retrieval. A small proof of concept shows a capable reader can still answer from parametric memory when retrieval has missed the needed evidence, which is why the framework scores the retrieved subgraph before generation.

Core claim

Automated, document-grounded graph-RAG pipelines can be instrumented end-to-end with stage-specific, mostly gold-free trust and cost metrics so that failures are localized to extraction, graph-and-schema, or retrieval rather than only observed as incorrect final answers. At usage time the metrics form a diagnostic chain of necessary conditions; the first broken link identifies a failure signature that maps to the corresponding stage levers.

What carries the argument

The usage-stage diagnostic chain of necessary conditions (query grounding → entity retrieval coverage → reasoning readiness → answer reasoning faithfulness → answer grounding). The first link that falls below threshold names the failure signature and selects the remediation stage.

Load-bearing premise

The structural retrieval metrics are predictive and localizing enough that their first broken link can reliably drive remediation, even though the supporting probe is small and synthetic and the full localization tests remain unrun.

What would settle it

On natural multi-hop benchmarks, if queries stratified by entity retrieval coverage and reasoning readiness show no monotonic rise in context coverage or answer correctness, or if the five failure signatures cannot be separated by their metric profiles, the localization claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Live deployments can compute gold-free signals on every query and flag retrieval failures before the generator runs.
  • Aggregate drops in stage metrics can trigger a targeted knowledge-graph update rather than an undifferentiated rebuild.
  • Retrieval cost becomes an explicit, hardware-independent quality dimension alongside trustworthiness.
  • When no ontology or gold standard is available, assessment degrades gracefully to the schema-free and gold-free subset of metrics.
  • Silent successes in which the model answers from parametric memory while retrieval fails become detectable rather than invisible.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same ordered-chain idea could be adapted to ordinary text RAG by replacing graph-connectivity metrics with passage-coverage and entailment signals.
  • If the chain proves localizing at scale, continuous health dashboards for production knowledge graphs become practical rather than aspirational.
  • Answer-level LLM-as-judge protocols likely understate retrieval failure rates whenever the model has strong parametric knowledge of the domain.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. TRIAGE proposes a stage-aware instrumentation framework for automated, document-grounded Graph-RAG. It attaches gold-free, stage-specific metrics to three phases—KG Implementation (e.g., TCS, SCR, SC/CVR, CSR_link), KG Validation (e.g., DNR, SRR, OC, LC; COR/CMP only offline), and KG Usage (QGR, ERC, RRS, AGR/AUR/ARF, RPC/RHD)—organized by reference requirement and white-/black-box visibility. At usage time the metrics form a diagnostic chain of necessary conditions (QGR→ERC→RRS→ARF→AGR) whose first broken link is said to localize a failure signature and map it to extraction, graph-and-schema, or retrieval levers. The paper is positioned as a theoretical framework with a usage-stage proof of concept on a synthetic PathQuestion/PQ-3H probe and a pre-specified falsification protocol for the remaining claims.

Significance. If the diagnostic chain and remediation map hold under realistic conditions, TRIAGE would fill a genuine gap: Graph-RAG systems and answer-level judges currently leave extraction, graph structure, and retrieval failures invisible until the final answer, while TKG-style methodologies do not cover automated LLM extraction or usage. The gold-free, stage-localized design, the explicit white-box/black-box and ontology axes, the primary/derived metric analysis, and the pre-inference triage idea are practically useful contributions. Credit is due for an honest PoC that isolates the parametric silent-success residual (CC separates sharply by ERC/RRS while Hits@1 does not) and for specifying a falsifiable evaluation protocol rather than claiming full validation. The work is therefore significant as a deployable instrumentation blueprint, provided the localization and remediation claims are either demonstrated or clearly scoped as hypotheses.

major comments (4)
  1. [§4.5, Fig. 5, §5.1–5.2, Table 5] §4.5, Fig. 5, and the abstract state that the first broken link in the usage chain localizes the failure and maps it to stage levers (extraction / graph-and-schema / retrieval). §5.1 and Table 5 only show that ERC and RRS stratify reader-free Context Coverage on a synthetic 96-question PQ-3H probe (480 non-independent question×retriever observations), with CC rising from ~44% at ERC<0.5 to 98.4% at ERC=1. They do not test mutual distinguishability of the five signatures, co-movement of upstream Implementation/Validation metrics (TCS, DNR, OC, CMP) with chain breaks, or that acting on the indicated lever remediates the failure. Localization, utility, and the remediation map are deferred to the unexecuted protocol in §5.2. Either run those tests or reframe the chain as a design hypothesis whose localization/remediation claims remain unvalidated.
  2. [§5.1, Table 5] The PoC (§5.1) uses synthetic distance-controlled questions, in-sample Youden thresholds (θ_ERC=0.67), and five retrieval views of the same 96 questions without clustered uncertainty. Table 5 is correctly labeled descriptive, but the text still treats the separation as preliminary evidence for the framework’s central diagnostic claim. For a journal contribution that hinges on predictive and localizing validity, the PoC should either (i) report clustered CIs / hold-out thresholds and natural multi-hop queries (WebQSP, MetaQA as planned in §5.2), or (ii) explicitly limit claims to “ERC/RRS associate with evidence presence on this controlled probe,” without implying support for first-broken-link diagnosis.
  3. [§4.5] Necessity of structural proxies does not establish root-cause attribution. By construction ERC≤QGR; later links are logical preconditions (§4.5). A connectivity gap (high ERC, low RRS) may stem from failed entity resolution (Validation/DNR), missing extraction (Implementation), or the retrieval primitive itself. The manuscript acknowledges this (“localizes which condition failed, not why”) but still presents the stage-lever map as the framework’s distinctive payoff. Without the localization alignment study in §5.2 (Spearman ρ of upstream metrics with ERC/RRS; signature separability), the map remains an untested design choice. Strengthen the text to separate condition localization (supported by chain order) from stage attribution (not yet shown).
  4. [§4.1–4.2, §5, Abstract] Implementation metrics (TCS, SCR, EC, CSC, SC/CVR, EAR) and Validation metrics (DNR, SRR, OC, LC) are load-bearing for the three-stage story and for the claim that aggregate breaches trigger a targeted KG Update, yet none are exercised empirically; only usage metrics appear in the PoC. §5.2 leaves extraction metrics and the remediation map to future work. At minimum, either (i) add a small document-grounded extraction experiment showing TCS/SCR/DNR are computable and co-vary with downstream ERC/RRS under controlled extraction faults, or (ii) clearly demote Implementation/Validation instrumentation to “specified but unvalidated” in the abstract and contributions so the paper’s evidence matches its claims.
minor comments (6)
  1. [Table 2, §4] Table 2 and §4 introduce a large metric suite; a short “minimal monitoring set” box (already hinted at the end of §3) would help practitioners. Consider moving derived aggregates (TRIAGE-TI, TFI) and black-box KGC rows to an appendix to keep the spine readable.
  2. [§4.3.2, Eq. (28)] Notation: γ_q vs γ_a and M_q vs M_a are clear in Table 1, but RRS’s dual definition for |M_q|=1 vs ≥2 (§4.3.2, Eq. 28) could use a one-line example next to the formula.
  3. [Fig. 4] Fig. 4’s worked example is helpful but uses illustrative values; label it explicitly as synthetic so readers do not treat TCS≈0.94 / DNR=0.18 as measured results.
  4. [§4, §5] Several free parameters (θ on chain links, ω_i for TRIAGE-TI, τ for TCS, δ for SRR) are acknowledged but not defaulted. A short “default settings for first deployment” paragraph would improve reproducibility of the protocol.
  5. [§4.3.3, Eq. (32)] ARF’s lexical p⊆A test (§4.3.3, Eq. 32) is a known limitation; the future-work NLI replacement is noted, but a brief sensitivity note (paraphrase failure rate on a few examples) would set expectations.
  6. [References, Fig. 4] Minor typos/consistency: “Chris Nolan” / “Christopher Nolan” in Fig. 4 is intentional; ensure arXiv id and venue placeholders are cleaned for camera-ready. References include several 2025–2026 arXiv items—fine for a fast-moving area, but double-check stable citations where available.

Circularity Check

1 steps flagged

No load-bearing circular derivation: gold-free metrics are defined on distinct pipeline artifacts; the diagnostic chain is a design hypothesis with one mild definitional ordering (ERC≤QGR), not a forced prediction.

specific steps
  1. self definitional [§4.3.2 Entity Retrieval Coverage; §4.5 diagnostic chain / Fig. 5]
    "Sharing the denominator |M_q| with QGR gives ERC≤QGR always; the gap QGR−ERC is exactly the retrieval failure (grounded entities not fetched). ... For the first two links this is exact, ERC≤QGR by construction; for the later links it is a logical precondition rather than a numeric bound. This ordering is what makes the chain diagnostic: reading the metrics in sequence, the first link that falls below its threshold θ localizes the failure to a single condition."

    ERC and QGR share the same mention denominator by definition, so ERC≤QGR is guaranteed rather than discovered. The paper treats this built-in ordering as part of what makes the chain diagnostic (first broken link = retrieval gap when QGR high and ERC low). This is mild: both metrics remain independently interpretable on different numerators, and the paper is transparent about the construction. It does not force the later links (RRS, ARF, AGR) or the remediation map.

full rationale

TRIAGE is a stage-aware instrumentation framework, not a first-principles derivation. Deployed metrics (TCS, SCR, DNR, QGR, ERC, RRS, AGR, etc.) are defined on extraction logs, the assembled graph, or the retrieved subgraph without gold answers; gold metrics (COR, CMP, GPC, CC, Hits@1) are explicitly offline calibration. The PoC association of ERC/RRS with Context Coverage is not tautological: ERC scores grounded query mentions while CC scores gold-answer entities, and the paper states this disjointness. TRIAGE-TI and TFI are openly weighted aggregates of other metrics (definitional complements, not independent predictions). The only mild circularity-adjacent feature is the definitional inequality ERC≤QGR (shared denominator |M_q|), which the paper uses as a diagnostic property of the chain rather than as a claimed empirical discovery. The Youden threshold θ_ERC=0.67 is fit in-sample but labeled descriptive, not a validated out-of-sample prediction. TKG [2] is cited for imported COR/CMP and lifecycle structure; author sets do not overlap with TRIAGE, so this is not self-citation load-bearing. The localization/remediation map is under-supported empirically (deferred to §5.2), which is weak evidence rather than circular reduction of outputs to inputs. Score 1 for the single definitional ordering used diagnostically; no step reduces a central claim to its own fitted inputs by construction.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 3 invented entities

The central claim rests on standard IR/KG measurement ideas plus domain assumptions about Graph-RAG pipelines and a small set of free thresholds/weights. Invented entities are the framework and its named metrics; independent evidence for the diagnostic claim is only the limited PoC.

free parameters (5)
  • Decision thresholds θ on QGR/ERC/RRS/ARF/AGR
    Chain localization depends on thresholds separating low vs adequate; PoC fits θ_ERC=0.67 in-sample via Youden; general θ left to empirical estimation (§4.5, §5.1).
  • TRIAGE-TI component weights ω_i
    Optional aggregate trustworthiness index is a weighted sum of TCS/SCR/EC/CSC/SC/CSR_link with application-chosen weights summing to 1 (§4.1.2).
  • TFI component weights λ_i
    Benchmark-only fidelity aggregate weights cosine, Soft-F1, and EntRatio (§4.1.1).
  • TCS confidence threshold τ
    SCR and flagging of low-confidence triples depend on a chosen τ (§4.1.2).
  • SRR similarity threshold δ
    Semantic redundancy depends on component-similarity cutoff δ (§4.2).
axioms (5)
  • domain assumption A correct graph-grounded answer requires the ordered necessary conditions QGR, ERC, RRS, ARF, AGR in sequence.
    Load-bearing for the diagnostic chain; stated as necessary proxies, not guarantees (§4.5).
  • domain assumption LLM token log-probabilities are a usable intrinsic proxy for extraction confidence (TCS).
    TCS definition; paper notes imperfect calibration for factual correctness (§4.1.2, Limitations).
  • domain assumption TKG phase structure (implementation/validation/usage) and effectiveness metrics (COR, CMP, LC) are appropriate to adapt to automated Graph-RAG.
    Framework is informed by TKG [2] while making independent design choices (§1, §3).
  • standard math Standard rank-based KGC metrics and Soft-F1/NLI grounding are valid black-box or fidelity probes.
    Imported definitions from link-prediction and IE evaluation literature (§4.1).
  • domain assumption Entity mentions can be extracted and grounded reliably enough that QGR/ERC/RRS/AGR remain meaningful.
    Usage metrics depend on M_q, M_a and grounding γ; paper lists propagation of extraction errors as a limitation (§4.3, Limitations).
invented entities (3)
  • TRIAGE three-stage instrumentation framework and remediation map no independent evidence
    purpose: Organize gold-free stage metrics into a deployable diagnostic and update loop for automated Graph-RAG.
    Core contribution; independent evidence limited to PoC + protocol.
  • Usage diagnostic chain signatures (out-of-scope, retrieval gap, connectivity gap, relational gap, generation gap) no independent evidence
    purpose: Localize first broken necessary condition at inference.
    Defined in §4.5; not yet shown separable at scale.
  • Named gold-free metrics (TCS, SCR, EC, CSC, SC, CVR, CSR_link, EAR, OC, DNR, SRR, QGR, ERC, ERP, RRS, RPC, AGR, AUR, ARF, RHD, etc.) no independent evidence
    purpose: Stage-localized trust and cost signals without gold at deployment.
    Many instantiate standard ideas; joint suite is paper-specific.

pith-pipeline@v1.1.0-grok45 · 38313 in / 3541 out tokens · 29514 ms · 2026-07-12T02:23:56.371901+00:00 · methodology

0 comments
read the original abstract

Knowledge graphs (KGs) that underpin Graph-based Retrieval-Augmented Generation (Graph-RAG) are increasingly built automatically by LLM-driven extraction rather than curated by experts. Proper evaluation would require instrumenting all pertinent stages: extraction, graph construction, and inference, coherently enough to localize failures, so that a failure at one stage is not discovered as a wrong answer at the end. We introduce TRIAGE, a stage-aware instrumentation framework for automated, document-grounded graph-RAG that asks not only whether the underlying graph can be trusted but at what cost it can be queried. TRIAGE attaches stage-specific, independently interpretable metrics to three stages: the KG Implementation (triple confidence, source coverage, and schema and canonicalization checks), the KG Validation by expert (graph-level structural quality, with correctness and completeness computed only as offline calibration when a reference is available), and the KG Usage (retrieval coverage, faithfulness, and retrieval cost); the deployed metrics need no gold annotations, the gold-requiring ones serving only as offline calibration. At usage time these metrics form a diagnostic chain of necessary conditions whose first broken link localizes the failure, and the diagnosis maps to the stage levers that can remedy it: extraction, graph and schema, or retrieval. TRIAGE is a theoretical framework with a proof of concept and a reproducible evaluation protocol.

Figures

Figures reproduced from arXiv: 2607.03447 by Axel TahmasebiMoradi, Lucas Schott, Martin Royer.

Figure 1
Figure 1. Figure 1: The TRIAGE framework organized into three phases. Each phase pro￾duces distinct knowledge artifacts together with stage-specific confidence metrics (gray). Low aggregate metric values may trigger a new KG Update cycle; the stage of the failing metric points to the levers that can remedy it (section 4.5). metrics), and KG Usage (graph-based inference). Each phase produces both a knowledge artifact and a set… view at source ↗
Figure 2
Figure 2. Figure 2: White-box instrumentation blueprint of the [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: White-box instrumentation blueprint of the [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: A single query threaded through the three [PITH_FULL_IMAGE:figures/full_fig_p016_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The usage-stage diagnostic chain as a decision cascade. Reading the met [PITH_FULL_IMAGE:figures/full_fig_p035_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

71 extracted references · 3 canonical work pages

  1. [1]

    arXiv preprint arXiv:2509.19209 (2025)

    Akindele, O.K., Mishra, B.K., Wertheim, K.Y.: A knowledge graph and a tripartite evaluation framework make retrieval-augmented generation scalable and transpar- ent. arXiv preprint arXiv:2509.19209 (2025)

    arXiv 2025

  2. [2]

    In: 16th International Conference on Performance, Safety and Robustness in Complex Systems and Applications (PESARO) (2026)

    Amdouni, E., Mattioli, L., Adjed, F., Awadid, A., Gonzalez, M., Cantat, L., Mat- tioli, J.: An end-to-end trustworthy knowledge graph engineering methodology. In: 16th International Conference on Performance, Safety and Robustness in Complex Systems and Applications (PESARO) (2026)

    2026

  3. [3]

    In: Proceedings of the 20th International Joint Conference on Artificial Intelligence (IJCAI)

    Banko, M., Cafarella, M.J., Soderland, S., Broadhead, M., Etzioni, O.: Open in- formation extraction from the web. In: Proceedings of the 20th International Joint Conference on Artificial Intelligence (IJCAI). pp. 2670–2676 (2007)

    2007

  4. [4]

    Journal of Machine Learning Research3, 1137–1155 (2003),http://www

    Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C.: A neural probabilistic language model. Journal of Machine Learning Research3, 1137–1155 (2003),http://www. jmlr.org/papers/v3/bengio03a.html

    2003

  5. [5]

    Bhardwaj, S., Aggarwal, S., Mausam: CaRB: A crowdsourced benchmark for open IE. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). pp. 6262–6267. Association for Computational Linguistics, Hong Kong, China (2019).https://doi....

    doi:10.18653/v1/d19-1651 2019

  6. [6]

    Bordes, A., Usunier, N., Garcia-Duran, A., Weston, J., Yakhnenko, O.: Trans- lating embeddings for modeling multi-relational data. In: Advances in Neural Information Processing Systems (2013),https://papers.nips.cc/paper/2013/ hash/1cecc7a77928ca8133fa24680a88d2f9-Abstract.html, introduces the stan- dard link prediction evaluation by ranking corrupted hea...

    2013

  7. [7]

    In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing

    Bowman, S.R., Angeli, G., Potts, C., Manning, C.D.: A large annotated corpus for learning natural language inference. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. pp. 632–642. Association for Computational Linguistics, Lisbon, Portugal (2015).https://doi.org/10.18653/ v1/D15-1075,https://aclanthology.org/D15-1075/

    2015

  8. [8]

    In: Advances in Neural Information Processing Systems

    Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. In: Advances in Neural Information Processing Systems. vol. 33, pp. 1877–1901 (2020),https://proceedings.neurips.cc/paper/2020/ hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html

    1901

  9. [9]

    In: Pro- ceedings of the AAAI conference on artificial intelligence

    Chen, B., Guo, Z., Yang, Z., Chen, Y., Chen, J., Liu, Z., Shi, C., Yang, C.: Pathrag: Pruning graph-based retrieval augmented generation with relational paths. In: Pro- ceedings of the AAAI conference on artificial intelligence. vol. 40, pp. 30183–30191 (2026)

    2026

  10. [10]

    Educational and Psychological Measurement20(1), 37–46 (1960).https://doi.org/10.1177/ 001316446002000104

    Cohen, J.: A coefficient of agreement for nominal scales. Educational and Psychological Measurement20(1), 37–46 (1960).https://doi.org/10.1177/ 001316446002000104

    1960

  11. [11]

    In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL)

    Cui, L., Wei, F., Zhou, M.: Neural open information extraction. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL). pp. 407–413 (2018).https://doi.org/10.18653/v1/P18-2065

    work page doi:10.18653/v1/p18-2065 2018

  12. [12]

    In: Proceedings of the AAAI Conference on Artificial Intel- ligence

    Dettmers, T., Minervini, P., Stenetorp, P., Riedel, S.: Convolutional 2d knowledge graph embeddings. In: Proceedings of the AAAI Conference on Artificial Intel- ligence. vol. 32, pp. 1811–1818 (2018).https://doi.org/10.1609/aaai.v32i1. TRIAGE: Trustworthy Retrieval Instrumentation And Graph Evaluation 45 11573,https://ojs.aaai.org/index.php/AAAI/article/v...

    doi:10.1609/aaai.v32i1 2018

  13. [13]

    arXiv preprint arXiv:2404.16130 (2024)

    Edge, D., Trinh, H., Cheng, N., Bradley, J., Chao, A., Mody, A., Truitt, S., Metropolitansky, D., Ness, R.O., Larson, J.: From local to global: A graph RAG approach to query-focused summarization. arXiv preprint arXiv:2404.16130 (2024)

    Pith/arXiv arXiv 2024

  14. [14]

    Journal of the American Statis- tical Association82(397), 171–185 (1987).https://doi.org/10.1080/01621459

    Efron, B.: Better bootstrap confidence intervals. Journal of the American Statis- tical Association82(397), 171–185 (1987).https://doi.org/10.1080/01621459. 1987.10478410,https://www.jstor.org/stable/2289144

    doi:10.1080/01621459 1987

  15. [15]

    Efron, B., Tibshirani, R.J.: An Introduction to the Bootstrap. Chapman and Hall/CRC, New York, NY, USA (1994).https://doi.org/10.1201/ 9780429246593,https://www.taylorfrancis.com/books/mono/10.1201/ 9780429246593/introduction-bootstrap-bradley-efron-tibshirani

    1994

  16. [16]

    arXiv preprint arXiv:2604.09666 (2026)

    Fan, D., Xue, Z., Liu, S., Tan, Q.: Do we still need graphrag? benchmarking rag and graphrag for agentic search systems. arXiv preprint arXiv:2604.09666 (2026)

    Pith/arXiv arXiv 2026

  17. [17]

    Semantic Web9(1), 77–129 (2018).https://doi.org/10.3233/SW-170275

    F¨ arber, M., Bartscherer, F., Menne, C., Rettinger, A.: Linked data quality of DB- pedia, Freebase, OpenCyc, Wikidata, and YAGO. Semantic Web9(1), 77–129 (2018).https://doi.org/10.3233/SW-170275

    work page doi:10.3233/sw-170275 2018

  18. [18]

    In: Producing Artificial Intelligent Systems, pp

    Gelin, R.: Confiance.ai program: Software engineering for a trustworthy AI. In: Producing Artificial Intelligent Systems, pp. 11–29. Studies in Computational In- telligence, Springer (2024).https://doi.org/10.1007/978-3-031-55817-7_2

    work page doi:10.1007/978-3-031-55817-7_2 2024

  19. [19]

    In: Proceedings of the 34th International Conference on Machine Learn- ing (ICML)

    Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On calibration of modern neural networks. In: Proceedings of the 34th International Conference on Machine Learn- ing (ICML). vol. 70, pp. 1321–1330 (2017),https://arxiv.org/abs/1706.04599

    Pith/arXiv arXiv 2017

  20. [20]

    arXiv preprint arXiv:2410.057792(3) (2024)

    Guo, Z., Xia, L., Yu, Y., Ao, T., Huang, C.: Lightrag: Simple and fast retrieval- augmented generation. arXiv preprint arXiv:2410.057792(3) (2024)

    Pith/arXiv arXiv 2024

  21. [21]

    graphrag: A systematic evaluation and key insights

    Han, H., Ma, L., Wang, Y., Shomer, H., Lei, Y., Qi, Z., Guo, K., Hua, Z., Long, B., Liu, H., et al.: Rag vs. graphrag: A systematic evaluation and key insights. arXiv preprint arXiv:2502.11371 (2025)

    arXiv 2025

  22. [22]

    High-Level Expert Group on Artificial Intelligence (AI HLEG): Ethics guidelines for trustworthy AI. Tech. rep., European Commission, Brus- sels (2019),https://digital-strategy.ec.europa.eu/en/library/ ethics-guidelines-trustworthy-ai

    2019

  23. [23]

    ACM Computing Surveys54(4), 1–37 (2021).https://doi.org/10.1145/3447772

    Hogan, A., Blomqvist, E., Cochez, M., d’Amato, C., de Melo, G., Gutierrez, C., Kirrane, S., Gayo, J.E.L., Navigli, R., Neumaier, S., et al.: Knowledge graphs. ACM Computing Surveys54(4), 1–37 (2021).https://doi.org/10.1145/3447772

    doi:10.1145/3447772 2021

  24. [24]

    In: Proceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies

    Honovich, O., Aharoni, R., Herzig, J., Taitelbaum, H., Kukliansy, D., Cohen, V., Scialom, T., Szpektor, I., Hassidim, A., Matias, Y.: TRUE: Re-evaluating factual consistency evaluation. In: Proceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 3905–3920. Associ...

    doi:10.18653/v1/2022.naacl-main.287 2022

  25. [25]

    Hoyt, C.T., Berrendorf, M., Galkin, M., Tresp, V., Gyori, B.M.: A unified frame- work for rank-based evaluation metrics for link prediction in knowledge graphs. arXiv preprint (2022),https://arxiv.org/pdf/2203.07544, surveys and formal- izes rank-based metrics for KG link prediction, motivating their use in the absence of explicit negatives

    Pith/arXiv arXiv 2022

  26. [26]

    Jaccard, P.: ´Etude comparative de la distribution florale dans une portion des alpes et du jura. Bulletin de la Soci´ et´ e Vaudoise des Sciences Naturelles37, 547–579 46 Axel TahmasebiMoradi, Lucas Schott, Martin Royer (1901),https://commons.wikimedia.org/wiki/File:%C3%89tude_comparative_ de_la_distribution_florale_dans_une_portion_des_Alpes_et_du_Jura.pdf

    1901

  27. [27]

    In: Proceedings of the Text Analysis Conference (TAC) (2014),http: //blender.cs.illinois.edu/paper/edl2014overview.pdf

    Ji, H., Nothman, J., Hachey, B.: Overview of TAC-KBP2014 entity discovery and linking tasks. In: Proceedings of the Text Analysis Conference (TAC) (2014),http: //blender.cs.illinois.edu/paper/edl2014overview.pdf

    2014

  28. [28]

    ACM Computing Surveys55(12), 1–38 (2023).https://doi.org/10

    Ji, Z., Lee, N., Frieske, R., et al.: Survey of hallucination in natural language generation. ACM Computing Surveys55(12), 1–38 (2023).https://doi.org/10. 1145/3571730

    2023

  29. [29]

    Jiang, Z., Araki, J., Ding, H., Neubig, G.: How can we know when lan- guage models know? on the calibration of language models for ques- tion answering. Transactions of the Association for Computational Lin- guistics9, 962–977 (2021).https://doi.org/10.1162/tacl_a_00407, https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00407/107277/ How-Can-We-Know-...

    doi:10.1162/tacl_a_00407 2021

  30. [30]

    05221,https://arxiv.org/abs/2207.05221

    Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., et al.: Language models (mostly) know what they know (2022).https://doi.org/10.48550/arXiv.2207. 05221,https://arxiv.org/abs/2207.05221

    Pith/arXiv arXiv doi:10.48550/arxiv.2207 2022

  31. [31]

    SAGE Publications, Thousand Oaks, CA, 2 edn

    Krippendorff, K.: Content Analysis: An Introduction to Its Methodology. SAGE Publications, Thousand Oaks, CA, 2 edn. (2004)

    2004

  32. [32]

    Naval Re- search Logistics Quarterly2(1–2), 83–97 (1955).https://doi.org/10.1002/nav

    Kuhn, H.W.: The hungarian method for the assignment problem. Naval Re- search Logistics Quarterly2(1–2), 83–97 (1955).https://doi.org/10.1002/nav. 3800020109

    doi:10.1002/nav 1955

  33. [33]

    arXiv preprint arXiv:2603.20252 (2026)

    Kumar, M., Sarmah, B., Pasquali, S.: Finreflectkg–hallubench: Graphrag hal- lucination benchmark for financial question answering systems. arXiv preprint arXiv:2603.20252 (2026)

    arXiv 2026

  34. [34]

    In: Proceedings of the 28th Interna- tional Conference on Information Fusion (FUSION) (2025)

    Laudy, C., Alonso, V., Reverdy, C., Dreo, J.: First high-level information fusion competition: Feedback and lessons learned. In: Proceedings of the 28th Interna- tional Conference on Information Fusion (FUSION) (2025)

    2025

  35. [35]

    In: Advances in Neural Information Processing Systems

    Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., K¨ uttler, H., Lewis, M., Yih, W.t., Rockt¨ aschel, T., Riedel, S., Kiela, D.: Retrieval-augmented generation for knowledge-intensive NLP tasks. In: Advances in Neural Information Processing Systems. vol. 33, pp. 9459–9474 (2020)

    2020

  36. [36]

    arXiv preprint arXiv:2509.26383 (2025)

    Lin, J., Liu, S., Song, J., Wang, S., Shun, J., Zhu, Y.: Efficient and transfer- able agentic knowledge graph RAG via reinforcement learning. arXiv preprint arXiv:2509.26383 (2025)

    Pith/arXiv arXiv 2025

  37. [37]

    arXiv preprint arXiv:2309.03433 (2023)

    Ling, C., Zhao, X., Zhang, X., Cheng, W., Liu, Y., Sun, Y., Oishi, M., Osaki, T., Matsuda, K., Chen, J., Bo, X., Zhang, T., et al.: Improving open information extraction with large language models: A study on demonstration uncertainty. arXiv preprint arXiv:2309.03433 (2023)

    Pith/arXiv arXiv 2023

  38. [38]

    arXiv preprint arXiv:2502.12442 (2025)

    Liu, H., et al.: HopRAG: Multi-hop reasoning for logic-aware retrieval-augmented generation. arXiv preprint arXiv:2502.12442 (2025)

    Pith/arXiv arXiv 2025

  39. [39]

    arXiv preprint arXiv:2410.13080 (2024)

    Luo, L., Zhao, Z., Haffari, G., Li, Y.F., Gong, C., Pan, S.: Graph-constrained reasoning: Faithful reasoning on knowledge graphs with large language models. arXiv preprint arXiv:2410.13080 (2024)

    Pith/arXiv arXiv 2024

  40. [40]

    Proceedings of the AAAI Symposium Series7(1), 206–213 (2025).https://doi.org/10.1609/ aaaiss.v7i1.36888

    Mattioli, J., Mattioli, L., Gonzalez, M.: A brief overview of key quality metrics for knowledge graph solution: Illustration on digital NOTAMs. Proceedings of the AAAI Symposium Series7(1), 206–213 (2025).https://doi.org/10.1609/ aaaiss.v7i1.36888

    2025

  41. [41]

    arXiv preprint arXiv:2405.20139 (2024) TRIAGE: Trustworthy Retrieval Instrumentation And Graph Evaluation 47

    Mavromatis, C., Karypis, G.: Gnn-rag: Graph neural retrieval for large language model reasoning. arXiv preprint arXiv:2405.20139 (2024) TRIAGE: Trustworthy Retrieval Instrumentation And Graph Evaluation 47

    Pith/arXiv arXiv 2024

  42. [42]

    In: Proceedings of ACL (2020)

    Maynez, J., Narayan, S., Bohnet, B., McDonald, R.: On faithfulness and factuality in abstractive summarization. In: Proceedings of ACL (2020)

    2020

  43. [43]

    Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., Zettlemoyer, L.: Rethinking the role of demonstrations: What makes in-context learning work? In: Proceedings of EMNLP (2022)

    2022

  44. [44]

    Proceedings of the IEEE104(1), 11–33 (2015)

    Nickel, M., Murphy, K., Tresp, V., Gabrilovich, E.: A review of relational machine learning for knowledge graphs. Proceedings of the IEEE104(1), 11–33 (2015). https://doi.org/10.1109/JPROC.2015.2483592

    doi:10.1109/jproc.2015.2483592 2015

  45. [45]

    OECD Legal In- struments, OECD/LEGAL/0449 (2019),https://legalinstruments.oecd.org/ en/instruments/OECD-LEGAL-0449

    OECD: Recommendation of the council on artificial intelligence. OECD Legal In- struments, OECD/LEGAL/0449 (2019),https://legalinstruments.oecd.org/ en/instruments/OECD-LEGAL-0449

    2019

  46. [46]

    Semantic Web8(3), 489–508 (2017).https://doi.org/10.3233/ SW-160218

    Paulheim, H.: Knowledge graph refinement: A survey of approaches and evalua- tion methods. Semantic Web8(3), 489–508 (2017).https://doi.org/10.3233/ SW-160218

    2017

  47. [47]

    Reimers, N., Gurevych, I.: Sentence-bert: Sentence embeddings using siamese bert- networks. In: Proceedings of the 2019 Conference on Empirical Methods in Nat- ural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). pp. 3980–3990. Association for Compu- tational Linguistics, Hong Kong, China (2019...

    doi:10.18653/v1/ 2019

  48. [48]

    In: International Conference on Learning Representations (ICLR) (2020),https://openreview.net/forum?id=BkxSmlBFvr

    Ruffinelli, D., Broscheit, S., Gemulla, R.: You can teach an old dog new tricks! on training knowledge graph embeddings. In: International Conference on Learning Representations (ICLR) (2020),https://openreview.net/forum?id=BkxSmlBFvr

    2020

  49. [49]

    McGraw- Hill, New York, NY, USA (1983)

    Salton, G., McGill, M.J.: Introduction to Modern Information Retrieval. McGraw- Hill, New York, NY, USA (1983)

    1983

  50. [50]

    Shen, W., Wang, J., Han, J.: Entity linking with a knowledge base: Issues, techniques, and solutions. IEEE Transactions on Knowledge and Data Engi- neering27(2), 443–460 (2015).https://doi.org/10.1109/TKDE.2014.2327028, https://www.bibsonomy.org/bibtex/27bedb76858c3a0d0094aa8f9392ce270

    doi:10.1109/tkde.2014.2327028 2015

  51. [51]

    In: Proceedings of the 2016 Conference on Empirical Methods in Nat- ural Language Processing

    Stanovsky, G., Dagan, I.: Creating a large benchmark for open information ex- traction. In: Proceedings of the 2016 Conference on Empirical Methods in Nat- ural Language Processing. pp. 2300–2305. Association for Computational Lin- guistics, Austin, Texas (2016).https://doi.org/10.18653/v1/D16-1252,https: //aclanthology.org/D16-1252/

    doi:10.18653/v1/d16-1252 2016

  52. [52]

    Sun, Z., Deng, Z.H., Nie, J.Y., Tang, J.: Rotate: Knowledge graph embedding by relational rotation in complex space. In: International Conference on Learning Representations (ICLR) (2019),https://openreview.net/forum?id=HkgEQnRqYQ, a standard KGE paper evaluating link prediction with rank-based metrics such as MRR and Hits@K

    2019

  53. [53]

    Tabassi, E.: Artificial intelligence risk management framework (AI RMF 1.0). Tech. Rep. NIST AI 100-1, National Institute of Standards and Technology (2023). https://doi.org/10.6028/NIST.AI.100-1

    doi:10.6028/nist.ai.100-1 2023

  54. [54]

    In: Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality (CVSC)

    Toutanova, K., Chen, D.: Observed versus latent features for knowledge base and text inference. In: Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality (CVSC). pp. 57–66. Association for Computational Linguistics, Beijing, China (2015).https://doi.org/10.18653/ v1/W15-4007,https://aclanthology.org/W15-4007/

    2015

  55. [55]

    In: Proceedings of the 33rd International Conference on Machine Learning (ICML)

    Trouillon, T., Welbl, J., Riedel, S., Gaussier, E., Bouchard, G.: Complex em- beddings for simple link prediction. In: Proceedings of the 33rd International Conference on Machine Learning (ICML). Proceedings of Machine Learning Re- search, vol. 48, pp. 2071–2080. PMLR (2016),https://proceedings.mlr.press/ 48 Axel TahmasebiMoradi, Lucas Schott, Martin Roye...

    2071

  56. [56]

    arXiv preprint arXiv:2510.15339 (2025)

    Tsang, H.T., Bai, J., Huang, H., Xiao, Q., Zheng, T., Xu, B., Liu, S., Song, Y.: AutoGraph-R1: End-to-end reinforcement learning for knowledge graph construc- tion. arXiv preprint arXiv:2510.15339 (2025)

    Pith/arXiv arXiv 2025

  57. [57]

    arXiv preprint arXiv:2602.02053 (2026)

    Wang, P., Xu, B., Zhang, L., Wang, S., Du, M., Zhu, C., Mao, Z.: Wildgraphbench: Benchmarking graphrag with wild-source corpora. arXiv preprint arXiv:2602.02053 (2026)

    arXiv 2026

  58. [58]

    In: International Conference on Learning Representations (ICLR) (2025), arXiv:2503.22166

    Wang, S., et al.: Reasoning of large language models over knowledge graphs with super-relations. In: International Conference on Learning Representations (ICLR) (2025), arXiv:2503.22166

    Pith/arXiv arXiv 2025

  59. [59]

    In: International Conference on Learning Representations (ICLR) (2023).https: //doi.org/10.48550/arXiv.2203.11171,https://arxiv.org/abs/2203.11171

    Wang, X., Wei, J., Schuurmans, D., Le, Q.V., Chi, E.H., Narang, S., Chowdhery, A., Zhou, D.: Self-consistency improves chain of thought reasoning in language models. In: International Conference on Learning Representations (ICLR) (2023).https: //doi.org/10.48550/arXiv.2203.11171,https://arxiv.org/abs/2203.11171

    Pith/arXiv arXiv doi:10.48550/arxiv.2203.11171 2023

  60. [60]

    In: Advances in Neural Information Processing Systems (2022)

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., et al.: Chain-of-thought prompting elicits reasoning in large language models. In: Advances in Neural Information Processing Systems (2022)

    2022

  61. [61]

    In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Technologies, Volume 1 (Long Papers)

    Williams, A., Nangia, N., Bowman, S.: A broad-coverage challenge corpus for sen- tence understanding through inference. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Technologies, Volume 1 (Long Papers). pp. 1112–1122. As- sociation for Computational Linguistics, New...

    doi:10.18653/v1/n18-1101 2018

  62. [62]

    arXiv preprint arXiv:2506.05690 (2025)

    Xiang, Z., Wu, C., Zhang, Q., Chen, S., Hong, Z., Huang, X., Su, J.: When to use graphs in rag: A comprehensive analysis for graph retrieval-augmented generation. arXiv preprint arXiv:2506.05690 (2025)

    arXiv 2025

  63. [63]

    In: International Conference on Learning Representations (2023)

    Yao, S., Zhao, J., Yu, D., et al.: React: Synergizing reasoning and acting in language models. In: International Conference on Learning Representations (2023)

    2023

  64. [64]

    In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL)

    Yih, W.t., Richardson, M., Meek, C., Chang, M.W., Suh, J.: The value of semantic parse labeling for knowledge base question answering. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL). pp. 201– 206 (2016).https://doi.org/10.18653/v1/P16-2033

    doi:10.18653/v1/p16-2033 2016

  65. [65]

    arXiv preprint arXiv:2603.14045 (2026)

    Zarrinkia, Y., Srinivasan, V., Thomo, A.: The reasoning bottleneck in graph-rag: Structured prompting and context compression for multi-hop qa. arXiv preprint arXiv:2603.14045 (2026)

    arXiv 2026

  66. [66]

    Semantic Web7(1), 63–93 (2016).https: //doi.org/10.3233/SW-150175

    Zaveri, A., Rula, A., Maurino, A., Pietrobon, R., Lehmann, J., Auer, S.: Quality assessment for linked data: A survey. Semantic Web7(1), 63–93 (2016).https: //doi.org/10.3233/SW-150175

    doi:10.3233/sw-150175 2016

  67. [67]

    arXiv preprint arXiv:2504.09823 (2025)

    Zhang, H., Si, J., Yan, G., Qi, B., Cai, P., Mao, S., Wang, D., Shi, B.: RAKG: Document-level retrieval augmented knowledge graph construction. arXiv preprint arXiv:2504.09823 (2025)

    Pith/arXiv arXiv 2025

  68. [68]

    In: Proceedings of the AAAI conference on artificial intelligence

    Zhang, Y., Dai, H., Kozareva, Z., Smola, A., Song, L.: Variational reasoning for question answering with knowledge graph. In: Proceedings of the AAAI conference on artificial intelligence. vol. 32 (2018)

    2018

  69. [69]

    In: Proceedings of ICML (2021)

    Zhao, T., Wallace, E., Feng, S., Klein, D., Singh, S.: Calibrate before use: Improving few-shot performance of language models. In: Proceedings of ICML (2021)

    2021

  70. [70]

    arXiv preprint arXiv:2508.08344 (2025) TRIAGE: Trustworthy Retrieval Instrumentation And Graph Evaluation 49

    Zhou, D., Zhu, Y., Wang, X., Zhou, H., He, Y., Chen, J., Staab, S., Kharlamov, E.: What breaks knowledge graph based rag? benchmarking and empirical insights into reasoning under incomplete knowledge. arXiv preprint arXiv:2508.08344 (2025) TRIAGE: Trustworthy Retrieval Instrumentation And Graph Evaluation 49

    arXiv 2025

  71. [71]

    In: Proceedings of the 27th international conference on computational linguistics

    Zhou, M., Huang, M., Zhu, X.: An interpretable reasoning network for multi- relation question answering. In: Proceedings of the 27th international conference on computational linguistics. pp. 2010–2022 (2018)

    2010