Pith. sign in

REVIEW 3 major objections 4 minor 36 references

SIRIN claims contextual hallucination detection is an infrastructure problem, unifying probing, judge-style, and uncertainty detectors under one interface, and shows the combination can gate memory-grounded LLMs.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 01:35 UTC pith:7YFUICPF

load-bearing objection A genuinely useful unification toolkit; the memory-gate demo numbers should be treated as preliminary until re-estimated with held-out thresholds and independent labels. the 3 major comments →

arxiv 2608.00033 v1 pith:7YFUICPF submitted 2026-07-20 cs.AI

SIRIN: A Unified Toolkit for Detecting Contextual Hallucinations in Retrieval-Augmented and Memory-Grounded LLM Systems

Julia Belikova , Rauf Parchiev , Mikhail Filimonov , Konstantin Polev , Andrey Savchenko , Maksim Makarenko This is my paper
classification cs.AI
keywords contextual hallucinationretrieval-augmented generationhallucination detectionrepresentation probinguncertainty estimationjudge-style verificationquery answerabilityselective prediction
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Contextual hallucinations — fluent answers unsupported by the given evidence — plague retrieval-augmented and memory-grounded LLM systems, but the detectors built to catch them are scattered across toolkits that each assume different data formats, model access, and output granularity. SIRIN claims this is an infrastructure problem: one shared report schema, configuration system, and evaluation pipeline can host all three detector families (representation probing, uncertainty estimation, judge-style verification) plus the pre-generation check of query answerability, at response and span level, in white-box and black-box settings. The paper backs this with evaluations on hallucination detection, answerability, and span localization, and by wiring the detectors as gates around a long-term memory agent: combining an answerability gate with a faithfulness gate raises served accuracy from 62.4% to 79.3% and cuts strict unfaithfulness from 20.2% to 9.8%. If the claim holds, teams can compare, combine, and deploy detectors without rewriting application code, and can inspect exactly which span of an answer is suspect.

Core claim

On its own terms, the paper's central discovery is that a single abstraction — an inspection report containing a calibrated probability, a predicted label, and optional evidence such as span scores — is expressive enough to unify probing, judge, and uncertainty detectors, and that this abstraction lets the same detectors serve both runtime gating and interactive span-level inspection. The experiments show that lightweight probes over internal states are annotation-efficient and competitive on ranking quality, that a fine-tuned judge is strongest when labeled data is plentiful, and that unsupervised uncertainty estimators are weaker but require no labels. The memory-gating experiment conclude

What carries the argument

The load-bearing abstraction is the inspection report Dθ(x) = (s, ŷ, E): a calibrated score, a binary or ternary label, and optional evidence like span scores or judge rationales. Every detector, from a hidden-state probe to an API judge, emits this same schema, which is what allows comparison, ensembling, and drop-in gating. The pipeline further separates where to look from what to extract via token locators and feature processors, and uses a splitter/aggregator so that response-level methods can produce span-level evidence. A composing processor extracts multiple feature families from a single forward pass, keeping probing cost close to one model pass.

Load-bearing premise

The headline gating gains rest on a protocol where the coverage threshold is chosen as a within-evaluation rank cut and the strict-unfaithfulness column in the main table is the probe's own training label; if that protocol inflates the numbers, the claimed lift from 62.4% to 79.3% served accuracy is not established.

What would settle it

Re-run the memory-gating experiment (on the same long-term-memory benchmark) with thresholds fit only on a training split and faithfulness scored by an independent judge model; if served accuracy and strict-unfaithfulness no longer show the reported gains, the trust-layer claim collapses.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Any new detector can enter as a feature processor, detector, or aggregator and immediately work across white-box and black-box backends at both response and span granularity.
  • Hallucination detection becomes a selective-prediction layer: a system can serve, retry, or abstain based on the two gates, without modifying the underlying memory or generator.
  • The strong answerability results suggest that query answerability can be checked as textual entailment, a simpler pre-generation filter than generation-based scoring.
  • Annotation budget determines the right detector family: probes win when labels are scarce, judges win when supervision is abundant, and uncertainty methods offer a label-free fallback.
  • Fusing many confidence estimators adds little over the best individual estimator, indicating a ceiling for current uncertainty signals.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural extension of the unification claim is a public leaderboard where every detector family runs through the same pipeline on shared benchmarks, which would let practitioners see which family transfers across domains and generators.
  • Because the headline memory-gating numbers use a within-evaluation threshold cut and a faithfulness label that is the probe's own training label, a direct test of robustness is to fix thresholds on a training split and score faithfulness with an independent judge; if the lift persists, the trust-layer claim is solid.
  • The one-pass multi-signal extraction could generalize beyond hallucination detection to other quality dimensions, such as factual consistency or safety, by reusing the same report schema with different probes.
  • Finding that answerability aligns with entailment hints that retrieval quality itself could be reframed as a single entailment check, potentially simplifying RAG evaluation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents SIRIN, an open-source toolkit that unifies three families of contextual hallucination detectors (representation probing, uncertainty estimation, and judge-style verification) plus query answerability under a single interface, configuration system, and evaluation pipeline. It also provides a Streamlit web UI for live inspection with span highlighting. The authors evaluate SIRIN on RAGTruth, SQuAD 2.0, PsiloQA, and as a selective-prediction memory gate on LongMemEval with SimpleMem, Mem0, and LightMem. The headline result is that combining answerability and faithfulness gates raises served accuracy on SimpleMem from 62.4% to 79.3% and reduces strict unfaithfulness from 20.2% to 9.8%. The paper is transparent about methodological caveats in the appendices.

Significance. If the toolkit performs as described, this is a valuable infrastructure contribution. SIRIN is, to my knowledge, the first open-source system to bring probing, judge, uncertainty, and answerability into a single shared report schema with plug-in extension points, white-box/black-box backends, and span-level inspection. The public code and reproducible configuration system are strengths, as is the authors' explicit disclosure of the transductive threshold and training-label circularity in the memory-gate evaluation. The independent Claude Sonnet 5 labels used for Mem0 and LightMem are a good-faith attempt to address the circularity, though the SimpleMem headline lacks such validation. These strengths make the toolkit claim credible, but the memory-gate demonstration needs revision before the "effective trust layer" conclusion can be accepted.

major comments (3)
  1. [§4, Table 5, Appendix E] The coverage threshold for the memory gates is a transductive within-evaluation rank cut, as the paper itself states in Appendix E: "The coverage threshold, as in Table 5, is a transductive within-evaluation rank cut." Setting the abstention threshold to achieve exactly 80% coverage on the evaluation set means the reported served accuracies (e.g., 62.4% to 79.3% on SimpleMem) are in-sample estimates; under distribution shift in deployment, the threshold would not necessarily yield 80% coverage or the same accuracy. This compromises the §4 claim that SIRIN is an effective trust layer. Please re-select thresholds on a validation split or report coverage-accuracy curves over the full ranking.
  2. [§4, Table 5, Appendix E] The Strict-unfaith column in Table 5 is the probe's own training label, as disclosed in Appendix E: "Its Strict-unfaith column is the probe's own training label, whereas here that column comes from an independent Claude Sonnet 5 judge." For the probing rows, the detector is evaluated on the labels it was trained on, so the reduction from 20.2% to 9.8% is an in-sample rank-quality measure, not an independent faithfulness assessment. For the judge rows, the metric is still not an independent judge. The independent Claude Sonnet 5 labels used in Table 8 for Mem0 and LightMem are a step forward, but the headline SimpleMem result lacks this validation. Please re-score Table 5 with an independent judge or clearly label the Strict-unfaith numbers as training-label-based.
  3. [§4, Limitations] The paper concludes that SIRIN "can provide an effective trust layer for memory-grounded agents" based on Table 5. Given the two protocol issues above, this conclusion is not yet established. Additionally, the Limitations section states "we have not yet evaluated gate transfer across unrelated backbones without refitting," so the generality of the gate results is limited. Please either add out-of-sample validation (independent labels and a validation-based threshold) for at least the SimpleMem setup, or substantially qualify the §4 conclusion to describe the result as an in-sample demonstration.
minor comments (4)
  1. [§4] The section heading "Trustworthy memory with SIRIN" is repeated immediately before the diagram; remove the duplicate.
  2. [§4, Table 5] The retry/expansion mechanism described in §4 ("serve, retry, or abstain") is not reflected in the evaluation tables; the tables only implement abstention. Please clarify whether retry is evaluated and how, or remove the retry description from the design.
  3. [Table 2] The footnote "Uncertainty estimation is not reported because uncertainty signals from a separate proxy model are poorly aligned..." is a reasonable caveat but should be stated in the main text as well, not only in the table note.
  4. [References] Several references are dated 2026 (e.g., Liu et al. 2026, Fang et al. 2026). If these are preprints, please note their status to avoid confusion.

Circularity Check

2 steps flagged

Table 5's Strict-unfaith drop for probe gates is scored on the probe's own training labels, and the coverage threshold is a transductive within-eval rank cut; the headline memory-gate numbers are partly in-sample, though the toolkit's unification claim is independent.

specific steps
  1. fitted input called prediction [Table 5 and §4; disclosed in Appendix E ('Trustworthy Memory on Additional Memory Frameworks')]
    "Table 5 instead validates evidence quotes and drops 24 SimpleMem rows to N=476, which puts its answerability abstention at 21.2% instead of 20.0%. Its Strict-unfaith column is the probe’s own training label, whereas here that column comes from an independent Claude Sonnet 5 judge."

    In Table 5, the probing gates are trained on the strict-unfaith labels and then report Strict-unfaith on the served subset using those same labels. The drop from 20.2% to 11.6–12.9% is therefore a measure of in-sample rank quality, not an independent faithfulness measurement; it is a fitted-input-called-prediction by construction. The §4 sentence 'reducing strict unfaithfulness from 20.2% to 9.8%' for the headline SimpleMem row has no independent-label counterpart in the main text; the independent Claude judge is introduced only for the Mem0/LightMem panels in Appendix E.

  2. fitted input called prediction [Appendix E, same paragraph; applies to Table 5]
    "The coverage threshold, as in Table 5, is a transductive within-evaluation rank cut."

    The served-accuracy columns in Table 5 are computed after choosing the abstention threshold on the evaluation set itself to hit a target coverage (80%), so the reported served accuracy and abstention rate are not a deployment prediction. The threshold is fitted to the test distribution, and the served set is then scored on the same distribution. This is not label leakage, but it is a parameter fit to the evaluation set being presented as the gate's operating point, which overstates expected deployment performance if the score distribution shifts.

full rationale

The paper is a toolkit paper rather than a formal derivation, so most of the circularity burden falls on evaluation self-reference. I did not count ordinary supervised evaluations (Tables 2–4) as circular because thresholds are selected on training splits and metrics are computed on held-out portions, which is standard practice. I also did not count self-citations (Belikova et al., Bazarova et al., Rykov et al.) as load-bearing reductions: they are references to prior work, datasets, or feature options, not uniqueness theorems or ansatzes that force the paper's conclusions. The load-bearing circularity is isolated to the memory-gate demonstration. Appendix E explicitly discloses that Table 5's Strict-unfaith column is the probe's own training label and that the coverage threshold is a transductive within-evaluation rank cut. For the probing rows, the reported reduction in strict unfaithfulness is thus an in-sample artifact of scoring the gate on the labels it was trained on; and for all Table 5 rows, served accuracy is contingent on a threshold fitted to the evaluation set. The paper deserves credit for adding an independent Claude Sonnet 5 judge in Table 8 for Mem0 and LightMem, but that independent validation does not cover the headline SimpleMem result (62.4% to 79.3%) that §4 presents as the main demonstration. Therefore the central numerical claim of the trust-layer demonstration is partially circular, while the toolkit's unification and interface contributions remain independently supported by the external benchmarks in Tables 2–4.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

Most of what SIRIN contributes is software, so the ledger is dominated by evaluation-side fitted parameters rather than novel physical/formal axioms. The coverage threshold and F1 thresholds are fitted to data, the probe layer and UE fusion are selected on training/fold data, and the proxy-model/entailment alignments are domain assumptions. No invented entities are introduced.

free parameters (5)
  • Coverage threshold / rank cut for memory gates = rank cut targeting 80% coverage, chosen transductively on the evaluation set
    Selective-prediction threshold for Tables 5 & 8 is chosen within the evaluation set; this is a parameter fitted to the test data and makes reported served accuracy optimistic.
  • Decision thresholds for F1 in detection tables = selected on training split to maximize F1
    F1 numbers in Tables 2-4 depend on thresholds tuned on the training split; this is a standard but free parameter.
  • Probe layer index = middle layer; layer 30 chosen in 27 of 30 folds (Appendix D)
    Layer choice is selected by validation/fold performance, not fixed a priori.
  • Combined UE estimator selection/weights = best-performing estimators on training split aggregated after normalization
    The fused uncertainty score is constructed from estimators selected on the training split; another fitted choice.
  • Probe/CatBoost/LoRA hyperparameters = not fully specified
    Regularization, CatBoost settings, LoRA rank/alpha are not reported; they influence reported numbers and reproducibility.
axioms (5)
  • domain assumption Proxy-model probing assumption: hidden states of Qwen3-4B over annotated samples from other generators carry hallucination signal.
    Used for RAGTruth and PsiloQA probing rows because original generator internals are unavailable (§3.1, §3.2). If false, probe AUROC numbers measure something other than the original generator's hallucination state.
  • domain assumption Answerability is closely aligned with textual entailment.
    Interpretation of DeBERTa-v3 NLI transfer result in §3.1; used to justify answerability gate design.
  • domain assumption Annotated labels (RAGTruth, SQuAD 2.0, PsiloQA, LongMemEval) are correct ground truth for hallucination/answerability.
    All evaluation metrics treat these labels as ground truth; label noise is not modeled.
  • domain assumption A single forward pass over frozen LLM hidden states yields features sufficient for probes; pooling/linear classifiers can separate faithful from hallucinated outputs.
    Core to probing pipeline in §2 and Figure 1b; standard in literature but assumed.
  • ad hoc to paper Selective-prediction rank cut on evaluation data can be used to estimate gate performance.
    The transductive within-evaluation rank cut in Tables 5/8 assumes in-sample thresholding is a valid evaluation protocol, which it is not for generalization claims.

pith-pipeline@v1.3.0-alltime-deepseek · 13633 in / 19195 out tokens · 177657 ms · 2026-08-04T01:35:43.997206+00:00 · methodology

0 comments
read the original abstract

SIRIN (Semantic Inconsistency Recognition and Inspection Nexus) is a unified toolkit and interactive web UI for detecting contextual hallucinations (fluent, plausible responses unsupported by the provided evidence) in retrieval-augmented, agentic, and memory-grounded LLM systems. SIRIN unifies three detector paradigms (representation probing, uncertainty estimation, and judge-style verification) and the complementary task of pre-generation query answerability under one interface, configuration system, and evaluation pipeline, supporting response- and span-level inspection in both white-box and black-box settings. The web UI enables live analysis of user-supplied context-query-answer triples through hallucination scores, unsupported-span highlighting, and side-by-side detector comparison, with a lightweight plug-in design for adding new detectors. We demonstrate SIRIN on hallucination detection, query answerability, and as a faithfulness gate within long-term memory systems. The source code is publicly available at https://github.com/sb-ai-lab/SIRIN.

Figures

Figures reproduced from arXiv: 2608.00033 by Andrey Savchenko, Julia Belikova, Konstantin Polev, Maksim Makarenko, Mikhail Filimonov, Rauf Parchiev.

Figure 1
Figure 1. Figure 1: (a) Three detector families: Judge (black-box) uses a second model to check for contradictions between evidence and answer; Probing (white-box) trains a classifier over hidden-state representations to separate faithful from hallucinated outputs; Uncertainty fuses 38 estimators (entropy, perplexity, SAR, P(True), etc.) into a single calibrated score based on generation confidence patterns. (b) The full pipe… view at source ↗
Figure 2
Figure 2. Figure 2: SIRIN’s interactive Web UI. A visitor supplies a context–question pair (or picks a curated example), selects a detector preset spanning the probing, judge, and uncertainty families, and inspects the results directly: the answer is scored and its unsupported spans are highlighted with per-span risk, alongside a summary of suspect spans and maximum risk. switching task, detector family, or granularity is a c… view at source ↗
Figure 3
Figure 3. Figure 3: SIRIN for trustworthy dynamic memory. An￾swerability and span-level faithfulness checks validate retrieved evidence before returning an answer or trigger retrieval expansion on failure. Long-term memory turns hallucination from an isolated mistake into a compounding one: an agent that misreads its own memory confidently contra￾dicts facts the user trusts it to remember. Since every agentic memory framework… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

36 extracted references · 8 linked inside Pith

  1. [1]

    Alexandra Bazarova, Aleksandr Yugay, Andrey Shulga, Alina Ermilova, Andrei Volodichev, Konstantin Polev, Julia Belikova, Rauf Parchiev, Dmitry Simakov, Maxim Savchenko, Andrey Savchenko, Serguei Barannikov, and Alexey Zaytsev. 2025. https://arxiv.org/abs/2504.10063 Hallucination detection in llms with topological divergence on attention graphs . Transacti...

    Pith/arXiv arXiv 2025

  2. [2]

    Julia Belikova, Konstantin Polev, Rauf Parchiev, and Dmitry Simakov. 2025. https://doi.org/10.1145/3726302.3731969 Data-efficient meta-models for evaluation of context-based questions and answers in llms . In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 4385--4389. ACM

    arXiv 2025

  3. [3]

    Sky CH-Wang, Benjamin Van Durme, Jason Eisner, and Chris Kedzie. 2024. https://doi.org/10.18653/v1/2024.findings-acl.260 Do androids know they're only dreaming of electric sheep? In Findings of the Association for Computational Linguistics: ACL 2024, pages 4401--4420. Association for Computational Linguistics

    doi:10.18653/v1/2024.findings-acl.260 2024

  4. [4]

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. 2020. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning (ICML), volume 119 of Proceedings of Machine Learning Research, pages 1597--1607. PMLR

    2020

  5. [5]

    Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. 2025. https://arxiv.org/abs/2504.19413 Mem0 : Building production-ready AI agents with scalable long-term memory . CoRR, abs/2504.19413

    Pith/arXiv arXiv 2025

  6. [6]

    Yung-Sung Chuang, Linlu Qiu, Cheng-Yu Hsieh, Ranjay Krishna, Yoon Kim, and James R. Glass. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.84 Lookback lens: Detecting and mitigating contextual hallucinations in large language models using only attention maps . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pag...

    doi:10.18653/v1/2024.emnlp-main.84 2024

  7. [7]

    Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. 2024. https://doi.org/10.18653/v1/2024.eacl-demo.16 Ragas: Automated evaluation of retrieval augmented generation . In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 150--158. Association for Comput...

    doi:10.18653/v1/2024.eacl-demo.16 2024

  8. [8]

    Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, Timothy Baldwin, and Artem Shelmanov. 2023. https://doi.org/10.18653/v1/2023.emnlp-demo.41 Lm-polygraph: Uncertainty estimation for language models . In Proceedings of the 2023 Confere...

    doi:10.18653/v1/2023.emnlp-demo.41 2023

  9. [9]

    Jizhan Fang, Xinle Deng, Haoming Xu, Ziyan Jiang, Yuqi Tang, Ziwen Xu, Shumin Deng, Yunzhi Yao, Mengru Wang, Shuofei Qiao, Huajun Chen, and Ningyu Zhang. 2026. https://arxiv.org/abs/2510.18866 LightMem : Lightweight and efficient memory-augmented generation . In The Fourteenth International Conference on Learning Representations (ICLR)

    Pith/arXiv arXiv 2026

  10. [10]

    Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. 2024. https://doi.org/10.1038/s41586-024-07421-0 Detecting hallucinations in large language models using semantic entropy . Nature, 630(8017):625--630

    doi:10.1038/s41586-024-07421-0 2024

  11. [11]

    Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Fr \'e d \'e ric Blain, Francisco Guzm \'a n, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. 2020. https://doi.org/10.1162/tacl_a_00330 Unsupervised quality estimation for neural machine translation . Transactions of the Association for Computational Linguistics, 8:539--555

    doi:10.1162/tacl_a_00330 2020

  12. [12]

    John Hewitt and Percy Liang. 2019. https://doi.org/10.18653/v1/D19-1275 Designing and interpreting probes with control tasks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2733--2743. Association for Computational Li...

    doi:10.18653/v1/d19-1275 2019

  13. [13]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. https://doi.org/10.1145/3703155 A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions . ACM Transactions on Information Systems

    doi:10.1145/3703155 2025

  14. [14]

    Tomczak, and Max Welling

    Maximilian Ilse, Jakub M. Tomczak, and Max Welling. 2018. Attention-based deep multiple instance learning. In Proceedings of the 35th International Conference on Machine Learning (ICML), volume 80 of Proceedings of Machine Learning Research, pages 2127--2136. PMLR

    2018

  15. [15]

    Jeffrey Ip. 2024. Deepeval: The open-source llm evaluation framework. https://github.com/confident-ai/deepeval

    2024

  16. [16]

    Ziwei Ji, Delong Chen, Etsuko Ishii, Samuel Cahyawijaya, Yejin Bang, Bryan Wilie, and Pascale Fung. 2024. https://doi.org/10.18653/v1/2024.blackboxnlp-1.6 Llm internal states reveal hallucination risk faced with a query . In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 88--104. Association for Comp...

    doi:10.18653/v1/2024.blackboxnlp-1.6 2024

  17. [17]

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Delong Chen, Wenliang Dai, Ho Shu Chan, Andrea Madotto, and Pascale Fung. 2023. https://doi.org/10.1145/3571730 Survey of hallucination in natural language generation . ACM Computing Surveys, 55(12):248:1--248:38

    doi:10.1145/3571730 2023

  18. [18]

    Malik, and Yarin Gal

    Jannik Kossen, Jiatong Han, Muhammed Razzak, Lisa Schut, Shreshth A. Malik, and Yarin Gal. 2024. https://arxiv.org/abs/2406.15927 Semantic entropy probes: Robust and cheap hallucination detection in llms . CoRR, abs/2406.15927

    Pith/arXiv arXiv 2024

  19. [19]

    \'A d \'a m Kov \'a cs and G \'a bor Recski. 2025. https://arxiv.org/abs/2502.17125 Lettucedetect: A hallucination detection framework for rag applications . CoRR, abs/2502.17125

    Pith/arXiv arXiv 2025

  20. [20]

    Xixun Lin, Yucheng Ning, Jingwen Zhang, Yan Dong, Yilong Liu, Yongxuan Wu, Xiaohua Qi, Nan Sun, Yanmin Shang, Pengfei Cao, Lixin Zou, Xu Chen, Chuan Zhou, Jia Wu, Shirui Pan, Bin Wang, Yanan Cao, Kai Chen, Songlin Hu, and Li Guo. 2025. https://arxiv.org/abs/2509.18970 Llm-based agents suffer from hallucinations: A survey of taxonomy, methods, and directio...

    arXiv 2025

  21. [21]

    Zhouhan Lin, Minwei Feng, C \' cero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. https://openreview.net/forum?id=BJC_jUqxe A structured self-attentive sentence embedding . In 5th International Conference on Learning Representations (ICLR). OpenReview.net

    2017

  22. [22]

    Jiaqi Liu, Yaofeng Su, Peng Xia, Siwei Han, Zeyu Zheng, Cihang Xie, Mingyu Ding, and Huaxiu Yao. 2026. https://arxiv.org/abs/2601.02553 Simplemem: Efficient lifelong memory for llm agents . CoRR, abs/2601.02553

    Pith/arXiv arXiv 2026

  23. [23]

    Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.557 Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9004--9017. Association for Computational Linguistics

    doi:10.18653/v1/2023.emnlp-main.557 2023

  24. [24]

    Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.741 Factscore: Fine-grained atomic evaluation of factual precision in long form text generation . In Proceedings of the 2023 Conference on Empirical Methods in Natural Langu...

    doi:10.18653/v1/2023.emnlp-main.741 2023

  25. [25]

    Cheng Niu, Yuanhao Wu, Juno Zhu, Siliang Xu, KaShun Shum, Randy Zhong, Juntong Song, and Tong Zhang. 2024. https://doi.org/10.18653/v1/2024.acl-long.585 Ragtruth: A hallucination corpus for developing trustworthy retrieval-augmented language models . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long...

    doi:10.18653/v1/2024.acl-long.585 2024

  26. [26]

    Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. https://doi.org/10.18653/v1/P18-2124 Know what you don't know: Unanswerable questions for squad . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784--789. Association for Computational Linguistics

    doi:10.18653/v1/p18-2124 2018

  27. [27]

    Elisei Rykov, Kseniia Petrushina, Maksim Savkin, Valerii Olisov, Artem Vazhentsev, Kseniia Titova, Alexander Panchenko, Vasily Konovalov, and Julia Belikova. 2025. https://doi.org/10.18653/v1/2025.findings-emnlp.626 When models lie, we learn: Multilingual span-level hallucination detection with psiloqa . In Findings of the Association for Computational Li...

    doi:10.18653/v1/2025.findings-emnlp.626 2025

  28. [28]

    Snowflake Inc. 2024. Trulens: Evaluation and tracking for llm experiments and ai agents. https://github.com/truera/trulens

    2024

  29. [29]

    Manveer Singh Tamber, Forrest Sheng Bao, Chenyu Xu, Ge Luo, Suleman Kazi, Minseok Bae, Miaoran Li, Ofer Mendelevitch, Renyi Qu, and Jimmy Lin. 2025. https://doi.org/10.18653/v1/2025.emnlp-industry.54 Benchmarking llm faithfulness in rag with evolving leaderboards . In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: ...

    doi:10.18653/v1/2025.emnlp-industry.54 2025

  30. [30]

    UpTrain AI . 2023. Uptrain: An open-source unified platform to evaluate and improve generative ai applications. https://github.com/uptrain-ai/uptrain

    2023

  31. [31]

    Roman Vashurin, Ekaterina Fadeeva, Artem Vazhentsev, Lyudmila Rvanova, Daniil Vasilev, Akim Tsvigun, Sergey Petrakov, Rui Xing, Abdelrahman Sadallah, Kirill Grishchenkov, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov, and Artem Shelmanov. 2025. https://doi.org/10.1162/tacl_a_00737 Benchmarking uncertainty quantification methods for larg...

    doi:10.1162/tacl_a_00737 2025

  32. [32]

    Artem Vazhentsev, Lyudmila Rvanova, Gleb Kuzmin, Ekaterina Fadeeva, Ivan Lazichny, Alexander Panchenko, Maxim Panov, Mrinmaya Sachan, Preslav Nakov, Timothy Baldwin, and Artem Shelmanov. 2026. Efficient hallucination detection for llms using uncertainty-aware attention heads. In Proceedings of the 43rd International Conference on Machine Learning (ICML), ...

    Pith/arXiv arXiv 2026

  33. [33]

    Ra \'u l V \'a zquez, Timothee Mickus, Elaine Zosa, Teemu Vahtola, J \"o rg Tiedemann, Aman Sinha, Vincent Segonne, Fernando S \'a nchez-Vega, Alessandro Raganato, Jind r ich Libovick \'y , Jussi Karlgren, Shaoxiong Ji, Jind r ich Helcl, Liane Guillou, Ona de Gibert, Jaione Bengoetxea, Joseph Attieh, and Marianna Apidianaki. 2025. https://aclanthology.org...

    2025

  34. [34]

    Min-Hsuan Yeh, Max Kamachee, Seongheon Park, and Yixuan Li. 2025. https://arxiv.org/abs/2502.11948 Halluentity: Benchmarking and understanding entity-level hallucination detection . Transactions on Machine Learning Research. ArXiv:2502.11948

    Pith/arXiv arXiv 2025

  35. [35]

    Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting Hu. 2023. https://doi.org/10.18653/v1/2023.acl-long.634 Alignscore: Evaluating factual consistency with a unified alignment function . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11328--11348. Association for Computational Linguistics

    doi:10.18653/v1/2023.acl-long.634 2023

  36. [36]

    Tianhang Zhang, Lin Qiu, Qipeng Guo, Cheng Deng, Yue Zhang, Zheng Zhang, Chenghu Zhou, Xinbing Wang, and Luoyi Fu. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.58 Enhancing uncertainty-based hallucination detection with stronger focus . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 915--932. Associat...

    doi:10.18653/v1/2023.emnlp-main.58 2023