Pith. sign in

REVIEW 5 major objections 5 minor 33 references

Too Consistent to Detect: A Study of Self-Consistent Errors in LLMs

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper shows that LLMs can repeatedly produce the same wrong answer across samples—self-consistent errors—that do not shrink with model scale and that today's error detectors largely miss, until a second external LLM's hidden states…

desk verdict A useful formalization of self-consistent errors with a real detection gap, but the cross-model probe's headline row is contaminated by labeler-verifier overlap. read the letter →

arxiv 2505.17656 v3 pith:P3TOMQJP submitted 2025-05-23 cs.CL

Hexiang Tan , Fei Sun , Sha Liu , Du Su , Qi Cao , Xin Chen , Jingang Wang , Xunliang Cai
show 3 more authors
Yuanzhuo Wang Huawei Shen Xueqi Cheng
This is my paper · ORCID
classification cs.CL
keywords self-consistenterrorsLLMerrordetectionhallucinationsemanticentropyconsistency-baseduncertaintysupervisedprobecross-modelverificationscaling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper identifies a class of large language model mistakes it calls self-consistent errors: for a given question, the model generates the same wrong answer across many random samples. It shows that, unlike inconsistent errors, these errors do not become less frequent as models grow larger; on the benchmarks studied they stay roughly constant or even increase. The paper then evaluates four families of error-detection methods—token probabilities, self-reported confidence, semantic entropy, and supervised probes on internal states—and finds all of them perform markedly worse on self-consistent errors, with semantic entropy falling to or below random guessing. To address this, it proposes a cross-model probe that trains a detector on hidden states of a separate verifier model and blends its score with the original probe, raising detection AUROC across three model families. If correct, the paper implies that a model's own uncertainty signals are structurally blind to its most confident mistakes, and evidence from an external model is needed.

What carries the argument

The central objects are the formal definition of a self-consistent error and the cross-model probe architecture. Definition 1 declares an error self-consistent when all k=15 stochastic samples taken at temperature 0.5 are semantically equivalent to the greedy response—judged by pairwise NLI mutual entailment—and the greedy response is labeled incorrect. The detection machinery is the cross-model probe: hidden states h_M from the generating model at a chosen layer and token position feed a probe called Probe_M, while hidden states h_V from an external verifier V fed the same question and response feed a second probe Probe_V; the final error score is (1-λ)·s_M + λ·s_V, with λ selected on validation. The load-bearing empirical observation is that self-consistent errors rarely overlap across models, which gives the external verifier's hidden states information the generating model's own states do not provide.

What would settle it

Take a few hundred of the paper's self-consistent-error cases and have independent human annotators judge both correctness and whether the 15 samples really mean the same thing; if many cases are reclassified as inconsistent or correct, the reported scale-resistance and detector gaps should shrink proportionally.

Watch

Extended reading notes

Core claim

When a language model is sampled multiple times at nonzero temperature, it sometimes gives the same wrong answer every time. The paper names this a self-consistent error and reports two central findings: first, the frequency of such errors does not decline with model size, remaining stable or even rising while inconsistent errors drop sharply; second, all four types of mainstream error detectors—probability averaging, P(True), semantic entropy, and supervised probes on hidden states—suffer substantial performance drops on these errors, with the consistency-based semantic entropy detector performing at or below chance. The paper further observes that self-consistent errors are largely model-specific, with the highest overlap between two tested models at only 28.7% of questions, and builds on this by proposing a cross-model probe: a probe trained on hidden states of an external verifier LLM looking at the same question-response pair, combined with the original model's probe through a scalar weight. On the tested datasets and model families, this combined probe improves AUROC on self-consistent errors while preserving or slightly improving performance on inconsistent errors.

Load-bearing premise

The paper's split between self-consistent and inconsistent errors depends on Llama-3.1-70B's rubric-based judgments of correctness and semantic equivalence, with only 1 of 300 labels checked by a human; if that labeler itself makes self-consistent errors, the error counts and detector scores are built on a wrong partition.

Editorial extensions

If this is right

  • Consistency-based uncertainty methods, including semantic entropy, cannot be trusted as error detectors for self-consistent errors, since they score at or below random guessing on CE subsets.
  • Scaling up model size will not by itself reduce the absolute number of self-consistent errors on these QA benchmarks, so this error class needs targeted mitigation beyond scale.
  • Supervised probes on the generating model's hidden states also degrade on self-consistent errors, showing that the generating model's internal evidence alone is insufficient.
  • A single forward pass through an external verifier's hidden states, fused with the original probe, improves detection of self-consistent errors across Qwen, Llama, and Mistral models without sacrificing performance on inconsistent errors.
  • The assumption that consistent outputs are more likely to be correct holds for inconsistent errors but fails specifically for self-consistent errors.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the low overlap between models suggests that mixing answers from two or more differently trained LLMs could reduce self-consistent errors at generation time, not just detect them.
  • Beyond the paper, the method's gains may shrink when the verifier shares the original model's training distribution or alignment data; the paper's own verifier-selection results, showing cross-series verifiers help more than same-series ones, point in that direction.
  • Beyond the paper, the CE frequency and AUROC numbers are tied to k=15 samples and to an NLI equivalence test; with longer, free-form generations or larger k the CE/IE boundary may shift, so the scale-resistance claim should be rechecked on open-ended tasks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper defines 'self-consistent errors' (SCEs): cases where an LLM's greedy response is incorrect and all k stochastic samples are semantically equivalent to that incorrect answer. It measures the prevalence of SCEs across nine Qwen/Llama scales on SciQ and TriviaQA, evaluates four families of error detectors (probability, P(True), semantic entropy, and supervised probes) on SCE versus inconsistent-error subsets, and proposes a cross-model probe that combines hidden states of the target model with hidden states of an external verifier model. The authors report that (1) SCE frequency is stable or increases with scale while inconsistent errors decrease, (2) all detector types, especially semantic entropy, perform substantially worse on SCEs, and (3) the cross-model probe improves CE detection across three LLM families without hurting IE detection.

Significance. If the findings hold, the paper makes a useful contribution: it identifies a failure mode that is invisible to consistency-based uncertainty estimation, documents that scaling alone does not remove the failure mode, and offers a cheap external-verifier remedy that avoids the 10-20 extra generations used by prior cross-model sampling methods. The paper is also unusually transparent: code is released, the definition of SCE is formal, the evaluation covers four detector categories, and the verifier-overlap analysis in Table 3 is a valuable empirical observation. However, several load-bearing points need to be fixed before the main claims can be accepted: the semantic-entropy result is confounded by the subset definition, the Qwen2.5-14B cross-model row is contaminated by verifier-labeler overlap, the scaling claim lacks statistical support, and the ground-truth labels rest on a very small manual audit.

major comments (5)
  1. [Section 2.2, Table 2] The semantic-entropy (SE) result on the CE subset is partly an artifact of the subset definition. By definition, every CE instance has all k stochastic samples semantically equivalent to the greedy response, so its semantic entropy is essentially minimal. Correct responses, in contrast, are not constrained to have minimal entropy. An AUROC below 0.5 can therefore arise even if SE is a perfectly calibrated uncertainty measure, simply because the negative set was constructed to have near-zero entropy. To support the claim that 'consistency-based detectors fail,' the paper should report SE AUROC on a matched control in which correct responses also have all samples semantically equivalent, or should otherwise control for the consistency distribution between CE and IE subsets.
  2. [Section 4.1, Table 2; Appendix A.4] The cross-model probe result for Qwen2.5-14B is contaminated by verifier-labeler overlap. All ground-truth labels z_i are produced by Llama-3.1-70B (Appendix A.4), and for Qwen2.5-14B the external verifier is also Llama3.1-70B (Table 2 caption). Probe_V is therefore trained to predict z_i from hidden states of the very model that generated z_i, so it can learn the labeler's internal grading behavior rather than a generalizable truthfulness signal. The reported CE gains for Qwen2.5-14B (SciQ 0.7473 to 0.8118; TQA 0.8512 to 0.9332) may be inflated by this leakage. Please re-run the Qwen2.5-14B row with a verifier that did not produce the labels, or re-annotate labels for that row with a different model or with human annotation.
  3. [Section 2.3, Figure 1, Figure 3] The claim that self-consistent errors 'remain stable or even increase' with model scale is supported only by raw frequency counts in two bar charts. No confidence intervals, per-question rates, normalization for dataset/model differences, or significance tests are reported. Since this is the paper's first central finding, please report per-question SCE rates with bootstrap confidence intervals and a trend test (or at least error bars) across the nine model scales.
  4. [Appendix A.4] The correctness labels z_i are produced by a single LLM, Llama-3.1-70B, and the audit is only 1 disagreement out of 300 samples. This is a weak basis for the entire CE/IE split, especially because the questions where the labeler itself makes a self-consistent error would systematically misclassify errors. Please report agreement on a larger, stratified sample that oversamples hard cases (e.g., responses that are semantically close to the gold answer), and separately report labeler agreement on the subset that is ultimately classified as self-consistent.
  5. [Section 2.2] The operationalization of 'semantic equivalence' via NLI-based mutual entailment is inherited from prior work but is never validated on the datasets used here. Noisy equivalence judgments would reclassify errors between the CE and IE subsets, and since the CE/IE split is the backbone of every experiment, the paper should report human agreement on a sample of the pairwise equivalence judgments, and ideally a sensitivity analysis over the equivalence threshold or a second equivalence method.
minor comments (5)
  1. [Table 2] The AUROC values are reported as point estimates with no variance. Supervised probe training is stochastic, so please report means and standard deviations over at least three random seeds, or state that the results are from a single run.
  2. [Section 4.1] The integration parameter lambda is selected on the validation set as the value with the best validation performance. Please clarify whether the reported test AUROC is based on the validation-selected lambda, and describe the validation/test split so readers can judge selection overfitting risk.
  3. [Appendix A.6] The evaluation prompt contains a duplicated 'Predicted answer 4:' line, which should be corrected to 'Predicted answer 5:' for readability.
  4. [Appendix A.5] The k=1 point in Figure 4 corresponds to comparing only the greedy response against itself, which is degenerate; please label this clearly or start the curve at k=2.
  5. [Table 3] The overlap analysis is only reported for Qwen2.5-7B as the target. Since Table 4 shows that verifier choice matters, an overlap matrix across all target/verifier pairs would strengthen the verifier-selection guidance.

Circularity Check

1 steps flagged · score 5.0 of 10

The semantic-entropy baseline's failure on self-consistent errors is definitionally forced, while the paper's other empirical claims remain non-circular.

  1. self definitional [Section 2.2 Definition 1; Section 3.1 SE baseline; Section 3.2 Table 2]
    "If all samples are semantically equivalent to the greedy response, ... and the greedy answer is judged incorrect (z_i = 0), then r^g_i is a self-consistent error. ... [SE] samples multiple responses and calculates the entropy of their semantic clusters. ... SE ... exhibits the most dramatic decline on CE subsets, performing at or below random guessing."

    The CE subset is defined as instances in which every stochastic sample is semantically equivalent to the greedy response. Semantic entropy is, by construction, at or near zero when all sampled responses fall into a single semantic cluster, so SE assigns its lowest error score to every CE instance. The reported AUROC at or below 0.5 is therefore an inevitable consequence of the definition, not an empirical discovery about detector quality. The IE subset necessarily contains multiple clusters, so SE's ranking across the CE and IE subsets is anti-correlated by design.

full rationale

The only step that reduces by construction is the SE baseline on the CE subset. Definition 1 defines CE as full semantic equivalence among stochastic samples, and SE scores uncertainty as semantic-cluster entropy; CE items therefore must receive near-minimal entropy scores, forcing the reported below-random AUROC. This is a genuine self-definitional artifact, and it affects one of the four detector families evaluated. The paper's other headline results are not circular: the probability, P(True), and probe baselines are empirical; the scale-resistance analysis depends on the LLM-produced correctness labels and the NLI equivalence check, which are assumptions rather than circular reductions; and the cross-model probe is evaluated against held-out data with an external verifier. A separate validity concern, not a circularity, is that for the Qwen2.5-14B row the verifier (Llama3.1-70B) is the same model that generated the correctness labels, potentially allowing Probe_V to emulate the labeler; this affects interpretation but does not make the result equivalent to its input by construction. No load-bearing self-citation or imported uniqueness argument was found.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The paper does not derive a law; it makes empirical claims. The central numbers depend on a hand-chosen sample count (k=15), a validation-tuned integration weight lambda, a validation-selected probe layer, an LLM judge for correctness, and an NLI criterion for semantic equivalence. There are no invented physical entities; 'self-consistent error' is a definitional category. The labeler and NLI assumptions are domain assumptions, not standard math.

free parameters (3)
  • lambda integration weight = selected from {0, 0.05, ..., 1.0} on validation set
    The cross-model probe combines original and verifier probes with lambda tuned by validation AUROC (Section 4.1).
  • number of stochastic samples k = 15
    CE/IE classification uses k=15; the count of self-consistent errors changes with k (Figure 4), so the definition depends on this hand-picked threshold.
  • probe layer l = layer with best validation AUROC
    The supervised probe uses hidden states from whichever layer maximizes validation AUROC (Appendix A.3).
assumptions (5)
  • domain assumption NLI-based mutual entailment is a valid semantic equivalence relation
    Used to decide when stochastic samples are 'the same'; if noisy, CE/IE labels shift. Section 2.2.
  • domain assumption Llama-3.1-70b correctness judgments are accurate
    Ground-truth labels are LLM-generated with only 1/300 manual check; systematic labeler errors would change all results. Section A.4.
  • domain assumption SciQ and TriviaQA gold answers are correct and representative
    The evaluation is limited to two QA datasets; domain bias could affect the scaling claim. Section 2.3.
  • standard math AUROC on class-balanced CE/IE subsets is a fair comparison metric
    AUROC is threshold-independent, but subset construction still decides which negatives are included. Section 3.1.
  • domain assumption Hidden states encode enough correctness signal for a probe
    The probe method assumes hidden states carry truthfulness features, following Azaria and Mitchell (2023). Section 3.1 and A.3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Too Consistent to Detect: A Study of Self-Consistent Errors in LLMs." pith.science (2026). https://pith.science/paper/P3TOMQJP

@misc{pith2026250517656,
  author       = {Pith},
  title        = {Pith review of: Too Consistent to Detect: A Study of Self-Consistent Errors in LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P3TOMQJP}},
  note         = {Machine review of arXiv:2505.17656}
}
read the original abstract

As large language models (LLMs) often generate plausible but incorrect content, error detection has become increasingly critical to ensure truthfulness. However, existing detection methods often overlook a critical problem we term as self-consistent error, where LLMs repeatedly generate the same incorrect response across multiple stochastic samples. This work formally defines self-consistent errors and evaluates mainstream detection methods on them. Our investigation reveals two key findings: (1) Unlike inconsistent errors, whose frequency diminishes significantly as the LLM scale increases, the frequency of self-consistent errors remains stable or even increases. (2) All four types of detection methods significantly struggle to detect self-consistent errors. These findings reveal critical limitations in current detection methods and underscore the need for improvement. Motivated by the observation that self-consistent errors often differ across LLMs, we propose a simple but effective cross-model probe method that fuses hidden state evidence from an external verifier LLM. Our method significantly enhances performance on self-consistent errors across three LLM families.

Figures

Figures reproduced from arXiv: 2505.17656 by the authors.

Figure 1
Figure 1. Frequency of self-consistent and inconsis [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Performance of cross-model under different [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. The number of self-consistent and inconsistent errors across different scales of LLMs on TriviaQA. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: The number of self-consistent errors with different sample numbers [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 12 canonical work pages

  1. [1]

    Amos Azaria and Tom Mitchell. 2023. The internal state of an LLM knows when it ' s lying. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore. Association for Computational Linguistics

    work page 2023

  2. [2]

    Mohammad Beigi, Ying Shen, Runing Yang, Zihao Lin, Qifan Wang, Ankith Mohan, Jianfeng He, Ming Jin, Chang-Tien Lu, and Lifu Huang. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.751 I nternal I nspector i^2 : Robust confidence estimation in LLM s through internal states . In Findings of the Association for Computational Linguistics: EMNLP 2024, pag...

    doi:10.18653/v1/2024.findings-emnlp.751 2024

  3. [3]

    Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. 2023. Discovering latent knowledge in language models without supervision. In The Eleventh International Conference on Learning Representations

    2023

  4. [4]

    Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. 2024 a . https://openreview.net/forum?id=Zj12nzlQbz INSIDE : LLM s' internal states retain the power of hallucination detection . In The Twelfth International Conference on Learning Representations

    2024

  5. [5]

    Zhiyu Chen, Jing Ma, Xinlu Zhang, Nan Hao, An Yan, Armineh Nourbakhsh, Xianjun Yang, Julian McAuley, Linda Ruth Petzold, and William Yang Wang. 2024 b . https://openreview.net/forum?id=upAWnMgpnH A survey on large language models for critical societal domains: Finance, healthcare, and law . Transactions on Machine Learning Research. Survey Certification

    work page 2024

  6. [6]

    Xuefeng Du, Chaowei Xiao, and Yixuan Li. 2024. Haloscope: Harnessing unlabeled LLM generations for hallucination detection. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    work page 2024

  7. [7]

    Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. 2024. Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5050--5063

    2024

  8. [8]

    Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. 2024. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625--630

    2024

Show all 33 references
  1. [9]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. https://doi.org/10.1145/3703155 A survey on hallucination in large language models: Principles, taxonomy, challenges, and o...

    2025 doi

  2. [10]

    Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p...

    2017

  3. [11]

    Saurav Kadavath, Tom Conerly, and et al. 2022. https://arxiv.org/abs/2207.05221 Language models (mostly) know what they know . Preprint, arXiv:2207.05221

    2022 arXiv

  4. [12]

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. https://openreview.net/forum?id=VD-AYtP0dve Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation . In The Eleventh International Conference on Learning Representations

    2023

  5. [13]

    Kenneth Li, Oam Patel, Fernanda Vi \'e gas, Hanspeter Pfister, and Martin Wattenberg. 2023. Inference-time intervention: Eliciting truthful answers from a language model. In Thirty-seventh Conference on Neural Information Processing Systems

    2023

  6. [14]

    Lin, Jacob Hilton, and Owain Evans

    Stephanie C. Lin, Jacob Hilton, and Owain Evans. 2022. https://api.semanticscholar.org/CorpusID:249191391 Teaching models to express their uncertainty in words . Trans. Mach. Learn. Res., 2022

    2022

  7. [15]

    Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2024. https://openreview.net/forum?id=DWkJCSxKU5 Generating with confidence: Uncertainty quantification for black-box large language models . Transactions on Machine Learning Research

    2024

  8. [16]

    Mat \'e o Mahaut, Laura Aina, Paula Czarnowska, Momchil Hardalov, Thomas M \"u ller, and Lluis Marquez. 2024. https://doi.org/10.18653/v1/2024.acl-long.250 Factual confidence of LLM s: on reliability and robustness of current estimators . In Proceedings of the 62nd Annual Meet...

    2024 doi

  9. [17]

    Andrey Malinin and Mark Gales. 2021. Uncertainty estimation in autoregressive structured prediction. In The Ninth International Conference on Learning Representations

    2021

  10. [18]

    Potsawee Manakul, Adian Liusie, Mark JF Poon, Yun-Sung Chuang, and Philip HS Torr. 2023. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processin...

    2023

  11. [19]

    Samuel Marks and Max Tegmark. 2024. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. In First Conference on Language Modeling

    2024

  12. [20]

    Hadas Orgad, Michael Toker, Zorik Gekhman, Roi Reichart, Idan Szpektor, Hadas Kotek, and Yonatan Belinkov. 2025. https://openreview.net/forum?id=KRnsX5Em3W LLM s know more than they show: On the intrinsic representation of LLM hallucinations . In The Thirteenth International C...

    2025

  13. [21]

    Weihang Su, Changyue Wang, Qingyao Ai, Yiran Hu, Zhijing Wu, Yujia Zhou, and Yiqun Liu. 2024. https://doi.org/10.18653/v1/2024.findings-acl.854 Unsupervised real-time hallucination detection based on the internal states of large language models . In Findings of the Association...

    2024 doi

  14. [22]

    Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. 2023. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedin...

    2023

  15. [23]

    Chaojun Wang and Rico Sennrich. 2020. https://doi.org/10.18653/v1/2020.acl-main.326 On exposure bias, hallucination and domain shift in neural machine translation . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3544--3552, On...

    2020 doi

  16. [24]

    Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. https://openreview.net/forum?id=1PL1NIMMrw Self-consistency improves chain of thought reasoning in language models . In The Eleventh International Conferenc...

    2023

  17. [25]

    Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. 2024. https://arxiv.org/abs/2411.04368 Measuring short-form factuality in large language models . Preprint, arXiv:2411.04368

    2024 arXiv

  18. [26]

    Johannes Welbl, Nelson F Liu, and Matt Gardner. 2017. Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 94--106

    2017

  19. [27]

    Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2024. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. In The Twelfth International Conference on Learning Representations

    2024

  20. [28]

    Yihao Xue, Kristjan Greenewald, Youssef Mroueh, and Baharan Mirzasoleiman. 2025. Verify when uncertain: Beyond self-consistency in black box hallucination detection. arXiv preprint arXiv:2502.15845

    2025 arXiv

  21. [29]

    An Yang, Baosong Yang, Beichen Zhang, and et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

    2024 arXiv

  22. [30]

    Jiaxin Zhang, Zhuohang Li, Kamalika Das, Bradley Malin, and Sricharan Kumar. 2023. SAC ^3 : Reliable hallucination detection in black-box language models via semantic-aware cross-check consistency. In Findings of the Association for Computational Linguistics: EMNLP 2023. Assoc...

    2023

  23. [31]

    Derui Zhu, Dingfan Chen, Qing Li, Zongxiong Chen, Lei Ma, Jens Grossklags, and Mario Fritz. 2024. P o LLM graph: Unraveling hallucinations in large language models via state transition dynamics. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 47...

    2024

  24. [32]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  25. [33]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.