Pith. sign in

REVIEW 4 major objections 4 minor 27 references

Eliminating Hallucination-Induced Errors in LLM Code Generation with Functional Clustering

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Functional clustering turns LLM sampling noise into a quantified confidence score and cuts returned-answer error from ~65% to 2% (and to 0% at a conservative threshold).

desk verdict A clean empirical wrapper for filtering LLM code output, but the headline 'exact confidence' and exponential guarantees do not survive contact with the actual test-based oracle and in-sample threshold tuning. read the letter →

arxiv 2506.11021 v1 pith:RMGZQHIF submitted 2025-05-16 cs.SE cs.AI

Chaitanya Ravuri , Saman Amarasinghe This is my paper · ORCID
classification cs.SEcs.AI
keywords functionalclusteringLLMcodegenerationhallucinationdetectionconfidenceestimationselectiveabstentionbehavioralequivalencetest-basedoracleChernoffbound
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Coding LLMs still produce plausible but subtly wrong programs, so autonomous deployment is unsafe. This paper proposes a black-box wrapper: sample many candidate programs, execute each on automatically generated test inputs, and group those with identical outputs. The size of the largest group becomes a confidence score, and a single threshold on that score lets the wrapper answer only when confident and abstain otherwise. On LiveCodeBench the wrapper reportedly preserves baseline pass@1 while cutting the error rate of returned answers from roughly 65% to 2%, and reaches 0% at a more conservative threshold while still answering 15.6% of prompts. Manual audits attribute the few residual failures to misreading the prompt rather than to random generation noise.

What carries the argument

The central object is the functional equivalence class: two programs are equivalent when their output vectors over the generated test inputs match exactly. The machinery is the routine that samples $n$ programs, runs each on $m$ generated inputs, and clusters them by these vectors, computing $\rho = s_{\max}/n$, the empirical mass of the largest class. This statistic is the confidence score, and the abstention rule is $\rho \ge \tau$. What carries the argument is the concentration inequality in Eq. 5, which bounds the probability of accepting a truly low-mass class by an exponentially decaying tail in $n$, and Eq. 7, which says a real behavioral difference is missed only if none of the $m$ random test inputs happen to hit the differing region.

What would settle it

Construct a task whose failure mode lies outside the input generator's reach—for example, a function with a subtle error on large integer arguments while the generated tests use only small integers. If the dominant cluster passes the self-generated suite, exceeds the acceptance threshold, and still fails hidden tests, the faithfulness premise is violated. A reader could also scan LiveCodeBench output for any dominant-cluster failure that is not a prompt misinterpretation but a genuine algorithmic bug shared by all samples.

Watch

Extended reading notes

Core claim

The central claim is that functional equivalence classes—programs returning identical outputs on an auto-generated test suite—concentrate the probability mass of correct solutions, while hallucinated bugs scatter mass across many small classes. The paper uses the empirical mass $\hat{C}_n$ of the largest cluster as the confidence estimate and returns a representative program only when $\hat{C}_n \ge \tau$. It proves a Chernoff-style bound, $\Pr[\hat{C}_n \ge \tau \mid C] \le \exp(-n D_{\mathrm{KL}}(\tau\|C))$, on the chance of accepting a cluster whose true mass is below threshold, giving an exponential reliability guarantee. The authors claim this turns any black-box LLM into a selective coder: on LiveCodeBench, error on returned answers drops from ~65% to 2%, and to 0% at a conservative threshold while still solving 15.6% of benchmark tasks, with HumanEval pass@1 preserved.

Load-bearing premise

The load-bearing premise is that agreement on the finite, automatically generated test-input set is a faithful stand-in for universal correctness: if the LLM's own input generator systematically misses the inputs that expose a bug, all sampled programs can agree, the largest cluster can be confidently wrong, and the confidence score will not reveal it.

Editorial extensions

If this is right

  • Because the wrapper needs only sampling and sandbox execution, it applies unchanged to closed-source APIs and future models, without logits, fine-tuning, or auxiliary classifiers.
  • A user can trade coverage for reliability with one scalar threshold; doubling the sample size roughly squares the false-acceptance bound from Eq. 5.
  • On LiveCodeBench, thresholding at 2% residual error answers 36.8% of tasks while preserving baseline pass@1 on solvable tasks, and a conservative threshold answers 15.6% with zero observed errors.
  • Manual audits attribute residual failures to prompt misinterpretation; rewriting four ambiguous HumanEval specifications turned all four into high-confidence correct solutions.
  • The same test-based oracle can validate optimized rewrites or ensemble outputs, because equivalence is checked behaviorally rather than syntactically.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If this result holds, token-level likelihood or embedding-distance confidence scores become less necessary for code: behavioral clustering captures probability mass that syntactic variants split, which could improve any downstream abstention or re-ranking system.
  • The faithfulness of the confidence score depends on the input generator's coverage; pairing the method with coverage-guided or mutation-based test generation could reveal blind spots that the current self-generated suites miss.
  • For tasks that admit several valid outputs—permutations, tie-breaking, regex matches—the single-class assumption excludes them by design; an extension that clusters under an equivalence relation appropriate to the task could recover them.
  • Across models, one model's dominant cluster can be wrong while another model's cluster disagrees; using cross-model disagreement as an additional abstention signal might reduce the residual prompt-misinterpretation errors the paper observes.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a black-box post-processing wrapper for LLM code generation. For each prompt, it samples n candidate programs, generates m test inputs using the same LLM, executes all candidate programs on all inputs, and clusters programs by identical output vectors. The empirical fraction of samples in the largest cluster, denoted rho, is used as a confidence estimate; the wrapper returns a representative of the largest cluster if rho >= tau and abstains otherwise. The authors derive a Chernoff bound (Eq. 5) for the probability that the empirical mass of a fixed equivalence class exceeds tau when its true mass is below tau, and report experiments on HumanEval and LiveCodeBench claiming error rates reduce from roughly 65% to 2% and to 0% at a conservative threshold.

Significance. The problem addressed is important and the method is simple, black-box, and potentially applicable to closed APIs. The paper ships code and reproduces its prompts in an appendix, which is good practice. However, the central theoretical guarantee does not apply to the decision rule actually used (selection of the largest class), and the test oracle is generated by the same model family as the candidate programs, so a systematic misinterpretation of the specification can produce a large, confidently wrong cluster. The headline zero-error result is obtained by tuning the threshold on the 219-task test set, so it is an in-sample description rather than a certified or predicted error rate. The method may have heuristic value as a consensus filter, but the paper does not substantiate the claimed exponential reliability guarantees for the measured object.

major comments (4)
  1. [Section 3.3, Eq. (5); Algorithm 1] The Chernoff bound in Eq. (5) bounds the tail probability for a fixed equivalence class with true probability mass C under the sampling distribution P. Algorithm 1, however, selects the largest empirical class after seeing the data. Because the number of classes can be large and the class identity is data-dependent, the probability that the selected class has empirical mass at least tau while its true mass is below tau is not controlled by Eq. (5). The paper does not provide a union bound or any argument that the selection step preserves the exponential guarantee, so the claimed reliability guarantee does not attach to the rule that is evaluated.
  2. [Section 3.4, Eq. (6)-(7); Section 4] The test inputs in Eq. (7) are drawn from a task-dependent distribution D, and Section 4 states that the self-generated test suites are produced by GPT-4o alone, the same model family that generates the candidate programs. If the model systematically misinterprets the prompt, its test inputs share that misinterpretation, so programs in the largest equivalence class can all implement the same wrong specification. The paper's own Limitations section concedes that behavioral agreement on a finite, auto-generated test set is assumed to be a faithful proxy for universal correctness, and the HC and HA error categories in Table 2 are direct instances of this failure mode. Eq. (7) requires D(B) = delta > 0 for the bug set B that matters, but no evidence is given that the self-generated distribution assigns positive measure to the hidden-test bug set.
  3. [Section 4.2, Table 2] The thresholds tau0%, tau1%, and tau2% are defined in Table 2 as thresholds tuned so that the returned-answer error rate does not exceed 0%, 1%, or 2% on the same 219 LiveCodeBench tasks. Consequently, the abstract's claim that the error rate drives it to 0% at a conservative threshold while still answering 15.6% of prompts is an in-sample statement obtained by fitting tau to the test set. No held-out split, cross-validation, or separate calibration set is reported, so the result does not support a general prediction of zero error at a fixed threshold.
  4. [Section 4.2, Figure 1b] The claim that the accuracy-coverage trade-off follows the exponential tail predicted by Eq. 5 is not quantitatively supported. Eq. (5) is an upper bound on a tail probability as a function of n and the gap tau - C, whereas Figure 1b reports observed correctness rates on 219 tasks with a single n = 100 and no confidence intervals. A quantitative comparison, for example fitting the predicted curve or reporting error bars, would be needed to substantiate this statement.
minor comments (4)
  1. [Section 3.1] The formalism assumes programs are deterministic functions f_phi: X -> Y, but many LLM-generated programs use randomness (for example, random.choice or hash-based iteration); the paper does not explain how output vectors are defined for such programs.
  2. [Abstract and Section 4] The abstract claims that the verifier preserves baseline pass@1 on solvable tasks, but Table 2 reports 'Expected' accuracy rather than pass@1, so the reader cannot verify that pass@1 is preserved; please clarify the metric and provide the corresponding comparison.
  3. [Figure 1] The figure caption describes cumulative wrong/correct step curves but does not define how they aggregate the 100 samples per task or how the y-axis is normalized; adding this detail would improve interpretability.
  4. [Appendix B] The prompt for test-case generation asks GPT-4o for a comprehensive list of valid input test cases, but there is no check on the diversity or validity of these inputs; a mis-parsed or degenerate test input could make all programs fall into one cluster trivially, and this risk is not discussed.

Circularity Check

2 steps flagged · score 6.0 of 10

The headline 2% and 0% error rates are in-sample operating points obtained by tuning τ on the same benchmark, and the 'exact confidence' is definitionally the frequency of agreement on tests generated by the same model, so the central reliability claims partially reduce to their own inputs.

  1. fitted input called prediction [Table 2 caption; Section 4.2 (LiveCodeBench); Abstract]
    "Threshold accuracies are measured after clustering with τ tuned so that the returned-answer error rate does not exceed 0%, 1%, or 2%; tasks falling below the threshold are returned as UNKNOWN."

    The 2% and 0% residual-error figures are produced by locating the threshold that makes the observed error on the same LiveCodeBench tasks at most 2% or 0%, then reporting that observed error. The Abstract's claim that the verifier 'drives it to 0% at a conservative threshold' is therefore a fitted, in-sample description, not an out-of-sample prediction. The reported error rates are forced by construction: τ is selected after seeing the benchmark labels so that the error does not exceed the target, so 'achieving' the target is tautological and is not covered by the exponential bound in Eq. 5, which concerns a pre-specified threshold and an unknown true class mass.

  2. self definitional [Section 3.1-3.4 (Eqs. 2, 6); Section 4; Limitations]
    "Because universal equivalence is undecidable, we approximate it with agreement on a finite, automatically generated test set and show that the resulting estimator inherits exponential reliability guarantees. ... The self-generated test suites are produced by GPT-4o alone."

    Eq. 2 defines confidence C(φ) as the probability that a sampled program is equivalent to φ, and Eq. 6 redefines equivalence as agreement on the self-generated test set. The 'empirical mass of the largest cluster' is therefore, by construction, the empirical frequency of mutual agreement on those tests, not a derived estimate of correctness on hidden tests. Because the same model (GPT-4o) generates both candidate programs and test inputs, a shared misinterpretation of the prompt makes the largest test-equivalence class large and confidently wrong. The paper's Limitations states the missing premise explicitly: 'behavioral agreement on a finite, auto-generated test set is a faithful proxy for universal correctness.' Eq.

full rationale

The mathematical concentration argument (Eq. 5 and Appendix A) is sound as a statement about Bernoulli sampling from a fixed equivalence-class mass C, and the method is not merely a renamed existing result. However, the paper's two central reliability claims do not follow from that argument. First, the abstract's headline error reductions from ~65% to 2% and to 0% are obtained after tuning τ on the same benchmark tasks until the observed error falls below the target; Table 2 states this explicitly, so the 'prediction' is a fitted operating point. Second, the equivalence relation used in the estimate (Eq. 6) is defined by agreement on tests generated by GPT-4o, the same model family that produces the candidate programs; C in Eq. 5 is then the mass of that test-agreement class, not a probability of correctness on universal or hidden-test behavior. The limitations section concedes the load-bearing proxy assumption. The 0% claim is also an in-sample observation on a small filtered subset of tasks, not a guaranteed bound. I found no load-bearing self-citation chain or imported uniqueness theorem; the circularity here is definitional substitution and fitted-threshold reporting rather than citation laundering. Overall, the central claims reduce partly to their own inputs, so a score of 6 is appropriate rather than 0-2.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method introduces no new entities. The central claim rests on two domain assumptions (single-correct-output tasks, test-proxy fidelity), one ad hoc distributional assumption about test coverage, and the i.i.d. sampling assumption. The threshold tau is a free parameter tuned on the evaluation set, and the test count m is unreported.

free parameters (3)
  • acceptance threshold tau = tuned per operating point (tau0%, tau1%, tau2%)
    In Section 4.2 and Table 2, tau is tuned so that the returned-answer error rate on the benchmark does not exceed 0%, 1%, or 2%; the zero-error result is therefore a fit to the evaluation set.
  • sample size n = 100 for LiveCodeBench, 50 for HumanEval
    Chosen by the authors; Section 4 states 50 per model, with HumanEval using only GPT-4o, while Section 3.3 says 'we take n = 100', an inconsistency.
  • number of test inputs m = not reported
    The number of self-generated test inputs per task is never stated, which prevents reproduction and makes the strength of the test-based oracle unknowable.
assumptions (4)
  • domain assumption Each programming task has a single correct output for any given input
    Section 3.1 asserts this and excludes tasks with multiple valid outputs; the authors acknowledge some benchmark tasks violate this assumption.
  • domain assumption Behavioral agreement on a finite, auto-generated test set is a faithful proxy for universal correctness
    Stated explicitly in Limitations; this is the core premise that makes the empirical confidence estimate meaningful.
  • ad hoc to paper Test inputs are drawn from a task-dependent distribution D that has positive measure on any bug set B
    Section 3.4 assumes D(B)=delta>0 for any behavioral divergence and that m random tests will likely hit B; the actual tests are LLM-generated, and no argument shows they cover hidden-test bug regions.
  • standard math Candidate programs are sampled i.i.d. from the LLM's predictive distribution P
    Eq. 3 requires i.i.d. samples; temperature sampling approximates this but the paper does not specify decoding parameters.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Eliminating Hallucination-Induced Errors in LLM Code Generation with Functional Clustering." pith.science (2026). https://pith.science/paper/RMGZQHIF

@misc{pith2026250611021,
  author       = {Pith},
  title        = {Pith review of: Eliminating Hallucination-Induced Errors in LLM Code Generation with Functional Clustering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RMGZQHIF}},
  note         = {Machine review of arXiv:2506.11021}
}
read the original abstract

Modern code-generation LLMs can already solve a large fraction of programming problems, yet they still hallucinate subtle bugs that make their outputs unsafe for autonomous deployment. We present functional clustering, a black-box wrapper that eliminates nearly all hallucination-induced errors while providing a tunable confidence score. The wrapper samples many candidate programs, executes each on a self-generated test suite, and clusters candidates whose I/O behavior is identical; the empirical mass of the largest cluster serves as an exact confidence estimate. A single scalar threshold on this estimate lets users trade coverage for reliability with exponential guarantees. On LiveCodeBench our verifier preserves baseline pass@1 on solvable tasks yet slashes the error rate of returned answers from ~65% to 2%, and drives it to 0% at a conservative threshold while still answering 15.6% of prompts. Manual audits show that the few residual mistakes stem from prompt misinterpretation, not random generation noise, narrowing future work to specification clarity. Because the method requires only sampling and sandbox execution, it applies unchanged to closed-source APIs and future models, offering a practical path toward dependable, autonomous code generation. Our code is available on Github (https://github.com/20ChaituR/functional-clustering).

Figures

Figures reproduced from arXiv: 2506.11021 by the authors.

Figure 1
Figure 1. Correctness versus estimated confidence across datasets. Colors encode correctness; Each point is a response that is either correct or wrong, with its x position denoting the model’s confidence in the response, and its y position meaningless. Dashed lines mark confidence thresholds, each guaranteeing an empirical error rate of at most the indicated percentage for returned answers. The Cum. Wrong / Cum. Correct step … view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 6 canonical work pages

  1. [1]

    The claude 3 model family: Opus, sonnet, haiku, March 2024

    Anthropic. The claude 3 model family: Opus, sonnet, haiku, March 2024. URL https: //www.anthropic.com/news/claude-3-family

    work page 2024

  2. [2]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    arXiv 2021

  3. [3]

    Calibration of pre-trained transformers

    Shrey Desai and Greg Durrett. Calibration of pre-trained transformers. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors,Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 295–302, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.21. URL http...

    doi:10.18653/v1/2020.emnlp-main.21 2020

  4. [4]

    Detecting hallucinations in large language models using semantic entropy

    Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625–630, June 2024. ISSN 1476-4687. doi: 10.1038/s41586-024-07421-0. URL https://doi.org/10.1038/ s41586-024-07421-0

    doi:10.1038/s41586-024-07421-0 2024

  5. [5]

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qiang- long Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transac- tions on Information Systems , November 2024. ISSN 1558-2868. doi: 10.1145/3703155. URL ...

    doi:10.1145/3703155 2024

  6. [6]

    Livecodebench: Holistic and contamination free evaluation of large language models for code, 2024

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code, 2024. URL https://arxiv.org/abs/ 2403.07974

    arXiv 2024

  7. [7]

    Language models (mostly) know what they know, 2022

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec,...

    arXiv 2022

  8. [8]

    Semantic uncertainty: Linguistic invari- ances for uncertainty estimation in natural language generation

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invari- ances for uncertainty estimation in natural language generation. In International Conference on Learning Representations (ICLR) , 2023. URL https://openreview.net/forum?id= VD-AYtP0dve

    work page 2023

Show all 27 references
  1. [9]

    Teaching models to express their uncertainty in words

    Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. Transactions on Machine Learning Research, 2022. Available athttps://openreview. net/forum?id=8s8K2UZGTZ

    2022

  2. [10]

    Exploring and evaluating hallucinations in llm-powered code generation,

    Fang Liu, Yang Liu, Lin Shi, Houkun Huang, Ruifeng Wang, Zhen Yang, Li Zhang, Zhongqi Li, and Yuchi Ma. Exploring and evaluating hallucinations in llm-powered code generation,

  3. [11]

    Litcab: Lightweight language model calibration over short and long-form responses

    Xin Liu, Muhammad Khalifa, and Lu Wang. Litcab: Lightweight language model calibration over short and long-form responses. In International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=jH67LHVOIO

    2024

  4. [12]

    Estimating llm uncertainty with logits, 2025

    Huan Ma, Jingdong Chen, Guangyu Wang, and Changqing Zhang. Estimating llm uncertainty with logits, 2025. URL https://arxiv.org/abs/2502.00290

    2025 arXiv

  5. [13]

    Introducing gpt-4.1 in the api, April 2025

    OpenAI. Introducing gpt-4.1 in the api, April 2025. URL https://openai.com/index/ gpt-4-1/

    2025

  6. [14]

    OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M ˛ adry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex ...

    2024 arXiv

  7. [15]

    Competitive programming with large reasoning models, 2025

    OpenAI, :, Ahmed El-Kishky, Alexander Wei, Andre Saraiva, Borys Minaiev, Daniel Selsam, David Dohan, Francis Song, Hunter Lightman, Ignasi Clavera, Jakub Pachocki, Jerry Tworek, Lorenz Kuhn, Lukasz Kaiser, Mark Chen, Max Schwarzer, Mostafa Rohaninejad, Nat McAleese, o3 contrib...

    2025 arXiv

  8. [16]

    Semantic density: Uncertainty quantification for large language models through confidence measurement in semantic space, 2024

    Xin Qiu and Risto Miikkulainen. Semantic density: Uncertainty quantification for large language models through confidence measurement in semantic space, 2024. URL https: //arxiv.org/abs/2405.13845

    2024 arXiv

  9. [17]

    Assessing correctness in llm-based code generation via uncertainty estimation, 2025

    Arindam Sharma and Cristina David. Assessing correctness in llm-based code generation via uncertainty estimation, 2025. URL https://arxiv.org/abs/2502.11620

    2025 arXiv

  10. [18]

    Magis: Llm-based multi-agent framework for github issue resolution

    Wei Tao, Yucheng Zhou, Yanlin Wang, Wenqiang Zhang, Hongyu Zhang, and Yu Cheng. Magis: Llm-based multi-agent framework for github issue resolution. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Pro...

    2024

  11. [19]

    Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D. Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models finetuned with human feedback. In Proceedings o...

    2023 doi

  12. [20]

    Codehalu: Investigating code hallucinations in llms via execution- based verification, 2024

    Yuchen Tian, Weixiang Yan, Qian Yang, Xuandong Zhao, Qian Chen, Wen Wang, Ziyang Luo, Lei Ma, and Dawn Song. Codehalu: Investigating code hallucinations in llms via execution- based verification, 2024. URL https://arxiv.org/abs/2405.00253

    2024 arXiv

  13. [21]

    M Towhidul Islam Tonmoy, S M Mehedi Zaman, Vinija Jain, Anku Rani, Vipula Rawte, Aman Chadha, and Amitava Das

    S. M Towhidul Islam Tonmoy, S M Mehedi Zaman, Vinija Jain, Anku Rani, Vipula Rawte, Aman Chadha, and Amitava Das. A comprehensive survey of hallucination mitigation techniques in large language models, 2024. URL https://arxiv.org/abs/2401.01313

    2024 arXiv

  14. [22]

    Llm perfor- mance assessment in computer science graduate entrance exams

    Arya VarastehNezhad, Reza Tavasoli, Mostafa Masumi, and Fattaneh Taghiyareh. Llm perfor- mance assessment in computer science graduate entrance exams. In 2024 11th International Symposium on Telecommunications (IST), pages 232–237, 2024. doi: 10.1109/IST64061.2024. 10843484

    2024

  15. [23]

    Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms, 2024

    Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms, 2024. URL https://arxiv.org/abs/2306.13063. 12

    2024 arXiv

  16. [24]

    Mitigating llm hallucinations via conformal abstention, 2024

    Yasin Abbasi Yadkori, Ilja Kuzborskij, David Stutz, András György, Adam Fisch, Arnaud Doucet, Iuliya Beloshapka, Wei-Hung Weng, Yao-Yuan Yang, Csaba Szepesvári, Ali Taylan Cemgil, and Nenad Tomasev. Mitigating llm hallucinations via conformal abstention, 2024. URL https://arxi...

    2024 arXiv

  17. [25]

    Wong, Emine Yilmaz, Shuming Shi, and Zhaopeng Tu

    Fanghua Ye, Yang MingMing, Jianhui Pang, Longyue Wang, Derek F. Wong, Emine Yilmaz, Shuming Shi, and Zhaopeng Tu. Benchmarking llms via uncertainty quantification, 2024. URL https://arxiv.org/abs/2401.12794

    2024 arXiv

  18. [26]

    explanation

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong W...

    2025 arXiv

  19. [2024]

    URL https://arxiv.org/abs/2404.00971. 10

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.