Pith. sign in

REVIEW 5 major objections 6 minor 79 references

A black-box uncertainty score built from execution consensus can flag wrong code from closed-source LLMs before tests run.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 06:35 UTC pith:O67IPUWE

load-bearing objection Execution-grounded uncertainty for code LLMs is a promising and novel idea, but the metric isn't always well-defined because the similarity matrix can have negative eigenvalues — fix that before trusting the -0.98. the 5 major comments →

arxiv 2607.12273 v2 pith:O67IPUWE submitted 2026-07-14 cs.SE cs.AIcs.CL

Code-MUE: Measuring Code LLMs' Uncertainty through Execution-based Semantic Interaction Graphs

Xiaoning Ren , Yinxing Xue , Lei Ma , Yuheng Huang This is my paper
classification cs.SE cs.AIcs.CL
keywords Code LLMsuncertainty estimationVon Neumann entropysemantic interaction graphexecution-basedblack-boxcode generationselective prediction
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Code-MUE tries to establish that a purely black-box uncertainty measure for code-generating LLMs can be computed by sampling many programs, running them on a small set of generated inputs, and measuring how often they behave identically. The paper reports that the Von Neumann entropy of this behavioral-agreement graph correlates strongly with functional correctness, with Spearman correlations up to -0.98, far exceeding text-based and embedding-based baselines. If true, the method would let systems abstain from uncertain predictions or select a good answer without access to logits, ground-truth tests, or internal model states. The core insight is that code's executability gives a semantic ground truth that text similarity misses: two programs that look different may behave alike, and two that look alike may behave oppositely.

Core claim

The central claim is that uncertainty in code LLMs can be grounded in observable runtime behavior rather than textual or embedding similarity. Code-MUE samples N programs per prompt, executes them on K hybrid probe inputs (half boundary values derived from static types, half LLM-synthesized nominal inputs), and builds a Semantic Interaction Graph whose edge weights are the fraction of inputs on which two programs produce identical execution outcomes: same return value, same exception type, or timeout. The graph's adjacency matrix is trace-normalized into a density matrix, and its Von Neumann entropy, normalized by ln(N), yields a scalar uncertainty score in [0,1]. The paper reports that this

What carries the argument

The Semantic Interaction Graph: a weighted graph whose nodes are sampled programs and whose edge weights are the fraction of probe inputs on which two programs produce identical execution outcomes. This graph is converted to a density matrix by trace-normalizing the similarity matrix, and the Von Neumann entropy of its eigenvalue spectrum quantifies whether the ensemble is concentrated in one behavioral consensus or split into multiple competing modes; syntax-invalid programs become isolated nodes, and timeouts are conservatively treated as non-agreement.

Load-bearing premise

The load-bearing premise is that the K=20 probe inputs (half boundary values, half LLM-synthesized nominal cases) hit the semantic regions where generated programs differ; if they miss those regions, behaviorally distinct programs merge and uncertainty is under-reported.

What would settle it

Take an ensemble of sampled programs and two probe sets: the default K=20 hybrid inputs and a larger adversarial set of random inputs. If Code-MUE assigns low uncertainty to ensembles that disagree on the adversarial inputs, the probe-coverage assumption fails; a benchmark measuring the correlation between U and correctness on hidden tests outside the generated input distribution would settle the central claim.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Code-MUE enables uncertainty estimation for closed-source LLMs without logits or internal states, using only sampled outputs and local execution.
  • Selective prediction works: rejecting the top 10–20% most uncertain samples sharply reduces error rates in the reported risk-coverage curves.
  • Unsupervised response selection via eigenvector centrality picks a correct program more often than pass@1 alone, often exceeding 80% accuracy on solvable tasks.
  • The metric separates overconfident errors (low uncertainty, wrong) from genuine epistemic uncertainty (high uncertainty), helping flag systematic model bias.
  • Performance degrades gracefully with smaller budgets: N=25 retains most of the quality of N=100, and even K=4 probe inputs give a meaningful signal.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Extension: The method is only as strong as its probe generator. A testable refinement would replace the static type-based boundary map with search-based input generation — fuzzing or symbolic execution — and measure whether the Spearman correlation improves on adversarial tasks.
  • Extension: The centrality-based selection assumes correct programs form one dense clique. For tasks with several equally valid algorithmic families, the graph may split into multiple dense clusters; a multi-centroid or voting strategy would be a natural variant to test.
  • Extension: The study executes Python only. Porting the pipeline to other sandboxed languages would require re-deriving the type-to-boundary-value mapping and exception taxonomy, a concrete next experiment.
  • Extension: The paper collapses uncertainty into a single scalar. The eigenvalue spectrum itself may carry more information — for instance, the number of significant modes could distinguish 'two competing solutions' from 'many scattered failures', which the scalar entropy conflates.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces Code-MUE, a black-box uncertainty estimation method for code LLMs. It samples N candidate programs for a prompt, executes them on K synthesized probe inputs, builds a pairwise behavioral-agreement matrix W, trace-normalizes it as rho = W/N, and reports the Von Neumann entropy of rho as an uncertainty score. The authors evaluate the method on eight LLMs and four code benchmarks (HumanEval, MBPP, QuixBugs, CodeNet), reporting Spearman correlations with functional correctness up to -0.98, AUROC-based discrimination of solved/unsolved instances, and applications to selective prediction and response selection via eigenvector centrality.

Significance. If the claims hold, Code-MUE is a valuable contribution: it is a purely black-box, execution-grounded alternative to token-level or text-similarity uncertainty estimates for code, addressing a real limitation for closed-source models. The paper includes a large evaluation matrix, a concrete pipeline, sensitivity analyses, and a public code link, which are strengths. The central idea of using execution-based behavioral agreement graphs is novel in this combination and the reported effect sizes are large. However, the current manuscript leaves the formal definition of the metric incomplete and the evaluation design has a shared-sample issue that affects the interpretation of the headline correlations; these need to be addressed before the claims can be accepted.

major comments (5)
  1. [§5.2.1/Table 2] The correctness labels and the uncertainty score are computed from the same N=100 sampled generations: 'Solved if at least one of the N sampled candidates passes all tests' while W is built from those same N programs. This shared-sample design makes the reported Spearman/AUROC quantify how well an ensemble statistic predicts a property of the same ensemble, not how well it predicts the correctness of an independent generation, and it can inflate the headline numbers. Please add a held-out evaluation (e.g., compute U on N samples and label an additional independent sample) or explicitly temper the 'proxy for functional correctness' claim.
  2. [§4.3.2/Eq. (3)] The formal definition of W is incomplete and internally contradictory. Eq. (3) defines agreement via I(o_i,k == o_j,k) and then lists only equal return values and equal exception types; the text immediately says matching timeouts do NOT count, but the indicator would naturally count two timeouts as equal. Timeout-vs-value is also not defined inside Eq. (3), and syntax-error isolation is specified outside the equation. A stress-test example claiming a negative eigenvalue misapplies the stated rule (under that rule W13=0 in the example). With a precise outcome-label mapping (timeouts/syntax errors mapped to unique per-program labels, exceptions mapped to their type), each input contributes a PSD matrix and W is PSD. Please replace the prose with a complete formal definition and include the PSD proof so that the entropy is well-defined and reproducible.
  3. [§5.2.2/Table 2] The claim of 'significantly outperforming' baselines is not supported by significance tests, confidence intervals, or effect sizes. Several cells contradict the claim (e.g., QuixBugs, Llama3.1-8B AUROC: Ours 0.591 vs Lexical 0.667; Qwen2.5-7B: Ours 0.797 vs Lexical 0.897; GPT-4.1-nano: Ours 0.677 vs Embedding 0.684). The QuixBugs sample size is acknowledged as only 40, but the paper should quantify variance (bootstrap or per-task confidence intervals) and run paired tests across the 32 model/dataset cells before claiming broad superiority.
  4. [§4.2.2/§7] The probe set is load-bearing: it is what allows execution traces to expose semantic divergence. The paper admits in §7 that 'LLM-generated inputs risk coverage bias,' but no empirical evidence is provided about how often the K=20 probes distinguish behaviorally distinct programs. Without such evidence, an ensemble of semantically different programs can collapse into one consensus cluster (low U) if the probes miss the distinguishing inputs. Please add a probe-coverage analysis (e.g., fraction of reference-test behaviors distinguished by the probe set, or mutation-style experiments) and discuss how probe construction affects the reported correlations.
  5. [§5.3.1/Figure 4] Response selection is evaluated only against Pass@1 of a single generation ('Baseline (Pass@1)'). Pass@1 is not a reranking baseline; the standard comparison would be to majority voting or cluster-based self-consistency using the same N samples. Without such comparisons, the claimed utility of SIG-based eigenvector centrality over existing multi-sample selection methods is not established.
minor comments (6)
  1. [§4.4.2] The assertion lambda_k >= 0 is stated without proof. Under the clarified outcome-label mapping it is true (each input contributes a sum of PSD rank-one blocks), so a one-line proof should be added.
  2. [§4.3.1] Please specify how compound return values (lists, dicts, tuples) are compared in the execution signature (deep equality? serialized string? repr?). The current text only mentions 'numerical/string value'.
  3. [Abstract and §1] The phrase 'eight state-of-the-art LLMs' overstates the evaluated models, several of which are small/cheap tiers (GPT-4.1-nano, Claude-3-Haiku, 7B open models). Suggest 'eight representative LLMs'.
  4. [Figure 2/Eq. (6)] Figure 2 labels entropy as 's in [0,1]' while the text defines H_spec and then U in Eq. (6). Make the notation consistent in the figure and the caption.
  5. [Table 2] The bold/underlining legend says 'best' and 'second-best,' but several AUROC rows contain ties (e.g., Gemini-2.5-Flash on QuixBugs: Ours 0.821 and Lexical 0.821). Mark ties explicitly or clarify the convention.
  6. [§4.3.2] There is a missing space in 'we require only validinputs to observe execution traces'.

Circularity Check

0 steps flagged

No significant circularity: U(P) is a defined function of execution traces, not fitted to correctness labels; self-citations are motivational, not load-bearing.

full rationale

The central derivation is self-contained. U(P) is the Von Neumann entropy of rho = W/Tr(W) (Eqs. 4-6), with W_ij defined in Eq. 3 as the mean number of probe inputs on which two sampled programs produce identical observable behavior (return value, exception type, or timeout marker). No parameter is fitted to correctness: N=100, K=20, and temperature are fixed hyperparameters, and the probe inputs are generated without expected outputs ('their expected outputs are not used when constructing the semantic similarity matrix', Sec. 5.1). The Spearman/AUROC results in Table 2 are thus external empirical evaluations, not consequences of the metric's definition. This is confirmed by the paper's own failure-mode analyses: low U can accompany Pass@1=0 (Sec. 5.4.1, the 'even digits' overconfident-error case) and high U can accompany correct but inefficient solutions (Sec. 5.4.2, the timeout case), so U is not definitionally tied to correctness. Self-citations from the authors' group ([28], [29], [30], [61], [68]) appear in related-work motivation and as support for the Length baseline; the core spectral-entropy construction rests on external references ([10], [12], [17], [19], [36]) and standard execution-based SE practice. No uniqueness theorem or ansatz is imported from self-citations. A non-circular soundness caveat: the timeout rule in Sec. 4.3.2 can produce a non-positive-semidefinite W (e.g., two mutual timeouts and one shared value on a single probe), so the Sec. 4.4.2 assertion that all eigenvalues satisfy lambda_k >= 0 is not guaranteed; this affects whether the Von Neumann entropy is well-defined for such matrices, but it does not make the metric circular.

Axiom & Free-Parameter Ledger

6 free parameters · 6 axioms · 2 invented entities

The method adds no fitted constants to compute VNE, but it introduces several hand-chosen hyperparameters (N, K, temperature, timeout, probe split, boundary values) and depends on domain assumptions about execution signatures, probe adequacy, and the consensus hypothesis. The most fragile is the unproven positive-semidefiniteness of W/N, which is needed for the Von Neumann entropy to be mathematically valid.

free parameters (6)
  • Number of sampled programs N = 100 (default; sensitivity 5–100)
    Ensemble size for Monte Carlo sampling; graph and entropy are computed over N nodes. Table 3 shows diminishing returns; default is hand-chosen.
  • Number of probe inputs K = 20 (default; sensitivity 4–20)
    Size of the hybrid test-input pool; determines how finely execution signatures are resolved; default hand-chosen.
  • Sampling temperature = 1.0
    Decoding temperature to expose variability; fixed across models rather than fitted.
  • Sandbox timeout = 5 seconds
    Executions beyond this are marked Timeout and contribute 0 to cross-program similarity; affects entropy for inefficient programs.
  • Edge/nominal probe split = 1:1
    Budget allocation between type-based boundary inputs and LLM-synthesized nominal inputs.
  • Boundary value set = 0, -1, sys.maxsize, 1e-9, empty/singleton sequences
    Hand-picked corner values for numeric and sequence types used to trigger boundary behaviors.
axioms (6)
  • domain assumption Sandboxed execution of generated code yields an observable signature (return value, exception type, timeout) that reflects program semantics.
    Invoked in Section 4.3.1; the entire similarity matrix is built from these signatures.
  • domain assumption Behavioral agreement on the probe set approximates functional equivalence; correct programs form a dense consensus cluster and incorrect programs fragment.
    Stated as the Semantic Consensus Hypothesis in Section 5.3.1 and used for both entropy interpretation and centrality-based selection.
  • ad hoc to paper W/N is a valid density matrix with eigenvalues lambda_k >= 0, so Von Neumann entropy is well-defined.
    Section 4.4.2 assumes this after Eq. (4); nonnegative similarity matrices are not generally positive semidefinite, so the assumption is unproven and can fail (e.g., a weighted path where endpoints do not agree).
  • domain assumption The K=20 hybrid probe set sufficiently discriminates semantically different programs.
    Needed for W to separate behavioral modes; the paper acknowledges coverage-bias risk in Section 7 (Internal Validity).
  • domain assumption Temperature-1.0 stochastic sampling of N outputs approximates the model's posterior predictive distribution over programs.
    Section 4.2.1 treats the sampled ensemble as the empirical proxy for the model's semantic distribution.
  • domain assumption Benchmark hidden tests define ground-truth functional correctness for evaluation.
    Section 5.1 uses pass@k and instance-level Pass@1 computed from benchmark tests.
invented entities (2)
  • Semantic Interaction Graph (SIG) no independent evidence
    purpose: Graph abstraction with programs as nodes and execution-trace agreement as edge weights; the basis for spectral uncertainty computation.
    Defined entirely within the paper; no external falsifiable handle beyond its own experiments.
  • Latent Semantic Modes no independent evidence
    purpose: Interpretation of eigenvalues of rho as probabilities over functional clusters of the generated solutions.
    Purely interpretive framing; not operationalized independently of the eigenvalue spectrum.

pith-pipeline@v1.3.0-alltime-deepseek · 25452 in / 17254 out tokens · 163463 ms · 2026-08-02T06:35:28.318881+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Code-MUE: Measuring Code LLMs' Uncertainty through Execution-based Semantic Interaction Graphs." pith.science (2026). https://pith.science/paper/O67IPUWE

@misc{pith2026260712273,
  author       = {Pith},
  title        = {Pith review of: Code-MUE: Measuring Code LLMs' Uncertainty through Execution-based Semantic Interaction Graphs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O67IPUWE}},
  note         = {Machine review of arXiv:2607.12273}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

As Code Large Language Models (LLMs) become central to modern software engineering, their inherent stochasticity poses significant real-world risks, where even minor errors can lead to severe functional, security, or safety consequences. Reliable automation, therefore, demands the ability to distinguish between confident, well-supported predictions and stochastic guessing. However, existing uncertainty estimation methods face a critical gap: white and grey-box techniques are often inapplicable to closed-source models, while standard "black-box" text metrics fail to capture the unique fragility of code, where syntactic variation does not always imply semantic divergence. To bridge this syntax-semantics gap, we introduce Code-MUE, a purely black-box framework that measures uncertainty through execution-based Semantic Interaction Graphs. Different from prior approaches that rely on superficial textual similarity, Code-MUE grounds uncertainty in observable runtime behavior, calculating the Von Neumann entropy of the solution space to quantify global semantic diversity. A large-scale empirical study across eight state-of-the-art LLMs demonstrates that Code-MUE achieves a strong negative correlation with functional correctness (Spearman's correlation up to -0.98), significantly outperforming lexical and embedding-based baselines while enabling robust risk detection and selective prediction in practical workflows.

Figures

Figures reproduced from arXiv: 2607.12273 by Lei Ma, Xiaoning Ren, Yinxing Xue, Yuheng Huang.

Figure 1
Figure 1. Figure 1: Motivating example showing the limitations of surface-level and embedding-based similarity across [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the proposed Graph-based Semantic Uncertainty Quantification framework. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Risk-Coverage curves on HumanEval across eight LLMs. The plots show the trade-off between coverage and selective risk. A lower curve indicates a more effective uncertainty estimator; our method (solid red line) consistently achieves the lowest risk profile and AURC scores compared to all baselines. CodeNet HumanEval MBPP QuixBugs 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy 0.64 +6% 0.87 +20% 0.82 +40% 0.74 +26% DeepS… view at source ↗
Figure 4
Figure 4. Figure 4: Reranking performance of Spectral Graph Entropy. Bars show selection accuracy across eight LLMs and four datasets. By selecting responses with the highest eigenvector centrality, our method con￾sistently identifies correct solutions (often > 80%), validating that correct implementations form a dense semantic consensus while incorrect ones diverge. 5.4 RQ3: Case Studies on Uncertainty-Accuracy Misalignments… view at source ↗
Figure 5
Figure 5. Figure 5: DeepSeek-Coder consistently misinterprets "even digits" as “even integers”. [PITH_FULL_IMAGE:figures/full_fig_p017_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: The model generates a correct but exponentially slow solution, leading to universal timeouts. [PITH_FULL_IMAGE:figures/full_fig_p017_6.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

79 extracted references · 3 canonical work pages

  1. [1]

    Website for This Work

    2026. Website for This Work. https://github.com/hnurxn/Code-Uncertainty

    2026

  2. [2]

    Vibhor Agarwal, Yulong Pei, Salwa Alamir, and Xiaomo Liu. 2024. Codemirage: Hallucinations in code generated by large language models.arXiv preprint arXiv:2408.08333(2024)

    Pith/arXiv arXiv 2024

  3. [3]

    DeepSeek AI. 2024. deepseek-ai/deepseek-coder-6.7b-instruct (model card). https://huggingface.co/deepseek-ai/ deepseek-coder-6.7b-instruct. Accessed: 2026-01-26

    2024

  4. [4]

    Mistral AI. 2024. mistralai/Mistral-7B-Instruct-v0.3 (model card). https://huggingface.co/mistralai/Mistral-7B-Instruct- v0.3. Accessed: 2026-01-26

    2024

  5. [5]

    Miltiadis Allamanis, Sheena Panthaplackel, and Pengcheng Yin. 2024. Unsupervised evaluation of code LLMs with round-trip correctness(ICML’24). JMLR.org, Article 44, 17 pages

    2024

  6. [6]

    2024.Claude 3 Haiku Large Language Model

    Anthropic. 2024.Claude 3 Haiku Large Language Model. Anthropic. https://www.anthropic.com/news/claude-3-haiku

    2024

  7. [7]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models.arXiv preprint arXiv:2108.07732 (2021)

    Pith/arXiv arXiv 2021

  8. [8]

    Yavuz Faruk Bakman, Duygu Nur Yaldiz, Sungmin Kang, Tuo Zhang, Baturalp Buyukates, Salman Avestimehr, and Sai Praneeth Karimireddy. 2025. Reconsidering LLM Uncertainty Estimation Methods in the Wild. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Wanxiang Che, Joyce Nabende, Ekaterina Shu...

    doi:10.18653/v1/2025.acl-lon 2025

  9. [9]

    Mohammad Beigi, Ying Shen, Runing Yang, Zihao Lin, Qifan Wang, Ankith Mohan, Jianfeng He, Ming Jin, Chang-Tien Lu, and Lifu Huang. 2024. InternalInspector 𝐼 2: Robust Confidence Estimation in LLMs through Internal States. In Findings of the Association for Computational Linguistics: EMNLP 2024, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Ass...

    doi:10.18653/v1/2024 2024

  10. [10]

    2017.Geometry of quantum states: an introduction to quantum entanglement

    Ingemar Bengtsson and Karol Życzkowski. 2017.Geometry of quantum states: an introduction to quantum entanglement. Cambridge university press

    2017

  11. [11]

    Viola Campos, Robin Kuschnereit, and Adrian Ulges. 2025. Multicalibration for LLM-based Code Generation.arXiv preprint arXiv:2512.08810(2025)

    arXiv 2025

  12. [12]

    Mark Chen, Jerry Tworek, et al . 2021. Evaluating Large Language Models Trained on Code. (2021). arXiv:2107.03374 [cs.LG]

    Pith/arXiv arXiv 2021

  13. [13]

    Trishul M Chilimbi, Ben Liblit, Krishna Mehra, Aditya V Nori, and Kapil Vaswani. 2009. Holmes: Effective statistical debugging via efficient path profiling. In2009 IEEE 31st International Conference on Software Engineering. IEEE, 34–44

    2009

  14. [14]

    Yu-Neng Chuang, Prathusha Kameswara Sarma, Parikshit Gopalan, John Boccio, Sara Bolouki, Xia Hu, and Helen Zhou. 2025. Learning to Route LLMs with Confidence Tokens. InForty-second International Conference on Machine Learning. https://openreview.net/forum?id=U08mUogGDM

    2025

  15. [15]

    2025.Gemini 2.5 Flash-Lite Large Language Model

    Google DeepMind. 2025.Gemini 2.5 Flash-Lite Large Language Model. Google DeepMind. https://blog.google/ innovation-and-ai/models-and-research/google-deepmind/gemini-model-thinking-updates-march-2025/

    2025

  16. [16]

    Ajinkya Deshpande, Anmol Agarwal, Shashank Shet, Arun Iyer, Aditya Kanade, Ramakrishna Bairi, and Suresh Parthasarathy. 2024. Natural Language to Class-level Code Generation by Iterative Tool-augmented Reasoning over Repository. InICML 2024 Workshop on Data-Centric Machine Learning Research. https://openreview.net/forum?id= yqjr7ojVYa

    2024

  17. [17]

    Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. 2024. Detecting hallucinations in large language models using semantic entropy.Nature630, 8017 (2024), 625–630

    2024

  18. [18]

    Yichao Fu, Xuewei Wang, Yuandong Tian, and Jiawei Zhao. 2025. Deep think with confidence.arXiv preprint arXiv:2508.15260(2025)

    Pith/arXiv arXiv 2025

  19. [19]

    Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. Ininternational conference on machine learning. PMLR, 1050–1059

    2016

  20. [20]

    Ido Galil, Mohammed Dabbah, and Ran El-Yaniv. 2023. What Can we Learn From The Selective Prediction And Uncertainty Estimation Performance Of 523 Imagenet Classifiers?. InThe Eleventh International Conference on Learning Representations. https://openreview.net/forum?id=p66AzKi6Xim

    2023

  21. [21]

    Tobias Groot and Matias Valdenegro Toro. 2024. Overconfidence is Key: Verbalized Uncertainty Evaluation in Large Language and Vision-Language Models. InProceedings of the 4th Workshop on Trustworthy Natural Language Processing (TrustNLP 2024), Anaelia Ovalle, Kai-Wei Chang, Yang Trista Cao, Ninareh Mehrabi, Jieyu Zhao, Aram Galstyan, Jwala Dhamala, Anoop ...

    doi:10.18653/v1/2024.trustnlp-1.13 2024

  22. [22]

    David Gros and Prem Devanbu. 2025. Localized Calibrated Uncertainty in Code Language Models.arXiv preprint arXiv:2512.24560(2025). Proc. ACM Softw. Eng., Vol. 3, No. ISSTA, Article ISSTA185. Publication date: October 2026. ISSTA185:22 Xiaoning Ren, Yinxing Xue, Lei Ma, and Yuheng Huang

    arXiv 2025

  23. [23]

    Weinberger

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On Calibration of Modern Neural Networks. In Proceedings of the 34th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 70), Doina Precup and Yee Whye Teh (Eds.). PMLR, 1321–1330

    2017

  24. [24]

    Sudheendra Hangal and Monica S Lam. 2002. Tracking down software bugs using automatic anomaly detection. In Proceedings of the 24th international conference on Software engineering. 291–301

    2002

  25. [25]

    Ishrak Hayet, Adam Scott, and Marcelo d’Amorim. 2024. Chatassert: Llm-based test oracle generation with external tools assistance.IEEE Transactions on Software Engineering51, 1 (2024), 305–319

    2024

  26. [26]

    Wenchong He, Zhe Jiang, Tingsong Xiao, Zelin Xu, and Yukun Li. 2025. A survey on uncertainty quantification methods for deep learning.Comput. Surveys(2025)

    2025

  27. [27]

    Soneya Binta Hossain, Raygan Taylor, and Matthew Dwyer. 2025. Doc2OracLL: Investigating the Impact of Doc- umentation on LLM-Based Test Oracle Generation.Proceedings of the ACM on Software Engineering2, FSE (2025), 1870–1891

    2025

  28. [28]

    Yuheng Huang, Lei Ma, Keizaburo Nishikino, and Takumi Akazaki. 2025. Risk assessment framework for code llms via leveraging internal states. InProceedings of the 33rd ACM International Conference on the Foundations of Software Engineering. 432–443

    2025

  29. [29]

    Yuheng Huang, Jiayang Song, Qiang Hu, Felix Juefei-Xu, and Lei Ma. 2025. AcTracer: Active Testing of Large Language Model via Multi-Stage Sampling.ACM Transactions on Software Engineering and Methodology(2025)

    2025

  30. [30]

    Yuheng Huang, Jiayang Song, Zhijie Wang, Shengming Zhao, Huaming Chen, Felix Juefei-Xu, and Lei Ma. 2025. Look before you leap: An exploratory study of uncertainty analysis for large language models.IEEE Transactions on Software Engineering(2025)

    2025

  31. [31]

    James A Jones and Mary Jean Harrold. 2005. Empirical evaluation of the tarantula automatic fault-localization technique. InProceedings of the 20th IEEE/ACM international Conference on Automated software engineering. 273–282

    2005

  32. [32]

    James A Jones, Mary Jean Harrold, and John Stasko. 2002. Visualization of test information to assist fault localization. InProceedings of the 24th international conference on Software engineering. 467–477

    2002

  33. [33]

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al . 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221(2022)

    Pith/arXiv arXiv 2022

  34. [34]

    Nikita Kotelevskii, Vladimir Kondratyev, Martin Takáč, Eric Moulines, and Maxim Panov. 2025. From Risk to Uncertainty: Generating Predictive Uncertainty Measures via Bayesian Estimation. InThe Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=cWfpt2t37q

    2025

  35. [35]

    Bonan Kou, Shengmai Chen, Zhijie Wang, Lei Ma, and Tianyi Zhang. 2024. Do large language models pay similar attention like human programmers when generating code?Proceedings of the ACM on Software Engineering1, FSE (2024), 2261–2284

    2024

  36. [36]

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation. InThe Eleventh International Conference on Learning Representations. https://openreview.net/forum?id=VD-AYtP0dve

    2023

  37. [37]

    Jiaqi Li, Yixuan Tang, and Yi Yang. 2025. Know the Unknown: An Uncertainty-Sensitive Method for LLM Instruction Tuning. InFindings of the Association for Computational Linguistics: ACL 2025, Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Computational Linguistics, Vienna, Austria, 2972–2989. doi:10.1865...

    work page doi:10.18653/v1/2025.findings-acl.153 2025

  38. [38]

    Lei Li, Hehuan Liu, Yaxin Zhou, ZhaoYang Gui, Xudong Weng, Yi Yuan, Zheng Wei, and Zang Li. 2025. Uncertainty- aware iterative preference optimization for enhanced llm reasoning. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 23996–24012

    2025

  39. [39]

    Ningke Li, Yuekang Li, Yi Liu, Ling Shi, Kailong Wang, and Haoyu Wang. 2024. Drowzee: Metamorphic testing for fact-conflicting hallucination detection in large language models.Proceedings of the ACM on Programming Languages 8, OOPSLA2 (2024), 1843–1872

    2024

  40. [40]

    Rui Li, Jing Long, Muge Qi, Heming Xia, Lei Sha, Peiyi Wang, and Zhifang Sui. 2025. Towards Harmonized Uncertainty Estimation for Large Language Models. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Assoc...

    doi:10.18653/v1/2025.acl-long.1118 2025

  41. [41]

    Yingzhen Li and Yarin Gal. 2017. Dropout inference in bayesian neural networks with alpha-divergences. InInternational conference on machine learning. PMLR, 2052–2061

    2017

  42. [42]

    Derrick Lin, James Koppel, Angela Chen, and Armando Solar-Lezama. 2017. QuixBugs: a multi-lingual program repair benchmark set based on the quixey challenge(SPLASH Companion 2017). Association for Computing Machinery, New York, NY, USA, 55–56. doi:10.1145/3135932.3135941

    arXiv 2017

  43. [43]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Teaching Models to Express Their Uncertainty in Words. Transactions on Machine Learning Research(2022). https://openreview.net/forum?id=8s8K2UZGTZ Proc. ACM Softw. Eng., Vol. 3, No. ISSTA, Article ISSTA185. Publication date: October 2026. Code-MUE: Measuring Code LLMs’ Uncertainty through Execution-based...

    2022

  44. [44]

    Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2024. Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models.Transactions on Machine Learning Research(2024). https://openreview.net/forum? id=DWkJCSxKU5

    2024

  45. [45]

    Chao Liu, Xifeng Yan, Long Fei, Jiawei Han, and Samuel P Midkiff. 2005. SOBER: statistical model-based bug localization. ACM SIGSOFT Software Engineering Notes30, 5 (2005), 286–295

    2005

  46. [46]

    Linyu Liu, Yu Pan, Xiaocheng Li, and Guanting Chen. 2024. Uncertainty estimation and quantification for llms: A simple supervised approach.arXiv preprint arXiv:2404.15993(2024)

    Pith/arXiv arXiv 2024

  47. [47]

    Zhijie Liu, Yutian Tang, Xiapu Luo, Yuming Zhou, and Liang Feng Zhang. 2024. No need to lift a finger anymore? assessing the quality of code generation by chatgpt.IEEE Transactions on Software Engineering50, 6 (2024), 1548–1584

    2024

  48. [48]

    Meta. 2024. meta-llama/Llama-3.1-8B-Instruct (model card). https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct. Accessed: 2026-01-26

    2024

  49. [49]

    Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau

    Sabrina J. Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. 2022. Reducing Conversational Agents’ Overconfi- dence Through Linguistic Calibration.Transactions of the Association for Computational Linguistics10 (2022), 857–872. doi:10.1162/tacl_a_00494

    doi:10.1162/tacl_a_00494 2022

  50. [50]

    Facundo Molina, Alessandra Gorla, and Marcelo d’Amorim. 2025. Test Oracle Automation in the era of LLMs.ACM Transactions on Software Engineering and Methodology34, 5 (2025), 1–24

    2025

  51. [51]

    Alexander Nikitin, Jannik Kossen, Yarin Gal, and Pekka Marttinen. 2024. Kernel language entropy: Fine-grained uncertainty quantification for llms from semantic similarities.Advances in Neural Information Processing Systems37 (2024), 8901–8929

    2024

  52. [52]

    OpenAI. 2025. GPT-4.1 mini model (OpenAI API documentation). https://platform.openai.com/docs/models/gpt-4.1- mini. Accessed: 2026-01-26

    2025

  53. [53]

    OpenAI. 2025. GPT-4.1 nano model (OpenAI API documentation). https://platform.openai.com/docs/models/gpt-4.1- nano. Accessed: 2026-01-26

    2025

  54. [54]

    Shuyin Ouyang, Jie M Zhang, Mark Harman, and Meng Wang. 2025. An empirical study of the non-determinism of chatgpt in code generation.ACM Transactions on Software Engineering and Methodology34, 2 (2025), 1–28

    2025

  55. [55]

    Rangeet Pan, Ali Reza Ibrahimzada, Rahul Krishna, Divya Sankar, Lambert Pouguem Wassi, Michele Merler, Boris Sobolev, Raju Pavuluri, Saurabh Sinha, and Reyhaneh Jabbarvand. 2024. Lost in translation: A study of bugs introduced by large language models while translating code. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering. 1–13

    2024

  56. [56]

    Ruchir Puri, David S Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, Veronika Thost, Luca Buratti, Saurabh Pujar, Shyam Ramji, Ulrich Finkler, Susan Malaika, and Frederick Reiss. 2021. CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks. InThirty...

    2021

  57. [57]

    Yuhao Qing, Boyu Zhu, Mingzhe Du, Zhijiang Guo, Terry Yue Zhuo, Qianru Zhang, Jie Zhang, Heming Cui, Siu Ming Yiu, Dong Huang, et al. 2026. Effibench-x: A multi-language benchmark for measuring efficiency of llm-generated code.Advances in Neural Information Processing Systems38 (2026)

    2026

  58. [58]

    Xin Qiu and Risto Miikkulainen. 2024. Semantic density: Uncertainty quantification for large language models through confidence measurement in semantic space.Advances in neural information processing systems37 (2024), 134507–134533

    2024

  59. [59]

    Arindam Sharma and Cristina David. 2025. Assessing Correctness in LLM-Based Code Generation via Uncertainty Estimation.arXiv preprint arXiv:2502.11620(2025)

    Pith/arXiv arXiv 2025

  60. [60]

    Ola Shorinwa, Zhiting Mei, Justin Lidard, Allen Z Ren, and Anirudha Majumdar. 2025. A survey on uncertainty quantification of large language models: Taxonomy, open research challenges, and future directions.Comput. Surveys (2025)

    2025

  61. [61]

    Da Song, Xuan Xie, Jiayang Song, Derui Zhu, Yuheng Huang, Felix Juefei-Xu, and Lei Ma. 2024. Luna: A model-based universal analysis framework for large language models.IEEE Transactions on Software Engineering50, 7 (2024), 1921–1948

    2024

  62. [62]

    Claudio Spiess, David Gros, Kunal Suresh Pai, Michael Pradel, Md Rafiqul Islam Rabin, Amin Alipour, Susmit Jha, Prem Devanbu, and Toufique Ahmed. 2025. Calibration and Correctness of Language Models for Code(ICSE ’25). IEEE Press, 540–552. doi:10.1109/ICSE55347.2025.00040

    arXiv 2025

  63. [63]

    Qwen Team. 2025. Qwen/Qwen2.5-7B-Instruct (model card). https://huggingface.co/Qwen/Qwen2.5-7B-Instruct. Accessed: 2026-01-26

    2025

  64. [64]

    Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. 2023. Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Ho...

    2023

  65. [65]

    Weixi Tong and Tianyi Zhang. 2024. CodeJudge: Evaluating Code Generation with Large Language Models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Computational Linguistics, Miami, Florida, USA, 20032–20051. doi:10.18653/v1/2024.emnlp-main.1118

    doi:10.18653/v1/2024.emnlp-main.1118 2024

  66. [66]

    Yuvraj Virk, Premkumar Devanbu, and Toufique Ahmed. 2025. Calibration of Large Language Models on Code Summarization.Proc. ACM Softw. Eng.2, FSE, Article FSE130 (June 2025), 21 pages. doi:10.1145/3729400

    work page doi:10.1145/3729400 2025

  67. [67]

    Yikun Wang, Rui Zheng, Liang Ding, Qi Zhang, Dahua Lin, and Dacheng Tao. 2024. Uncertainty Aware Learning for Language Model Alignment. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, ...

    work page doi:10.18653/v1/2024.acl-long.597 2024

  68. [68]

    Zhijie Wang, Zijie Zhou, Da Song, Yuheng Huang, Shengmai Chen, Lei Ma, and Tianyi Zhang. 2025. Towards understanding the characteristics of code generation errors made by large language models. In2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE Computer Society, 717–717

    2025

  69. [69]

    Menghua Wu, Cai Zhou, Stephen Bates, and Tommi Jaakkola. 2025. Thought calibration: Efficient and confident test-time scaling. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (Eds.). Association for Computational Linguistics, Suzhou, Ch...

    doi:10.18653/v1/2025.emnlp-main.722 2025

  70. [70]

    Zeguan Xiao, Diyang Dou, Boya Xiong, Yun Chen, and Guanhua Chen. 2025. Enhancing Uncertainty Estimation in LLMs with Expectation of Aggregated Internal Belief.arXiv preprint arXiv:2509.01564(2025)

    arXiv 2025

  71. [71]

    Xiaoyuan Xie, Tsong Yueh Chen, Fei-Ching Kuo, and Baowen Xu. 2013. A theoretical analysis of the risk evaluation formulas for spectrum-based fault localization.ACM Transactions on software engineering and methodology (TOSEM) 22, 4 (2013), 1–40

    2013

  72. [72]

    Miao Xiong, Zhiyuan Hu, Xinyang Lu, YIFEI LI, Jie Fu, Junxian He, and Bryan Hooi. 2024. Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs. InThe Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=gjeQKFxFpZ

    2024

  73. [73]

    Borui Yang, Md Afif Al Mamun, Jie M Zhang, and Gias Uddin. 2025. Hallucination Detection in Large Language Models with Metamorphic Relations.Proceedings of the ACM on Software Engineering2, FSE (2025), 425–445

    2025

  74. [74]

    Xiyue Zhang, Xiaofei Xie, Lei Ma, Xiaoning Du, Qiang Hu, Yang Liu, Jianjun Zhao, and Meng Sun. 2020. Towards characterizing adversarial defects of deep learning software from the lens of uncertainty. InProceedings of the ACM/IEEE 42nd International Conference on Software Engineering. 739–751

    2020

  75. [75]

    Ziyao Zhang, Chong Wang, Yanlin Wang, Ensheng Shi, Yuchi Ma, Wanjun Zhong, Jiachi Chen, Mingzhi Mao, and Zibin Zheng. 2025. Llm hallucinations in practical code generation: Phenomena, mechanism, and mitigation.Proceedings of the ACM on Software Engineering2, ISSTA (2025), 481–503

    2025

  76. [76]

    Xuandong Zhao, Zhewei Kang, Aosong Feng, Sergey Levine, and Dawn Song. 2025. Learning to reason without external rewards.arXiv preprint arXiv:2505.19590(2025)

    Pith/arXiv arXiv 2025

  77. [77]

    Blaschko

    Han Zhou, Jordy Van Landeghem, Teodora Popordanoska, and Matthew B. Blaschko. 2025. A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimators. InForty-second International Conference on Machine Learning. https://openreview.net/forum?id=LBBUJkqkOM

    2025

  78. [78]

    Zhi Quan Zhou, Shaowen Xiang, and Tsong Yueh Chen. 2015. Metamorphic testing for software quality assessment: A study of search engines.IEEE Transactions on Software Engineering42, 3 (2015), 264–284

    2015

  79. [79]

    Yuqi Zhu, Ge Li, Xue Jiang, Jia Li, Hong Mei, Zhi Jin, and Yihong Dong. 2025. Uncertainty-guided chain-of-thought for code generation with llms.arXiv preprint arXiv:2503.15341(2025). Proc. ACM Softw. Eng., Vol. 3, No. ISSTA, Article ISSTA185. Publication date: October 2026

    Pith/arXiv arXiv 2025