Pith. sign in

REVIEW 4 major objections 5 minor 34 references

The paper claims that agreement among a model's own samples, and even across different models, is a weak and regime-dependent proxy for correctness rather than a trustworthy confidence score, and that the most self-consistent frontier model

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 07:55 UTC pith:JTGXWGS3

load-bearing objection Useful, unusually honest audit: the agreement–correctness link is real but weak, and the headline 48% frontier error rate is plausible yet rests on unverifiable snapshots—trust the direction, treat the magnitudes as provisional. the 4 major comments →

arxiv 2607.08065 v2 pith:JTGXWGS3 submitted 2026-07-09 cs.AI

When LLMs Agree, Are They Right? Auditing Self-Consistency and Cross-Model Agreement as Confidence Signals

Kaihua Ding This is my paper
classification cs.AI
keywords self-consistencyagreement as confidenceLLM calibrationmajority votingGPQAAIMEconfidence routingLLM evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether agreement signals correctness in large language models: if a model repeats the same answer across many samples, or if different judges concur, is the answer more likely to be right? The author's central claim is that self-consistency is not accuracy, but is a positive, regime-dependent empirical proxy: useful for allocating compute and for mid-tier models, but unreliable as a standalone confidence score. Across 265,000 samples from 53 independent runners on GPQA and AIME, agreement correlates positively but weakly with correctness (Spearman ρ 0.20–0.59), and weakest for the most consistent frontier model, whose high-agreement answers are wrong 48% of the time. If true, this matters because enterprise evaluation pipelines increasingly treat agreement as a confidence signal for routing, abstention, and judging, and would inherit that error rate. The paper also shows a meaningful share of confident errors recur across runners and prompts, and an exploratory cross-family check suggests some are shared across providers. The author is explicit that the runner data are not guaranteed independent and model snapshots were unlogged, so the results establish recurrence across runners and prompts, not across time.

Core claim

On the paper's own terms, the central discovery is a quantified audit of self-consistency C—the fraction of K samples that agree on the majority answer—as a deployed confidence signal. The paper finds that C correlates positively but weakly with majority-correctness M in every one of twelve experimental cells (Spearman ρ 0.20–0.59), with the weakest correlation (about 0.20 on GPQA) occurring for the frontier model, which is the most self-consistent (mean C = 0.89; C ≥ 0.8 on 77% of GPQA cases) yet no more accurate than the mid-tier model and worse calibrated. A high-agreement frontier answer on GPQA is wrong 48% of the time (95% CI [0.40, 0.56]). The paper also shows that chain-of-thought im

What carries the argument

The central object is the self-consistency score C = n_maj / K, the share of K temperature-1 samples that choose the majority answer, treated as the confidence score under audit. Its performance is measured against the deployment label M, the indicator that the majority answer matches ground truth, using Spearman rank correlation. Because entries are not independent (each runner contributes many rows, each case is seen by many runners), confidence intervals come from a hierarchical runner-clustered bootstrap that resamples runners and then cases within runners, with a case-clustered bootstrap as robustness. The cross-runner replication design—53 runners each drawing K = 50 samples on overlap

Load-bearing premise

The load-bearing premise is that the 53 runners' submitted runs are approximately exchangeable draws from the same model-and-prompt distribution; the paper concedes that runners 'may have exchanged code or discussion,' wrote their own prompts, and ran unlogged, unrecoverable model snapshots, so if the runs actually differ in prompt style, model version, or shared course context, the recurrence and correlation estimates are confounded by the measurement process rather than ref

What would settle it

A controlled, logged-snapshot replication with K = 50, fixed prompts, and recorded model versions on the same GPQA cases that found the frontier model's high-agreement (C ≥ 0.8) error rate at or below the mid-tier model's—or a nonnegative Δρ(C, M) between the frontier and mid-tier model—would contradict the paper's central frontier over-confidence conclusion.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A confidence-routing system that auto-trusts high-agreement answers from the frontier model on GPQA will inherit roughly a 48% error rate, since agreement is piled at the ceiling and loses discriminative power.
  • Self-consistency correlates positively but weakly with correctness throughout (ρ 0.20–0.59), so it cannot serve as a standalone abstention or deferral threshold on hard reasoning benchmarks.
  • Chain-of-thought robustly improves accuracy (paired ΔA ≈ +0.067 on GPQA) but at best marginally improves the agreement–correctness signal, so it is an accuracy intervention, not a calibration fix.
  • Adaptive sampling that stops when agreement is reached uses 60–82% fewer samples at equal majority error, so agreement remains a useful compute-allocation signal even where it is weak for trusting an answer.
  • A confidence-routed cascade that escalates low-agreement cases to the frontier model is dominated by simply always using the mid-tier model, making escalation to the over-confident frontier counterproductive in this audit.
  • Part of GPQA 'confidence' is positional: under option shuffling, majority accuracy drops and the model under-selects option D regardless of content, so agreement on a letter can reflect position priors rather than semantic confidence.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the positional component is as large as the shuffle control suggests, then randomizing option order in evaluation harnesses could directly reduce the measured confident-wrongness rate; this is a testable intervention the paper does not run.
  • The recurrence metric—the same wrong majority answer appearing across independent runners—could be developed into a benchmark-cleaning filter: items where many runners converge on a non-key answer are likely ambiguous or mislabeled, so removing them would sharpen both accuracy and calibration estimates.
  • For routing systems, the results imply a cheap pre-filter: use the mid-tier model's agreement to allocate sample budget, but never to escalate to the frontier model; a direct extension is whether a learned router that ignores C for escalation outperforms the C-routed cascade the paper evaluates.
  • The exploratory cross-family shared-error pattern suggests cross-model agreement should not be treated as independent confirmation; a decisive test would compare same-wrong-answer rates against the marginal-preserving null on far more than 46 shared items, with matched sampling paths.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper audits whether self-consistency C (the fraction of K samples agreeing on the majority answer) is a usable proxy for correctness on GPQA Diamond and AIME. It analyzes a cross-runner dataset: 53 runners, ~5,300 case-result rows, 265,000 samples, across twelve model/prompt/benchmark cells. All twelve Spearman correlations ρ(C,M) between self-consistency and majority-correctness are positive (0.20–0.59) and survive both runner- and case-clustered resampling. The central negative finding is that the frontier model gpt-4.1 has the highest mean self-consistency (C=0.89) but the worst calibration (ECE=0.41) and the lowest ρ(C,M)=0.20, with 48% of high-agreement (C≥0.8) GPQA answers wrong. The paper also reports that CoT improves accuracy but only marginally improves the agreement–correctness signal, that confident errors recur across runners and are partly positional, that an exploratory cross-family Claude check reproduces the frontier over-confidence, and that simulated adaptive sampling saves compute while confidence-routed cascades are dominated by always using a mid-tier model. The manuscript explicitly discloses its main limitations: unlogged, unrecoverable model snapshots, runners who may have shared code/discussion, and small exploratory controls.

Significance. If the central claim holds, the paper makes a useful and timely contribution: it provides a large-scale, multi-cell audit showing that agreement is a weak, regime-dependent proxy for correctness, and it quantifies the deployment consequence for confidence-routing systems (a ~48% error rate when auto-trusting high-agreement frontier answers). The design is a direct empirical audit, not a derivation, and no parameters are fitted to force the results. The paper is commendably honest about its null results (CoT improves accuracy but not the signal; a confidence-routed cascade is dominated by always using the mid-tier model) and about the exploratory nature of the cross-family check. It also releases de-identified per-run rows and a clustered bootstrap analysis pipeline, which is a strength. However, the headline quantitative claim is not independently verifiable from the released artifact: the primary data rest on unlogged model snapshots and non-independent runners, and the logged-snapshot confirmation is too small (48 GPQA cases, K=20) to confirm the magnitude of the reported effect.

major comments (4)
  1. [§2, §4, §11] The headline quantity—the 48% majority-error rate for gpt-4.1 at C≥0.8 on GPQA and the Δρ(C,M) degradation relative to mini—is load-bearing for the paper's negative conclusion. It is derived from 53 runners whose exact model snapshots and timestamps were not logged and are unrecoverable (§2, §11), and whose submissions may not be statistically independent. The logged-snapshot re-run reported in §4 uses only 48 GPQA cases and K=20; the key contrasts are not individually significant (Δρ=−.06 [−.37,.26]; ΔECE=+.07 [−.07,.22]). The re-run does reproduce the over-confidence direction (higher C at equal accuracy), but it cannot verify the specific 48% rate or the correlation gap. I recommend that the abstract and conclusion either present this figure explicitly as a cohort-specific descriptive estimate, or be backed by a larger logged re-run with adequate power. The current framing overstates
  2. [§2, §7] The recurrence analysis in §7 (28% of unique GPQA cases confidently wrong for every runner, 50% for at least one) is interpreted as evidence that confident errors are not sampling noise. But because runners 'may have exchanged code or discussion' and wrote their own prompts (§2), recurring wrong answers could reflect shared implementation artifacts or a common prompt template rather than model-intrinsic confident errors. The runner-clustered bootstrap addresses dependence in variance estimation, but not this source of confounding. I ask for per-runner heterogeneity diagnostics (e.g., distribution of C and M across runners, or sensitivity of the recurrence rates to excluding runners with unusually similar outputs) or, failing that, a more prominent caveat that recurrence is established across submitted runs, not across independent implementations.
  3. [§8, Tables 7–8] The cross-family shared-confident-error analysis is used in the abstract and conclusion to support 'confident errors recurring across providers.' The Claude samples, however, have K=10, no temperature control, unverifiable backend independence, and internal reasoning that cannot be disabled; the paper itself calls the check exploratory. Despite this, Table 8 reports p=.003 and p=.005 for two tiers. Given the small, uncontrolled samples, these p-values should be labeled as descriptive only, and the cross-provider sharing claim should be presented as a hypothesis for a future logged, matched-design study. As written, the exploratory check carries more inferential weight than its design can support.
  4. [§4, Table 2] The same under-powering issue affects the E2 focal contrast on which much of the frontier-degradation conclusion rests. E2 survives a Bonferroni-for-two correction and a case-clustered bootstrap (GPQA Δρ=−.18 [−.31,−.03], p=.025), so the directional claim is defensible. But the width of the CI means the magnitude is not pinned down. The paper should make explicit that the contribution is the direction and robustness of the degradation, not a precise estimate of its size, and should avoid implying that the logged re-run confirms the numerical gaps (which it explicitly does not at K=20).
minor comments (5)
  1. [Abstract] The abstract states '53 runners drew K=50 samples for assigned overlapping cases' and gives 265,000 total samples. The arithmetic requires the reader to infer that each runner ran both conditions of an axis on the same cases; a one-sentence clarification of the unit structure would help.
  2. [§6, Table 6] The comparison of confidence signals is based on n=50 cases and, for verbalized confidence, only 39/50 parsed GPQA cases. The paper notes this, but the concluding sentence that self-consistency is 'competitive with, not dominated by' dedicated signals is stronger than the wide CIs support. I would describe §6 as an underpowered pilot.
  3. [§7, Figure 4] The option-shuffle control reports a drop in mini's majority accuracy from 0.50 to 0.29 with overlapping 95% CIs at n=48. The accompanying text appropriately calls this 'suggestive,' but the figure caption's 'and GPQA accuracy drops under shuffling' is too categorical; I suggest aligning the caption with the cautious wording in the main text.
  4. [§9, Table 9] The cascade simulation reports a range for C-routed cascade (0.30–0.36) and random routing (0.30–0.34) but the text would benefit from stating that these are ranges across axes A+C cells, to avoid the implication of a continuous frontier.
  5. [Appendix A] The hierarchical cluster bootstrap is described as correcting runner dependence 'rather than implementing a full two-way crossed resample.' Since the paper also reports a case-clustered bootstrap, it would be useful to state explicitly that neither procedure fully accounts for crossed random effects, or to report the sensitivity of the headline CI to a two-way clustered bootstrap if computationally feasible.

Circularity Check

0 steps flagged

No circular derivation: the paper is a direct empirical audit whose quantities are measured, not fitted, and whose simulations are disclosed as such.

full rationale

The paper is a direct empirical audit, not a derivation. Its central quantities — self-consistency C = n_maj/K, sample accuracy A = n_correct/K, and majority-correct M = 1[majority answer = ground truth] — are measured, not fitted; no parameter is estimated from a subset of data and then renamed a prediction. The headline relation "high-agreement (C≥0.8) GPQA answers are wrong 48% of the time" is a conditional empirical rate computed from the same K samples, which is exactly the deployment setting under audit rather than a construction that forces the conclusion. The adaptive-sampling and cascade numbers in §9 are disclosed simulations over the measured sample banks, and the paper explicitly reports null or marginal results (CoT improves accuracy but only marginally improves ρ(C,M); C-routed cascade is dominated by always-mini), which would be difficult to force by construction. The logged-snapshot re-run in §4 is an external check (single logged snapshot, fixed prompt, subset of cases) and is reported with its wide CIs, including non-significant gaps. No uniqueness theorem or ansatz is imported from the author's own prior work; the cited self-references (Ding 2018, 2025a/b; Ding et al. 2024/2025) are context or methodological pointers and none is load-bearing for the main findings. Any concerns about unlogged model snapshots or runner non-independence are correctness/verifiability risks, not circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The central claim does not rest on fitted parameters in the usual sense, but the hand-chosen threshold C≥0.8 is central to the headline number. Inferential axioms are the trustworthiness of benchmark labels, the adequacy of runner-clustered inference despite non-independent runners, the unverifiable identity of model snapshots, and majority voting as the deployment rule.

free parameters (3)
  • high-agreement threshold = 0.8 (C≥0.8)
    Hand-selected cutoff for defining 'confident' entries; the headline 48% wrong-at-C≥0.8 and Pr[C≥0.8] depend on it.
  • sample budget K = 50
    Number of samples per case; C and M are computed from these 50 samples, so the measured signal strength is conditional on this choice.
  • sampling temperature = 1.0
    Fixed by design; controls the spread of C across samples and therefore the scale of agreement values.
axioms (4)
  • domain assumption GPQA Diamond and AIME ground-truth labels are correct for computing majority-correctness M.
    The paper itself notes 'a minority of GPQA items appear ambiguous or mislabeled' (§11), which would inflate counted confident-wrongness if those labels are wrong.
  • domain assumption The runner-clustered bootstrap adequately corrects dependence among the 53 runners.
    The paper says runners are 'independently submitted but not guaranteed statistically independent' (§2); the bootstrap assumes exchangeability at runner level despite this admission.
  • domain assumption The submitted runs were generated by the named model versions (gpt-4.1, gpt-4.1-mini, Claude tiers).
    Exact model snapshots and run timestamps were not logged and are unrecoverable (§11), so the study assumes the unlogged outputs came from the models they are labeled with.
  • domain assumption The deployment decision rule is majority voting over K samples.
    Defining M as majority-correct encodes the assumption that deployed systems return the majority answer; this frames the audit around M rather than per-sample accuracy (§2).

pith-pipeline@v1.3.0-alltime-deepseek · 13751 in / 12983 out tokens · 122006 ms · 2026-08-02T07:55:33.622846+00:00 · methodology

0 comments
read the original abstract

LLM-as-judge (Zheng et al., 2023) is increasingly the default for evaluating AI systems in enterprise pipelines, often scaled to ensembles (Verga et al., 2024) or "mixture-of-experts" (Shazeer et al., 2017) panels of judges. These systems share a key assumption: that consistency -- agreement among judges, or among a model's own samples -- indicates correctness. We show this assumption is unreliable. Agreement is not accuracy: a model can agree with itself, and different models can agree with each other, out of shared bias, a memorized heuristic, or an option-position prior rather than truth. We ask when agreement is nonetheless a usable proxy, in a large-scale cross-runner study: 53 runners drew K=50 samples for assigned overlapping cases across comparisons of model tier, prompting, and scale on GPQA Diamond and AIME -- 265,000 samples. Using majority-correctness as the deployment label and a hierarchical runner-clustered bootstrap, agreement is a positive but weak predictor (rho 0.20-0.59, all positive under item-clustered resampling) whose usefulness is regime-dependent: best for unsaturated mid-tier models and for allocating compute, and worst -- over-confident yet no more accurate -- for the most consistent frontier model (agreement >=0.8 on 77% of GPQA case-result entries, 48% of those wrong). An exploratory cross-family check on three Claude tiers shows the same frontier over-confidence, with confident errors recurring across providers above a marginal-preserving null. Self-consistency is thus a conditional proxy for correctness, not a standalone confidence score. We publicly release the de-identified per-run rows and answer distributions.

Figures

Figures reproduced from arXiv: 2607.08065 by Kaihua Ding.

Figure 1
Figure 1. Figure 1: GPQA reliability: binned self-consistency [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 1
Figure 1. Figure 1: GPQA reliability: binned self-consistency [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: GPQA risk–coverage by self-consistency for [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Risk–coverage curves by confidence signal [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Majority-answer letter distribution under orig [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

34 extracted references · 17 linked inside Pith

  1. [1]

    The Eleventh International Conference on Learning Representations (ICLR) , year =

    Self-Consistency Improves Chain of Thought Reasoning in Language Models , author =. The Eleventh International Conference on Learning Representations (ICLR) , year =. 2203.11171 , archivePrefix =

    Pith/arXiv arXiv

  2. [2]

    Advances in Neural Information Processing Systems 35 (NeurIPS) , year =

    Chain-of-Thought Prompting Elicits Reasoning in Large Language Models , author =. Advances in Neural Information Processing Systems 35 (NeurIPS) , year =. 2201.11903 , archivePrefix =

    Pith/arXiv arXiv

  3. [3]

    , booktitle =

    Rein, David and Hou, Betty Li and Stickland, Asa Cooper and Petty, Jackson and Pang, Richard Yuanzhe and Dirani, Julien and Michael, Julian and Bowman, Samuel R. , booktitle =. 2024 , url =. 2311.12022 , archivePrefix =

    Pith/arXiv arXiv 2024

  4. [4]

    2024 , howpublished =

    The. 2024 , howpublished =

    2024

  5. [5]

    arXiv preprint arXiv:2207.05221 , year =

    Language Models (Mostly) Know What They Know , author =. arXiv preprint arXiv:2207.05221 , year =. 2207.05221 , archivePrefix =

    Pith/arXiv arXiv

  6. [6]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages =

    Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback , author =. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages =. 2023 , publisher =. doi:10.18653/v1/2023.emnlp-main.330 , eprint =

    doi:10.18653/v1/2023.emnlp-main.330 2023

  7. [7]

    Transactions on Machine Learning Research (TMLR) , year =

    Teaching Models to Express Their Uncertainty in Words , author =. Transactions on Machine Learning Research (TMLR) , year =. 2205.14334 , archivePrefix =

    Pith/arXiv arXiv

  8. [8]

    2303.08774 , archivePrefix =

    arXiv preprint arXiv:2303.08774 , year =. 2303.08774 , archivePrefix =

    Pith/arXiv arXiv

  9. [9]

    Let's Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with

    Aggarwal, Pranjal and Madaan, Aman and Yang, Yiming and Mausam , booktitle =. Let's Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with. 2023 , publisher =. doi:10.18653/v1/2023.emnlp-main.761 , eprint =

    doi:10.18653/v1/2023.emnlp-main.761 2023

  10. [10]

    2023 , url =

    Chen, Lingjiao and Zaharia, Matei and Zou, James , journal =. 2023 , url =. 2305.05176 , archivePrefix =

    Pith/arXiv arXiv 2023

  11. [11]

    Advances in Neural Information Processing Systems 30 (NIPS) , year =

    Selective Classification for Deep Neural Networks , author =. Advances in Neural Information Processing Systems 30 (NIPS) , year =. 1705.08500 , archivePrefix =

    Pith/arXiv arXiv

  12. [12]

    Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL) , pages =

    Selective Question Answering under Domain Shift , author =. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL) , pages =. 2020 , publisher =. doi:10.18653/v1/2020.acl-main.503 , eprint =

    doi:10.18653/v1/2020.acl-main.503 2020

  13. [13]

    The Twelfth International Conference on Learning Representations (ICLR) , year =

    Large Language Models Are Not Robust Multiple Choice Selectors , author =. The Twelfth International Conference on Learning Representations (ICLR) , year =. 2309.03882 , archivePrefix =

    Pith/arXiv arXiv

  14. [14]

    Findings of the Association for Computational Linguistics: ACL 2024 , pages =

    Unveiling Selection Biases: Exploring Order and Token Sensitivity in Large Language Models , author =. Findings of the Association for Computational Linguistics: ACL 2024 , pages =. 2024 , publisher =. 2406.03009 , archivePrefix =

    Pith/arXiv arXiv 2024

  15. [15]

    Findings of the Association for Computational Linguistics: NAACL 2024 , year =

    Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions , author =. Findings of the Association for Computational Linguistics: NAACL 2024 , year =. 2308.11483 , archivePrefix =

    Pith/arXiv arXiv 2024

  16. [16]

    Manakul, Potsawee and Liusie, Adian and Gales, Mark J. F. , booktitle =. 2023 , address =. 2303.08896 , archivePrefix =

    Pith/arXiv arXiv 2023

  17. [17]

    The Eleventh International Conference on Learning Representations (ICLR) , year =

    Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation , author =. The Eleventh International Conference on Learning Representations (ICLR) , year =. 2302.09664 , archivePrefix =

    Pith/arXiv arXiv

  18. [18]

    Nature , volume =

    Detecting hallucinations in large language models using semantic entropy , author =. Nature , volume =. 2024 , publisher =. doi:10.1038/s41586-024-07421-0 , url =

    doi:10.1038/s41586-024-07421-0 2024

  19. [19]

    and Zhang, Hao and Gonzalez, Joseph E

    Zheng, Lianmin and Chiang, Wei-Lin and Sheng, Ying and Zhuang, Siyuan and Wu, Zhanghao and Zhuang, Yonghao and Lin, Zi and Li, Zhuohan and Li, Dacheng and Xing, Eric P. and Zhang, Hao and Gonzalez, Joseph E. and Stoica, Ion , booktitle =. Judging. 2023 , url =. 2306.05685 , archivePrefix =

    Pith/arXiv arXiv 2023

  20. [20]

    and Feng, Shi , booktitle =

    Panickssery, Arjun and Bowman, Samuel R. and Feng, Shi , booktitle =. 2024 , url =. 2404.13076 , archivePrefix =

    Pith/arXiv arXiv 2024

  21. [21]

    Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL) , pages =

    A Survey of Confidence Estimation and Calibration in Large Language Models , author =. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL) , pages =. 2024 , address =. 2311.08298 , archivePrefix =

    Pith/arXiv arXiv 2024

  22. [22]

    Xiong, Miao and Hu, Zhiyuan and Lu, Xinyang and Li, Yifei and Fu, Jie and He, Junxian and Hooi, Bryan , booktitle =. Can. 2024 , url =. 2306.13063 , archivePrefix =

    Pith/arXiv arXiv 2024

  23. [23]

    and Roe, Philip L

    Ding, Kaihua and Fidkowski, Krzysztof J. and Roe, Philip L. , title =. 54th AIAA Aerospace Sciences Meeting , year =

  24. [24]

    and Roe, Philip L

    Ding, Kaihua and Fidkowski, Krzysztof J. and Roe, Philip L. , title =. Eighth International Conference on Computational Fluid Dynamics (ICCFD8) , year =

  25. [25]

    and Roe, Philip L

    Ding, Kaihua and Fidkowski, Krzysztof J. and Roe, Philip L. , title =. 21st AIAA Computational Fluid Dynamics Conference , year =

  26. [26]

    , title =

    Ding, Kaihua and Fidkowski, Krzysztof J. , title =. Computers & Fluids , volume =. 2020 , doi =

    2020

  27. [27]

    , title =

    Ding, Kaihua and Fidkowski, Krzysztof J. , title =. 23rd AIAA Computational Fluid Dynamics Conference , year =

  28. [28]

    2018 , url =

    Ding, Kaihua , title =. 2018 , url =

    2018

  29. [29]

    2025 , eprint =

    Ding, Kaihua , title =. 2025 , eprint =

    2025

  30. [30]

    PMSA Journal , pages =

    Ding, Kaihua and Cui, Jingsong and Soltani, Mohammad and Jin, Jing , title =. PMSA Journal , pages =

  31. [31]

    2024 , eprint =

    Ding, Kaihua and Cui, Jingsong and Soltani, Mohammad and Jin, Jing , title =. 2024 , eprint =

    2024

  32. [32]

    , title =

    Ding, Kaihua and Fidkowski, Krzysztof J. , title =. AIAA Scitech 2021 Forum , year =

    2021

  33. [33]

    2024 , eprint =

    Verga, Pat and Hofst\"atter, Sebastian and Althammer, Sophia and Su, Yixuan and Piktus, Aleksandra and Arkhangorodsky, Arkady and Xu, Minjie and White, Naomi and Lewis, Patrick , title =. 2024 , eprint =

    2024

  34. [34]

    International Conference on Learning Representations (ICLR) , year =

    Shazeer, Noam and Mirhoseini, Azalia and Maziarz, Krzysztof and Davis, Andy and Le, Quoc and Hinton, Geoffrey and Dean, Jeff , title =. International Conference on Learning Representations (ICLR) , year =