Pith. sign in

REVIEW 6 major objections 7 minor 40 references

When frontier LLMs agree most, they are wrong nearly half the time

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-10 00:41 UTC pith:JTGXWGS3

load-bearing objection Solid empirical audit of self-consistency as a confidence signal; the frontier over-confidence finding is real but partly confounded by label quality. the 6 major comments →

arxiv 2607.08065 v1 pith:JTGXWGS3 submitted 2026-07-09 cs.AI

When LLMs Agree, Are They Right? Auditing Self-Consistency and Cross-Model Agreement as Confidence Signals

Kaihua Ding This is my paper
classification cs.AI
keywords agreementmodelagreesamplesacrossassumptionconfidencecorrectness
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper audits whether self-consistency—when a language model returns the same answer across many stochastic samples—can serve as a confidence signal for deployment decisions like routing, abstention, or auto-accepting answers. Drawing on 265,000 samples from 53 independent runners across two hard reasoning benchmarks (GPQA Diamond and AIME) and three model tiers, the author finds that agreement is a positive but weak predictor of correctness (Spearman rho 0.20-0.59). The relationship is regime-dependent: mid-tier models with unsaturated agreement levels show the most useful signal, while the most consistent frontier model (gpt-4.1) exhibits the worst calibration, reaching high agreement (C>=0.8) on 77% of GPQA cases yet being wrong 48% of the time on those high-confidence answers. Chain-of-thought prompting improves accuracy but only marginally improves the agreement-correctness link. An option-shuffle control reveals that part of the apparent confidence on multiple-choice questions is positional bias (systematic under-selection of option D) rather than semantic certainty. Confident errors recur across different runners and prompts, and an exploratory cross-family check on three Claude tiers shows the same frontier over-confidence pattern, with confident wrong answers shared across providers above a marginal-preserving null—suggesting shared pretraining biases rather than independent corroboration.

Core claim

The central discovery is a non-monotonic relationship between model capability and the reliability of self-consistency as a confidence signal. As models become more self-consistent (higher agreement), the agreement signal does not become more informative about correctness—it becomes less so, because the most consistent frontier model piles its agreement at the ceiling (C>=0.8 on 77% of GPQA cases), destroying the discriminative power of the signal. This means the deployment practice of routing hard (low-agreement) cases to frontier models is counterproductive: the frontier model receives the cases where reliable confidence is most needed, yet its high agreement on those cases is only weaklyt

What carries the argument

The paper operationalizes self-consistency as C = n_majority / K (the fraction of K=50 samples that produce the majority answer), and measures its relationship to majority-correctness M (whether the majority answer equals the ground truth). The core statistical machinery is a hierarchical runner-clustered bootstrap (resampling runners, then cases within runners; B=2000) that corrects for the non-independence introduced by 53 runners each contributing many rows on shared cases. A case-clustered bootstrap (resampling items rather than runners) serves as a robustness check. An option-shuffle control permutes answer-choice positions to separate positional confidence from semantic confidence. A l

Load-bearing premise

The paper treats majority-correctness (whether the majority-voted answer matches the benchmark answer key) as the ground-truth label for all correlation and calibration analyses, but acknowledges that a minority of GPQA items appear ambiguous or mislabeled—if the answer keys contain errors, the agreement-correctness correlation is measured against a noisy target, which could inflate or deflate the reported confident-wrongness rates, particularly for the frontier model.

What would settle it

If a subsequent study with independently verified, cleaned answer keys found that the frontier model's apparent over-confidence disappeared (i.e., its high-agreement answers were mostly on items the original key mislabeled), or if the non-monotonic relationship between model tier and agreement-correctness correlation failed to replicate across additional model families and benchmarks beyond the exploratory Claude check, the central claim would be undermined.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Deployment systems that auto-trust high-agreement LLM answers inherit a substantial error rate, especially when using frontier models—roughly 48% of high-confidence frontier answers on GPQA are wrong.
  • Confidence-routed cascades that escalate low-agreement cases to frontier models are counterproductive; simply always using the mid-tier model achieves higher accuracy than any agreement-based routing policy tested.
  • Adaptive sampling rules (stop when consensus is reached) can save 60-82% of compute at equal error rates, making self-consistency useful for compute allocation even where it is unreliable for trusting individual answers.
  • Cross-model agreement panels (ensemble judges) cannot be assumed to provide independent confirmation if models share pretraining biases, option-position priors, or memorized heuristics that produce the same confident wrong answers.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

6 major / 7 minor

Summary. This paper audits whether self-consistency (agreement among a model's own K=50 samples) functions as a reliable confidence proxy for correctness on GPQA Diamond and AIME. Using 265,000 samples from 53 runners across three controlled axes (model tier, prompt strategy, scale), the author finds that agreement is a positive but weak predictor (Spearman ρ 0.20–0.59), with a regime-dependent usefulness: best for unsaturated mid-tier models and compute allocation, worst for the over-confident frontier model (gpt-4.1). The frontier model shows the highest agreement (C=0.89) yet the lowest correlation (ρ=0.20) and worst calibration (ECE=0.41), with high-agreement answers wrong 48% of the time on GPQA. An exploratory cross-family check on Claude tiers reproduces the frontier over-confidence pattern, and a label-permutation test suggests confident errors are partly shared across providers. The paper releases de-identified per-run data.

Significance. The paper addresses a timely and practically important question: whether self-consistency, already deployed in routing, cascading, and abstention systems, is a trustworthy confidence signal. The large-scale cross-runner design (53 runners, overlapping cases, 265K samples) and the hierarchical runner-clustered bootstrap with case-clustered robustness checks are methodological strengths. The finding that the frontier model is uniquely over-confident—high agreement at no accuracy advantage—is a concrete, deployable caution. The option-shuffle control (§7) and the cross-family shared-error permutation test (§8) add value by separating positional bias and shared pretraining bias from genuine confidence. The released dataset enables replication. These strengths make the paper a worthwhile contribution to the LLM evaluation literature.

major comments (6)
  1. §4, Table 5: The headline finding that the frontier model (gpt-4.1) is the 'worst-calibrated voter' (ECE=0.41, err|C≥0.8=0.48, ρ(C,M)=0.20) is the most novel and striking claim, yet it is also the most vulnerable to a structural confound with ground-truth label quality. The paper acknowledges in §7 that 'a minority of GPQA items appear ambiguous or mislabeled' and notes one item 'answered consistently against its listed key.' When a GPQA item's key is wrong, a highly consistent model commits to its (actually correct) answer with high C, producing a 'confident error' (high C, M=0). A less consistent model spreads probability mass, producing a lower-C entry that contributes less to the high-C error rate. This mechanism would systematically inflate the frontier's err|C≥0.8 and ECE relative to mid-tier models, independent of any genuine calibration regression. The paper does not quantify how
  2. many of the 28% of frontier confidently-wrong-for-every-runner cases (§7) are actually key errors, nor does it recompute headline metrics with corrected or excluded keys. This is a load-bearing gap: the 'frontier is uniquely worst' finding cannot be fully distinguished from a label-quality artifact without such an analysis. The controlled re-run (§4, 48 GPQA, K=20) does not reproduce the ρ or ECE gaps at significance (Δρ=−0.06 [−0.37, 0.26], ΔECE=+0.07 [−0.07, 0.22]), which is consistent with the gap being partly driven by item-level artifacts in the larger course dataset. The author should either (a) identify and exclude/relabel the suspect items and recompute Table 5's frontier metrics, or (b) provide a sensitivity bound showing how many mislabeled items would be needed to overturn the frontier degradation finding. Without this, the novel claim is not fully established.
  3. §2, Table 3 and §11: The 53 runners are graduate-course participants who 'may have exchanged code or discussion,' and each implemented their own prompts. The paper treats prompt variance as 'part of the measurement' (§2), which is defensible for external validity, but the runner-clustered bootstrap only corrects for runner-level dependence—it does not account for the fact that runners within the same course context may share systematic prompt strategies or code patterns that are not independent. The mean replication count is 2.5 runners per cell (max 6–9), which is thin for estimating the runner-level variance component. The paper should report the intraclass correlation or an equivalent diagnostic showing how much of the total variance is at the runner level versus the case level, so readers can assess whether the runner-clustered CIs are adequately calibrated. This is particularly load
  4. bearing for the paired Axis-B and Axis-C contrasts (Table 2), where the effective sample size for the paired difference is the number of runners (17–18), not the number of case-result rows.
  5. §8, Table 7: The exploratory cross-family check on Claude tiers uses agent-mediated sessions with K=10, no temperature control, and unverifiable backend/session independence. The author states these caveats clearly, which is appropriate. However, the claim that 'confident errors are partly shared across providers' (§8, Table 8) relies on a label-permutation null that preserves each model's empirical wrong-answer marginal. This null is reasonable for option-position bias, but it does not account for the possibility that the shared wrong answers are on items that are genuinely difficult or mislabeled—i.e., both models fail for the same reason (item difficulty or key error) rather than 'shared bias.' The paper notes six items answered wrongly by all four models, including one whose listed key is B that every model answers D. This is more consistent with an item flaw than shared pretraining
  6. bias. The cross-family shared-error claim should be qualified to distinguish 'shared bias' from 'shared item difficulty or label error,' or the permutation null should be conditioned on item difficulty.
minor comments (7)
  1. §2, Metrics: The distinction between 'over-confident' (high C co-occurring with errors) and formal probability miscalibration is stated, but the use of ECE and Brier (which are formal calibration metrics) creates slight tension. A clarifying sentence noting that ECE here is used as an operational diagnostic of the deployment use, not as a claim of probabilistic miscalibration, would help.
  2. Table 1: The † marker on the nano–GPQA ρ(C,A) CI crossing zero is noted, but the table caption states 'All twelve ρ(C,M) are positive and survive Holm correction.' It would help to also note explicitly in the table that ρ(C,A) is the secondary metric and that the zero-crossing is expected given the floor effect on GPQA.
  3. §4: The phrase 'worst-calibrated voter' is used in the section title and text. Given that the paper explicitly disclaims formal calibration (§2), 'worst-agreement-calibrated' or 'most over-confident' would be more precise.
  4. Figure 1: The reliability diagram bins C into 0.2-width bins. With C=0.89 for gpt-4.1, most mass is in the highest bin. A finer-grained binning or a rug plot showing the C distribution would make the over-confidence visually clearer.
  5. §7: The option-shuffle control uses 48 GPQA cases (12 per ground-truth letter). The CIs for the accuracy drop under shuffling overlap (mini: 0.50 [0.35, 0.65] to 0.29 [0.19, 0.41]). The text says 'suggestive, though the CIs overlap at n=48,' but the drop is from 0.50 to 0.29, which is a large effect. The overlap may be due to the small n; a power analysis or a note on the expected detectable effect size would contextualize this.
  6. References: The self-citations (Ding 2018, 2025a, 2025b) are to prior work on error estimation and assessment design. They are relevant but somewhat tangential to the LLM focus. Consider whether all three are necessary for the argument.
  7. §11, Limitations: The statement 'We do not study reasoning-trained models, whose internal reasoning may already supply the calibration lift we measure for prompted CoT' is important. Given that gpt-4.1 and Claude opus may already reason internally, this could interact with the frontier over-confidence finding. A brief note on how this limitation affects the interpretation of §4 would strengthen the limitations section.

Circularity Check

0 steps flagged

No circularity: empirical audit with correlations computed against external ground-truth labels

full rationale

This paper is an empirical audit measuring the correlation between self-consistency (C = n_maj/K) and majority-correctness (M = 𝟙[majority answer = ground truth]) using external benchmark answer keys (GPQA Diamond, AIME). There is no derivation chain that could reduce to its inputs by construction. The correlations ρ(C, M) are computed against externally provided ground-truth labels, not against quantities the authors define. The self-citations (Ding 2018, 2025a, 2025b, Ding et al. 2024/2025) are to prior methodological work on error estimation and assessment design; none are invoked as load-bearing mathematical facts or uniqueness theorems that the current paper's conclusions depend on. The option-shuffle control (§7) and cross-family Claude check (§8) are independent experiments, not renamings of prior results. The adaptive sampling and cascade simulations (§9) use the collected data to simulate deployment scenarios but do not claim to 'predict' quantities that are fitted by construction. The skeptic's concern about label quality inflating frontier miscalibration is a confounding/correctness issue, not circularity—the paper does not define its own ground truth and then recover it. No circular steps found.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The paper is an empirical audit with no new theoretical entities, particles, forces, or postulated constructs. It introduces no new mathematical objects beyond standard statistical measures (Spearman ρ, ECE, Brier, AURC). The 'regime-dependent proxy' framing is an interpretive lens, not a new entity. Free parameters are design choices (K, temperature, thresholds) rather than fitted model constants.

free parameters (3)
  • K (samples per case) = 50
    Number of stochastic samples drawn per case; chosen by design, not derived. Affects the granularity of the agreement distribution.
  • temperature = 1.0
    Sampling temperature for all main-study runs; chosen by design. Affects the spread of answers and thus the agreement distribution.
  • C≥0.8 threshold = 0.8
    Threshold for 'high agreement' used in confident-wrongness analysis (Table 5, §4). Chosen post-hoc as a practically meaningful cutoff; the paper also reports a threshold sweep in supplementary material.
axioms (4)
  • domain assumption Majority-correctness M (whether the majority answer equals ground truth) is the correct deployment label for evaluating self-consistency as a confidence signal.
    Stated in §2 (Metrics): 'Because deployed majority-vote systems return the majority answer, M is our primary outcome.' This is reasonable for the deployment context but assumes majority-vote is the deployment mechanism.
  • domain assumption GPQA Diamond and AIME answer keys are correct ground truth.
    Used throughout as the correctness target. The paper acknowledges this is imperfect: 'a minority of GPQA items appear ambiguous or mislabeled' (§7).
  • domain assumption Runners' independently submitted runs are sufficiently independent for hierarchical clustering, despite shared course context.
    Stated in §2 (Data provenance): 'participants shared a course context and may have exchanged code or discussion, so runs are independently submitted but not guaranteed statistically independent.' The hierarchical bootstrap corrects for runner-level dependence but assumes the residual independence structure is adequate.
  • standard math Spearman ρ is an appropriate measure of the agreement-correctness relationship for this analysis.
    Used throughout Table 1 and all correlation analyses. Standard non-parametric rank correlation; appropriate for the ordinal-ish nature of C and binary M.

pith-pipeline@v1.1.0-glm · 17649 in / 3174 out tokens · 372291 ms · 2026-07-10T00:41:36.978280+00:00 · methodology

0 comments
read the original abstract

LLM-as-judge (Zheng et al., 2023) is increasingly the default for evaluating AI systems in enterprise pipelines, often scaled to ensembles (Verga et al., 2024) or "mixture-of-experts" (Shazeer et al., 2017) panels of judges. These systems share a key assumption: that consistency -- agreement among judges, or among a model's own samples -- indicates correctness. We show this assumption is unreliable. Agreement is not accuracy: a model can agree with itself, and different models can agree with each other, out of shared bias, a memorized heuristic, or an option-position prior rather than truth. We ask when agreement is nonetheless a usable proxy, in a large-scale cross-runner study: 53 runners drew K=50 samples for assigned overlapping cases across comparisons of model tier, prompting, and scale on GPQA Diamond and AIME -- 265,000 samples. Using majority-correctness as the deployment label and a hierarchical runner-clustered bootstrap, agreement is a positive but weak predictor (rho 0.20-0.59, all positive under item-clustered resampling) whose usefulness is regime-dependent: best for unsaturated mid-tier models and for allocating compute, and worst -- over-confident yet no more accurate -- for the most consistent frontier model (agreement >=0.8 on 77% of GPQA case-result entries, 48% of those wrong). An exploratory cross-family check on three Claude tiers shows the same frontier over-confidence, with confident errors recurring across providers above a marginal-preserving null. Self-consistency is thus a conditional proxy for correctness, not a standalone confidence score. We publicly release the de-identified per-run rows and answer distributions.

Figures

Figures reproduced from arXiv: 2607.08065 by Kaihua Ding.

Figure 1
Figure 1. Figure 1: GPQA reliability: binned self-consistency [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: GPQA risk–coverage by self-consistency for [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Risk–coverage curves by confidence signal [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

40 extracted references · 40 canonical work pages · 19 internal anchors

  1. [22]

    and Roe, Philip L

    Ding, Kaihua and Fidkowski, Krzysztof J. and Roe, Philip L. , title =. 54th AIAA Aerospace Sciences Meeting , year =

    work page

  2. [23]

    and Roe, Philip L

    Ding, Kaihua and Fidkowski, Krzysztof J. and Roe, Philip L. , title =. Eighth International Conference on Computational Fluid Dynamics (ICCFD8) , year =

    work page

  3. [24]

    and Roe, Philip L

    Ding, Kaihua and Fidkowski, Krzysztof J. and Roe, Philip L. , title =. 21st AIAA Computational Fluid Dynamics Conference , year =

    work page

  4. [25]

    , title =

    Ding, Kaihua and Fidkowski, Krzysztof J. , title =. Computers & Fluids , volume =. 2020 , doi =

    work page 2020

  5. [26]

    , title =

    Ding, Kaihua and Fidkowski, Krzysztof J. , title =. 23rd AIAA Computational Fluid Dynamics Conference , year =

    work page

  6. [27]

    2018 , url =

    Ding, Kaihua , title =. 2018 , url =

    work page 2018

  7. [28]

    2025 , eprint =

    Ding, Kaihua , title =. 2025 , eprint =

    work page 2025

  8. [29]

    PMSA Journal , pages =

    Ding, Kaihua and Cui, Jingsong and Soltani, Mohammad and Jin, Jing , title =. PMSA Journal , pages =

    work page

  9. [30]

    2024 , eprint =

    Ding, Kaihua and Cui, Jingsong and Soltani, Mohammad and Jin, Jing , title =. 2024 , eprint =

    work page 2024

  10. [31]

    , title =

    Ding, Kaihua and Fidkowski, Krzysztof J. , title =. AIAA Scitech 2021 Forum , year =

    work page 2021

  11. [32]

    2024 , eprint =

    Verga, Pat and Hofst\"atter, Sebastian and Althammer, Sophia and Su, Yixuan and Piktus, Aleksandra and Arkhangorodsky, Arkady and Xu, Minjie and White, Naomi and Lewis, Patrick , title =. 2024 , eprint =

    work page 2024

  12. [33]

    International Conference on Learning Representations (ICLR) , year =

    Shazeer, Noam and Mirhoseini, Azalia and Maziarz, Krzysztof and Davis, Andy and Le, Quoc and Hinton, Geoffrey and Dean, Jeff , title =. International Conference on Learning Representations (ICLR) , year =

    work page

  13. [34]

    Pranjal Aggarwal, Aman Madaan, Yiming Yang, and Mausam. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.761 Let's sample step by step: Adaptive-consistency for efficient reasoning and coding with LLM s . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 12375--12396, Singapore. Association for Compu...

    work page doi:10.18653/v1/2023.emnlp-main.761 2023

  14. [35]

    Lingjiao Chen, Matei Zaharia, and James Zou. 2023. https://arxiv.org/abs/2305.05176 FrugalGPT : How to use large language models while reducing cost and improving performance . arXiv preprint arXiv:2305.05176

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  15. [36]

    Kaihua Ding. 2018. https://deepblue.lib.umich.edu/handle/2027.42/144065 Efficient Output-Based Adaptation Mechanics for High-Order Computational Fluid Dynamics Methods . Ph.D. thesis, University of Michigan

    work page 2018

  16. [37]

    Kaihua Ding. 2025 a . https://doi.org/10.48550/arXiv.2512.10758 Designing AI-Resilient Assessments Using Interconnected Problems: A Theoretically Grounded and Empirically Validated Framework . Preprint, arXiv:2512.10758. Accepted, IEEE Frontiers in Education (FIE) 2026

    work page doi:10.48550/arxiv.2512.10758 2025

  17. [38]

    Kaihua Ding. 2025 b . https://doi.org/10.48550/arXiv.2509.22751 Variance-Bounded Evaluation of Entity-Centric AI Systems Without Ground Truth: Theory and Measurement . Preprint, arXiv:2509.22751

    work page doi:10.48550/arxiv.2509.22751 2025

  18. [39]

    Kaihua Ding, Jingsong Cui, Mohammad Soltani, and Jing Jin. 2024. https://doi.org/10.48550/arXiv.2405.14743 Iterative Causal Segmentation: Filling the Gap between Market Segmentation and Marketing Strategy . Preprint, arXiv:2405.14743

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2405.14743 2024

  19. [40]

    Kaihua Ding, Jingsong Cui, Mohammad Soltani, and Jing Jin. 2025. Iterative Causal Segmentation . PMSA Journal, pages 21--33

    work page 2025

  20. [41]

    Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. 2024. https://doi.org/10.1038/s41586-024-07421-0 Detecting hallucinations in large language models using semantic entropy . Nature, 630(8017):625--630

    work page doi:10.1038/s41586-024-07421-0 2024

  21. [42]

    Yonatan Geifman and Ran El-Yaniv. 2017. https://arxiv.org/abs/1705.08500 Selective classification for deep neural networks . In Advances in Neural Information Processing Systems 30 (NIPS). ArXiv:1705.08500

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  22. [43]

    Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. 2024. https://arxiv.org/abs/2311.08298 A survey of confidence estimation and calibration in large language models . In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), ...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  23. [44]

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, and 17 others. 2022. https://arxiv.org/abs/2207.05221 Language models (mostly...

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  24. [45]

    Amita Kamath, Robin Jia, and Percy Liang. 2020. https://doi.org/10.18653/v1/2020.acl-main.503 Selective question answering under domain shift . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 5684--5696. Association for Computational Linguistics

    work page doi:10.18653/v1/2020.acl-main.503 2020

  25. [46]

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. https://arxiv.org/abs/2302.09664 Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation . In The Eleventh International Conference on Learning Representations (ICLR)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  26. [47]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. https://arxiv.org/abs/2205.14334 Teaching models to express their uncertainty in words . Transactions on Machine Learning Research (TMLR). ArXiv:2205.14334

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  27. [48]

    Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. 2023. https://arxiv.org/abs/2303.08896 SelfCheckGPT : Zero-resource black-box hallucination detection for generative large language models . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9004--9017, Singapore. Association for Computational Linguistics

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  28. [49]

    OpenAI . 2023. https://arxiv.org/abs/2303.08774 GPT-4 technical report . arXiv preprint arXiv:2303.08774

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  29. [50]

    LLM Evaluators Recognize and Favor Their Own Generations

    Arjun Panickssery, Samuel R. Bowman, and Shi Feng. 2024. https://arxiv.org/abs/2404.13076 LLM evaluators recognize and favor their own generations . In Advances in Neural Information Processing Systems 37 (NeurIPS 2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  30. [51]

    Pouya Pezeshkpour and Estevam Hruschka. 2024. https://arxiv.org/abs/2308.11483 Large language models sensitivity to the order of options in multiple-choice questions . In Findings of the Association for Computational Linguistics: NAACL 2024. Association for Computational Linguistics

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  31. [52]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2024. https://arxiv.org/abs/2311.12022 GPQA : A graduate-level google-proof Q&A benchmark . In First Conference on Language Modeling (COLM). ArXiv:2311.12022

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  32. [53]

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations (ICLR)

    work page 2017

  33. [54]

    Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D. Manning. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.330 Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback . In Proceedings of the 2023 Conference on...

    work page doi:10.18653/v1/2023.emnlp-main.330 2023

  34. [55]

    Pat Verga, Sebastian Hofst\"atter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. 2024. https://arxiv.org/abs/2404.18796 Replacing judges with juries: Evaluating LLM generations with a panel of diverse models . Preprint, arXiv:2404.18796

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  35. [56]

    Self-Consistency Improves Chain of Thought Reasoning in Language Models

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. https://arxiv.org/abs/2203.11171 Self-consistency improves chain of thought reasoning in language models . In The Eleventh International Conference on Learning Representations (ICLR). ArXiv:2203.11171

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  36. [57]

    Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. https://arxiv.org/abs/2201.11903 Chain-of-thought prompting elicits reasoning in large language models . In Advances in Neural Information Processing Systems 35 (NeurIPS). ArXiv:2201.11903

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  37. [58]

    Sheng-Lun Wei, Cheng-Kuang Wu, Hen-Hsen Huang, and Hsin-Hsi Chen. 2024. https://arxiv.org/abs/2406.03009 Unveiling selection biases: Exploring order and token sensitivity in large language models . In Findings of the Association for Computational Linguistics: ACL 2024, pages 5598--5621. Association for Computational Linguistics

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  38. [59]

    Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2024. https://arxiv.org/abs/2306.13063 Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs . In The Twelfth International Conference on Learning Representations (ICLR)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  39. [60]

    Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2024. https://arxiv.org/abs/2309.03882 Large language models are not robust multiple choice selectors . In The Twelfth International Conference on Learning Representations (ICLR). ArXiv:2309.03882

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  40. [61]

    Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. https://arxiv.org/abs/2306.05685 Judging LLM -as-a-judge with MT-Bench and chatbot arena . In Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Datasets...

    work page internal anchor Pith review Pith/arXiv arXiv 2023