Pith. sign in

REVIEW 3 major objections 5 minor 28 references

The Calibration Floor: Format Repair Can Masquerade as Self-Correction at Small-to-Mid Scale

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Most measured self-correction accuracy gains are answer-format artifacts, not reasoning improvements.

desk verdict A strong measurement paper whose causal control needs a missing fidelity check before the headline claim is fully sold. read the letter →

arxiv 2608.04355 v1 pith:BJH4GYKI submitted 2026-08-05 cs.CL

Mingguang Chen , Bo Qu , Licheng Wang This is my paper · ORCID
classification cs.CL
keywords self-correctionanswerextractionmargindecompositioncalibrationfloorconfidencegatingformatrecoverylanguagemodelevaluationscaledependence
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that what the field has been measuring as self-correction in language models is mostly an artifact of answer extraction. When a model revises its own answer, the benchmark's extractor can fail to find a parseable answer before revision and succeed after (or vice versa), producing apparent accuracy gains or losses that have nothing to do with changed reasoning. The paper splits the always-revise accuracy shift into a content margin for rows where both answers parse, and format-recovery and format-loss margins for rows where parseability changes, and shows the format margins dominate on every cell with a meaningful extraction-failure rate. A causal control that re-extracts already-generated reasoning under grammar constraints, so every answer is parseable by construction, closes a median 71% of the gap between the naive total effect and the content margin. If the paper is right, published self-correction gains and harms need re-auditing before being interpreted as reasoning ability.

What carries the argument

The load-bearing object is the margin decomposition identity, together with the calibration-floor identity for confidence gating. The decomposition partitions each revision trajectory by whether the initial answer and final answer are parseable: bothok (content margin), recover (initial unparseable, final parseable), loss (initial parseable, final unparseable), and dead (both unparseable). It yields $\Delta_{\mathrm{total}} = \Delta_{\mathrm{content}} + \Delta_{\mathrm{format\ recover}} + \Delta_{\mathrm{format\ loss}}$, exact over the full sample. The gating identity $\Delta(\tau) = (1-\mathrm{acc}_0)\,\mathrm{TPR}(\tau)\,r(\tau) - \mathrm{acc}_0\,\mathrm{FPR}(\tau)\,d(\tau)$ gives the calibration floor: the minimum signal quality needed for confidence-triggered revision to beat the always-revise endpoint, and the paper re-derives it on the content margin. The causal control is a grammar-constrained decoding that forces one token from the valid answer set after the frozen reasoning text, guaranteeing parseability and turning the observational recover/loss split into a closure experiment.

What would settle it

The C1 claim would be falsified by finding a model-task cell with an active extraction channel (at least 5% unparseable initial answers) in which $|\Delta_{\mathrm{content}}|$ exceeds $|\Delta_{\mathrm{format}}|$, or by running the constrained-decoding control on active-channel cells and observing median closure well below the reported 71%, such as closure near zero or negative. A capable-scale (4B-12B) cell with a systematically positive content margin above 0.03 on non-ARC tasks would falsify C2.

Watch

Extended reading notes

Core claim

The central claim is that apparent always-revise self-correction effects are dominated by extraction artifacts, in both directions, not by changes in the model's answer. Formally, for every sample the final-minus-initial accuracy delta decomposes exactly as $\Delta_{\mathrm{total}} = \Delta_{\mathrm{content}} + \Delta_{\mathrm{format\ recover}} + \Delta_{\mathrm{format\ loss}}$, where $\Delta_{\mathrm{content}}$ counts only rows in which both the initial and revised answers are parseable. Across 12 admitted cells with at least 5% unparseable initial answers, $|\Delta_{\mathrm{format}}|$ exceeds $|\Delta_{\mathrm{content}}|$ (one-sided Wilcoxon p=1.7e-3). The causal control forces a guaranteed-parseable continuation from the already-generated reasoning text: across 14 cells it closes a median 71% of the gap between the naive total effect and the content margin, with two cells converging exactly and a residual on the two largest-effect cells. The paper also claims content flips are real but harmful only at small scale (0.8B/2B), while at 4B-12B and, provisionally, at frontier scale the content margin is near zero even when total effects reach +0.275.

Load-bearing premise

The causal control assumes that forcing a short continuation after already-generated reasoning is a minimal elicitation that reads out an answer already latent in the text, so the observed closure of the total-versus-content gap is attributable to guaranteeing parseability rather than to the forced continuation changing the model's thinking.

Editorial extensions

If this is right

  • A benchmark accuracy delta after self-revision cannot be read as reasoning change unless parseability is stable or the content margin is reported.
  • Reported self-correction gains on tasks with verbose free-form outputs, such as math word problems and multiple-choice questions, are likely to shrink or vanish once format-recovery rows are removed.
  • Confidence-gated selective revision has at most a marginal niche: floor-scale models have content headroom but no signal clears the floor, while capable-scale models have signal but almost no headroom.
  • Small models (0.8B-2B) genuinely change parseable answers on the content margin, and these changes are typically harmful, so self-correction at floor scale is risky even when it appears to work.
  • The same format artifact can manufacture both apparent gain and apparent harm depending only on which side of the extraction boundary fails, so prompt wording alone can flip the sign of a headline result.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct corollary the paper leaves implicit: any evaluation pipeline that scores free-form text against an extracted answer, including agentic and code-generation harnesses, should report parseability rates before interpreting accuracy deltas.
  • The paper's squeeze result suggests a testable design: revision operators that force structured output from the start, such as constrained decoding, may eliminate the format channel entirely and leave the content margin as the only possible effect.
  • Because the frontier arm found format-dominance intensifying with scale, a larger-scale confirmation on a model with available logprobs and complete data would close the scale question the paper leaves open.
  • The two MMLU residuals (content margin near zero but constrained effect +0.048 and +0.053) likely indicate either imputation underestimation or a probe-like elicitation effect; distinguishing these would require tracking correctness on originally unparseable rows under both interpretations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript proposes an exact, per-sample decomposition of the always-revise accuracy shift in LLM self-correction into a content margin (both pre- and post-revision answers parseable) and two format margins (recover and loss), together with a 'calibration floor' criterion for when confidence-gated revision can beat always/never baselines. Empirically, across 29 admitted cells (Qwen3.5 0.8B-9B, Gemma-4-12B) plus a five-cell frontier API arm, the paper claims format margins dominate apparent self-correction effects, content margin is inert at capable scale but real and harmful at floor scale, and only a marginal gating niche exists. A grammar-constrained re-extraction control is reported to close a median 71% of the gap between naive total and content-margin estimates on eight active-channel cells. The paper is transparent about post-hoc design choices, exploratory arms, and residuals.

Significance. If the claims hold, the paper makes a valuable methodological contribution: it identifies and quantifies a measurement confound that can flip the sign of self-correction results, provides an additive decomposition applicable to any accuracy pipeline, and supports it with an unusual combination of exact synthetic validation, reproducible code and trajectory release, cross-family and scale checks, and honest reporting of exceptions. The scale contrast (content flips at 0.8B/2B, inertia at 4B-12B) and the frontier finding (content margin exactly zero despite large total effects) are striking and falsifiable. The main caveat is that the causal interpretation of the control rests on a read-out fidelity premise that is not empirically checked, and the headline statistics depend on post-hoc admission thresholds; these weaken but do not destroy the central measurement claim.

major comments (3)
  1. [§4.12 / §9.1] The constrained-decoding causal control lacks a read-out fidelity check on bothok rows. Unlike the §4.10 forced-continuation probe, which requires ≥90% reproduction of already-extracted a0 answers on a0-OK rows before its results are interpreted, §4.12 reports no equivalent check that grammar-constrained re-extraction reproduces the originally extracted a0/af answers on rows where both answers were already parseable. Without this check, closure of the total-vs-content gap could be driven by the forced continuation acting as a new elicitation rather than by guaranteeing parseability. The paper concedes in the same section that the control 'is itself a new (if minimal) elicitation event, not a passive read-out of the frozen text,' and §11 states C1 is not fully causally settled on the two MMLU cells; however, the abstract's 'test the failure causally' and the median-71% closure statistic are presented as the causal evidence for C1. Because the required check is a straightforward re-run of the released constrained-decoding harness on bothok rows, this omission is fixable and is load-bearing for the causal interpretation.
  2. [§4.9 / §6.3 / §9.1] The two thresholds that define the headline C1 comparison — the 25% extraction-completeness admission gate and the ≥5% 'active extraction channel' threshold for the 12-cell Wilcoxon test — are both post-hoc and data-informed. The admission gate was introduced after the first grid audit, and the activity threshold selects the cells on which |Δ_format| > |Δ_content| is claimed with p=1.7e-3. The paper discloses this in §6.3 and §11, but reports no sensitivity analysis over reasonable threshold choices (e.g., gate at 20% or 30%, activity at 3% or 8%). Since the central claim is defined by these thresholds, the conclusion would be more robust if the qualitative result were shown to persist under a range of values.
  3. [§9.1, closure statistic] The closure fraction is defined as 1 − |Δ_constrained_total − Δ_content| / |Δ_original_total − Δ_content|, where Δ_content is computed on the original bothok subgroup under the convention that unparseable answers are wrong. The two largest-effect cells (4B/9B MMLU) retain residuals of +0.048/+0.053 against a content-margin estimate near zero; on 4B MMLU the constrained total (+0.048) is 46% of the original total (+0.105). The paper offers two candidate explanations without adjudicating, which is honest, but the residual means the quantitative 'median 71% closure' may overstate the format-repair share if any of that residual reflects new elicitation. Reporting the closure against the imputation bounds of §3.3, or per-row answer agreement on bothok rows, would delimit the range of plausible format-repair shares and make the headline statistic less ambiguous.
minor comments (5)
  1. [§4.6] Signal D is referenced in §3.4 and §6.1 but not defined in the signal list (A, A', B, C); please define it or state explicitly that it is out of scope.
  2. [Abstract] 'Content is a minority share of what the field has measured as self-correction' is a generalization beyond the tested benchmarks and models; the limitations section is appropriately cautious, but the abstract should echo that caution (e.g., 'in the tested cells').
  3. [Throughout] There are several typographical issues: inconsistent Unicode fi ligature 'insufficient', 'AL W A YS' in §3.2, 'T otal' in §6.1, and occasional 'SS3.2' instead of '§3.2'; these should be cleaned.
  4. [Figure 4] Figure 4 relies on color (amber/blue/purple) without explicit grayscale markers; please add hatching or labels for accessibility.
  5. [§4.10] The probes were run against reconstructed pre-fix prompts and a fallback parser was added during offline reparse; this is disclosed, but the two truncation-regime ARC cells that fail the fidelity control (0.28–0.43) suggest the pre-fix MCQ extraction is unreliable even for 'successful' reads; consider stating this more prominently in the main text rather than only in the probe subsection.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the margin decomposition is exact bookkeeping, the causal control is an independent intervention with honestly reported residuals, and self-citations are contextual rather than load-bearing.

full rationale

I walked the paper's derivation chain from the per-sample identity (§3.2) through the margin decomposition (§3.3), the causal control (§4.12), the scale contrast (§9.2), and the calibration-floor criterion (§9.3). The decomposition Δtotal = Δcontent + Δformat_recover + Δformat_loss is an exact accounting identity over the paper's own extraction categories (Appendix B), not a fitted quantity, and the claim that format margins dominate is an empirical comparison on measured cells rather than a consequence of the definition. The constrained-decoding control is not circular: it intervenes on already-frozen reasoning text by forcing grammar-parseable output, and the paper explicitly disclaims that this is a passive read-out ('it does not, and cannot, identify whether the model's original free-form generation already carried the same latent answer, because forcing a constrained continuation is itself a new (if minimal) elicitation event'), reports residuals on 4B/9B MMLU, and gives falsification conditions in §3.4. AUROC* is solved numerically from measured (acc0, r, d) under a fixed δ=0.01 threshold, so it is not an input fitted to reproduce the conclusion. Self-citations [8], [16], [17] are contextual positioning and are not used to justify any load-bearing premise. The potential validity threat noted by reviewers—the absence of a bothok read-out fidelity check for the constrained control—is a correctness/identification concern, not circularity, and the paper acknowledges the narrower sense of its causal claim. No step reduces, by the paper's own equations or by self-citation, to its own inputs.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

No new physical or mechanistic entities are postulated. The central claims depend on the extraction convention (unparseable scored wrong), the post-hoc 25% admission gate, the frozen-trajectory protocol, the hand-set delta threshold, and the exploratory best-signal selection; all are disclosed, but several are data-informed rather than prespecified.

free parameters (4)
  • delta (profitability threshold) = 0.01
    Hand-set minimum profitable gain in the delta-corrected calibration floor; determines AUROC* and which cells pass the floor.
  • extraction-completeness admission threshold = 25% bad (0.25)
    Post-hoc cell admission threshold: both a0-bad and yf-bad rates must be below 0.25; all primary analyses use cells admitted under this data-informed rule.
  • C2 inertia bounds = |Delta_content| <= 0.03; content change rate <= 0.05
    Hand-chosen bounds defining inert capable-scale cells in claim C2; the count of cells satisfying the bounds depends on these thresholds.
  • forced-continuation probe decision thresholds = probe acc >= 0.8x revision acc; fidelity >= 0.90
    Hand-set decision rule for whether the forced continuation counts as evidence; affects which probe cells are classified as pass or fail.
assumptions (6)
  • domain assumption Unparseable answers are scored wrong on both sides of the comparison.
    Defines Delta_content, Delta_format-recover, Delta_format-loss and the imputation bounds in Section 3.3; also matches standard accuracy pipelines.
  • domain assumption The frozen always-revise trajectory with offline policy evaluation supports counterfactual gating estimates.
    Gating only decides whether to keep the initial answer or adopt the revision; used to evaluate NEVER, ALWAYS, and gated policies on one trajectory per cell (Section 4.1).
  • ad hoc to paper The grammar-constrained continuation in Section 4.12 is a minimal elicitation that does not induce new reasoning.
    The causal control's closure statistic is interpretable as format removal only if the forced continuation does not change the latent answer; the paper itself flags this as a new elicitation event.
  • domain assumption Benchmark labels and reference answers are ground truth for correctness.
    Correctness y0/yf are measured against human-annotated labels for GSM8K, MATH, MMLU, ARC, HumanEval, MBPP, TriviaQA, CommonsenseQA, and TruthfulQA-MC1 (Section 4.4).
  • ad hoc to paper The binormal ROC model is an adequate approximation for computing AUROC* thresholds.
    Used in Sections 3.2 and 9.3 for the floor criterion; the paper later checks fit (R-squared 0.898-0.985) and supplements with a nonparametric floor call that reverses two binormal false positives.
  • standard math Law of total probability and the definitions of conditional probability underpin the per-sample identity in Section 3.2.
    The identity Delta(tau) = (1-acc0)*TPR*r - acc0*FPR*d is derived from these definitions; no new mathematics is introduced.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Calibration Floor: Format Repair Can Masquerade as Self-Correction at Small-to-Mid Scale." pith.science (2026). https://pith.science/paper/BJH4GYKI

@misc{pith2026260804355,
  author       = {Pith},
  title        = {Pith review of: The Calibration Floor: Format Repair Can Masquerade as Self-Correction at Small-to-Mid Scale},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BJH4GYKI}},
  note         = {Machine review of arXiv:2608.04355}
}
read the original abstract

Accuracy changes after language-model self-revision are usually interpreted as changes in reasoning. We show this can fail at the answer-extraction boundary, and test the failure causally rather than only observationally. Across Qwen3.5 (0.8B-9B), Gemma-4-12B, and two frontier models via API (Tencent Hy3, Nvidia Nemotron-3-Ultra-550B) in 29 primary cells plus a frontier arm, we decompose the always-revise accuracy shift into a content margin (both answers parseable) and format-recovery/loss margins (parseability changes). On 12 cells with meaningful unparseable-answer rates, format effects exceed content effects (Wilcoxon p=1.7e-3). To test this causally, we force already-generated reasoning through grammar-constrained decoding so every answer is parseable by construction: across 14 cells this closes a median 71% of the gap between the naive total effect and the content-margin estimate, with two cells converging exactly and a residual on the two largest-effect cells reported rather than dismissed. A clustered model confirms floor-scale (0.8B/2B) models have far higher odds of content-level change and harm than capable-scale models (p<1e-7). Replicating a cited confidence-gating protocol verbatim on Qwen3.5 does not reproduce its reported gain and shows the same near-zero content margin. A frontier check on much larger models shows format-dominance intensifying with scale: content margin is exactly zero in all 5 cells despite total effects up to +0.275, though this arm is lower-powered. The calibration-floor criterion on the content margin reveals a squeeze: floor-scale cells have headroom but insufficient signal, capable-scale cells have signal but little headroom; only one cell is marginally viable, with negligible sealed-holdout gain. Content is a minority share of what the field has measured as self-correction. We release the instrument, code, and derived results.

Figures

Figures reproduced from arXiv: 2608.04355 by the authors.

Figure 1
Figure 1. Left — Δ(𝜏) from helpful−harmful vs. the identity formula; points on the diagonal. Right — ΔALWAYS vs. 𝜆; profit only for 𝜆 < 1. Check 3: 𝛿-floor separates pass/fail cells. On a five-cell synthetic grid with known (acc0 , 𝑟, 𝑑, AUROC), the floor (𝛿 = 0.01) predicts whether max𝜏 Δ(𝜏) ≥ 𝛿 with 100% agreement (5/5) [PITH_FULL_IMAGE:figures/full_fig_p013_1.png] view at source ↗
Figure 2
Figure 2. Left — binormal ROCs vs. the 𝜆 floor ray for a high-𝑑 cell. Right — measured AUROC vs. 𝛿-floor AUROC* across synthetic cells [PITH_FULL_IMAGE:figures/full_fig_p014_2.png] view at source ↗
Figure 3
Figure 3. Left — explicit ROC curves with implicit flip points ( [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Margin decomposition of the ALWAYS-revise effect for all 29 admitted cells. Signed [PITH_FULL_IMAGE:figures/full_fig_p020_4.png]
Figure 5
Figure 5. Figure 5: The squeeze plane. x: content headroom (oracle [PITH_FULL_IMAGE:figures/full_fig_p023_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 8 canonical work pages

  1. [1]

    Self-refine: Iterative refinement with self-feedback

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegr- effe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bod- hisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. InAdvances in Neural Infor- mation Processi...

    arXiv 2023

  2. [2]

    Reflexion: Language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. InAdvances in Neural Information Processing Systems (NeurIPS) , 2023. URLhttps://arxiv.org/abs/2303.1 1366

    work page 2023

  3. [3]

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. STaR: Bootstrapping reasoning with reasoning. InAdvances in Neural Information Processing Systems (NeurIPS) , 2022. URL https://arxiv.org/abs/2203.14465

    arXiv 2022

  4. [4]

    Self-rewarding language models

    Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. In International Conference on Machine Learning (ICML), 2024. URLhttps://arxiv.org/abs/2401.10020

    arXiv 2024

  5. [5]

    Large language models cannot self-correct reasoning Yet

    Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning Yet. InInterna- tional Conference on Learning Representations (ICLR) , 2024. URLhttps://arxiv.org/abs/23 10.01798

    work page 2024

  6. [6]

    When can LLMs actually correct their own mistakes? A critical survey of self-correction of LLMs.Transactions of the Association for Computational Linguistics , 12, 2024

    Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han, and Rui Zhang. When can LLMs actually correct their own mistakes? A critical survey of self-correction of LLMs.Transactions of the Association for Computational Linguistics , 12, 2024. URLhttps://arxiv.org/abs/2406.01297

    arXiv 2024

  7. [7]

    Whendoesintrinsicself-correction help? a task-sensitive analysis, 2026

    ElroyStav, DvirBerlowitz, MaayanOrner, andSaritKraus. Whendoesintrinsicself-correction help? a task-sensitive analysis, 2026. URLhttps://arxiv.org/abs/2606.23196

    arXiv 2026

  8. [8]

    Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops

    Mingguang Chen, Licheng Wang, and Bo Qu. Recursive self-improvement in AI: From bounded self-refinement to autonomous research loops. arXiv preprint arXiv:2607.07663 , 2026. URL https://arxiv.org/abs/2607.07663

    work page Pith review arXiv 2026

Show all 28 references
  1. [9]

    LLMs cannot find reasoning errors, but can correct them given the error location

    Madeline Tyen, Hongjin Su, Colin Raffel, Noah Shinn, Federico Cassano, Fazl Barez, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. LLMs cannot find reasoning errors, but can correct them given the error location. InFindings of the Association for Computational Linguistics...

    2024 arXiv

  2. [10]

    Xing, and Kun Zhang

    Loka Li, Zhenhao Chen, Guangyi Chen, Yixuan Zhang, Yusheng Su, Eric P. Xing, and Kun Zhang. Confidence matters: Revisiting intrinsic self-correction capabilities of large language models, 2024. URLhttps://arxiv.org/abs/2402.12563. 34

    2024 arXiv

  3. [11]

    Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, John Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom B

    Saurav Kadavath, Tom Conerly, Amanda Askell, Thomas Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zachary Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, SheerEl-Showk, AndyJones, NelsonElhage, TristanHume, AnnaChen, YuntaoBai, SamuelR. Bowman, Stanisl...

    2022 arXiv

  4. [12]

    C. K. Chow. An optimum character recognition system using decision functions.IRE Trans- actions on Electronic Computers , EC-6(4):247–254, 1957

    1957

  5. [13]

    Selectiveclassificationfordeepneuralnetworks

    YonatanGeifmanandRanEl-Yaniv. Selectiveclassificationfordeepneuralnetworks. Advances in Neural Information Processing Systems (NeurIPS) , 2017. URLhttps://arxiv.org/abs/1705 .08500

    2017

  6. [14]

    When are likely answers right? on sequence probability and correctness in LLMs, 2026

    Johannes Zenn and Jonas Geiping. When are likely answers right? on sequence probability and correctness in LLMs, 2026. URLhttps://arxiv.org/abs/2606.27359

    2026 arXiv

  7. [15]

    Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. SelfCheckGPT: Zero-resource black- box hallucination detection for generative large language models. In Empirical Methods in Natural Language Processing (EMNLP), 2023. URLhttps://arxiv.org/abs/2303.08896

    2023 arXiv

  8. [16]

    Separating error structure from marginal accuracy in self-refinement, 2026

    Mingguang Chen. Separating error structure from marginal accuracy in self-refinement, 2026. Companion preprint in the same series

    2026

  9. [17]

    Excess self-confirmation: Separating self-evaluation drift from the opti- mizer’s curse, 2026

    Mingguang Chen. Excess self-confirmation: Separating self-evaluation drift from the opti- mizer’s curse, 2026. Companion preprint in the same series

    2026

  10. [18]

    Qwen3.5-omni technical report

    Qwen Team. Qwen3.5-omni technical report. arXiv preprint arXiv:2604.15804 , 2026. URL https://arxiv.org/abs/2604.15804

    2026 arXiv

  11. [19]

    Qwen3 technical report

    Qwen Team. Qwen3 technical report. arXiv preprint arXiv:2505.09388 , 2025. URL https: //arxiv.org/abs/2505.09388

    2025 arXiv

  12. [20]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. U...

    arXiv

  13. [21]

    Measuring mathematical problem solving with the MATH dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In NeurIPS Datasets and Benchmarks Track , 2021. URLhttps://arxiv.org/abs/2103.03874

    2021 arXiv

  14. [23]

    Program synthesis with large language models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 , 2021. URL https: //arxiv.org/abs/2108...

    2021 arXiv

  15. [24]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR) , 2021. URLhttps://arxiv.org/abs/2009.033 00

    2021

  16. [25]

    Think you have solved question answering? try ARC, the AI2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the AI2 reasoning challenge. In arXiv preprint arXiv:1803.05457 , 2018. URLhttps://arxiv.org/abs/1803.05457

    2018 arXiv

  17. [26]

    TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension

    Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics , 2017. URLhttps://arxiv.or g/abs/1...

    2017 arXiv

  18. [27]

    CommonsenseQA: A question answering challenge targeting commonsense knowledge

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. InProceedings of NAACL- HLT, 2019. URLhttps://arxiv.org/abs/1811.00937

    2019 arXiv

  19. [28]

    TruthfulQA: Measuring how models mimic human falsehoods

    Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics, 2022. URLhttps://arxiv.org/abs/2109.07958. 36

    2022 arXiv

  20. [2021]

    HumanEval

    URL https://arxiv.org/abs/2107.03374. HumanEval

    arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.