Heteroskedastic Signals in Budgeted LLM Verification: Structural Heterogeneity Limits Optimization Gains

Jinlong Yang

REVIEW 5 major objections 5 minor 5 references

Heteroskedastic Signals in Budgeted LLM Verification: Structural Heterogeneity Limits Optimization Gains

T0 review · 5 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Budgeted LLM verification fails when uncertainty quality differs across cost strata, and simple cost-stratified thresholds outperform stronger global learning.

desk verdict A well-engineered empirical study with an honest limitations section, but the headline CST gains are not cleanly separated from a budget-spending confound; it deserves a serious referee but needs per-policy cost utilization reported. read the letter →

arxiv 2606.15841 v2 pith:3G6U7UKQ submitted 2026-06-14 cs.AI

Jinlong Yang This is my paper · ORCID

classification cs.AI

keywords budgetedverificationselectivecomputeglobalsignalcomparabilitycoststratificationuncertaintyheteroskedasticityLLMresourceallocationonlineadaptationthresholdingpolicies

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tests a common assumption in selective-compute LLM systems: that a single uncertainty score means the same thing across all inputs. In budgeted verification, it finds the assumption breaks down—signal quality varies sharply across cost strata, and some high-error regions have near-random confidence signals. It builds an intervention hierarchy that separates weak signals, optimizer instability, and structural mismatch, and shows that a deliberately simple cost-stratified thresholding policy improves hit rate by up to 17 percentage points where heterogeneity is strong, while global online adaptation gives inconsistent gains. The broader claim is that misaligned feedback structure cannot always be repaired by stronger optimization: before optimizing harder over a shared proxy, test whether the proxy is decision-comparable across observable regimes.

What carries the argument

The central object is cost-stratified thresholding (CST): a no-learning policy that partitions samples into cost strata by an ex-ante input-length-normalized proxy cost, then applies independent running quantile thresholds on the cost-normalized score s/chat within each stratum. It is deliberately weaker than the global learners it is compared against. The theoretical load-bearer is Proposition 1, an allocation-distortion bound: the gap between global and stratified error discovery is bounded by a constant times the cross-stratum dispersion of per-stratum discriminability delta_k (standardized separation between error and correct score distributions, monotonically related to signal-error cor

What would settle it

A direct check: rerun the MBPP/MATH comparisons with the budget constraint enforced on the true output-length cost c_t instead of the proxy, and record each policy's actual true-cost spend. If CST's hit-rate advantage over Threshold shrinks to zero (or reverses) once all policies are matched to equal true-cost expenditure, the claim that structural heterogeneity—rather than differential budget utilization—drives the gains is falsified. A second check: on a new benchmark, compute per-stratum rho_k and test whether CST gain monotonically tracks the dispersion term in Proposition 1; a setting wit

Watch

Extended reading notes

Core claim

The paper's central discovery is that the global signal comparability assumption—that equal scores carry equal decision value across inputs—fails in budgeted LLM verification. Uncertainty quality is heteroskedastic across cost strata: some strata have strong signal-error correlation while others concentrate errors with near-random confidence. Under a local Gaussian location model, a single global quantile induces an allocation distortion whose upper bound is controlled by cross-stratum discriminability dispersion; when all strata share the same signal quality the distortion vanishes. Empirically, global online adaptation (MP-Adapt) does not reliably beat static global thresholding, while the

Load-bearing premise

The comparison assumes that hit-rate differences between policies are meaningful without verifying that every policy consumes the same fraction of the true budget; decisions are thresholded on an input-length-normalized proxy cost that correlates only about 0.3 with the true output-length cost that drains the budget, and per-stratum quantiles select a target fraction of samples, not a target fraction of cost. If the policies spend different true budget amounts, the stratifica

Editorial extensions

If this is right

Stronger global online adaptation does not reliably improve over static thresholding when uncertainty quality varies across cost strata; aligning the decision structure matters more than optimizer sophistication.
A no-learning, cost-stratified threshold policy can recover up to 17 percentage points of hit rate in strongly heterogeneous settings, while underperforming when heterogeneity is weak or misaligned, so stratification should be gated on a comparability test.
The theoretical bound predicts that gains from stratification scale with cross-stratum signal-quality dispersion; equal signal quality across strata leaves no room for structural gain.
The diagnostic protocol extends beyond verification: before spending more compute on optimizing a shared proxy, practitioners should measure conditional signal quality across observable decision-relevant strata and compare an aligned partition against negative-control partitions.
Negative controls (random partitions, signal-strength partitions) degrade performance, confirming that the gains come from structural alignment, not from regularization or partitioning alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If this holds, global calibration or global AUROC numbers for LLM uncertainty can be actively misleading: a proxy can look useful on average while being near-useless in exactly the high-error, high-cost regions that matter for allocation.
A natural extension is to learn the stratification itself: replace the fixed cost quartile partition with adaptive or context-dependent strata driven by per-stratum signal quality, or by any observable that correlates with decision value.
One testable prediction follows from the paper's own heterogeneity index: across new benchmarks and models, the CST-over-threshold gain should be predicted by the variance of per-stratum rho_k and by cost-error correlation, not by global signal strength alone.
The result also suggests that budget-constrained verification systems should report actual cost-budget consumption per policy, not just hit rate; otherwise apparent gains may reflect differential spending on expensive high-error samples.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The thing to know: this paper's central empirical claim is plausible but not yet cleanly established. The up-to-17pp gain for cost-stratified thresholding (CST) over global thresholding may partly reflect spending more of the true cost budget, not just fixing comparability.

What's genuinely new is the intervention hierarchy (Threshold → MP-Adapt → MP-Strat → CST) that separates optimizer strength, optimization noise, and structural alignment, plus the diagnostic protocol. The theory (Proposition 1) is internally correct under its stated assumptions and honestly labeled as a bound on error-discovery value, not on hit rate. The negative controls—random partitions, signal-strength partitions, homogeneous-cost control—are good practice and support the diagnosis. The limitations section is candid, explicitly noting that policies may differ in budget utilization.

The soft spot is real and load-bearing. Decisions use a proxy cost ĉ (input-length-normalized) with only ~0.3 correlation to the true cost c (output-length-normalized) that actually consumes the budget. CST targets a fixed fraction of samples per stratum via quantiles of s/ĉ; it does not enforce a target fraction of true cost. The paper's own Table 2 shows CST boosts audit coverage in Q4 from 0.151 to 0.191 while hit rate there drops—a coverage gain that likely costs extra true budget. Without reporting per-policy true-cost consumption, the comparison is not apples-to-apples. The homogeneous-cost control rules out cost heterogeneity alone but does not address the proxy-vs-true-cost mismatch in the heterogeneous setting. The theory also bounds a different quantity (V, error-discovery value) than the metric reported throughout (hit rate). The authors acknowledge both issues, but acknowledgment is not the same as fixing them.

For whom: practitioners building selective-compute systems will find the diagnostic protocol useful; researchers in LLM resource allocation will want to see the budget confound resolved. It should go to peer review, not desk reject, because the question is important and the experimental design is thoughtful. But the revision should report actual budget utilization per policy, ideally equalize it or adjust hit rate accordingly, and reconcile the theory with the reported metric. Code/data release would also help.

I'd recommend sending to a serious venue with the request that the authors address the budget-utilization confound before acceptance.

Referee Report

5 major / 5 minor

Summary. The paper studies budgeted LLM verification, where a limited verification budget must be allocated across candidate outputs. It argues that the standard assumption of globally comparable uncertainty signals fails in practice: the discriminative quality of uncertainty scores varies across cost strata. The authors propose a controlled intervention hierarchy (Threshold, MP-Adapt, MP-Strat, CST) to separate weak signals, optimizer instability, and structural heterogeneity, and report that the deliberately simple cost-stratified thresholding (CST) improves hit rate by up to 17 percentage points over global thresholding, while global online adaptation does not reliably help. They derive a theoretical bound (Proposition 1) linking allocation distortion to cross-stratum discriminability dispersion, and present negative controls and per-stratum diagnostics. The paper concludes that structural heterogeneity, rather than optimizer weakness, is the primary bottleneck in the observed settings.

Significance. If the empirical claims are sound, this is a useful and timely contribution. The intervention hierarchy is well designed: CST is intentionally the weakest learner, so its success isolates structural alignment from optimization sophistication. The negative controls (random partitions, signal-strength partitions, Qwen3-MATH) are good practice, and reporting paired CIs over 10 seeds is a strength. Proposition 1 is transparent about its assumptions and is a genuinely diagnostic bound rather than a fitted model. The paper also has an unusually candid Limitations section that identifies several threats to validity. However, the central empirical claim is currently undermined by a budget-utilization confound: policies select a fixed fraction of samples by a proxy cost, not a fixed fraction of true cost, and no realized true-cost consumption is reported. In addition, the abstract promises an HGA method that does not appear anywhere in the body. These issues are fixable, but they must be addressed before the headline conclusion can be accepted.

major comments (5)

[§5.1, Table 2] The headline comparison is not budget-equivalent. The formal constraint in §3.1 is Σ a_t c_t ≤ β Σ c_t with c_t the true output-length-normalized cost, but every implemented policy (Threshold, MP-Adapt, MP-Strat, CST) selects by running quantiles of s(h)/ĉ_t, i.e., a fraction of samples, not a fraction of true cost. Since ĉ and c have correlation ≈0.3, the two are not interchangeable. Table 2 shows CST increases audit rate in high-cost Q4 from 0.151 to 0.191; if Q4 is also high true-cost, CST may simply spend more of the real budget. The Limitations concede 'policies may differ in budget utilization,' but no realized budget-consumption numbers are reported. Without equalizing or reporting actual true-cost utilization per policy, the +17pp CST gain and the conclusion that structural heterogeneity is the primary bottleneck are not cleanly separated from a budget-spending effect.
[Abstract] The abstract and the claimed contributions introduce Heterogeneity-Gated Allocation (HGA), described as a warm-up comparability test that chooses between global and cost-stratified allocation and 'preserves most gains and avoids blind stratification.' I could not find HGA in Sections 3–6 or in Appendices A–J: there is no definition of the warm-up test, no algorithm, and no experimental result. The paper cannot claim this result if it is not defined and evaluated. Please add the HGA method and its evaluation, or remove the claim from the abstract and introduction.
[§4.1, Eq. (2)] Proposition 1 is proved under (A3), which assumes shared error prevalence p and common scale, so the bound isolates discriminability dispersion (Source A). However, the empirical effect the paper emphasizes includes Source B — base-rate heterogeneity. In Table 2, Q4 has error rate 0.714 vs Q1 0.423, and the text explicitly credits coverage gains in high-error strata as a component of CST's improvement. Eq. (2) cannot quantify Source B, and the paper provides no bound for it. This is partly acknowledged ('analyzed separately as Source B'), but it means the theoretical result does not cover a major part of the empirical phenomenon. State this limitation wherever Proposition 1 is invoked, and ideally provide a companion bound for prevalence heterogeneity.
[§5.1, Tables 6 and 7] The Oracle results appear to violate the nominal sample budget. For MATH Qwen3, the error rate is 0.247. With β=0.50 and a 50% sample budget, the highest possible hit rate is 0.494 (select all 1235 errors among 2500 audited samples). Table 6 reports Oracle = 0.526 at β=50% and 0.938 at β=30%, both above the sample-budget ceiling (0.823 at β=30%). This suggests the Oracle is optimized with respect to true cost or uses a different constraint than the other policies. Because Oracle serves as the upper-bound reference for the hierarchy, it must be evaluated under the same budget definition as Threshold/CST, or the caption must state explicitly that it is not budget-comparable.
[§4.1 vs. §5.2] Proposition 1 bounds the error-discovery value V(π)=E[y·1{π verifies}], but the headline metric throughout Section 5 is HitRate=Σ a_t y_t / Σ a_t. These quantities respond differently to changes in audit volume: a policy that audits more samples may increase V while decreasing HitRate. The paper acknowledges this in Section 4.1, but then uses Proposition 1 to explain empirical hit-rate gains (e.g., Section 5.3: 'consistent with its smaller CST gains (Proposition 1)'). Because the bound is on V, not HitRate, and is unsigned, it does not predict the positive hit-rate gains in Table 1. Present the theory strictly as a diagnostic for allocation distortion, or report V alongside HitRate in the empirical tables.

minor comments (5)

[§5.1] The 'correlation ≈0.3' between ĉ and c is reported without per-dataset values or variability. Since the budget confound depends on this correlation, please report it for each dataset and model.
[Table 1 caption] The column header 'C+M' is not defined in the caption; it is only mentioned in §5.1 as CST+Memory. Expand it in the caption.
[Figure 3] The per-stratum ρ_k significance asterisks are difficult to distinguish in grayscale. Consider labeling the numerical values or using distinct marker shapes.
[Eq. (1)] The symbol λ is used as a generic threshold parameter in Eq. (1), but as a learned dual variable in MP-Adapt. Use different symbols to avoid confusion.
[Appendix J] In Figure 5, the label '= 0.157' in the left panel is missing the 'ρ' prefix; similarly for the other quartile labels.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the theory is a standalone bound and the empirical claims rest on external benchmarks with negative controls; the budget-utilization limitation is a confound, not a circular step.

full rationale

The paper's central derived result, Proposition 1, is a self-contained analytic bound (Appendix A) proved from explicit assumptions (A1)-(A3); it is not defined in terms of the empirical hit-rate comparisons, and the paper explicitly states it 'does not determine the sign of the difference or directly bound the empirical hit-rate ratio.' The empirical core uses external benchmarks (MBPP, MATH) and multiple model families, with negative controls (random partitions, signal-strength partitions, homogeneous-cost control, Qwen3-MATH) that make the CST-vs-Threshold contrast an actual empirical test rather than a tautology. The heterogeneity index H=Var_k(rho_k) is used descriptively/post hoc, not fitted and then used to 'predict' the same data. There are no self-citations, uniqueness imports, or ansatz-by-citation chains. The Limitations passage ('Hit rate is cost-unweighted, policies may differ in budget utilization') identifies a genuine validity concern about comparing hit rates under a proxy-cost budget, but that is a confounding/budget-utilization threat, not an equivalence of the derivation to its inputs. It does not make any claimed result definitionally forced by construction. Overall, the derivation chain is not circular.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central claim rests on the choice of cost strata (K=4), the fixed score weights, and the injected noise. The theory relies on assumptions A1–A3 that are partially contradicted by the empirical base-rate heterogeneity (Source B).

free parameters (5)

w0 (initial score weights) = not reported (estimated from first 50 samples)
Used by Threshold and CST to combine three uncertainty signals; fitted from warm-up data, affecting all subsequent decisions.
K (number of cost strata) = 4
Chosen by hand; Appendix I shows sensitivity, with over-partitioning degrading performance. Central to CST.
Label noise / proxy noise = 10% Bernoulli label noise; Gaussian proxy noise σ=0.1
Injected into the data-generating process by the authors; affects observed signal quality and heteroskedasticity.
MP-Adapt hyperparameters (ηθ, decay, γ, λ_floor) = ηθ=0.5, decay=0.002, γ=0.9, λ_floor=0.1
Hand-chosen; determine MP-Adapt/MP-Strat behavior and hence the negative result that stronger global optimization does not help.
Warm-up window T0 = 50 samples
Used to fix w0 in Threshold/CST; affects early decisions.

assumptions (6)

domain assumption (A1) The error-weighted density g_k(z) = p f_k(z|y=1) is bounded above by M.
Used in Proposition 1 to bound change in discovered error mass under threshold perturbation.
standard math (A2) The mixture density of z is lower bounded by ℓ > 0 near the relevant quantiles.
Needed for the quantile-perturbation bound (Eq. 3).
ad hoc to paper (A3) Strata share error prevalence and scale and differ only via Gaussian location-separation parameter δ_k.
A stylized local model chosen to make the dispersion bound tractable; contradicted by observed base-rate variation (Source B) which the paper analyzes separately.
domain assumption Ex-ante cost ĉ_t is an observable, decision-relevant stratification variable.
The paper stratifies by ĉ_t and shows it captures heterogeneous signal quality; if ĉ_t were unrelated to signal quality, CST would lose its advantage.
domain assumption The three token-level log-probability features (average negative log-likelihood, min margin, margin dispersion) are valid uncertainty proxies.
These features are used as the basis of the decision score s(ht) = θ^T h_t; no validation of their calibration is provided.
standard math Paired differences across 10 seeds are approximately normal for Student-t CIs.
Used to report 95% CIs; with 10 seeds, normality is assumed.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Heteroskedastic Signals in Budgeted LLM Verification: Structural Heterogeneity Limits Optimization Gains." pith.science (2026). https://pith.science/paper/3G6U7UKQ

@misc{pith2026260615841,
  author       = {Pith},
  title        = {Pith review of: Heteroskedastic Signals in Budgeted LLM Verification: Structural Heterogeneity Limits Optimization Gains},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3G6U7UKQ}},
  note         = {Machine review of arXiv:2606.15841}
}

read the original abstract

Selective-compute LLM systems decide which outputs merit verification, additional reasoning, tool execution, or human audit under a limited budget. It is natural to expect that stronger online optimization over a shared uncertainty or reward signal should improve these decisions. We take a critical look at this assumption and ask: when does optimizing harder fail because the signal is not decision-comparable across inputs? In budgeted LLM verification, we find that uncertainty quality is heteroskedastic across cost strata: some regions exhibit near-random discriminability while concentrating many errors. Under an explicit local model, we characterize the resulting distortion of global allocation and show that its upper bound scales with cross-stratum signal-quality dispersion. To separate weak signals from optimizer instability and structural mismatch, we introduce a controlled intervention hierarchy: Threshold, MP-Adapt, MP-Strat, and cost-stratified thresholding (CST). We then turn the diagnosis into Heterogeneity-Gated Allocation (HGA), which uses a warm-up comparability test to choose between global and cost-stratified allocation. Across MBPP and MATH using Qwen3-8B, LLaMA3-8B, and GPT-4o-mini, global online adaptation yields inconsistent gains over static thresholding; CST improves hit rate by up to 17 percentage points in strongly heterogeneous settings, while HGA preserves most gains and avoids blind stratification when the partition is not useful. These findings suggest a resource-allocation principle for LLM systems: before optimizing harder over a shared proxy, test whether the proxy is decision-comparable across observable operating regimes, and gate structural specialization on that test.

Figures

Figures reproduced from arXiv: 2606.15841 by the authors.

**Figure 2.** Signal strength sweep under homogeneous synthetic signals ( [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** Per-stratum ρk across cost quartiles. Filled: p<.05; open: n.s.; shaded: ρk<0.10 (near-random). MBPP models exhibit strong heterogeneity in opposite directions; MATH shows weaker variation (Proposition 1). 5 10 15 20 25 30 40 50 Budget β (\%) −20 −10 0 10 20 Δ Hit R ate (p p) MBPP Qwen3 MBPP LLaMA3 MBPP GPT-4o MATH Qwen3 MATH LLaMA3 [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: CST gain over Threshold (∆ hit rate, pp) across budgets β. CST consistently improves MBPP, but exhibits heterogeneous behavior on MATH depending on model and budget. (H=0.0065) yields +5.1 vs. +15.1pp; GPT-4omini (H=0.0025) achieves +6.5pp via extreme Q4 signal collap…

**Figure 5.** Figure 5: Global thresholding under heteroskedastic costs. Synthetic score distributions parameterized by empirical [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

5 extracted references · 4 linked inside Pith

[4]

InInternational Conference on Learning Representations, volume 2024, pages 39578–39601

Let’s verify step by step. InInternational Conference on Learning Representations, volume 2024, pages 39578–39601. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, and 1 others. 2023. Self-refine: Iterative refinement with self-feedback.Advances in neural informat...

arXiv 2024
[1134]

Santiago Balseiro, Haihao Lu, and Vahab Mirrokni

PMLR. Santiago Balseiro, Haihao Lu, and Vahab Mirrokni
[2020]

InInternational Conference on Machine Learn- ing, pages 613–628

Dual mirror descent for online allocation prob- lems. InInternational Conference on Machine Learn- ing, pages 613–628. PMLR. Lingjiao Chen, Matei Zaharia, and James Zou. 2023. Frugalgpt: How to use large language models while reducing cost and improving performance.arXiv preprint arXiv:2305.05176. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique...

arXiv 2023
[2023]

InProceedings of the 40th International Conference on Machine Learning (ICML’23)

Lever: Learning to verify language-to-code generation with execution. InProceedings of the 40th International Conference on Machine Learning (ICML’23). Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E Gonzalez, M Waleed Kadous, and Ion Stoica. 2024. Routellm: Learning to route llms with preference data.arXiv preprint arXiv:2406...

arXiv 2024
[2024]

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others

Large language model validity via enhanced conformal prediction methods.Advances in Neural Information Processing Systems, 37:114812–114842. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problem...

arXiv 2021

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.

[1] [4]

InInternational Conference on Learning Representations, volume 2024, pages 39578–39601

Let’s verify step by step. InInternational Conference on Learning Representations, volume 2024, pages 39578–39601. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, and 1 others. 2023. Self-refine: Iterative refinement with self-feedback.Advances in neural informat...

arXiv 2024

[2] [1134]

Santiago Balseiro, Haihao Lu, and Vahab Mirrokni

PMLR. Santiago Balseiro, Haihao Lu, and Vahab Mirrokni

[3] [2020]

InInternational Conference on Machine Learn- ing, pages 613–628

Dual mirror descent for online allocation prob- lems. InInternational Conference on Machine Learn- ing, pages 613–628. PMLR. Lingjiao Chen, Matei Zaharia, and James Zou. 2023. Frugalgpt: How to use large language models while reducing cost and improving performance.arXiv preprint arXiv:2305.05176. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique...

arXiv 2023

[4] [2023]

InProceedings of the 40th International Conference on Machine Learning (ICML’23)

Lever: Learning to verify language-to-code generation with execution. InProceedings of the 40th International Conference on Machine Learning (ICML’23). Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E Gonzalez, M Waleed Kadous, and Ion Stoica. 2024. Routellm: Learning to route llms with preference data.arXiv preprint arXiv:2406...

arXiv 2024

[5] [2024]

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others

Large language model validity via enhanced conformal prediction methods.Advances in Neural Information Processing Systems, 37:114812–114842. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problem...

arXiv 2021