Beyond Post-Hoc Temperature Scaling: Bilevel Optimization for LLM Calibration

Bojian Hou; Jiancong Xiao; Ruochen Jin; Zhanliang Wang; Zongyu Dai

REVIEW 4 major objections 4 minor 300 references

Beyond Post-Hoc Temperature Scaling: Bilevel Optimization for LLM Calibration

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that entropy-based bilevel fine-tuning, not post-hoc temperature scaling, keeps aligned LLMs calibrated when the domain shifts.

desk verdict A sensible training-time calibration method whose OOD headline numbers may reflect target-domain hyperparameter selection; worth serious review and a revision, but the zero-shot generality claim is overreaching as written. read the letter →

arxiv 2608.07419 v1 pith:BWRBMXAL submitted 2026-08-07 cs.LG

Ruochen Jin , Zhanliang Wang , Zongyu Dai , Jiancong Xiao , Bojian Hou This is my paper · ORCID

classification cs.LG

keywords LLMcalibrationbileveloptimizationtemperaturescalingentropymaximizationout-of-domaingeneralizationexpectederrorpreferencealignmentoverconfidence

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Preference-aligned LLMs tend to be overconfident, and the standard remedy — fitting a temperature on a validation set after training — transfers poorly across domains. This paper argues the fix belongs in the training loop: fine-tune the model so that a temperature-like adjustment generalizes, by formulating a bilevel problem in which the lower level minimizes the ordinary next-token cross-entropy while the upper level tunes per-token logit adjustments to maximize the entropy of the predictive distribution on held-out inputs. The claim is that this entropy target, kept in check by the task-optimality constraint, moves overconfident aligned models toward confidence that matches accuracy — and does so where a single fitted temperature cannot, with the largest gains in out-of-domain multiple-choice and generative question answering. If the paper is right, calibration can be built into alignment-time fine-tuning instead of bolted on per dataset, at the price of a few hours of extra training and no inference-time overhead.

What carries the argument

The load-bearing object is a bilevel optimization whose upper objective is entropy maximization applied to a temperature-parameterized predictive distribution. The distribution uses per-vocabulary adjustments $(d_y, l_y)$ — a multiplicative rescaling $d_y := \tau_y^{-1}$ of each token logit plus an additive shift $l_y$ — which generalize the single global temperature of temperature scaling and can redistribute confidence across answer options rather than only sharpen or flatten the whole distribution. The upper level maximizes predictive entropy on held-out inputs; the lower level requires the model parameters to remain optimal for the ordinary cross-entropy task loss. Second-order hypergradients are avoided by a first-order, value-function-based scheme that enforces lower-level optimality through a plug-in residual $\hat{q} = L_{\mathrm{tr}}(\theta;\alpha) - L_{\mathrm{tr}}(\tilde{\theta};\alpha)$ built from a short inner optimization, with the joint update direction $d = g_{\mathrm{ent}} + \lambda g_q$ combining the entropy gradient with an adaptive multiple of the residual gradient. The ablations identify that coupling term — not the entropy penalty alone — as what the calibration gains require.

What would settle it

Take the released code and one model from Table 1, record the learned per-vocabulary adjustments $(d_y, l_y)$, apply them only to the logits of the four answer tokens A, B, C, and D, and recompute confidence ECE on the MCQA test set. If the reported out-of-domain improvement (for example, Llama-3.1 from $0.1784$ to $0.1050$) does not reproduce under that restriction, then the claimed transfer from full-vocabulary entropy maximization to four-option calibration does not follow from the proposed mechanism.

Watch

Extended reading notes

Core claim

The paper's central claim is that maximizing the Shannon entropy of the predictive distribution, as the objective of an upper-level optimization, is the right training-time objective for calibrating aligned LLMs. Post-hoc temperature scaling is reframed as a special two-stage procedure — fit the model, then rescale logits — and generalized into a bilevel problem: the lower level trains the model under a parametric cross-entropy loss, and the upper level selects temperature-like hyperparameters $\alpha = (l,\tau)$, consisting of additive logit shifts and per-class temperatures, to maximize entropy on held-out inputs subject to the model remaining optimal for the task loss. Because miscalibration in aligned models is predominantly overconfidence, raising entropy moves the model from an overconcentrated regime toward a point where confidence matches accuracy, while the lower-level constraint stops the entropy objective from eroding discrimination. The reported experiments, on four 7-8B DPO- or RLHF-aligned models, show that this procedure (CALM) lowers confidence ECE and classwise ECE relative to the uncalibrated baseline and to training-time baselines including calibration-aware fine-tuning and label smoothing, with the clearest margins in out-of-domain multiple-choice QA and in cross-domain generative QA measured by semantic ECE, while preserving most of the task accuracy.

Load-bearing premise

The central premise is that teaching an already-aligned model on general instruction data to spread its next-word predictions more evenly will also make its four-option multiple-choice answers better calibrated, yet the paper never specifies exactly how the learned per-word adjustments are applied to the A/B/C/D options or why full-vocabulary entropy should transfer to a four-way softmax.

Editorial extensions

If this is right

In the out-of-domain multiple-choice setting, CALM attains the best confidence ECE on three of the four models, cutting confidence ECE substantially relative to the uncalibrated baseline (Llama-3.1: $0.1784 \to 0.1050$, about a 41% reduction) while keeping accuracy within about two percentage points.
Post-hoc temperature scaling is inconsistent under domain shift: on Llama-3.1 it raises confidence ECE from 0.1784 to 0.2513, whereas the training-time adjustments learned by CALM remain effective on unseen domains.
The ablations imply that the bilevel constraint itself does the work: a flat single-level entropy penalty (Regularization), even with its weight grid-tuned, and an alternating scheme without the coupling term (Iterate) both fail to match CALM, and Iterate can collapse accuracy.
In cross-domain free-form generation (trained on PopQA, tested on TriviaQA), CALM achieves the lowest mean semantic ECE ($0.0671$ versus $0.0846$ for the uncalibrated baseline), while label smoothing collapses under the shift and a tuning-budget-matched calibration-aware fine-tuning baseline still trails.
CALM preserves general capability: its out-of-domain-to-in-domain accuracy gap on five standard benchmarks is the smallest among training-based methods, and because all added machinery lives in training, inference cost is unchanged.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the upper-level entropy objective needs no ground-truth labels, a natural extension is to run CALM before preference alignment, on the base model with unlabeled text, to see whether it preempts the overconfidence that DPO and RLHF later induce; the paper only tests already-aligned models.
The large gap between scalar and per-vocabulary adjustments (confidence ECE 0.3140 for the scalar variant versus 0.0822 for CALM on Mistral-7B) suggests that aligned-model miscalibration is asymmetric across tokens, which invites testing whether the same vectorization helps other calibration objectives, such as classwise temperature scaling or binning-based losses, not just entropy maximization.
A stress test of the method's own domain-dependence would be to select CALM's hyperparameters on a validation split drawn from a different domain than the test set; if the out-of-domain gains shrink when validation and test domains are separated, some of the reported advantage may reflect tuning on the evaluation distribution rather than genuine transfer.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: CALM is a plausible training-time calibration method, and the bilevel formulation is a genuinely useful generalization of temperature scaling. But the headline OOD MCQA result is not verifiable as written, because the paper never says which validation set is used for hyperparameter selection. If it is the MCQA calibration split, the 'zero-shot' claim partly evaporates.

What is genuinely good: the entropy-maximizing upper level over a task-loss lower level, made tractable with a BOME-style first-order residual approximation, is a real methodological step and not just post-hoc TS with extra parameters. The ablations are well chosen: Regularization failing at every grid point supports the claim that the bilevel coupling, not entropy alone, does the work. They also release code and honestly report three-seed variance, showing Vicuna and OLMo stable but Llama and Mistral with large std—so the main-table margins for those two should be read cautiously, and the authors mostly acknowledge this.

The soft spots are proportionate but real. First and load-bearing: Appendix C.3 says hyperparameters are selected by lowest conf-ECE. In the OOD MCQA setup, training is on Alpaca, which has no four-option labels, so the only sensible place to compute conf-ECE is the MCQA validation split. If that is where the learning rates were chosen, then CALM, Regularization, and CFT are all being tuned on the target benchmark family before the 'zero-shot' test—and CALM's twelve grid configurations give it more chances to fit that family than Regularization's four. The paper never states which split was used, so the headline OOD generalization claim cannot be fully verified. The stress-test note is correct on this.

Second, the paper never explicitly specifies how the full-vocabulary d_y, l_y adjustments are applied to the A/B/C/D logits. That is a reproducibility gap, though likely minor if the code is clear.

Third, the open-ended evaluation uses 300 questions and a self-cited Sem-ECE metric; worth a footnote but not a dealbreaker.

Who this is for: people working on LLM calibration or training-time uncertainty methods. They should read it carefully, with an eye on the selection protocol. My recommendation: send to peer review—the core idea deserves referee time—but the revision must state the validation split explicitly, ideally report results with model selection done on Alpaca alone, and specify the logit mapping. As written, I would be skeptical of the OOD numbers until those details are pinned down.

Referee Report

4 major / 4 minor

Summary. The paper proposes CALM, a training-time calibration method for aligned LLMs. The method is formulated as a bilevel optimization problem: the lower level fine-tunes the model under a cross-entropy loss with per-vocabulary logit adjustments (multiplicative d_y and additive l_y, generalizing temperature scaling), while the upper level maximizes the entropy of the predictive distribution on held-out inputs. A first-order approximation inspired by BOME is used to avoid Hessian computation. The empirical section evaluates CALM against post-hoc temperature scaling, label smoothing, CFT, and two ablations (Regularization and Iterate) on four aligned LLMs, under multiple-choice QA in both in-domain and out-of-domain settings and under open-ended generative QA under domain shift, reporting conf-ECE, cw-ECE, and Sem-ECE. The central claim is that CALM produces well-calibrated LLMs with particular advantages in out-of-domain generalization, at a small accuracy cost and without test-time overhead.

Significance. If the empirical claims hold, the paper makes a useful contribution: it offers a training-time alternative to post-hoc temperature scaling, with an entropy-based upper-level objective that does not require labels, and a computationally tractable first-order bilevel scheme. The study is reasonably broad (four models, two task families, two calibration metrics plus a semantic ECE), and it explicitly reports seed variance, including honest acknowledgment that Llama-3.1 and Mistral-7B results are less stable. The paper also makes a concrete design claim that the bilevel coupling rather than entropy alone is responsible for the gains, which is testable. However, the central out-of-domain claims currently rest on two underspecified protocol details: the split used for hyperparameter selection, and the mapping from full-vocabulary per-token adjustments to the four-option MCQA logits. These issues are load-bearing for the paper's main narrative that calibration transfers without fitting the target distribution.

major comments (4)

[Appendix C.3 and C.7] The manuscript never states which split is used for hyperparameter selection in the OOD MCQA setting. Appendix C.3 says the best configuration per model is selected by lowest conf-ECE while maintaining accuracy above 80% of baseline, but the only split in the OOD setup on which conf-ECE is well-defined for four-option MCQA is the MCQA calibration/validation split: the Alpaca S_V split described in Appendix C.7 has no A/B/C/D labels, so conf-ECE cannot be computed there. If the MCQA validation split was used for model selection, then the 'zero-shot' OOD claim in Section 5.2 is partly a target-domain model-selection outcome, and CALM's larger grid (12 configurations, Table 5) has more opportunities to fit that target than Regularization (4) or CFT, whose MCQA alpha is fixed. The authors must state exactly which split was used; if MCQA validation was used, they must re-run the comparison with hyperparameters selected on a split that does not overlap the OOD evaluation distribution and report whether the Table 1 advantages survive.
[Section 4, Eq. (2); Section 5.1] The paper defines d_y and l_y as per-vocabulary logit adjustments over the full token vocabulary (e.g., 32,000 or 128,256 tokens), and the upper-level entropy objective is computed on the full next-token predictive distribution. The manuscript does not specify how these learned adjustments are mapped to the four extracted A/B/C/D logits in the MCQA evaluation: whether d_y and l_y are indexed by the token IDs of the option letters, whether the option logits are taken from vocabulary positions of 'A', 'B', 'C', 'D' with or without leading spaces, or whether a reduced class set is used. Without this mapping, Equation (2) cannot be instantiated on the MCQA setup, and the claimed transfer from full-vocabulary entropy training to four-way softmax calibration is not reproducible.
[Section 5.2; Appendix C.2] The text claims that Regularization is 'exactly CALM's entropy objective as a grid-tuned single-level penalty' and that its failure at 'any grid point' demonstrates that the bilevel coupling is necessary. This is overstated: Regularization runs for 30 epochs in the OOD setting versus 5 for CALM, uses only a 2x2 grid ({1e-3, 1e-4} x {1e-5, 1e-6}), and applies the entropy penalty to the training set, whereas CALM's upper level evaluates entropy on held-out validation inputs. The comparison therefore does not hold constant the data split, epoch budget, or search density, and the quoted sentence overinterprets the ablation as evidence for the necessity of the coupling.
[Section 5.3; Appendix C.7] For the generative OOD experiments in Table 3, the paper does not report which split was used to select CALM's hyperparameters or the CFT alpha sweep. Appendix C.3 describes selection by lowest conf-ECE, but conf-ECE is not defined for free-form generation, and Appendix C.7 states that Sem-ECE is computed on 300 evaluation questions with 20 samples each, without mentioning a validation partition. If the hyperparameters were chosen using Sem-ECE on the target-domain evaluation questions, the Table 3 comparison would again be affected by selection on the evaluation distribution. The authors should specify the validation protocol for the generative setting and confirm whether the reported CALM improvements persist under a fixed, non-overlapping selection procedure.

minor comments (4)

[Appendix C.2] In the CALM configuration, 'Inner optimization steps: 20 batches per epoch for the lower-level surrogate ˆx' uses an undefined symbol \hat{x}; this should refer to the surrogate model \tilde{\theta} used in Appendix A.
[Appendix C.2] In the CFT paragraph, 'no Logit 2levelwrapper' appears to be a formatting artifact; it should read 'no logit-level wrapper'.
[Table 3] The label-smoothed (LS) row reports Sem-ECE values near 0.9-1.0. The caption calls this 'LS (collapsed)', but the main text does not explain what 'collapsed' means operationally; please clarify how LS is applied in the generative setting and why it yields near-maximal Sem-ECE.
[Appendix D.2, Table 12] The scalar-versus-vector ablation is reported for a single seed on one model. Given the seed variance documented in Table 11 for Mistral-7B, the 0.0822 versus 0.3140 conf-ECE gap should ideally be accompanied by a multi-seed check, or the text should state that this is a point estimate.

Circularity Check

1 steps flagged · score 4.0 of 10

The bilevel entropy derivation is not circular, but the headline OOD MCQA advantage is partly a target-domain hyperparameter-selection outcome rather than a pure zero-shot prediction.

fitted input called prediction [Appendix C.3 (Hyperparameter Search Strategy); Table 6; Section 5.2 'OOD calibration (Primary Evaluation)']
"The best configuration per model is selected based on the lowest conf-ECE while maintaining accuracy above 80% of the baseline."

In the OOD MCQA setting, training is on Alpaca, which has no A/B/C/D labels, so conf-ECE can only be computed on the MCQA target benchmark. The paper never states whether the grid search uses the MCQA calibration split or the test split. Table 6 reports the OOD 'selected' configurations with conf-ECE values identical to the test results in Table 1 (e.g., Llama-3.1 0.1050, Vicuna-7B 0.0380). Thus the reported zero-shot OOD advantage is not an independent prediction of the entropy-bilevel objective alone: the hyperparameters, and therefore the reported conf-ECE, are chosen by the same calibration metric on the same benchmark family used for evaluation. The central empirical claim is therefore partially a model-selection outcome rather than a pure out-of-domain test.

full rationale

The method derivation in Sections 3-4 is self-contained and not circular in the mathematical sense. The upper-level entropy objective (Eq. 6) is not defined in terms of ECE; the lower-level cross-entropy (Eq. 7) preserves task performance; and the first-order BOME-inspired update is adopted from Liu et al. (2022), with the full algorithm given in Appendix A. There is no self-definitional step where entropy 'calibrates' by construction, nor is post-hoc temperature scaling re-renamed as training: the paper explicitly treats temperature scaling as a special case and extends it to per-vocabulary adjustments. The main circularity-adjacent issue is empirical. Hyperparameters for all methods are selected by lowest conf-ECE (Appendix C.3), and for OOD MCQA the only labeled data on which conf-ECE can be measured is the MCQA benchmark itself. The paper does not disclose which MCQA split is used for this selection, and Table 6's OOD conf-ECE values exactly match the test-table values, indicating that the reported OOD results are conditioned on target-benchmark model selection. This weakens the claim that 'the entropy objective does not fit a particular test distribution.' Self-citations are present (CFT from Xiao et al. 2025a and Sem-ECE from Wang et al. 2026 both involve overlapping authors), but they are not load-bearing in the derivation: Sem-ECE is fully defined in Appendix B, and CFT is a comparison baseline. Accordingly, the score is 4: the core bilevel-entropy proposal has independent content, but the flagship OOD empirical claim is partly a selected outcome rather than a clean first-principles prediction.

Assumptions & free parameters 7 free parameters · 6 assumptions · 0 invented entities

The central claim rests on learned per-vocabulary logit adjustments, a hand-chosen bilevel update coefficient, and a validation-based hyperparameter selection protocol. No new physical or mathematical entities are introduced. The largest assumptions are that full-vocabulary entropy transfer to subset logits, that the BOME surrogate is adequate at LLM scale, and that the semantic-sampling metric is a faithful calibration ground truth.

free parameters (7)

d_y multiplicative per-vocabulary logit adjustments = Learned per token, size |V|
Central calibration parameters optimized by the upper-level entropy objective during bilevel training.
l_y additive per-vocabulary logit adjustments = Learned per token, size |V|
Second set of central calibration parameters jointly optimized with d_y.
Upper learning rate for alpha = 1e-4 to 1e-7 per model
Selected by grid search using lowest conf-ECE; affects the final calibration result.
Lower learning rate for theta = 0.001 to 0.01 per model
Selected by grid search; controls the task-loss adaptation.
BOME balancing coefficient gamma = 0.5
Hand-chosen coefficient for the dynamic barrier in the joint update direction.
Inner optimization steps T = 20 batches per epoch
Chosen by hand to approximate the lower-level solution with the plug-in surrogate.
QLoRA rank = 64
Standard hyperparameter for parameter-efficient fine-tuning; affects capacity of the adaptation.

assumptions (6)

domain assumption Entropy maximization on held-out inputs improves calibration without hard labels
Section 4 Equation (8) relies on overconfidence being the dominant failure mode and on moving toward higher entropy as a route to better confidence-accuracy alignment. No proof links the bilevel solution to low ECE.
domain assumption The lower-level cross-entropy constraint preserves task performance and discrimination
Section 4 Equations (7) and (8) assume that keeping theta optimal for the task loss prevents the entropy objective from collapsing accuracy.
standard math The BOME first-order residual surrogate approximates the true bilevel solution
Appendix A relies on Danskin's theorem and the BOME framework. The plug-in residual may not bound the exact residual away from the current alpha, though it is used only at the current point.
domain assumption Per-vocabulary logit adjustments trained on next-token data transfer to four-way MCQA logits
Sections 3 and 5.1 train on Alpaca next-token data but evaluate on A/B/C/D logits. The mapping is not specified, and this transfer is load-bearing for the OOD MCQA claim.
domain assumption Sem-ECE is a valid calibration metric for open-ended generation
Section 5.1 and Appendix B use Sem-ECE from Wang et al. 2026, a paper with overlapping authors. Its validity is assumed rather than independently established here.
domain assumption Overconfidence is the predominant calibration failure mode in aligned LLMs
Section 1 motivates the entropy objective from prior work and Figure 1, rather than from a formal characterization of aligned-model miscalibration.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Beyond Post-Hoc Temperature Scaling: Bilevel Optimization for LLM Calibration." pith.science (2026). https://pith.science/paper/BWRBMXAL

@misc{pith2026260807419,
  author       = {Pith},
  title        = {Pith review of: Beyond Post-Hoc Temperature Scaling: Bilevel Optimization for LLM Calibration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BWRBMXAL}},
  note         = {Machine review of arXiv:2608.07419}
}

read the original abstract

Preference alignment often makes large language models (LLMs) overconfident and poorly calibrated. Traditional post-hoc temperature scaling is inherently domain-dependent: a temperature fitted on one domain does not generalize across domains. This motivates us to modify model parameters during training to improve calibration. We propose maximizing the entropy of predictive distributions as the calibration objective, which directly targets overconfidence by discouraging overly concentrated predictions. Inspired by temperature scaling, we realize this through a bilevel optimization formulation, where the lower level trains the model under a parametric loss and the upper level selects loss hyperparameters to maximize entropy. To make the framework practical at LLM scale, we adopt an efficient first-order approximation that avoids explicit second-order computation. Across both multiple-choice and open-ended generative question answering, experiments demonstrate that our method yields well-calibrated LLMs with particular advantages in out-of-domain generalization.

Figures

Figures reproduced from arXiv: 2608.07419 by the authors.

**Figure 2.** Classwise (Total) reliability diagrams in the [PITH_FULL_IMAGE:figures/full_fig_p027_2.png] view at source ↗

**Figure 3.** Classwise (Total) reliability diagrams in the [PITH_FULL_IMAGE:figures/full_fig_p028_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Classwise (Total) reliability diagrams in the [PITH_FULL_IMAGE:figures/full_fig_p029_4.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

300 extracted references · 5 canonical work pages

[1]

Proceedings of the International Conference on Learning Representations (ICLR) , year=

Measuring Massive Multitask Language Understanding , author=. Proceedings of the International Conference on Learning Representations (ICLR) , year=
[2]

Proceedings of the Conference on Health, Inference, and Learning (CHIL) , year=

MedMCQA: A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering , author=. Proceedings of the Conference on Health, Inference, and Learning (CHIL) , year=
[3]

Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) , year=

Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering , author=. Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) , year=
[4]

Advances in Neural Information Processing Systems , volume=

Stochastic Regret Guarantees for Online Zeroth-and First-Order Bilevel Optimization , author=. Advances in Neural Information Processing Systems , volume=
[5]

International Conference on Artificial Intelligence and Statistics , pages=

Online bilevel optimization: Regret analysis of online alternating gradient methods , author=. International Conference on Artificial Intelligence and Statistics , pages=. 2024 , organization=

2024
[6]

See https://vicuna

Vicuna: An open-source chatbot impressing gpt-4 with 90\ author=. See https://vicuna. lmsys. org , year=
[7]

arXiv preprint arXiv:2307.09288 , year=

Llama 2: Open foundation and fine-tuned chat models , author=. arXiv preprint arXiv:2307.09288 , year=

arXiv
[8]

arXiv preprint arXiv:2310.06825 , year=

Mistral 7B , author=. arXiv preprint arXiv:2310.06825 , year=

arXiv

Show all 300 references

[9]

arXiv preprint arXiv:2408.00118 , year=

Gemma 2: Improving open language models at a practical size , author=. arXiv preprint arXiv:2408.00118 , year=

arXiv
[10]

arXiv preprint arXiv:2501.00656 , year=

2 OLMo 2 Furious , author=. arXiv preprint arXiv:2501.00656 , year=

arXiv
[11]

arXiv preprint arXiv:2411.15124 , year=

TULU 3: Pushing Frontiers in Open Language Model Post-Training , author=. arXiv preprint arXiv:2411.15124 , year=

arXiv
[12]

International conference on machine learning , pages=

On calibration of modern neural networks , author=. International conference on machine learning , pages=. 2017 , organization=

2017
[13]

arXiv preprint arXiv:1701.06548 , year=

Regularizing neural networks by penalizing confident output distributions , author=. arXiv preprint arXiv:1701.06548 , year=

arXiv
[14]

Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

Rethinking the inception architecture for computer vision , author=. Proceedings of the IEEE conference on computer vision and pattern recognition , pages=
[15]

Advances in Neural Information Processing Systems , volume=

Direct preference optimization: Your language model is secretly a reward model , author=. Advances in Neural Information Processing Systems , volume=
[16]

Advances in neural information processing systems , volume=

Deep reinforcement learning from human preferences , author=. Advances in neural information processing systems , volume=
[17]

Advances in neural information processing systems , volume=

Qlora: Efficient finetuning of quantized llms , author=. Advances in neural information processing systems , volume=
[18]

Crafting Papers on Machine Learning , year =

Pat Langley , bibsource =. Crafting Papers on Machine Learning , year =. Proceedings of the Seventeenth International Conference on Machine Learning
[19]

T. M. Mitchell , institution =. The Need for Biases in Learning Generalizations , year =
[20]

M. J. Kearns , school =. Computational Complexity of Machine Learning , year =
[21]

I , year =

Machine Learning: An Artificial Intelligence Approach, Vol. I , year =
[22]

R. O. Duda and P. E. Hart and D. G. Stork , edition =. Pattern Classification , year =
[23]

Author, N. N. , title =
[24]

Newell and P

A. Newell and P. S. Rosenbloom , booktitle =. Mechanisms of Skill Acquisition and the Law of Practice , year =
[25]

A. L. Samuel , journal =. Some Studies in Machine Learning Using the Game of Checkers , volume =
[26]

On divergences and informations in statistics and information theory , volume =

Liese, Friedrich and Vajda, Igor , journal =. On divergences and informations in statistics and information theory , volume =
[27]

doi:10.57967/hf/0181 , publisher =

distilbert-base-uncased-finetuned-sst-2-english (Revision bfdd146) , url =. doi:10.57967/hf/0181 , publisher =

doi
[28]

Simple statistical gradient-following algorithms for connectionist reinforcement learning , volume =

Williams, Ronald J , journal =. Simple statistical gradient-following algorithms for connectionist reinforcement learning , volume =
[29]

f -divergences , url =

Yury Polyanskiy , keywords =. f -divergences , url =
[30]

Convex analysis , volume =

Rockafellar, R Tyrrell , publisher =. Convex analysis , volume =
[31]

PyTorch: An Imperative Style, High-Performance Deep Learning Library , url =

Adam Paszke and Sam Gross and Francisco Massa and Adam Lerer and James Bradbury and Gregory Chanan and Trevor Killeen and Zeming Lin and Natalia Gimelshein and Luca Antiga and Alban Desmaison and Andreas K. PyTorch: An Imperative Style, High-Performance Deep Learning Library ,...
[32]

Convex optimization , year =

Boyd, Stephen and Boyd, Stephen P and Vandenberghe, Lieven , publisher =. Convex optimization , year =
[33]

Convex analysis and minimization algorithms I: Fundamentals , volume =

Hiriart-Urruty, Jean-Baptiste and Lemar. Convex analysis and minimization algorithms I: Fundamentals , volume =
[34]

Information theory and statistics: A tutorial , volume =

Csisz. Information theory and statistics: A tutorial , volume =. Foundations and Trends
[35]

Entity-level Factual Consistency of Abstractive Text Summarization , url =

Nan, Feng and Nallapati, Ramesh and Wang, Zhiguo and Nogueira dos Santos, Cicero and Zhu, Henghui and Zhang, Dejiao and McKeown, Kathleen and Xiang, Bing , booktitle =. Entity-level Factual Consistency of Abstractive Text Summarization , url =. doi:10.18653/v1/2021.eacl-main.2...

doi
[36]

Lin, Chin-Yew , booktitle =
[37]

A Distributional Approach to Controlled Text Generation , url =

Muhammad Khalifa and Hady Elsahar and Marc Dymetman , bibsource =. A Distributional Approach to Controlled Text Generation , url =. Proc. of ICLR , publisher =
[38]

f -divergence Inequalities , volume =

Sason, Igal and Verd. f -divergence Inequalities , volume =. IEEE Transactions on Information Theory , number =
[39]

Neural Text Generation from Structured Data with Application to the Biography Domain , url =

Lebret, R. Neural Text Generation from Structured Data with Application to the Biography Domain , url =. Proc. of EMNLP , doi =
[40]

Controlling Conditional Language Models without Catastrophic Forgetting , url =

Korbak, Tomasz and Elsahar, Hady and Kruszewski, German and Dymetman, Marc , booktitle =. Controlling Conditional Language Models without Catastrophic Forgetting , url =
[41]

On f-divergences: Integral representations, local behavior, and inequalities , volume =

Sason, Igal , journal =. On f-divergences: Integral representations, local behavior, and inequalities , volume =
[42]

and Della Pietra, Stephen A

Berger, Adam L. and Della Pietra, Stephen A. and Della Pietra, Vincent J. , journal =. A Maximum Entropy Approach to Natural Language Processing , url =
[43]

Abstractive Text Summarization using Sequence-to-sequence

Nallapati, Ramesh and Zhou, Bowen and dos Santos, Cicero and Gul. Abstractive Text Summarization using Sequence-to-sequence. Proceedings of the 20th. doi:10.18653/v1/K16-1028 , pages =

doi
[44]

Variational Inference with Tail-adaptive f-Divergence , url =

Dilin Wang and Hao Liu and Qiang Liu , bibsource =. Variational Inference with Tail-adaptive f-Divergence , url =. Proc. of NeurIPS , editor =
[45]

Transformers: State-of-the-Art Natural Language Processing , url =

Wolf, Thomas and Debut, Lysandre and Sanh, Victor and Chaumond, Julien and Delangue, Clement and Moi, Anthony and Cistac, Pierric and Rault, Tim and Louf, Remi and Funtowicz, Morgan and Davison, Joe and Shleifer, Sam and von Platen, Patrick and Ma, Clara and Jernite, Yacine an...
[46]

Kingma and Jimmy Ba , bibsource =

Diederik P. Kingma and Jimmy Ba , bibsource =. Adam:. Proc. of ICLR , editor =
[47]

Hierarchical Neural Story Generation , url =

Fan, Angela and Lewis, Mike and Dauphin, Yann , booktitle =. Hierarchical Neural Story Generation , url =. doi:10.18653/v1/P18-1082 , pages =

doi
[48]

Courville and Yoshua Bengio , bibsource =

Dzmitry Bahdanau and Philemon Brakel and Kelvin Xu and Anirudh Goyal and Ryan Lowe and Joelle Pineau and Aaron C. Courville and Yoshua Bengio , bibsource =. An Actor-Critic Algorithm for Sequence Prediction , url =. Proc. of ICLR , publisher =
[49]

Reinforced Video Captioning with Entailment Rewards , url =

Pasunuru, Ramakanth and Bansal, Mohit , booktitle =. Reinforced Video Captioning with Entailment Rewards , url =. doi:10.18653/v1/D17-1103 , pages =

doi
[50]

Martin and Animesh Mehta and Brent Harrison and Mark O

Pradyumna Tambwekar and Murtaza Dhuliawala and Lara J. Martin and Animesh Mehta and Brent Harrison and Mark O. Riedl , bibsource =. Controllable Neural Story Plot Generation via Reward Shaping , url =. Proc. of IJCAI , doi =
[51]

Deep Reinforcement Learning for Dialogue Generation , url =

Li, Jiwei and Monroe, Will and Ritter, Alan and Jurafsky, Dan and Galley, Michel and Gao, Jianfeng , booktitle =. Deep Reinforcement Learning for Dialogue Generation , url =. doi:10.18653/v1/D16-1127 , pages =

doi
[52]

Liu, Chia-Wei and Lowe, Ryan and Serban, Iulian and Noseworthy, Mike and Charlin, Laurent and Pineau, Joelle , booktitle =. How. doi:10.18653/v1/D16-1230 , pages =

doi
[53]

The 22nd international conference on artificial intelligence and statistics , pages=

Evaluating model calibration in classification , author=. The 22nd international conference on artificial intelligence and statistics , pages=. 2019 , organization=

2019
[54]

Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining , pages=

Transforming classifier scores into accurate multiclass probability estimates , author=. Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining , pages=
[55]

Exploring the limits of transfer learning with a unified text-to-text transformer

Raffel, Colin and Shazeer, Noam and Roberts, Adam and Lee, Katherine and Narang, Sharan and Matena, Michael and Zhou, Yanqi and Li, Wei and Liu, Peter J and others , journal =. Exploring the limits of transfer learning with a unified text-to-text transformer. , volume =
[56]

Probabilistic model for code with decision trees , volume =

Raychev, Veselin and Bielik, Pavol and Vechev, Martin , journal =. Probabilistic model for code with decision trees , volume =
[57]

Black, Sid and Gao, Leo and Wang, Phil and Leahy, Connor and Biderman, Stella , doi =
[58]

The Woman Worked as a Babysitter: On Biases in Language Generation , url =

Sheng, Emily and Chang, Kai-Wei and Natarajan, Premkumar and Peng, Nanyun , booktitle =. The Woman Worked as a Babysitter: On Biases in Language Generation , url =. doi:10.18653/v1/D19-1339 , pages =

doi
[59]

On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting , url =

Tomasz Korbak and Hady Elsahar and Germ. On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting , url =. Proc. of NeurIPS , editor =
[60]

and Daly, Raymond E

Maas, Andrew L. and Daly, Raymond E. and Pham, Peter T. and Huang, Dan and Ng, Andrew Y. and Potts, Christopher , booktitle =. Learning Word Vectors for Sentiment Analysis , url =
[61]

A Diversity-Promoting Objective Function for Neural Conversation Models , url =

Li, Jiwei and Galley, Michel and Brockett, Chris and Gao, Jianfeng and Dolan, Bill , booktitle =. A Diversity-Promoting Objective Function for Neural Conversation Models , url =. doi:10.18653/v1/N16-1014 , pages =

doi
[62]

Texygen:

Yaoming Zhu and Sidi Lu and Lei Zheng and Jiaxian Guo and Weinan Zhang and Jun Wang and Yong Yu , bibsource =. Texygen:. Proc. of SIGIR , doi =
[63]

Language models are unsupervised multitask learners , volume =

Radford, Alec and Wu, Jeffrey and Child, Rewon and Luan, David and Amodei, Dario and Sutskever, Ilya and others , journal =. Language models are unsupervised multitask learners , volume =
[64]

Fine-tuning language models from human preferences , url =

Ziegler, Daniel M and Stiennon, Nisan and Wu, Jeffrey and Brown, Tom B and Radford, Alec and Amodei, Dario and Christiano, Paul and Irving, Geoffrey , journal =. Fine-tuning language models from human preferences , url =
[65]

A tutorial on energy-based learning , volume =

LeCun, Yann and Chopra, Sumit and Hadsell, Raia and Ranzato, M and Huang, Fujie , journal =. A tutorial on energy-based learning , volume =
[66]

Infinite-horizon policy-gradient estimation , volume =

Baxter, Jonathan and Bartlett, Peter L , journal =. Infinite-horizon policy-gradient estimation , volume =
[67]

Training products of experts by minimizing contrastive divergence , volume =

Hinton, Geoffrey E , journal =. Training products of experts by minimizing contrastive divergence , volume =
[68]

Global Autoregressive Models for Data-Efficient Sequence Learning , url =

Parshakova, Tetiana and Andreoli, Jean-Marc and Dymetman, Marc , booktitle =. Global Autoregressive Models for Data-Efficient Sequence Learning , url =. doi:10.18653/v1/K19-1084 , pages =

doi
[69]

Distributional reinforcement learning for energy-based sequential models , url =

Parshakova, Tetiana and Andreoli, Jean-Marc and Dymetman, Marc , journal =. Distributional reinforcement learning for energy-based sequential models , url =
[70]

Riedmiller , bibsource =

Volodymyr Mnih and Koray Kavukcuoglu and David Silver and Alex Graves and Ioannis Antonoglou and Daan Wierstra and Martin A. Riedmiller , bibsource =. Playing Atari with Deep Reinforcement Learning , url =. CoRR , timestamp =. 1312.5602 , eprinttype =

arXiv
[71]

Jordan and Pieter Abbeel , bibsource =

John Schulman and Philipp Moritz and Sergey Levine and Michael I. Jordan and Pieter Abbeel , bibsource =. High-Dimensional Continuous Control Using Generalized Advantage Estimation , url =. Proc. of ICLR , editor =
[72]

Proximal policy optimization algorithms , url =

Schulman, John and Wolski, Filip and Dhariwal, Prafulla and Radford, Alec and Klimov, Oleg , journal =. Proximal policy optimization algorithms , url =
[73]

Tuning recurrent neural networks with re-inforcement learning , url =

Jaques, Natasha and Gu, Shixiang and Turner, Richard E and Eck, Douglas , journal =. Tuning recurrent neural networks with re-inforcement learning , url =
[74]

COLD decoding: Energy-based constrained text generation with langevin dynamics , url =

Qin, Lianhui and Welleck, Sean and Khashabi, Daniel and Choi, Yejin , journal =. COLD decoding: Energy-based constrained text generation with langevin dynamics , url =
[75]

An Explanation of In-context Learning as Implicit Bayesian Inference , url =

Xie, Sang Michael and Raghunathan, Aditi and Liang, Percy and Ma, Tengyu , journal =. An Explanation of In-context Learning as Implicit Bayesian Inference , url =
[76]

Goodman, Noah D and Stuhlm\"
[77]

Buckley , bibsource =

Tomasz Korbak and Ethan Perez and Christopher L. Buckley , bibsource =. CoRR , timestamp =. doi:10.48550/arXiv.2205.11275 , eprint =

doi arXiv
[78]

Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback , url =

Yuntao Bai and Andy Jones and Kamal Ndousse and Amanda Askell and Anna Chen and Nova DasSarma and Dawn Drain and Stanislav Fort and Deep Ganguli and Tom Henighan and Nicholas Joseph and Saurav Kadavath and Jackson Kernion and Tom Conerly and Sheer El-Showk and Nelson Elhage an...
[79]

and Lowe, Ryan and Voss, Chelsea and Radford, Alec and Amodei, Dario and Christiano, Paul , booktitle =

Stiennon, Nisan and Ouyang, Long and Wu, Jeff and Ziegler, Daniel M. and Lowe, Ryan and Voss, Chelsea and Radford, Alec and Amodei, Dario and Christiano, Paul , booktitle =. Learning to Summarize from Human Feedback , year =
[80]

Advances in neural information processing systems , volume=

Training language models to follow instructions with human feedback , author=. Advances in neural information processing systems , volume=
[81]

Teaching language models to support answers with verified quotes , url =

Menick, Jacob and Trebacz, Maja and Mikulik, Vladimir and Aslanides, John and Song, Francis and Chadwick, Martin and Glaese, Mia and Young, Susannah and Campbell-Gillingham, Lucy and Irving, Geoffrey and McAleese, Nat , journal =. Teaching language models to support answers wi...
[82]

Improving alignment of dialogue agents via targeted human judgements , url =

Amelia Glaese and Nat McAleese and Maja Trebacz and John Aslanides and Vlad Firoiu and Timo Ewalds and Maribeth Rauh and Laura Weidinger and Martin Chadwick and Phoebe Thacker and Lucy Campbell. Improving alignment of dialogue agents via targeted human judgements , url =. CoRR...

doi arXiv
[83]

Language model cascades , url =

Dohan, David and Xu, Winnie and Lewkowycz, Aitor and Austin, Jacob and Bieber, David and Lopes, Raphael Gontijo and Wu, Yuhuai and Michalewski, Henryk and Saurous, Rif A and Sohl-Dickstein, Jascha and others , journal =. Language model cascades , url =
[84]

ArXiv preprint , title =

Scheurer, J. ArXiv preprint , title =
[85]

Mescheder and Andreas Geiger and Sebastian Nowozin , bibsource =

Lars M. Mescheder and Andreas Geiger and Sebastian Nowozin , bibsource =. Which Training Methods for GANs do actually Converge? , url =. Proc. of ICML , editor =
[86]

Generative Adversarial Networks , url =

Goodfellow, Ian and Pouget-Abadie, Jean and Mirza, Mehdi and Xu, Bing and Warde-Farley, David and Ozair, Sherjil and Courville, Aaron and Bengio, Yoshua , doi =. Generative Adversarial Networks , url =. Commun. ACM , number =
[87]

Mode Regularized Generative Adversarial Networks , url =

Tong Che and Yanran Li and Athul Paul Jacob and Yoshua Bengio and Wenjie Li , bibsource =. Mode Regularized Generative Adversarial Networks , url =. Proc. of ICLR , publisher =
[88]

f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization , url =

Sebastian Nowozin and Botond Cseke and Ryota Tomioka , bibsource =. f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization , url =. Proc. of NeurIPS , editor =
[89]

Wasserstein Generative Adversarial Networks , url =

Mart. Wasserstein Generative Adversarial Networks , url =. Proc. of ICML , editor =
[90]

How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary? , url =

Ferenc Huszar , journal =. How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary? , url =
[91]

Eric Zelikman and Yuhuai Wu and Jesse Mu and Noah Goodman , booktitle =
[92]

Scaling instruction-finetuned language models , url =

Chung, Hyung Won and Hou, Le and Longpre, Shayne and Zoph, Barret and Tay, Yi and Fedus, William and Li, Eric and Wang, Xuezhi and Dehghani, Mostafa and Brahma, Siddhartha and others , journal =. Scaling instruction-finetuned language models , url =
[93]

Self-critiquing models for assisting human evaluators , url =

Saunders, William and Yeh, Catherine and Wu, Jeff and Bills, Steven and Ouyang, Long and Ward, Jonathan and Leike, Jan , journal =. Self-critiquing models for assisting human evaluators , url =
[94]

Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets , url =

Solaiman, Irene and Dennison, Christy , journal =. Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets , url =
[95]

Sanh, Victor and Webson, Albert and Raffel, Colin and Bach, Stephen H. and Sutawika, Lintang and Alyafeai, Zaid and Chaffin, Antoine and Stiegler, Arnaud and Scao, Teven Le and Raja, Arun and Dey, Manan and Bari, M Saiful and Xu, Canwen and Thakker, Urmish and Sharma, Shanya S...
[96]

Challenges in Detoxifying Language Models , url =

Welbl, Johannes and Glaese, Amelia and Uesato, Jonathan and Dathathri, Sumanth and Mellor, John and Hendricks, Lisa Anne and Anderson, Kirsty and Kohli, Pushmeet and Coppin, Ben and Huang, Po-Sen , booktitle =. Challenges in Detoxifying Language Models , url =. doi:10.18653/v1...

doi
[97]

ArXiv preprint , title =

Ngo, Helen and Raterink, Cooper and Ara. ArXiv preprint , title =
[98]

ArXiv preprint , title =

Laura Weidinger and John Mellor and Maribeth Rauh and Conor Griffin and Jonathan Uesato and Po. ArXiv preprint , title =
[99]

Owen , booktitle =

Art B. Owen , booktitle =
[100]

Recipes for Building an Open-Domain Chatbot , url =

Roller, Stephen and Dinan, Emily and Goyal, Naman and Ju, Da and Williamson, Mary and Liu, Yinhan and Xu, Jing and Ott, Myle and Smith, Eric Michael and Boureau, Y-Lan and Weston, Jason , booktitle =. Recipes for Building an Open-Domain Chatbot , url =. doi:10.18653/v1/2021.ea...

doi
[101]

Ziegler and Seraphina Nix and Lawrence Chan and Tim Bauman and Peter Schmidt

Daniel M. Ziegler and Seraphina Nix and Lawrence Chan and Tim Bauman and Peter Schmidt. Adversarial Training for High-Stakes Reliability , url =. CoRR , timestamp =. doi:10.48550/arXiv.2205.01663 , eprint =

doi arXiv
[102]

An approximate sampler for energy-based models with divergence diagnostics , url =

Bryan Eikema and Germ. An approximate sampler for energy-based models with divergence diagnostics , url =. Transactions of Machine Learning Research , note =
[103]

Kappen and Vicen

Hilbert J. Kappen and Vicen. Optimal Control as a Graphical Model Inference Problem , url =. Proceedings of the Twenty-Third International Conference on Automated Planning and Scheduling,
[104]

Linearly-solvable Markov decision problems , url =

Emanuel Todorov , bibsource =. Linearly-solvable Markov decision problems , url =. Proc. of NeurIPS , editor =
[105]

Plug and Play Language Models:

Sumanth Dathathri and Andrea Madotto and Janice Lan and Jane Hung and Eric Frank and Piero Molino and Jason Yosinski and Rosanne Liu , bibsource =. Plug and Play Language Models:. Proc. of ICLR , publisher =
[106]

COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamics , url =

Qin, Lianhui and Welleck, Sean and Khashabi, Daniel and Choi, Yejin , journal =. COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamics , url =
[107]

arXiv preprint arXiv:2112.00861 , url =

A general language assistant as a laboratory for alignment , author=. arXiv preprint arXiv:2112.00861 , url =

arXiv
[108]

A note on the evaluation of generative models , url =

Lucas Theis and A. A note on the evaluation of generative models , url =. Proc. of ICLR , editor =
[109]

Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review , url =

Sergey Levine , journal =. Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review , url =
[110]

Reward Augmented Maximum Likelihood for Neural Structured Prediction , url =

Mohammad Norouzi and Samy Bengio and Zhifeng Chen and Navdeep Jaitly and Mike Schuster and Yonghui Wu and Dale Schuurmans , bibsource =. Reward Augmented Maximum Likelihood for Neural Structured Prediction , url =. Proc. of NeurIPS , editor =
[111]

Reinforcement learning by reward-weighted regression for operational space control , url =

Jan Peters and Stefan Schaal , bibsource =. Reinforcement learning by reward-weighted regression for operational space control , url =. Proc. of ICML , doi =
[112]

Srinivasa , bibsource =

Liyiming Ke and Sanjiban Choudhury and Matt Barnes and Wen Sun and Gilwoo Lee and Siddhartha S. Srinivasa , bibsource =. Imitation Learning as f-Divergence Minimization , url =. Algorithmic Foundations of Robotics XIV, Proceedings of the Fourteenth Workshop on the Algorithmic ...

doi
[113]

A Divergence Minimization Perspective on Imitation Learning Methods , url =

Ghasemipour, Seyed Kamyar Seyed and Zemel, Richard and Gu, Shixiang , booktitle =. A Divergence Minimization Perspective on Imitation Learning Methods , url =
[114]

Training a helpful and harmless assistant with reinforcement learning from human feedback , url =

Bai, Yuntao and Jones, Andy and Ndousse, Kamal and Askell, Amanda and Chen, Anna and DasSarma, Nova and Drain, Dawn and Fort, Stanislav and Ganguli, Deep and Henighan, Tom and others , journal =. Training a helpful and harmless assistant with reinforcement learning from human ...
[115]

doi:10.18653/v1/2022.acl-long.229 , pages =

Lin, Stephanie and Hilton, Jacob and Evans, Owain , booktitle =. doi:10.18653/v1/2022.acl-long.229 , pages =

2022 doi
[116]

Ethical-Advice Taker: Do Language Models Understand Natural Language Interventions? , url =

Zhao, Jieyu and Khashabi, Daniel and Khot, Tushar and Sabharwal, Ashish and Chang, Kai-Wei , booktitle =. Ethical-Advice Taker: Do Language Models Understand Natural Language Interventions? , url =. doi:10.18653/v1/2021.findings-acl.364 , pages =

2021 doi
[117]

ArXiv preprint , title =

Romal Thoppilan and Daniel De Freitas and Jamie Hall and Noam Shazeer and Apoorv Kulshreshtha and Heng. ArXiv preprint , title =
[118]

, booktitle =

Gehman, Samuel and Gururangan, Suchin and Sap, Maarten and Choi, Yejin and Smith, Noah A. , booktitle =. doi:10.18653/v1/2020.findings-emnlp.301 , pages =

2020 doi
[119]

and Gebru, Timnit and McMillan-Major, Angelina and Shmitchell, Shmargaret , booktitle =

Bender, Emily M. and Gebru, Timnit and McMillan-Major, Angelina and Shmitchell, Shmargaret , booktitle =. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? , year =
[120]

Theory-Grounded Measurement of

Cao, Yang and Sotnikova, Anna and Daum. Theory-Grounded Measurement of. Proc. of NAACL-HLT , doi =
[121]

Detoxifying Language Models Risks Marginalizing Minority Voices , url =

Xu, Albert and Pathak, Eshaan and Wallace, Eric and Gururangan, Suchin and Sap, Maarten and Klein, Dan , booktitle =. Detoxifying Language Models Risks Marginalizing Minority Voices , url =. doi:10.18653/v1/2021.naacl-main.190 , pages =

2021 doi
[122]

Process for adapting language models to society (palms) with values-targeted datasets , volume =

Solaiman, Irene and Dennison, Christy , journal =. Process for adapting language models to society (palms) with values-targeted datasets , volume =
[123]

On Faithfulness and Factuality in Abstractive Summarization , url =

Maynez, Joshua and Narayan, Shashi and Bohnet, Bernd and McDonald, Ryan , booktitle =. On Faithfulness and Factuality in Abstractive Summarization , url =. doi:10.18653/v1/2020.acl-main.173 , pages =

2020 doi
[124]

A Deep Reinforced Model for Abstractive Summarization , url =

Romain Paulus and Caiming Xiong and Richard Socher , bibsource =. A Deep Reinforced Model for Abstractive Summarization , url =. Proc. of ICLR , publisher =
[125]

Sequence Level Training with Recurrent Neural Networks , url =

Marc'Aurelio Ranzato and Sumit Chopra and Michael Auli and Wojciech Zaremba , bibsource =. Sequence Level Training with Recurrent Neural Networks , url =. Proc. of ICLR , editor =
[126]

Tom B. Brown and Benjamin Mann and Nick Ryder and Melanie Subbiah and Jared Kaplan and Prafulla Dhariwal and Arvind Neelakantan and Pranav Shyam and Girish Sastry and Amanda Askell and Sandhini Agarwal and Ariel Herbert. Language Models are Few-Shot Learners , url =. Proc. of ...
[127]

Brown and Adam Santoro and Aditya Gupta and Adri

Aarohi Srivastava and Abhinav Rastogi and Abhishek Rao and Abu Awal Md Shoeb and Abubakar Abid and Adam Fisch and Adam R. Brown and Adam Santoro and Aditya Gupta and Adri. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models , url =. CoR...

doi arXiv
[128]

Bot-Adversarial Dialogue for Safe Conversational Agents , url =

Xu, Jing and Ju, Da and Li, Margaret and Boureau, Y-Lan and Weston, Jason and Dinan, Emily , booktitle =. Bot-Adversarial Dialogue for Safe Conversational Agents , url =. doi:10.18653/v1/2021.naacl-main.235 , pages =

2021 doi
[129]

Optimal control as a graphical model inference problem , volume =

Kappen, Hilbert J and G. Optimal control as a graphical model inference problem , volume =. Machine learning , number =
[130]

Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control , url =

Natasha Jaques and Shixiang Gu and Dzmitry Bahdanau and Jos. Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control , url =. Proc. of ICML , editor =
[131]

ArXiv preprint , title =

Natasha Jaques and Asma Ghandeharioun and Judy Hanwen Shen and Craig Ferguson and. ArXiv preprint , title =
[132]

The Thirty-Third

Ning Miao and Hao Zhou and Lili Mou and Rui Yan and Lei Li , bibsource =. The Thirty-Third. doi:10.1609/aaai.v33i01.33016834 , pages =

doi
[133]

Exposing the Implicit Energy Networks behind Masked Language Models via Metropolis--Hastings , url =

Kartik Goyal and Chris Dyer and Taylor Berg. Exposing the Implicit Energy Networks behind Masked Language Models via Metropolis--Hastings , url =. Proc. of ICLR , publisher =
[134]

arXiv preprint arXiv:2403.07183 , year=

Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews , author=. arXiv preprint arXiv:2403.07183 , year=

arXiv
[135]

arXiv preprint arXiv:2312.11805 , year=

Gemini: a family of highly capable multimodal models , author=. arXiv preprint arXiv:2312.11805 , year=

arXiv
[136]

arXiv preprint arXiv:2303.12712 , year=

Sparks of artificial general intelligence: Early experiments with gpt-4 , author=. arXiv preprint arXiv:2303.12712 , year=

arXiv
[137]

arXiv preprint arXiv:1710.09412 , year=

mixup: Beyond empirical risk minimization , author=. arXiv preprint arXiv:1710.09412 , year=

arXiv
[138]

Advances in Neural Information Processing Systems , volume=

Calibrating deep neural networks using focal loss , author=. Advances in Neural Information Processing Systems , volume=
[139]

Advances in Neural Information Processing Systems , volume=

Verified uncertainty calibration , author=. Advances in Neural Information Processing Systems , volume=
[140]

Advances in Neural Information Processing Systems , volume=

Robust calibration with multi-domain temperature scaling , author=. Advances in Neural Information Processing Systems , volume=
[141]

Proceedings of the AAAI Conference on Artificial Intelligence , volume=

Sample-dependent adaptive temperature scaling for improved calibration , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=
[142]

Advances in neural information processing systems , volume=

Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration , author=. Advances in neural information processing systems , volume=
[143]

Journal of Machine Learning Research , volume=

Palm: Scaling language modeling with pathways , author=. Journal of Machine Learning Research , volume=
[144]

Journal of the American Statistical Association , volume=

On the algorithmic bias of aligning large language models with rlhf: Preference collapse and matching regularization , author=. Journal of the American Statistical Association , volume=. 2025 , publisher=

2025
[145]

International Conference on Machine Learning , pages=

Whose opinions do language models reflect? , author=. International Conference on Machine Learning , pages=. 2023 , organization=

2023
[146]

arXiv preprint arXiv:2303.08774 , year=

GPT-4 Technical Report , author=. arXiv preprint arXiv:2303.08774 , year=

arXiv
[147]

arXiv preprint arXiv:2307.15217 , year=

Open problems and fundamental limitations of reinforcement learning from human feedback , author=. arXiv preprint arXiv:2307.15217 , year=

arXiv
[148]

arXiv preprint arXiv:2404.01730 , year=

Asymptotics of Language Model Alignment , author=. arXiv preprint arXiv:2404.01730 , year=

arXiv
[149]

Advances in neural information processing systems , volume=

Language models are few-shot learners , author=. Advances in neural information processing systems , volume=
[150]

Proceedings of NAACL-HLT , pages=

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , author=. Proceedings of NAACL-HLT , pages=
[151]

Syntactic wordclass tagging , pages=

Tokenization , author=. Syntactic wordclass tagging , pages=. 1999 , publisher=

1999
[152]

arXiv preprint arXiv:2404.12358 , year=

From r to Q^* : Your Language Model is Secretly a Q-Function , author=. arXiv preprint arXiv:2404.12358 , year=

arXiv
[153]

arXiv preprint arXiv:2404.08495 , year=

Dataset reset policy optimization for rlhf , author=. arXiv preprint arXiv:2404.08495 , year=

arXiv
[154]

arXiv preprint arXiv:2404.09656 , year=

Learn Your Reference Model for Real Good Alignment , author=. arXiv preprint arXiv:2404.09656 , year=

arXiv
[155]

Proceedings of the 41st International Conference on Machine Learning , pages =

Munos, Remi and Valko, Michal and Calandriello, Daniele and Gheshlaghi Azar, Mohammad and Rowland, Mark and Guo, Zhaohan Daniel and Tang, Yunhao and Geist, Matthieu and Mesnard, Thomas and Fiegel, C\^. Proceedings of the 41st International Conference on Machine Learning , page...

2024
[156]

International conference on machine learning , pages=

Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor , author=. International conference on machine learning , pages=. 2018 , organization=

2018
[157]

ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models , year=

Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint , author=. ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models , year=

2024
[158]

The Journal of the Acoustical Society of America , volume=

Perplexity—a measure of the difficulty of speech recognition tasks , author=. The Journal of the Acoustical Society of America , volume=. 1977 , publisher=

1977
[159]

arXiv preprint arXiv:2402.07319 , year=

ODIN: Disentangled Reward Mitigates Hacking in RLHF , author=. arXiv preprint arXiv:2402.07319 , year=

arXiv
[160]

arXiv preprint arXiv:2404.18922 , year=

DPO Meets PPO: Reinforced Token Optimization for RLHF , author=. arXiv preprint arXiv:2404.18922 , year=

arXiv
[161]

arXiv preprint arXiv:2402.07314 , year=

A theoretical analysis of nash learning from human feedback under general kl-regularized preference , author=. arXiv preprint arXiv:2402.07314 , year=

arXiv
[162]

arXiv preprint arXiv:2402.18571 , year=

Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards , author=. arXiv preprint arXiv:2402.18571 , year=

arXiv
[163]

arXiv e-prints , pages=

RLHF Workflow: From Reward Modeling to Online RLHF , author=. arXiv e-prints , pages=
[164]

arXiv preprint arXiv:2309.06657 , year=

Statistical rejection sampling improves preference optimization , author=. arXiv preprint arXiv:2309.06657 , year=

arXiv
[165]

arXiv preprint arXiv:2303.10130 , year=

Gpts are gpts: An early look at the labor market impact potential of large language models , author=. arXiv preprint arXiv:2303.10130 , year=

arXiv
[166]

2012 , publisher=

Social choice and individual values , author=. 2012 , publisher=

2012
[167]

arXiv preprint arXiv:2310.10826 , year=

Mechanism design for large language models , author=. arXiv preprint arXiv:2310.10826 , year=

arXiv
[168]

Proceedings of the fourth Berkeley symposium on mathematical statistics and probability, volume 1: contributions to the theory of statistics , volume=

On measures of entropy and information , author=. Proceedings of the fourth Berkeley symposium on mathematical statistics and probability, volume 1: contributions to the theory of statistics , volume=. 1961 , organization=

1961
[169]

arXiv preprint arXiv:2506.12350 , year=

Theoretical tensions in rlhf: Reconciling empirical success with inconsistencies in social choice theory , author=. arXiv preprint arXiv:2506.12350 , year=

arXiv
[170]

arXiv preprint arXiv:2405.08448 , year=

Understanding the performance gap between online and offline alignment algorithms , author=. arXiv preprint arXiv:2405.08448 , year=

arXiv
[171]

, author=

Maximum entropy inverse reinforcement learning. , author=. Aaai , volume=. 2008 , organization=

2008
[172]

2005 , publisher=

Individual choice behavior: A theoretical analysis , author=. 2005 , publisher=

2005
[173]

Findings of the Association for Computational Linguistics: EMNLP 2022 , pages=

Truncation Sampling as Language Model Desmoothing , author=. Findings of the Association for Computational Linguistics: EMNLP 2022 , pages=

2022
[174]

arXiv preprint arXiv:2302.08215 , year=

Aligning language models with preferences through f-divergence minimization , author=. arXiv preprint arXiv:2302.08215 , year=

arXiv
[175]

Proxy objectives in reinforcement learning from human feedback , organization =

Schulman, John , url =. Proxy objectives in reinforcement learning from human feedback , organization =
[176]

arXiv preprint arXiv:2308.01320 , year=

DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales , author=. arXiv preprint arXiv:2308.01320 , year=

arXiv
[177]

arXiv preprint arXiv:2403.19159 , year=

Disentangling length from quality in direct preference optimization , author=. arXiv preprint arXiv:2403.19159 , year=

arXiv
[178]

arXiv preprint arXiv:2403.05006 , year=

Provable Multi-Party Reinforcement Learning with Diverse Human Feedback , author=. arXiv preprint arXiv:2403.05006 , year=

arXiv
[179]

arXiv preprint arXiv:2404.10719 , year=

Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study , author=. arXiv preprint arXiv:2404.10719 , year=

arXiv
[180]

arXiv preprint arXiv:2404.14367 , year=

Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data , author=. arXiv preprint arXiv:2404.14367 , year=

arXiv
[181]

International Conference on Artificial Intelligence and Statistics , pages=

A general theoretical paradigm to understand learning from human preferences , author=. International Conference on Artificial Intelligence and Statistics , pages=. 2024 , organization=

2024
[182]

arXiv preprint arXiv:2205.01068 , year=

Opt: Open pre-trained transformer language models , author=. arXiv preprint arXiv:2205.01068 , year=

arXiv
[183]

arXiv preprint arXiv:2402.08925 , year=

MaxMin-RLHF: Towards Equitable Alignment of Large Language Models with Diverse Human Preferences , author=. arXiv preprint arXiv:2402.08925 , year=

arXiv
[184]

The Twelfth International Conference on Learning Representations , year=

Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints , author=. The Twelfth International Conference on Learning Representations , year=
[185]

arXiv preprint arXiv:2305.17608 , year=

Reward Collapse in Aligning Large Language Models , author=. arXiv preprint arXiv:2305.17608 , year=
[186]

International Conference on Learning Representations , year=

Maximum Entropy RL (Provably) Solves Some Robust RL Problems , author=. International Conference on Learning Representations , year=
[187]

International Conference on Knowledge Discovery and Information Retrieval , volume=

Item-user preference mapping with mixture models-data visualization for item preference , author=. International Conference on Knowledge Discovery and Information Retrieval , volume=. 2009 , organization=

2009
[188]

arXiv preprint arXiv:1909.08593 , year=

Fine-tuning language models from human preferences , author=. arXiv preprint arXiv:1909.08593 , year=

1909 arXiv
[189]

arXiv preprint arXiv:2310.11564 , year=

Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging , author=. arXiv preprint arXiv:2310.11564 , year=

arXiv
[190]

arXiv preprint arXiv:1702.01806 , year=

Beam search strategies for neural machine translation , author=. arXiv preprint arXiv:1702.01806 , year=

arXiv
[191]

arXiv preprint arXiv:1904.09751 , year=

The curious case of neural text degeneration , author=. arXiv preprint arXiv:1904.09751 , year=

1904 arXiv
[192]

2018 , eprint=

Hierarchical Neural Story Generation , author=. 2018 , eprint=

2018
[193]

2022 , eprint=

Learning to summarize from human feedback , author=. 2022 , eprint=

2022
[194]

The method of paired comparisons , author=

Rank analysis of incomplete block designs: I. The method of paired comparisons , author=. Biometrika , volume=. 1952 , publisher=

1952
[195]

Journal of the Royal Statistical Society Series C: Applied Statistics , volume=

The analysis of permutations , author=. Journal of the Royal Statistical Society Series C: Applied Statistics , volume=. 1975 , publisher=

1975
[196]

2012 , publisher=

Individual choice behavior: A theoretical analysis , author=. 2012 , publisher=

2012
[197]

arXiv preprint arXiv:2306.04751 , year=

How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources , author=. arXiv preprint arXiv:2306.04751 , year=

arXiv
[198]

arXiv preprint arXiv:2304.03277 , year=

Instruction tuning with gpt-4 , author=. arXiv preprint arXiv:2304.03277 , year=

arXiv
[199]

2020 , publisher=

The alignment problem: Machine learning and human values , author=. 2020 , publisher=

2020
[200]

Collective Choice and Social Welfare , title =

Amartya Sen , publisher =. Collective Choice and Social Welfare , title =. 2017 , lastchecked =. doi:doi:10.4159/9780674974616 , isbn =

2017 doi
[201]

arXiv preprint arXiv:2307.12966 , year=

Aligning large language models with human: A survey , author=. arXiv preprint arXiv:2307.12966 , year=

arXiv
[202]

2022 , eprint=

Training language models to follow instructions with human feedback , author=. 2022 , eprint=

2022
[203]

2023 , eprint=

DICES Dataset: Diversity in Conversational AI Evaluation for Safety , author=. 2023 , eprint=

2023
[204]

2023 , eprint=

The Reasonable Effectiveness of Diverse Evaluation Data , author=. 2023 , eprint=

2023
[205]

2021 , eprint=

Whose Ground Truth? Accounting for Individual and Collective Identities Underlying Dataset Annotation , author=. 2021 , eprint=

2021
[206]

AI Magazine , author=

Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation , volume=. AI Magazine , author=. 2015 , month=. doi:10.1609/aimag.v36i1.2564 , abstractNote=

2015 doi
[207]

Why Don ' t You Do It Right? Analysing Annotators ' Disagreement in Subjective Tasks

Sandri, Marta and Leonardelli, Elisa and Tonelli, Sara and Jezek, Elisabetta. Why Don ' t You Do It Right? Analysing Annotators ' Disagreement in Subjective Tasks. Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. 2023...

2023 doi
[208]

Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection

Sap, Maarten and Swayamdipta, Swabha and Vianna, Laura and Zhou, Xuhui and Choi, Yejin and Smith, Noah A. Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection. Proceedings of the 2022 Conference of the North American Chapter of the Asso...

2022 doi
[209]

Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators’ Disagreement , url=

Leonardelli, Elisa and Menini, Stefano and Palmero Aprosio, Alessio and Guerini, Marco and Tonelli, Sara , year=. Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators’ Disagreement , url=. doi:10.18653/v1/2021.emnlp-main.822 , booktitle=

doi
[210]

Identifying and Measuring Annotator Bias Based on Annotators ' Demographic Characteristics

Al Kuwatly, Hala and Wich, Maximilian and Groh, Georg. Identifying and Measuring Annotator Bias Based on Annotators ' Demographic Characteristics. Proceedings of the Fourth Workshop on Online Abuse and Harms. 2020. doi:10.18653/v1/2020.alw-1.21

2020 doi
[211]

2023 , eprint=

Principled Reinforcement Learning with Human Feedback from Pairwise or K -wise Comparisons , author=. 2023 , eprint=

2023
[212]

Blog post, April , volume=

Koala: A dialogue model for academic research , author=. Blog post, April , volume=
[213]

arXiv preprint arXiv:2305.14387 , year=

Alpacafarm: A simulation framework for methods that learn from human feedback , author=. arXiv preprint arXiv:2305.14387 , year=

arXiv
[214]

arXiv preprint arXiv:2311.10702 , year=

Camels in a changing climate: Enhancing lm adaptation with tulu 2 , author=. arXiv preprint arXiv:2311.10702 , year=

arXiv
[215]

2020 , eprint=

Fine-Tuning Language Models from Human Preferences , author=. 2020 , eprint=

2020
[216]

2023 , eprint=

RRHF: Rank Responses to Align Language Models with Human Feedback without tears , author=. 2023 , eprint=

2023
[217]

2023 , eprint=

Aligning Language Models with Preferences through f-divergence Minimization , author=. 2023 , eprint=

2023
[218]

2023 , eprint=

A Survey of Reinforcement Learning from Human Feedback , author=. 2023 , eprint=

2023
[219]

2023 , eprint=

Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization , author=. 2023 , eprint=

2023
[220]

2023 , eprint=

Safe RLHF: Safe Reinforcement Learning from Human Feedback , author=. 2023 , eprint=

2023
[221]

Ethics and Information Technology , year=

Human-aligned artificial intelligence is a multiobjective problem , author=. Ethics and Information Technology , year=
[222]

Australasian Conference on Artificial Intelligence , year=

On the Limitations of Scalarisation for Multi-objective Reinforcement Learning of Pareto Fronts , author=. Australasian Conference on Artificial Intelligence , year=
[223]

2022 , eprint=

Fine-tuning language models to find agreement among humans with diverse preferences , author=. 2022 , eprint=

2022
[224]

2023 , eprint=

'Generative CI' through Collective Response Systems , author=. 2023 , eprint=

2023
[225]

2022 , eprint=

Constitutional AI: Harmlessness from AI Feedback , author=. 2022 , eprint=

2022
[226]

2023 , eprint=

Large Language Models as Superpositions of Cultural Perspectives , author=. 2023 , eprint=

2023
[227]

2023 , eprint=

Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards , author=. 2023 , eprint=

2023
[228]

2023 , eprint=

BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset , author=. 2023 , eprint=

2023
[229]

arXiv preprint arXiv:2401.01335 , year=

Self-play fine-tuning converts weak language models to strong language models , author=. arXiv preprint arXiv:2401.01335 , year=

arXiv
[230]

2019 , url=

Language Models are Unsupervised Multitask Learners , author=. 2019 , url=

2019
[231]

and Daly, Raymond E

Maas, Andrew L. and Daly, Raymond E. and Pham, Peter T. and Huang, Dan and Ng, Andrew Y. and Potts, Christopher , title =. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies , month =. 2011 , address =

2011
[232]

arXiv preprint arXiv:2310.10556, AAAI , year=

Sample Complexity of Preference-Based Nonparametric Off-Policy Evaluation with Deep Networks , author=. arXiv preprint arXiv:2310.10556, AAAI , year=

arXiv
[233]

The Twelfth International Conference on Learning Representations (ICLR) , year=

PARL: A Unified Framework for Policy Alignment in Reinforcement Learning , author=. The Twelfth International Conference on Learning Representations (ICLR) , year=
[234]

arXiv preprint arXiv:2305.18438 , year=

Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimism , author=. arXiv preprint arXiv:2305.18438 , year=

arXiv
[235]

arXiv preprint arXiv:2307.12975 , year=

Provable Benefits of Policy Learning from Human Preferences in Contextual Bandit Problems , author=. arXiv preprint arXiv:2307.12975 , year=

arXiv
[236]

arXiv preprint arXiv:2306.07528 , year=

Unified Off-Policy Learning to Rank: a Reinforcement Learning Perspective , author=. arXiv preprint arXiv:2306.07528 , year=

arXiv
[237]

arXiv preprint arXiv:1811.00511 , year=

Towards Coherent and Cohesive Long-form Text Generation , author=. arXiv preprint arXiv:1811.00511 , year=

arXiv
[238]

Pew Research Center , volume=

The state of online harassment , author=. Pew Research Center , volume=
[239]

Robust Solutions of Optimization Problems Affected by Uncertain Probabilities , urldate =

Aharon Ben-Tal and Dick den Hertog and Anja De Waegenaere and Bertrand Melenberg and Gijs Rennen , journal =. Robust Solutions of Optimization Problems Affected by Uncertain Probabilities , urldate =
[240]

2018 , eprint=

Statistics of Robust Optimization: A Generalized Empirical Likelihood Approach , author=. 2018 , eprint=

2018
[241]

2020 , eprint=

Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization , author=. 2020 , eprint=

2020
[242]

2021 , eprint=

On the Convergence and Sample Efficiency of Variance-Reduced Policy Gradient Method , author=. 2021 , eprint=

2021
[243]

2020 , eprint=

Variational Policy Gradient Method for Reinforcement Learning with General Utilities , author=. 2020 , eprint=

2020
[244]

Lindsay , journal =

Bruce G. Lindsay , journal =. Mixture Models: Theory, Geometry and Applications , urldate =
[245]

Mixture Models

Seidel, Wilfried. Mixture Models. International Encyclopedia of Statistical Science. 2011. doi:10.1007/978-3-642-04898-2_368

2011 doi
[246]

2016 , isbn =

Goodfellow, Ian and Bengio, Yoshua and Courville, Aaron , title =. 2016 , isbn =

2016
[247]

Bertsekas , title =

Dimitri P. Bertsekas , title =. 2019 , address =

2019
[248]

Sutton and Andrew G

Richard S. Sutton and Andrew G. Barto , title =. 2018 , edition =

2018
[249]

2024 , volume =

Li, Ziniu and Xu, Tian and Zhang, Yushun and Lin, Zhihang and Yu, Yang and Sun, Ruoyu and Luo, Zhi-Quan , booktitle =. 2024 , volume =

2024
[250]

arXiv preprint arXiv:2310.06452 , year=

Understanding the effects of rlhf on llm generalisation and diversity , author=. arXiv preprint arXiv:2310.06452 , year=

arXiv
[251]

arXiv preprint arXiv:2312.10584 , year=

Policy Optimization in RLHF: The Impact of Out-of-preference Data , author=. arXiv preprint arXiv:2312.10584 , year=

arXiv
[252]

Gomez and Lukasz Kaiser and Illia Polosukhin , title =

Ashish Vaswani and Noam Shazeer and Niki Parmar and Jakob Uszkoreit and Llion Jones and Aidan N. Gomez and Lukasz Kaiser and Illia Polosukhin , title =. Advances in Neural Information Processing Systems 30 , pages =
[253]

Claude-3 Model Card , year=

The claude 3 model family: Opus, sonnet, haiku , author =. Claude-3 Model Card , year=
[254]

Transactions of the Association for Computational Linguistics , volume=

How can we know when language models know? on the calibration of language models for question answering , author=. Transactions of the Association for Computational Linguistics , volume=
[255]

Findings of the Association for Computational Linguistics: EMNLP 2022 , pages=

Uncertainty quantification with pre-trained language models: A large-scale empirical analysis , author=. Findings of the Association for Computational Linguistics: EMNLP 2022 , pages=

2022
[256]

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

A close look into the calibration of pre-trained language models , author=. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=
[257]

Transactions on Machine Learning Research , year=

Teaching Models to Express Their Uncertainty in Words , author=. Transactions on Machine Learning Research , year=
[258]

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , pages=

Knowing more about questions can help: Improving calibration in question answering , author=. Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , pages=

2021
[259]

arXiv preprint arXiv:2207.05221 , year=

Language models (mostly) know what they know , author=. arXiv preprint arXiv:2207.05221 , year=

arXiv
[260]

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback

Tian, Katherine and Mitchell, Eric and Zhou, Allan and Sharma, Archit and Rafailov, Rafael and Yao, Huaxiu and Finn, Chelsea and Manning, Christopher. Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Fee...

2023
[261]

arXiv preprint arXiv:2306.13063 , year=

Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms , author=. arXiv preprint arXiv:2306.13063 , year=

arXiv
[262]

The Eleventh International Conference on Learning Representations , year=

Prototypical Calibration for Few-shot Learning of Language Models , author=. The Eleventh International Conference on Learning Representations , year=
[263]

Journal of the American Medical Informatics Association , volume=

Large language model uncertainty proxies: discrimination and calibration for medical diagnosis and treatment , author=. Journal of the American Medical Informatics Association , volume=. 2025 , publisher=

2025
[264]

Proceedings of The 27th International Conference on Artificial Intelligence and Statistics , pages =

Enhancing In-context Learning via Linear Probe Calibration , author =. Proceedings of The 27th International Conference on Artificial Intelligence and Statistics , pages =
[265]

Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=

A Survey of Confidence Estimation and Calibration in Large Language Models , author=. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=

2024
[266]

International Conference on Machine Learning , pages=

Thermometer: Towards Universal Calibration for Large Language Models , author=. International Conference on Machine Learning , pages=. 2024 , organization=

2024
[267]

Findings of the Association for Computational Linguistics: EMNLP 2023 , pages=

Generative calibration for in-context learning , author=. Findings of the Association for Computational Linguistics: EMNLP 2023 , pages=

2023
[268]

The Thirteenth International Conference on Learning Representations , year=

Magnetic Preference Optimization: Achieving Last-iterate Convergence for Language Model Alignment , author=. The Thirteenth International Conference on Learning Representations , year=
[269]

arXiv preprint arXiv:2503.10990 , year=

Statistical impossibility and possibility of aligning llms with human preferences: From condorcet paradox to nash equilibrium , author=. arXiv preprint arXiv:2503.10990 , year=

arXiv
[270]

The Thirteenth International Conference on Learning Representations , year=

Preserving Diversity in Supervised Fine-Tuning of Large Language Models , author=. The Thirteenth International Conference on Learning Representations , year=
[271]

The Thirteenth International Conference on Learning Representations , year=

Fine-Tuning Attention Modules Only: Enhancing Weight Disentanglement in Task Arithmetic , author=. The Thirteenth International Conference on Learning Representations , year=
[272]

International conference on machine learning , pages=

Calibrate before use: Improving few-shot performance of language models , author=. International conference on machine learning , pages=. 2021 , organization=

2021
[273]

Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

Sayself: Teaching llms to express confidence with self-reflective rationales , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

2024
[274]

Advances in neural information processing systems , volume=

When does label smoothing help? , author=. Advances in neural information processing systems , volume=
[275]

The Annals of statistics , pages=

On the convergence properties of the EM algorithm , author=. The Annals of statistics , pages=. 1983 , publisher=

1983
[276]

arXiv preprint arXiv:2404.10315 , year=

Enhancing Confidence Expression in Large Language Models Through Learning from Past Experience , author=. arXiv preprint arXiv:2404.10315 , year=

arXiv
[277]

Findings of the Association for Computational Linguistics ACL 2024 , pages=

When to Trust LLMs: Aligning Confidence with Response Quality , author=. Findings of the Association for Computational Linguistics ACL 2024 , pages=

2024
[278]

arXiv preprint arXiv:2410.09724 , year=

Taming overconfidence in llms: Reward calibration in rlhf , author=. arXiv preprint arXiv:2410.09724 , year=

arXiv
[279]

International Conference on Machine Learning , pages=

Trainable calibration measures for neural networks from kernel mean embeddings , author=. International Conference on Machine Learning , pages=. 2018 , organization=

2018
[280]

Advances in Neural Information Processing Systems , volume=

Improving model calibration with accuracy versus uncertainty optimization , author=. Advances in Neural Information Processing Systems , volume=
[281]

Advances in Neural Information Processing Systems , volume=

Soft calibration objectives for neural networks , author=. Advances in Neural Information Processing Systems , volume=
[282]

Advances in Neural Information Processing Systems , volume=

A consistent and differentiable lp canonical calibration error estimator , author=. Advances in Neural Information Processing Systems , volume=
[283]

The Thirty-eighth Annual Conference on Neural Information Processing Systems , year=

Information-theoretic Generalization Analysis for Expected Calibration Error , author=. The Thirty-eighth Annual Conference on Neural Information Processing Systems , year=
[284]

arXiv preprint arXiv:2406.06227 , year=

PAC-Bayes Analysis for Recalibration in Classification , author=. arXiv preprint arXiv:2406.06227 , year=

arXiv
[285]

International conference on machine learning , pages=

Distribution-free calibration guarantees for histogram binning without sample splitting , author=. International conference on machine learning , pages=. 2021 , organization=

2021
[286]

Advances in Neural Information Processing Systems , volume=

Distribution-free binary classification: prediction sets, confidence intervals and calibration , author=. Advances in Neural Information Processing Systems , volume=
[287]

Advances in Neural Information Processing Systems , volume=

Better uncertainty calibration via proper scores for classification and beyond , author=. Advances in Neural Information Processing Systems , volume=
[288]

Advances in Neural Information Processing Systems , volume=

Minimum-risk recalibration of classifiers , author=. Advances in Neural Information Processing Systems , volume=
[289]

Advances in Neural Information Processing Systems , volume=

Pac-bayesian spectrally-normalized bounds for adversarially robust generalization , author=. Advances in Neural Information Processing Systems , volume=
[290]

The Thirty Seventh Annual Conference on Learning Theory , pages=

Bridging the gap: Rademacher complexity in robust and standard generalization , author=. The Thirty Seventh Annual Conference on Learning Theory , pages=. 2024 , organization=

2024
[291]

International Conference on Machine Learning , pages=

Uniformly Stable Algorithms for Adversarial Training and Beyond , author=. International Conference on Machine Learning , pages=. 2024 , organization=

2024
[292]

arXiv preprint arXiv:2505.20627 , year=

Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matching , author=. arXiv preprint arXiv:2505.20627 , year=

arXiv
[293]

International Conference on Machine Learning , pages=

Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach , author=. International Conference on Machine Learning , pages=. 2025 , organization=

2025
[294]

Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

Towards Objective Fine-tuning: How LLMs’ Prior Knowledge Causes Potential Poor Calibration? , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=
[295]

International Conference on Machine Learning , pages=

Calibrated Language Models and How to Find Them with Label Smoothing , author=. International Conference on Machine Learning , pages=. 2025 , organization=

2025
[296]

Advances in neural information processing systems , volume=

Bome! bilevel optimization made easy: A simple first-order approach , author=. Advances in neural information processing systems , volume=
[297]

Hashimoto , title =

Rohan Taori and Ishaan Gulrajani and Tianyi Zhang and Yann Dubois and Xuechen Li and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto , title =. GitHub repository , howpublished =. 2023 , publisher =

2023
[298]

arXiv preprint arXiv:2310.11732 , year=

Investigating uncertainty calibration of aligned language models under the multiple-choice setting , author=. arXiv preprint arXiv:2310.11732 , year=

arXiv
[299]

Taming Overconfidence in

Jixuan Leng and Chengsong Huang and Banghua Zhu and Jiaxin Huang , booktitle=. Taming Overconfidence in. 2025 , url=

2025
[300]

2024 , url=

Xin Liu and Muhammad Khalifa and Lu Wang , booktitle=. 2024 , url=

2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Proceedings of the International Conference on Learning Representations (ICLR) , year=

Measuring Massive Multitask Language Understanding , author=. Proceedings of the International Conference on Learning Representations (ICLR) , year=

[2] [2]

Proceedings of the Conference on Health, Inference, and Learning (CHIL) , year=

MedMCQA: A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering , author=. Proceedings of the Conference on Health, Inference, and Learning (CHIL) , year=

[3] [3]

Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) , year=

Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering , author=. Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) , year=

[4] [4]

Advances in Neural Information Processing Systems , volume=

Stochastic Regret Guarantees for Online Zeroth-and First-Order Bilevel Optimization , author=. Advances in Neural Information Processing Systems , volume=

[5] [5]

International Conference on Artificial Intelligence and Statistics , pages=

Online bilevel optimization: Regret analysis of online alternating gradient methods , author=. International Conference on Artificial Intelligence and Statistics , pages=. 2024 , organization=

2024

[6] [6]

See https://vicuna

Vicuna: An open-source chatbot impressing gpt-4 with 90\ author=. See https://vicuna. lmsys. org , year=

[7] [7]

arXiv preprint arXiv:2307.09288 , year=

Llama 2: Open foundation and fine-tuned chat models , author=. arXiv preprint arXiv:2307.09288 , year=

arXiv

[8] [8]

arXiv preprint arXiv:2310.06825 , year=

Mistral 7B , author=. arXiv preprint arXiv:2310.06825 , year=

arXiv