Plug-and-Play Training Framework for Preference Optimization

Jingyuan Ma; Lei Sha; Rui Li; Zheng Li; Zhifang Sui

REVIEW 4 major objections 4 minor 1 cited by

Plug-and-Play Training Framework for Preference Optimization

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A plug-and-play weighting scheme uses repeated sampling to focus preference optimization on hard math problems, lifting accuracy on GSM8K and MATH500 across DPO, SimPO, DPOP, and IPO.

desk verdict Reasonable reweighting idea, but the paper's own tables don't support 'consistent improvements' and the evaluation confounds weight with data construction. read the letter →

arxiv 2412.20996 v1 pith:JVODK4BI submitted 2024-12-30 cs.CL

Jingyuan Ma , Rui Li , Zheng Li , Lei Sha , Zhifang Sui This is my paper · ORCID

classification cs.CL

keywords preferenceoptimizationmathematicalreasoningsampleweightingrepeatedsamplingDPOoutputdistributiontrainingframeworkLLMalignment

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that standard preference optimization methods like DPO waste training signal by treating every preference pair equally, which is especially damaging in mathematical reasoning where some problems are already mastered and others are systematically misunderstood. It proposes a plug-and-play weighting framework: sample each training question multiple times, count how often the model produces a correct answer, and derive a per-question weight that upweights problems the model repeatedly fails. Those weights are multiplied into the reward gap of any pairwise preference loss, and the paper reports that this improves GSM8K and MATH500 accuracy across DPO, SimPO, DPOP, and IPO with several model families. If the claim holds, any pairwise preference optimization method can be upgraded with little extra compute to focus training on the examples that matter most.

What carries the argument

The load-bearing object is the weight $w$ from Equation (2): $w = 1 + \alpha P_e/N$ when $P_c = 0$, and $w = \max(1,\, 1 + \alpha P_e/((P_c+\epsilon)N))$ when $P_c > 0$, where $P_c$ and $P_e$ are the numbers of correct and incorrect answers among $N$ samples, and $\alpha$ is a hyperparameter that controls the adjustment magnitude. This scalar is a direct estimate of per-question difficulty from the model's own output distribution, and it is inserted as a multiplicative factor on the reward gap in the pairwise preference loss. It carries the argument by turning an observed sampling statistic into a training signal that shifts optimization effort toward questions the model has not mastered.

What would settle it

Shuffle the computed weights across training questions while keeping their distribution fixed, retrain with the same preference optimizer, and compare GSM8K and MATH500 accuracy against the proposed weighting; if shuffled weights match the gains, the per-question difficulty signal carries no explanatory power.

Watch

Extended reading notes

Core claim

The paper's central claim is that a model's own repeated-sampling answer distribution provides a usable, model-specific difficulty measure, and that feeding this measure back as a scalar weight on each preference pair improves mathematical reasoning. For each question, the model samples N=16 responses at temperature 0.7; the count of correct and incorrect extractions (Pc and Pe) is turned into a weight w via Equation (2). This weight multiplies the reward difference in the preference objective, so pairs on which the model consistently fails get a larger gradient. The paper further shows that choosing the model's own correct response as the 'chosen' example, rather than the dataset's gold answer, helps training on mathematical tasks. The reported experiments show that this weighting generally outperforms the unweighted baselines across four preference optimization methods and several open-weight model families.

Load-bearing premise

The method assumes that a handful of sampled answers at one temperature reliably indicates how well the model has mastered each problem, so weighting by that count reflects true difficulty rather than sampling noise.

Editorial extensions

If this is right

Any pairwise preference optimization method (DPO, SimPO, DPOP, IPO) can incorporate the weights with negligible added compute: one extra sampling pass over the training set and a scalar multiply in the loss.
Training emphasis shifts to systematic mistakes: questions where the model repeats the same wrong answer receive the largest weights, so the gradient concentrates on undoing those misconceptions.
Using the model's own correct response as the chosen example keeps the optimized policy close to the original model's distribution, reducing the style-mimicry failure mode that gold-answer training can trigger in math.
The gains should be most visible on harder, accuracy-critical benchmarks like MATH500 and on models with intermediate error rates, because models that are too strong or too weak yield less informative sampling distributions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

One testable extension the paper does not run: shuffle the computed weights across questions while keeping their distribution fixed. If shuffled weights reproduce the gains, the per-question difficulty signal is not what matters; the improvement would come merely from uneven weighting.
The difficulty estimate is a snapshot of the model before training. As the model improves, the weights become stale; recomputing them mid-training or using an online estimator could yield larger or more persistent gains.
The framework's dependence on answer equivalence classes, acknowledged in the paper's limitation, means the same recipe could be applied to open-ended tasks by replacing exact numeric matching with semantic-equivalence clustering.
Because the chosen answer is sampled from the model itself, the method may compound existing biases or style quirks when the model is weak; a stronger model or a higher sampling temperature would reduce the risk of locking in errors.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a classic case of a plug-and-play trick that sounds good on paper but whose evaluation doesn't separate the two things it changes at once. The core idea—sample each question N times, count distinct numeric answers, weight each preference pair inversely to how often the model gets it right—is clean and cheap. Integrating a scalar weight into DPO/SimPO/DPOP/IPO is straightforward, and the paper does that clearly. The data distribution plots (Figures 3, 5, 6) give an honest look at what repeated sampling reveals, and the limitation section is a real attempt to bound the method's scope.

The soft spots are mostly about attribution and statistics. Table 1 shows 6 of 12 MATH500 cells and 5 of 12 GSM8K cells improve; several weighted rows are worse than their unweighted counterparts. The abstract's 'consistent improvements' is simply not supported by the paper's own numbers. More importantly, the framework bundles two changes: it builds preference pairs from the model's own samples (chosen = correct self-generated response, rejected = most common wrong answer, gold only if no correct sample) and then multiplies the loss by w. The baselines are just called 'DPO', 'SimPO', etc. If those baselines use different preference data (e.g., golden-answer pairs), any gain could come from the data construction, not from w. Figure 8 already shows a large boost from self-answer over golden-answer chosen responses, which makes this confound concrete. The paper never states the baselines use identical pairs with uniform weights, and without that control, the claimed mechanism is untested.

There are smaller issues: no error bars or multiple seeds, so the mixed differences (±0.5–1 point) are within plausible noise; Equation (2) has a redundant max(1,·) and, as written with N=16 and α=1, produces weights in a narrow [1,2] range, which suggests the rescaling is weak; and the model selection (ChatGLM3 for GSM, Qwen for MATH) is justified but leaves the cross-generalization question open.

Who is this for? Someone working on preference optimization for math reasoning might find the weighting formula worth a closer look. But as a referee, I'd want to see an identical-pairs baseline with uniform weights, a random-weights control, and a small seed analysis. The idea deserves that follow-up; the current paper doesn't support its headline. I'd accept it for peer review with major revision expected.

Referee Report

4 major / 4 minor

Summary. The paper proposes a plug-and-play weighting framework for preference optimization in mathematical reasoning. For each training question, the base model is sampled N times; the responses are grouped by extracted numeric answer, the numbers of correct and incorrect samples are counted, and a per-question weight w is computed by Eq. (2). Preference pairs are then constructed from the model's own samples (a self-generated correct answer as chosen, the most frequent wrong answer as rejected), and the pairwise preference loss of DPO, SimPO, DPOP, or IPO is multiplied by w. Experiments on GSM8K and MATH500 across Qwen2-1.5B/7B, GLM4-9B, and ChatGLM3-6B are reported in Tables 1-2, and the Abstract claims consistent improvements.

Significance. If the central claim were established, a simple scalar reweighting of pairwise preference losses that focuses training on questions the model has not mastered would be a useful and cheap addition to existing preference optimization methods. The framework is simple, applies to several losses, and the authors honestly discuss the answer-equivalence limitation and include a failure-case analysis for LLaMA3 in Appendix C. However, the current evidence does not support the claimed consistency: Table 1 contains many negative weighted-vs-unweighted differences, the design does not isolate the weight w from the self-sampled data-construction change, and no variance or significance information is provided. The value of the contribution will be clear only after a controlled comparison on identical preference pairs with error bars.

major comments (4)

[§4.5, Table 1] The Abstract's claim of "consistent improvements" is contradicted by the reported numbers. Across the 24 weighted-vs-unweighted comparisons in Table 1, only 13 show a positive difference and several are negative, e.g., Qwen2-1.5B GSM8K DPO 64.59 vs 64.74 (-0.15), Qwen2-1.5B GSM8K DPOP 64.36 vs 64.74 (-0.38), GLM4-9B GSM8K SimPO 57.92 vs 58.75 (-0.83), and GLM4-9B GSM8K IPO 58.68 vs 58.83 (-0.15). Table 2 adds another negative entry (IPO 54.28 vs 55.34, -1.06). The text should be revised to report this mixed pattern accurately, and the consistency claim should be either supported with an appropriate aggregate statistic or removed.
[§3.2 and §4.5] The experimental design does not isolate the effect of the weight w from the data-construction changes. The treatment changes two factors simultaneously: (i) preference pairs are built from self-sampled responses, with the chosen response being a self-generated correct answer and the rejected response being the most common wrong answer, and (ii) the pairwise loss is multiplied by w. The baselines in Tables 1 and 2 are described only as "DPO", "SimPO", "DPOP", and "IPO"; the paper never states whether these baselines use the identical self-constructed preference pairs with uniform weights or a different preference dataset. Because Figure 8 shows large accuracy gains from using self-generated answers over golden answers as the chosen response, the observed improvements in Tables 1-2 could be attributable entirely to the data construction rather than to Eq. (2). The authors should add an unweighted control trained on the same self-constructed pairs.
[§3.2, Eq. (2)] Equation (2) is not fully reproducible as written and its effect size is unclear. In the Pc>0 branch, the expression 1 + α·Pe/((Pc+ϵ)·N) is always greater than 1 for valid counts, making the max(1, ·) redundant; the placement of "· 1/N" is ambiguous and the first branch (1 + α·Pe/N) does not share the same scaling structure as the second branch. With N=16 and α=1, the weights lie approximately in [1, 1.94], which is a weak rescaling. The paper reports no sensitivity analysis for α, N, or sampling temperature, and no evidence that 16 samples at temperature 0.7 produce a stable estimate of per-question difficulty; if the counts are noisy, the weights add variance without a clear signal.
[§4.4, Tables 1 and 2] All experimental results are single-run and no error bars, multiple seeds, or significance tests are reported. Many weighted-vs-unweighted differences are within ±1 point on test sets of 500 (MATH500) or 1,319 (GSM8K) examples, so these differences cannot be distinguished from sampling noise without variance estimates. The authors should report multiple seeds or bootstrap confidence intervals, and a significance test for any aggregate improvement across models and methods.

minor comments (4)

[Throughout] There are multiple typos and grammatical errors, including "we assigns" (Section 1), "responds" in the Figure 3 and Figure 5 axis labels, "Bdfore" in Figure 5, "Comparision" in the Figure 9 caption, and "T ype" in Figure 6. The manuscript should be proofread.
[§3.3, Eqs. (3)-(4)] The derivation from Eq. (3) to Eq. (4) multiplies the logit difference by w, but it is not stated whether this is a heuristic modification or a consequence of a weighted Bradley-Terry likelihood. If the latter, the likelihood should be defined explicitly.
[§4.2] The sentence "we use ChatGLM3-6B for alignment with GSM data" is unclear; it should specify which training subset is used and that GSM8K is the evaluation set. The model-selection criterion in the same paragraph ("selecting models that maintain a certain error rate") should be made more precise.
[Appendix B] The GLM4-9B prompt discussion reports an inability to reproduce official accuracy but does not state the reproduced numbers or explain how the resulting baseline differences affect the comparison. Please provide the actual reproduced accuracies and clarify that the prompts are held fixed across methods.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the difficulty weights are an externally defined function of sampling counts, and the reported training objective is a standard weighted preference optimization.

full rationale

The derivation chain is self-contained and does not reduce to its inputs. The weight w in Equation (2) is defined directly from the base model's repeated sampling counts (Pc and Pe) on each question; it is not defined in terms of the test accuracy or the training loss being optimized. The objective in Equations (3)-(6) is the standard Bradley-Terry / DPO loss with w inserted as a scalar multiplier, so the prediction (that weighting affects optimization) is not equivalent to the construction of w. The chosen and rejected responses are selected by correctness and frequency, not by the weight formula, and no fitted parameter is renamed as a prediction: α, N, and temperature are stated as fixed settings rather than tuned to the test set. The paper contains no meaningful self-citations: the cited prior work (Brown et al., Snell et al., Rafailov et al., etc.) is external and is used only for standard sampling and preference-optimization machinery, not to justify the central claim. The limitation section and Appendix C candidly discuss boundary conditions and style mismatch, but they do not reveal a circular step. Some empirical results in Tables 1 and 2 are negative or mixed, which weakens the 'consistent improvements' claim, and the baselines may not have used the same self-constructed preference pairs, but those are attribution and experimental-design concerns, not circularity. Accordingly, no circular step meets the quoted reduction bar, and the score is 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the sampling-based difficulty proxy and the ad hoc weight formula. The main free parameters are alpha, N, epsilon, and temperature, none of which are ablated. The axioms are standard preference-modeling assumptions plus the paper-specific choices about answer equivalence and self-generated chosen answers.

free parameters (4)

alpha (weight magnitude) = 1
Equation (2): controls how much incorrect samples are upweighted; set to 1 without ablation or sensitivity analysis.
N (sampling count per question) = 16
Section 4.1: number of responses sampled per training question; chosen as a compromise, no ablation reported.
epsilon (stabilizer in Eq. 2) = unspecified small constant
Used in the denominator of Equation (2) for the Pc>0 branch; exact value is not reported.
sampling temperature = 0.7
Section 4.1: temperature for generating the N samples; affects the output distribution from which weights are derived, with no sensitivity analysis.

assumptions (5)

standard math Bradley-Terry preference model: P(yw > yl) follows a logistic function of reward differences.
Invoked in Section 3.3 to write the optimization objective of pairwise preference methods.
domain assumption Repeated sampling with N=16 at temperature 0.7 gives a reliable estimate of the model's per-question output distribution and mastery.
The entire weight computation in Equation (2) depends on counts of correct and incorrect answers from these samples; Sections 3.1 and 3.2.
domain assumption Numeric answer equivalence classes are sufficient to determine correctness for math questions.
The framework counts unique extracted answers and correctness per question; the Limitation section acknowledges this is nontrivial for other tasks.
domain assumption The model's own correct response is a better chosen answer than the golden answer.
Section 3.2 and Figure 8 assert that self-generated chosen answers minimize style mismatch and improve results; this is an empirical design assumption, not derived.
ad hoc to paper The weight formula in Equation (2) is a valid difficulty measure.
The formula is proposed without derivation; no theoretical or empirical justification beyond the intuition that more incorrect samples should mean higher weight.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Plug-and-Play Training Framework for Preference Optimization." pith.science (2026). https://pith.science/paper/JVODK4BI

@misc{pith2026241220996,
  author       = {Pith},
  title        = {Pith review of: Plug-and-Play Training Framework for Preference Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JVODK4BI}},
  note         = {Machine review of arXiv:2412.20996}
}

read the original abstract

Recently, preference optimization methods such as DPO have significantly enhanced large language models (LLMs) in wide tasks including dialogue and question-answering. However, current methods fail to account for the varying difficulty levels of training samples during preference optimization, leading to mediocre performance in tasks with high accuracy requirements, particularly in mathematical reasoning. To address this limitation, we propose a novel training framework, which employs multiple sampling to analyze output distributions, assign different weights to samples, and incorporate these weights into the preference optimization process. This plug-and-play approach enables LLMs to prioritize challenging examples during training, improving learning efficiency. Experimental results demonstrate that our framework integrates seamlessly with various preference optimization methods and achieves consistent improvements in mathematical reasoning tasks.

Figures

Figures reproduced from arXiv: 2412.20996 by the authors.

**Figure 2.** Overall process of the framework. In Step 1, we begin by sampling the model multiple times to collect the distribution of responses for each question. In step 2, we identify the responses with the highest number of incorrect answers as well as the correct answers. These are then weighted according to their frequency of occurrence. In step 3, various pairwise comparison alignment methods can be applied to incorporate… view at source ↗

**Figure 3.** The distribution of the responses of the model [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 5.** Figure 5: Data point distribution before and after train [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

**Figure 4.** Figure 4: Results of the model evaluated with multiple [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 6.** Figure 6: Training data distribution of different model. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

**Figure 7.** Figure 7: Reward between training with weight and without weight. Here we use DPO training on Qwen-2- 7B-Instruct as example. and the unweighted training method during the training process, we present the results of both methods in [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

**Figure 9.** Figure 9: Comparision of examples on Qwen2-7B-Instruct with and without weighted traning [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Progressive Mastery: Customized Curriculum Learning with Guided Prompting for Mathematical Reasoning
cs.CL 2025-06 conditional novelty 4.0 of 10

CCL orders LLM training data by the model's own measured accuracy and converts the hardest problems into hinted completion tasks, reporting higher average benchmark scores than uniform training.

Reference graph

Works this paper leans on

26 extracted references · 3 canonical work pages · cited by 1 Pith paper

[1]

Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, R \' e mi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. 2024. https://proceedings.mlr.press/v238/gheshlaghi-azar24a.html A general theoretical paradigm to understand learning from human preferences . In International Conference on Artificial Intelligence and Statistics, 2-4 May 2024,...

work page 2024
[2]

Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324--345

1952
[3]

Bradley C. A. Brown, Jordan Juravsky, Ryan Saul Ehrlich, Ronald Clark, Quoc V. Le, Christopher R \' e , and Azalia Mirhoseini. 2024. https://doi.org/10.48550/ARXIV.2407.21787 Large language monkeys: Scaling inference compute with repeated sampling . CoRR, abs/2407.21787

arXiv doi:10.48550/arxiv.2407.21787 2024
[4]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pond \' e de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bava...

arXiv 2021
[5]

Christiano, Jan Leike, Tom B

Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. https://proceedings.neurips.cc/paper/2017/hash/d5e2c0adad503c91f91df240d0cd4e49-Abstract.html Deep reinforcement learning from human preferences . In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing System...

2017
[6]

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. https://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems . CoRR, abs/2110.14168

arXiv 2021
[7]

Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. 2024. https://doi.org/10.1038/S41586-024-07421-0 Detecting hallucinations in large language models using semantic entropy . Nat., 630(8017):625--630

doi:10.1038/s41586-024-07421-0 2024
[8]

Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang, Qinkai Zheng, Rui Lu, Shuaiqi Duan, Shudan Zhang, Shulin Cao, ...

arXiv 2024

Show all 26 references

[9]

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. NeurIPS

2021
[10]

Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. https://openreview.net/forum?id=nZeVKeeFYf9 Lora: Low-rank adaptation of large language models . In The Tenth International Conference on Learning Representat...

2022
[11]

Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. 2024. Step-dpo: Step-wise preference optimization for long-chain reasoning of llms. arXiv:2406.18629

2024 arXiv
[12]

Hashimoto

Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval

2023
[13]

Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. https://openreview.net/forum?id=v8L0pN6EOi Let's verify step by step . In The Twelfth International Conference on Learning Re...

2024
[14]

Tianlin Liu, Shangmin Guo, Leonardo Bianco, Daniele Calandriello, Quentin Berthet, Felipe Llinares - L \' o pez, Jessica Hoffmann, Lucas Dixon, Michal Valko, and Mathieu Blondel. 2024. https://openreview.net/forum?id=n8g6WMxt09 Decoding-time realignment of language models . In...

2024
[15]

Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. https://doi.org/10.48550/ARXIV.2405.14734 Simpo: Simple preference optimization with a reference-free reward . CoRR, abs/2405.14734

2024 doi arXiv
[16]

Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. 2024. https://doi.org/10.48550/ARXIV.2402.13228 Smaug: Fixing failure modes of preference optimisation with dpo-positive . CoRR, abs/2402.13228

2024 doi arXiv
[17]

Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. 2019. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177

2019 arXiv
[18]

Jan Peters and Stefan Schaal. 2007. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th international conference on Machine learning, pages 745--750

2007
[19]

Manning, Stefano Ermon, and Chelsea Finn

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html Direct preference optimization: Your language model is sec...

2023
[20]

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. https://arxiv.org/abs/1707.06347 Proximal policy optimization algorithms . CoRR, abs/1707.06347

2017 arXiv
[21]

Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. https://doi.org/10.48550/ARXIV.2408.03314 Scaling LLM test-time compute optimally can be more effective than scaling model parameters . CoRR, abs/2408.03314

2024 doi arXiv
[22]

Le, Ed H

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. https://openreview.net/forum?id=1PL1NIMMrw Self-consistency improves chain of thought reasoning in language models . In The Eleventh International Conferen...

2023
[23]

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng...

2024 arXiv
[24]

Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu

Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2024. https://openreview.net/forum?id=N8N0hgNDRt Metamath: Bootstrap your own mathematical questions for large language models . In The Twelfth I...

2024
[25]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[26]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, R \' e mi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. 2024. https://proceedings.mlr.press/v238/gheshlaghi-azar24a.html A general theoretical paradigm to understand learning from human preferences . In International Conference on Artificial Intelligence and Statistics, 2-4 May 2024,...

work page 2024

[2] [2]

Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324--345

1952

[3] [3]

Bradley C. A. Brown, Jordan Juravsky, Ryan Saul Ehrlich, Ronald Clark, Quoc V. Le, Christopher R \' e , and Azalia Mirhoseini. 2024. https://doi.org/10.48550/ARXIV.2407.21787 Large language monkeys: Scaling inference compute with repeated sampling . CoRR, abs/2407.21787

arXiv doi:10.48550/arxiv.2407.21787 2024

[4] [4]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pond \' e de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bava...

arXiv 2021

[5] [5]

Christiano, Jan Leike, Tom B

Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. https://proceedings.neurips.cc/paper/2017/hash/d5e2c0adad503c91f91df240d0cd4e49-Abstract.html Deep reinforcement learning from human preferences . In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing System...

2017

[6] [6]

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. https://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems . CoRR, abs/2110.14168

arXiv 2021

[7] [7]

Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. 2024. https://doi.org/10.1038/S41586-024-07421-0 Detecting hallucinations in large language models using semantic entropy . Nat., 630(8017):625--630

doi:10.1038/s41586-024-07421-0 2024

[8] [8]

Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang, Qinkai Zheng, Rui Lu, Shuaiqi Duan, Shudan Zhang, Shulin Cao, ...

arXiv 2024