Artificial Intelligence · arXiv:2607.25091 v1 · submitted 27 Jul 2026 · 26 references
Towards Robust Reinforcement Learning for Small-Scale Language Model Agents
The paper says
“The alignment of Small Language Models (SLMs) in the 70–500M parameter range using reinforcement learning is often considered unstable, though the underlying failure mechanisms have not been systematically investigated.”
more of the abstract
In the State-of-the-Art (SOTA) research, fifteen (model, corpus) configurations were trained using Proximal Policy Optimization (PPO). The experiments included Pythia-70M, 160M, 410M and SmolLM2-135M, 360M on the TinyStories, CNN/DailyMail, and Wikitext-103 corpora. Three reproducible failure modes were identified in small-scale language models: silent LoRA parameter freezing in standard PEFT/TRL pipelines, numerical overflow in importance ratios when using bfloat16, and catastrophic policy collapse due to reward-model error. These issues were addressed using a merge-and-reinitialize adapter technique, float32 precision during PPO updates, and a three-layer safety mechanism comprising reward whitening, importance-ratio guarding, and weight rollback. In this paper, a capacity-headroom hypothesis is proposed, which states that PPO performance at the SLM scale depends on both a fluent supervised model (PPL < 20) and a discriminative reward signal, rather than on the number of model parameters. The proposed system converged stably in all experiments and improved preference win rate over the SFT baseline in configurations with a fluent prior and an informative reward signal. Furthermore, it outperformed instruction-tuned baselines while requiring significantly less training data. All checkpoints, preference datasets, and training scripts are publicly released$^{\S}$.
from the abstract
Pith says
Machine review · grok-4.5 · 31 Jul 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. Solid engineering paper on why PPO breaks on 70–500M models and how to keep it stable; the preference/SOTA claims are weaker than the stability work. This line is wrong
5 major objections · 6 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 26 references say.
The paper says, at §IV.D, Eq. (3), Table II
The review did not quote this passage.
Pith says: Objection 1 of 5 · major
The central claim that PPO 'improved preference win rate over the SFT baseline' is evaluated exclusively with r_ϕ, the same Bradley–Terry model used as the PPO reward in Eq. (3). Reporting that r_ϕ scores increase after 250 steps of optimizing r_ϕ demonstrates that optimization occurred, not that response quality improved.
more of objection 1
The concern is acute because r_ϕ is trained only on three mechanical degradations (truncation, sentence shuffle, cross-example mismatch), so it plausibly encodes surface heuristics a policy can exploit. The paper's own numbers are consistent with reward gaming rather than improvement: for Pythia-410M on TinyStories, reward rises +1.355 while generation PPL degrades 6.5→7.3 and D-1/D-2/ROUGE-1 all fall (Tables II and IV). A load-bearing fix: evaluate PPO vs. SFT generations with an instrument independent of the training reward — e.g., a reward model trained on a differe
The paper says, at §IV.D, Table II, Fig. 2
The review did not quote this passage.
Pith says: Objection 2 of 5 · major
The 'win rate' column is not a measured preference: it is an analytical estimate Φ(Δ/√(σ²_PPO+σ²_SFT)) assuming independent Gaussian reward distributions. PPO and SFT generations are produced from the same prompts, so per-prompt rewards are paired and correlated, and reward distributions after PPO are typically non-Gaussian (e.g., whitened/clipped during training).
more of objection 2
Since the per-prompt reward distributions were already collected (§V.A states CIs are computed from the complete per-prompt distribution), the empirical pairwise fraction P(r_PPO > r_SFT) on shared prompts should be reported instead of, or alongside, the Gaussian approximation. If the two disagree, all statements conditioned on win% (including the abstract) need revision.
The paper says, at §V.C, Table III, Abstract
The review did not quote this passage.
Pith says: Objection 3 of 5 · major
The claim that the pipeline 'outperformed instruction-tuned baselines' rests on scoring SmolLM2/Qwen Instruct outputs with the authors' own per-dataset r_ϕ. This reduces to 'our pipeline's outputs score higher on our pipeline's reward': r_ϕ is trained on domain-specific synthetic degradations and is plausibly calibrated to reward exactly the style the domain SFT/PPO pipeline produces, while the instruct baselines are evaluated zero-shot off-domain.
more of objection 3
Combined with Major Comment 1, this claim is not currently supported. Either add an independent evaluation instrument for the cross-model comparison, or restrict the claim to perplexity and diversity (where domain-specific SFT beating general instruct models is expected anyway and should be stated as such).
The paper says, at §IV.D vs. Table III caption
The review did not quote this passage.
Pith says: Objection 4 of 5 · major
The PPL metric is defined inconsistently: §IV.D says 'Perplexity (PPL) was computed on the generated continuations', while the Table III caption says 'PPL is computed using reference text'. These are different quantities with different interpretations, and the distinction matters for the stability claim in §V.E that all runs stayed 'within 15% of the SFT perplexity' — if PPL is measured …
more of objection 4
… on the model's own generations (and by which model?), a policy drifting toward high-confidence degenerate text could show stable PPL while quality collapses. The metric definition must be stated unambiguously and used consistently, including which model scores the perplexity.
The paper says, at §III.D, §V.E
The review did not quote this passage.
Pith says: Objection 5 of 5 · major
Contribution 1 frames three failure modes as 'specific to the small-scale architectures', but the evidence supports this only partially. Failure Mode I (silent LoRA non-trainability in PEFT/TRL) is a library integration bug that can occur at any scale; Failure Mode II is argued to bite below 200M parameters but no precision-versus-scale measurement is shown; Failure Mode III is a generic RLHF failure.
more of objection 5
Second, the ablation uses a single configuration (Pythia-70M/TinyStories) and compares only naive/manual/full pipelines — it does not isolate the three safety layers (whitening, ratio guard, rollback), so the claim that each layer addresses a distinct failure mode, and the sensitivity claim in §VI that the ratio cap is the most critical parameter, are not experimentally supported. A per-layer ablation (even on 2–3 configurations) and a scale-controlled check of whether FM-I also occurs at 1
6 minor objections
The paper says, at §V.B, Fig. 3
Pith says: minor objection 1 of 6
The decision rule PPL_SFT < 20 and the 'inflection point' in Fig. 3 are derived post hoc from 15 configurations. The regression region (PPL ∈ [20,50]) is supported essentially by one significant negative point (Pythia-410M/Wikitext-103). The text should state explicitly that the threshold was selected on the same data used to evaluate it, and ideally reserve a held-out set of configurations or seeds to validate it.
The paper says, at §III.C, §V.B
Pith says: minor objection 2 of 6
Condition (ii) of the capacity-headroom hypothesis — 'a discriminative reward signal' — is never operationalized. Held-out preference accuracy of r_ϕ on synthetic pairs (and ideally on non-degraded text) is not reported. Adding reward-model validation accuracy per configuration would let readers assess whether reward discriminability, rather than SFT fluency alone, explains the Fig. 3 pattern.
show 4 more
The paper says, at §V.A
Pith says: minor objection 3 of 6
No seed variance is reported anywhere: each configuration appears to be a single training run, and the CIs in §V.A reflect per-prompt variance, not training variance. At minimum, state this explicitly; ideally repeat the three key configurations across ≥3 seeds, since conclusions about stability/convergence are central to the paper.
The paper says, at Table V
Pith says: minor objection 4 of 6
Table V presents a single qualitative example. Either label it explicitly as illustrative/cherry-picked or add a small systematic sample (e.g., 5–10 randomly selected prompts) in an appendix.
The paper says, at Abstract, Fig. 1
Pith says: minor objection 5 of 6
The abstract phrase 'In the State-of-the-Art (SOTA) research, fifteen (model, corpus) configurations were trained' is awkward and misuses 'SOTA'; suggest 'In this work'. Also, Fig. 1's sidebar notes that Φ denotes both an update operator and the standard normal CDF — rename one symbol.
The paper says, at §IV.C, Limitations
Pith says: minor objection 6 of 6
The 250-step PPO budget is acknowledged as a limitation, but given that overoptimization against a synthetic reward model is the main threat (Major Comment 1), reward/PPL/diversity trajectories over training steps would be highly informative and cheap to include from existing logs.
What would settle it
Re-score the PPO vs. SFT generations on the 200 held-out prompts for the three significant configs (Pythia-410M/TS, SmolLM2-360M/TS, SmolLM2-360M/WT) using an independent evaluator the policy never optimized: e.g., a publicly available reward model trained on human preferences (ArmoRM-Llama3-8B or Skywork-Reward-8B) plus an LLM-judge pairwise comparison. If the PPO>SFT ordering replicates (win rate materially >50%) under the independent judge, the gains are real and the claim survives; if it flips or collapses to ~50%, the reported Δ is reward hacking of the synthetic r_ϕ. As a cheaper pre-check, measure r_ϕ's pairwise accuracy on a small human-annotated or GPT-4-labeled preference set disjoint from the truncation/shuffle/mismatch distribution; accuracy near chance would confirm the RM does not generalize beyond its synthetic degradations.
from the skeptic's reading
How the review went
4 readings by 2 models · 31 Jul 2026
- Reader holds up, with conditions
- Skeptic agrees
- Editor wrote the verdict above
- Referee major revision: 5 major objections, 6 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
The useful core here is not a new RL objective. It is a clean, reproducible map of three failure modes that actually show up when you run standard PEFT/TRL PPO on small models: silent LoRA non-trainability, bfloat16 importance-ratio blowups below ~200M, and reward-driven collapse. They fix them with merge-and-reinit, float32 in the PPO loop, and a simple safety stack (whiten/clip, ratio guard, rollback). All fifteen fixed-hyperparameter runs finish. That is real, checkable work, and the ablation on naive PEFT vs their stack is convincing. The released code, checkpoints, and data make it more than a blog post.
The capacity-headroom framing is also fair to the data. Fig. 3 and Table II show a clear pattern: when SFT PPL is already low (roughly <20) and the reward has something to grab, Δ goes positive; when the prior is weak, PPO does nothing useful. That is a practical go/no-go rule for people training on-device agents, not a deep theory claim, and they mostly present it that way.
Where it softens is the preference story. Win rates and the “beats instruct baselines” table are scored by the same Bradley–Terry model PPO was trained against, built only on truncation/shuffle/mismatch. Raising rϕ after optimizing rϕ is close to tautological; it does not independently establish better responses. Their own TinyStories numbers (reward up, PPL and diversity sometimes worse) fit surface-heuristic gaming. They flag synthetic prefs in Limitations, but the abstract and §V-C still state the preference/SOTA claims flat. Stability and the PPL rule do not need that claim; the alignment-quality claim does.
Math is standard PPO/GAE/BT—no red flags. Citations are appropriate. No multi-seed variance, no human prefs, no DPO control, single-turn only. Still: for anyone doing SLM RLHF or edge agents, the failure-mode writeup and the recipe are worth reading. I would send it to referees. Engage for the engineering; discount the win-rate language until someone scores with held-out humans or an independent judge.
The strongest objection
The central empirical claim — that PPO "improved preference win rate over the SFT baseline" in fluent-prior configurations — rests on evaluation by r_ϕ, the very Bradley–Terry reward model used as the PPO reward (Eq. 3). Optimizing r_ϕ for 250 steps and then reporting that r_ϕ scores went up is close to circular: Δ > 0 demonstrates that optimization occurred, not that response quality improved. The concern is sharpened by how r_ϕ was trained (Stage 0): its entire preference signal is three mechanical degradations (truncation, sentence shuffling, cross-example mismatch). Such a reward model plausibly encodes a small set of surface heuristics — don't stop mid-clause, don't permute sentences, stay on the prompt's topic. A policy can raise r_ϕ by exploiting those heuristics without any genuine improvement, and the paper's own numbers are consistent with that signature: for Pythia-410M on TinyStories, reward rises +1.355 while PPL degrades 6.5→7.3 and D-1/D-2/ROUGE all fall (Tables II, IV); the qualitative example (Table V) is a single cherry-picked prompt. The "win rate" is also not a pairwise preference measurement — it is an analytical Φ(Δ/√(σ²_PPO+σ²_SFT)) assuming independent Gaussian reward distributions, which inherits all of the above. The SOTA comparison in Table III ("highest reward scores" vs. SmolLM2/Qwen instruct models) uses the same in-house r_ϕ, so the outperformance claim reduces to "our pipeline's outputs score higher on our pipeline's reward." None of this touches the stability contributions — the three failure modes and their fixes are independently checkable from the released code and the ablation in §V-E is convincing — but the strongest_claim as stated (win-rate improvement, outperformance of instruct baselines) is load-bearing on r_ϕ being a valid proxy for quality, which nothing in the paper independently establishes. The authors partially acknowledge this in Limitations, but the abstract and §V-C state the claims unconditionally.
The reader's weakest_assumption and my load-bearing concern are the same one, so I agree. My analysis adds specificity rather than a new objection: (1) the circularity is not merely "synthetic RM might not generalize" but "evaluation by the optimized RM," which makes positive Δ near-tautological; (2) the paper's own auxiliary metrics (PPL, D-1, ROUGE regressions in exactly the configs with the largest Δ) display the classic reward-hacking signature, which is internal evidence the concern may already be landing; (3) the win rate is an analytical normal-CDF estimate under an independence assumption, not a measured pairwise preference, so "win rate 59.9%" overstates what was observed. Despite this, I recommend UNCHANGED rather than REJECT: the reader's CONDITIONAL verdict already carves the paper correctly — accept the stability contribution (three failure modes, fixes, fifteen-config convergence, ablation, released artifacts) as solid and checkable, while conditioning the alignment/win-rate/SOTA claims on exactly the independent preference evaluation I propose. The failure-mode findings have independent support (the released code and the §V-E ablation directly demonstrate them), and the capacity-headroom hypothesis is stated as corpus- and scale-bounded guidance, which is the appropriate epistemic register. What needs to change is on the authors' side (independent evaluation before strong alignment claims), which the existing verdict already requires.
The referee's report
If the engineering claims hold, the paper is a useful practical contribution: it documents three concrete, reproducible failure modes of PPO at the 70–500M scale (silent LoRA freezing in PEFT/TRL, bfloat16 importance-ratio overflow, reward-driven collapse) with specific fixes, trains fifteen matched (model, corpus) configurations under fixed hyperparameters, reports z-test CIs on n=200 held-out prompts, and releases checkpoints, preference datasets, and training scripts. The merge-and-reinitialize trick and the float32-PPO-loop recommendation are actionable for practitioners, and the capacity-headroom hypothesis (PPL_SFT < 20 as a gating condition) is a falsifiable decision rule. The limitations section is unusually candid about scope (synthetic preferences, single-turn MDP, additive invariance of rewards). The preference-improvement and SOTA-comparison claims, however, are currently unsupported by the evaluation design (see Major Comments 1–3), which substantially limits the significance of those specific claims as stated.
Pith's reading of the paper (a machine's summary, not the authors')
Small language models are widely thought to be too unstable for classical PPO alignment, so practitioners often skip reinforcement learning entirely. This paper runs a full supervised-fine-tuning, reward-model, and PPO pipeline across fifteen model–corpus pairs in the 70–500M range and shows that three concrete engineering failures—not model size itself—drive the instability: LoRA adapters that silently stop training, bfloat16 overflow in importance ratios, and unbounded rewards that collapse the policy. Fixing those with a merge-and-reinitialize adapter step, float32 PPO updates, and a three-layer safety loop (reward whitening, ratio guarding, weight rollback) makes every run converge. The central claim is a capacity-headroom rule: PPO improves preference scores over the supervised baseline only when the supervised model is already fluent (perplexity under about 20) and the reward model can tell good answers from bad ones. When those conditions hold, the aligned small models can beat much more heavily trained instruction-tuned baselines while using far less data.
What carries the argument. The capacity-headroom hypothesis, enforced by a three-layer safety mechanism (reward whitening with 3σ clipping, importance-ratio guarding that skips mini-batches above ratio 5, and weight rollback on NaN/Inf) together with merge-and-reinitialize LoRA and float32 PPO updates. These keep the PPO feedback loop stable so the fluency-and-discriminability rule can be tested cleanly.
Core claim. PPO effectiveness at the small-language-model scale depends on a fluent supervised prior (empirical decision rule: SFT perplexity below 20) plus a discriminative reward signal, not on raw parameter count. With three identified failure modes fixed, PPO converges in all fifteen tested configurations and raises preference win rate over SFT precisely when that headroom condition is met.
The premise it rests on. The paper assumes that synthetic preference pairs made only by truncating, shuffling sentences, and swapping continuations produce a reward model that is a fair enough stand-in for real response quality.
If the paper is right (Pith's inferences, not the authors')
Practitioners can use SFT perplexity under ~20 as a go/no-go gate before spending compute on PPO for on-device agents.
Standard PEFT/TRL LoRA pipelines need an explicit merge-and-reinitialize (or gradient re-enable) step before PPO, or the policy never updates.
Models under ~200M parameters should run PPO importance-ratio math in float32 to avoid silent overflow.
When SFT perplexity sits between 20 and 50, effort is better spent improving the supervised prior than running PPO.
The same stability controls are necessary conditions for any multi-turn or tool-using agent MDP built on the same loop.
If the headroom rule is real, group-relative methods (GRPO, RLOO) may still fail on weak SFT priors for the same distributional reason, even without an explicit reward model.
On-device agent stacks could treat “fluent prior + cheap synthetic RM + guarded PPO” as a default recipe instead of importing billion-scale instruction-tuning data.
A natural next measurement is whether the same PPL threshold predicts PPO gains when the reward comes from verifiable tool success rather than Bradley–Terry text preferences.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by grok-4.5, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
The alignment of Small Language Models (SLMs) in the 70–500M parameter range using reinforcement learning is often considered unstable, though the underlying failure mechanisms have not been systematically investigated. In the State-of-the-Art (SOTA) research, fifteen (model, corpus) configurations were trained using Proximal Policy Optimization (PPO). The experiments included Pythia-70M, 160M, 410M and SmolLM2-135M, 360M on the TinyStories, CNN/DailyMail, and Wikitext-103 corpora. Three reproducible failure modes were identified in small-scale language models: silent LoRA parameter freezing in standard PEFT/TRL pipelines, numerical overflow in importance ratios when using bfloat16, and catastrophic policy collapse due to reward-model error. These issues were addressed using a merge-and-reinitialize adapter technique, float32 precision during PPO updates, and a three-layer safety mechanism comprising reward whitening, importance-ratio guarding, and weight rollback. In this paper, a capacity-headroom hypothesis is proposed, which states that PPO performance at the SLM scale depends on both a fluent supervised model (PPL < 20) and a discriminative reward signal, rather than on the number of model parameters. The proposed system converged stably in all experiments and improved preference win rate over the SFT baseline in configurations with a fluent prior and an informative reward signal. Furthermore, it outperformed instruction-tuned baselines while requiring significantly less training data. All checkpoints, preference datasets, and training scripts are publicly released$^{\S}$.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-09-23
The authors' main claim, as the classifier picked it out 0.91
“In this paper, a capacity-headroom hypothesis is proposed, which states that PPO performance at the SLM scale depends on both a fluent supervised model ($\text{PPL}<20$) and a discriminative reward signal, rather than on the number of model parameters.”
How the main claim is stated: speculative 0.47
What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.28 · human participants 0.04 · controlled comparison 0.96 · randomized assignment 0.02 · states quantitative result 0.93 · compares with prior results 0.97 · mentions code or data availability 0.98
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| The alignment of Small Language Models (SLMs) in the 70--500M parameter range using reinforcement learning is often considered unstable, though the underlying failure mechanisms have not been systematically investigated. | gap 0.73 | 0.05 | 0.03 | asserted 0.69 |
| In the State-of-the-Art (SOTA) research, fifteen (model, corpus) configurations were trained using Proximal Policy Optimization (PPO). | method 0.89 | 0.47 | 0.02 | asserted 1.00 |
| The experiments included Pythia-70M, 160M, 410M and SmolLM2-135M, 360M on the TinyStories, CNN/DailyMail, and Wikitext-103 corpora. | method 0.99 | 0.78 | 0.02 | asserted 1.00 |
| Three reproducible failure modes were identified in small-scale language models: silent LoRA parameter freezing in standard PEFT/TRL pipelines, numerical overflow in importance ratios when using bfloat16, and catastrophic policy collapse due to reward-model error. | result 0.97 | 0.92 | 0.08 | asserted 1.00 |
| These issues were addressed using a merge-and-reinitialize adapter technique, float32 precision during PPO updates, and a three-layer safety mechanism comprising reward whitening, importance-ratio guarding, and weight rollback. | method 0.91 | 0.91 | 0.07 | asserted 1.00 |
| In this paper, a capacity-headroom hypothesis is proposed, which states that PPO performance at the SLM scale depends on both a fluent supervised model ($\text{PPL}<20$) and a discriminative reward signal, rather than on the number of model parameters. | contribution 0.49 | 0.63 | 0.73 | speculative 0.47 |
| The proposed system converged stably in all experiments and improved preference win rate over the SFT baseline in configurations with a fluent prior and an informative reward signal. | result 1.00 | 0.94 | 0.35 | asserted 1.00 |
| Furthermore, it outperformed instruction-tuned baselines while requiring significantly less training data. | result 1.00 | 0.90 | 0.11 | asserted 1.00 |
| All checkpoints, preference datasets, and training scripts are publicly released$^{\S}$. | contribution 0.91 | 0.74 | 0.02 | asserted 1.00 |
What kind of paper the classifier reports this is: empirical study 0.60 · evidence from observation or experiment 0.99
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 7,169 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (3)
Figures are the authors', reproduced from the paper.
References (26)
- [1] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askellet al., “Language Models are Few-Shot Learners,”Advances in Neural Information Processing Systems, vol. 33, pp. 1877–1901, 2020 on Pith
- [2] R. Eldan and Y . Li, “TinyStories: How Small Can Language Models Be and Still Speak Coherent English?”arXiv:2305.07759, 2023 on PithPith's reviewarXiv
- [3] S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O’Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raffet al., “Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling,” inInternational Conference on Machine Learning. PMLR, 2023... on Pith
- [4] L. B. Allal, A. Lozhkov, E. Bakouch, G. M. Bl ´azquez, G. Penedo, L. Tunstall, A. Marafioti, H. Kydl ´ıˇcek, A. P. Lajar ´ın, V . Srivastavet al., “SmolLM2: When Smol Goes Big–Data-Centric Training of a Small Language Model,”arXiv:2502.02737, 2025 on PithPith's reviewarXiv
- [5] Z. Liu, C. Zhao, F. Iandola, C. Lai, Y . Tian, I. Fedorov, Y . Xiong, E. Chang, Y . Shi, R. Krishnamoorthiet al., “MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases,” arXiv:2402.14905, 2024 on PithPith's reviewarXiv
- [6] A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Weiet al., “Qwen2.5 Technical Report,”arXiv:2412.15115, 2024 on PithPith's reviewarXiv
- [7] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei, “Deep Reinforcement Learning from Human Preferences,”Advances in Neural Information Processing Systems, vol. 30, 2017 on Pith
- [8] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Rayet al., “Training Language Models to Follow Instructions with Human Feedback,”Advances in Neural Information Processing Systems, vol. 35, pp. 27 730–27 744, 2022 on Pith
- [9] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal Policy Optimization Algorithms,”arXiv:1707.06347, 2017 on PithPith's reviewarXiv
- [10] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct Preference Optimization: Your Language Model is Secretly a Reward Model,”Advances in Neural Information Processing Systems, vol. 36, pp. 53 728–53 741, 2023 on Pith
- [11] R. Nallapati, B. Zhou, C. dos Santos, C. Gulcehre, and B. Xiang, “Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond,” inProceedings of the SIGNLL Conference on Computational Natural Language Learning, 2016, pp. 280–290 on Pith
- [12] S. Merity, C. Xiong, J. Bradbury, and R. Socher, “Pointer Sentinel Mixture Models,”arXiv:1609.07843, 2016 on PithPith's reviewarXiv
- [13] L. von Werra, Y . Belkada, L. Tunstall, E. Beeching, T. Thrush, N. Lambert, and S. Huang, “TRL: Transformer Reinforcement Learning,” GitHub, 2022. [Online]. Available: https://github.com/huggingface/trl on Pith
- [14] N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. V oss, A. Radford, D. Amodei, and P. F. Christiano, “Learning to Summarize with Human Feedback,”Advances in Neural Information Processing Systems, vol. 33, pp. 3008–3021, 2020 on Pith
- [15] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosaleet al., “Llama 2: Open Foundation and Fine-Tuned Chat Models,”arXiv:2307.09288, 2023 on PithPith's reviewarXiv
- [16] K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela, “KTO: Model Alignment as Prospect Theoretic Optimization,” arXiv:2402.01306, 2024 on PithPith's reviewarXiv
- [17] Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, M. Zhang, Y . Li, Y . Wu, and D. Guo, “DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models,”arXiv:2402.03300, 2024 on PithPith's reviewarXiv
- [18] E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, W. Chenet al., “LoRA: Low-Rank Adaptation of Large Language Models,”International Conference on Learning Representations (ICLR), vol. 1, no. 2, p. 3, 2022 on Pith
- [19] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “QLoRA: Ef- ficient Finetuning of Quantized LLMs,”Advances in Neural Information Processing Systems, vol. 36, pp. 10 088–10 115, 2023 on Pith
- [20] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving, “Fine-Tuning Language Models from Human Preferences,”arXiv:1909.08593, 2019 on PithPith's reviewarXiv
show 6 more
- [21] J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel, “High- Dimensional Continuous Control Using Generalized Advantage Estima- tion,”arXiv:1506.02438, 2015 on PithPith's reviewarXiv
- [22] N. Jain, P.-y. Chiang, Y . Wen, J. Kirchenbauer, H.-M. Chu, G. Somepalli, B. Bartoldson, B. Kailkhura, A. Schwarzschild, A. Sahaet al., “NEFTune: Noisy Embeddings Improve Instruction Finetuning,” inInternational Conference on Learning Representations, vol. 2024, 2024, pp. 17 5... on Pith
- [23] L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshimaet al., “The Pile: An 800gb Dataset of Diverse Text for Language Modeling,”arXiv:2101.00027, 2020 on PithPith's reviewarXiv
- [24] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” arXiv:1711.05101, 2017 on PithPith's reviewarXiv
- [25] R. A. Bradley and M. E. Terry, “Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons,”Biometrika, vol. 39, no. 3/4, pp. 324–345, 1952 on Pith
- [26] C.-Y . Lin, “ROUGE: A Package for Automatic Evaluation of Summaries,” inText Summarization Branches Out, 2004, pp. 74–81 on Pith
Cited by 2 Pith papers
Reviewed papers on Pith that cite this one. The full citation record
- Do Small Language Models Learn to Negotiate? A Controlled Scaling Study of RL-Trained Sellers cs.AI · Oct 2026
- A Survey on RLOps for Large Language Models cs.SE · Nov 2026
How the literature uses it
What this paper does with the arXiv papers it cites
What it cites · 17 sentences without a confident label
Uses the method or tool of 5 papers
-
[equation] PPO [ref] is applied with GAE [ref] ($λ=0.95, γ=1.0$), a clipping ratio $ϵ=0.2$, and an adaptive KL coefficient [ref] targeting a mean per-token KL of $6.0$ nats, implemented using TRL [ref] .
Fine-Tuning Language Models from Human Preferences · Stage 3: PPO Fine-Tuning and Stabilization · 1.00
-
[equation] PPO [ref] is applied with GAE [ref] ($λ=0.95, γ=1.0$), a clipping ratio $ϵ=0.2$, and an adaptive KL coefficient [ref] targeting a mean per-token KL of $6.0$ nats, implemented using TRL [ref] .
High-Dimensional Continuous Control Using Generalized Advantage Estimation · Stage 3: PPO Fine-Tuning and Stabilization · 1.00
-
[equation] PPO [ref] is applied with GAE [ref] ($λ=0.95, γ=1.0$), a clipping ratio $ϵ=0.2$, and an adaptive KL coefficient [ref] targeting a mean per-token KL of $6.0$ nats, implemented using TRL [ref] .
Proximal Policy Optimization Algorithms · Stage 3: PPO Fine-Tuning and Stabilization · 1.00
Uses data or values from 2 papers
-
Three corpora with different linguistic characteristics were selected: TinyStories ($𝒟_TS$) [ref] for simple narratives, CNN/DailyMail ($𝒟_CNN$) [ref] for formal journalism, and Wikitext-103 ($𝒟_WT$) [ref] for technical content.
TinyStories: How Small Can Language Models Be and Still Speak Coherent English? · Datasets · 0.97
-
Three corpora with different linguistic characteristics were selected: TinyStories ($𝒟_TS$) [ref] for simple narratives, CNN/DailyMail ($𝒟_CNN$) [ref] for formal journalism, and Wikitext-103 ($𝒟_WT$) [ref] for technical content.
Pointer Sentinel Mixture Models · Datasets · 0.97
Compares with 2 papers
-
The proposed system achieves performance comparable to publicly available instruction-tuned baselines, including SmolLM2-Instruct [ref] and Qwen2.5-Instruct [ref] , while using significantly less training data.
SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model · Introduction · 1.00
-
The proposed system achieves performance comparable to publicly available instruction-tuned baselines, including SmolLM2-Instruct [ref] and Qwen2.5-Instruct [ref] , while using significantly less training data.
Qwen2.5 Technical Report · Introduction · 1.00
Cites as background 6 papers
-
Existing SLM-alignment research primarily focuses on developing high-quality supervised models through SFT [ref] or using preference-learning methods that avoid the explicit reinforcement-learning process, such as DPO [ref] , KTO [ref] , and GRPO [ref] .
TinyStories: How Small Can Language Models Be and Still Speak Coherent English? · Related Works · 0.86
-
Existing SLM-alignment research primarily focuses on developing high-quality supervised models through SFT [ref] or using preference-learning methods that avoid the explicit reinforcement-learning process, such as DPO [ref] , KTO [ref] , and GRPO [ref] .
KTO: Model Alignment as Prospect Theoretic Optimization · Related Works · 0.86
-
Existing SLM-alignment research primarily focuses on developing high-quality supervised models through SFT [ref] or using preference-learning methods that avoid the explicit reinforcement-learning process, such as DPO [ref] , KTO [ref] , and GRPO [ref] .
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models · Related Works · 0.86
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment