Machine Learning · arXiv:2610.21404 v1 · submitted 15 Oct 2026 · 26 references
Multi-Reward Hacking: From Aggregation-Induced Drift to Trust-Anchored Defense
The paper says
“To improve task performance while encouraging desirable response properties, LLM post-training often combines a primary reward (e.g., correctness) with auxiliary rewards (e.g., brevity or efficiency). However, how these rewards are aggregated can critically affect training outcomes.”
more of the abstract
We show that symmetric aggregation, which treats reward channels equally, can induce a failure where aggregate reward improves while primary-task performance degrades, even when individual rewards are meaningful. We investigate this aggregation-induced failure through a one-step multi-reward decision formulation, characterizing when it constitutes multi-reward hacking. We prove that, under symmetric aggregation, deterministic conditional-mean GRPO converges to auxiliary-favoring behavior when auxiliary rewards are more reliably obtained, regardless of initialization. We propose Trust-Anchored Gating (TAG), which first elects an anchor channel based on the policy drift induced by optimizing each channel alone, then gates auxiliary credit on anchor satisfaction. Unlike calibration-sensitive static reweighting, TAG structurally eliminates credit for auxiliary-only outcomes while preserving additional credit for joint success. In the same analytical setting, TAG provably reverses this selection, converging to the primary-favoring action from any initialization when the primary channel is anchored. Experiments across Qwen and Llama models on mathematics and program synthesis reveal substantial degradation under symmetric, constrained, and normalized aggregation, whereas TAG mitigates this failure. On Qwen3-4B with a brevity reward, TAG achieves 0.581 pass@4 versus 0.220 for additive aggregation, while shortening responses by 44% relative to correctness-only training.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 16 Oct 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up. A well-executed paper that identifies a real failure mode in multi-reward LLM post-training and provides a principled defense, with strong theoretical and empirical support despite a manageable gap between the simplified theory and full practical algorithm. This line is wrong
1 major objection · 1 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 26 references say.
The paper says, at Appendix A.1, Section 3.3, Section 6.1
The review did not quote this passage.
Pith says: Objection 1 of 1 · major
The theoretical analysis (Theorems 1 and 2) is conducted under the deterministic conditional-mean GRPO recursion (Eq. 8, Appendix A.1), which assumes the logit update equals its exact expectation conditional on the current policy. The experiments use the full stochastic GRPO with clipping and no KL penalty (except control arms).
more of objection 1
While experimental results are qualitatively consistent with the theory, the paper does not provide a formal link (e.g., via stochastic approximation or mean-field analysis) between the deterministic recursion and the actual training dynamics. Clipping alters the update when the probability ratio deviates, and sampling noise can cause deviations from the expected update. The paper should discuss conditions under which the deterministic dynamics approximately describe the stochastic behavior, or provide numerical evidence (e.g., from multiple seeds) that the stochastic trajectories track the deterministic one on average. This gap is load-bearing because the central claim that symmetric aggregation drives GRPO toward auxiliary-favoring behavior is proven only for the simplified deterministic dynamics, and it is not a priori clear that clipping and stochasticity preserve the selection result.
1 minor objection
The paper says, at General
Pith says: minor objection 1 of 1
The paper is well-written and the theoretical exposition is clear. Minor issues: (1) In Figure 3, the label 'MO-GRPO' is included but Theorem 1 does not apply to it; the text states that MO-GRPO also selects a1 in that regime, but the proof does not cover it.
more of objection 1
This should be clarified in the caption or main text. (2) Table 1 and Table 2 label pass@4 with an upward arrow; the meaning of arrows is defined in the caption but not consistently used in all tables (e.g., Table 7, Table 11). Please unify the notation. (3) In the abstract and Section 1, the claim that TAG shortens responses by 44% relative to correctness-only training is reported only for one setting; it would be helpful to mention the setting and model. (4) The paper uses 'reverse KL per token' in Eq. (5); clarify that the denominator is the expected number of generated tokens and that the estimate is pooled over all generated tokens, not averaged over responses. (5) In the proof of Theorem 1 (Appendix A.1), Step 4 discusses L_t -> L^* < infinity and then derives a contradiction; the argument is dense and would benefit from a brief intuitive summary before the formal steps.
What would settle it
Simulate the tabular three-action MDP (Section 3.3) with the actual stochastic GRPO algorithm (n=8, clipping ε=0.2, no KL) for a range of (η,ρ) values (e.g., grid η∈{0,0.1,0.5}, ρ∈{0.3,0.6,0.9}) and compare the long-run distribution of actions to the deterministic conditional-mean prediction (Theorem 1). If the empirical selection does not match the prediction (e.g., a2 persists for some η<1,ρ<1), the theoretical result may not extend to the practical algorithm.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 16 Oct 2026
- Reader holds up
- Skeptic partly agrees
- Editor wrote the verdict above
- Referee minor revision: 1 major objection, 1 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
The core claim of this paper is important and largely holds up: symmetric reward aggregation can cause a previously unrecognized failure mode where the policy drifts toward an easier auxiliary objective even when the primary objective is correctly rewarded. The paper does a good job separating this "aggregation-induced drift" from known proxy misspecification, and the three-action MDP used for the theoretical analysis is simple but effective for illustrating the mechanism.
What the paper does well: it formally proves (Theorem 1) that under symmetric, monotone aggregation the deterministic conditional-mean GRPO dynamics converge to the auxiliary-favoring action, and it shows (Theorem 2) that the proposed TAG method reverses this selection. The empirical work is extensive: experiments across two model families (Qwen, Llama) at multiple scales (1.7B-30B) and two tasks (math, program synthesis) with several reward schemes. The magnitude of degradation under symmetric methods is striking, and TAG consistently recovers task performance while still achieving auxiliary benefits like brevity. The anchor-election method via reverse KL drift is novel and well-motivated.
The main soft spot is the gap between the theory and experiments. The theoretical analysis uses a deterministic conditional-mean recursion with binary rewards and assumes the update is applied in expectation, while the actual experiments use the full stochastic GRPO with clipping and no KL penalty. The paper acknowledges this limitation (the proof does not apply to the actual GRPO used in training), but it remains a load-bearing gap. That said, the empirical evidence is so consistent that I think the central message survives this gap. It would strengthen the paper to provide a heuristic argument (e.g., via mean-field approximation) or empirical comparison of deterministic vs. stochastic dynamics.
Other limitations:the analysis assumes a designated primary channel and requires short probe runs for anchor election. These are acknowledged, and TAG still adds some computational cost. The binary reward and three-action MDP is a simplification;extending to continuous rewards and more complex action spaces is left open.
Who this paper is for: anyone working on multi-reward Rl for LLMs in industry or academia. It should change how practitioners think about reward aggregation.
Verdict: I would accept this paper with minor re vis ion. The gap between theory and practice is not fatal,given the strength of the empirical results. The paper is clear, honest about its limitations, and makes a substantial contribution.
The strongest objection
The orem 1 proves convergence to the auxiliary-favoring action under the deterministic conditional-mean GRPO recursion (Eq. 8and Step 0 in Appendix A.1). The actual algorithm used in expriments (Section 6) is the full stochastic GRPO with clipping and no KL penalty (except control arms). The paper does not establish a formal link (e.g., via mean-field or stochastic approximation) between the two dynamics. The proof relies on the update being exactly the conditional mean, which holds only in expectation over groups and requires the update to be applied after each group rather than as a batch. In practice, the update is stochastic, and clipping alters the update when the probability ratio deviates. While the experiments are consistent, the theoretical guarantee does not directly apply to the trained algorithm. This is a recognized limitation (the reader's weakest assumption), but it remains the most load-bearing gap in the central claim that 'symmetric aggregation drives GRPO toward auxiliary-favoring behavior'.
The paper's theoretical core is Theorem 1, which proves selection of the auxiliary action under a specific deterministic recursion. The paper then uses this to explain empirical collapses in full LLM training with stochastic GRPO. However, the distance between the analyzed dynamics and the actual algorithm is significant: the deterministic recursion assumes an exact conditional-mean update, while GRPO uses per-step stochastic gradients with clipping and without KL penalty. The paper includes a stochastic check only at a boundary point (ρ=1,η=0, Proposition A.1). No general argument is given that the stochastic process inherits the same fixed points or convergence. Given that the central claim is about the failure mode in practice, this gap is the most load-bearing. The paper's limitations statement acknowledges the simplified model but does not specifically address the deterministic-vs-stochastic gap. Therefore, I propose a targeted simulation test to verify the correspondence.
The referee's report
The paper makes a significant contribution to the understanding of reward hacking in multi-reward RL for LLMs by identifying aggregation itself, rather than individual proxy misspecification, as a root cause of failure. The theoretical analysis in a simplified MDP is rigorous and provides clear mechanistic insight. The proposed TAG method is principled, addresses the identified failure, and is demonstrated effective across a wide range of models and tasks. The empirical evaluation is extensive, covering multiple baselines, model sizes, and two distinct tasks with both compatible and incompatible auxiliary rewards. If the central claims hold, this work has strong practical implications for the design of reward aggregation in LLM post-training. A key limitation is the gap between the deterministic conditional-mean analysis and the actual stochastic training algorithm; the experiments are consistent with the theory, and the limitation is acknowledged.
Pith's reading of the paper (a machine's summary, not the authors')
The paper establishes that combining multiple reward signals symmetrically (treating all channels equally) causes the aggregate reward to improve even as the primary-task performance degrades, a failure the authors term multi-reward hacking. They prove that under deterministic conditional-mean GRPO, a symmetric aggregation rule drives the policy to converge to an auxiliary-favoring action whenever the auxiliary reward is more reliably obtained, regardless of initialization. To remedy this, they propose Trust-Anchored Gating (TAG), which first elects a trusted anchor channel by measuring policy drift under isolated optimization, then conditions auxiliary credit on anchor satisfaction. TAG provably reverses the selection in the same analytical model, converging to the primary-favoring action. Experiments on Qwen and Llama models across mathematics and program synthesis show that symmetric, normalized, and constraint-based aggregations all suffer severe correctness degradation, while TAG substantially mitigates it while retaining auxiliary benefits.
What carries the argument. The analysis uses a two-channel MDP with binary rewards (0/1), three actions (a0: null; a1: auxiliary-favoring, earns auxiliary reward with certainty and primary reward with probability η; a2: primary-favoring, earns primary reward with probability η + (1-η)ρ and auxiliary reward with probability η). The central mechanism is the conditional-mean GRPO advantage, which is non-decreasing in the own scalar reward. Theorem 1's proof relies on comparing expected advantages through a shared joint-success coupling: because a1 receives the higher f(0,1) more reliably than a2 receives f(1,0), the advantage gap m1 - m2 is strictly positive at every interior state, driving log-odds towards a1. TAG reverses this because f_TAG maps auxiliary-only outcomes to the same zero as null outcomes, making the primary action's expected advantage always strictly positive whenever primary success is possible (ρ > 0). The anchor election uses channel-isolated probe policies and picks the one with smallest per-token reverse KL divergence from the base policy, which is sensitive to mode invention.
Core claim. The central claim is that symmetric reward aggregation systematically selects auxiliary-favoring behavior over the primary objective when the auxiliary reward is easier to obtain, even when both channels are individually well-specified. From a two-channel binary-reward MDP with compatibility η and primary success rate ρ, the paper proves that any symmetric, monotone aggregation f satisfying f(0,0) < f(0,1) ≤ f(1,1) and η,ρ < 1 causes deterministic conditional-mean GRPO to converge to the auxiliary-favoring action from any initialization (Theorem 1). The mechanism is a shared-component coupling showing that the auxiliary action's expected advantage dominates that of the primary action at every state. Trust-Anchored Gating (TAG) breaks the symmetry by gating auxiliary credit on anchor satisfaction: with aggregation f_TAG(R_P,R_A) = R_P (1 + λ R_A)/(1 + λ), auxiliary-only success gets zero credit, and the policy converges to the primary-favoring action for any λ > 0 when the primary channel is the anchor (Theorem 2). Empirical results across Qwen-1.7B/4B, Llama-3.2-3B/8B, and Qwen3-30B-A3B confirm the collapse and the defense.
The premise it rests on. The theoretical analysis (Theorems 1 and 2) assumes a two-channel MDP with binary rewards (0/1), three actions with fixed outcome distributions parameterized by compatibility η and primary success rate ρ, and the deterministic conditional-mean GRPO recursion; if these simplifications are violated (e.g., continuous rewards, correlated samples, more complex action spaces, or stochastic optimization), the convergence guarantees may not hold or may require additional conditions.
If the paper is right (Pith's inferences, not the authors')
Any aggregation rule that treats all reward channels equally (additive, normalized, constrained) can undermine the primary objective in LLM post-training, even when each individual reward signal is meaningful.
Practitioners using auxiliary rewards such as brevity, efficiency, or style must adopt an asymmetric aggregation structure to prevent the policy from collapsing onto easier auxiliary behaviors.
TAG provides a practical, theoretically grounded defense: it eliminates credit for auxiliary-only outcomes while preserving incentives for joint success, requiring no careful calibration of reward weights.
The anchor election method - measuring isolated policy drift via reverse KL - can identify which reward channel is least exploitable, offering a general tool for reward auditing before joint training.
Multi-reward hacking can occur even with compatible rewards (where correct and auxiliary-satisfying responses exist), because the auxiliary action's advantage dominates at every state.
The paper's MDP model (binary rewards, three actions, fixed η,ρ) abstracts away crucial features of real LLM training, including continuous rewards, correlated responses within a group, and stochastic optimization; the theoretical guarantees may weaken or change qualitatively in those regimes, though the empirical results across multiple models suggest robustness.
The anchor election procedure (isolated probe runs) introduces extra computational cost and assumes at least one channel admits no low-base-probability shortcut; scaling to many reward channels or dynamic anchor selection during training could face combinatorial or stability issues.
The definition of multi-reward hacking (Definition 2) requires the proxy optimum to strictly degrade true reward; in practice, even a non-strict degradation (plateau with lost potential) could be problematic, and the paper's characterization of hacking conditions only covers two specific true reward forms (primary-only and weighted sum).
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
To improve task performance while encouraging desirable response properties, LLM post-training often combines a primary reward (e.g., correctness) with auxiliary rewards (e.g., brevity or efficiency). However, how these rewards are aggregated can critically affect training outcomes. We show that symmetric aggregation, which treats reward channels equally, can induce a failure where aggregate reward improves while primary-task performance degrades, even when individual rewards are meaningful. We investigate this aggregation-induced failure through a one-step multi-reward decision formulation, characterizing when it constitutes multi-reward hacking. We prove that, under symmetric aggregation, deterministic conditional-mean GRPO converges to auxiliary-favoring behavior when auxiliary rewards are more reliably obtained, regardless of initialization. We propose Trust-Anchored Gating (TAG), which first elects an anchor channel based on the policy drift induced by optimizing each channel alone, then gates auxiliary credit on anchor satisfaction. Unlike calibration-sensitive static reweighting, TAG structurally eliminates credit for auxiliary-only outcomes while preserving additional credit for joint success. In the same analytical setting, TAG provably reverses this selection, converging to the primary-favoring action from any initialization when the primary channel is anchored. Experiments across Qwen and Llama models on mathematics and program synthesis reveal substantial degradation under symmetric, constrained, and normalized aggregation, whereas TAG mitigates this failure. On Qwen3-4B with a brevity reward, TAG achieves 0.581 pass@4 versus 0.220 for additive aggregation, while shortening responses by 44% relative to correctness-only training.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-16
The authors' main claim, as the classifier picked it out 0.81
“Experiments across Qwen and Llama models on mathematics and program synthesis reveal substantial degradation under symmetric, constrained, and normalized aggregation, whereas TAG mitigates this failure.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.16 · human participants 0.03 · controlled comparison 0.90 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.94 · mentions code or data availability 0.02
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| To improve task performance while encouraging desirable response properties, LLM post-training often combines a primary reward (e.g., correctness) with auxiliary rewards (e.g., brevity or efficiency). | background 1.00 | 0.05 | 0.02 | asserted 0.98 |
| However, how these rewards are aggregated can critically affect training outcomes. | background 0.61 | 0.07 | 0.04 | asserted 0.73 |
| We show that symmetric aggregation, which treats reward channels equally, can induce a failure where aggregate reward improves while primary-task performance degrades, even when individual rewards are meaningful. | result 0.96 | 0.94 | 0.31 | asserted 0.99 |
| We investigate this aggregation-induced failure through a one-step multi-reward decision formulation, characterizing when it constitutes multi-reward hacking. | contribution 0.78 | 0.93 | 0.06 | asserted 0.90 |
| We prove that, under symmetric aggregation, deterministic conditional-mean GRPO converges to auxiliary-favoring behavior when auxiliary rewards are more reliably obtained, regardless of initialization. | result 0.92 | 0.94 | 0.12 | asserted 1.00 |
| We propose Trust-Anchored Gating (TAG), which first elects an anchor channel based on the policy drift induced by optimizing each channel alone, then gates auxiliary credit on anchor satisfaction. | method 0.83 | 0.91 | 0.19 | asserted 0.99 |
| Unlike calibration-sensitive static reweighting, TAG structurally eliminates credit for auxiliary-only outcomes while preserving additional credit for joint success. | method 0.80 | 0.83 | 0.19 | asserted 1.00 |
| In the same analytical setting, TAG provably reverses this selection, converging to the primary-favoring action from any initialization when the primary channel is anchored. | result 0.95 | 0.94 | 0.26 | asserted 1.00 |
| Experiments across Qwen and Llama models on mathematics and program synthesis reveal substantial degradation under symmetric, constrained, and normalized aggregation, whereas TAG mitigates this failure. | result 0.99 | 0.96 | 0.27 | asserted 1.00 |
| On Qwen3-4B with a brevity reward, TAG achieves 0.581 pass@4 versus 0.220 for additive aggregation, while shortening responses by 44% relative to correctness-only training. | result 1.00 | 0.96 | 0.17 | asserted 1.00 |
What kind of paper the classifier reports this is: method or tool 0.95 · evidence from proof or derivation 0.50
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-16: 1 request, 7,671 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (12)
Figures are the authors', reproduced from the paper.
References (26)
- [1] Alon Albalak, Duy Phung, Nathan Lile, Rafael Rafailov, Kanishk Gandhi, Louis Castricato, Anikait Singh, Chase Blagden, Violet Xiang, Dakota Mahan, and Nick Haber. Big-Math: A large- 11 Preprint scale, high-quality math dataset for reinforcement learning in language models.arXi... on PithPith's reviewarXiv
- [4] Mohammad Beigi, Ming Jin, Junshan Zhang, Qifan Wang, and Lifu Huang. Adversarial reward auditing for active detection and mitigation of reward hacking.arXiv preprint arXiv:2602.01750, on PithPith's reviewarXiv
- [5] Lichang Chen, Chen Zhu, Davit Soselia, Jiuhai Chen, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. ODIN: Disentangled reward mitigates hacking in RLHF.arXiv preprint arXiv:2402.07319, on PithPith's reviewarXiv
- [8] Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Yunzhong He, Bing Liu, and Sean Hendryx. Rubrics as rewards: Reinforcement learning beyond verifiable domains.arXiv preprint arXiv:2507.17746, on PithPith's reviewarXiv
- [9] Daya Guo, Dejian Yang, Haowei Zhang, et al. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning.arXiv preprint arXiv:2501.12948, on PithPith's reviewarXiv
- [10] doi: 10.1162/tacl.a.782. Jacek Karwowski, Oliver Hayman, Xingjian Bai, Klaus Kiendlhofer, Charlie Griffin, and Joar Skalse. Goodhart’s law in reinforcement learning. InInternational Conference on Learning Representations, on Pithdoi:10.1162/tacl.a.782
- [11] doi: 10.1162/tacl.a.696. Michael Luo, Sijun Tan, Roy Huang, Ameen Patel, Alpay Ariyak, Qingyang Wu, Xiaoxiang Shi, Rachel Xin, Colin Cai, Maurice Weber, Ce Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica. DeepCoder: A fully open-source 14B coder at O3-mini level.https://ww... on Pithdoi:10.1162/tacl.a.696
- [12] Thomas P. Minka. Divergence measures and message passing. Technical Report MSR-TR-2005- 173, Microsoft Research, on Pith
- [15] Joar Skalse, Lewis Hammond, Charlie Griffin, and Alessandro Abate. Lexicographic multi-objective reinforcement learning. InProceedings of the 31st International Joint Conference on Artificial Intelligence, pp. 3430–3436, 2022a. Joar Skalse, Nikolaus Howe, Dmitrii Krasheninniko... on PithPith's reviewarXiv
- [16] Peter Vamplew, John Yearwood, Richard Dazeley, and Adam Berry. On the limitations of scalarisa- tion for multi-objective reinforcement learning of Pareto fronts. InAI 2008: Advances in Artificial Intelligence, pp. 372–378. Springer, on Pith
- [18] Zihao Wang, Chirag Nagpal, Jonathan Berant, Jacob Eisenstein, Alex D’Amour, Sanmi Koyejo, and Victor Veitch. Transforming and combining rewards for aligning large language models. In International Conference on Machine Learning (ICML), 2024b. Tengyu Xu, Eryk Helenowski, Karthi... on PithPith's reviewarXiv
- [19] An Yang et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, on PithPith's reviewarXiv
- [20] Minglai Yang et al. Rubric dropout: A simple way to mitigate reward hacking in rubric-as-reward RL.arXiv preprint arXiv:2608.11669, on PithPith's reviewarXiv
- [21] Zhanhui Zhou, Jie Liu, Jing Shao, Xiangyu Yue, Chao Yang, Wanli Ouyang, and Yu Qiao. Beyond one-preference-fits-all alignment: Multi-objective direct preference optimization. InFindings of the Association for Computational Linguistics: ACL 2024, on Pith
- [22] the realized rewards are then0,w 2 andw 1 for neither, auxiliary-only and primary-only, so a sampled group can contain three reward levels and the two-level identity (11) no longer applies; we therefore evaluate the conditional-mean GRPO recursion by exact enumeration (n= 8,α=... on Pith
- [23] Direction of the log-ratio.The quantityKL revT,k of Section 5.1 evaluates the log-ratio under re- sponses generated by the probed policyπ T,k . The behavior of interest is the one created by isolated optimization, so the divergence is evaluated under the policy that produces i... on Pith
- [24] and are kept as this check; the row is omitted from Table 1.Lagrangian(Moskovitz et al., 2024):R P +ν tRA, the multiplier updated once per rollout batch by dual ascent on the constraint E[RA]≥τ,ν t+1 = max{0, νt +α dual(τ− ¯RA)}withτ= 0.5,α dual = 0.05,ν 0 = 0.5; once the auxi... on Pith
- [25] on(R A +R P)/2:A i =r i −¯r without the group-standard-deviation division, and the per-sequence1/|o i|loss normalizer replaced by the constant1/L max, implemented by scalingA i by|o i|/Lmax.Rubric Dropout(Yang et al., 2026): with probability0.5per group the auxiliary channel i... on Pith
- [26] 0.550 0.282 0.332 0.386 0.118 0.164 2 0.620 0.256 0.294 0.451 0.087 0.169 4 0.612 0.254 0.298 0.447 0.089 0.165 Table 9:λsensitivity of TAG(Qwen3-1.7B, Correct & brief, cap 512, end of training). Held-out short-response rateR A and accuracyR P, the four outcome cells (first di... on Pith
- [2008] Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. InFindings of the Association for Computational Linguistics: EMNLP 2024, pp. 10582–10592, Miami, Florida, USA, 2024a. As- soc... on Pithdoi:10.18653/v1/2024.findings-emnlp.620
show 6 more
- [2013] 13 Preprint Zhihong Shao et al. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, on PithPith's reviewarXiv
- [2020] doi: 10.1038/s42256-020-00257-z. Aaron Grattafiori et al. The Llama 3 herd of models.arXiv preprint arXiv:2407.21783, on PithPith's reviewarXiv
- [2022] Alexandre Ram ´e, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: Towards Pareto-optimal alignment by inter- polating weights fine-tuned on diverse rewards. InAdvances in Neural Information Processing... on Pith
- [2024] Wenlong Deng, Jiaji Huang, Kaan Ozkara, Yushu Li, Christos Thrampoulidis, Xiaoxiao Li, and Youngsuk Park. Directional alignment mitigates reward hacking in reinforcement learning for language models.arXiv preprint arXiv:2605.25189, on PithPith's reviewarXiv
- [2025] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Man´e. Con- crete problems in AI safety.arXiv preprint arXiv:1606.06565, on PithPith's reviewarXiv
- [2026] Bowen Baker, Joost Huizinga, Leo Gao, Zehao Dou, Melody Y . Guan, Aleksander Madry, Wojciech Zaremba, Jakub Pachocki, and David Farhi. Monitoring reasoning models for misbehavior and the risks of promoting obfuscation.arXiv preprint arXiv:2503.11926, on PithPith's reviewarXiv
Cited by 1 Pith paper
Reviewed papers on Pith that cite this one. The full citation record
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment