Pith. sign in

REVIEW 4 major objections 5 minor 57 references

PRISM claims the optimal multi-reward LLM policy is a weighted product of per-reward positive policies and one global negative policy, so preferences can be mixed at inference time as a logit sum instead of being scalarized during training.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 10:50 UTC pith:FZA44P5L

load-bearing objection PRISM is a clever policy-space alternative to reward mixing with solid empirical gains, but its core logit-composition step is an unproven approximation and the paper overstates how consistent the wins are. the 4 major comments →

arxiv 2607.29246 v1 pith:FZA44P5L submitted 2026-07-31 cs.AI

Don't Mix Rewards, Mix Policies: Policy Decomposition and Optimization for Multi-Reward RL

Ruiming Liang , Yi Zhong , Yizhen Yuan , Yinan Zheng , Tianyi Tan , Tianyue Wang , Haiyun Guo , Jinqiao Wang
show 1 more author
Xianyuan Zhan
This is my paper
classification cs.AI
keywords multi-reward reinforcement learningpolicy decompositionlogit-level policy compositionpositive-negative policy optimizationalignment taxLLM alignmentinference-time controllabilityprefix-conditioned policies
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to change where preferences are combined in multi-reward RL for LLMs: instead of scalarizing rewards before the policy update, it composes policies after training. Its central claim is that the optimal policy for a weighted multi-reward objective factorizes into a product of individually trained positive policies, one per reward, and a single global negative policy that captures the union of failure modes; at decoding this factorized optimum is realized as a weighted sum of token logits. This lets one training run produce an entire family of trade-off policies, with the merge weights acting as an interpretable inference-time control, and it avoids the 'multi-reward alignment tax' that comes from letting conflicting rewards compete in the same gradient step. On scientific QA, tool-use reasoning, and helpfulness-safety benchmarks, PRISM reports higher joint scores than reward-scalarization baselines and keeps correctness stable as more rewards are added, converging in roughly half the steps.

Core claim

The central claim is the decomposition identity π⋆(o|q) ∝ ∏_{k=1}^N (π+_k(o|q))^{α_k} (π−(o|q))^γ, where each π+_k is the single-reward optimal policy for reward R_k relative to a reference, and π− is the optimal policy for a penalty that fires whenever any reward is low. The paper derives this from a KL-regularized multi-reward objective and then implements it as logit-level mixing, z*_t = Σ α_k z+_{k,t} − γ z−_t, with all sub-policies sharing one backbone through learnable prefixes. Training is asymmetric: positive branches update the backbone, the negative branch updates only its prefix, and the composed policy is used both for rollouts and at serving time. The paper argues this decouplin

What carries the argument

The carrying mechanism is a product-of-policies composition in logit space. A shared autoregressive model is split into N+1 branches by learnable prefix embeddings; branch k is trained on reward R_k, and the negative branch is trained so that its advantage is large exactly on the union of reward-specific failure modes. At each decoding step the N+1 branch logits are computed in one batched forward pass and combined as z*_t = Σ α_k z+_{k,t} − γ z−_t; the single sampled token is broadcast to all branches. This makes the composition both the training rollout policy and the serving policy, and makes the weights α_k, γ an explicit preference interface.

Load-bearing premise

The argument rests on treating the weighted per-token logit sum as an exact realization of the sequence-level product-of-policies optimum, and on using the current composed policy as the reference for training each sub-policy; the first is an unproved approximation and the second has no convergence or fixed-point analysis.

What would settle it

On a small task with two rewards, train the positive and negative branches, then compute the exact sequence-level product policy (Eq. 11) and the distribution induced by logit mixing (Eq. 14). If their KL divergence is material, or if the logit-mixture policy achieves lower joint reward than the product policy on held-out prompts, the central identity is refuted. A cheaper check: compare the per-token conditional of the product against the normalized product of the per-token conditionals.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A single training run yields a whole family of policies indexed by the merge weights, so preference trade-offs can be adjusted at inference time without retraining.
  • Per-reward optimization avoids the alignment tax of scalarized updates: reported correctness stays nearly flat as format and length rewards are added, while reward-space baselines drop.
  • The global negative policy concentrates all failure modes in one branch, so suppressing generic bad behavior does not need to be attributed to any particular reward.
  • Composed-policy rollouts are both training and serving distribution, so the method needs no separate distillation or reward-model phase to deploy the mixed policy.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The token-level logit sum is an approximation of the sequence-level product identity: for autoregressive models the conditional of a product distribution is not generally the normalized product of conditionals, so the theoretical guarantee is weaker than Eq. 11 implies. A direct comparison of the two sampling rules on held-out prefixes would settle how much the approximation costs.
  • Because the backbone is updated only through positive branches, the negative policy is expressed entirely in its prefix; safety behaviors that require new backbone knowledge may be poorly captured, suggesting a scaling limit as failure modes grow more complex.
  • Shared positive branches still collide in the backbone, so 'no gradient interference' is only partially realized; an ablation with fully independent sub-policy backbones would quantify the remaining interference.
  • Editorial note: §3.1 refers to 'Appendix??' for negative-advantage ablations; the corresponding ablation is present in Table 5, so the content exists but the reference is unresolved.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. PRISM is a multi-reward RL method that trains N reward-specific positive policies and one global negative policy using a shared LLM with per-branch prefixes; at inference it composes them via a weighted sum of token logits. The paper derives a product-of-experts optimal policy for a KL-regularized, scalarized objective (Eqs. 6–11) and claims that token-level logit mixing (Eq. 14) implements this policy exactly. Experiments on scientific QA, tool-use reasoning, and helpfulness–safety alignment report consistent gains over GRPO Sum/Product and GDPO baselines, plus a demonstration of inference-time controllability.

Significance. If the theoretical derivation were complete, PRISM would be a practically valuable contribution: it offers an efficient shared-backbone implementation, inference-time trade-off control without retraining, and a principled alternative to reward-space scalarization. The empirical evaluation is broad (three backbones, three task families), and the ablations in Tables 4–5 and Figure 5 provide useful evidence that the individual components matter. However, the central bridge from the sequence-level product-of-experts policy to the implemented token-level logit mixture is not proven, and the iterative reference-policy update is not analyzed. Until these gaps are addressed, the paper's exact-optimality claims are not established.

major comments (4)
  1. [§3.1–3.2, Eq. (11)/(14)] Eq. (11) is a product identity over full responses. For autoregressive generation, the conditional distribution of that product is π*(x|h) ∝ [∏_k π+_k(x|h)^{α_k} π^-(x|h)^{-γ}] · C(h,x), where h=(q,o_<t) and C(h,x)=∫_{suffix} ∏_k π+_k(suffix|h,x)^{α_k} π^-(suffix|h,x)^{-γ} dsuffix. Unless C(h,x) is independent of x, the logit-level composition in Eq. (14) is not the exact conditional of π*. The paper provides no bound, approximation argument, or experiment quantifying this future-normalizer effect. Because Eq. (14) is the implemented sampler and the basis for the inference-time controllability claims, this is a load-bearing gap.
  2. [§3.1, final paragraph] The derivation of Eqs. (9)–(11) treats μ as a fixed reference policy. The final paragraph of §3.1 states that training proceeds iteratively with the current composed policy π* serving as μ. This makes the sub-policy targets shift as the composition changes, so the optimality of Eq. (11) and the validity of the importance ratios in Eqs. (12)–(13) are no longer guaranteed. No convergence or fixed-point analysis is supplied. The authors should either provide such an analysis or explicitly characterize the iterative procedure as a heuristic and test its sensitivity to the reference-update schedule.
  3. [§3.1, Eq. (11)] As printed, Eq. (11) places (π^-)^γ in the numerator. This contradicts Eq. (8), where the negative term appears in the denominator, and Eq. (14), where the negative logits are subtracted. The exponent must be -γ, not +γ. Please correct the display and check that all downstream equations and the discussion are sign-consistent.
  4. [§3.1, Eq. (6)] Eq. (6) is itself a linear scalarization of the reward functions R_k plus a ψ penalty, with weights α_k and -γ. The optimal policy is therefore the same as that of a scalarized-reward RL problem. The 'mix policies, not rewards' distinction is operational — how sub-policies are trained and combined — rather than a difference in the underlying optimization objective. The text should state this explicitly and position the contribution relative to reward-space scalarization (e.g., GDPO) accordingly.
minor comments (5)
  1. [§3.1] The phrase 'alternatives are ablated in Appendix??' contains a missing appendix reference. Please fill in the correct section.
  2. [Table 1] Numeric cells in Table 1 are typeset with values and deltas run together (e.g., '69.91+3.2768.51+68.51'), making the table difficult to read. Please add proper spacing or separators.
  3. [§4.3, Figure 3] The normalization procedure for Figure 3 is described only as 'normalized using the corresponding rewards of PRISM under three-reward setting.' Please specify the exact formula and the reference values used.
  4. [Appendix B.4.2] Equation numbering jumps from Eq. (17) in the main text to Eq. (18) in Appendix B.4.2; no equations 15–17 appear in that appendix. Please renumber consistently.
  5. [§2.3, Eq. (5)] Eq. (5) uses a single coefficient α for the single-reward product-of-experts form, but Section 3 uses per-reward coefficients α_k and a separate γ. The relation between the two notations should be clarified.

Circularity Check

0 steps flagged

No substantive circularity; core derivation is self-contained, with a redundant self-citation and non-circular approximation gaps.

full rationale

The claimed derivation from the KL-regularized multi-reward objective (Eq. 6) to the product-of-policies form (Eq. 11) is arithmetic: decompose μ using ∑α_k−γ=1 and regroup exponentials. The resulting composed policy is, by construction, the optimal policy for a linear scalarization plus the ψ penalty; this is an identity, not an empirical prediction built from fitted parameters. No parameter is fit to the evaluation benchmarks, so there is no fitted-input-called-prediction. The DIPOLE self-citation in §2.3 is redundant—DExperts [20] and the re-derivation in §3.1 carry the weight—hence at most a minor self-citation. The substantive gaps in the paper are non-circular: Eq. 14 replaces a sequence-level product-of-experts by per-token logit summation, dropping a token-dependent future normalizer, and the iterative choice μ=π* makes the sub-policy targets self-referential without a fixed-point proof. These are approximation/convergence concerns, not reductions of the output to the input. The sign inconsistency in Eq. 11 relative to Eq. 8 is an internal algebra error, not circularity. Therefore the central derivation is not circular, and only a minor redundant self-citation prevents a score of 0.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

PRISM rests on standard RLHF optimal-policy results, a heuristic failure-mode penalty ψ, an unproved token-level composition approximation, and an iterative training loop whose reference policy is the composed policy itself. The central claim therefore depends on several domain assumptions and ad hoc choices beyond standard math.

free parameters (4)
  • merge weights α_k, γ = [0.55, 0.25, 0.25] with γ=0.05 in Figure 5; training-time values not reported
    Hand-chosen composition weights define the trade-off; central to Eq. 6 and Eq. 14.
  • advantage offset c
    Appears in \tilde A^+_k = σ(\hat A^k_i)−c and \tilde A^-_i = c−(∏σ)^{1/N}; value never stated.
  • KL coefficient η = 0.001 in helpfulness-safety (Table 8); unstated for science QA and tool-use
    Controls strength of reference-policy regularization in Eq. 6-10; omitted in two of three experiment settings.
  • prefix embedding length
    Learnable prefixes are introduced in Eq. 15 and initialized in Appendix D, but their token length is never specified.
axioms (5)
  • standard math Optimal KL-regularized policy has form π⋆ ∝ μ exp(ηR)
    Invoked at Eq. 9-10 to define sub-policies; standard RLHF result (Ziegler et al. 2019, DPO 2023).
  • ad hoc to paper A single global negative policy trained on soft-conjunction advantage captures the union of reward-specific failure modes
    Eq. 13 and surrounding text define ψ via \tilde A^-_i = c − (∏ σ(\hat A^k_i))^{1/N}; no proof that this is the right ψ, and the planned ablation lives in a missing 'Appendix??'.
  • ad hoc to paper Token-level logit composition equals sequence-level product-of-experts composition
    Eq. 14 is presented as a direct consequence of Eq. 11; not generally valid for autoregressive conditionals without additional assumptions.
  • domain assumption Prefix-conditioned shared backbone instantiates independent sub-policies anchored on a common reference μ
    Eq. 15 shares θ across branches; branch outputs are not independent policies from the same μ, so the theoretical decomposition is approximate.
  • domain assumption Iterative training with the current composed policy as reference μ converges and preserves the decomposition
    Final paragraph of §3.1; no fixed-point or convergence argument is supplied.

pith-pipeline@v1.3.0-daily-deepseek · 16791 in / 17390 out tokens · 155877 ms · 2026-08-03T10:50:46.093441+00:00 · methodology

0 comments
read the original abstract

Modern large language models (LLMs) are expected not just to answer correctly, but to adapt their behavior to different human values and use cases. As a result, multi-reward reinforcement learning (RL) has become an increasingly important problem for LLMs, where each reward captures a different aspect of desired behavior. However, optimizing with multiple rewards suffers from a more severe alignment tax issue, where different optimization objectives can trade off or even conflict with each other, leading to unstable and inefficient post-training. In this work, we propose PRISM, a new multi-reward RL framework built upon the idea of policy-space decomposition and composition. Instead of compositing different rewards, PRISM optimizes a set of standalone positive policies and a global negative policy. This alleviates the potential conflict during multi-reward policy optimization, while enabling controllability during inference by flexible policy composition. Experiments on scientific reasoning, tool-use reasoning, and helpfulness-safety alignment show that PRISM consistently outperforms existing multi-reward RL baselines, with extra controllability for inference-time preference control.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

57 extracted references · 20 linked inside Pith

  1. [1]

    Amo.https://github.com/Artessay/Amo, 2025

    2025

  2. [2]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  3. [3]

    Constrained policy optimization

    Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. InInternational conference on machine learning, pages 22–31. Pmlr, 2017

    2017

  4. [4]

    A general language assistant as a laboratory for alignment.arXiv preprint arXiv:2112.00861, 2021

    Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment.arXiv preprint arXiv:2112.00861, 2021

    Pith/arXiv arXiv 2021

  5. [5]

    Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

    Pith/arXiv arXiv 2022

  6. [6]

    Safe rlhf: Safe reinforcement learning from human feedback

    Juntao Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback. InInternational Conference on Learning Representations, volume 2024, pages 50750–50777, 2024

    2024

  7. [7]

    Dynamic multi-reward weighting for multi-style controllable generation

    Karin De Langis, Ryan Koo, and Dongyeop Kang. Dynamic multi-reward weighting for multi-style controllable generation. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6783–6800, 2024

    2024

  8. [8]

    Controlled text generation via language model arithmetic.arXiv preprint arXiv:2311.14479, 2023

    Jasper Dekoninck, Marc Fischer, Luca Beurer-Kellner, and Martin Vechev. Controlled text generation via language model arithmetic.arXiv preprint arXiv:2311.14479, 2023

    Pith/arXiv arXiv 2023

  9. [9]

    Sciknoweval: Evaluating multi-level scientific knowledge of large language models.arXiv preprint arXiv:2406.09098, 2024

    Kehua Feng, Xinyi Shen, Weijie Wang, Xiang Zhuang, Yuqi Tang, Qiang Zhang, and Keyan Ding. Sciknoweval: Evaluating multi-level scientific knowledge of large language models.arXiv preprint arXiv:2406.09098, 2024

    arXiv 2024

  10. [10]

    Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.arXiv preprint arXiv:2209.07858, 2022

    Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.arXiv preprint arXiv:2209.07858, 2022

    Pith/arXiv arXiv 2022

  11. [11]

    Training products of experts by minimizing contrastive divergence.Neural computation, 14(8): 1771–1800, 2002

    Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence.Neural computation, 14(8): 1771–1800, 2002

    2002

  12. [12]

    Personalized soups: Personalized large language model alignment via post-hoc parameter merging.arXiv preprint arXiv:2310.11564, 2023

    Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized soups: Personalized large language model alignment via post-hoc parameter merging.arXiv preprint arXiv:2310.11564, 2023

    Pith/arXiv arXiv 2023

  13. [13]

    Beavertails: Towards improved safety alignment of llm via a human-preference dataset

    Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. Advances in Neural Information Processing Systems, 36:24678–24704, 2023. 11

    2023

  14. [14]

    The power of scale for parameter-efficient prompt tuning

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 conference on empirical methods in natural language processing, pages 3045–3059, 2021

    2021

  15. [15]

    Gradient-adaptive policy optimization: Towards multi-objective alignment of large language models

    Chengao Li, Hanyu Zhang, Yunkun Xu, Hongyan Xue, Xiang Ao, and Qing He. Gradient-adaptive policy optimization: Towards multi-objective alignment of large language models. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11214–11232, 2025

    2025

  16. [16]

    Prefix-tuning: Optimizing continuous prompts for generation

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582–4597, 2021

    2021

  17. [17]

    Dichotomous diffusion policy optimization.arXiv preprint arXiv:2601.00898, 2025

    Ruiming Liang, Yinan Zheng, Kexin Zheng, Tianyi Tan, Jianxiong Li, Liyuan Mao, Zhihao Wang, Guang Chen, Hangjun Ye, Jingjing Liu, et al. Dichotomous diffusion policy optimization.arXiv preprint arXiv:2601.00898, 2025

    Pith/arXiv arXiv 2025

  18. [18]

    Mitigating the alignment tax of rlhf

    Yong Lin, Hangyu Lin, Wei Xiong, Shizhe Diao, Jianmeng Liu, Jipeng Zhang, Rui Pan, Haoxiang Wang, Wenbin Hu, Hanning Zhang, et al. Mitigating the alignment tax of rlhf. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 580–606, 2024

    2024

  19. [19]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Pith/arXiv arXiv 2024

  20. [20]

    Dexperts: Decoding-time controlled text generation with experts and anti-experts

    Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A Smith, and Yejin Choi. Dexperts: Decoding-time controlled text generation with experts and anti-experts. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol...

    2021

  21. [21]

    Gdpo: Group reward-decoupled normalization policy optimization for multi-reward rl optimization, 2026

    Shih-Yang Liu, Xin Dong, Ximing Lu, Shizhe Diao, Peter Belcak, Mingjie Liu, Min-Hung Chen, Hongxu Yin, Yu-Chiang Frank Wang, Kwang-Ting Cheng, Yejin Choi, Jan Kautz, and Pavlo Molchanov. Gdpo: Group reward-decoupled normalization policy optimization for multi-reward rl optimization, 2026. URLhttps://arxiv. org/abs/2601.05242

    Pith/arXiv arXiv 2026

  22. [22]

    Contrastive decoding improves reasoning in large language models.arXiv preprint arXiv:2309.09117, 2023

    Sean O’Brien and Mike Lewis. Contrastive decoding improves reasoning in large language models.arXiv preprint arXiv:2309.09117, 2023

    Pith/arXiv arXiv 2023

  23. [23]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022

    2022

  24. [24]

    Patil, Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E

    Shishir G. Patil, Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models. In Forty-second International Conference on Machine Learning, 2025

    2025

  25. [25]

    Efficiently scaling transformer inference.arXiv preprint arXiv:2211.05102, 2022

    Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference.arXiv preprint arXiv:2211.05102, 2022

    Pith/arXiv arXiv 2022

  26. [26]

    Toolrl: Reward is all tool learning needs.Advances in Neural Information Processing Systems, 38:105523–105553, 2026

    Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tur, Gokhan Tur, and Heng Ji. Toolrl: Reward is all tool learning needs.Advances in Neural Information Processing Systems, 38:105523–105553, 2026

    2026

  27. [27]

    Dmoerm: Recipes of mixture-of-experts for effective reward modeling

    Shanghaoran Quan. Dmoerm: Recipes of mixture-of-experts for effective reward modeling. InFindings of the Association for Computational Linguistics: ACL 2024, pages 7006–7028, 2024

    2024

  28. [28]

    Direct preference optimization: Your language model is secretly a reward model.Advances in neural information processing systems, 36:53728–53741, 2023

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model.Advances in neural information processing systems, 36:53728–53741, 2023

    2023

  29. [29]

    Warm: On the benefits of weight averaged reward models.arXiv preprint arXiv:2401.12187, 2024

    Alexandre Ramé, Nino Vieillard, Léonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. Warm: On the benefits of weight averaged reward models.arXiv preprint arXiv:2401.12187, 2024

    Pith/arXiv arXiv 2024

  30. [30]

    Gpqa: A graduate-level google-proof q&a benchmark.arXiv preprint arXiv:2311.12022, 2023

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark.arXiv preprint arXiv:2311.12022, 2023

    Pith/arXiv arXiv 2023

  31. [31]

    A survey of multi-objective sequential decision-making.Journal of Artificial Intelligence Research, 48:67–113, 2013

    Diederik M Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley. A survey of multi-objective sequential decision-making.Journal of Artificial Intelligence Research, 48:67–113, 2013. 12

    2013

  32. [32]

    Scienceqa: A novel resource for question answering on scholarly articles.International Journal on Digital Libraries, 23(3):289–301, 2022

    Tanik Saikh, Tirthankar Ghosal, Amish Mittal, Asif Ekbal, and Pushpak Bhattacharyya. Scienceqa: A novel resource for question answering on scholarly articles.International Journal on Digital Libraries, 23(3):289–301, 2022

    2022

  33. [33]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  34. [34]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  35. [35]

    MIT press Cambridge, 1998

    Richard S Sutton, Andrew G Barto, et al.Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998

    1998

  36. [36]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model.https://github.com/tatsu-lab/ stanford_alpaca, 2023

    2023

  37. [37]

    Qwen2.5: A party of foundation models, September 2024

    Qwen Team. Qwen2.5: A party of foundation models, September 2024. URLhttps://qwenlm.github.io/blog/ qwen2.5/

    2024

  38. [38]

    Interpretable preferences via multi- objective reward modeling and mixture-of-experts

    Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi- objective reward modeling and mixture-of-experts. Association for Computational Linguistics, 2024

    2024

  39. [39]

    Aligning large language models with human: A survey.arXiv preprint arXiv:2307.12966, 2023

    Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. Aligning large language models with human: A survey.arXiv preprint arXiv:2307.12966, 2023

    Pith/arXiv arXiv 2023

  40. [40]

    Multi-objective reinforcement learning from ai feedback.arXiv preprint arXiv:2406.07295, 2024

    Marcus Williams. Multi-objective reinforcement learning from ai feedback.arXiv preprint arXiv:2406.07295, 2024

    Pith/arXiv arXiv 2024

  41. [41]

    Fine-grained human feedback gives better rewards for language model training.Advances in Neural Information Processing Systems, 36:59008–59033, 2023

    Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training.Advances in Neural Information Processing Systems, 36:59008–59033, 2023

    2023

  42. [42]

    Rewards-in-context: Multi-objective alignment of foundation models with dynamic preference adjustment

    Rui Yang, Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong, Dong Yu, and Jianshu Chen. Rewards-in-context: Multi-objective alignment of foundation models with dynamic preference adjustment. InInternational Conference on Machine Learning, pages 56276–56297. PMLR, 2024

    2024

  43. [43]

    Dapo: An open-source llm reinforcement learning system at scale.Advances in Neural Information Processing Systems, 38:113222–113244, 2026

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale.Advances in Neural Information Processing Systems, 38:113222–113244, 2026

    2026

  44. [44]

    Negative preference optimization: From catastrophic collapse to effective unlearning.arXiv preprint arXiv:2404.05868, 2024

    Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse to effective unlearning.arXiv preprint arXiv:2404.05868, 2024

    Pith/arXiv arXiv 2024

  45. [45]

    Group sequence policy optimization.arXiv preprint arXiv:2507.18071, 2025

    Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization.arXiv preprint arXiv:2507.18071, 2025

    Pith/arXiv arXiv 2025

  46. [46]

    Beyond one-preference- fits-all alignment: Multi-objective direct preference optimization

    Zhanhui Zhou, Jie Liu, Jing Shao, Xiangyu Yue, Chao Yang, Wanli Ouyang, and Yu Qiao. Beyond one-preference- fits-all alignment: Multi-objective direct preference optimization. InFindings of the Association for Computational Linguistics: ACL 2024, pages 10586–10613, 2024

    2024

  47. [47]

    Fine-tuning language models from human preferences.arXiv preprint arXiv:1909.08593, 2019

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences.arXiv preprint arXiv:1909.08593, 2019. 13 A Experimental details of scientific question answering A.1 Dataset and Prompt Format We train on the SciKnowEval dataset, restricting to t...

    Pith/arXiv arXiv 1909

  48. [48]

    The <think> section must contain the complete reasoning process

  49. [49]

    The <answer> section must appear after </think>

  50. [50]

    The <answer> section must contain only one option letter: A, B, C, or D

  51. [51]

    Do not include explanation, extra words, or additional formatting in <answer>

  52. [52]

    A", "B",

    Always complete the reasoning before writing the final answer. The user turn is rendered as: User Prompt Given a question and four options, please select the right answer. Your answer should be "A", "B", "C" or "D". {question} Choices: A. {choice_a} B. {choice_b} C. {choice_c} D. {choice_d} A.2 Reward Functions Training uses two reward signals: correctnes...

  53. [53]

    Provide at least one of<tool_call> or <response>

    You must always include the<think> field to outline your reasoning. Provide at least one of<tool_call> or <response>. Decide whether to use<tool_call>possibly multiple times,<response>, or both

  54. [54]

    name" field and a

    You can invoke multiple tool calls simultaneously in the<tool_call> fields. Each tool call should be a JSON object with a"name" field and a"parameters" field containing a dictionary of parameters. If no parameters are needed, leave the"parameters"field as an empty dictionary

  55. [55]

    Refer to the previous dialogue records in the history, including the user’s queries, previous<tool_call>, <response>, and any tool feedback noted as<obs>if exists. User Prompt for ToolRL Training Dialogue History <user>{{ Initial User Input }}</user> <think>Round 1 Model Thought</think> {{ Round 1 model output<tool_call>or<response>}} <obs>Round 1 Observa...

  56. [56]

    Tool name matching: a multiset overlap score in[0, 1]is computed between the predicted and ground-truth tool names

  57. [57]

    name ":

    Parameter matching: ground-truth calls are processed one by one. For each ground-truth call, every unmatched predicted call with the same tool name is assigned a parameter-level score. This score is the sum of a parameter-name overlap term in[0, 1]and one point for each ground-truth parameter whose value is predicted exactly. The highest parameter-level s...

    2048