pith. sign in

arxiv: 2605.15207 · v1 · pith:IGJCGWICnew · submitted 2026-05-01 · 💻 cs.LG · cs.MA

TeamTR: Trust-Region Fine-Tuning for Multi-Agent LLM Coordination

Yi Xie , Siao Liu , Falong Fan , Yuanqi Yao , Yue Zhao , Bo Liu This is my paper

Pith reviewed 2026-05-19 17:56 UTC · model grok-4.3

classification 💻 cs.LG cs.MA
keywords multi-agent LLMtrust-region optimizationfine-tuningcoordinationoccupancy shiftsequential updatesreinforcement learning
0
0 comments X

The pith

Sequential fine-tuning of multi-agent LLM teams incurs a compounding occupancy shift that scales quadratically with agent count, which TeamTR corrects to linear scaling via trust-region resampling and per-agent divergence control.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper identifies that updating one agent in a shared-context team changes the context distribution seen by the others, so scoring later updates on cached old rollouts creates a mismatch that compounds. The resulting penalty grows quadratically with the number of agents, while resampling fresh trajectories after each update reduces it to linear growth. TeamTR implements this insight by resampling full team trajectories after every component update and adding per-agent limits on policy change inside a trust region, which supplies mathematical lower bounds guaranteeing improvement at each step and each stage. This addresses why current multi-agent LLM systems often underperform single-model baselines despite added resources. If the bounds hold, the method enables stable sequential training without coordination regressions.

Core claim

We identify a structural failure mode in sequential fine-tuning of shared-context teams: updating one agent shifts the team's context distribution, and when subsequent updates are evaluated on cached rollouts, this mismatch compounds. We formalize this as the compounding occupancy shift and prove that stale-occupancy evaluation incurs a penalty that scales quadratically with the number of agents. In contrast, intermediate-occupancy evaluation reduces this to linear scaling. We propose TeamTR, a trust-region framework that resamples trajectories after each component update and enforces per-agent divergence control, yielding rigorous per-update and per-stage improvement lower bounds.

What carries the argument

Compounding occupancy shift, the mismatch between updated agent policies and stale team context distributions, countered by TeamTR's trust-region resampling of full trajectories after each component update plus per-agent divergence control.

If this is right

  • TeamTR supplies rigorous lower bounds on performance improvement after each individual component update and after each full training stage.
  • The method outperforms both single-agent baselines and prior sequential fine-tuning approaches by 7.1 percent on average while reducing coordination regressions.
  • Plug-and-play replacement of any single agent or component becomes feasible without harming overall team performance.
  • Switching to intermediate-occupancy evaluation after each update changes the scaling of the penalty from quadratic to linear in the number of agents.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The linear scaling result could make training feasible for teams larger than those where quadratic penalties become prohibitive.
  • Similar resampling and per-component trust-region ideas might transfer to sequential training in non-LLM multi-agent reinforcement learning settings.
  • Direct measurement of context distribution shifts in deployed multi-agent systems would test whether the formal occupancy shift appears outside the paper's experimental tasks.

Load-bearing premise

That resampling full team trajectories after every component update stays computationally tractable and that per-agent divergence control does not introduce new coordination failures outside the analyzed lower bounds.

What would settle it

A controlled test on teams of 2, 4, 6, and 8 agents that measures whether the performance penalty from stale rollouts grows quadratically while the penalty from fresh intermediate rollouts grows only linearly would directly confirm or refute the central scaling claim.

Figures

Figures reproduced from arXiv: 2605.15207 by Bo Liu, Falong Fan, Siao Liu, Yi Xie, Yuanqi Yao, Yue Zhao.

Figure 1
Figure 1. Figure 1: Update trajectories on the team objective landscape. Left: Joint updates suffer from coupled drift and uncoordinated leaps. Middle: Naive sequential updates with cached rollouts drift away from the target due to stale occupancy. Right: TeamTR resamples under fresh occupancy after each update, reaching the target stably. Inset: Occupancy distributions (top), penalty term scaling (bottom). improves performan… view at source ↗
Figure 2
Figure 2. Figure 2: Stale-occupancy gap within a training stage. We plot the occupancy gap across training stages k for within-stage update indices i = 2 and i = 3. For the baselines-reuse stage, the gap is larger for the later update (i = 3), consistent with Remark 3.3. cost remains rollout generation rather than trust-region mon￾itoring. The turn-taking assumption remains the formal scope of the guarantee. Lemma 3.1 relies … view at source ↗
Figure 3
Figure 3. Figure 3: Training dynamics under matched rollout budgets. We compare TeamTR against sequential baselines instantiated with PPO/GRPO/DAPO. Baselines exhibit occasional regressions during training, whereas TeamTR maintains more stable improvement by evaluating each update on intermediate rollouts and enforcing per-update trust regions. Shaded regions indicate variation across seeds. 5.5. Training Dynamics [PITH_FULL… view at source ↗
Figure 4
Figure 4. Figure 4: Trust-region enforcement and certificate tracking (AIME25). (a) Distribution of per-update token-level KL divergence; the red dashed line marks the threshold δ, and percentages indicate out-of-region rates (D[KLtok > δ). (b) Cumulative measured improvement versus certificate lower bound (Theorem 3.6); ρ denotes rank correlation. (c) Per-stage calibration of certificate values against empirical improvements… view at source ↗
Figure 5
Figure 5. Figure 5: Token-level logit shifts by pre-update probability rank. Left: an in-region update (D[KLtok ≤ δ) produces localized changes on top tokens. Right: an out-of-region update (D[KLtok > δ) reshuffles probability mass among alternatives. close alternatives and can flip the dominant token, consistent with the instabilities observed in Sec. 5.5. 5.8. Plug-and-Play Component Replacement Proposition 3.8 implies that… view at source ↗
Figure 6
Figure 6. Figure 6: Plug-and-play component replacement. A Qwen2.5-Instruct (1.5B/3B/7B) team is trained over 20 stages; and the 1.5B agent is replaced with Qwen3-8B. Stage-0 alignment mitigates the swap shock and achieves the best performance. 6. Conclusion and Limitations We analyzed sequential fine-tuning of shared-context LLM teams and identified compounding occupancy shift: later component updates can be optimized or eva… view at source ↗
read the original abstract

Multi-agent LLM systems have shown promise for complex reasoning, yet recent evaluations reveal they often underperform single-model baselines. We identify a structural failure mode in sequential fine-tuning of shared-context teams: updating one agent shifts the team's context distribution, and when subsequent updates are evaluated on cached rollouts, this mismatch compounds. We formalize this as the compounding occupancy shift and prove that stale-occupancy evaluation incurs a penalty that scales quadratically with the number of agents. In contrast, intermediate-occupancy evaluation reduces this to linear scaling. We propose TeamTR, a trust-region framework that resamples trajectories after each component update and enforces per-agent divergence control, yielding rigorous per-update and per-stage improvement lower bounds. Experiments show that TeamTR outperforms single-agent and sequential baselines with 7.1% on average, mitigates coordination regressions, and supports plug-and-play component replacement. Code is available at https://github.com/Yydc/TeamTR.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper identifies a compounding occupancy shift in sequential fine-tuning of shared-context multi-agent LLM teams. It proves that stale-occupancy evaluation incurs a quadratic penalty in the number of agents while intermediate-occupancy evaluation (via resampling after each component update) reduces the penalty to linear scaling. TeamTR is proposed as a trust-region method that performs full-team trajectory resampling after each update and enforces per-agent divergence constraints, yielding per-update and per-stage improvement lower bounds. Experiments report a 7.1% average improvement over single-agent and sequential baselines, reduced coordination regressions, and support for plug-and-play component replacement.

Significance. If the occupancy-shift analysis and improvement bounds hold under the stated assumptions, the work provides a principled theoretical framework for multi-agent LLM fine-tuning that directly addresses a structural failure mode. The combination of rigorous lower bounds with empirical gains and code release would strengthen the case for trust-region methods in coordinated LLM systems.

major comments (2)
  1. [§3] §3 (Occupancy Shift Analysis): The proof that stale-occupancy evaluation scales quadratically while intermediate-occupancy scales linearly assumes exact full-team trajectory resampling after every single-agent update. The manuscript does not quantify the computational cost of this O(N) resampling per stage or demonstrate that it remains tractable for teams larger than those in the experiments or for long-context LLMs; without such analysis the claimed linear scaling and improvement bounds may not survive practical approximations.
  2. [§4.2] §4.2 (Per-Agent Divergence Control): The per-agent KL constraints are claimed to suffice for controlling joint occupancy shift, yet the manuscript provides no explicit bound showing that these local constraints prevent new coordination failures that could arise from the resampling process itself. This assumption is load-bearing for the per-stage improvement lower bound.
minor comments (2)
  1. The abstract states that 'Code is available at https://github.com/Yydc/TeamTR' but the main text lacks a dedicated reproducibility paragraph detailing random seeds, hyperparameter ranges, and exact evaluation protocols used for the 7.1% gain.
  2. Notation for occupancy measures and the precise definition of 'intermediate-occupancy evaluation' should be introduced earlier (ideally in §2) to improve readability before the formal proofs.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback, which identifies key practical and theoretical considerations. We address each major comment below and commit to revisions that strengthen the manuscript without altering its core claims.

read point-by-point responses

  1. Referee: [§3] §3 (Occupancy Shift Analysis): The proof that stale-occupancy evaluation scales quadratically while intermediate-occupancy scales linearly assumes exact full-team trajectory resampling after every single-agent update. The manuscript does not quantify the computational cost of this O(N) resampling per stage or demonstrate that it remains tractable for teams larger than those in the experiments or for long-context LLMs; without such analysis the claimed linear scaling and improvement bounds may not survive practical approximations.

    Authors: We agree that the linear-scaling result in §3 is derived under exact full-team resampling after each update. This O(N) cost per stage is explicit in the intermediate-occupancy procedure and is the mechanism that prevents quadratic compounding. In the experiments, team sizes were small (N ≤ 5) and trajectory lengths moderate, rendering the overhead acceptable within the simulator. For larger N or long-context settings, full resampling may indeed require approximations such as importance-weighted reuse of prior trajectories or batched sampling. We will insert a dedicated paragraph in §3 (and a corresponding note in the experiments section) that states the exact complexity, discusses when the linear bound remains approximately valid under controlled approximations, and clarifies that the theoretical claims hold exactly only under the stated resampling assumption. revision: yes

  2. Referee: [§4.2] §4.2 (Per-Agent Divergence Control): The per-agent KL constraints are claimed to suffice for controlling joint occupancy shift, yet the manuscript provides no explicit bound showing that these local constraints prevent new coordination failures that could arise from the resampling process itself. This assumption is load-bearing for the per-stage improvement lower bound.

    Authors: The per-agent KL constraints are intended to keep each component close to its pre-update version while the full-team trajectories are regenerated with the newly updated agents. This combination is what allows the per-stage improvement lower bound to be stated. We acknowledge, however, that the current text does not supply an explicit lemma that directly bounds the probability of newly introduced coordination failures attributable to the resampling step itself. In the revision we will add a short supporting argument (or lemma) in §4.2 that relates the per-agent KL radii to the total-variation distance of the joint occupancy under resampling, thereby making the load-bearing assumption explicit and tightening the justification for the per-stage bound. revision: yes

Circularity Check

0 steps flagged

No circularity: claims rest on formalization and stated proofs without self-referential reduction

full rationale

The abstract formalizes compounding occupancy shift, states a quadratic-vs-linear scaling result for stale vs intermediate occupancy evaluation, and claims per-update and per-stage improvement lower bounds from the TeamTR resampling and per-agent KL controls. No equations appear in the provided text that define any quantity in terms of itself or rename a fitted parameter as a prediction. The derivation chain is presented as proceeding from the occupancy-shift formalization to the bounds, with no load-bearing self-citation or ansatz imported from prior author work visible. The central improvement guarantees therefore remain independent of the inputs they are claimed to bound.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Only the abstract is available; no free parameters, axioms, or invented entities are described in sufficient detail to populate the ledger.

pith-pipeline@v0.9.0 · 5704 in / 1139 out tokens · 59377 ms · 2026-05-19T17:56:47.322611+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

67 extracted references · 67 canonical work pages · 18 internal anchors

  1. [1]

    Towards a Science of Scaling Agent Systems

    Towards a Science of Scaling Agent Systems , author =. arXiv preprint arXiv:2512.08296 , year =. doi:10.48550/arXiv.2512.08296 , url =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2512.08296

  2. [2]

    2025 , howpublished=

    DeepScaleR: Surpassing O1-Preview with a 1.5B Model by Scaling RL , author=. 2025 , howpublished=

    work page 2025

  3. [3]

    Proceedings of the Twentieth European Conference on Computer Systems , pages=

    Hybridflow: A flexible and efficient rlhf framework , author=. Proceedings of the Twentieth European Conference on Computer Systems , pages=

    work page

  4. [4]

    The Llama 3 Herd of Models

    The llama 3 herd of models , author=. arXiv preprint arXiv:2407.21783 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [5]

    2025 , eprint=

    Qwen2.5 Technical Report , author=. 2025 , eprint=

    work page 2025

  6. [6]

    Qwen3 Technical Report

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  7. [7]

    Why Do Multi-Agent LLM Systems Fail?

    Why Do Multi-Agent LLM Systems Fail? , author =. arXiv preprint arXiv:2503.13657 , year =. doi:10.48550/arXiv.2503.13657 , url =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2503.13657

  8. [8]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning , author=. arXiv preprint arXiv:2501.12948 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  9. [9]

    QwQ-32B: Embracing the Power of Reinforcement Learning , url =

    Qwen Team , month =. QwQ-32B: Embracing the Power of Reinforcement Learning , url =

    work page

  10. [10]

    Self-Consistency Improves Chain of Thought Reasoning in Language Models

    Self-consistency improves chain of thought reasoning in language models , author=. arXiv preprint arXiv:2203.11171 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [11]

    Advances in neural information processing systems , volume=

    Chain-of-thought prompting elicits reasoning in large language models , author=. Advances in neural information processing systems , volume=

    work page

  12. [12]

    Yan, Walden , year =. Don

    work page

  13. [13]

    2025 , month = jun, howpublished =

    How we built our multi-agent research system , author =. 2025 , month = jun, howpublished =

    work page 2025

  14. [14]

    AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation

    AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation , author =. arXiv preprint arXiv:2308.08155 , year =. doi:10.48550/arXiv.2308.08155 , url =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2308.08155

  15. [15]

    Mixture-of-Agents Enhances Large Language Model Capabilities

    Mixture-of-Agents Enhances Large Language Model Capabilities , author =. arXiv preprint arXiv:2406.04692 , year =. doi:10.48550/arXiv.2406.04692 , url =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2406.04692

  16. [16]

    ReAct: Synergizing Reasoning and Acting in Language Models

    ReAct: Synergizing Reasoning and Acting in Language Models , author =. arXiv preprint arXiv:2210.03629 , year =. doi:10.48550/arXiv.2210.03629 , url =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2210.03629

  17. [17]

    Advances in neural information processing systems , volume=

    Tree of thoughts: Deliberate problem solving with large language models , author=. Advances in neural information processing systems , volume=

    work page

  18. [18]

    DAPO: An Open-Source LLM Reinforcement Learning System at Scale

    Dapo: An open-source llm reinforcement learning system at scale , author=. arXiv preprint arXiv:2503.14476 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  19. [19]

    Advances in Neural Information Processing Systems , volume=

    Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change , author=. Advances in Neural Information Processing Systems , volume=

    work page

  20. [20]

    arXiv preprint arXiv:2506.08295 , year=

    From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information? , author=. arXiv preprint arXiv:2506.08295 , year=

    work page arXiv

  21. [21]

    Advances in neural information processing systems , volume=

    Training language models to follow instructions with human feedback , author=. Advances in neural information processing systems , volume=

    work page

  22. [22]

    Z ebra L ogic: On the Scaling Limits of LLM s for Logical Reasoning

    Zebralogic: On the scaling limits of llms for logical reasoning , author=. arXiv preprint arXiv:2502.01100 , year=

    work page arXiv

  23. [23]

    Measuring Mathematical Problem Solving With the MATH Dataset

    Measuring mathematical problem solving with the math dataset , author=. arXiv preprint arXiv:2103.03874 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  24. [24]

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models , author=. arXiv preprint arXiv:2402.03300 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  25. [25]

    Proximal Policy Optimization Algorithms

    Proximal policy optimization algorithms , author=. arXiv preprint arXiv:1707.06347 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  26. [26]

    Proceedings of the Nineteenth International Conference on Machine Learning (ICML) , year =

    Approximately Optimal Approximate Reinforcement Learning , author =. Proceedings of the Nineteenth International Conference on Machine Learning (ICML) , year =

    work page

  27. [27]

    Proceedings of the 32nd International Conference on Machine Learning (ICML) , series =

    Trust Region Policy Optimization , author =. Proceedings of the 32nd International Conference on Machine Learning (ICML) , series =. 2015 , url =

    work page 2015

  28. [28]

    International Conference on Learning Representations (ICLR) , year =

    High-Dimensional Continuous Control Using Generalized Advantage Estimation , author =. International Conference on Learning Representations (ICLR) , year =

    work page

  29. [29]

    Advances in neural information processing systems , volume=

    Multi-agent actor-critic for mixed cooperative-competitive environments , author=. Advances in neural information processing systems , volume=

    work page

  30. [30]

    Journal of Machine Learning Research , volume=

    Monotonic value function factorisation for deep multi-agent reinforcement learning , author=. Journal of Machine Learning Research , volume=

    work page

  31. [31]

    Advances in neural information processing systems , volume=

    The surprising effectiveness of ppo in cooperative multi-agent games , author=. Advances in neural information processing systems , volume=

    work page

  32. [32]

    Advances in Neural Information Processing Systems , volume=

    Camel: Communicative agents for" mind" exploration of large language model society , author=. Advances in Neural Information Processing Systems , volume=

    work page

  33. [33]

    Proceedings of the 36th annual acm symposium on user interface software and technology , pages=

    Generative agents: Interactive simulacra of human behavior , author=. Proceedings of the 36th annual acm symposium on user interface software and technology , pages=

    work page

  34. [34]

    Forty-first International Conference on Machine Learning , year=

    Improving factuality and reasoning in language models through multiagent debate , author=. Forty-first International Conference on Machine Learning , year=

    work page

  35. [35]

    Proceedings of the 2024 conference on empirical methods in natural language processing , pages=

    Encouraging divergent thinking in large language models through multi-agent debate , author=. Proceedings of the 2024 conference on empirical methods in natural language processing , pages=

    work page 2024

  36. [36]

    The twelfth international conference on learning representations , year=

    MetaGPT: Meta programming for a multi-agent collaborative framework , author=. The twelfth international conference on learning representations , year=

    work page

  37. [37]

    URL https://arxiv

    Chatdev: Communicative agents for software development, 2024 , author=. URL https://arxiv. org/abs/2307 , volume=

    work page 2024

  38. [38]

    First Conference on Language Modeling , year=

    Autogen: Enabling next-gen LLM applications via multi-agent conversations , author=. First Conference on Language Modeling , year=

    work page

  39. [39]

    arXiv preprint arXiv:2308.04592 , year=

    Shepherd: A critic for language model generation , author=. arXiv preprint arXiv:2308.04592 , year=

    work page arXiv

  40. [40]

    Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems , pages=

    ACORN: Acyclic Coordination with Reachability Network to Reduce Communication Redundancy in Multi-Agent Systems , author=. Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems , pages=

    work page

  41. [41]

    arXiv preprint arXiv:2310.05915 , year=

    Fireact: Toward language agent fine-tuning , author=. arXiv preprint arXiv:2310.05915 , year=

    work page arXiv

  42. [42]

    arXiv preprint arXiv:2306.02561 , year=

    Llm-blender: Ensembling large language models with pairwise ranking and generative fusion , author=. arXiv preprint arXiv:2306.02561 , year=

    work page arXiv

  43. [43]

    arXiv preprint arXiv:2305.16960 , volume=

    Training socially aligned language models in simulated human society , author=. arXiv preprint arXiv:2305.16960 , volume=

    work page arXiv

  44. [44]

    Simple synthetic data reduces sycophancy in large language models

    Simple synthetic data reduces sycophancy in large language models , author=. arXiv preprint arXiv:2308.03958 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  45. [45]

    Advances in Neural Information Processing Systems , volume=

    Gorilla: Large language model connected with massive apis , author=. Advances in Neural Information Processing Systems , volume=

    work page

  46. [46]

    Findings of the association for computational linguistics: ACL 2023 , pages=

    Discovering language model behaviors with model-written evaluations , author=. Findings of the association for computational linguistics: ACL 2023 , pages=

    work page 2023

  47. [47]

    arXiv preprint arXiv:2506.08292 , year=

    From debate to equilibrium: Belief-driven multi-agent llm reasoning via bayesian nash equilibrium , author=. arXiv preprint arXiv:2506.08292 , year=

    work page arXiv

  48. [48]

    arXiv preprint arXiv:2508.04652 , year=

    Llm collaboration with multi-agent reinforcement learning , author=. arXiv preprint arXiv:2508.04652 , year=

    work page arXiv

  49. [49]

    arXiv preprint arXiv:2502.16906 , year=

    AutoLogi: Automated generation of logic puzzles for evaluating reasoning abilities of large language models , author=. arXiv preprint arXiv:2502.16906 , year=

    work page arXiv

  50. [50]

    arXiv preprint arXiv:2501.05707 , year=

    Multiagent finetuning: Self improvement with diverse reasoning chains , author=. arXiv preprint arXiv:2501.05707 , year=

    work page arXiv

  51. [51]

    Proceedings of the IEEE/CVF international conference on computer vision , pages=

    Improving generalization in visual reinforcement learning via conflict-aware gradient agreement augmentation , author=. Proceedings of the IEEE/CVF international conference on computer vision , pages=

    work page

  52. [52]

    Towards Understanding Sycophancy in Language Models

    Towards understanding sycophancy in language models , author=. arXiv preprint arXiv:2310.13548 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  53. [53]

    IEEE Transactions on Industrial Electronics , year=

    Improving Robotic Grasp Detection Under Sparse Annotations Via Grasp Transformer With Pixel-Wise Contrastive Learning , author=. IEEE Transactions on Industrial Electronics , year=

    work page

  54. [54]

    Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems , pages=

    Heuristics-Assisted Experience Replay Strategy for Cooperative Multi-Agent Reinforcement Learning , author=. Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems , pages=

    work page

  55. [55]

    Self-consistency preference optimization, 2025

    Self-consistency preference optimization , author=. arXiv preprint arXiv:2411.04109 , year=

    work page arXiv

  56. [56]

    RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback

    Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback , author=. arXiv preprint arXiv:2309.00267 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  57. [57]

    Findings of the Association for Computational Linguistics: ACL 2024 , pages=

    Agenttuning: Enabling generalized agent abilities for llms , author=. Findings of the Association for Computational Linguistics: ACL 2024 , pages=

    work page 2024

  58. [58]

    International conference on machine learning , pages=

    Trust region policy optimization , author=. International conference on machine learning , pages=. 2015 , organization=

    work page 2015

  59. [59]

    Proceedings of the nineteenth international conference on machine learning , pages=

    Approximately optimal approximate reinforcement learning , author=. Proceedings of the nineteenth international conference on machine learning , pages=

    work page

  60. [60]

    arXiv preprint arXiv:2109.11251 , year=

    Trust region policy optimisation in multi-agent reinforcement learning , author=. arXiv preprint arXiv:2109.11251 , year=

    work page arXiv

  61. [61]

    International conference on machine learning , pages=

    Stabilising experience replay for deep multi-agent reinforcement learning , author=. International conference on machine learning , pages=. 2017 , organization=

    work page 2017

  62. [62]

    International conference on machine learning , pages=

    Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures , author=. International conference on machine learning , pages=. 2018 , organization=

    work page 2018

  63. [63]

    Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback

    Open problems and fundamental limitations of reinforcement learning from human feedback , author=. arXiv preprint arXiv:2307.15217 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  64. [64]

    A long way to go: Investigating length correlations in rlhf.arXiv preprint arXiv:2310.03716, 2023

    A long way to go: Investigating length correlations in rlhf , author=. arXiv preprint arXiv:2310.03716 , year=

    work page arXiv

  65. [65]

    Editing Models with Task Arithmetic

    Editing models with task arithmetic , author=. arXiv preprint arXiv:2212.04089 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  66. [66]

    International conference on machine learning , pages=

    Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time , author=. International conference on machine learning , pages=. 2022 , organization=

    work page 2022

  67. [67]

    Huang, Q

    Lorahub: Efficient cross-task generalization via dynamic lora composition , author=. arXiv preprint arXiv:2307.13269 , year=

    work page arXiv