Pith. sign in

REVIEW 4 major objections 5 minor 44 references

RLAE: Reinforcement Learning-Assisted Ensemble for LLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A reinforcement-learning agent that sets LLM ensemble weights per prompt beats fixed-weight baselines by up to 3.3 accuracy points on seven benchmarks.

desk verdict The RL-for-ensemble weighting idea is new enough to deserve a serious referee, but the missing train/test split makes the headline numbers unverifiable. read the letter →

arxiv 2506.00439 v1 pith:OCRY7AAY submitted 2025-05-31 cs.LG cs.AI

Yuqian Fu , Yuanheng Zhu , Jiajun Chai , Guojun Yin , Wei Lin , Qichao Zhang , Dongbin Zhao This is my paper
classification cs.LGcs.AI
keywords LLMensemblereinforcementlearningmulti-agentPPOspan-leveladaptiveweightallocationMarkovdecisionprocesslargelanguagemodelsmodelfusion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes RLAE, a framework that treats LLM ensembling as a Markov decision process and trains a reinforcement learning agent to assign each base model a weight for the next span of generated tokens, based on the prompt and what has been generated so far. The central claim is that these learned, context-dependent weights outperform fixed weights, perplexity-based weights, and ranker-based selection across general knowledge, math, science, and code benchmarks, by up to 3.3 accuracy points. This matters because it would make ensembles of comparable LLMs self-tuning: the same policy can adapt to a prompt's demands and even transfer to a new task without retraining, while keeping inference latency close to token-level heuristic fusion. The authors report that a multi-agent variant, where each LLM is controlled by its own RL agent, works best when base models are heterogeneous, and that a single-agent variant is better for code generation. The paper's own Limitations section flags that the terminal accuracy reward may not fully capture generation quality.

What carries the argument

The load-bearing mechanism is a span-level Markov decision process. At each span of $L=4$ tokens, the state is the prompt plus generated history, the action is a vector of ensemble weights on the probability simplex, and the reward is the final task-specific accuracy; the transition is the weighted mixture of base-model token probabilities from Eq. (1). The policy and value function share a DeBERTa-v3-large backbone, trained by PPO for the single-agent version and by MAPPO (one agent per base model, centralized critic, softmax over logits) for the multi-agent version. Span-level decisions cut the number of weighting choices from $H$ token steps to $\lceil H/L\rceil$, and tokenizer mismatch is handled by projecting all model vocabularies into a unified vocabulary through a mapping matrix following GAC. This construction is what lets the policy adapt weights per context rather than per dataset.

What would settle it

Re-run RLAE on one benchmark with an explicit held-out split: train the policy on half of MMLU and evaluate on the other half, and compare against a policy trained and evaluated on the same prompts. If the claimed gains over uniform or fixed weighting appear only when training and evaluation share prompts, the central claim collapses. A second check is to replace the terminal accuracy reward with a process reward or human-preference score; if RLAE then no longer beats the fixed-weight baselines, the result is tied to the specific reward rather than to adaptive weighting.

Watch

Extended reading notes

Core claim

On the paper's terms, the discovery is that ensemble weights for LLMs do not have to be fixed or hand-designed. A policy trained with PPO or MAPPO reads the input prompt and the response generated so far and outputs a weight vector over base models for the next span of tokens; at test time this policy is applied with no reward or supervision signal. Trained with terminal reward equal to task accuracy, the policy gives the best per-benchmark scores in most configurations examined, including strong gains on ARC-C, GSM8K, and other tasks in the three-model setting, while losing only 0.4-0.6 points when transferred from MMLU to ARC-C compared to a 3.8-point loss for the PAIRRANKER baseline. The explanation offered is path dependence: which model should lead depends on where the response is in its generation, so choosing weights span-by-span captures complementarity that response-level selection and token-level fixed fusion miss.

Load-bearing premise

The load-bearing premise is that the accuracy reward used to train each benchmark policy is computed on prompts genuinely separate from those used to report the benchmark's accuracy (a split the paper never states), so the reported gains reflect adaptive weighting rather than in-sample reward fitting, and that accuracy is a sufficient proxy for generation quality, which the Limitations section itself questions.

Editorial extensions

If this is right

  • A weighting policy trained on one benchmark can be applied to a related benchmark with only 0.4-0.6 point degradation, compared to a 3.8-point drop for the PAIRRANKER ranker baseline.
  • Span-level weight decisions keep inference latency close to token-level heuristic fusion, even with the extra 400M-parameter policy network, and far below ranker-based pairwise selection.
  • The multi-agent RLAE_MAPPO exceeds single-agent RLAE_PPO on three-model ensembles of heterogeneous LLMs, especially on tasks requiring different domain expertise, while single-agent RLAE_PPO remains stronger on code generation.
  • Adaptive weights remove per-task manual tuning: RLAE beats both uniform and perplexity-based weightings on MMLU and GSM8K with no change to the policy.
  • Gains are largest when base models are comparable in strength, and shrink when one model dominates the ensemble.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same MDP formulation could be trained with dense per-span process rewards or human-preference feedback, not just terminal accuracy; a direct comparison of these reward schemes would test whether the terminal accuracy reward is the binding limitation the paper itself flags.
  • Beyond the paper, if the policy's cross-task transfer reflects a general 'which model to trust at this point in the response' heuristic, then it should also transfer to very different prompt formats such as open-ended dialogue or tool-use tasks, which are not among the seven benchmarks tested.
  • Beyond the paper, the reported gains are per-benchmark accuracy; a deployment-oriented analysis would additionally measure per-token improvement, worst-case prompts, and whether the policy sometimes assigns near-uniform weights, since those cases would reveal when the RL overhead is unnecessary.
  • Beyond the paper, the framework currently trains policies per benchmark with benchmark-specific accuracy; a natural extension is meta-training across many tasks so that a new task needs only a few adaptation steps rather than a fresh RL run.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes RLAE, a reinforcement learning-assisted framework for ensembling LLMs. It formulates span-level ensemble weight selection as an MDP and trains a policy with PPO (single-agent) or MAPPO (multi-agent), where the reward is a task-specific metric such as accuracy. The authors evaluate on seven benchmarks with several 7-8B open-weight LLMs, reporting that RLAE improves over previous ensemble methods by up to 3.3 accuracy points, transfers across tasks without retraining, and has competitive latency. The main claims are that adaptive RL-based weighting outperforms fixed or heuristic weighting and that the method generalizes to new tasks.

Significance. The high-level idea of learning context-dependent ensemble weights with RL is interesting and potentially valuable: existing ensemble methods largely use fixed or manually designed weights, and a span-level formulation is a reasonable efficiency-accuracy trade-off. If the reported gains were obtained on held-out prompts with proper statistical controls, the paper would be a useful contribution to LLM ensembling and to the growing area of RL-based LLM coordination. The paper also deserves credit for comparing against several strong baseline ensemblers (PAIRRANKER, GAC, DEEPEN) and for including a cross-task transfer experiment, whose transfer row (MMLU-trained policy evaluated on ARC-C) is a genuinely held-out evaluation that shows only small degradation.

major comments (4)
  1. [Section 4.1 / Algorithm 2] The experimental protocol does not specify any train/test split. Section 4.1 states that the reward is a rule-based sparse reward using 'benchmark-specific evaluation metrics such as accuracy,' and Algorithm 2 loops 'for prompt x in D' and computes reward r_t from that same metric. If D is the entire benchmark and Tables 1 and 2 report accuracy on the same prompts, then the reported numbers are in-sample reward optimization rather than evidence of adaptive ensemble quality. The authors must state whether each benchmark is split into disjoint training and evaluation subsets, report the sizes of those subsets, and confirm that all table/figure accuracies are computed on held-out prompts. Without this information the headline 'up to 3.3%' improvement is not interpretable.
  2. [Section 4.2, Tables 1 and 2] The claim that RLAE 'consistently outperforms existing methods in most scenarios' is contradicted by several entries. In Table 2, RLAEMAPPO scores 67.9 on GSM8K versus OpenChat-3.5's 73.4 and 80.8 on PIQA versus 87.1; in Table 1, RLAEMAPPO scores 35.3 on GPQA with three LLMs versus Qwen-2.5-7B-Instruct's 36.4. These are losses to a single base model of 5.5, 6.3, and 1.1 points, respectively, which is larger than the reported 3.3-point maximum gain over ensemble baselines. The authors should report per-benchmark wins and losses against the best base model and against each baseline, and temper the abstract and Section 4.2 claims accordingly.
  3. [Tables 1-3 and Figures 4-5] No error bars, confidence intervals, or multiple-seed results are reported anywhere. RL training with PPO/MAPPO is stochastic, and the differences in Tables 1 and 2 are often within 1-2 accuracy points, so a single run cannot support the claimed improvements. Please report means and standard deviations over at least three seeds, and where possible paired significance tests (e.g., bootstrap or McNemar-style tests) for the key comparisons.
  4. [Section 4.3, Table 3] The cross-task generalization experiment is a step in the right direction, but the 'direct training' rows in Table 3 and Table A2 appear to train and evaluate on the same benchmark (ARC-C or GPQA). If so, those numbers are in-sample and should be labeled as such. The transfer rows (MMLU→ARC-C, ARC-C→GPQA) are valid held-out evaluations and show only small degradation, which is a point in the paper's favor; however, the authors should clearly state the training/evaluation split for both direct and transfer settings.
minor comments (5)
  1. [Figure 3] The label 'DAEEPEN' in the latency figure is a typo and should read 'DEEPEN.'
  2. [Table 1 caption] The caption says 'Highlight indicates the best performing method, while bold indicates the second best,' but no highlighting is visible in the rendered table and the bold usage is inconsistent; please make the formatting unambiguous.
  3. [Algorithm 2 vs. Section 4.1] Algorithm 2 computes a reward r_t at every span, while Section 4.1 says the reward is sparse and provided only at the terminal state. Please reconcile the pseudocode with the sparse-reward description, or clarify whether a dense process reward is used in training.
  4. [Section 3.1] The MDP definition says intermediate states use 'a process reward model that provides dense feedback,' but the experiments use only terminal accuracy. Please clarify which reward is actually used in the reported experiments.
  5. [Section 4.3] The latency comparison reports RLAE as comparable to GAC, but the RL agent adds a 400M-parameter DeBERTa model; please specify whether the reported latency includes the agent's forward passes and how the span length L affects the measurement.

Circularity Check

1 steps flagged · score 6.0 of 10

Main accuracy results may be in-sample reward optimization: the RL reward is the benchmark accuracy itself, and Section 4 specifies no train/test split, so the reported gains can reduce to a fitted objective.

  1. fitted input called prediction [Section 4.1 Implementation Details; Algorithm 2 (Appendix A); Section 4.2 Main Results]
    "For reward design, we employ a rule-based sparse reward model that provides feedback only at the terminal state of generation, utilizing benchmark-specific evaluation metrics such as accuracy. ... Algorithm 2: for prompt x in D do ... Compute reward r_t ... Tables 1 and 2 demonstrate the performance of our proposed RLAE framework."

    The paper defines the RL reward as the same benchmark-specific accuracy that later appears in the main performance tables. Algorithm 2 trains the policy by iterating over 'dataset D' and computing reward r_t at the terminal state, but Section 4.1 never states that D is split into training and held-out evaluation subsets. As written, the policy is rewarded on the same prompts whose accuracies are reported in Tables 1 and 2. The RL objective is then identical to the reported evaluation metric, so the headline 'up to 3.3% accuracy points' improvement over baselines is statistically forced if no split exists: it is the training reward, not an out-of-sample prediction of adaptive ensemble quality.

full rationale

The only substantive circularity risk is the absence of any documented train/test separation between the RL reward signal and the reported accuracy numbers. The paper states in Section 4.1 that the reward is 'benchmark-specific evaluation metrics such as accuracy,' and Algorithm 2 trains over a dataset D while computing exactly such rewards. Because Section 4.1 describes no split of MMLU, ARC-C, GSM8K, MBPP, etc., into training and evaluation subsets, the main comparison in Tables 1 and 2 can reduce by construction to fitting the policy to the evaluation metric on the evaluation prompts themselves. This is a fitted-input-called-prediction issue rather than a self-citation or definitional-uniqueness issue: there is no load-bearing self-citation chain, no imported uniqueness theorem, and no renamed known result. The cross-task generalization experiment (Table 3) trains on MMLU and evaluates on ARC-C, which is genuinely out-of-sample and provides independent content, so the paper is not wholly circular. However, that experiment does not rescue the primary per-benchmark claims, which the paper never verifies on held-out prompts. Accordingly, the central empirical claim carries a substantial circularity risk under the documented protocol, warranting a score of 6 rather than a lower score reserved for papers whose results are clearly self-contained and externally verified.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on standard RL/PPO machinery plus the domain assumption that weighted token-probability fusion across language models is a sound ensemble operation. The only hand-tuned numeric choices are common PPO hyperparameters and the span length. No new theoretical entities are introduced; the RL agent is a learned model, not an invented physical or conceptual entity.

free parameters (4)
  • span length L = 4
    Chosen by hand as a balanced choice (Section 4.3, Figure 5); affects the number of decision points and performance.
  • PPO clip range epsilon = 0.2
    Standard value from PPO, not tuned for this task; Table A1.
  • GAE discount gamma and lambda = 0.99, 0.95
    Standard values from PPO; Table A1.
  • entropy coefficient = 0.01
    Table A1.
assumptions (4)
  • domain assumption Weighted combination of token probabilities from different LLMs after vocabulary projection is a valid ensemble operation (Eq. 1).
    Assumed from GAC (Yu et al., 2024) and prior ensemble work; the paper does not validate that the projected probability space is semantically aligned.
  • domain assumption The state (prompt plus partial generation) is sufficient for the Markov property of the ensemble MDP (Section 3.1).
    The paper assumes the history up to span t captures all relevant information for choosing weights for the next span.
  • domain assumption Task-specific accuracy is an appropriate reward signal for training the weight policy (Section 3.1, 4.1).
    The paper acknowledges in Limitations that accuracy may not align with generation quality.
  • domain assumption The base LLMs generate from a comparable distribution after zero-probability assignment for out-of-vocabulary tokens (Appendix D).
    This is borrowed from GAC; the paper does not analyze its effect on the learned policy.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RLAE: Reinforcement Learning-Assisted Ensemble for LLMs." pith.science (2026). https://pith.science/paper/OCRY7AAY

@misc{pith2026250600439,
  author       = {Pith},
  title        = {Pith review of: RLAE: Reinforcement Learning-Assisted Ensemble for LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OCRY7AAY}},
  note         = {Machine review of arXiv:2506.00439}
}
abstract

Ensembling large language models (LLMs) can effectively combine diverse strengths of different models, offering a promising approach to enhance performance across various tasks. However, existing methods typically rely on fixed weighting strategies that fail to adapt to the dynamic, context-dependent characteristics of LLM capabilities. In this work, we propose Reinforcement Learning-Assisted Ensemble for LLMs (RLAE), a novel framework that reformulates LLM ensemble through the lens of a Markov Decision Process (MDP). Our approach introduces a RL agent that dynamically adjusts ensemble weights by considering both input context and intermediate generation states, with the agent being trained using rewards that directly correspond to the quality of final outputs. We implement RLAE using both single-agent and multi-agent reinforcement learning algorithms ($\text{RLAE}_\text{PPO}$ and $\text{RLAE}_\text{MAPPO}$ ), demonstrating substantial improvements over conventional ensemble methods. Extensive evaluations on a diverse set of tasks show that RLAE outperforms existing approaches by up to $3.3\%$ accuracy points, offering a more effective framework for LLM ensembling. Furthermore, our method exhibits superior generalization capabilities across different tasks without the need for retraining, while simultaneously achieving lower time latency.

Figures

Figures reproduced from arXiv: 2506.00439 by the authors.

Figure 1
Figure 1. Overview of Ranker-based Ensemble, Heuris [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The RLAE framework. Top row: at each generation step, our RL agent makes actions (ensemble weights) [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Time latency comparison of different methods. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Span length ablation study on different tasks. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

44 extracted references · 20 canonical work pages

  1. [1]

    Anthropic. 2024. https://www.anthropic.com/news/claude-3-family Introducing the next generation of Claude

    work page 2024

  2. [2]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732

    arXiv 2021

  3. [3]

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. PIQA : Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence

    work page 2020

  4. [4]

    Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. 2016. OpenAI Gym . arXiv preprint arXiv:1606.01540

    arXiv 2016

  5. [5]

    Jiajun Chai, Sicheng Li, Yuqian Fu, Dongbin Zhao, and Yuanheng Zhu. 2025. Empowering LLM agents with zero-shot optimal decision-making through q-learning. In The Thirteenth International Conference on Learning Representations

    work page 2025

  6. [6]

    Souradip Chakraborty, Sujay Bhatt, Udari Madhushani Sehwag, Soumya Suvra Ghosal, Jiahao Qiu, Mengdi Wang, Dinesh Manocha, Furong Huang, Alec Koppel, and Sumitra Ganesh. 2025. Collab: Controlled decoding using mixture of agents for LLM alignment. In The Thirteenth International Conference on Learning Representations

    work page 2025

  7. [7]

    Zhijun Chen, Jingzheng Li, Pengpeng Chen, Zhuoran Li, Kai Sun, Yuankai Luo, Qianren Mao, Dingqi Yang, Hailong Sun, and Philip S Yu. 2025. Harnessing multiple large language models: A survey on LLM ensemble. arXiv preprint arXiv:2502.18036

    arXiv 2025

  8. [8]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try ARC , the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457

    arXiv 2018

Show all 44 references
  1. [9]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    2021 arXiv

  2. [10]

    Evan Frick, Connor Chen, Joseph Tennyson, Tianle Li, Wei-Lin Chiang, Anastasios N Angelopoulos, and Ion Stoica. 2025. Prompt-to-leaderboard. arXiv preprint arXiv:2502.14855

    2025 arXiv

  3. [11]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Art...

    2024 arXiv

  4. [12]

    Sven Gronauer and Klaus Diepold. 2022. Multi-agent deep reinforcement learning: a survey. Artificial Intelligence Review, 55(2):895--943

    2022

  5. [13]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. DeepSeek-R1 : Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948

    2025 arXiv

  6. [14]

    Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. DeBERTaV3 : Improving DeBERTa using ELECTRA -style pre-training with gradient-disentangled embedding sharing. arXiv preprint arXiv:2111.09543

    2021 arXiv

  7. [15]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In International Conference on Learning Representations

    2021

  8. [16]

    Shengyi Huang, Rousslan Fernand Julien Dossa, Antonin Raffin, Anssi Kanervisto, and Weixun Wang. 2022. The 37 implementation details of proximal policy optimization. The ICLR Blog Track 2023

    2022

  9. [17]

    Yichong Huang, Xiaocheng Feng, Baohang Li, Yang Xiang, Hui Wang, Ting Liu, and Bing Qin. 2024. Ensemble learning for heterogeneous large language models with deep parallel collaboration. Advances in Neural Information Processing Systems, 37:119838--119860

    2024

  10. [18]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L \' e lio Renard Lavaud, Marie - Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, ...

    2023 arXiv

  11. [19]

    Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. 2023 b . https://doi.org/10.18653/v1/2023.acl-long.792 LLM -blender: Ensembling large language models with pairwise ranking and generative fusion . In Proceedings of the 61st Annual Meeting of the Association for Computational Ling...

    2023 doi

  12. [20]

    Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. https://doi.org/10.18653/v1/P17-1147 T rivia QA : A large scale distantly supervised challenge dataset for reading comprehension . In Proceedings of the 55th Annual Meeting of the Association for Computational...

    2017 doi

  13. [21]

    Michael L Littman. 1994. Markov games as a framework for multi-agent reinforcement learning. In Machine learning proceedings 1994, pages 157--163. Elsevier

    1994

  14. [22]

    Jinliang Lu, Ziliang Pang, Min Xiao, Yaochen Zhu, Rui Xia, and Jiajun Zhang. 2024. Merge, ensemble, and cooperate! a survey on collaborative strategies in the era of large language models. arXiv preprint arXiv:2407.06089

    2024 arXiv

  15. [23]

    Bo Lv, Chen Tang, Yanan Zhang, Xin Liu, Yue Yu, and Ping Luo. 2024. SpecFuse : Ensembling large language models via next-segment prediction. arXiv preprint arXiv:2412.07380

    2024 arXiv

  16. [24]

    Costas Mavromatis, Petros Karypis, and George Karypis. 2024. https://openreview.net/forum?id=5Nsl0nlStc Pack of LLM s: Model fusion at test-time via perplexity optimization . In First Conference on Language Modeling

    2024

  17. [25]

    OpenAI. 2024. https://openai.com/index/gpt-4o-system-card/ GPT-4o system card

    2024

  18. [26]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 3...

    2022

  19. [27]

    Bei Peng, Tabish Rashid, Christian Schroeder de Witt, Pierre-Alexandre Kamienny, Philip Torr, Wendelin B \"o hmer, and Shimon Whiteson. 2021. FACMAC : Factored multi-agent centralised policy gradients. Advances in Neural Information Processing Systems, 34:12208--12221

    2021

  20. [28]

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, ...

    2024 arXiv

  21. [29]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. GPQA : A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling

    2024

  22. [30]

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. 2016. High-dimensional continuous control using generalized advantage estimation. In International Conference on Learning Representations

    2016

  23. [31]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347

    2017 arXiv

  24. [32]

    Selim Furkan Tekin, Fatih Ilhan, Tiansheng Huang, Sihao Hu, and Ling Liu. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.698 LLM - TOPLA : Efficient LLM ensemble by maximising diversity . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 11...

    2024 doi

  25. [33]

    Songjun Tu, Jingbo Sun, Qichao Zhang, Xiangyuan Lan, and Dongbin Zhao. 2025. Online preference-based reinforcement learning with self-augmented feedback from large language model. In Proceedings of the 24rd International Conference on Autonomous Agents and Multiagent Systems

    2025

  26. [34]

    Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. 2024. OpenChat : Advancing open-source language models with mixed-quality data. In The Twelfth International Conference on Learning Representations

    2024

  27. [35]

    Kaixuan Xu, Jiajun Chai, Sicheng Li, Yuqian Fu, Yuanheng Zhu, and Dongbin Zhao. 2025 a . DipLLM : Fine-tuning LLM for strategic decision-making in Diplomacy . In International Conference on Machine Learning. PMLR

    2025

  28. [36]

    Yangyifan Xu, Jianghao Chen, Junhong Wu, and Jiajun Zhang. 2025 b . https://aclanthology.org/2025.coling-main.555/ Hit the sweet spot! span-level ensemble for large language models . In Proceedings of the 31st International Conference on Computational Linguistics, pages 8314--...

    2025

  29. [37]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze...

    2024 arXiv

  30. [38]

    Yuxuan Yao, Han Wu, Mingyang LIU, Sichun Luo, Xiongwei Han, Jie Liu, Zhijiang Guo, and Linqi Song. 2025. Determine-Then-Ensemble : Necessity of top-k union for large language model ensembling. In The Thirteenth International Conference on Learning Representations

    2025

  31. [39]

    Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. 2022. The surprising effectiveness of PPO in cooperative multi-agent games. Advances in neural information processing systems, 35:24611--24624

    2022

  32. [40]

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. 2025. DAPO : An open-source LLM reinforcement learning system at scale. arXiv preprint arXiv:2503.14476

    2025 arXiv

  33. [41]

    Yao-Ching Yu, Chun Chih Kuo, Ye Ziqi, Chang Yucheng, and Yueh-Se Li. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.99 Breaking the ceiling of the LLM community by treating token generation as a classification for ensembling . In Findings of the Association for Computat...

    2024 doi

  34. [42]

    Wenhao Zheng, Yixiao Chen, Weitong Zhang, Souvik Kundu, Yun Li, Zhengzhong Liu, Eric P Xing, Hongyi Wang, and Huaxiu Yao. 2025. CITER : Collaborative inference for efficient large language model decoding with token-level routing. arXiv preprint arXiv:2502.01976

    2025 arXiv

  35. [43]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  36. [44]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.