Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Multi-Agent Sampling: Scaling Inference Compute for Data Synthesis with Tree Search-Based Agentic Collaboration

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A tree-search coordinator lets several LLMs share inference compute more efficiently than any fixed workflow, the paper claims, and the synthetic data it produces transfers to fine-tuning.

desk verdict Applying MCTS to multi-agent sampling is a sensible engineering step, and the paper is honest about its reward-hacking limitation; the compute-optimality claim is plausible but the current evidence does not fully nail it down. read the letter →

arxiv 2412.17061 v2 pith:PIGM2P2M submitted 2024-12-22 cs.CL

Hai Ye , Mingbao Lin , Hwee Tou Ng , Shuicheng Yan This is my paper · ORCID
classification cs.CL
keywords multi-agentsamplinginferencecomputescalingMonteCarloTreeSearchmodelcoordinationdatasynthesisbest-of-Nrewardalignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that when inference compute is spent on generating multiple outputs from several different language models, how you coordinate those models matters as much as how many samples you draw. It proposes TOA, a Monte Carlo Tree Search that decides, for each input question, which model should refine which earlier response, guided by a reward model. Across alignment, machine translation, and math, the paper reports that this dynamic coordination beats random sampling from a single model, parallel ensembles, sequential refinement, and a fixed mixture-of-agents workflow at equal FLOPs. If true, multi-agent sampling becomes a practical way to buy better performance and better synthetic training data with compute rather than with larger models.

What carries the argument

The central object is TOA, a Monte Carlo Tree Search whose tree alternates between model nodes and response nodes. Each action is a pair (model, previous response), the transition draws a new response from that model conditioned on that response, and the reward is a reward-model score for the new response. UCB balances visiting low-count branches against high-reward branches; response nodes are pruned to keep the best-scoring children while model nodes are not. This lets the search discover, per question, which refinement path among the K models yields the highest rewards, instead of following a fixed pipeline.

What would settle it

Re-run TOA and the baselines at matched FLOPs while evaluating the selected outputs with a reward model different from the one guiding the search, and check whether TOA's advantage persists. The paper already reports a decline on KIWI-XXL at high compute for translation; if that decline is large enough to eliminate TOA's edge over parallel ensemble under an independent judge, the compute-optimality claim would be an artifact of reward hacking.

Watch

Extended reading notes

Core claim

The paper's central claim is that optimal multi-agent coordination is instance-specific, so a fixed workflow wastes compute, and that viewing coordination as sequential decision making and searching it with MCTS yields the most compute-efficient data synthesis. TOA formalizes best-of-N sampling from K models as an MDP: each action picks a model and a previously generated response to refine, and a reward model scores each new response. The search tree alternates model layers and response layers, uses UCB to trade exploration against reward, prunes low-scoring responses, and backpropagates rewards. The paper reports that this method is the most compute-optimal across AlpacaEval, WMT, and MATH, reaches a 72.2% length-controlled win rate on AlpacaEval, sets new SOTA results on WMT, and that fine-tuning on TOA-synthesized alignment data beats SimPO on AlpacaEval and Arena-Hard.

Load-bearing premise

The paper assumes the reward model that guides the search — ArmoRM for alignment, KIWI for translation, Qwen2.5-Math-RM for math — is a trustworthy measure of response quality, and its own translation experiments show that optimizing KIWI eventually lowers scores on the independent KIWI-XXL judge.

Editorial extensions

If this is right

  • Multi-agent best-of-N sampling is more compute-efficient than single-model sampling at the same FLOPs, across alignment, translation, and math.
  • A tree-search coordinator can beat parallel ensemble, sequential refinement, and mixture-of-agents baselines while using no more inference FLOPs.
  • Synthetic alignment data generated by TOA transfers to fine-tuning, outperforming preference-learning baselines like SimPO on AlpacaEval and Arena-Hard.
  • TOA sets a new state of the art on WMT translation benchmarks, above GPT-4 and CPO, at lower total FLOPs than the nearest competitor MoA.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the reward-model dependence is tamed — for example with a more robust or multi-objective reward — the same MCTS coordination could be applied to other multi-agent settings such as code generation, tool use, or retrieval-augmented generation, where instance-specific workflows likely matter too.
  • The finding that successor models along optimal paths are usually different from predecessors suggests that diversity, not just quality, is what makes multi-agent refinement effective; a testable extension is whether deliberately maximizing pairwise model diversity in the MCTS action space improves results further.
  • The paper's own reward-hacking observation implies that compute-efficiency comparisons of search-based agents should be re-run with held-out reward models or external judges before treating FLOPs-versus-quality curves as stable.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript proposes Tree Search-based Orchestrated Agents (TOA) for multi-agent sampling, in which the coordination of several language models is framed as a Markov decision process and solved with Monte Carlo Tree Search guided by a reward model. The method is evaluated on alignment (AlpacaEval), machine translation (WMT'21/'22), and mathematical reasoning (MATH), where the authors report scaling curves as a function of inference FLOPs and claim that TOA is the most compute-optimal method compared with parallel ensemble, sequential refinement, and mixture-of-agents baselines. The paper also demonstrates that fine-tuning with TOA-synthesized alignment data improves AlpacaEval and Arena-Hard scores over several preference-learning baselines.

Significance. If the compute-optimality claim holds, TOA would be a practical and timely contribution to inference-time scaling and synthetic data generation. The formalization of existing fixed workflows as special cases of a general multi-agent coordination framework is useful, and the release of code supports reproducibility. The paper is also transparent in acknowledging reward hacking in the machine-translation experiments. However, the demonstrated reward hacking raises a load-bearing concern: because the same reward model guides the search and is used to report the internal-quality scaling curves, the observed gains may partly reflect overfitting to that reward model. The absence of external-evaluator scaling curves for alignment and math prevents the reader from resolving this concern, and the central claim as stated is therefore stronger than the evidence currently supports.

major comments (4)
  1. [Section 6.2, Figure 4] The paper's own reward-hacking result directly undercuts the headline compute-optimality claim. The left panel of Figure 4 evaluates translations with the same KIWI reward model that guides TOA's MCTS (Section 5.1, Eq. (5)), while the right panel shows that when the selected best translation is re-scored by the external KIWI-XXL model, the score declines after an initial improvement at high compute budgets. Because the same reward-guided selection mechanism is used for alignment (ArmoRM, Figure 3) and math (Qwen2.5-Math-RM-72B, Figure 5), and no external-evaluator scaling curves are provided for those tasks, the claim that TOA remains compute-optimal across all tasks is not established. Moreover, the plotted compute range in Figure 4 stops before the decline fully plays out, so the assertion that TOA remains the best overall approach is untested at larger budgets.
  2. [Section 5.1, Eq. (2)-(5)] Because the reward model directly determines the MCTS selection, expansion, simulation, and backpropagation steps, the compute-optimality of TOA is conditional on the reward model being a faithful proxy for true quality. The paper provides no evidence that ArmoRM or Qwen2.5-Math-RM-72B maintains fidelity to external judges (GPT-4, KIWI-XXL, ground-truth answers) across the full compute range considered. I ask for per-task external-evaluator scaling curves, or at least a demonstration that the internal reward remains monotonically related to external quality over the tested budgets, to support the transferability of the method.
  3. [Figures 3-5] The absence of error bars, the downsampling of AlpacaEval to 200 prompts (Figure 3 caption), and the moving-average smoothing applied in Figures 4 and 5 make it difficult to assess whether the observed ordering among TOA, sequential refinement, and mixture-of-agents is statistically reliable, especially near crossing points of the scaling curves. The compute-optimality conclusion should be supported by confidence intervals or multiple random seeds at each compute budget, and the raw unsmoothed curves should be reported alongside the smoothed versions.
  4. [Section 6.2] The fitted scaling law R = a·log10(C)^2 + b·log10(C) + c is presented as a 'reliable means to predict the reward', but it is based on only seven points, has no uncertainty estimates, and uses the internal ArmoRM reward rather than an external quality measure. The extrapolative claim should be validated on held-out compute budgets and against external evaluators before it is described as reliable.
minor comments (5)
  1. [Section 5.1] The sentence 'The above stages of a −d repeat' should read 'a–d' or 'a to d'.
  2. [Section 6.2] The text says 'the left figure illustrates' and 'the right figure reveals'; for clarity, use 'left panel' and 'right panel' when referring to subfigures of Figure 4.
  3. [Appendix B.3] The statement 'For MoA, we use the default prompt for generation when alignment is involved' should specify what the default prompt is and where it appears in the appendix.
  4. [Figure 5] The legend entries such as 'Par. Ensemble_2' and 'TOA_4' are not explained in the caption; please state that the subscript denotes the number of models combined.
  5. [Section 6.1] The MATH evaluation uses only 100 problems sampled from MATH500 (Table 4); this small test set should be listed as a limitation or at least mentioned explicitly in the main text.

Circularity Check

2 steps flagged · score 5.0 of 10

Partial circularity: TOA's left-panel scaling curves optimize and then read the same reward model, and the fitted 'prediction' curve is a fit to that same objective; external evaluations (GPT-4, KIWI-XXL, MATH) provide independent support for the headline SOTA.

  1. self definitional [Figure 3 (left) and Figure 4 (left) captions; Section 5.1 Eq. (5); Section 6.2 'TOA is the most compute-optimal.']
    "Left: ArmoRM is used as the reward model for guidance and evaluation."

    The left-panel metric is the average top-10 reward assigned by ArmoRM (alignment) or KIWI (translation). TOA's MCTS selects and backpropagates with exactly this reward (Eq. 5: ri ← R(x, yi)). Thus TOA's left-panel curve plots the value of the very objective being maximized; improvement in that metric is partly definitional. The paper invokes 'the scaling results in Figures 3, 4, and 5' as the basis for 'TOA is the most compute-optimal,' so this self-reward curve is load-bearing evidence for the compute-optimality claim, even though external GPT-4 and KIWI-XXL curves are also provided.

  2. fitted input called prediction [Section 6.2, paragraph 'TOA is the most compute-optimal.']
    "To analyze the scaling behavior of TOA, we fit its scaling curve in Figure 3 (left) using a function where the input is FLOPs (C) and the output is the avg. top-10 reward (R)."

    The fitted 'scaling curve' is fit to the same avg. top-10 reward R that TOA's search maximizes at generation time (Eq. 5). Calling the learned function a 'reliable means to predict the reward for a given compute budget' is therefore a description of the optimized objective's own growth, not a prediction of an independent held-out quality metric. The curve is reconstructed from the training signal that drove the search, which is the fitted-input-as-prediction pattern, though the paper's external evaluations are not fitted in this way.

full rationale

The paper's headline achievements—72.2% LC win rate on AlpacaEval, SOTA WMT scores, and fine-tuning gains—are evaluated externally (GPT-4, KIWI-XXL, MetricX, Arena-Hard, and answer accuracy on MATH), so the central results are not entirely circular. However, the compute-optimality claim is explicitly supported by 'the scaling results in Figures 3, 4, and 5,' and the left panels of Figures 3 and 4 use the same reward model for both guiding TOA's search and scoring the reported samples. Because TOA selects and backpropagates with that exact reward, the left-panel improvement is partly an artifact of optimizing and then reading the same scalar. The paper itself acknowledges this weakness in the 'Reward hacking exists in TOA' paragraph: the external KIWI-XXL score initially improves but later declines, which confirms that the internally guided reward overstates true quality at high compute. The fitted reward-scaling curve is likewise a curve fit to the optimized objective rather than an external prediction. The PRS baseline is from the authors' prior work but is used only as a comparison method, not as a load-bearing justification for TOA, so it does not add circularity. On balance, the self-reward panels and fitted curve introduce partial circularity, but the independent external evaluations keep the core SOTA claims meaningfully supported; score 5 reflects that partial rather than total circularity.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

No new physical entities are introduced. The main load-bearing choices are the reward model as a quality oracle and the FLOPs accounting; these are assumptions, not invented entities.

free parameters (6)
  • alpha (UCB exploration) = 0.01 (alignment, math), 0.05 (MT)
    Used in Eq. (3) to balance exploration vs. exploitation; set per task in Appendix B.3.
  • Maximum tree width / depth = floor(N/3) for alignment and MT, floor(N/2) for math
    Controls the shape of the search tree; chosen by hand in Appendix B.3.
  • Sample count N = 64 to 1280 depending on experiment
    Sets the compute budget; varied to build scaling curves (Figures 3-5).
  • Sampling temperature and top_p = temperature 0.7, top_p 1.0
    Generation noise in Eq. (4); fixed in Appendix B.3.
  • Scaling curve coefficients a, b, c = a=-0.0031, b=0.11, c=-0.71
    Fitted to TOA rewards in Figure 3 (left) to describe reward vs. FLOPs; no validation on held-out budgets.
  • DPO negative rank 30 = rank 30
    For synthetic data DPO, the 30th-ranked response is used as the reject sample; chosen from 'optimal ranking experiments' (Section 6.2).
assumptions (6)
  • standard math UCB and MCTS converge to good actions in the given finite tree.
    MCTS is applied as in Browne et al. (2012); the search tree is finite (at most N responses), so standard exploration guarantees apply.
  • domain assumption FLOPs from Hoffmann et al. (2022) is the correct compute measure for comparison.
    FLOPs are summed over all models and all generations; the comparison ignores memory cost and API latency, which are noted as a limitation.
  • domain assumption The selected reward models provide reliable real-time feedback.
    Section 5.1 relies on reward model R to score new responses; the paper later admits reward hacking (Section 6.2), so this assumption is load-bearing and only partially valid.
  • domain assumption Multiple distinct LLMs have complementary strengths that can be combined.
    Introduction states this hypothesis; the MATH result (4 models underperforming 2 models) shows it does not always hold.
  • domain assumption Best-of-N sampling with reward selection improves quality.
    Used as the evaluation protocol throughout; standard in prior work (Liu et al., 2024).
  • ad hoc to paper The quadratic scaling law R = a log10(C)^2 + b log10(C) + c is the correct functional form.
    Section 6.2 introduces this form without prior justification; the coefficients are fitted, not derived.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-Agent Sampling: Scaling Inference Compute for Data Synthesis with Tree Search-Based Agentic Collaboration." pith.science (2026). https://pith.science/paper/PIGM2P2M

@misc{pith2026241217061,
  author       = {Pith},
  title        = {Pith review of: Multi-Agent Sampling: Scaling Inference Compute for Data Synthesis with Tree Search-Based Agentic Collaboration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PIGM2P2M}},
  note         = {Machine review of arXiv:2412.17061}
}
read the original abstract

Scaling laws for inference compute in multi-agent systems remain under-explored compared to single-agent scenarios. This work aims to bridge this gap by investigating the problem of data synthesis through multi-agent sampling, where synthetic responses are generated by sampling from multiple distinct language models. Effective model coordination is crucial for successful multi-agent collaboration. Unlike previous approaches that rely on fixed workflows, we treat model coordination as a multi-step decision-making process, optimizing generation structures dynamically for each input question. We introduce Tree Search-based Orchestrated Agents~(TOA), where the workflow evolves iteratively during the sequential sampling process. To achieve this, we leverage Monte Carlo Tree Search (MCTS), integrating a reward model to provide real-time feedback and accelerate exploration. Our experiments on alignment, machine translation, and mathematical reasoning demonstrate that multi-agent sampling significantly outperforms single-agent sampling as inference compute scales. TOA is the most compute-efficient approach, achieving SOTA performance on WMT and a 72.2\% LC win rate on AlpacaEval. Moreover, fine-tuning with our synthesized alignment data surpasses strong preference learning methods on challenging benchmarks such as Arena-Hard and AlpacaEval.

Figures

Figures reproduced from arXiv: 2412.17061 by the authors.

Figure 1
Figure 1. Illustration of multi-agent sampling: We scale [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of previous methods (a, b and c) and our method TOA for multi-agent sampling. The methods used to sample N responses per question share the same model structure but differ in coordination strategies. TOA casts the problem as multi-step decision making and uses Monte Carlo Tree Search (MCTS) to decide which model and response to use to generate a new sample. Stages d.1 to d.4 display how a new sample is … view at source ↗
Figure 3
Figure 3. Scaling results for alignment on AlpacaEval with 4 small-scale models, varying the number of samples [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Scaling results for machine translation (zh to en) on WMT’22 using 5 large-scale models. The number [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Scaling results for math where the 4 small-scale models are combined. We generate solutions iteratively [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: For TOA with 5 large models, we identify the [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: With the large-scale models, we report the [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: The refinement prompt for alignment. You are provided with a text originally written in a foreign language and its corresponding English translation. Your task is to revise this translation to improve its precision, natural flow, consistency, and completeness. When ref…
Figure 9
Figure 9. Figure 9: The refinement prompt for machine transla [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 11
Figure 11. Figure 11: Ablation of the number of models for combi [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]
Figure 12
Figure 12. Figure 12: Effect of depth for TOA. We vary the max [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]
Figure 13
Figure 13. Figure 13: Performance of different language models [PITH_FULL_IMAGE:figures/full_fig_p014_13.png]
Figure 14
Figure 14. Figure 14: On the machine translation task, for TOA [PITH_FULL_IMAGE:figures/full_fig_p015_14.png]
Figure 15
Figure 15. Figure 15: Example (a) of a decision tree constructed [PITH_FULL_IMAGE:figures/full_fig_p016_15.png]
Figure 17
Figure 17. Figure 17: Example (c) of a decision tree constructed [PITH_FULL_IMAGE:figures/full_fig_p017_17.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Theory of Inference Compute Scaling: Reasoning through Directed Stochastic Skill Search

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A skill-graph random-walk model gives closed-form accuracy-versus-compute formulas for four reasoning strategies and connects them to training scaling.

  2. ThinkLess: A Training-Free Inference-Efficient Method for Reducing Reasoning Redundancy

    cs.CL 2025-05 conditional novelty 4.0 of 10

    Skipping explicit chain-of-thought reasoning entirely, and prompting for a formatted answer, matches full CoT accuracy on several benchmarks while cutting latency and token counts.

Reference graph

Works this paper leans on

51 extracted references · 20 canonical work pages · cited by 2 Pith papers

  1. [1]

    Mistral AI. 2024 a . https://mistral.ai/news/mixtral-8x22b/ Cheaper, better, faster, stronger . Accessed: 2024-12-01

    work page 2024

  2. [2]

    Mistral AI. 2024 b . Mistral large 2. https://mistral.ai/news/mistral-large-2407/. Accessed: 2024-12-01

    work page 2024

  3. [3]

    Tran, and Mehran Kazemi

    Hritik Bansal, Arian Hosseini, Rishabh Agarwal, Vinh Q. Tran, and Mehran Kazemi. 2024. http://arxiv.org/abs/2408.16737 Smaller, weaker, yet better: Training LLM reasoners via compute-optimal sampling . CoRR, abs/2408.16737

    arXiv 2024

  4. [4]

    Bradley C. A. Brown, Jordan Juravsky, Ryan Saul Ehrlich, Ronald Clark, Quoc V. Le, Christopher R \' e , and Azalia Mirhoseini. 2024. http://arxiv.org/abs/2407.21787 Large language monkeys: Scaling inference compute with repeated sampling . CoRR, abs/2407.21787

    arXiv 2024

  5. [5]

    Lucas, Peter I

    Cameron Browne, Edward Jack Powley, Daniel Whitehouse, Simon M. Lucas, Peter I. Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez Liebana, Spyridon Samothrakis, and Simon Colton. 2012. https://doi.org/10.1109/TCIAIG.2012.2186810 A survey of monte carlo tree search methods . IEEE Trans. Comput. Intell. AI Games , 4(1):1--43

    arXiv 2012

  6. [6]

    Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. 2024. http://arxiv.org/abs/2406.20094 Scaling synthetic data creation with 1,000,000,000 personas . CoRR, abs/2406.20094

    arXiv 2024

  7. [7]

    Justin Chih - Yao Chen, Swarnadeep Saha, and Mohit Bansal. 2024 a . https://doi.org/10.18653/v1/2024.acl-long.381 Reconcile: Round-table conference improves reasoning via consensus among diverse llms . In ACL

    doi:10.18653/v1/2024.acl-long.381 2024

  8. [8]

    Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chi - Min Chan, Heyang Yu, Yaxi Lu, Yi - Hsin Hung, Chen Qian, Yujia Qin, Xin Cong, Ruobing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou. 2024 b . https://openreview.net/forum?id=EHg5GDnyq1 Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors . In ICLR

    work page 2024

Show all 51 references
  1. [9]

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. 2024. https://openreview.net/forum?id=BOorDpKHiJ ULTRAFEEDBACK: boosting language models with scaled AI feedback . In ICML

    2024

  2. [10]

    Elvis Dohmatob, Yunzhen Feng, Pu Yang, Fran c ois Charton, and Julia Kempe. 2024. https://openreview.net/forum?id=KVvku47shW A tale of tails: Model collapse as a change of scaling laws . In ICML

    2024

  3. [11]

    Tenenbaum, and Igor Mordatch

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. 2024. https://openreview.net/forum?id=zj7YuTE4t8 Improving factuality and reasoning in language models through multiagent debate . In ICML

    2024

  4. [12]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al - Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Z...

    2024 arXiv

  5. [13]

    Yunzhen Feng, Elvis Dohmatob, Pu Yang, Fran c ois Charton, and Julia Kempe. 2024. http://arxiv.org/abs/2406.07515 Beyond model collapse: Scaling up with synthesized data requires reinforcement . CoRR, abs/2406.07515

    2024 arXiv

  6. [14]

    Donoho, and Sanmi Koyejo

    Matthias Gerstgrasser, Rylan Schaeffer, Apratim Dey, Rafael Rafailov, Tomasz Korbak, Henry Sleight, Rajashree Agrawal, John Hughes, Dhruv Bhandarkar Pai, Andrey Gromov, Dan Roberts, Diyi Yang, David L. Donoho, and Sanmi Koyejo. 2024. https://openreview.net/forum?id=5B2K4LRgmz ...

    2024

  7. [15]

    C aglar G \" u l c ehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, Wolfgang Macherey, Arnaud Doucet, Orhan Firat, and Nando de Freitas. 2023. http://arxiv.org/abs/2308.08998 Rei...

    2023 arXiv

  8. [16]

    Yanzhu Guo, Guokan Shang, Michalis Vazirgiannis, and Chlo \' e Clavel. 2024. https://doi.org/10.18653/v1/2024.findings-naacl.228 The curious decline of linguistic diversity: Training language models on synthetic text . In Findings of NAACL

    2024 doi

  9. [17]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/be83ab3ecd0db773eb2dc1b0a17836a1-Abstract-round2.html Measuring mathematical problem so...

    2021

  10. [18]

    Rae, Oriol Vinyals, and Laurent Sifre

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osin...

    2022 arXiv

  11. [19]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  12. [20]

    Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. 2023 b . https://doi.org/10.18653/v1/2023.acl-long.792 Llm-blender: Ensembling large language models with pairwise ranking and generative fusion . In ACL

    2023 doi

  13. [21]

    Juraj Juraska, Mara Finkelstein, Daniel Deutsch, Aditya Siddhant, Mehdi Mirzazadeh, and Markus Freitag. 2023. https://aclanthology.org/2023.wmt-1.63 M etric X -23: The G oogle submission to the WMT 2023 metrics shared task . In Proceedings of the Eighth Conference on Machine T...

    2023

  14. [22]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. http://arxiv.org/abs/2001.08361 Scaling laws for neural language models . CoRR, abs/2001.08361

    2020 arXiv

  15. [23]

    Smith, and Hannaneh Hajishirzi

    Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Raghavi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. 2024. https://doi.org/10.48550/ARXIV.2403.13787 Rewardbench: Evaluating reward model...

    2024 doi arXiv

  16. [24]

    Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023 a . http://papers.nips.cc/paper\_files/paper/2023/hash/a3621ee907def47c1b952ade25c67698-Abstract-Conference.html CAMEL: communicative agents for "mind" exploration of large language model societ...

    2023

  17. [25]

    Gonzalez, and Ion Stoica

    Tianle Li, Wei - Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. 2024. http://arxiv.org/abs/2406.11939 From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline . CoRR, abs/2406.11939

    2024 arXiv

  18. [26]

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023 b . Alpacaeval: An automatic evaluator of instruction-following models

    2023

  19. [27]

    Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. 2024. https://aclanthology.org/2024.emnlp-main.992 Encouraging divergent thinking in large language models through multi-agent debate . In EMNLP

    2024

  20. [28]

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. https://openreview.net/forum?id=v8L0pN6EOi Let's verify step by step . In ICLR

    2024

  21. [29]

    Liu, and Jialu Liu

    Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J. Liu, and Jialu Liu. 2024. https://openreview.net/forum?id=xbjSwwrQOe Statistical rejection sampling improves preference optimization . In ICLR

    2024

  22. [30]

    Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. 2023. http://arxiv.org/abs/2310.02170 Dynamic llm-agent network: An llm-agent collaboration framework with agent team optimization . CoRR, abs/2310.02170

    2023 arXiv

  23. [31]

    Keming Lu, Hongyi Yuan, Runji Lin, Junyang Lin, Zheng Yuan, Chang Zhou, and Jingren Zhou. 2024. https://doi.org/10.18653/v1/2024.naacl-long.109 Routing to the expert: Efficient reward-guided ensemble of large language models . In NAACL

    2024 doi

  24. [32]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. http://papers.n...

    2023

  25. [33]

    Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. http://arxiv.org/abs/2405.14734 Simpo: Simple preference optimization with a reference-free reward . CoRR, abs/2405.14734

    2024 arXiv

  26. [34]

    Manning, Stefano Ermon, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html Direct preference optimization: Your language model is sec...

    2023

  27. [35]

    Treviso, Nuno Miguel Guerreiro, Chrysoula Zerva, Ana C

    Ricardo Rei, Marcos V. Treviso, Nuno Miguel Guerreiro, Chrysoula Zerva, Ana C. Farinha, Christine Maroti, Jos \' e G. C. de Souza, Taisiya Glushkova, Duarte M. Alves, Lu \' sa Coheur, Alon Lavie, and Andr \' e F. T. Martins. 2022. https://aclanthology.org/2022.wmt-1.60 Cometki...

    2022

  28. [36]

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. http://arxiv.org/abs/2408.03314 Scaling LLM test-time compute optimally can be more effective than scaling model parameters . CoRR, abs/2408.03314

    2024 arXiv

  29. [37]

    Ye Tian, Baolin Peng, Linfeng Song, Lifeng Jin, Dian Yu, Haitao Mi, and Dong Yu. 2024. http://arxiv.org/abs/2404.12253 Toward self-improvement of llms via imagination, searching, and criticizing . CoRR, abs/2404.12253

    2024 arXiv

  30. [38]

    Ziyu Wan, Xidong Feng, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. 2024. https://openreview.net/forum?id=C4OpREezgj Alphazero-like tree-search can guide large language model decoding and training . In ICML

    2024

  31. [39]

    Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. 2024 a . https://aclanthology.org/2024.findings-emnlp.620 Interpretable preferences via multi-objective reward modeling and mixture-of-experts . In Findings of EMNLP

    2024

  32. [40]

    Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. 2024 b . http://arxiv.org/abs/2406.04692 Mixture-of-agents enhances large language model capabilities . CoRR, abs/2406.04692

    2024 arXiv

  33. [41]

    Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. 2024. http://arxiv.org/abs/2408.00724 An empirical analysis of compute-optimal inference for problem-solving with language models . CoRR, abs/2408.00724

    2024 arXiv

  34. [42]

    Kai Xiong, Xiao Ding, Yixin Cao, Ting Liu, and Bing Qin. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.508 Examining inter-consistency of large language models collaboration: An in-depth analysis via debate . In Findings EMNLP

    2023 doi

  35. [43]

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. 2024 a . https://openreview.net/forum?id=CfXh93NDgH Wizard LM : Empowering large pre-trained language models to follow complex instructions . In ICLR

    2024

  36. [44]

    Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. 2024 b . https://openreview.net/forum?id=51iwkioZpn Contrastive preference optimization: Pushing the boundaries of LLM performance in machine translation . In ICML

    2024

  37. [45]

    Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. 2024 c . http://arxiv.org/abs/2406.08464 Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing . CoRR, abs/2406.08464

    2024 arXiv

  38. [46]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze...

    2024 arXiv

  39. [47]

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. 2024 b . http://arxiv.org/abs/2409.12122 Qwen2.5-math technical report...

    2024 arXiv

  40. [48]

    Hai Ye and Hwee Tou Ng. 2024. https://aclanthology.org/2024.emnlp-main.1206 Preference-guided reflective sampling for aligning language models . In EMNLP

    2024

  41. [49]

    Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yu...

    2024 arXiv

  42. [50]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  43. [51]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.