Pith. sign in

REVIEW 4 major objections 5 minor 25 references

The paper claims that whether diversity helps a multi-agent LLM team depends entirely on the quality of the mechanism that selects or blends their answers: below a crossover threshold diversity hurts, above it diversity helps, and in a 42-t

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 17:47 UTC pith:3KIRJWIG

load-bearing objection Real empirical contrast worth a referee, but the crossover threshold is fitted, not derived, and the synthesis claim rests on one implementation. the 4 major comments →

arxiv 2603.20324 v2 pith:3KIRJWIG submitted 2026-03-20 cs.MA cs.AI

When Agents Disagree: The Selection Bottleneck in Multi-Agent LLM Pipelines

Artem Maryanskyy , Dmitry Budnikov , Alibek T. Kaliyev This is my paper
classification cs.MA cs.AI
keywords multi-agent LLM pipelinesdiversityselection bottleneckaggregation qualitycrossover thresholdMixture-of-AgentsLLM-as-judgewin rate
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that conflicting results about whether diverse teams of language models outperform single models can be reconciled by one quantity: the quality of the mechanism that picks or blends their answers. It defines a selector-quality parameter s and proves there is a crossover threshold s* below which a diverse team's extra variance hurts and above which it helps. In a 42-task experiment, a judge that selects the best candidate beats a synthesis-style aggregator that blends all candidates in every task, with the blended outputs falling below a single-model baseline. If the model is right, practitioners should spend effort on better selectors rather than on adding more generators.

Core claim

The central claim is that aggregation, not generation, determines whether team diversity pays off in single-round multi-agent LLM pipelines. The paper models expected output quality as a linear blend of the team's average candidate quality and its best-candidate (oracle) quality, weighted by a selector-quality parameter s, and derives a closed-form threshold s* that separates the regime where diversity hurts from the regime where it helps. Empirically, with a diverse team and judge-based selection, win rates against a single-model baseline reach 0.810, while the same team under majority vote sits near chance (0.496) and under MoA-style synthesis falls to 0.179. Selection beats synthesis in a

What carries the argument

Selector quality s, defined as how much of the gap between the team mean and the team oracle a selector captures, and the crossover threshold s* = (µ_best − M)/(O − M) from Proposition 1. The paper's Assumption 1 — that Q(T, s) = s·O + (1 − s)·M — turns the qualitative claim 'diversity helps only with a good aggregator' into a testable prediction: diverse teams beat homogeneous ones exactly when s exceeds s*. Assumption 2 says homogeneous teams offer nothing to select among, which explains their near-chance performance.

Load-bearing premise

The load-bearing premise is Assumption 1, the linear interpolation Q(T, s) = s·O + (1 − s)·M, which the paper itself labels a modeling choice rather than a derived result; if real selectors do not interpolate linearly between team mean and oracle, the closed-form threshold and the regime split lose their precise meaning.

What would settle it

Run the same five-cell design with a multi-round synthesizer or a stronger synthesis model, and also measure Q(T, s) at several intermediate selector strengths; if a different synthesizer reaches or exceeds the judge-based cell's win rate, or if measured Q(T, s) deviates systematically from the straight line between M and O, the paper's central crossover claim would not survive.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • In single-round pipelines, replacing a synthesis aggregator with a judge-based selector should improve quality without changing the generator team.
  • Homogeneous teams of identical models provide essentially no multi-agent benefit; the diversity advantage requires distinct model families.
  • Majority voting can make a diverse team perform no better than a single model, because votes cannot exploit the best-candidate variance.
  • The exploratory weak-model result suggests that adding a cheaper, weaker model can raise quality and lower cost when selection is strong, but this is not pre-registered.
  • Aggregator quality, not just team composition, is the primary lever; synthesis may even produce outputs below the single-model baseline.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • We infer a direct engineering rule: estimate s* with a small pilot before committing to a multi-agent architecture; teams with weak selectors should stay homogeneous.
  • A natural next test is whether multi-round iterative synthesis raises effective s above the threshold; if it does, the 'selection beats synthesis' headline may be limited to one-shot aggregation.
  • The weak-model paradox, if replicated, suggests that cost-optimal teams may deliberately include weaker, diverse members rather than only frontier models — a reversal of usual scaling intuition.
  • The linearity assumption could be tested directly by sweeping judge strength; a nonlinear Q(T, s) would change the threshold but not necessarily the qualitative crossover.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a 'selection bottleneck' model for multi-agent LLM pipelines: output quality is Q(T,s) = s·O(T) + (1−s)·M(T) (Assumption 1), homogeneous teams have O≈M (Assumption 2), and synthesis acts at s≈0 (Assumption 3). Proposition 1 derives a crossover threshold s* below which diversity hurts and above which it helps. The empirical study crosses three team compositions with three selectors across 42 tasks (N=210), reporting that diverse+judge selection achieves BT-WR = 0.810, homogeneous+judge 0.512, majority vote 0.496, and MoA-style synthesis 0.179, with a decoupled evaluation preserving rank order (Spearman ρ = 0.90). An exploratory result claims that adding a weaker model improves win rate and lowers cost. The paper concludes that selector quality is a more impactful design lever than generator diversity in single-round generate-then-select pipelines.

Significance. If the empirical pattern holds, the paper addresses a genuine and practically important contradiction in the multi-agent LLM literature. The study's strengths are its pre-registered confirmatory contrasts, confidence intervals, robust regression specifications, a decoupled evaluation pass, and a public reproducibility package. The formal crossover threshold is a useful expository device, and the weak-model paradox is intriguing. However, the central selection-vs-synthesis claim rests on a single synthesis implementation, the decoupled evidence is only reported at the aggregate level, and the key quantity s is never directly measured; these limit the generality of the headline conclusion. The contribution is therefore significant but more provisional and narrower than the abstract suggests.

major comments (4)
  1. [§6.2, §6.5] The headline contrast, selection vs. synthesis (g = 3.86), operationalizes synthesis as a single-round aggregation by Claude Sonnet. Original MoA (Wang et al., ref. 22) is a layered, iterative architecture, and the paper concedes in §6.2 that multi-round iterative synthesis 'may yield different results.' As written, the claim that 'MoA-style synthesis' falls below the crossover threshold conflates this one implementation with synthesis as a mechanism. The central claim should be restricted to single-round synthesis with this aggregator, or additional synthesizers/rounds should be tested. This is load-bearing because the all-42-task superiority of selection is measured against this specific cell.
  2. [§4.4, Table 2] The decoupled evaluation shows large attenuation: the selection-vs-synthesis gap shrinks from Δ=0.631 on the original panel to Δ≈0.299 on the independent 2-judge panel (0.611 − 0.312). Yet the paper's strongest statement — 'selection wins in all 42 tasks' — is based on the original judge panel, which also performed the selection. The decoupled pass is reported only at the aggregate level (Spearman ρ = 0.90 over five cells), not per task. Without per-task decoupled results, the 'all 42 tasks' claim and the practical 'select, don't synthesize' rule are not supported by the independent evaluation. Please report the per-task decoupled comparison or explicitly qualify the claim.
  3. [Table 2, §4.6] There is an internal inconsistency in the homogeneous cell. Table 2 reports decoupled 3-judge WR = 0.000 with the footnote 'All pairwise verdicts are ties; independent judges cannot distinguish homogeneous outputs,' while §4.6 states ties are coded as 0.5 in the BT-WR computation. If all pairwise verdicts are ties, the BT-WR should be 0.500, not 0.000. The text in §4.4 says the cell 'yields WR = 0.500 under independent evaluation,' so the table entry appears to be an error. Please reconcile the table and the text, since this affects trust in the decoupled analysis.
  4. [§3.2, §5.4] The theoretical contribution is largely a parameterization rather than a falsifiable prediction. Assumption 1 is explicitly 'a modeling choice, not a derived result,' Assumption 2 encodes the homogeneous near-chance outcome that the paper later treats as evidence for the model, and the crossover threshold s* is fitted from an 8-task pilot (s* ≈ 0.567) and then used to classify the V4 regimes. Selector quality s is never directly measured. Consequently, the statement that vote operates below s*, synthesis far below, and judge above is a post-hoc calibration, not an out-of-sample confirmation. The model is a useful framework, but the paper should consistently describe the V4 results as qualitatively consistent with the calibrated model, not as independent confirmation of the threshold.
minor comments (5)
  1. [Abstract] The phrase 'the synthesis approach is preferred over the baseline in zero of 42 tasks' should be qualified as 'by the original judge panel'; the decoupled panel gives a non-zero win rate (0.312) for synthesis.
  2. [§5.1, Table 3] The low inter-rater agreement for the flagship cell (mean κ = 0.095 for div strong+judge) is discussed, but the reader would benefit from a sentence on how BT-WR behaves when judges strongly disagree on pairwise preferences but agree on the aggregate direction.
  3. [§5.2] Calling the weak-model effect 'statistically significant' while also labeling it exploratory is potentially confusing; consider 'statistically significant in this exploratory analysis' to avoid implying pre-registered confirmation.
  4. [Figure 1] Panel (b) plots empirical win rates on the same y-axis as the theoretical Q(T,s) but the x-axis positions are not measured values of s; adding an explicit 's not directly measured; cell placement is schematic' note would improve clarity.
  5. [§5.1] The sentence 'The judge wins in all 42 tasks and all 7 categories—consistent across all 42 tasks and 7 categories, with no exceptions' is redundant; trim to a single statement.

Circularity Check

3 steps flagged

Model 'predictions' for homogeneous teams and synthesis are assumptions fitted to the same data; crossover classification relabels observed win rates, while the main empirical contrast has independent decoupled support.

specific steps
  1. self definitional [§3.2 (Assumption 2); §5.1 ('The 0.500 phenomenon')]
    "This assumption is motivated by the observation that same-model copies at moderate temperature produce functionally identical outputs—our decoupled evaluation confirms this: all 756 pairwise verdicts from independent judges were ties for the homogeneous cell (Section 4.4). ... The near-chance result (WR = 0.512) indicates that within-model sampling variation provides negligible material for selection to exploit, confirming that cross-model diversity—not merely multiple candidates—drives the diversity advantage."

    Assumption 2 is chosen because the homogeneous cell's outputs were observed to be indistinguishable, so the model's homogeneous-team quality is set to the observed near-chance value. The same WR = 0.512 observation is then reported as 'confirming' the model. The 'prediction' that selector quality has no effect for homogeneous teams is the input observation relabeled, not an independent consequence.

  2. fitted input called prediction [§3.3 (Assumption 3 and Remark 3); §6.2 ('Why Synthesis Fails')]
    "A synthesis-based aggregator that blends all candidates into a single output, rather than selecting among them, operates at effective selector quality s_synth ≈ 0. ... The observed synthesis win rate of 0.179 (Section 5.1) is consistent with s_synth ≤ 0 under our model. ... In the selection bottleneck model, this is exactly what Assumption 3 predicts: when s≈0 (synthesis has no selection capacity), diversity has no mechanism to help."

    The synthesis result is not a prediction: the model stipulates s_synth ≈ 0, and then the observed WR = 0.179 is used to say the assumption is 'consistent.' Since s_synth ≤ 0 is inferred from the very outcome it is supposed to explain, the 'confirmation' reduces to fitting the parameter to the result. Calling this a prediction of Assumption 3 treats an input as an output.

  3. renaming known result [§5.4 (Calibrating the Crossover Threshold)]
    "The five V4 cells confirm the qualitative predictions of Proposition 1. Judge-based selection (BT-WR = 0.810) operates well above the crossover threshold; majority vote (0.496) sits at or below it; and synthesis (0.179) falls far below. ... Monte Carlo calibration on a pilot study (8 tasks, N=136) yielded s*≈0.567 ... The V4 scaled experiment is qualitatively consistent: the estimated selector qualities place vote below and judge above this threshold."

    Because s* is the model point where output quality equals the homogeneous baseline, any cell with observed win rate above baseline is arithmetically 'above s*' and any cell below baseline is 'below s*' under the linear model. Without a separate measurement of selector quality s, the V4 'confirmation' is a relabeling of the empirical ordering (synth < vote < baseline < judge). The pilot-calibrated threshold is not independently tested by V4.

full rationale

The paper's main empirical claim — that judge-based selection beats single-round MoA synthesis (Δ_WR=+0.631, g=3.86) and wins all 42 tasks — is not definitionally forced: Table 2's decoupled 2-judge panel preserves the ordering (Spearman ρ=0.90) even though magnitudes attenuate by 53–67%. The paper itself flags this as 'consistent with partial circularity in the original design' (§4.4), and I weigh that as measurement overlap rather than a derivation collapse. The formal model, however, contains genuine circular steps: Assumption 2 is justified by the homogeneous-cell ties it later 'predicts'; Assumption 3 stipulates s_synth≈0 and then the synthesis WR=0.179 is presented as confirming it; and §5.4's crossover 'confirmation' is a relabeling of observed win rates using a pilot-calibrated s*. Assumption 1 is explicitly 'a modeling choice, not a derived result,' so the closed-form threshold is an algebraic consequence of the linear ansatz rather than an independent derivation; I do not separately score that because the paper is candid about it. No load-bearing self-citations were found; the cited prior work is external. Overall: partial circularity concentrated in the formal model's confirmatory framing, while the headline empirical contrast retains independent support.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 1 invented entities

The paper's formal contributions rest on three explicit modeling assumptions, none derived from first principles; the empirical central claim rests on LLM-as-judge validity and task representativeness. The crossover threshold is a fitted/algebraic quantity rather than a measured constant.

free parameters (2)
  • s* (crossover threshold) = ≈0.567 (bootstrap 95% CI 0.48–0.65, B=10,000)
    Estimated by Monte Carlo calibration on a pilot study (8 tasks, N=136) in §5.4; the paper then uses it to partition V4 cells into above/below crossover. This is a fitted constant, not derived from independent data.
  • s (effective selector quality per cell) = not directly estimated
    Defined in Eq. (1) from latent quality distributions; never measured. The paper qualitatively assigns cells above/near/below s* from win rates rather than estimating s, so the model's central moderator is not independently quantified.
axioms (5)
  • ad hoc to paper Assumption 1: Q(T,s)=s·O(T)+(1−s)·M(T)
    Stated in §3.2 as a modeling choice, not derived; the crossover threshold s* follows algebraically from it.
  • ad hoc to paper Assumption 2: homogeneous teams have O≈M, so Q≈μ for all s
    Justified in §3.2 by observed indistinguishability of same-model outputs and then treated as an explanation of the homogeneous cell's near-chance WR—circularity risk.
  • ad hoc to paper Assumption 3: synthesis operates at s≈0
    §3.3 postulates synthesis is non-selective; the low synthesis WR is then read as confirming this assumption.
  • domain assumption LLM judge pairwise preferences approximate output quality
    All BT-WR values depend on judge preferences; decoupled evaluation partially checks this but judges remain LLMs with documented biases (referenced in §2.5).
  • domain assumption Tasks are representative of multi-agent use cases
    42 tasks across 7 categories; authors note in §6.5 that this demonstrates existence, not magnitude across all tasks.
invented entities (1)
  • selector quality s (latent) no independent evidence
    purpose: Central moderator in the selection bottleneck model indexing aggregation quality; the crossover threshold s* is defined from it.
    s is defined in Eq. (1) through latent qualities that are not observed; no direct measurement or falsifiable handle is provided beyond win rates interpreted through the model.

pith-pipeline@v1.3.0-alltime-deepseek · 15408 in / 14400 out tokens · 138814 ms · 2026-08-02T17:47:59.411444+00:00 · methodology

0 comments
read the original abstract

Multi-agent LLM pipelines produce contradictory evidence on whether team diversity improves output quality: heterogeneous Mixture-of-Agents teams outperform single models, yet homogeneous Self-MoA teams consistently win under synthesis-based aggregation. We propose a resolution by identifying the selection bottleneck -- a crossover threshold in aggregation quality that determines whether diversity helps or hurts. Under this model, we obtain a closed-form crossover threshold $s^*$ (Proposition 1) that separates the regimes where diversity helps and hurts. In a targeted experiment spanning 42 tasks across 7 categories ($N=210$), a diverse team with judge-based selection achieves a win rate of 0.810 against a single-model baseline, while a homogeneous team scores 0.512 -- near chance (Glass's $\Delta = 2.07$). Judge-based selection outperforms MoA-style synthesis by $\Delta_{\mathrm{WR}} = +0.631$ -- the synthesis approach is preferred over the baseline in zero of 42 tasks by the judge panel. A decoupled evaluation with independent judges confirms all directional findings (Spearman $\rho = 0.90$). Exploratory evidence suggests that including a weaker model improves performance while reducing cost ($p < 10^{-4}$, not pre-registered). Our results suggest that selector quality may be a more impactful design lever than generator diversity in single-round generate-then-select pipelines.

Figures

Figures reproduced from arXiv: 2603.20324 by Alibek T. Kaliyev, Artem Maryanskyy, Dmitry Budnikov.

Figure 1
Figure 1. Figure 1: The selection bottleneck. (a) Theoretical output quality Q(T, s) as a function of selector quality s. The diverse team (blue) crosses the homogeneous baseline (gray) at the crossover threshold s ∗ (dashed vertical). Below s ∗ , diversity hurts; above it, diversity helps. (b) Empirical operating regimes from V4 data. Judge-based selection operates well above s ∗ (WR = 0.810), majority vote sits near s ∗ (WR… view at source ↗
Figure 2
Figure 2. Figure 2: BT-corrected consensus win rates for all five experimental cells, averaged over 42 tasks. High performance appears only when diversity and judge-based selection are combined. The synthesis cell (MoA) performs worst, falling well below the single-model baseline [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Per-task diversity advantage (∆ WR = diverse strong+judge minus homo opus+judge) across all 42 tasks. Positive values favor diversity. 38 of 42 tasks show a positive effect (4 ties, 0 negative; Clopper–Pearson 95% CI: [0.774, 0.973]). Error bars are approximate 95% CIs. ∆ = +0.119 [0.060, 0.179], Hedges’ g = 0.87 [0.42, 1.32], p = 1.18 × 10−4 . While statistically significant, this comparison was not pre￾r… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

25 extracted references · 1 canonical work pages

  1. [1]

    Barr, Roger Levy, Christoph Scheepers, and Harry J

    Dale J. Barr, Roger Levy, Christoph Scheepers, and Harry J. Tily. Random effects structure for confirmatory hypothesis testing: Keep it maximal.Journal of Memory and Language, 68(3):255–278, 2013. doi: 10.1016/j.jml. 2012.11.001

    work page doi:10.1016/j.jml 2013

  2. [2]

    Ralph Allan Bradley and Milton E. Terry. Rank analy- sis of incomplete block designs: I. the method of paired comparisons.Biometrika, 39(3/4):324–345, 1952. doi: 10.2307/2334029

    doi:10.2307/2334029 1952

  3. [3]

    LLMSelector: Se- lecting the right LLM for any task.arXiv preprint arXiv:2502.14815, 2025

    Jiangjie Chen, Xintao Wang, Rui Xu, Siyu Yuan, Lida Chen, and Yanghua Xiao. LLMSelector: Se- lecting the right LLM for any task.arXiv preprint arXiv:2502.14815, 2025. URLhttps://arxiv. org/abs/2502.14815

    Pith/arXiv arXiv 2025

  4. [4]

    Gon- zalez, and Ion Stoica

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anasta- sios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gon- zalez, and Ion Stoica. Chatbot arena: An open plat- form for evaluating LLMs by human preference.arXiv preprint arXiv:2403.04132, 2024. URLhttps:// arxiv.org/abs/2403.04132

    Pith/arXiv arXiv 2024

  5. [5]

    Iden- tity bias in large language model debate.arXiv preprint arXiv:2510.07517, 2025

    Jinheon Choi, Seungbeen Lee, and Jungseul Ok. Iden- tity bias in large language model debate.arXiv preprint arXiv:2510.07517, 2025. URLhttps://arxiv. org/abs/2510.07517

    Pith/arXiv arXiv 2025

  6. [6]

    Imprimerie Royale, Paris, 1785

    Marquis de Condorcet.Essai sur l’application de l’analyse `a la probabilit´e des d´ecisions rendues `a la plu- ralit´e des voix. Imprimerie Royale, Paris, 1785

  7. [7]

    Tenenbaum, and Igor Mordatch

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factual- ity and reasoning in language models through multia- gent debate. InProceedings of the 40th International Conference on Machine Learning (ICML), 2023. URL https://arxiv.org/abs/2305.14325

    Pith/arXiv arXiv 2023

  8. [8]

    Lu Hong and Scott E. Page. Groups of diverse problem solvers can outperform groups of high-ability problem solvers.Proceedings of the National Academy of Sci- ences, 101(46):16385–16389, 2004. doi: 10.1073/pnas. 0403723101

    doi:10.1073/pnas 2004

  9. [9]

    MetaGPT: Meta programming for a multi-agent collaborative framework

    Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, and Chenglin Wu. MetaGPT: Meta programming for a multi-agent collaborative framework. arXiv preprint arXiv:2308.00352, 2023. URLhttps: //arxiv.org/abs/2308.00352

    Pith/arXiv arXiv 2023

  10. [10]

    LLM- Blender: Ensembling large language models with pair- wise ranking and generative fusion

    Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LLM- Blender: Ensembling large language models with pair- wise ranking and generative fusion. InProceedings of the 61st Annual Meeting of the Association for Com- putational Linguistics (ACL), 2023. URLhttps:// arxiv.org/abs/2306.02561

    Pith/arXiv arXiv 2023

  11. [11]

    Krishna K. Ladha. The Condorcet jury theorem, free speech, and correlated votes.American Journal of Po- litical Science, 36(3):617–634, 1992. doi: 10.2307/ 2111584

    1992

  12. [12]

    CAMEL: Communicative agents for “mind” explo- ration of large language model society.arXiv preprint arXiv:2303.17760, 2023

    Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. CAMEL: Communicative agents for “mind” explo- ration of large language model society.arXiv preprint arXiv:2303.17760, 2023. URLhttps://arxiv. org/abs/2303.17760

    Pith/arXiv arXiv 2023

  13. [13]

    More agents is all you need.arXiv preprint arXiv:2402.05120, 2024

    Junyou Li, Qin Zhang, Yangbin Yu, Qiang Fu, and De- heng Ye. More agents is all you need.arXiv preprint arXiv:2402.05120, 2024. URLhttps://arxiv. org/abs/2402.05120

    Pith/arXiv arXiv 2024

  14. [14]

    More agents is all you need: Self-MoA outperforms mixed-MoA.arXiv preprint arXiv:2502.00674, 2025

    Xuechen Li, Linjun Zhang, Zhenyu Zhang, Yuqing Yang, and Zhangyang Wang. More agents is all you need: Self-MoA outperforms mixed-MoA.arXiv preprint arXiv:2502.00674, 2025. URLhttps://arxiv. org/abs/2502.00674

    Pith/arXiv arXiv 2025

  15. [15]

    Bowman, and Shi Feng

    Nina Panickssery, Samuel R. Bowman, and Shi Feng. LLM evaluators recognize and favor their own gener- ations.arXiv preprint arXiv:2404.13076, 2024. URL https://arxiv.org/abs/2404.13076

    Pith/arXiv arXiv 2024

  16. [16]

    When mod- els think alike: The limits of multi-agent debate.arXiv preprint arXiv:2311.17371, 2023

    Kris Smit, Ian Keane, and Weiquan Mao. When mod- els think alike: The limits of multi-agent debate.arXiv preprint arXiv:2311.17371, 2023. URLhttps:// arxiv.org/abs/2311.17371

    Pith/arXiv arXiv 2023

  17. [17]

    Scaling LLM test-time compute optimally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314, 2024

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Ku- mar. Scaling LLM test-time compute optimally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314, 2024. URLhttps:// arxiv.org/abs/2408.03314

    Pith/arXiv arXiv 2024

  18. [18]

    Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul Christiano

    Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. InAdvances in Neural Infor- mation Processing Systems (NeurIPS), volume 33, 2020. URLhttps://arxiv.org/abs/2009.01325

    Pith/arXiv arXiv 2020

  19. [19]

    Large language models are inconsistent and bi- ased evaluators.arXiv preprint arXiv:2405.01724, 2024

    Rickard Stureborg, Dimitris Alikaniotis, and Yoshi Suhara. Large language models are inconsistent and bi- ased evaluators.arXiv preprint arXiv:2405.01724, 2024. URLhttps://arxiv.org/abs/2405.01724

    Pith/arXiv arXiv 2024

  20. [20]

    The value of diversity in multi-agent systems.arXiv preprint arXiv:2602.07186, 2026

    Zhiwei Tang, Dmitrii Kharlapenko, Bonnie Li, Fengyu Li, Haifeng Chen, and Deng Cai. The value of diversity in multi-agent systems.arXiv preprint arXiv:2602.07186, 2026. URLhttps://arxiv. org/abs/2602.07186. 11

    arXiv 2026

  21. [21]

    Re- placing judges with juries: Evaluating LLM genera- tions with a panel of diverse models.arXiv preprint arXiv:2404.18796, 2024

    Pat Verga, Sebastian Hofst ¨atter, Sophia Althammer, Yix- uan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. Re- placing judges with juries: Evaluating LLM genera- tions with a panel of diverse models.arXiv preprint arXiv:2404.18796, 2024. URLhttps://arxiv. org/abs/2404.18796

    Pith/arXiv arXiv 2024

  22. [22]

    Mixture-of-agents yields state-of-the- art on AlpacaEval 2.0, MT-Bench, and FLASK.arXiv preprint arXiv:2406.04692, 2024

    Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. Mixture-of-agents yields state-of-the- art on AlpacaEval 2.0, MT-Bench, and FLASK.arXiv preprint arXiv:2406.04692, 2024. URLhttps:// arxiv.org/abs/2406.04692

    Pith/arXiv arXiv 2024

  23. [23]

    Self-consistency improves chain of thought reasoning in language models

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. InProceedings of the International Conference on Learning Representations (ICLR), 2023. URLhttps://arxiv.org/abs/ 2203.11171

    Pith/arXiv arXiv 2023

  24. [24]

    White, Doug Burger, and Chi Wang

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W. White, Doug Burger, and Chi Wang. Auto- Gen: Enabling next-gen LLM applications via multi- agent conversation.arXiv preprint arXiv:2308.08155, 2023

    Pith/arXiv arXiv 2023

  25. [25]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuo- han Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. InAdvances in Neural Information Processing Systems (NeurIPS), vol- ume 36, 2023. URLhttps://arxiv.org/abs/ 2306.05685. 12

    Pith/arXiv arXiv 2023