pith. sign in

arxiv: 2606.27288 · v1 · pith:ME4AXOJVnew · submitted 2026-06-25 · 💻 cs.AI · cs.LG

When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models

Pith reviewed 2026-06-26 04:16 UTC · model grok-4.3

classification 💻 cs.AI cs.LG
keywords LLM ensemblesmodel routingvotingmixture of agentsco-failure rateerror correlationfrontier modelsaccuracy bounds
0
0 comments X

The pith

Any policy selecting one model answer from an ensemble cannot exceed accuracy of one minus the rate at which all models fail together.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows that ensembles outputting exactly one member answer face a hard ceiling of 1 minus beta, where beta is the fraction of queries on which every model errs. Pairwise error correlations cannot identify this beta because multiple joint distributions can share the same marginals and pairwise terms yet differ in the all-wrong probability. Measurements across 67 frontier models from 21 providers find beta of 0.052 on open-ended math and 0.079 on code execution, exceeding single-factor and Gaussian-copula predictions. On these tasks, low-correlation heterogeneous sets still rarely surpass the single best model without strong per-query routing. Re-asking questions in free-response format raises beta, locating part of the co-failure in answer format rather than domain knowledge.

Core claim

For any policy whose output is one member model answer, accuracy cannot exceed one minus beta, where beta is the rate at which every model is wrong on the same query. The usual diagnostic of average pairwise error correlation rho cannot identify beta. A Clopper-Pearson bound on beta supplies a finite-sample certificate on the largest gain any router, vote, or cascade could deliver. Across 67 models, observed beta exceeds copula predictions, and gains appear only when models fail on different questions and routing exploits that difference.

What carries the argument

The co-failure rate beta, the probability that all models in the ensemble err simultaneously on a given query.

If this is right

  • Routers, voters, and cascades cannot exceed 1-beta no matter how they choose among member answers.
  • Identical pairwise correlations can hide different beta values, so rho alone does not certify ensemble headroom.
  • On checkable tasks, adding models without query-level routing rarely beats the single best model.
  • Changing answer format from multiple-choice to free-response can reopen the co-failure tail even on the same questions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Systems that synthesize new answers or add external verification steps can in principle exceed the 1-beta ceiling that applies to pure selection policies.
  • Better tail modeling beyond Gaussian copulas would be needed to predict the largest achievable gains before training routers.
  • Reducing simultaneous failures across models may matter more for ensemble performance than further lowering average pairwise correlation.

Load-bearing premise

The policy must output exactly one of the member models' answers rather than synthesizing a new answer or using external verification.

What would settle it

An ensemble policy that selects one member answer and achieves accuracy higher than one minus the empirically measured beta on the same query set would contradict the bound.

Figures

Figures reproduced from arXiv: 2606.27288 by Josef Chen.

Figure 1
Figure 1. Figure 1: Orchestration is allocation. A query is routed, cascaded, or fused over a priced, correlated, fast-churning pool of 67 frontier-to-cheap models; the optimal policy is a per-type bang-per-buck rule with a single shadow price λB on the inference dollar (Prop. 4). No selection policy can exceed the ceiling 1 − β set by the rate β at which all models fail at once (Prop. 1). The field reports pairwise error cor… view at source ↗
Figure 2
Figure 2. Figure 2: The co-failure residual is a common-mode atom, not copula misspecification (MATH-500, [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Format, not content, sets the regime. The same 79 GPQA-Diamond questions, asked multiple-choice (top) and free-response (bottom; options stripped, 5-judge panel, κ 0.73–0.92). Each cell is one question; an orange cell is one on which every model is wrong. Changing only the format opens a co-failure block of 10/79 (β=0.127, CP[0.062, 0.220]) where multiple-choice had none (β ≈ 0), while mean accuracy falls … view at source ↗
Figure 4
Figure 4. Figure 4: Two regimes across the domains. All-models-wrong rate β per domain with 95% Clopper–Pearson intervals, recomputed from the committed outcome matrices. Ceiling-bound domains (open-ended math and code, free-response GPQA) carry a co-failure tail β > 0 that caps every selection policy at 1 − β and that pairwise ρ underprices by 2.5–8.3× (tetrachoric); realizability-bound domains (multiple-choice GPQA, MMLU-Pr… view at source ↗
Figure 5
Figure 5. Figure 5: Pillar A cost–quality frontier (re￾graded): the per-query oracle (green) sits just above single-best (black); cheap models populate the frontier. 0.9 0.8 0.7 0.6 0.5 verifier AUC (error-detector) 0.00 0.02 0.04 0.06 0.08 0.10 0.12 cascade advantage over random mixing AUC = 1/2 (collapse to random mixing) advantage over random mixing vanishes as the verifier degrades to chance (Pillar C) Cascade collapse id… view at source ↗
Figure 8
Figure 8. Figure 8: Equal-quality break-even (S=9, matched 6-model band): low-ρ heterogeneous fusion (ρ=0.42) vs. high-ρ Self-MoA (ρ=0.80), both on distinct draws so Self-MoA scales with￾out recycling samples. Under the pre-registered distinct-draw aggregation the heterogeneous en￾semble beats Self-MoA from the information-fair k=3 onward (∗: query-bootstrap 95% CI excludes zero), supporting the diversification mechanism at m… view at source ↗
Figure 9
Figure 9. Figure 9: Pillar A, realizable routing: a held-out learned router (TF-IDF+domain) vs. the cost-aware [PITH_FULL_IMAGE:figures/full_fig_p014_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Pillar B, k ∗ (ρ) at matched quality: across 57 sub-bands of the matched 6-model pool (MMLU-Pro), majority-vote gain over the best member vs. inter-model ρ, controlling for member accuracy. The slope is negative (diversification-limit direction; CI spans zero), reversing the positive slope in the unmatched pool ( [PITH_FULL_IMAGE:figures/full_fig_p020_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Optionality under churn (secondary). Frontier accuracy on MMLU-Pro over 2024–2026 [PITH_FULL_IMAGE:figures/full_fig_p026_11.png] view at source ↗
read the original abstract

Multi-model LLM systems such as routing, voting, cascades, fusion, and mixture-of-agents are used to beat single-model accuracy. We show that their gain is capped by a quantity the field rarely reports. For any policy whose output is one member model answer, accuracy cannot exceed one minus beta, where beta is the rate at which every model is wrong on the same query. In contrast, the usual diagnostic, average pairwise error correlation rho, cannot identify beta: error laws with identical marginals and pairwise correlations can have different all-wrong rates. A Clopper-Pearson bound on beta gives a finite-sample certificate on the largest gain any router, vote, or cascade could deliver before training a router. Across 67 models from 21 providers, a tetrachoric-calibrated single-factor model still underprices the all-wrong tail: on open-ended mathematics, observed beta is 0.052 versus 0.023 under the full 67-model Gaussian copula, about 2.5 times underpricing, with 90 percent CI 1.7 to 3.4 and k equals 17. The effect recurs on execution-graded code, where beta is 0.079. Re-asking the same GPQA-Diamond questions in free-response rather than multiple-choice form reopens the tail, with beta 0.127 and a five-judge panel with kappa 0.73 to 0.92, locating co-failure in answer format rather than subject. At matched quality, low-rho heterogeneous ensembles beat high-rho Self-MoA, but on checkable tasks in our pool, combining models rarely beats the single best model without a strong query-level routing signal. Gains come from models failing on different questions, not from adding more models.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The manuscript defines β as the probability that every model in an ensemble is simultaneously wrong on a given query and proves that any policy whose output must be exactly one of the member models' answers cannot exceed accuracy 1−β. It shows that the conventional diagnostic of average pairwise error correlation ρ is insufficient to identify β, supplies a Clopper-Pearson finite-sample upper bound on β, and reports that across 67 frontier models a Gaussian-copula model underestimates observed β (0.052 vs. 0.023 on open-ended math; 0.079 on code; 0.127 on free-response GPQA). The paper concludes that, on checkable tasks, gains from combining models are driven by query-level routing rather than by adding models or lowering ρ.

Significance. If the logical bound is accepted and the empirical β measurements prove robust, the work supplies a simple, reportable quantity that places a hard ceiling on the upside of any selection-based router, vote, or cascade before any training occurs. The demonstration that identical marginal error rates and pairwise ρ can produce materially different β values is a clarifying observation for the ensemble literature. The scale of the study (67 models, multiple task types, execution grading) lends weight to the claim that co-failure tails are heavier than low-dimensional factor models predict. These elements constitute a constructive, falsifiable contribution to understanding the limits of multi-LLM systems.

major comments (2)
  1. [Abstract / Title] Abstract and title: The central bound is derived only for policies that output exactly one member model's answer, yet the title and abstract list 'routing, voting, cascades, fusion, and mixture-of-agents' as the systems under study and state that 'their gain is capped.' Mixture-of-Agents and fusion frameworks commonly synthesize a new token sequence rather than emit a raw member output; on queries where all members fail, such synthesis can still be correct. The manuscript must either restrict the title/abstract/claim to selection policies or derive/qualify the bound for synthesis methods.
  2. [Empirical results (math / code)] § on Gaussian-copula comparison (math and code results): The statement that a 'tetrachoric-calibrated single-factor model still underprices the all-wrong tail' (observed β = 0.052 vs. 0.023) requires an explicit statement of how the copula parameters are estimated, whether the fitted marginals exactly match the empirical per-model error rates, and whether the 90 % CI (1.7–3.4) accounts for dependence across the 67 models. Without these details the factor-model underpricing claim cannot be evaluated.
minor comments (1)
  1. [Abstract] The abstract states 'k equals 17' without defining k; the main text should state what this integer represents (e.g., number of models or a hyper-parameter).

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for these precise comments on scope and methodological transparency. Both points are addressable by targeted revisions that preserve the paper's central claims and results.

read point-by-point responses
  1. Referee: [Abstract / Title] Abstract and title: The central bound is derived only for policies that output exactly one member model's answer, yet the title and abstract list 'routing, voting, cascades, fusion, and mixture-of-agents' as the systems under study and state that 'their gain is capped.' Mixture-of-Agents and fusion frameworks commonly synthesize a new token sequence rather than emit a raw member output; on queries where all members fail, such synthesis can still be correct. The manuscript must either restrict the title/abstract/claim to selection policies or derive/qualify the bound for synthesis methods.

    Authors: We agree the 1−β bound applies strictly to selection policies. The body already qualifies the claim accordingly. We will revise the title and abstract to read 'selection-based ensembles (routing, voting, cascades)' and insert a clarifying sentence noting that synthesis methods such as Mixture-of-Agents can in principle exceed the bound by generating novel outputs. This aligns front matter with the theorems without altering any proofs or experiments. revision: yes

  2. Referee: [Empirical results (math / code)] § on Gaussian-copula comparison (math and code results): The statement that a 'tetrachoric-calibrated single-factor model still underprices the all-wrong tail' (observed β = 0.052 vs. 0.023) requires an explicit statement of how the copula parameters are estimated, whether the fitted marginals exactly match the empirical per-model error rates, and whether the 90 % CI (1.7–3.4) accounts for dependence across the 67 models. Without these details the factor-model underpricing claim cannot be evaluated.

    Authors: We will add a dedicated paragraph in the methods section stating: pairwise tetrachoric correlations are estimated directly from the 67×query error-indicator matrix; marginal error probabilities are fixed exactly to the empirical per-model rates; the single-factor Gaussian copula is then sampled to obtain the model-predicted β. The 90 % CI on the ratio (observed β / copula β) is a nonparametric bootstrap over queries and does not incorporate cross-model dependence; we will explicitly note this limitation. These additions make the underpricing claim (factor ≈2.5 on math) fully evaluable from the released data. revision: yes

Circularity Check

0 steps flagged

No circularity: bound is direct logical consequence of beta definition and single-output restriction

full rationale

The paper's core claim states that for policies outputting exactly one member model answer, accuracy ≤ 1 − β where β is the all-wrong rate. This follows immediately from the definitions without any fitting, parameter estimation, or reduction to prior self-citations. The abstract explicitly qualifies the bound to 'any policy whose output is one member model answer,' distinguishing it from synthesis methods. No equations or steps in the provided text reduce the result to its inputs by construction. The scope mismatch with MoA/fusion noted by the skeptic is a potential overgeneralization in presentation but does not constitute circularity in the derivation itself. The analysis is self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the definitional bound and the single-output policy restriction; the Clopper-Pearson bound is a standard statistical tool with no free parameters introduced for the main result.

axioms (1)
  • standard math Clopper-Pearson bound applies to estimating the binomial proportion beta from finite samples
    Invoked to provide a finite-sample certificate on the largest possible gain from any router or vote.

pith-pipeline@v0.9.1-grok · 5867 in / 1363 out tokens · 61404 ms · 2026-06-26T04:16:29.088449+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

41 extracted references · 15 canonical work pages · 4 internal anchors

  1. [1]

    Tsitsiklis.Introduction to Linear Optimization

    Dimitris Bertsimas and John N. Tsitsiklis.Introduction to Linear Optimization. Athena Scientific, 1997

  2. [2]

    FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance

    Lingjiao Chen, Matei Zaharia, and James Zou. FrugalGPT: How to use large language models while reducing cost and improving performance.arXiv preprint arXiv:2305.05176, 2023

  3. [3]

    Dekoninck, M

    Jasper Dekoninck, Maximilian Baader, and Martin Vechev. A unified approach to routing and cascading for LLMs.arXiv preprint arXiv:2410.10347, 2024

  4. [4]

    Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor R¨ uhle, Laks V. S. Lakshmanan, and Ahmed Hassan Awadallah. Hybrid LLM: Cost-efficient and quality-aware query routing. InInternational Conference on Learning Representations (ICLR), 2024

  5. [5]

    Dixit and Robert S

    Avinash K. Dixit and Robert S. Pindyck.Investment Under Uncertainty. Princeton University Press, 1994

  6. [6]

    The devil is in the tails: actuarial mathematics and the subprime mortgage crisis.ASTIN Bulletin, 40(1):1–33, 2010

    Catherine Donnelly and Paul Embrechts. The devil is in the tails: actuarial mathematics and the subprime mortgage crisis.ASTIN Bulletin, 40(1):1–33, 2010

  7. [7]

    McNeil, and Daniel Straumann

    Paul Embrechts, Alexander J. McNeil, and Daniel Straumann. Correlation and dependence in risk management: properties and pitfalls. InRisk Management: Value at Risk and Beyond, pages 176–223. Cambridge University Press, 2002

  8. [8]

    Cost-of-pass: An economic framework for evaluating language models.arXiv preprint arXiv:2504.13359, 2025

    Mehmet Hamza Erol, Batu El, Mirac Suzgun, et al. Cost-of-pass: An economic framework for evaluating language models.arXiv preprint arXiv:2504.13359, 2025

  9. [9]

    Representation of multivariate bernoulli distributions with a given set of specified moments.Journal of Multivariate Analysis, 168:290–303, 2018

    Roberto Fontana and Patrizia Semeraro. Representation of multivariate bernoulli distributions with a given set of specified moments.Journal of Multivariate Analysis, 168:290–303, 2018

  10. [10]

    Selective classification for deep neural networks

    Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2017

  11. [11]

    Ronald A. Howard. Information value theory.IEEE Transactions on Systems Science and Cybernetics, 2(1):22–26, 1966

  12. [12]

    RouterBench: A Benchmark for Multi-LLM Routing System

    Qitian Jason Hu, Jacob Bieker, et al. RouterBench: A benchmark for multi-LLM routing system.arXiv preprint arXiv:2403.12031, 2024

  13. [13]

    arXiv preprint arXiv:2306.02561 (2023)

    Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LLM-Blender: Ensembling large language models with pairwise ranking and generative fusion. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), 2023. arXiv:2306.02561

  14. [14]

    When does confidence-based cascade deferral suffice? InAdvances in Neural Information Processing Systems (NeurIPS), 2023

    Wittawat Jitkrittum, Neha Gupta, Aditya Krishna Menon, Harikrishna Narasimhan, Ankit Singh Rawat, and Sanjiv Kumar. When does confidence-based cascade deferral suffice? InAdvances in Neural Information Processing Systems (NeurIPS), 2023. arXiv:2307.02764. 16

  15. [15]

    Correlated errors in large language models

    Elliot Kim, Avi Garg, Kenny Peng, and Nikhil Garg. Correlated errors in large language models. InInternational Conference on Machine Learning (ICML), 2025. arXiv:2506.07962

  16. [16]

    Algorithmic monoculture and social welfare.Proceedings of the National Academy of Sciences, 118(22):e2018340118, 2021

    Jon Kleinberg and Manish Raghavan. Algorithmic monoculture and social welfare.Proceedings of the National Academy of Sciences, 118(22):e2018340118, 2021

  17. [17]

    Competition when consumers have switching costs.The Review of Economic Studies, 62(4):515–539, 1995

    Paul Klemperer. Competition when consumers have switching costs.The Review of Economic Studies, 62(4):515–539, 1995

  18. [18]

    Neural network ensembles, cross validation, and active learning

    Anders Krogh and Jesper Vedelsby. Neural network ensembles, cross validation, and active learning. InAdvances in Neural Information Processing Systems (NeurIPS), volume 7, pages 231–238, 1995

  19. [19]

    Kuncheva.Combining Pattern Classifiers: Methods and Algorithms

    Ludmila I. Kuncheva.Combining Pattern Classifiers: Methods and Algorithms. Wiley, 2004

  20. [20]

    Kuncheva and Christopher J

    Ludmila I. Kuncheva and Christopher J. Whitaker. Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy.Machine Learning, 51(2):181–207, 2003

  21. [21]

    Kuncheva, Christopher J

    Ludmila I. Kuncheva, Christopher J. Whitaker, Catherine A. Shipp, and Robert P. W. Duin. Limits on the majority vote accuracy in classifier fusion.Pattern Analysis & Applications, 6(1): 22–31, 2003

  22. [22]

    David X. Li. On default correlation: A copula function approach.Journal of Fixed Income, 9 (4):43–54, 2000

  23. [23]

    arXiv preprint arXiv:2402.05120 , year=

    Junyou Li, Qin Zhang, Yangbin Yu, Qiang Fu, and Deheng Ye. More agents is all you need. Transactions on Machine Learning Research, 2024. arXiv:2402.05120

  24. [24]

    arXiv preprint arXiv:2502.00674 , year =

    Wenzhe Li, Yong Lin, Mengzhou Xia, and Chi Jin. Rethinking mixture-of-agents: Is mixing different large language models beneficial?arXiv preprint arXiv:2502.00674, 2025

  25. [25]

    AutoMix: Automatically mixing language models

    Aman Madaan, Pranjal Aggarwal, et al. AutoMix: Automatically mixing language models. arXiv preprint arXiv:2310.12963, 2023

  26. [26]

    Portfolio selection.The Journal of Finance, 7(1):77–91, 1952

    Harry Markowitz. Portfolio selection.The Journal of Finance, 7(1):77–91, 1952

  27. [27]

    Marshall and Ingram Olkin

    Albert W. Marshall and Ingram Olkin. A multivariate exponential distribution.Journal of the American Statistical Association, 62(317):30–44, 1967

  28. [28]

    The value of waiting to invest.The Quarterly Journal of Economics, 101(4):707–727, 1986

    Robert McDonald and Daniel Siegel. The value of waiting to invest.The Quarterly Journal of Economics, 101(4):707–727, 1986

  29. [29]

    Robert C. Merton. Option pricing when underlying stock returns are discontinuous.Journal of Financial Economics, 3(1-2):125–144, 1976

  30. [30]

    Maximum likelihood estimation of the polychoric correlation coefficient.Psychome- trika, 44(4):443–460, 1979

    Ulf Olsson. Maximum likelihood estimation of the polychoric correlation coefficient.Psychome- trika, 44(4):443–460, 1979

  31. [31]

    RouteLLM: Learning to Route LLMs with Preference Data

    Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M. Waleed Kadous, and Ion Stoica. RouteLLM: Learning to route LLMs with preference data. InInternational Conference on Learning Representations (ICLR), 2025. arXiv:2406.18665

  32. [32]

    Mathematical contributions to the theory of evolution

    Karl Pearson. Mathematical contributions to the theory of evolution. VII. on the correlation of characters not quantitatively measurable.Philosophical Transactions of the Royal Society A, 195:1–47, 1900. 17

  33. [33]

    Bivariate extreme statistics, i.Annals of the Institute of Statistical Mathematics, 11(2):195–210, 1959

    Masaaki Sibuya. Bivariate extreme statistics, i.Annals of the Institute of Statistical Mathematics, 11(2):195–210, 1959

  34. [34]

    How many stocks make a diversified portfolio?Journal of Financial and Quantitative Analysis, 22(3):353–363, 1987

    Meir Statman. How many stocks make a diversified portfolio?Journal of Financial and Quantitative Analysis, 22(3):353–363, 1987. doi: 10.2307/2330969

  35. [35]

    Don’t always pick the highest- performing model: An information-theoretic view of LLM ensemble selection.arXiv preprint arXiv:2602.08003, 2026

    Yigit Turkmen, Baturalp Buyukates, and Melih Bastopcu. Don’t always pick the highest- performing model: An information-theoretic view of LLM ensemble selection.arXiv preprint arXiv:2602.08003, 2026

  36. [36]

    Generalization error of ensemble estimators

    Naonori Ueda and Ryohei Nakano. Generalization error of ensemble estimators. InIEEE International Conference on Neural Networks (ICNN), 1996

  37. [37]

    Mixture-of-Agents Enhances Large Language Model Capabilities

    Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. Mixture-of-agents enhances large language model capabilities.arXiv preprint arXiv:2406.04692, 2024

  38. [38]

    orchestration is allocation

    Danny Wood, Tingting Mu, Andrew M. Webb, Henry W. J. Reeve, Mikel Luj´ an, and Gavin Brown. A unified theory of diversity in ensemble learning.Journal of Machine Learning Research, 24(359):1–49, 2023. arXiv:2301.03962. A Economic scaffolding: routing, diversification, and cascades This appendix gives the economic scaffolding deferred from the main text (§...

  39. [39]

    reproduceeveryone- and two-dimensional marginal— hence every pairwise Pearson and tetrachoric correlation, which are functions of that common bivariate table—yet have β = 0 and β = 1

  40. [40]

    thinking

    So β is not a function of the pairwise law. Padding with independent coordinates extends the example to any m≥ 3, and the common-shock mixture 22 (β∞ = π > 0) versus a matched- ¯ρGaussian copula ( β∞ = 0 by zero lower tail dependence) realize the two extremes asm→ ∞, both consistent with the same pairwise ¯ρ. C Reproducibility Data and code availability.W...

  41. [41]

    we take problems and retain only those whoseownaccepted Python-3 reference our grader accepts (so an all-wrong event is genuine co-failure, not a grader artifact); 63 problems pass (of 140 fetched; 77 dropped, mostly for lacking a short Python-3 reference), each with a median of 20 tests. Each model writes a stdin/stdout program; we execute it against the...