When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models
Pith reviewed 2026-06-26 04:16 UTC · model grok-4.3
The pith
Any policy selecting one model answer from an ensemble cannot exceed accuracy of one minus the rate at which all models fail together.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
For any policy whose output is one member model answer, accuracy cannot exceed one minus beta, where beta is the rate at which every model is wrong on the same query. The usual diagnostic of average pairwise error correlation rho cannot identify beta. A Clopper-Pearson bound on beta supplies a finite-sample certificate on the largest gain any router, vote, or cascade could deliver. Across 67 models, observed beta exceeds copula predictions, and gains appear only when models fail on different questions and routing exploits that difference.
What carries the argument
The co-failure rate beta, the probability that all models in the ensemble err simultaneously on a given query.
If this is right
- Routers, voters, and cascades cannot exceed 1-beta no matter how they choose among member answers.
- Identical pairwise correlations can hide different beta values, so rho alone does not certify ensemble headroom.
- On checkable tasks, adding models without query-level routing rarely beats the single best model.
- Changing answer format from multiple-choice to free-response can reopen the co-failure tail even on the same questions.
Where Pith is reading between the lines
- Systems that synthesize new answers or add external verification steps can in principle exceed the 1-beta ceiling that applies to pure selection policies.
- Better tail modeling beyond Gaussian copulas would be needed to predict the largest achievable gains before training routers.
- Reducing simultaneous failures across models may matter more for ensemble performance than further lowering average pairwise correlation.
Load-bearing premise
The policy must output exactly one of the member models' answers rather than synthesizing a new answer or using external verification.
What would settle it
An ensemble policy that selects one member answer and achieves accuracy higher than one minus the empirically measured beta on the same query set would contradict the bound.
Figures
read the original abstract
Multi-model LLM systems such as routing, voting, cascades, fusion, and mixture-of-agents are used to beat single-model accuracy. We show that their gain is capped by a quantity the field rarely reports. For any policy whose output is one member model answer, accuracy cannot exceed one minus beta, where beta is the rate at which every model is wrong on the same query. In contrast, the usual diagnostic, average pairwise error correlation rho, cannot identify beta: error laws with identical marginals and pairwise correlations can have different all-wrong rates. A Clopper-Pearson bound on beta gives a finite-sample certificate on the largest gain any router, vote, or cascade could deliver before training a router. Across 67 models from 21 providers, a tetrachoric-calibrated single-factor model still underprices the all-wrong tail: on open-ended mathematics, observed beta is 0.052 versus 0.023 under the full 67-model Gaussian copula, about 2.5 times underpricing, with 90 percent CI 1.7 to 3.4 and k equals 17. The effect recurs on execution-graded code, where beta is 0.079. Re-asking the same GPQA-Diamond questions in free-response rather than multiple-choice form reopens the tail, with beta 0.127 and a five-judge panel with kappa 0.73 to 0.92, locating co-failure in answer format rather than subject. At matched quality, low-rho heterogeneous ensembles beat high-rho Self-MoA, but on checkable tasks in our pool, combining models rarely beats the single best model without a strong query-level routing signal. Gains come from models failing on different questions, not from adding more models.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript defines β as the probability that every model in an ensemble is simultaneously wrong on a given query and proves that any policy whose output must be exactly one of the member models' answers cannot exceed accuracy 1−β. It shows that the conventional diagnostic of average pairwise error correlation ρ is insufficient to identify β, supplies a Clopper-Pearson finite-sample upper bound on β, and reports that across 67 frontier models a Gaussian-copula model underestimates observed β (0.052 vs. 0.023 on open-ended math; 0.079 on code; 0.127 on free-response GPQA). The paper concludes that, on checkable tasks, gains from combining models are driven by query-level routing rather than by adding models or lowering ρ.
Significance. If the logical bound is accepted and the empirical β measurements prove robust, the work supplies a simple, reportable quantity that places a hard ceiling on the upside of any selection-based router, vote, or cascade before any training occurs. The demonstration that identical marginal error rates and pairwise ρ can produce materially different β values is a clarifying observation for the ensemble literature. The scale of the study (67 models, multiple task types, execution grading) lends weight to the claim that co-failure tails are heavier than low-dimensional factor models predict. These elements constitute a constructive, falsifiable contribution to understanding the limits of multi-LLM systems.
major comments (2)
- [Abstract / Title] Abstract and title: The central bound is derived only for policies that output exactly one member model's answer, yet the title and abstract list 'routing, voting, cascades, fusion, and mixture-of-agents' as the systems under study and state that 'their gain is capped.' Mixture-of-Agents and fusion frameworks commonly synthesize a new token sequence rather than emit a raw member output; on queries where all members fail, such synthesis can still be correct. The manuscript must either restrict the title/abstract/claim to selection policies or derive/qualify the bound for synthesis methods.
- [Empirical results (math / code)] § on Gaussian-copula comparison (math and code results): The statement that a 'tetrachoric-calibrated single-factor model still underprices the all-wrong tail' (observed β = 0.052 vs. 0.023) requires an explicit statement of how the copula parameters are estimated, whether the fitted marginals exactly match the empirical per-model error rates, and whether the 90 % CI (1.7–3.4) accounts for dependence across the 67 models. Without these details the factor-model underpricing claim cannot be evaluated.
minor comments (1)
- [Abstract] The abstract states 'k equals 17' without defining k; the main text should state what this integer represents (e.g., number of models or a hyper-parameter).
Simulated Author's Rebuttal
We thank the referee for these precise comments on scope and methodological transparency. Both points are addressable by targeted revisions that preserve the paper's central claims and results.
read point-by-point responses
-
Referee: [Abstract / Title] Abstract and title: The central bound is derived only for policies that output exactly one member model's answer, yet the title and abstract list 'routing, voting, cascades, fusion, and mixture-of-agents' as the systems under study and state that 'their gain is capped.' Mixture-of-Agents and fusion frameworks commonly synthesize a new token sequence rather than emit a raw member output; on queries where all members fail, such synthesis can still be correct. The manuscript must either restrict the title/abstract/claim to selection policies or derive/qualify the bound for synthesis methods.
Authors: We agree the 1−β bound applies strictly to selection policies. The body already qualifies the claim accordingly. We will revise the title and abstract to read 'selection-based ensembles (routing, voting, cascades)' and insert a clarifying sentence noting that synthesis methods such as Mixture-of-Agents can in principle exceed the bound by generating novel outputs. This aligns front matter with the theorems without altering any proofs or experiments. revision: yes
-
Referee: [Empirical results (math / code)] § on Gaussian-copula comparison (math and code results): The statement that a 'tetrachoric-calibrated single-factor model still underprices the all-wrong tail' (observed β = 0.052 vs. 0.023) requires an explicit statement of how the copula parameters are estimated, whether the fitted marginals exactly match the empirical per-model error rates, and whether the 90 % CI (1.7–3.4) accounts for dependence across the 67 models. Without these details the factor-model underpricing claim cannot be evaluated.
Authors: We will add a dedicated paragraph in the methods section stating: pairwise tetrachoric correlations are estimated directly from the 67×query error-indicator matrix; marginal error probabilities are fixed exactly to the empirical per-model rates; the single-factor Gaussian copula is then sampled to obtain the model-predicted β. The 90 % CI on the ratio (observed β / copula β) is a nonparametric bootstrap over queries and does not incorporate cross-model dependence; we will explicitly note this limitation. These additions make the underpricing claim (factor ≈2.5 on math) fully evaluable from the released data. revision: yes
Circularity Check
No circularity: bound is direct logical consequence of beta definition and single-output restriction
full rationale
The paper's core claim states that for policies outputting exactly one member model answer, accuracy ≤ 1 − β where β is the all-wrong rate. This follows immediately from the definitions without any fitting, parameter estimation, or reduction to prior self-citations. The abstract explicitly qualifies the bound to 'any policy whose output is one member model answer,' distinguishing it from synthesis methods. No equations or steps in the provided text reduce the result to its inputs by construction. The scope mismatch with MoA/fusion noted by the skeptic is a potential overgeneralization in presentation but does not constitute circularity in the derivation itself. The analysis is self-contained against external benchmarks.
Axiom & Free-Parameter Ledger
axioms (1)
- standard math Clopper-Pearson bound applies to estimating the binomial proportion beta from finite samples
Reference graph
Works this paper leans on
-
[1]
Tsitsiklis.Introduction to Linear Optimization
Dimitris Bertsimas and John N. Tsitsiklis.Introduction to Linear Optimization. Athena Scientific, 1997
1997
-
[2]
FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance
Lingjiao Chen, Matei Zaharia, and James Zou. FrugalGPT: How to use large language models while reducing cost and improving performance.arXiv preprint arXiv:2305.05176, 2023
work page internal anchor Pith review Pith/arXiv arXiv 2023
-
[3]
Jasper Dekoninck, Maximilian Baader, and Martin Vechev. A unified approach to routing and cascading for LLMs.arXiv preprint arXiv:2410.10347, 2024
-
[4]
Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor R¨ uhle, Laks V. S. Lakshmanan, and Ahmed Hassan Awadallah. Hybrid LLM: Cost-efficient and quality-aware query routing. InInternational Conference on Learning Representations (ICLR), 2024
2024
-
[5]
Dixit and Robert S
Avinash K. Dixit and Robert S. Pindyck.Investment Under Uncertainty. Princeton University Press, 1994
1994
-
[6]
The devil is in the tails: actuarial mathematics and the subprime mortgage crisis.ASTIN Bulletin, 40(1):1–33, 2010
Catherine Donnelly and Paul Embrechts. The devil is in the tails: actuarial mathematics and the subprime mortgage crisis.ASTIN Bulletin, 40(1):1–33, 2010
2010
-
[7]
McNeil, and Daniel Straumann
Paul Embrechts, Alexander J. McNeil, and Daniel Straumann. Correlation and dependence in risk management: properties and pitfalls. InRisk Management: Value at Risk and Beyond, pages 176–223. Cambridge University Press, 2002
2002
-
[8]
Mehmet Hamza Erol, Batu El, Mirac Suzgun, et al. Cost-of-pass: An economic framework for evaluating language models.arXiv preprint arXiv:2504.13359, 2025
-
[9]
Representation of multivariate bernoulli distributions with a given set of specified moments.Journal of Multivariate Analysis, 168:290–303, 2018
Roberto Fontana and Patrizia Semeraro. Representation of multivariate bernoulli distributions with a given set of specified moments.Journal of Multivariate Analysis, 168:290–303, 2018
2018
-
[10]
Selective classification for deep neural networks
Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2017
2017
-
[11]
Ronald A. Howard. Information value theory.IEEE Transactions on Systems Science and Cybernetics, 2(1):22–26, 1966
1966
-
[12]
RouterBench: A Benchmark for Multi-LLM Routing System
Qitian Jason Hu, Jacob Bieker, et al. RouterBench: A benchmark for multi-LLM routing system.arXiv preprint arXiv:2403.12031, 2024
work page internal anchor Pith review Pith/arXiv arXiv 2024
-
[13]
arXiv preprint arXiv:2306.02561 (2023)
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LLM-Blender: Ensembling large language models with pairwise ranking and generative fusion. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), 2023. arXiv:2306.02561
-
[14]
Wittawat Jitkrittum, Neha Gupta, Aditya Krishna Menon, Harikrishna Narasimhan, Ankit Singh Rawat, and Sanjiv Kumar. When does confidence-based cascade deferral suffice? InAdvances in Neural Information Processing Systems (NeurIPS), 2023. arXiv:2307.02764. 16
-
[15]
Correlated errors in large language models
Elliot Kim, Avi Garg, Kenny Peng, and Nikhil Garg. Correlated errors in large language models. InInternational Conference on Machine Learning (ICML), 2025. arXiv:2506.07962
-
[16]
Algorithmic monoculture and social welfare.Proceedings of the National Academy of Sciences, 118(22):e2018340118, 2021
Jon Kleinberg and Manish Raghavan. Algorithmic monoculture and social welfare.Proceedings of the National Academy of Sciences, 118(22):e2018340118, 2021
2021
-
[17]
Competition when consumers have switching costs.The Review of Economic Studies, 62(4):515–539, 1995
Paul Klemperer. Competition when consumers have switching costs.The Review of Economic Studies, 62(4):515–539, 1995
1995
-
[18]
Neural network ensembles, cross validation, and active learning
Anders Krogh and Jesper Vedelsby. Neural network ensembles, cross validation, and active learning. InAdvances in Neural Information Processing Systems (NeurIPS), volume 7, pages 231–238, 1995
1995
-
[19]
Kuncheva.Combining Pattern Classifiers: Methods and Algorithms
Ludmila I. Kuncheva.Combining Pattern Classifiers: Methods and Algorithms. Wiley, 2004
2004
-
[20]
Kuncheva and Christopher J
Ludmila I. Kuncheva and Christopher J. Whitaker. Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy.Machine Learning, 51(2):181–207, 2003
2003
-
[21]
Kuncheva, Christopher J
Ludmila I. Kuncheva, Christopher J. Whitaker, Catherine A. Shipp, and Robert P. W. Duin. Limits on the majority vote accuracy in classifier fusion.Pattern Analysis & Applications, 6(1): 22–31, 2003
2003
-
[22]
David X. Li. On default correlation: A copula function approach.Journal of Fixed Income, 9 (4):43–54, 2000
2000
-
[23]
arXiv preprint arXiv:2402.05120 , year=
Junyou Li, Qin Zhang, Yangbin Yu, Qiang Fu, and Deheng Ye. More agents is all you need. Transactions on Machine Learning Research, 2024. arXiv:2402.05120
-
[24]
arXiv preprint arXiv:2502.00674 , year =
Wenzhe Li, Yong Lin, Mengzhou Xia, and Chi Jin. Rethinking mixture-of-agents: Is mixing different large language models beneficial?arXiv preprint arXiv:2502.00674, 2025
-
[25]
AutoMix: Automatically mixing language models
Aman Madaan, Pranjal Aggarwal, et al. AutoMix: Automatically mixing language models. arXiv preprint arXiv:2310.12963, 2023
-
[26]
Portfolio selection.The Journal of Finance, 7(1):77–91, 1952
Harry Markowitz. Portfolio selection.The Journal of Finance, 7(1):77–91, 1952
1952
-
[27]
Marshall and Ingram Olkin
Albert W. Marshall and Ingram Olkin. A multivariate exponential distribution.Journal of the American Statistical Association, 62(317):30–44, 1967
1967
-
[28]
The value of waiting to invest.The Quarterly Journal of Economics, 101(4):707–727, 1986
Robert McDonald and Daniel Siegel. The value of waiting to invest.The Quarterly Journal of Economics, 101(4):707–727, 1986
1986
-
[29]
Robert C. Merton. Option pricing when underlying stock returns are discontinuous.Journal of Financial Economics, 3(1-2):125–144, 1976
1976
-
[30]
Maximum likelihood estimation of the polychoric correlation coefficient.Psychome- trika, 44(4):443–460, 1979
Ulf Olsson. Maximum likelihood estimation of the polychoric correlation coefficient.Psychome- trika, 44(4):443–460, 1979
1979
-
[31]
RouteLLM: Learning to Route LLMs with Preference Data
Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M. Waleed Kadous, and Ion Stoica. RouteLLM: Learning to route LLMs with preference data. InInternational Conference on Learning Representations (ICLR), 2025. arXiv:2406.18665
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[32]
Mathematical contributions to the theory of evolution
Karl Pearson. Mathematical contributions to the theory of evolution. VII. on the correlation of characters not quantitatively measurable.Philosophical Transactions of the Royal Society A, 195:1–47, 1900. 17
1900
-
[33]
Bivariate extreme statistics, i.Annals of the Institute of Statistical Mathematics, 11(2):195–210, 1959
Masaaki Sibuya. Bivariate extreme statistics, i.Annals of the Institute of Statistical Mathematics, 11(2):195–210, 1959
1959
-
[34]
Meir Statman. How many stocks make a diversified portfolio?Journal of Financial and Quantitative Analysis, 22(3):353–363, 1987. doi: 10.2307/2330969
-
[35]
Yigit Turkmen, Baturalp Buyukates, and Melih Bastopcu. Don’t always pick the highest- performing model: An information-theoretic view of LLM ensemble selection.arXiv preprint arXiv:2602.08003, 2026
-
[36]
Generalization error of ensemble estimators
Naonori Ueda and Ryohei Nakano. Generalization error of ensemble estimators. InIEEE International Conference on Neural Networks (ICNN), 1996
1996
-
[37]
Mixture-of-Agents Enhances Large Language Model Capabilities
Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. Mixture-of-agents enhances large language model capabilities.arXiv preprint arXiv:2406.04692, 2024
work page internal anchor Pith review Pith/arXiv arXiv 2024
-
[38]
Danny Wood, Tingting Mu, Andrew M. Webb, Henry W. J. Reeve, Mikel Luj´ an, and Gavin Brown. A unified theory of diversity in ensemble learning.Journal of Machine Learning Research, 24(359):1–49, 2023. arXiv:2301.03962. A Economic scaffolding: routing, diversification, and cascades This appendix gives the economic scaffolding deferred from the main text (§...
-
[39]
reproduceeveryone- and two-dimensional marginal— hence every pairwise Pearson and tetrachoric correlation, which are functions of that common bivariate table—yet have β = 0 and β = 1
-
[40]
thinking
So β is not a function of the pairwise law. Padding with independent coordinates extends the example to any m≥ 3, and the common-shock mixture 22 (β∞ = π > 0) versus a matched- ¯ρGaussian copula ( β∞ = 0 by zero lower tail dependence) realize the two extremes asm→ ∞, both consistent with the same pairwise ¯ρ. C Reproducibility Data and code availability.W...
2026
-
[41]
we take problems and retain only those whoseownaccepted Python-3 reference our grader accepts (so an all-wrong event is genuine co-failure, not a grader artifact); 63 problems pass (of 140 fetched; 77 dropped, mostly for lacking a short Python-3 reference), each with a median of 20 tests. Each model writes a stdin/stdout program; we execute it against the...
1900
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.