Pith. sign in

REVIEW 2 major objections 5 minor 41 references

Are Diversity Metrics Measuring Diversity? A Capability-Controlled Audit of Majority-Vote Gain in LLM Ensembles

T0 review · 2 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Common diversity measures for LLM ensembles mostly re-express capability: strict diversity is nearly one minus mean accuracy, and after control only more shared error robustly predicts lower majority-vote gain.

desk verdict A careful, honest audit showing diversity metrics mostly re-express capability in LLM pools; the core collinearity is real but the non-neutral parse slice is a legitimate caveat. read the letter →

arxiv 2607.20768 v1 pith:VD2JDSUV submitted 2026-07-22 cs.CL cs.AIcs.LG

Donghwan Kim This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords majorityvotingLLMensemblesdiversitymeasurescapabilityconfoundingco-failureMMLU-Proensembleselectioncontingencytable
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Everyone knows that diverse models should vote better than any single model, so diversity scores are used to pick LLM ensembles. This paper argues that, on modern LLM pools, the most commonly used diversity scores are mostly just measuring average member skill: strict diversity, for instance, is a nearly perfect mirror of one minus mean accuracy, leaving almost no independent variation once capability is controlled. After that control, the only signal that reliably predicts whether majority voting beats the strongest member is a modest negative one: pairs of models that share more errors produce lower gains. In other words, raw diversity–gain correlations are misleading, and the practical takeaway is to evaluate ensembles against the best member, control for capability, and treat contingency-table diversity measures as algebraically coupled rather than independent quantities.

What carries the argument

The central object is the 2×2 contingency table of pairwise joint correctness (a: both correct, b/c: split errors, d: both wrong). Two exact algebraic identities do the heavy lifting: strict diversity = disagreement + double-fault, and 1 − mean accuracy = double-fault + ½ disagreement. The first makes three audited statistics linearly dependent, so raw-space regressions treating them as independent predictors are rank-deficient; the second forces any linear residualization that controls mean accuracy to produce a perfectly collinear residual axis (Pearson r = −1, slope −1/2). The empirical complement is that on modern LLM pools, strict diversity is nearly collinear with 1 − mean accuracy (ρ

What would settle it

Recompute the strict-diversity vs. 1−mean-accuracy Spearman correlation and the capability-controlled double-fault partial correlation on the 144 dropped hard items alone, or on a harder benchmark with per-subset denominators. If ρ drops well below ~0.99 on those items, or if the controlled double-fault association becomes positive or near-zero in that regime, the paper's central claim of capability entanglement with no separable diversity signal would be falsified for exactly the items where diversity would matter most.

Watch

Extended reading notes

Core claim

The paper audits five diversity-related statistics — strict diversity (complement of joint correctness), disagreement, double-fault (co-failure), pairwise Jaccard error overlap, and focal diversity — as predictors of majority-vote gain over the best member across all 31,900 size-2–4 subsets of 30 LLMs on MMLU-Pro (29 on TruthfulQA). It establishes that the three contingency-table statistics are linearly dependent: strict = disagreement + double-fault and 1 − mean accuracy = double-fault + ½ disagreement, so raw-space linear control for capability forces a one-dimensional residual and joint regressions are rank-deficient. Empirically, strict diversity is nearly collinear with one minus mean a

Load-bearing premise

The headline numbers come from the 356-item common-parse slice created by keeping only items every model could answer; the 144 dropped items are harder (mean accuracy 0.659 vs. 0.791) and more disagreed-over (0.316 vs. 0.178), and if the regime where diversity information lives is exactly this filtered-out region, the measured capability entanglement and the residual co-failure direction would be slice-induced.

Editorial extensions

If this is right

  • Raw diversity–gain correlations should not be read as evidence that diversity hurts or helps: most associations flip or vanish once member capability is controlled.
  • Because strict, disagreement, and double-fault are algebraically coupled and mostly re-express mean accuracy, using them as three independent predictors in a regression is rank-deficient and meaningless.
  • Majority voting converts latent complementarity into a win only rarely: oracle gain is positive in 100% of subsets, but the vote beats the best member in just 9.98% of size-3 subsets (18.71% when the best member is chosen on a held-out split).
  • The one directionally robust residual signal is pairwise co-failure: more shared error predicts lower majority-vote gain, so ensemble-selection heuristics should focus on reducing pairwise error overlap rather than maximizing generic diversity.
  • Capability controls, including nonlinear ones, should be standard in any future claim that a diversity measure drives ensemble gain.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the near-collinearity of joint-correctness with mean accuracy holds across mainstream LLM pools, then any diversity measure defined over joint correctness is redundant with average skill for selection; the useful signal is not 'diversity' but the structure of shared errors, which only shows up in the double-fault cell.
  • The paper's evidence suggests a concrete selection rule to test: among subsets of comparable mean and best accuracy, pick the one with the smallest pairwise double-fault residual; the paper's own held-out AUC of 0.597 shows this is a weak but non-random predictor of rare positive gains, and a direct interventional study could quantify its lift.
  • The finding that the co-failure signal weakens on hard items implies that filtering by parseability may be removing exactly the items where ensembles could differentiate models; a deliberate sampling design that oversamples hard, high-disagreement items would test whether the entanglement is intrinsic or slice-induced.
  • The algebraic non-separability result generalizes beyond LLMs to any ensemble of classifiers where the three contingency-table statistics are computed, so the audit method transfers to classical ensemble learning and other prediction settings.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper asks whether five diversity-related statistics (strict diversity, disagreement, double-fault, pairwise Jaccard error-set similarity, and focal diversity) predict the majority-vote gain over the best member in LLM ensembles, once member capability is controlled. The audit enumerates all 31,900 size-2--4 subsets of 30 LLMs on MMLU-Pro (29 on TruthfulQA) and applies six linear control specifications plus nonlinear, matched, model-resampling, and leave-one-model-out robustness checks. Three headline findings are reported: (i) oracle complementarity is universal but realized majority-vote gain over the best member is rare (9.98% at size 3 under in-sample best selection); (ii) strict diversity is nearly collinear with one minus mean accuracy (Spearman rho = +0.991/+0.988); and (iii) after capability control, the only directionally stable contingency-table signal is a modest residual pairwise co-failure association, with more shared error associated with lower gain. The paper is careful to separate algebraic identities from empirical claims, and it repeatedly stresses that the findings are associational and slice-conditional.

Significance. If the results hold, the paper makes a useful measurement-level contribution: several diversity metrics commonly used to select LLM ensembles largely re-express member capability in current model pools, and the linear coupling among strict diversity, disagreement, and double-fault makes raw-space joint regressions rank-deficient by construction. The paper's strengths include exhaustive subset enumeration, explicit separation of algebraic identities from empirical regularities, a wide battery of control and robustness specifications, model-level resampling rather than inflated subset-level p-values, cross-benchmark reproduction on TruthfulQA, and a planned release of scripts and derived correctness matrices sufficient for independent reproduction. The authors are also unusually candid about the non-neutrality of their item filter and other limitations. The main risks are data-conditionality issues concerning which items and which prompt versions enter the correctness matrix.

major comments (2)
  1. [Section 4 / Table 1 / Section 5.6] All headline numbers, including the keystone collinearity strict-diversity vs. 1-mean-accuracy (rho = +0.991/+0.988) and the residual double-fault association (-0.432), are computed on the 356-item common-parse slice. Table 1 shows the 144 dropped items are substantially harder (mean accuracy 0.659 vs. 0.791) and more disagreed (0.316 vs. 0.178), so the filter is not neutral. The checks in Section 5.6 (per-subset denominators, 451-item slice) still select on parsed items and cannot fully rule out slice-induced inflation; Limitation 1 concedes the issue is 'mitigated, not eliminated.' I request a full-500 analysis scoring unparsed responses as incorrect, consistent with the full-500 accuracy definitions in Appendix A.1, or a formal argument why that specification would be invalid. This is load-bearing because both the entanglement diagnosis and the residual co-failure claim are reported o
  2. [Appendix A.2 / Section 8, Limitation 6] The retry protocol re-queried every previously unparsed response with progressively simplified prompts, with Retry 2 and Retry 3 dropping the chain-of-thought instruction. The final correctness matrix therefore mixes initial-prompt responses with simplified-prompt responses, and the pre-retry intersection parsed by all 30 models is only 18 items. A per-subset initial-response-only analysis would not require all 30 models to have parsed an item, so the 18-item figure does not by itself justify omitting such an analysis. Please report the number and fraction of retry-derived responses in the common slice and add a sensitivity analysis using only initial responses on per-subset denominators, or a unified re-prompting of a subsample. Without this, the internal comparability of model predictions underlying every result is uncertain.
minor comments (5)
  1. [Appendix A.1] The MMLU-Pro roster uses anthropic/claude-haiku-4.5 while the TruthfulQA roster uses anthropic/claude-haiku-4-5. Please clarify whether these are the same underlying model; if not, the TruthfulQA 'reproduction' uses a slightly different roster beyond the exclusion of qwen3.6-plus, and this should be stated explicitly.
  2. [Table 3 footnote] The dagger footnote 'Linear-control positive residuals in the full pool only' is cryptic. The text explains that the positive strict/disagreement residuals are roster-dependent, but the footnote should say this directly, since readers may otherwise interpret the +0.339/+0.292 values as robust effects.
  3. [Section 3.1] The focal diversity definition would benefit from one sentence of intuition: rho_i measures, for items on which member i fails, how rarely other members also fail, normalized so that fully disjoint failures give high diversity. Currently the formula is given without a plain-language interpretation.
  4. [Figure 4] The figure reports a descriptive in-sample R^2 from an OLS projection onto ranked best and mean accuracy. The text states this, but the axis label 'variance accounted for' may be misread as predictive or held-out. Consider labeling it 'descriptive in-sample R^2' on the figure itself.
  5. [Appendix C] The 'attenuation-corrected (approx.)' value of -0.53 is based on the classical attenuation formula applied to a partial correlation, which the paper notes treats the controls as measured without error. This is a useful sanity check, but the caveat should appear next to the table entry as well as in the text.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: algebraic identities are explicitly labeled algebraic and empirical claims are tested, not fitted.

full rationale

The paper's derivation chain is not circular. The two identities in Section 3.1 (strict = disagreement + double-fault; 1 − Acc = double-fault + 1/2 disagreement) are presented as exact algebra, and Section 5.4 explicitly says: 'This raw-space one-dimensionality is algebraically inevitable; it is not an empirical discovery.' The near-collinearity of strict diversity with 1 − mean accuracy is treated as an empirical registry property of the pool, not derived from the identities alone, and it is probed with per-subset denominators, a less-filtered 451-item slice, threshold changes, and a second benchmark. The residual pairwise co-failure association is a measured partial-Spearman quantity after rank-space residualization, not a fitted parameter renamed as a prediction; the paper repeatedly stresses its direction is robust but its magnitude is configuration-dependent. The only self-citation (Kim, 2026) is an illustrative aside about closed-loop table recognition and is not load-bearing for the audit's claims. The common-parse slice is a real data-selection threat, acknowledged as 'mitigated, not eliminated' in Section 8, but selection bias is a validity risk rather than definitional circularity. No load-bearing step reduces to its own input by construction.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central claims rest on accuracy-based capability controls, rank-space residualization, a non-neutral common-parse slice, and a specific OpenRouter roster. No fitted model parameters or new theoretical entities are introduced; the only hand-chosen input affecting the roster is the accuracy band, with sensitivity analysis.

free parameters (1)
  • Accuracy inclusion band [0.40, 0.92] = 0.40, 0.92
    Pragmatic model-inclusion filter, not preregistered or literature-derived (Section 4). Non-binding on MMLU-Pro; on TruthfulQA it excludes qwen3.6-plus (full-500 accuracy 0.922). A post-hoc no-upper-cutoff sensitivity check is reported; still a hand-chosen threshold that affects roster composition.
assumptions (5)
  • domain assumption Member accuracy (best/mean/min/median/std/full vector) adequately operationalizes 'capability' for confounding control.
    If capability has components not captured by accuracy on the same benchmark, residual diversity associations could be confounded rather than diversity-specific. The paper controls only on accuracy-based quantities.
  • standard math Rank-space linear residualization (partial Spearman) is a valid control procedure for heavily overlapping subset data.
    Used throughout; the paper avoids subset-level p-values because subsets reuse 30 models, and supports inference with model-level resampling intervals and leave-one-model-out.
  • domain assumption The common parsed item slice (356/338 items) is representative enough for estimating the associations.
    Section 4/Table 1: the filter is not neutral—dropped items are harder and more disagreed—and the authors say the issue is mitigated, not eliminated, with per-subset denominators and a less-filtered 451-item slice.
  • domain assumption The 30/29-model rosters via OpenRouter default routing represent the population of modern LLMs relevant to ensemble selection.
    Roster composition is provider-dependent and time-specific; the paper itself shows near-zero co-failure association in a restricted Chinese-origin pool, so magnitudes are pool-sensitive.
  • domain assumption Majority vote under the correct-count rule with gain over the best member is a meaningful baseline for auditing diversity.
    This defines the dependent variable and deliberately makes the bar stringent; alternatives (held-out best, plurality rule) are reported and change rates but not the qualitative conclusions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Are Diversity Metrics Measuring Diversity? A Capability-Controlled Audit of Majority-Vote Gain in LLM Ensembles." pith.science (2026). https://pith.science/paper/VD2JDSUV

@misc{pith2026260720768,
  author       = {Pith},
  title        = {Pith review of: Are Diversity Metrics Measuring Diversity? A Capability-Controlled Audit of Majority-Vote Gain in LLM Ensembles},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VD2JDSUV}},
  note         = {Machine review of arXiv:2607.20768}
}
read the original abstract

Majority voting over LLMs is widely assumed to benefit from diversity, and diversity measures are used to choose which models to combine. We ask whether five such measures track diversity or mainly re-express capability, auditing them as predictors of majority-vote gain over the best member across 31,900 subsets of 30 LLMs on MMLU-Pro (29 on TruthfulQA) under explicit capability controls. Three findings emerge. First, latent complementarity is ubiquitous: oracle gain is positive in 100% of subsets, yet simple voting beats the strongest member in only 9.98% of all canonical size-3 subsets (18.71% with held-out best selection); the pooled size-2-4 rate is 1.27%, partly reflecting deterministic even-size voting behavior. Second, a joint-correctness proxy (strict diversity) is nearly collinear with one minus mean accuracy (size-3 Spearman rho = +0.991 / +0.988); raw diversity-gain associations are strongly capability-entangled and, with one exception, unstable under control. Third, three linear contingency-table statistics are algebraically non-separable; after capability control, the empirically stable remainder is a modest residual pairwise co-failure association in which more shared error corresponds to lower gain. This direction is robust, but its magnitude is configuration-dependent. Joint rawspace linear regressions treating strict diversity, disagreement, and double-fault as independent predictors are rank-deficient by construction.

Figures

Figures reproduced from arXiv: 2607.20768 by the authors.

Figure 1
Figure 1. Capability-controlled audit pipeline. Top: benchmark-specific routes, fixed filters, and common-parse [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Oracle opportunity versus realized majority [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Strict diversity against one minus mean member accuracy, size 3, on MMLU-Pro (left) and TruthfulQA [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Capability control changes apparent diversity–gain associations (size 3; values from Ta￾ble 3): raw vs. partial Spearman controlling best and best+mean. Only double-fault stays negative through￾out. the association is strong on easy/medium items (≈−0.75 to −0.86) but m…
Figure 6
Figure 6. Figure 6: Residual pairwise co-failure association (size [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 1 canonical work pages

  1. [1]

    Beyond Majority Voting:

    Ai, Rui and Pan, Yuqi and Simchi-Levi, David and Tambe, Milind and Xu, Haifeng , booktitle =. Beyond Majority Voting:. 2026 , month = jul, eprint =

    2026

  2. [3]

    2026 , month = jul, url =

    Baek, David and Estornell, Andrew and Zhang, Yichi and Taufiq, Muhammad Faaiz and Ton, Jean-Francois and Mei, Jie and Wang, Tao , booktitle =. 2026 , month = jul, url =

    2026

  3. [5]

    2026 , month = jun, eprint =

    When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models , author =. 2026 , month = jun, eprint =

    2026

  4. [6]

    ICML 2024 Workshop on In-Context Learning , year =

    Universal Self-Consistency for Large Language Models , author =. ICML 2024 Workshop on In-Context Learning , year =

    2024

  5. [10]

    2023 , address =

    Jiang, Dongfu and Ren, Xiang and Lin, Bill Yuchen , booktitle =. 2023 , address =. doi:10.18653/v1/2023.acl-long.792 , url =. 2306.02561 , archiveprefix =

    arXiv 2023

  6. [13]

    Advances in Neural Information Processing Systems 7 , editor =

    Neural Network Ensembles, Cross Validation, and Active Learning , author =. Advances in Neural Information Processing Systems 7 , editor =. 1995 , publisher =

    1995

  7. [18]

    2024 , address =

    Tekin, Selim Furkan and Ilhan, Fatih and Huang, Tiansheng and Hu, Sihao and Liu, Ling , booktitle =. 2024 , address =. doi:10.18653/v1/2024.findings-emnlp.698 , url =. 2410.03953 , archiveprefix =

    arXiv 2024

  8. [23]

    Journal of Machine Learning Research , year =

    A Unified Theory of Diversity in Ensemble Learning , author =. Journal of Machine Learning Research , year =

Show all 41 references
  1. [24]

    Journal of Machine Learning Research , year =

    Ensemble Pruning Via Semi-definite Programming , author =. Journal of Machine Learning Research , year =

  2. [28]

    Proceedings of the 41st International Conference on Machine Learning , pages =

    Improving Factuality and Reasoning in Language Models through Multiagent Debate , author =. Proceedings of the 41st International Conference on Machine Learning , pages =. 2024 , volume =

    2024

  3. [32]

    Rui Ai, Yuqi Pan, David Simchi-Levi, Milind Tambe, and Haifeng Xu. 2026. https://arxiv.org/abs/2510.01499 Beyond majority voting: LLM aggregation by leveraging higher-order information . In Forty-Third International Conference on Machine Learning

    2026 arXiv

  4. [33]

    Junade Ali. 2026. https://arxiv.org/abs/2607.17384 Quantifying diversity of thought: A predictive law of weighted LLM ensemble lift . Preprint, arXiv:2607.17384

    2026 arXiv

  5. [34]

    David Baek, Andrew Estornell, Yichi Zhang, Muhammad Faaiz Taufiq, Jean-Francois Ton, Jie Mei, and Tao Wang. 2026. https://icml.cc/virtual/2026/poster/63783 D-FUSEr : Diverse failure, unified success via error-distribution shaping in LLM reasoning . In Forty-Third International...

    2026

  6. [35]

    Gavin Brown, Jeremy Wyatt, Rachel Harris, and Xin Yao. 2005. https://doi.org/10.1016/j.inffus.2004.04.004 Diversity creation methods: a survey and categorisation . Information Fusion, 6(1):5--20

    2005 doi

  7. [36]

    Josef Chen. 2026. https://arxiv.org/abs/2606.27288 When does combining language models help? a co-failure ceiling on routing, voting, and mixture-of-agents across 67 frontier models . Preprint, arXiv:2606.27288

    2026 arXiv

  8. [37]

    Lingjiao Chen, Matei Zaharia, and James Zou. 2024 a . https://arxiv.org/abs/2305.05176 FrugalGPT : How to use large language models while reducing cost and improving performance . Transactions on Machine Learning Research

    2024 arXiv

  9. [38]

    Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, and Denny Zhou. 2024 b . https://openreview.net/forum?id=LjsjHF7nAN Universal self-consistency for large language models . In ICML 2024 Workshop on In-Contex...

    2024 arXiv

  10. [39]

    Zhijun Chen, Xiaodong Lu, Jingzheng Li, Pengpeng Chen, Zhuoran Li, Kai Sun, Yuankai Luo, Qianren Mao, Ming Li, Likang Xiao, Dingqi Yang, Xiao Huang, Yikun Ban, Hailong Sun, and Philip S. Yu. 2025. https://arxiv.org/abs/2502.18036 Harnessing multiple large language models: A su...

    2025 arXiv

  11. [40]

    Seffi Cohen, Nurit Cohen Inger, Niv Goldshlager, Bracha Shapira, and Lior Rokach. 2026. https://doi.org/10.18653/v1/2026.findings-eacl.282 DFPE : A diverse fingerprint ensemble for enhancing LLM performance . In Findings of the Association for Computational Linguistics: EACL 2...

    2026 doi

  12. [41]

    Dietterich

    Thomas G. Dietterich. 2000. https://doi.org/10.1007/3-540-45014-9_1 Ensemble methods in machine learning . In Multiple Classifier Systems, volume 1857 of Lecture Notes in Computer Science, pages 1--15, Berlin, Heidelberg. Springer

    2000 doi

  13. [42]

    Tenenbaum, and Igor Mordatch

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. 2024. https://proceedings.mlr.press/v235/du24e.html Improving factuality and reasoning in language models through multiagent debate . In Proceedings of the 41st International Conference on Machine L...

    2024

  14. [43]

    Shashwat Goel, Joschka Str \"u ber, Ilze Amanda Auzina, Karuna K Chandra, Ponnurangam Kumaraguru, Douwe Kiela, Ameya Prabhu, Matthias Bethge, and Jonas Geiping. 2025. https://arxiv.org/abs/2502.04313 Great models think alike and this undermines AI oversight . In Proceedings of...

    2025 arXiv

  15. [44]

    Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. 2023. https://doi.org/10.18653/v1/2023.acl-long.792 LLM-Blender : Ensembling large language models with pairwise ranking and generative fusion . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguis...

    2023 doi

  16. [45]

    Donghwan Kim. 2026. https://arxiv.org/abs/2607.13347 LLM -as-a-judge scores are unreliable optimization signals in closed-loop table recognition . Preprint, arXiv:2607.13347

    2026 arXiv

  17. [46]

    Elliot Myunghoon Kim, Avi Garg, Kenny Peng, and Nikhil Garg. 2025. https://arxiv.org/abs/2506.07962 Correlated errors in large language models . In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pag...

    2025 arXiv

  18. [47]

    Anders Krogh and Jesper Vedelsby. 1995. https://proceedings.neurips.cc/paper/1994/hash/b8c37e33defde51cf91e1e03e51657da-Abstract.html Neural network ensembles, cross validation, and active learning . In Advances in Neural Information Processing Systems 7, pages 231--238. MIT P...

    1995

  19. [48]

    Kuncheva and Christopher J

    Ludmila I. Kuncheva and Christopher J. Whitaker. 2003. https://doi.org/10.1023/A:1022859003006 Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy . Machine Learning, 51(2):181--207

    2003 doi

  20. [49]

    Baptiste Lefort, Eric Benhamou, Jean-Jacques Ohana, B \'e atrice Guez, David Saltiel, and Thomas Jacquot. 2024. https://doi.org/10.48550/arXiv.2409.00094 Examining independence in ensemble sentiment analysis: A study on the limits of large language models using the condorcet j...

    2024 doi arXiv

  21. [50]

    Junyou Li, Qin Zhang, Yangbin Yu, Qiang Fu, and Deheng Ye. 2024. https://arxiv.org/abs/2402.05120 More agents is all you need . Transactions on Machine Learning Research

    2024 arXiv

  22. [51]

    Wenzhe Li, Yong Lin, Mengzhou Xia, and Chi Jin. 2026. https://arxiv.org/abs/2502.00674 Rethinking mixture-of-agents: Is mixing different large language models beneficial? Transactions on Machine Learning Research

    2026 arXiv

  23. [52]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. https://doi.org/10.18653/v1/2022.acl-long.229 TruthfulQA : Measuring how models mimic human falsehoods . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), page...

    2022 doi

  24. [53]

    Chenglei Si, Weijia Shi, Chen Zhao, Luke Zettlemoyer, and Jordan Boyd-Graber. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.552 Getting MoRE out of mixture of language model reasoning experts . In Findings of the Association for Computational Linguistics: EMNLP 2023, p...

    2023 doi

  25. [54]

    Selim Furkan Tekin, Fatih Ilhan, Tiansheng Huang, Sihao Hu, and Ling Liu. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.698 LLM-TOPLA : Efficient LLM ensemble by maximising diversity . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1195...

    2024 doi

  26. [55]

    Yigit Turkmen, Baturalp Buyukates, and Melih Bastopcu. 2026. https://arxiv.org/abs/2602.08003 Don't always pick the highest-performing model: An information theoretic view of LLM ensemble selection . Preprint, arXiv:2602.08003

    2026

  27. [56]

    Vardeman and Max D

    Stephen B. Vardeman and Max D. Morris. 2013. https://doi.org/10.1080/00031305.2013.778788 Majority voting by independent classifiers can increase error rates . The American Statistician, 67(2):94--96

    2013 arXiv

  28. [57]

    Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. 2025. https://arxiv.org/abs/2406.04692 Mixture-of-agents enhances large language model capabilities . In International Conference on Learning Representations. Spotlight

    2025 arXiv

  29. [58]

    Le, Ed H

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. https://arxiv.org/abs/2203.11171 Self-consistency improves chain of thought reasoning in language models . In International Conference on Learning Representations

    2023 arXiv

  30. [59]

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024. https://arxiv.org/abs/2406.01574 MMLU-Pro : A more robust an...

    2024 arXiv

  31. [60]

    Webb, Henry W

    Danny Wood, Tingting Mu, Andrew M. Webb, Henry W. J. Reeve, Mikel Luj \'a n, and Gavin Brown. 2023. https://jmlr.org/papers/v24/23-0041.html A unified theory of diversity in ensemble learning . Journal of Machine Learning Research, 24(359):1--49

    2023

  32. [61]

    Nick Street

    Yi Zhang, Samuel Burer, and W. Nick Street. 2006. https://jmlr.org/papers/v7/zhang06a.html Ensemble pruning via semi-definite programming . Journal of Machine Learning Research, 7(48):1315--1338

    2006

  33. [62]

    Yichi Zhang, Kevin Lu, Yuang Zhang, Jie Gao, Lirong Xia, and Fang-Yi Yu. 2026. https://arxiv.org/abs/2605.24048 Mixture of complementary agents for robust LLM ensemble . Preprint, arXiv:2605.24048. Associated ICLR 2026 submission withdrawn

    2026 arXiv

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.