Pith. sign in

REVIEW 2 major objections 5 minor 48 references

This paper claims that the errors of language-model judges co-occur in a systematic, state-dependent way, leaving majority-vote committees with a permanent error floor, and that thresholds chosen from measured correlations reduce cost-weigh

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-07-31 23:30 UTC pith:DDPANE5U

load-bearing objection Solid, honest paper that delivers on its title: state-dependent error correlations in LLM committees are measurable and useful for threshold design; the Gaussian-copula assumption is the main soft spot, but held-out validation backs the central claim. the 2 major comments →

arxiv 2607.23931 v1 pith:DDPANE5U submitted 2026-07-27 cs.CY

State-dependent error correlations shape voting thresholds in committees of AI agents

Haifeng Li , Mo Hai This is my paper
classification cs.CY MSC 62H2091B14
keywords correlated errorsLLM committeesGaussian copulavoting thresholdsscreening theoryensemble diversitymajority votingpairwise error correlation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that the errors of language-model judges are far from independent: models trained on similar data fail on the same cases, and they do so to different degrees on good versus bad cases. That makes classical majority-voting guarantees, which assume independent votes, misleading for AI committees. The authors build a screening model in which each judge has a state-conditional approval rate and a state-conditional correlation with the other judges, and they show that positive correlation leaves a residual error floor that no amount of added judges removes. They estimate the model from 174,384 votes by 28 language models on four binary-screening tasks and find that the full correlation structure predicts held-out committee losses for every k-of-n rule better than an independence assumption (identity-line R² of 0.967 versus 0.840); using the fitted structure to set the approval threshold cuts cost-weighted loss by about 15.7% relative to majority rule. If right, the paper turns measured error overlap into a practical design input: choose committees and voting thresholds from each state's pairwise error correlations rather than from individual accuracies alone.

Core claim

The paper's central claim is that state-conditional error dependence is a large, stable, and usable property of AI-committee judgments. In the homogeneous one-factor model, shared errors mean a majority committee's error rate does not vanish as the committee grows; it converges to a positive Gaussian-tail floor Φ(−Φ⁻¹(p)/√ρ), so dependence caps what vote-only aggregation can achieve. In the heterogeneous model, member-specific approval probabilities and two state-specific correlation matrices determine the cost-weighted loss of every k-of-n threshold, and the authors show these inputs can be estimated from one half of the data and predict loss on the held-out half. The dependence-aware model

What carries the argument

The central object is a one-factor Gaussian latent-threshold model (an exchangeable Gaussian copula): judge i's latent score in state θ is √ρθ·W + √(1−ρθ)·εi, with a shared factor W plus idiosyncratic noise, and the judge approves when the score falls below a threshold set by the approval probability. This yields the acceptance identity A_{n,k}(p,ρ) = Pr(√ρ·W + √(1−ρ)·ε_(k) ≤ Φ⁻¹(p)), where ε_(k) is the k-th order statistic of the idiosyncratic terms: the shared factor's variance ρ survives every threshold, while the idiosyncratic part is what design can diversify. The heterogeneous extension replaces the single factor with a full state-specific pairwise correlation matrix (good state and ba

Load-bearing premise

The model assumes that the joint chance of any set of judges approving the same case is fully captured by pairwise vote correlations in a Gaussian-copula form; if judges fail together in ways pairwise correlations cannot represent, the predicted losses could be off even though the held-out fit is close.

What would settle it

On a new benchmark domain with a fresh model roster, estimate the full correlation matrices on one half of the items, then compare realized loss of the dependence-selected threshold against the independence-selected threshold on the other half; if the dependence model's mean loss is not below the independence model's (or the incremental 1.73-unit gain reverses), the applied claim fails. A sharper check: test whether triple-wise vote coincidences exceed what the fitted Gaussian copula implies; if higher-order dependence changes the loss of unanimous rules materially, the joint model is misspeci

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X LinkedIn Reddit HN

If this is right

  • Under the measured correlations, majority voting cannot drive committee error to zero: for a judge of accuracy 0.8, the infinite-majority residual retains roughly 21–90% of single-agent error across the measured correlation range.
  • The loss-minimizing approval threshold depends jointly on state prevalence, relative mistake costs, member approval rates, and state-conditional dependence; costlier false acceptances call for stricter (higher-k) rules.
  • Dependence can differ between good and bad states, and unequal state correlations can make a unanimous approval signal weaker than an 8-of-9 approval, so committees should enumerate all candidate thresholds rather than assume a monotone likelihood ratio.
  • A model using measured pairwise dependence predicts held-out losses of every k-of-n rule with identity-line R² 0.967 versus 0.840 for an otherwise identical independence model, implying independence-based estimates misstate committee loss.
  • Dependence-aware threshold selection adds a further loss reduction beyond cost-sensitive independence: 1.73 scaled units (95% CI 0.68–2.33), for a total 15.73% (95% CI 13.41–16.75%) reduction from majority rule.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Since the same-family-versus-cross-family correlation difference is statistically indistinguishable after controlling for capability, the practical roster-selection signal may be the measured pair-level correlation itself, not model-family labels.
  • The existence of an error floor implies a hard ceiling for vote-only committees of a fixed roster; going below it would require breaking shared error sources (for example, diversifying training data or eliciting reasons before votes), a direction the paper does not develop.
  • A direct testable extension is to apply the same estimation and threshold-selection protocol to human review boards or mixed human-AI panels; the formalism does not depend on the judges being machines.
  • The loss decomposition (about 82% of the gain from cost-sensitive threshold choice, 18% from dependence modeling) suggests that even without a fitted correlation model, organizations should at minimum replace majority rule with cost-weighted threshold selection.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper studies committee voting by AI agents whose errors are positively correlated within each state, combining Sah–Stiglitz screening with a Gaussian one-factor (Vasicek-style) dependence model. The theoretical part derives an order-statistic representation of k-of-n acceptance probabilities, proves that correlation weakens the protection of hierarchies and polyarchies, establishes a positive Vasicek error floor for large correlated majorities, and proves a collapse of fixed-size committees as latent correlation tends to one. The empirical part estimates state-conditional marginals and tetrachoric correlations from 174,384 votes by 28 LLMs on four binary screening tasks, then uses an odd/even item split to validate held-out loss predictions for every k-of-n rule. An independence model, a one-factor projection, and a full-matrix Gaussian-copula model are compared. The full-matrix model raises identity-line R² from 0.840 to 0.967, and a design-balanced bootstrap analysis reports that dependence-aware threshold selection reduces scaled loss by 1.73 units beyond independence-based selection, for a 15.73% total reduction from majority voting. The authors openly disclose that the finite-grid check, not a proof, underlies global majority-loss monotonicity, and that the empirical correlation estimates require Fréchet clipping and PSD projection in a substantial fraction of cases.

Significance. If the empirical claims hold, the paper makes a useful contribution to the growing literature on LLM juries and multi-agent committees: it supplies an analytic framework in which state-conditional error dependence is not a nuisance but a measurable design input, and it provides a reproducible pipeline that goes from raw votes to threshold recommendations. The theoretical core is clean and well supported: Eq. (1)/Lemma S2 gives an illuminating decomposition into a diversifiable order statistic and a non-diversifiable common factor; the Vasicek limit (Corollary S8) and the collapse theorem (Theorem S6) are correctly stated and proved; and the numerical certification table (Table S1) is a genuine strength. The odd/even parameter split, the bootstrap procedure that fully refits and reselects thresholds, the code-to-result map in Table S2, and the archived data are exemplary for this literature. The main risk is empirical: the headline dependence-aware gain rests on a Gaussian-copula assumption for binary vote vectors, and the paper’s own diagnostics show heavy regularization; the pooled validation metric may not be sensitive enough in the strict-threshold, high-κ regime where threshold

major comments (2)
  1. [§S7.1, §S9.4, Table S6] The central empirical claim — that modeling measured dependence improves held-out threshold selection — is not yet robustly established for the strict-rule regime. The heterogeneous model assumes that each committee’s binary vote vector is a Gaussian copula determined by marginals and pairwise tetrachoric correlations. For binary data this is a strong functional-form assumption, and the authors’ own diagnostics show that 76.80% of committee state matrices required PSD projection, 18.48% of eligible inversions saturated a Fréchet boundary, and 17.09% used an unbracketed clip. The reported identity-line R²=0.967 is pooled over 73,410 threshold–cost rows and can be dominated by moderate-k thresholds, while the design-relevant choices for large κ are exactly the near-unanimity thresholds. The paper does not report calibration or the dependence-aware benefit separately by κ, nor by threshold
  2. [Result 1 / §S8, check C6] The main-text Result 1 states that correlation 'erodes the state-specific protection supplied by each canonical rule.' This is proved globally for hierarchies and polyarchies, but for majority rule only the local derivative at ρ=0 is proved analytically and the global statement rests on a finite-grid check over 2,700 curves. The paper is honest about this in §S8 (C6) and in the Results paragraph, but the summary sentence in Result 1 is stronger than the demonstrated statement. Please rephrase the summary so that the analytic result is distinguished from the numerical-grid evidence, and state the grid and tolerance again in the main text. This does not undermine the empirical section, but it matters because the 'correlation erodes majority protection' claim is one of the paper’s conceptual takeaways.
minor comments (5)
  1. [§S10.4 / Fig. 2B] The raw same-family versus cross-family difference (0.48 vs 0.60) is likely to be read as a family effect, but the adjusted regression gives a same-family coefficient whose 95% interval includes zero. Please state the adjusted result in the main text near Fig. 2B to prevent overinterpretation.
  2. [§S9.2] The sentence 'Collector-level evidence records an 8-token default for other base local terse runs' is unclear. Define 'collector-level evidence' or rephrase to describe the actual token-limit setting.
  3. [Fig. S2 caption] The caption refers to 'Eq. 2 of the main text,' but the main-text equation numbering is not tied to Eq. (2) as a standalone numbered equation in the displayed text. Check the cross-reference or use the equation number consistently.
  4. [Table S6] The table mixes scaled-loss units (×100) and percentages without a unified notation. Please add a footnote or column header that explicitly says which rows are scaled by 100 and which are percentages, to avoid confusion with the main-text numbers.
  5. [§S9.4, Fig. 3C] The cluster bootstrap used for the R² intervals in Fig. 3C is described only in the main text. Add the resampling scheme to the figure caption, since the caption currently reports R² values without stating how the intervals were obtained.

Circularity Check

0 steps flagged

No significant circularity; derivation and validation are self-contained.

full rationale

The paper's theoretical results follow by explicit proof from the stated one-factor Gaussian-copula model (Lemmas S1–S3, Theorems S6–S7, Corollary S8, Propositions S10 and S13), not from the empirical estimates. The empirical claims are supported by a clean odd/even split: marginals and pairwise tetrachoric correlations are estimated on odd-indexed items, and all k-of-n losses are evaluated on even-indexed items; the design-balanced bootstrap fully refits parameters and reselects thresholds in each replicate. The dependence-aware model is therefore not predicting its own fitting target. The Gaussian-copula assumption is an explicit modeling choice, and the paper reports regularizations and diagnostics (PSD projection, Fréchet bounds) rather than presenting them as derived facts. Result 4's direct enumeration is a tautological identity by the definition of loss, but it is used as an algorithm, not as evidence for the empirical conclusions. No load-bearing self-citation was found; the only self-reference is the replication package, which is not used to justify any result. The paper is appropriately self-contained against held-out benchmarks, so the honest finding is no significant circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The central claims rest on estimated reliability parameters (marginals and pairwise latent correlations) and on a parametric Gaussian-copula assumption for how votes co-occur; no new entity is introduced. The heavy PSD/boundary regularization is disclosed, but it is part of what the model contributes versus what is assumed.

free parameters (3)
  • state-conditional approval probabilities p̂θ,i = not reported numerically in main text (data-dependent)
    Estimated from odd-indexed vote frequencies (S9.4); every k-of-n predicted loss is a function of these marginals.
  • state-conditional latent correlations ρ̂θ,ij = not reported numerically in main text (data-dependent)
    Estimated by tetrachoric inversion with Fréchet bounds and PSD projection (S9.4); the full-matrix dependence model is built from them.
  • cost-weighted prior odds κ = {0.25, 0.5, 1, 2, 4}
    Analyst-specified design grid, not fitted; threshold-optimality comparisons and the primary estimand weight these values equally.
axioms (5)
  • domain assumption Exchangeable one-factor Gaussian latent-threshold (Vasicek) model, Eq. S1
    Homogeneous theory assumes votes are conditionally independent given a shared normal factor W with state correlation ρθ.
  • domain assumption Heterogeneous Gaussian copula: Z_i^(θ) ~ N(0, Σθ) with unit diagonal, S7.1
    Empirical predictions assume pairwise tetrachoric correlations plus Gaussianity fully determine joint vote distributions; non-Gaussian or higher-order dependence would bias k-of-n loss.
  • domain assumption Benchmark labels are correct ground truth
    State labels come from VitaminC, GSM8K perturbations, MBPP mutations, and TruthfulQA (S9.1); if labels are noisy, p̂θ and ρ̂θ are biased.
  • domain assumption Tetrachoric inversion is valid after Fréchet bounds and PSD projection
    Regularization diagnostics show 18.48% Fréchet-boundary saturation and 76.8% PSD projection; this assumption is load-bearing for the full-matrix model.
  • standard math Consistency of empirical quantiles and Slutsky's theorem in Theorem S7 proof
    The large-n Vasicek limit uses these standard tools.

reviewed 2026-07-31 · how reviews work

0 comments
Cite this review

Pith. "Pith review of State-dependent error correlations shape voting thresholds in committees of AI agents." pith.science (2026). https://pith.science/paper/DDPANE5U

@misc{pith2026260723931,
  author       = {Pith},
  title        = {Pith review of: State-dependent error correlations shape voting thresholds in committees of AI agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DDPANE5U}},
  note         = {Machine review of arXiv:2607.23931}
}
Share X LinkedIn Reddit HN
read the original abstract

The aggregation benefit of a committee of artificial intelligence (AI) agents comes from complementary information across members. Classical voting guarantees assume independent errors. Language-model errors often co-occur on the same cases. We combine Sah-Stiglitz screening with error dependence that can differ between good and bad cases. In a homogeneous exchangeable Gaussian-copula model, shared errors create a positive asymptotic error floor for majority voting and can change the approval threshold that minimizes expected loss. We estimate a heterogeneous extension from 174,384 votes cast by 28 language models on four binary-screening benchmarks. Parameters estimated from odd-indexed items predicted committee loss on even-indexed items. For the sampled committee composition, the full-matrix dependence model increased identity-line R^2 from 0.840 under independence to 0.967. In a design-balanced analysis, cost-sensitive threshold selection under independence reduced scaled loss from 60.25 for majority to 52.50. Modeling dependence reduced it further to 50.77, an incremental improvement of 1.73 units (95% bootstrap CI, 0.68-2.33). The overall reduction from majority was 15.73% (95% bootstrap CI, 13.41-16.75%).

Figures

Figures reproduced from arXiv: 2607.23931 by Haifeng Li, Mo Hai.

Figure 1
Figure 1. Figure 1: State-dependent error correlation and committee thresholds. (A) The screening model. Agents share a latent factor 𝑊, and their latent scores have correlation 𝜌. A 𝑘-of-𝑛 threshold maps votes to a decision evaluated by cost-weighted loss. (B) Acceptance probabilities versus 𝜌 for hierarchy, majority, and polyarchy (𝑛 = 5) in the good state (𝑝𝐺 = 0.8, solid) and bad state (𝑝𝐵 = 0.25, dashed); each rule’s sig… view at source ↗
Figure 2
Figure 2. Figure 2: Measured correlation structure of machine judgments. (A) Latent error-correlation matrices Σˆ 𝐵 (bad state; tetrachoric estimates between the first temperature sample of each judge, all items; Phi-4 is excluded in code review, Appendix, §S9), by domain; gray cells mark pairs in which a member approves or rejects nearly everything, leaving the tetrachoric correlation unidentifiable, and concentrate in the n… view at source ↗
Figure 3
Figure 3. Figure 3: Validation of committee-loss prediction on held-out items. (A) Heterogeneous full-matrix dependence predictions versus realized loss for all 𝑘-of-𝑛 thresholds under the archived committee composition (73,410 threshold–cost rows). Parameters are estimated from odd-indexed items and evaluated on even-indexed items. (B) Predictions after forcing independence. (C) Descriptive identity-line 𝑅 2 for the full-mat… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

48 extracted references · 1 canonical work pages

  1. [1]

    Stiglitz

    Raaj Kumar Sah and Joseph E. Stiglitz. Human fallibility and economic organization.American Economic Review (Papers and Proceedings), 75(2):292–297, 1985

    1985

  2. [2]

    Stiglitz

    Raaj Kumar Sah and Joseph E. Stiglitz. The architecture of economic systems: Hierarchies and polyarchies.American Economic Review, 76(4):716–727, 1986

    1986

  3. [3]

    Committees,hierarchiesandpolyarchies.TheEconomicJournal, 98(391):451–470, 1988

    RaajKumarSahandJosephE.Stiglitz. Committees,hierarchiesandpolyarchies.TheEconomicJournal, 98(391):451–470, 1988

    1988

  4. [4]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. InAdvances in Neural Information Processing Systems 36, 2023

    2023

  5. [5]

    Replacing judges with juries: Evaluating LLM generations with a panel of diverse models.arXiv preprint arXiv:2404.18796, 2024

    Pat Verga, Sebastian Hofstätter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhang- orodsky, Minjie Xu, Naomi White, and Patrick Lewis. Replacing judges with juries: Evaluating LLM generations with a panel of diverse models.arXiv preprint arXiv:2404.18796, 2024

    Pith/arXiv arXiv 2024

  6. [6]

    Improvingfactuality and reasoning in language models through multiagent debate

    YilunDu,ShuangLi,AntonioTorralba,JoshuaB.Tenenbaum,andIgorMordatch. Improvingfactuality and reasoning in language models through multiagent debate. InProceedings of the 41st International Conference on Machine Learning, volume 235 ofProceedings of Machine Learning Research, pages 11733–11763, 2024

    2024

  7. [7]

    Mixture-of-agents enhances large language model capabilities

    Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. Mixture-of-agents enhances large language model capabilities. InInternational Conference on Learning Representations, 2025

    2025

  8. [8]

    Autogen: Enabling next-gen LLM applications via multi-agent conversations

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, ShaokunZhang,JialeLiu,AhmedH.Awadallah,RyenW.White,DougBurger,andChiWang. Autogen: Enabling next-gen LLM applications via multi-agent conversations. InConference on Language Modeling, 2024

    2024

  9. [9]

    Metagpt: Meta programming for a multi-agent collaborative framework

    SiruiHong,MingchenZhuge,JonathanChen,XiawuZheng,YuhengCheng,CeyaoZhang,JinlinWang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. Metagpt: Meta programming for a multi-agent collaborative framework. In International Conference on Learning Representations, 2024

    2024

  10. [10]

    Chateval: Towards better LLM-based evaluators through multi-agent debate

    Chi-MinChan,WeizeChen,YushengSu,JianxuanYu,WeiXue,ShanghangZhang,JieFu,andZhiyuan Liu. Chateval: Towards better LLM-based evaluators through multi-agent debate. InInternational Conference on Learning Representations, 2024

    2024

  11. [11]

    Imprimerie Royale, Paris, 1785

    Marquis de Condorcet.Essai sur l’application de l’analyse à la probabilité des décisions rendues à la pluralité des voix. Imprimerie Royale, Paris, 1785. 33

  12. [12]

    Self-consistency improves chain of thought reasoning in language models

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations, 2023

    2023

  13. [13]

    Moreagentsisallyouneed.Transactions on Machine Learning Research, 2024

    JunyouLi,QinZhang,YangbinYu,QiangFu,andDehengYe. Moreagentsisallyouneed.Transactions on Machine Learning Research, 2024. arXiv:2402.05120

    Pith/arXiv arXiv 2024

  14. [14]

    Correlated errors in large language models

    Elliot Kim, Avi Garg, Kenny Peng, and Nikhil Garg. Correlated errors in large language models. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 ofProceedings of Machine Learning Research, pages 30038–30066, 2025

    2025

  15. [15]

    Chandra, Ponnurangam Kumaraguru, Douwe Kiela, Ameya Prabhu, Matthias Bethge, and Jonas Geiping

    Shashwat Goel, Joschka Strüber, Ilze Amanda Auzina, Karuna K. Chandra, Ponnurangam Kumaraguru, Douwe Kiela, Ameya Prabhu, Matthias Bethge, and Jonas Geiping. Great models think alike and this undermines AI oversight. InProceedings of the 42nd International Conference on Machine Learning, volume 267 ofProceedings of Machine Learning Research, pages 19621–1...

    2025

  16. [16]

    Krishna K. Ladha. The condorcet jury theorem, free speech, and correlated votes.American Journal of Political Science, 36(3):617–634, 1992

    1992

  17. [17]

    Modellingdependenceinsimpleandindirectmajority systems.Journal of Applied Probability, 26(1):81–88, 1989

    PhilipJ.Boland,FrankProschan,andY.L.Tong. Modellingdependenceinsimpleandindirectmajority systems.Journal of Applied Probability, 26(1):81–88, 1989

    1989

  18. [18]

    MonotonicityinCondorcet’sjurytheoremwithdependentvoters.Social Choice and Welfare, 28(3):507–528, 2007

    DanielBerendandLubaSapir. MonotonicityinCondorcet’sjurytheoremwithdependentvoters.Social Choice and Welfare, 28(3):507–528, 2007

    2007

  19. [19]

    Optimal jury design for homogeneous juries with correlated votes.Theory and Decision, 71(4):439–459, 2011

    Serguei Kaniovski and Alexander Zaigraev. Optimal jury design for homogeneous juries with correlated votes.Theory and Decision, 71(4):439–459, 2011

    2011

  20. [20]

    Optimal decision rules in uncertain dichotomous choice situations

    Shmuel Nitzan and Jacob Paroush. Optimal decision rules in uncertain dichotomous choice situations. International Economic Review, 23(2):289–297, 1982

    1982

  21. [21]

    Random forests.Machine Learning, 45(1):5–32, 2001

    Leo Breiman. Random forests.Machine Learning, 45(1):5–32, 2001

    2001

  22. [22]

    Kuncheva and Christopher J

    Ludmila I. Kuncheva and Christopher J. Whitaker. Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy.Machine Learning, 51(2):181–207, 2003

    2003

  23. [23]

    Towards a science of scaling agent systems.arXiv preprint arXiv:2512.08296, 2025

    Yubin Kim, Ken Gu, Chanwoo Park, Chunjong Park, Samuel Schmidgall, et al. Towards a science of scaling agent systems.arXiv preprint arXiv:2512.08296, 2025

    Pith/arXiv arXiv 2025

  24. [24]

    Too good to be true: When overwhelming evidence fails to convince.Proceedings of the Royal Society A, 472:20150748, 2016

    LachlanJ.Gunn,FrançoisChapeau-Blondeau,MarkD.McDonnell,AndrewAllison,andDerekAbbott. Too good to be true: When overwhelming evidence fails to convince.Proceedings of the Royal Society A, 472:20150748, 2016

    2016

  25. [25]

    Get your vitamin C! robust fact verification with contrastive evidence

    Tal Schuster, Adam Fisch, and Regina Barzilay. Get your vitamin C! robust fact verification with contrastive evidence. InProceedings of NAACL-HLT, pages 624–643, 2021

    2021

  26. [26]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert,JerryTworek,JacobHilton,ReiichiroNakano,ChristopherHesse,andJohnSchulman. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Pith/arXiv arXiv 2021

  27. [27]

    Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021. 34

    Pith/arXiv arXiv 2021

  28. [28]

    Truthfulqa: Measuring how models mimic human falsehoods

    Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. InProceedings of ACL, pages 3214–3252, 2022

    2022

  29. [29]

    Replication package: Correlated errors determine the optimal architecture of AI-agent organizations

    Haifeng Li and Mo Hai. Replication package: Correlated errors determine the optimal architecture of AI-agent organizations. Zenodo, 2026. URLhttps://doi.org/10.5281/zenodo.21332551. Version v7.3.1

    work page doi:10.5281/zenodo.21332551 2026

  30. [30]

    Yale University Press, New Haven, 1972

    Jacob Marschak and Roy Radner.Economic Theory of Teams. Yale University Press, New Haven, 1972

    1972

  31. [31]

    Design of decision-making organizations.Management Science, 56(1):71–89, 2010

    Michael Christensen and Thorbjørn Knudsen. Design of decision-making organizations.Management Science, 56(1):71–89, 2010

    2010

  32. [32]

    Felipe A. Császár. An efficient frontier in organization design: Organizational structure as a determinant of exploration and exploitation.Organization Science, 24(4):1083–1101, 2013

    2013

  33. [33]

    Neural network ensembles, cross validation, and active learning

    Anders Krogh and Jesper Vedelsby. Neural network ensembles, cross validation, and active learning. In Advances in Neural Information Processing Systems 7, pages 231–238, 1995

    1995

  34. [34]

    Dietterich

    Thomas G. Dietterich. Ensemble methods in machine learning. InMultiple Classifier Systems (LNCS 1857), pages 1–15, 2000

    2000

  35. [35]

    Lu Hong and Scott E. Page. Groups of diverse problem solvers can outperform groups of high-ability problem solvers.Proceedings of the National Academy of Sciences, 101(46):16385–16389, 2004

    2004

  36. [36]

    Davis-Stober, David V

    Clintin P. Davis-Stober, David V. Budescu, Jason Dana, and Stephen B. Broomell. When is a crowd wise?Decision, 1(2):79–101, 2014

    2014

  37. [37]

    Encouraging divergent thinking in large language models through multi-agent debate

    Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking in large language models through multi-agent debate. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17889–17904, 2024. doi: 10.18653/v1/2024.emnlp-main.992

    doi:10.18653/v1/2024.emnlp-main.992 2024

  38. [38]

    Algorithmic monoculture and social welfare.Proceedings of the National Academy of Sciences, 118(22):e2018340118, 2021

    Jon Kleinberg and Manish Raghavan. Algorithmic monoculture and social welfare.Proceedings of the National Academy of Sciences, 118(22):e2018340118, 2021

    2021

  39. [39]

    Creel, Ananya Kumar, Dan Jurafsky, and Percy Liang

    Rishi Bommasani, Kathleen A. Creel, Ananya Kumar, Dan Jurafsky, and Percy Liang. Picking on the same person: Does algorithmic monoculture lead to outcome homogenization? InAdvances in Neural Information Processing Systems 35, 2022

    2022

  40. [40]

    Multi-agent risks from advanced AI.arXiv preprint arXiv:2502.14143, 2025

    Lewis Hammond, Alan Chan, Jesse Clifton, Jason Hoelscher-Obermaier, Akbir Khan, Euan McLean, Chandler Smith, Wolfram Barfuss, Jakob Foerster, Tomáš Gavenčiak, et al. Multi-agent risks from advanced AI.arXiv preprint arXiv:2502.14143, 2025

    Pith/arXiv arXiv 2025

  41. [41]

    AI models collapse when trained on recursively generated data.Nature, 631:755–759, 2024

    Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. AI models collapse when trained on recursively generated data.Nature, 631:755–759, 2024

    2024

  42. [42]

    The distribution of loan portfolio value.Risk, 15(12):160–162, 2002

    Oldrich Vasicek. The distribution of loan portfolio value.Risk, 15(12):160–162, 2002

    2002

  43. [43]

    Michael B. Gordy. A risk-factor model foundation for ratings-based bank capital rules.Journal of Financial Intermediation, 12(3):199–232, 2003

    2003

  44. [44]

    The one-sided barrier problem for Gaussian noise.Bell System Technical Journal, 41(2): 463–501, 1962

    David Slepian. The one-sided barrier problem for Gaussian noise.Bell System Technical Journal, 41(2): 463–501, 1962. 35

    1962

  45. [45]

    Plackett

    Robin L. Plackett. A reduction formula for normal multivariate integrals.Biometrika, 41(3–4):351–360, 1954

    1954

  46. [46]

    Mathematicalcontributionstothetheoryofevolution.VII.onthecorrelationofcharacters not quantitatively measurable.Philosophical Transactions of the Royal Society A, 195:1–47, 1900

    KarlPearson. Mathematicalcontributionstothetheoryofevolution.VII.onthecorrelationofcharacters not quantitatively measurable.Philosophical Transactions of the Royal Society A, 195:1–47, 1900

    1900

  47. [47]

    George Shanthikumar.Stochastic Orders

    Moshe Shaked and J. George Shanthikumar.Stochastic Orders. Springer, New York, 2007

    2007

  48. [48]

    Princeton UniversityPress,Princeton,1998

    DonaldM.Topkis.SupermodularityandComplementarity. Princeton UniversityPress,Princeton,1998. 36

    1998

This paper was first reviewed by deepseek-v4-flash on July 31, 2026.