{"id":"503498da-f04a-4c7f-8296-9de291cd24ef","arxiv_id":"2606.27288","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Any single-output LLM ensemble is accuracy-capped at 1-beta where beta is the all-models-wrong rate, a quantity not captured by pairwise correlations and frequently underestimated by copula models.","lead":"This paper shows that any LLM combination method outputting one member model's answer cannot exceed accuracy of 1 minus beta, where beta is the fraction of queries on which every model fails. Standard pairwise error correlations do not determine beta, and measurements across 67 models indicate the bound is often tight on math and code tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Bound holds only for selection policies; title/abstract include synthesis methods (MoA, fusion) without explicit restriction","rationale":"The reader's weakest_assumption matches the load-bearing gap exactly. The mathematical bound itself is tautological once the selection restriction is granted; the risk lies in whether the listed methods satisfy that restriction. Because the reader could not access the full text, the current UNVERDICTED verdict is appropriate until the scope clarification is verified.","tokens_in":1827,"tokens_out":331,"duration_ms":51841,"concrete_test":"Search the full text for every occurrence of 'mixture-of-agents' or 'fusion'; extract the operational definition used and whether the 1-\beta statement is applied to those methods or explicitly restricted to selection-only policies. If the bound is invoked for MoA without the restriction, the headline claim overreaches.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim states that accuracy ≤ 1-\beta only for policies whose output is exactly one member model answer. The title and abstract list 'routing, voting, cascades, fusion, and mixture-of-agents' as the systems under study and then state the bound. Mixture-of-Agents and fusion frameworks typically allow an aggregator to synthesize a new token sequence rather than emit one of the raw member outputs; on queries where all members are wrong, such synthesis can still be correct. The paper therefore applies the 1-\beta ceiling to a broader class than the stated hypothesis covers. No section quoted in the provided material derives or qualifies the bound for synthesis.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript defines β as the probability that every model in an ensemble is simultaneously wrong on a given query and proves that any policy whose output must be exactly one of the member models' answers cannot exceed accuracy 1−β. It shows that the conventional diagnostic of average pairwise error correlation ρ is insufficient to identify β, supplies a Clopper-Pearson finite-sample upper bound on β, and reports that across 67 frontier models a Gaussian-copula model underestimates observed β (0.052 vs. 0.023 on open-ended math; 0.079 on code; 0.127 on free-response GPQA). The paper concludes that, on checkable tasks, gains from combining models are driven by query-level routing rather than by adding models or lowering ρ.","tokens_in":1990,"tokens_out":649,"duration_ms":40492,"significance":"If the logical bound is accepted and the empirical β measurements prove robust, the work supplies a simple, reportable quantity that places a hard ceiling on the upside of any selection-based router, vote, or cascade before any training occurs. The demonstration that identical marginal error rates and pairwise ρ can produce materially different β values is a clarifying observation for the ensemble literature. The scale of the study (67 models, multiple task types, execution grading) lends weight to the claim that co-failure tails are heavier than low-dimensional factor models predict. These elements constitute a constructive, falsifiable contribution to understanding the limits of multi-LLM systems.","major_comments":[{"comment":"Abstract and title: The central bound is derived only for policies that output exactly one member model's answer, yet the title and abstract list 'routing, voting, cascades, fusion, and mixture-of-agents' as the systems under study and state that 'their gain is capped.' Mixture-of-Agents and fusion frameworks commonly synthesize a new token sequence rather than emit a raw member output; on queries where all members fail, such synthesis can still be correct. The manuscript must either restrict the title/abstract/claim to selection policies or derive/qualify the bound for synthesis methods.","section":"Abstract / Title"},{"comment":"§ on Gaussian-copula comparison (math and code results): The statement that a 'tetrachoric-calibrated single-factor model still underprices the all-wrong tail' (observed β = 0.052 vs. 0.023) requires an explicit statement of how the copula parameters are estimated, whether the fitted marginals exactly match the empirical per-model error rates, and whether the 90 % CI (1.7–3.4) accounts for dependence across the 67 models. Without these details the factor-model underpricing claim cannot be evaluated.","section":"Empirical results (math / code)"}],"minor_comments":[{"comment":"The abstract states 'k equals 17' without defining k; the main text should state what this integer represents (e.g., number of models or a hyper-parameter).","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for these precise comments on scope and methodological transparency. Both points are addressable by targeted revisions that preserve the paper's central claims and results.","responses":[{"response":"We agree the 1−β bound applies strictly to selection policies. The body already qualifies the claim accordingly. We will revise the title and abstract to read 'selection-based ensembles (routing, voting, cascades)' and insert a clarifying sentence noting that synthesis methods such as Mixture-of-Agents can in principle exceed the bound by generating novel outputs. This aligns front matter with the theorems without altering any proofs or experiments.","revision_made":"yes","referee_comment":"[Abstract / Title] Abstract and title: The central bound is derived only for policies that output exactly one member model's answer, yet the title and abstract list 'routing, voting, cascades, fusion, and mixture-of-agents' as the systems under study and state that 'their gain is capped.' Mixture-of-Agents and fusion frameworks commonly synthesize a new token sequence rather than emit a raw member output; on queries where all members fail, such synthesis can still be correct. The manuscript must either restrict the title/abstract/claim to selection policies or derive/qualify the bound for synthesis methods."},{"response":"We will add a dedicated paragraph in the methods section stating: pairwise tetrachoric correlations are estimated directly from the 67×query error-indicator matrix; marginal error probabilities are fixed exactly to the empirical per-model rates; the single-factor Gaussian copula is then sampled to obtain the model-predicted β. The 90 % CI on the ratio (observed β / copula β) is a nonparametric bootstrap over queries and does not incorporate cross-model dependence; we will explicitly note this limitation. These additions make the underpricing claim (factor ≈2.5 on math) fully evaluable from the released data.","revision_made":"yes","referee_comment":"[Empirical results (math / code)] § on Gaussian-copula comparison (math and code results): The statement that a 'tetrachoric-calibrated single-factor model still underprices the all-wrong tail' (observed β = 0.052 vs. 0.023) requires an explicit statement of how the copula parameters are estimated, whether the fitted marginals exactly match the empirical per-model error rates, and whether the 90 % CI (1.7–3.4) accounts for dependence across the 67 models. Without these details the factor-model underpricing claim cannot be evaluated."}],"tokens_in":1718,"tokens_out":548,"duration_ms":23959,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main takeaway is that any system forced to output exactly one member model's answer cannot beat 1 minus beta, the rate at which all models fail the same query. Pairwise error correlations do not pin down beta, so the usual rho diagnostic misses the tail. That separation is the cleanest part of the work.\n\nThe empirical section does something useful: across 67 frontier models the observed beta on math is 0.052 while a Gaussian copula fitted to the same marginals and pairwise correlations gives only 0.023, and the gap repeats on code and free-response GPQA. The Clopper-Pearson certificate on beta is also a practical addition for anyone sizing a router. Those numbers are worth having even if the grading details need scrutiny.\n\nThe soft spot is the framing. The abstract and title list routing, voting, cascades, fusion, and mixture-of-agents together, then state the 1-beta ceiling. The bound is derived only for policies that emit one raw member output; synthesis or external verification can still recover on all-wrong queries. The stress-test concern lands. The paper is careful in the body but the headline claim is broader than the proof.\n\nThe citation pattern looks light on prior ensemble bounds, but the copula comparison and the format effect on GPQA are new enough to stand on their own. This is for people shipping multi-model systems on checkable tasks who want a quick limit on expected gains. It is not a full theory of ensembles, but the observation is sharp enough that a serious editor should send it out for review rather than desk-reject.","headline":"The paper's core bound on selection policies is straightforward and the 67-model data shows copulas underpredict joint failures, but the title and abstract fold in synthesis methods where the bound does not apply.","tokens_in":2465,"tokens_out":411,"would_cite":true,"duration_ms":27417,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Any policy selecting one model answer from an ensemble cannot exceed accuracy of one minus the rate at which all models fail together.","keywords":["LLM ensembles","model routing","voting","mixture of agents","co-failure rate","error correlation","frontier models","accuracy bounds"],"falsifier":"An ensemble policy that selects one member answer and achieves accuracy higher than one minus the empirically measured beta on the same query set would contradict the bound.","tokens_in":2722,"feed_emoji":"📉","tokens_out":698,"duration_ms":29012,"temperature":0.7,"pith_summary":"The paper shows that ensembles outputting exactly one member answer face a hard ceiling of 1 minus beta, where beta is the fraction of queries on which every model errs. Pairwise error correlations cannot identify this beta because multiple joint distributions can share the same marginals and pairwise terms yet differ in the all-wrong probability. Measurements across 67 frontier models from 21 providers find beta of 0.052 on open-ended math and 0.079 on code execution, exceeding single-factor and Gaussian-copula predictions. On these tasks, low-correlation heterogeneous sets still rarely surpass the single best model without strong per-query routing. Re-asking questions in free-response format raises beta, locating part of the co-failure in answer format rather than domain knowledge.","feed_headline":"Ensemble accuracy capped by all-models-wrong rate","feed_subtitle":"Across 67 models, queries where every model fails occur at 5-13 percent, limiting routers and voters even at low pairwise correlation","key_machinery":"The co-failure rate beta, the probability that all models in the ensemble err simultaneously on a given query.","core_discovery":"For any policy whose output is one member model answer, accuracy cannot exceed one minus beta, where beta is the rate at which every model is wrong on the same query. The usual diagnostic of average pairwise error correlation rho cannot identify beta. A Clopper-Pearson bound on beta supplies a finite-sample certificate on the largest gain any router, vote, or cascade could deliver. Across 67 models, observed beta exceeds copula predictions, and gains appear only when models fail on different questions and routing exploits that difference.","pith_inferences":["Systems that synthesize new answers or add external verification steps can in principle exceed the 1-beta ceiling that applies to pure selection policies.","Better tail modeling beyond Gaussian copulas would be needed to predict the largest achievable gains before training routers.","Reducing simultaneous failures across models may matter more for ensemble performance than further lowering average pairwise correlation."],"forward_implications":["Routers, voters, and cascades cannot exceed 1-beta no matter how they choose among member answers.","Identical pairwise correlations can hide different beta values, so rho alone does not certify ensemble headroom.","On checkable tasks, adding models without query-level routing rarely beats the single best model.","Changing answer format from multiple-choice to free-response can reopen the co-failure tail even on the same questions."],"fun_headline_variants":["Co-failure rate caps LLM routing and voting gains","All-wrong rate sets bound on 67-model ensemble accuracy","Beta limits mixture-of-agents beyond pairwise rho","Shared failures bound gains from model mixing","67 models expose copula underprediction of co-failure"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The policy must output exactly one of the member models' answers rather than synthesizing a new answer or using external verification.","fun_headline_variants_meta":{"raw":{"variants":["Co-failure rate caps LLM routing and voting gains","All-wrong rate sets bound on 67-model ensemble accuracy","Beta limits mixture-of-agents beyond pairwise rho","Shared failures bound gains from model mixing","67 models expose copula underprediction of co-failure"]},"model":"grok-4.3","cost_usd":0.004847,"raw_usage":{"total_tokens":2460,"prompt_tokens":827,"num_sources_used":0,"completion_tokens":73,"cost_in_usd_ticks":48474500,"prompt_tokens_details":{"text_tokens":827,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1560,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":827,"tokens_out":73,"duration_ms":21514,"temperature":1.0,"reasoning_tokens":1560,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-26T04:16:29.088449+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"An ensemble policy that selects one member answer and achieves accuracy higher than one minus the empirically measured beta on the same query set would contradict the bound.","supporting_citations":[],"review_version":1}