{"id":"dcaaad3c-5690-42d1-af2f-a773fe3cebbe","arxiv_id":"2611.16315","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Teams of 30 language-model agents reduce single-agent Fermi-estimation error by only 1.7% because 90.6% of the error variance is shared across all draws on an item.","lead":"This paper runs 2.51 million Fermi-estimation answers from 15 open-weight language models and finds that most of a model's error is shared across all of its answers to the same question. Because of that shared error, averaging 30 answers barely improves accuracy, giving only about a 2% reduction instead of the 74-79% reduction that independent, unbiased answers would provide.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"The one-way ANOVA estimate of ρ assumes the item-level component c_j is homogeneous across runs; the run component accounts for up to 11% of variance in 6 models, and the main sweep mixes two prompt pairs unevenly across models, so the headline ρ=0.906 may be inflated by run-level and…","rationale":"The reader's weakest_assumption correctly identifies that the decomposition treats c_j as common to all draws of a model on an item, and notes the run component of up to 11% variance in 6 models. However, the reader does not fully develop the consequence: that the one-way ANOVA by item conflates item-level variation with run-level and prompt-level variation, potentially inflating ρ. The paper's own Table D.2 shows that for some models (e.g., mistral-7b, run component 10.6%; smollm3-3b, 8.1%; llama3.1-8b, 6.7%) the run component is substantial, and the prompt-pair mixing adds further unmeasured variation. The second corpus replication (ρ=0.915) uses a single prompt pair but still includes run variation; it is reassuring but not a full control. The concern is not that the paper is wrong, but that the headline ρ may be an overestimate of the truly item-level share, and the paper's own data provide the means to check this. The concrete test is feasible with the existing data (if released) or by the authors. The verdict remains CONDITIONAL because the paper's own limitations and the second corpus support the qualitative conclusion, but the precise magnitude of ρ is load-bearing for the quantitative claim about the 1.7% reduction. I agree with the reader that the central result is plausible and the paper is honest about limitations; my concern is more specific about the variance decomposition's sensitivity to uncontrolled factors.","tokens_in":28743,"tokens_out":868,"duration_ms":212760,"concrete_test":"Re-estimate ρ using a mixed-effects model that includes random intercepts for item, run, and prompt pair (where known), on the round-1 draws of the main sweep. If the item-level variance component shrinks by more than 0.05 relative to the one-way ANOVA estimate, then the headline ρ is inflated by run/prompt variation and the central claim about the limit of aggregation is weaker than reported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that item-level error accounts for 90.6% of variance (ρ=0.906) rests on a one-way ANOVA by item that pools all round-1 draws of a model, treating every draw as sharing the same item-level component c_j. However, the authors report that a run component (deviation of a specific run from the model's other runs) accounts for up to 11% of error variance in 6 models (Table D.2), and that the main sweep mixes two prompt pairs unevenly across models, with the revised pair lowering MAE@5 by −0.20 OOM in models that ran both (Table B.2). Because runs and prompt pairs are not crossed with items in the ANOVA, the between-item variance A absorbs run-level and prompt-level variation that is common to all draws of a run but not truly a property of the item. The second corpus uses a single prompt pair and reproduces ρ=0.915 (Table 2), which is close but not identical; the run component remains. If the run and prompt variation were removed, ρ would decrease, and the available reduction from aggregation would increase. The paper acknowledges the run component but does not quantify how much of the between-item variance A is actually due to runs/prompts rather than items, nor does it report a decomposition that controls for run. This is load-bearing because the headline claim that aggregation is nearly useless depends on ρ being near 0.9; a ρ of, say, 0.8 would imply Neff(30)=1.27 instead of 1.10 and a larger potential gain.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper decomposes the log10 error of language-model Fermi estimates into an item-level component common to all draws of a model on an item and a residual, and estimates both from 2.51M answers by 15 open-weight models on realFP and other benchmarks. The central claim is that the item-level component accounts for a median 0.906 of the error variance (ρ=0.906), so teams of up to 30 agents reduce single-agent error by only a few percent, far less than the 74–79% reduction that would be possible if errors were unbiased and independent given the item. The paper further shows that debate, temperature changes, and activation steering do not reduce the item-level component, and that mixing models yields modest gains without surpassing the best single model. The results are conditional on the models, benchmarks, prompts, and settings tested.","tokens_in":29062,"tokens_out":2071,"duration_ms":20610,"significance":"If the decomposition and the headline ρ ≈ 0.9 hold, the paper provides a clear statistical explanation for why multi-agent aggregation of language models on continuous estimation tasks yields much smaller gains than the 1/√N averaging argument would predict. The finding that the item-level error is largely irreducible by standard aggregation or communication protocols is practically important for the design of multi-agent systems. The paper is thorough: it includes two corpora, multiple benchmarks, robustness checks (median rule, item-clustered intervals, alternative reference), and explicit limitations. The decomposition itself is a standard one-way ANOVA, but its application to Fermi estimation is novel and the empirical scale of the study (2.51M answers, 15 models) is a strength.","major_comments":[{"comment":"The one-way ANOVA estimate of ρ pools all round-1 draws of a model, treating every draw as sharing the same item-level component c_j. However, the authors report that a run component (deviation of a specific run from the model's other runs) accounts for up to 11% of error variance in 6 models (Table D.2), and that the main sweep mixes two prompt pairs unevenly across models, with the revised pair lowering MAE@5 by −0.20 OOM in models that ran both (Table B.2). Because runs and prompt pairs are not crossed with items in the ANOVA, the between-item variance A absorbs run-level and prompt-level variation that is common to all draws of a run but not truly a property of the item. The second corpus uses a single prompt pair and reproduces ρ=0.915 (Table 2), which is close but not identical; the run component remains. If the run and prompt variation were removed, ρ would decrease, and the available reduction from aggregation would increase. The paper acknowledges the run component but does not quantify how much of the between-item variance A is actually due to runs/prompts rather than items, nor does it report a decomposition that controls for run. This is load-bearing because the headline claim that aggregation is nearly useless depends on ρ being near 0.9; a ρ of, say, 0.8 would imply Neff(30)=1.27 instead of 1.10 and a larger potential gain. I request a sensitivity analysis: re-estimate ρ after partialling out run-level means (or using a two-way ANOVA by item and run) for the subset of models with multiple runs, and report the range of ρ under that decomposition.","section":"§3.1, Eq. (1)–(2); §4.1; Table D.2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The 'unbiased given the item' reference (draws with independent random signs) is used to compute the share of the available reduction that real teams realise (median 2.2%). This reference removes µ + c_j entirely, which is the strongest possible counterfactual. The paper also reports a 'pooled' reference (draws across items with random signs) in Appendix D, which gives a smaller available reduction (51–63%) and a share realised of at most 10.7%. The choice of reference affects the interpretation of 'how much aggregation could in principle achieve'. The paper should state more prominently that the 74–79% figure is an upper bound that assumes not only independence but also zero item-level error, and that the more realistic reference (across items) still leaves the real teams far short. This is not a flaw but a clarity issue; the current presentation in the abstract and Section 5 emphasises the 74–79% figure without immediately noting that it is an extreme bound.","section":"§3.2; §5; Table 2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The per-agent scaffold arm improves team MAE@5 by −0.104 OOM (p=0.022 unadjusted, p=0.089 after Holm correction). The paper states that this does not survive correction for multiplicity and is not significant under the median rule. However, the conclusion in the abstract and Section 7 that 'per-agent scaffold prompts (not significant after multiplicity correction)' is accurate but the presentation in the main text (Table 4) shows the unadjusted p-value and the Holm-adjusted p-value, which is appropriate. No change required, but I note that the evidence for scaffolds is weak and the paper's cautious language is correct.","section":"§7; Table 4; Appendix G","authors_acknowledge":"The screen stored only per-item team errors computed with the earlier parser and the median rule, and covers one model and 150 items, so it identifies a candidate rather than an established effect.","authors_acknowledge_p":0.81},{"comment":"The defective parser in rounds 2 and 3 (22.5% of homogeneous teams, 32.6% of heterogeneous teams) is a serious data-quality issue for the debate results. The authors acknowledge it and state that excluding those teams leaves the homogeneous results unchanged but removes the significance of the SciOly difference and the degradation of the heterogeneous pool. They also state 'We therefore draw no conclusion about debate in heterogeneous teams.' This is appropriate. However, the paper still reports debate results for homogeneous teams (Table 2, Table F.1) and uses them to support the claim that debate does not reduce item-level error. Given that the parser error affects what agents were shown, not just the scoring, and that the error rate is substantial, I would like the authors to confirm that the homogeneous-team debate results are robust to excluding teams with misparsed inputs (they state this in Section 6 but only for the comparison with self-revision; please also confirm for the main debate statistics in Table 2 and Table F.1).","section":"§6; Table 3; §8","authors_acknowledge":"Heterogeneous teams and misparsed inputs.Rounds 2 and 3 were generated by a harness whose parser misread some answers written as a mantissa times a power of ten, so in 22.5% of the homogeneous and 32.6% of the heterogeneous teams an agent was shown a peer value that differed from the one writ- ten.","authors_acknowledge_p":0.65}],"minor_comments":[{"comment":"The abstract states 'Teams of thirty therefore reduce a single agent's error by a median 1.7% (at most 5.4%), whereas draws that were also unbiased on each item would reduce it by 74–79%.' The phrase 'draws that were also unbiased on each item' could be misinterpreted as a realistic alternative; consider adding 'under the strong counterfactual that the item-level error is zero' or similar.","section":"Abstract","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The definition of Neff(N) = N / (1 + (N-1)ρ) is the Kish design effect. It would be helpful to note that this formula assumes equal cluster sizes and homoscedasticity, which approximately hold here.","section":"§3.1, Eq. (2)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The run component is reported as a percentage of error variance for each model. It would be useful to also report the run component as a share of the between-item variance A, to directly assess how much of A might be inflated by run-level variation.","section":"§4.1, Table D.2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The right panel of Figure 2 shows the median over models of three references. The kink at N=2 for the 'unbiased given the item' reference is explained in the caption as arising because identical draws with opposite signs cancel exactly. This is correct but the explanation could be expanded slightly for clarity.","section":"§5, Figure 2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The column '∆A(%)' for the calculator-offload arms is marked as not interpretable because winsorization lowers A when an arm inflates the within-item spread. Consider adding a footnote or note in the table caption to this effect, as it is currently only in the text.","section":"§7, Table 4","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The activation screen uses unwinsorized A and the median rule. The paper notes this. It would be helpful to state explicitly that the screen's results are not directly comparable to the main sweep's winsorized estimates.","section":"Appendix G, Table G.3","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper uses 'OOM' as an abbreviation for 'orders of magnitude'. This is standard but should be defined at first use (it is defined in the introduction).","section":"Throughout","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"Hey,\n\nThis paper is worth reading. It does something clean: it decomposes the error of LLM estimates on Fermi problems into an item-level component that every draw of a model shares and a residual that averages out. The headline result—that item-level error accounts for a median 90.6% of variance, so teams of thirty reduce error by only 1.7%—is internally consistent and supported by two corpora, a second estimator via team means, and robustness checks. The paper is unusually honest about its limitations: the prompt-pair mixing, the defective parser in debate rounds, the unregistered analysis plan, the small Guesstimate set. That candor earns trust.\n\nWhat is new: the quantitative decomposition itself. Prior work reported correlated errors on discrete benchmarks, but here the continuous scale lets them measure the common component in the units of the answer and show that it is the item, not dependence between agents, that kills aggregation. The finding that debate collapses dispersion without changing the item-level error, and that self-revision matches peer revision, is a clean replication of earlier results in a new setting.\n\nThe soft spots are real but not fatal. The one-way ANOVA by item pools all round-1 draws of a model, treating every draw as sharing the same item-level component c_j. But the authors report that a run component accounts for up to 11% of variance in 6 models, and the main sweep mixes two prompt pairs unevenly across models. Because runs and prompt pairs are not crossed with items in the ANOVA, the between-item variance A absorbs run-level and prompt-level variation that is common to all draws of a run but not truly a property of the item. The second corpus uses a single prompt pair and reproduces rho=0.915, which is close, but the run component remains. If that variation were removed, rho would decrease and the available reduction from aggregation would increase. The paper acknowledges the run component but does not quantify how much of A is actually due to runs/prompts rather than items, nor does it report a decomposition that controls for run. This is load-bearing because the headline claim depends on rho being near 0.9; a rho of 0.8 would imply Neff(30)=1.27 instead of 1.10 and a larger potential gain.\n\nThe other soft spot is reproducibility: the paper claims a complete rerun reproduces all numbers exactly, but no code or data is released. Given the size of the dataset and the complexity of the pipeline, that claim cannot be verified. The authors should release the code and scored answers.\n\nWho this is for: anyone working on multi-agent LLM systems, forecast combination, or wisdom of crowds. It deserves a serious referee. I would accept peer review and recommend the same, with the expectation that the authors address the run/prompt confound and release code and data.\n\nBest,\n[Your name]","headline":"The paper convincingly shows that on continuous estimation tasks, the 1/sqrt(N) averaging argument fails because item-level error dominates, but the headline rho=0.906 may be slightly inflated by run/prompt variation not controlled in the ANOVA.","tokens_in":29739,"tokens_out":713,"duration_ms":8396},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"Shared item-level error limits multi-agent LLM aggregation in Fermi estimation","keywords":["Fermi estimation","multi-agent systems","language models","aggregation","error decomposition","intraclass correlation","wisdom of crowds"],"falsifier":"If a team of 30 agents from a single model reduced error by more than 10% on a large, diverse Fermi benchmark (e.g., realFP), the claim that item-level error dominates would be contradicted.","tokens_in":30043,"feed_emoji":"📊","tokens_out":1334,"duration_ms":10840,"temperature":0.7,"pith_summary":"The paper tries to establish that the dominant source of error in language-model-based Fermi estimation is not the variance between draws of the same model on an item, but the shared error that every draw of that model makes on that item. This item-level component accounts for about 90% of the error variance, so aggregating many agents (even 30) reduces error by only a few percent, far less than the 1/√N rate that motivates multi-agent systems. The finding matters because it challenges the premise that simply adding more agents can substitute for a better model, and redirects attention to changing the item-level error itself.","feed_headline":"Shared error limits multi-agent LLM gains in Fermi estimation","feed_subtitle":"Item-level component accounts for 90% of error variance; teams of 30 reduce error by only 1.7%","key_machinery":"The error decomposition ejk = μ + cj + ujk, where cj is the item-level component common to all draws of a model on item j, and ρ = A/(A+B) is the intraclass correlation over items. This decomposition isolates the part of the error that aggregation cannot remove (μ+cj) from the part it can (ujk). The design effect Neff(N) = N/(1+(N-1)ρ) then quantifies the effective number of independent agents.","core_discovery":"The paper decomposes the log10 error of each answer into a model-item mean error (cj) and a residual (ujk). The item-level component cj accounts for a median 90.6% of the error variance over items (ρ=0.906). Given the item, teammates' residuals are uncorrelated, so teams behave as independent draws, but the shared error passes through unchanged under aggregation rules like mean, median, or trimmed mean. Consequently, teams of 30 reduce single-agent error by a median 1.7% (max 5.4%), whereas unbiased independent draws would reduce it by 74–79%. The paper also finds that all models compress estimates toward $10^{6}$, explaining a median 17% of cj, and that communication (debate) reduces within-team dispersion by 93% but leaves cj unchanged.","pith_inferences":["If the item-level error is mostly due to shared training data or model architecture, then combining models from different families or training paradigms may reduce cj more than combining similar models.","The findings suggest that for Fermi estimation, the practical ceiling on aggregation is low; a single strong model (like gpt-oss-20b) already outperforms any mixture of weaker models.","The result may extend to other tasks where errors are dominated by item-level factors (e.g., factuality, numerical reasoning), implying that multi-agent systems in those domains also face a similar limit.","The paper's decomposition method could be applied to other continuous-output tasks to diagnose whether aggregation is worth pursuing."],"forward_implications":["Multi-agent LLM systems that rely on averaging or voting without addressing shared item-level error will see diminishing returns beyond a few agents.","Benchmarks like realFP and SciOly have item-level error structures that may generalize to other continuous estimation tasks, limiting the wisdom-of-crowds benefit.","Interventions that reduce the item-level component (e.g., model edits, different prompts, or recalibration) are more promising than increasing team size or debate.","The compression effect (estimates regressing toward 10^6) suggests a systematic bias that linear recalibration can partially remove but not fully eliminate.","Temperature increases and activation steering do not meaningfully change team error, so diversity interventions that only increase variance are ineffective."],"supporting_citations":[{"why":"Provides the design effect formula Neff(N) = N/(1+(N-1)ρ) that the paper uses to quantify the effective number of independent agents.","marker":"Kish (1965)"},{"why":"Establishes that positively correlated errors limit the benefit of averaging forecasts, which the paper tests empirically.","marker":"Clemen & Winkler (1986)"},{"why":"Provides the Guesstimate benchmark and reports large gains from aggregating sampled estimates of one model, which the paper replicates and explains via lower ρ.","marker":"Chuang et al. (2025)"},{"why":"Introduces the realFP benchmark used as the primary dataset in the paper.","marker":"Kalyan et al. (2021)"},{"why":"Reports correlated errors across language models on discrete benchmarks, which the paper extends to continuous Fermi estimation.","marker":"Kim et al. (2025)"},{"why":"Proposes multi-agent debate to improve factuality and reasoning, which the paper tests and finds ineffective for reducing item-level error.","marker":"Du et al. (2024)"},{"why":"Introduces self-consistency (sampling and voting) that motivates multi-agent aggregation, which the paper's results qualify.","marker":"Wang et al. (2023)"}],"fun_headline_variants":[],"cache_read_input_tokens":0,"weakest_assumption_plain":"The decomposition ejk = μ + cj + ujk assumes that the item-level component cj is a property of the model-item pair that is common to all draws, and that the benchmark's items are a representative distribution of the quantities of interest.","fun_headline_variants_meta":{"raw":{"variants":[]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":8.8e-05,"raw_usage":{"cost":8.82882e-05,"is_byok":false,"cost_details":{"upstream_inference_cost":8.82882e-05,"upstream_inference_prompt_cost":8.79242e-05,"upstream_inference_completions_cost":3.64e-07},"total_tokens":1173,"prompt_tokens":1171,"completion_tokens":2,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":0}},"tokens_in":1171,"tokens_out":2,"cost_source":"openrouter_usage","duration_ms":2545,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":0,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-11-13T06:50:03.783654+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"If a team of 30 agents from a single model reduced error by more than 10% on a large, diverse Fermi benchmark (e.g., realFP), the claim that item-level error dominates would be contradicted.","supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}