Pith. sign in

REVIEW 3 major objections 4 minor 40 references

Learned linear aggregation of 15 language models' forecasts beats every individual model and all classical rules, with the gain traced to error-pattern diversity; training-cutoff contamination inflates apparent model rankings.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-02 12:59 UTC pith:QXY54JEB

load-bearing objection Learned aggregation of LLM forecasts works out-of-sample and the contamination analysis is valuable, but the symbolic-regression evidence for the diversity mechanism is partially in-sample and needs a corrected evaluation. the 3 major comments →

arxiv 2607.18269 v2 pith:QXY54JEB submitted 2026-05-26 cs.AI cs.LG

Wisdom of LLM Crowds: Aggregation and Contamination in Language Model Ensembles

Igor Douven This is my paper
classification cs.AI cs.LG
keywords wisdom of crowdslarge language modelsprobability aggregationensemble methodstraining data contaminationprediction marketssymbolic regressionerror diversity
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks whether a 'crowd' of large language models displays the wisdom-of-crowds effect documented in human forecasters. It claims yes, with a twist: the benefit of aggregating 15 models' probability estimates comes not from picking expert models or from complex nonlinear interaction, but from learning a linear combination that exploits how models' errors decorrelate. The strongest evidence is that a logistic regression matches a trained neural network, and that symbolic regression distills the network's strategy into a simple difference between two models' estimates. The paper also shows that training-cutoff contamination is pervasive: it inflates the apparent cloud-vs-local capability gap from 8.9% to 35.8% on contaminated questions, so contamination-free evaluation is essential for any such comparison.

Core claim

On a set of 94 prediction questions that resolve after every model's training cutoff, both learned aggregators—a multilayer perceptron and a logistic regression—beat all 15 individual models and all classical aggregators, with the linear model matching the nonlinear one. This near-equivalence, together with the symbolic-regression finding that the network's mapping collapses to a two-model contrastive formula, leads the author to conclude that the aggregation benefit is carried by error-pattern diversity: the learned weights track how independent a model's errors are from the rest of the crowd, not how accurate the model is. The same diversity mechanism is what has been documented in human c

What carries the argument

The central machinery is a 'temporal transfer' evaluation design: the learned aggregators (an MLP and an L2-regularized logistic regression) are trained on 160 questions that resolve before all models' cutoffs and tested on 94 questions that resolve after them, so the test questions are genuinely novel. The paper's main interpretive tool is symbolic regression, which distills the trained MLP's input-output map into interpretable formulas; the lowest-complexity useful formula on the Pareto frontier is a logistic transform of the difference between two models' probability estimates, a pure disagreement signal. A coefficient analysis of the logistic regression completes the machinery, showing t

Load-bearing premise

The clean-subset evaluation of the recovered two-model formula is not fully out-of-sample: the formula was distilled from an MLP trained on 196 items, 48 of which are also in the 94-item clean set, so those 48 clean scores may reflect indirect fitting to ground truth.

What would settle it

Recompute the Brier score of the two-model contrastive formula on only the 46 clean items that are not in the 208-item training set; if the formula's advantage over the arithmetic mean disappears on that disjoint subset, the symbolic-regression evidence for the diversity mechanism is nullified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If the diversity mechanism is real, then a simple linear reweighting of model outputs is sufficient to harvest most of the wisdom of an LLM crowd; elaborate nonlinear ensembles add little.
  • Any evaluation of LLM forecasting on resolved events must control for training-cutoff contamination; otherwise published capability gaps and model rankings are systematically distorted.
  • LLM crowds can match or beat their best individual member, so ensemble forecasting with diverse models is a viable, scalable alternative to single-model systems.
  • The persistence of a roughly twofold Brier-score gap against a dynamically updated market, even at matched cutoffs, sets a clear target: frozen-snapshot LLMs lack real-time information aggregation, so the ceiling is structural rather than an artifact.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Testable extension: an explicitly diversity-weighted linear aggregator, with weights proportional to measured error decorrelation and no learned parameters, should reproduce much of the MLP's gain; if it does not, the linearity story is incomplete.
  • If contamination distorts rankings this much, the same mechanism likely affects other benchmark comparisons on dated event data; re-running such evaluations on post-cutoff items could shrink reported capability gaps elsewhere.
  • The U-shaped model selection (strong cloud plus weak local models) suggests an ensemble design rule: inject deliberately worse but decorrelated models rather than only the best performers; this is implicit in the paper but a prospective test would harden the claim.
  • A deliberation protocol in which models observe each other's estimates could narrow the gap to dynamic markets, but the cutoff-matched result implies a residual gap would remain because no fixed training snapshot can incorporate new information.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper reports an experiment in which probability estimates on 254 binary prediction-market questions are elicited from 15 LLMs, and four classical aggregators (arithmetic mean, median, geometric/log-odds mean, harmonic mean) are compared with two learned aggregators (L2-regularized logistic regression and a two-hidden-layer MLP). The learned aggregators are trained on 160 items resolving before 1 September 2025 and evaluated on a 94-item 'clean' subset resolving after that date, a genuine temporal-transfer design. The paper's central claims are that the learned aggregators outperform all individual models and all classical aggregators; that logistic regression performs comparably to the MLP, implying the benefit comes from a linear combination of diverse outputs; that symbolic regression recovers a simple disagreement formula σ(p_GeminiPro − p_ClaudeHaiku) that nearly replicates the learned-aggregation benefit; and that training-cutoff contamination substantially inflates apparent model performance and distorts model rankings, with the cloud--local gap collapsing from 35.8% to 8.9% on clean items.

Significance. If the claims hold, the paper makes a meaningful contribution to both collective-intelligence research and LLM evaluation methodology. The temporal-transfer design for the learned aggregators is a real strength: it avoids the circularity that would arise if aggregators were trained and tested on the same items. The contamination analyses are thoughtful and include useful robustness checks (cutoff-date perturbations and a nested clean subset). The supplementary materials appear to ship code and data, which is welcome. However, two load-bearing points currently outrun the evidence: the claim that learned aggregators beat every individual model is not backed by explicit clean-subset individual scores, and the symbolic-regression clean-subset evaluation is partially in-sample. Both are fixable, so I view this as a major-revision rather than a rejection.

major comments (3)
  1. [§3.1, Fig. 2] The central claim that the learned aggregators 'achieved lower Brier scores than all individual models and all classical aggregators' (§4.1, also the abstract) is not directly supported by the numerical results presented. Table 2 reports individual Brier scores only on the full 208-item dataset; the right panel of Fig. 2 shows clean-subset individual scores visually but gives no numerical values. The reader therefore cannot verify, e.g., that the MLP's BS=0.264 and logistic regression's BS=0.241 are actually below the best clean-subset individual model. Please add a table reporting Brier, accuracy, and AUC for all 15 models, the four classical aggregators, and the two learned aggregators on the 94-item clean subset, together with paired tests (e.g., bootstrap or Wilcoxon) comparing each learned aggregator against the best individual model. This is necessary to support the paper's headlin
  2. [§2.5, §3.3] The symbolic-regression clean-subset score is partially in-sample. Section 2.5 states that the MLP distilled by SR was retrained on all complete rows of the full 208-item dataset (n=196) with ground-truth labels, and SR searched for formulas minimizing MSE against this MLP on those same 196 items. The 94-item clean subset includes 48 items drawn from the original 208-item dataset (Section 2.1), so for those 48 items the MLP's predictions—and hence the SR formula fitted to them—are indirectly fitted to the ground-truth outcomes. Scoring the recovered formula on the 94-item clean subset is therefore not an out-of-sample test for 48 of the 94 items. The paper's defense in Section 2.5 that 'ground truth enters at the evaluation stage instead' does not address this overlap, because ground-truth labels were used to train the MLP on exactly those items. Please re-run the SR evaluation on the 46
  3. [§3.1, §4.1] The core finding that 'a linear model performs comparably to the nonlinear model' is not statistically assessed. The paper reports that logistic regression has a lower Brier score (0.241 vs. 0.264) while the MLP has better AUC (0.657 vs. 0.633) and accuracy (0.606 vs. 0.574), but no test quantifies whether these differences are consistent or whether the two learned aggregators are statistically distinguishable. Additionally, the logistic regression's Brier advantage over the arithmetic mean is not item-consistent: the sign test is far from significant (46/94, p=.837) and the Wilcoxon test is not significant (p=.122); only the permutation test (p=.002) supports the mean reduction. The paper does discuss this pattern, but the linear-vs-nonlinear conclusion in Section 4.1 is stated more firmly than the inferential evidence warrants. Please provide confidence intervals or paired tests for LR
minor comments (4)
  1. [§3.1 vs. §3.3] The arithmetic-mean Brier score on the clean subset is reported as 0.313 in Section 3.1, but Section 3.3 gives the symbolic-regression formula's clean-subset score as 0.243 without an explicit comparison to the arithmetic mean on the same 94 items. The text also states that the formula achieves BS=0.231 on 203 full-dataset items against arithmetic-mean 0.265. Please add an explicit clean-subset comparison (0.243 vs. 0.313, if that is the correct pairing) so the reader does not have to infer it from different subsets.
  2. [Fig. 1 caption] The caption says no bootstrap brackets are drawn for the learned aggregators because 'their temporal transfer evaluation design is not comparable to the all-data bootstrap used for other methods.' This is a reasonable caveat, but the main text and figure would benefit from stating that the point comparisons in the figure are not uncertainty-adjusted; otherwise the visual impression of a large gap is stronger than the statistical evidence.
  3. [§2.5] The sentence 'Ground truth enters at the evaluation stage instead' is at best incomplete, because the MLP is trained with binary cross-entropy against ground-truth outcomes on the same items used for SR distillation. Please rephrase to acknowledge the overlap and to make clear which reported scores are out-of-sample and which are not.
  4. [§3.3, Table 5] The relationship between the 'model selection frequencies' in Table 5 and the recovery of the specific formula σ(p_GeminiPro − p_ClaudeHaiku) is not fully explained. Does the selection frequency refer to variables appearing in Pareto-frontier formulas, or to the specific lowest-complexity formula? Please clarify the connection, since the stability of the two-variable formula across runs is what supports the mechanistic reading.

Circularity Check

1 steps flagged

Symbolic-regression clean-subset Brier is partly in-sample; core aggregation result remains out-of-sample.

specific steps
  1. fitted input called prediction [Section 2.5 (Symbolic regression) and Section 3.3 (Symbolic regression results)]
    "we retrained the MLP on all complete rows of the full 208-item dataset (n = 196; here, no imputation is applied, so the 12 rows containing a missing cell are dropped) rather than on the 160 non-clean items used for the temporal transfer evaluation. ... Ground truth enters at the evaluation stage instead: any formula recovered by SR is subsequently scored against the actual resolutions, on both the full dataset and the clean subset (Sect. 3.3), so no performance claim in this paper rests on a formula's agreement with the MLP."

    The clean subset is not disjoint from the SR training set: Section 2.1 says 'Of the 208 items in the main dataset, 48 resolved after this date' and the 94-item clean subset is formed by supplementing those 48 with 46 new items. Section 2.5 retrains the MLP on all complete rows of the same 208-item dataset, so those 48 items are in the MLP's training set. The SR formula is fit to the MLP's predictions on exactly those training rows; the MLP's predictions on training rows are fit to the ground-truth resolutions via binary cross-entropy. Therefore scoring the recovered formula on the clean subset (BS = 0.243) is an indirect in-sample evaluation for 48 of the 94 items, not an out-of-sample prediction for them. The paper's statement that 'ground truth enters at the evaluation stage instead' is

full rationale

The learned-aggregation core result is genuinely out-of-sample: the MLP and logistic regression were trained on the 160 non-clean items and evaluated on the 94 clean items, so their Brier/accuracy/AUC advantage over individuals and classical aggregators is not circular. The linear-vs-nonlinear comparison is likewise based on the same temporal-transfer evaluation. The symbolic-regression analysis, however, contains a partial in-sample evaluation: the SR-target MLP was retrained on the full 208-item dataset, which includes 48 of the 94 clean items, and the recovered formula is then scored on those same clean items. For those 48 items, the formula's clean-subset Brier is inherited from the MLP's training fit to ground truth. This weakens the '0.243 on clean subset' quantification and the diversity-mechanism support drawn from it. The mechanism claim retains independent support from the logistic-regression weight analysis, which correlates weights trained on 160 non-clean items with error-decorrelation measured on the 94 clean items, and from the SR selection-frequency pattern (though that pattern is itself based on the full contaminated dataset). Self-citations to Douven et al. [15,16] are methodological or interpretational and not load-bearing; no uniqueness theorem or ansatz is smuggled in via citation. Net: one partially constructed 'prediction' in a supporting analysis; the central aggregation result is independent.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The central claims rest on several domain assumptions and hand-chosen hyperparameters, but no new conceptual or physical entities are postulated. The learned aggregators' weights are fitted to the training data, which is the intended method rather than a hidden free parameter. The main extra-statistical assumptions are the reliability of training cutoffs and the validity of the diversity metric.

free parameters (5)
  • Logistic regression L2 penalty λ = 0.01
    Chosen ad hoc in Section 2.4; no sensitivity analysis is reported. Affects the learned weights and thus the central learned-vs-classical comparison.
  • MLP hyperparameters (hidden units 64/32, dropout 0.2, learning rate 5e-3, epochs 25, batch size 16) = 64/32, 0.2, 5e-3, 25, 16
    Selected after preliminary experiments that showed no benefit from larger networks (Section 2.4). These are hand-chosen and not tuned on the clean evaluation set, but still introduce degrees of freedom.
  • Minimum trader threshold for market selection = 75
    Arbitrary threshold (Section 2.1) to ensure community attention; may bias question difficulty or information availability.
  • Clean subset cutoff date = 2025-09-01
    Operationalized as the first month-day after the latest model training cutoff (August 2025); not fitted, but the entire contamination analysis depends on this boundary.
  • Symbolic regression complexity penalty / Pareto-frontier selection = unspecified
    The 'lowest-complexity useful formula' is chosen from the Pareto frontier across 50 runs (Section 2.5). The exact complexity metric and selection criterion are not specified, introducing a modeling choice.
axioms (4)
  • domain assumption Training cutoff dates from official documentation are accurate at month granularity.
    The contamination analysis and clean-subset construction depend entirely on these cutoffs (Table 1, Section 2.7). If a model's true training window extends beyond the reported cutoff, the clean subset is not actually clean.
  • domain assumption Manifold Markets play-money probabilities are well-calibrated and comparable to real-money markets.
    Invoked in Section 2.1 (footnote 3) via Wolfers & Zitzewitz to justify using Manifold as a human benchmark. If false, the human-crowd comparisons and market baselines are biased.
  • domain assumption Mean pairwise correlation of squared errors is a valid operationalization of error-pattern diversity.
    Used in Section 3.4 to show that logistic-regression weights track diversity (r_s=+0.482). If this metric does not capture the diversity relevant to aggregation, the mechanism claim loses a key piece of support.
  • domain assumption The temporal-transfer design (training on 160 contaminated items, evaluating on 94 clean items) is a fair test of learned aggregation.
    The paper argues that contamination-induced distribution shift is a handicap, but the validity of the comparison assumes that the error structure learned on contaminated items transfers to clean items. This is flagged as a limitation in Section 4.5, making it a load-bearing premise.

reviewed 2026-08-02 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Wisdom of LLM Crowds: Aggregation and Contamination in Language Model Ensembles." pith.science (2026). https://pith.science/paper/QXY54JEB

@misc{pith2026260718269,
  author       = {Pith},
  title        = {Pith review of: Wisdom of LLM Crowds: Aggregation and Contamination in Language Model Ensembles},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QXY54JEB}},
  note         = {Machine review of arXiv:2607.18269}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

The wisdom of crowds -- the finding that aggregating judgments across individuals often outperforms the best individual -- has been extensively studied with human forecasters. Whether the same phenomenon emerges when the ``crowd'' consists of large language models (LLMs) is an open question with both theoretical and practical implications. We elicited probability estimates from 15 LLMs on 254 binary prediction market questions and evaluated classical and learned aggregation methods. Learned aggregators -- a multilayer perceptron and a logistic regression -- outperformed all individual models and classical methods. The logistic regression was found to match the neural network, suggesting that the benefit of learned aggregation derives from learning a linear combination of diverse model outputs rather than from nonlinear interactions. Symbolic regression applied to the neural network's learned mapping recovered a pure model-disagreement signal as the lowest-complexity useful formula on the Pareto frontier, further supporting this interpretation. Training cutoff contamination proved a pervasive confound: the apparent capability gap between frontier cloud models and smaller local models collapsed from 35.8% to 8.9% on a clean subset of questions resolving after all models' training cutoffs, and individual model rankings showed only moderate stability. Even when the prediction market is evaluated at each model's training cutoff, LLMs remained substantially less accurate, indicating a genuine gap in collective information aggregation. These findings suggest that LLM crowds can exhibit wisdom-of-crowds effects, but that contamination-free evaluation is essential for reliable assessment.

Figures

Figures reproduced from arXiv: 2607.18269 by Igor Douven.

Figure 1
Figure 1. Figure 1: Brier scores on the 94-item clean subset for the 15 models and the aggregators (sorted by Brier score). Both learned aggregator [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Individual model Brier scores on the initial 208-item dataset (left) and the 94-item clean subset (right), sorted by Brier score [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

40 extracted references · 5 canonical work pages

  1. [1]

    Argyle, Ethan C

    Lisa P. Argyle, Ethan C. Busby, Nancy Fulda, Joshua R. Gubler, Christopher Rytting, and David Wingate. 2023. Out of one, many: Using language models to simulate human samples.Political Analysis31 (2023), 337–351

    2023

  2. [2]

    Joshua Becker, Ethan Porter, and Damon Centola. 2019. The wisdom of partisan crowds.Proceedings of the National Academy of Sciences116 (2019), 10717–10722

    2019

  3. [3]

    Jeff Bezanson, Alan Edelman, Stefan Karpinski, and Viral B. Shah. 2017. Julia: A fresh approach to numerical computing.SIAM Rev.59 (2017), 65–98

    2017

  4. [4]

    Marcel Binz and Eric Schulz. 2023. Using cognitive psychology to understand GPT-3.Proceedings of the National Academy of Sciences120 (2023), e2218523120. doi:10.1073/pnas.2218523120

    doi:10.1073/pnas.2218523120 2023

  5. [5]

    Glenn W. Brier. 1950. Verification of forecasts expressed in terms of probability.Monthly Weather Review78 (1950), 1–3

    1950

  6. [6]

    Peter S. Bullen. 2003.Handbook of Means and Their Inequalities(2nd ed.). Kluwer, Amsterdam

    2003

  7. [7]

    Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. 2021. Extracting training data from large language models. In30th USENIX Security Symposium (USENIX Security 21). 2633–2650

    2021

  8. [8]

    Miles Cranmer. 2023. Interpretable machine learning for science with PySR and SymbolicRegression.jl. arXiv:2305.01582 [astro-ph.IM] https: //arxiv.org/abs/2305.01582

    Pith/arXiv arXiv 2023

  9. [9]

    Miles Cranmer, Alvaro Sanchez-Gonzalez, Peter Battaglia, Rui Xu, Kyle Cranmer, David Spergel, and Shirley Ho. 2020. Discovering symbolic models from deep learning with inductive biases. InAdvances in Neural Information Processing Systems, Vol. 33. 17429–17442

    2020

  10. [10]

    Davis-Stober, David V

    Clintin P. Davis-Stober, David V. Budescu, Jason Dana, and Stephen B. Broomell. 2014. When is a crowd wise?Decision1 (2014), 79–101

    2014

  11. [11]

    Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. 2024. Investigating data contamination in modern benchmarks for large language models. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). Association for Computation...

    doi:10.18653/v1/ 2024

  12. [12]

    Igor Douven. 2019. Optimizing group learning: An evolutionary computing approach.Artificial Intelligence275 (2019), 235–251

    2019

  13. [13]

    2022.The Art of Abduction

    Igor Douven. 2022.The Art of Abduction. MIT Press, Cambridge, MA

    2022

  14. [14]

    Igor Douven and Nikolaus Kriegeskorte. 2026. Predicting Lockean from gradational accuracy.International Journal of Approximate Reasoning192 (2026), 109636. doi:10.1016/j.ijar.2026.109636

    arXiv 2026

  15. [15]

    Igor Douven, Nikolaus Kriegeskorte, and Patrick Stinson. 2026. Three and a half stages of crowd wisdom.Collective Intelligence5 (2026). https://doi.org/10.1177/26339137261435121

    work page doi:10.1177/26339137261435121 2026

  16. [16]

    Igor Douven, Nikolaus Kriegeskorte, Patrick Stinson, and Josh Ying. 2026. Condorcet and beyond: An empirical comparison of voting rules.Cognitive Science(2026). In press

    2026

  17. [17]

    Francis Galton. 1907. Vox populi.Nature75 (1907), 450–451

    1907

  18. [18]

    Zhenyu Gao, Wenxi Jiang, and Yutong Yan. 2025. A test of lookahead bias in LLM forecasts. arXiv:2512.23847 [q-fin.GN] https://arxiv.org/ abs/2512.23847

    arXiv 2025

  19. [19]

    Christian Genest and James V. Zidek. 1986. Combining probability distributions: A critique and an annotated bibliography.Statist. Sci.1 (1986), 114–135

    1986

  20. [20]

    Tilmann Gneiting, Fadoua Balabdaoui, and Adrian E. Raftery. 2007. Probabilistic forecasts, calibration and sharpness.Journal of the Royal Statistical Society B69 (2007), 243–268

    2007

  21. [21]

    Thilo Hagendorff, Sarah Fabi, and Michal Kosinski. 2023. Human-like intuitive behavior and reasoning biases emerged in large language models but disappeared in ChatGPT.Nature Computational Science3 (2023), 833–838. doi:10.1038/s43588-023-00527-x

    doi:10.1038/s43588-023-00527-x 2023

  22. [22]

    Danny Halawi, Fred Zhang, Chen Yueh-Han, and Jacob Steinhardt. 2024. Approaching human-level forecasting with language models. InAdvances in Neural Information Processing Systems, Vol. 37.https://arxiv.org/abs/2402.18563

    Pith/arXiv arXiv 2024

  23. [23]

    Ryuji Hashimoto, Takehiro Takayanagi, Masahiro Suzuki, and Kiyoshi Izumi. 2026. LLM agents reveal how human bias shapes path-dependent market dynamics.Journal of Computational Social Science9 (2026), 32. doi:10.1007/s42001-026-00465-4

    work page doi:10.1007/s42001-026-00465-4 2026

  24. [24]

    Lu Hong and Scott E. Page. 2004. Groups of diverse problem solvers can outperform groups of high-ability problem solvers.Proceedings of the National Academy of Sciences101 (2004), 16385–16389

    2004

  25. [25]

    Lu Hong and Scott E. Page. 2025. The range of collective accuracy for binary classifications under majority rule.Economic Theory79 (2025), 275–300. doi:10.1007/s00199-024-01570-z

    work page doi:10.1007/s00199-024-01570-z 2025

  26. [26]

    John R. Koza. 1992.Genetic Programming: On the Programming of Computers by Means of Natural Selection. MIT Press, Cambridge, MA

    1992

  27. [27]

    2020.Open Democracy: Reinventing Popular Rule for the Twenty-first Century

    Hélène Landemore. 2020.Open Democracy: Reinventing Popular Rule for the Twenty-first Century. Princeton University Press, Princeton, NJ

    2020

  28. [28]

    Larrick and Jack B

    Richard P. Larrick and Jack B. Soll. 2006. Intuitions about combining opinions: Misappreciation of the averaging principle.Management Science52 (2006), 111–127

    2006

  29. [29]

    Peter A. Morris. 1983. An axiomatic approach to expert resolution.Management Science29 (1983), 24–32

    1983

  30. [30]

    Scott E. Page. 2007.The Difference: How the Power of Diversity Creates Better Groups, Firms, Schools, and Societies. Princeton University Press, Princeton, NJ

    2007

  31. [31]

    Sebastian Seung, and John McCoy

    Dražen Prelec, H. Sebastian Seung, and John McCoy. 2017. A solution to the single-question crowd wisdom problem.Nature541 (2017), 532–535. Manuscript submitted to ACM 18 Igor Douven

    2017

  32. [32]

    Reia and José F

    Sandro M. Reia and José F. Fontanari. 2021. Wisdom of crowds: Much ado about nothing.Journal of Statistical Mechanics: Theory and Experiment 2021, 5 (2021), 053402. doi:10.1088/1742-5468/abfa1f

    work page doi:10.1088/1742-5468/abfa1f 2021

  33. [33]

    Manley Roberts, Himanshu Thakur, Christine Herlihy, Colin White, and Samuel Dooley. 2024. To the cutoff . . . and beyond? A longitudinal perspective on LLM data contamination. InThe Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=m2NVG4Htxs

    2024

  34. [34]

    Michael Schmidt and Hod Lipson. 2009. Distilling free-form natural laws from experimental data.Science324 (2009), 81–85

    2009

  35. [35]

    Park, Rafael Valdece Sousa Bastos, and Philip E

    Philipp Schoenegger, Indre Tuminauskaite, Peter S. Park, Rafael Valdece Sousa Bastos, and Philip E. Tetlock. 2024. Wisdom of the silicon crowd: LLM ensemble prediction capabilities rival human crowd accuracy.Science Advances10 (2024), eadp1528. doi:10.1126/sciadv.adp1528

    doi:10.1126/sciadv.adp1528 2024

  36. [36]

    Siqueira Neto and José F

    Nilton S. Siqueira Neto and José F. Fontanari. 2023. On the efficacy of the wisdom of crowds to forecast economic indicators.The European Physical Journal B96, 1 (2023), 6. doi:10.1140/epjb/s10051-023-00482-6

    work page doi:10.1140/epjb/s10051-023-00482-6 2023

  37. [37]

    2004.The Wisdom of Crowds: Why the Many are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies, and Nations

    James Surowiecki. 2004.The Wisdom of Crowds: Why the Many are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies, and Nations. Random House, New York, NY

    2004

  38. [38]

    Justin Wolfers and Eric Zitzewitz. 2004. Prediction markets.Journal of Economic Perspectives18 (2004), 107–126

    2004

  39. [39]

    Cheng Xu, Shuhao Guan, Derek Greene, and Mohand-Tahar Kechadi. 2024. Benchmark data contamination of large language models: A survey. arXiv:2406.04244 [cs.CL]https://arxiv.org/abs/2406.04244

    Pith/arXiv arXiv 2024

  40. [40]

    Andy Zou, Tristan Xiao, Ryan Jia, Joe Kwon, Mantas Mazeika, Richard Li, Dawn Song, Jacob Steinhardt, Owain Evans, and Dan Hendrycks. 2022. Forecasting future world events with neural networks. InAdvances in Neural Information Processing Systems, Vol. 35. NeurIPS 2022 Datasets and Benchmarks Track. Manuscript submitted to ACM

    2022

This paper was first reviewed by deepseek-v4-flash on August 2, 2026.