Pith. sign in

REVIEW 2 major objections 5 minor 46 references

LLM-SoccerArena: Benchmarking LLMs on Real-World Predictions in Sports

T0 review · 2 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A live benchmark tests LLMs on World Cup matches before the outcomes are known.

desk verdict A genuinely prospective, open benchmark with careful stats; the headline web-access gain is plausible but the prompt-wording confound means the causal claim needs softening. read the letter →

arxiv 2607.24573 v1 pith:QETQ66BD submitted 2026-07-27 cs.AI

Jonas Schröder , Jonas Schweisthal , Oliver Müller , Markus Weinmann , Stefan Feuerriegel This is my paper · ORCID
classification cs.AI MSC 68T5062F1562P30
keywords largelanguagemodelsforecastingprospectivebenchmarklivesoccersportsanalyticsBrierscorewebsearch
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LLM-SoccerArena is a prospective, open-source benchmark that records forecasts of football matches before the outcomes are known, so the results cannot be contaminated by memorized post-hoc knowledge. In its first large deployment, seven LLMs forecast all 104 matches of the 2026 FIFA World Cup and 15 tournament questions under a factorial design varying model, web access, prompt order, and forecast horizon. The central empirical claim is that giving models web access yields the clearest controlled improvement in forecast quality, lowering the mean Brier score from 0.535 to 0.512 (paired difference 0.0228, Holm-adjusted p=0.045), while no single model wins after correction and prompting order has no measurable effect. The paper argues that sports, with their scheduled, objectively resolved, continuously recurring events, are a suitable setting for testing whether LLMs can synthesize changing information into calibrated probabilistic judgments before an outcome is resolved.

What carries the argument

The load-bearing mechanism is the prospective live benchmark protocol: forecasts are registered, timestamped, schema-validated, and archived while the event outcome is still unresolved, with an outcome barrier that prevents known results from entering the recorded forecasts. The factorial design Z=M×A×P×H (model version, information access, prompting strategy, forecast horizon) keeps the forecasting task identical across all configurations so that each factor can be compared on the same matches; the statistical machinery is a paired permutation test with 10,000 sign-flips and Holm correction within prespecified comparison families. The named evaluation instrument is the multiclass Brier score on the home/draw/away probability vector after 90 minutes, plus log loss, modal accuracy, exact-score accuracy, calibration curves, and a Scoring System for scorelines.

What would settle it

A reader could look at the public archive and check whether the open-book advantage of 0.0228 in Brier score is replicated on the next tournament, and more directly, could run a controlled experiment that varies the prompt-header wording independently of tool availability; if open-book headers without any actual search still produce the improvement, the information-access attribution fails.

Watch

Extended reading notes

Core claim

The paper's central claim, stated on its own terms, is that LLM-SoccerArena provides the first prospective, live, factorial benchmark for LLM forecasting of unresolved real-world sports events, and that its first large-scale deployment yields new evidence about how information access shapes LLM forecasts. Concretely: open-book web access improves forecast quality by a small but statistically significant margin (mean Brier 0.512 vs 0.535 at T–24h; within-match closed-minus-open difference 0.0228, 95% CI [0.0044, 0.0403], Holm-adjusted p=0.045), while no model version emerges as a clear winner (Brier 0.506 to 0.546, no paired comparison significant after Holm correction) and prompt order does not change accuracy (difference −0.0008, adjusted p=0.693). The paper also reports that the seven models' forecasts are highly correlated (mean pairwise correlation 0.943, Jensen–Shannon divergence 0.0044), that equal-weight ensembling adds little, that open-book rationales mention concrete current evidence far more often (recent form +68.0 pp, markets/odds +60.2 pp, injuries/lineups +53.0 pp) and generic unsupported claims less often (−18.1 pp), and that the best open-book LLM's T–2h forecasts tie the de-vigged closing bookmaker consensus (Gemini Brier 0.497 vs market 0.498).

Load-bearing premise

The headline attribution assumes that the open-book and closed-book conditions differ only in whether web access is available, but the two prompt headers also differ in wording, so the measured Brier improvement is not purely an information-access effect if those wording differences shift calibration independently.

Editorial extensions

If this is right

  • If the open-book result is correct, practitioners should give LLMs web access when decisions depend on current information, rather than relying on a larger or newer model alone.
  • The finding that forecasting closer to kickoff adds little suggests that late-breaking information beyond T–24h is not being effectively synthesized by these models, pointing to where retrieval design matters.
  • If the high pairwise forecast correlation persists on future tournaments, ensemble aggregation of diverse LLMs is unlikely to yield large gains without mechanisms that incentivize genuinely distinct forecasts.
  • The tie with closing bookmaker odds at T–2h, if it replicates, means a frontier LLM can match an aggregated market signal on a well-covered event without dedicated sports-analytics features.
  • The platform itself extends to league competitions, so the same protocol can produce a longitudinal record of whether successive model generations improve forecast quality, calibration, and search behavior over time.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implicit extension is that soccer's standardized, continuously recurring event stream lets the benchmark separate memorization from forecasting ability better than retrospective question sets, because no model can have seen the outcome.
  • A testable extension would be to randomize the header wording of the closed-book and open-book prompts independently of tool availability, to check whether the measured 0.0228 Brier improvement is purely an information-access effect or partly a wording effect.
  • The evidence-category results suggest a concrete mechanism for the open-book gain: the delivered rationales shift from generic claims to current, citable facts; verifying whether that shift predicts per-match Brier improvement would connect the qualitative and quantitative findings.
  • A neighbouring application is to use the same protocol benchmark for other low-scoring, widely-covered sports or for structured events with frequent unresolved outcomes, where the factorial design can be reused without modification.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper introduces LLM-SoccerArena, an open-source, prospective live benchmark for evaluating LLM forecasts of unresolved real-world sports events, demonstrated on all 104 matches and 15 tournament questions of the 2026 FIFA World Cup. The platform records timestamped, schema-validated forecasts together with prompts, model versions, tool traces, and costs, and implements a factorial design over model version, information access, prompting strategy, and forecast horizon. The case study evaluates seven LLMs with 8,736 match forecasts and 420 tournament forecasts, reporting that open-book access improves mean Brier score by 0.0228 (from 0.535 to 0.512) at T-24h, that no model clearly wins, that prompting strategy has no average effect, and that LLM forecasts are competitive with de-vigged closing bookmaker odds. The paper also provides extensive reproducibility artifacts, including frozen snapshots, hashes, validation logs, and code.

Significance. If the empirical claims are supported, this is a valuable contribution: it provides a standardized, auditable, and continuously operating benchmark for a class of real-world forecasting tasks, with a transparent prospective protocol and a strong set of reproducibility practices. The paper's strengths include machine-checked artifact provenance, frozen analysis snapshots, matched paired comparisons with bootstrap and permutation inference, Holm correction within declared families, leave-one-match-out robustness checks, and an external de-vigged odds baseline. The comparison of LLMs to a market baseline is a useful reference point. However, the headline causal claim about web access is not yet cleanly identified, because the open-book and closed-book conditions differ in prompt wording as well as tool availability.

major comments (2)
  1. [§4.3, Appendix F.3] The headline open-book improvement (mean Brier 0.535 to 0.512; paired difference 0.0228, 95% CI [0.0044, 0.0403], Holm-adjusted p=0.045) is attributed to 'web access,' but the two conditions differ in prompt wording as well as tool availability. The closed-book prompt instructs the model to use only the match information plus internal knowledge, while the open-book prompt instructs it to use the web-search tool, to 'base the final forecast on public information,' and it labels the condition OPEN_BOOK. These wording differences can affect calibration independently of retrieved information, so the measured effect is not a clean estimate of information access. Because the effect is small and only marginally significant, a tool-only vs. wording-only ablation (e.g., open-book wording with the search tool disabled) is needed before the causal claim in the abstract and Section 4.3 can be sustained; alternatively, the claim should be softened to an association.
  2. [§4.3, Appendix C.3] The intent-to-treat analysis includes open-book calls in which no search was observed (search was observed in only 84.5% of open-book forecasts), and the secondary observed-search sensitivity analysis compares self-selected groups with model-specific search propensities. That analysis is correlational and cannot separate the effect of retrieved information from the effect of the open-book wording; for example, if the wording alone improves calibration, the observed-search subset would show an effect even when no information is retrieved. The Limitations section correctly states that generated rationales do not establish causal influence, but this caution is not applied to the headline information-access claim, which is stated causally in the abstract and in Section 4.3.
minor comments (5)
  1. [Table 1] The row label 'Ex.source' appears truncated; it should read 'External sources' or a more descriptive phrase.
  2. [Section 2] In the first paragraph, 'i.e., )LLM evaluation' contains a stray parenthesis, and the sentence introducing the three research streams is awkwardly punctuated.
  3. [Appendix A] The sentence 'Continuous operation enables evaluation at scale and create a longitudinal record' should be 'creates a longitudinal record.'
  4. [Appendix F.3] The open-book prompt instructs the model not to use the project's stored predictions; this is an instruction that cannot be independently verified from the recorded tool traces, so it should be listed as a limitation of the audit trail.
  5. [Table 3] The model registry honestly reports 'Public weights: Unverified' for all models; the paper should briefly explain the implications for third-party reproduction, since exact deployed endpoints cannot be independently reconstructed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark's forecasts and scores are computed prospectively without fitted parameters, and the open/closed prompt-wording confound is an internal-validity concern, not a circularity.

full rationale

I walked the paper's derivation chain and found no step in which a claimed prediction or first-principles result is equivalent to its inputs by construction. The central empirical claim is that open-book access improves Brier score at T-24h (0.535 closed vs. 0.512 open, paired difference 0.0228, Holm-adjusted p=0.045). This quantity is computed directly from timestamped, schema-validated forecasts recorded before outcomes were known and from official 90-minute outcomes resolved afterward, as described in Section 3.1.3 and Section 4.1. No parameter is fitted to the outcome data, no model is trained on the held-out matches, and the Brier and log-loss measures are standard proper scoring rules applied to stored forecasts. The appendix-level comparison against closing bookmaker odds uses an independently sourced external baseline with a de-vigging procedure (Appendix E), not the LLMs' own outputs. The self-citations [9,10] concern a reporting checklist and NLP annotation methodology; they are not load-bearing for the forecasting result. The skeptical concern that the open-book and closed-book prompts differ in wording beyond tool availability (Appendix F.3) is a threat to causal attribution of the information-access effect, and the paper itself acknowledges in Limitations that the evidence analyses do not establish which information causally influenced a forecast. That is an internal-validity or confound issue, not circularity: the reported Brier difference is still a genuine, externally checkable comparison of two recorded forecast conditions. None of the enumerated circularity patterns applies.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters are fitted to outcomes and no new entities are introduced. The benchmark conclusions rest on outcome integrity, repair fidelity, proper scoring rules, and the clean interpretation of the web-access manipulation. The prompt-wording caveat is the weakest of these.

assumptions (4)
  • standard math Brier score and log loss are strictly proper scoring rules, so lower values correspond to better probability forecasts.
    Used to rank forecasts in Section 4.1; standard result from Gneiting and Raftery [12].
  • domain assumption Official 90-minute plus stoppage time outcomes are the ground truth and are resolved only after all forecasts for that event are stored.
    Section 3.1.3 step 4; the 'outcome barrier' is what makes the benchmark prospective.
  • domain assumption The validation and at-most-one repair step does not change the substantive forecast.
    Section 3.1.3 step 3 and Appendix F.5; relevant because 206 of 8,736 main-panel responses needed repair.
  • domain assumption The open-book and closed-book conditions differ only in information access, not in prompt wording effects.
    Appendix F.3; the two conditions use different access headers, so the measured 0.023 Brier improvement is attributed to web access.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLM-SoccerArena: Benchmarking LLMs on Real-World Predictions in Sports." pith.science (2026). https://pith.science/paper/QETQ66BD

@misc{pith2026260724573,
  author       = {Pith},
  title        = {Pith review of: LLM-SoccerArena: Benchmarking LLMs on Real-World Predictions in Sports},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QETQ66BD}},
  note         = {Machine review of arXiv:2607.24573}
}
read the original abstract

Large language models (LLMs) increasingly support decisions about uncertain future events, yet evaluating their ability to forecast real-world outcomes remains difficult. In particular, existing benchmarks are typically static and retrospective, and therefore cannot test how information is synthesized by LLMs to predict future events under uncertainty. We introduce LLM-SoccerArena (https://llm-soccerarena.com), a prospective live benchmark that evaluates how well LLMs forecast real-world sports events before the outcomes are known. LLM-SoccerArena provides (1) a prospective live benchmark protocol, (2) a public open-source platform, and (3) a factorial benchmark design together with tournament-related questions (e.g., which team will win). LLM-SoccerArena automatically records timestamped, schema-validated forecasts of unresolved events, together with prompts, model versions, tool traces, and costs. The factorial design varies along four dimensions: (1) model version (e.g., GPT-5.5, Claude Opus 4.8); (2) information access; (3) prompting strategy, and (4) forecast horizon. We demonstrate LLM-SoccerArena through a large-scale evaluation of the 2026 FIFA World Cup, in which seven LLMs generated forecasts for all 104 matches and 15 tournament-related questions. We provide a detailed analysis of model performance across information access, prompting strategy, and forecast horizon. As a result, LLM-SoccerArena provides new evidence about the forecasting performance of state-of-the-art LLMs. For example, LLMs with web access outperform those without, but only by a small margin (i.e., a 0.023 improvement in Brier score). Overall, LLM-SoccerArena provides a flexible, open-source platform for prospective benchmarking of unresolved events. LLM-SoccerArena will be continuously updated, and can be directly applied to future national and international tournaments and league competitions.

Figures

Figures reproduced from arXiv: 2607.24573 by the authors.

Figure 1
Figure 1. LLM-SoccerArena architecture and public interface. 3 LLM-SoccerArena LLM-SoccerArena consists of three components (see [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. (a) Model-specific and (b) match-level effects of [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Public interfaces to the auditable benchmark archive. Figure 1 summarizes the platform architecture; this figure [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: Performance across the tournament. Cumulative trajectories use the balanced T–24h aggregation across both [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]
Figure 5
Figure 5. Figure 5: Forecast similarity and same-condition ensemble performance. The matrices report pairwise Jensen–Shannon [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: Information-access results. Panels show the pooled T–24h access effect, horizon comparisons, forecast changes across [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: Model-specific distributions of T–24h closed-minus-open Brier-score differences. The pooled access contrast is [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Prompting-strategy effects. The left panel reports paired Brier contrasts. The right panel reports changes in score [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Agreement between the outcome implied by the most likely scoreline and the outcome with the highest reported [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Probability calibration and self reported confidence. Home, draw, and away outcomes use separate reliability panels [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: Sum of the 48 reported semifinal inclusion probabilities for each model version and benchmark condition. The [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]
Figure 12
Figure 12. Figure 12: Complete evidence-category results for the 1,456 T–24h probabilistic forecast rationales. Points mark closed-book [PITH_FULL_IMAGE:figures/full_fig_p023_12.png]
Figure 13
Figure 13. Figure 13: Model-specific median and interquartile range of rationale length by information-access condition. Claude Fable [PITH_FULL_IMAGE:figures/full_fig_p024_13.png]
Figure 14
Figure 14. Figure 14: Exploratory fixed-seed t-SNE projection of frozen, L2-normalized MPNet embeddings for the 1,456 rationale texts. [PITH_FULL_IMAGE:figures/full_fig_p025_14.png]
Figure 15
Figure 15. Figure 15: Absolute performance of closing odds, individual model versions, and the equal-weight LLM ensemble in the matched [PITH_FULL_IMAGE:figures/full_fig_p027_15.png]
Figure 16
Figure 16. Figure 16: Paired LLM advantage over closing odds for Brier score, log loss, ranked probability score, and modal H/D/A accuracy. [PITH_FULL_IMAGE:figures/full_fig_p028_16.png]
Figure 17
Figure 17. Figure 17: Cumulative Brier advantage relative to closing odds. Positive values indicate a lower cumulative mean Brier score [PITH_FULL_IMAGE:figures/full_fig_p029_17.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 25 canonical work pages

  1. [1]

    Shubham Atreja, Joshua Ashkinaze, Lingyao Li, Julia Mendelsohn, and Libby Hemphill. 2025. What’s in a Prompt?: A Large-Scale Experiment to Assess the Impact of Prompt Design on the Compliance and Accuracy of LLM-Generated Text Annotations.Proceedings of the International AAAI Conference on Web and Social Media19 (June 2025), 122–145. doi:10.1609/icwsm.v19i1.35807

    doi:10.1609/icwsm.v19i1.35807 2025

  2. [2]

    Mislav Balunovic, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, and Martin Vechev. 2025. MathArena: Evaluating LLMs on Uncon- taminated Math Competitions. InAdvances in Neural Information Processing Systems, Vol. 38. https://proceedings.neurips.cc/paper_ files/paper/2025/hash/1d27c01ebd3e3aebe226b44fc970d803-Abstract- Datasets_and_Benchmarks_Track.html

    work page 2025

  3. [3]

    Glenn W. Brier. 1950. Verification of Forecasts Expressed in Terms of Probability. Monthly Weather Review78, 1 (1950), 1–3

    work page 1950

  4. [4]

    Rory Bunker, Calvin Yeung, and Keisuke Fujii. 2024. Machine Learning for Soccer Match Result Prediction. https://arxiv.org/abs/2403.07669v1

    work page Pith review arXiv 2024

  5. [5]

    Simin Chen, Yiming Chen, Zexin Li, Yifan Jiang, Zhongwei Wan, Yixin He, Dezhi Ran, Tianle Gu, Haizhou Li, Tao Xie, and Baishakhi Ray. 2025. Benchmarking Large Language Models Under Data Contamination: A Survey from Static to Dynamic Evaluation. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Association for Comput...

    doi:10.18653/v1/2025.emnlp-main.511 2025

  6. [6]

    Gonzalez, and Ion Stoica

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. arXiv:2403.04132 [cs.AI] https://arxiv.org/abs/2403. 04132

    arXiv 2024

  7. [7]

    Day, Sandy Gordon, and Corinna Fink

    David V. Day, Sandy Gordon, and Corinna Fink. 2012. The Sporting Life: Explor- ing Organizations through the Lens of Sport.Academy of Management Annals6, 1 (2012), 397–433. doi:10.1080/19416520.2012.678697

    arXiv 2012

  8. [8]

    Dixon and Stuart G

    Mark J. Dixon and Stuart G. Coles. 1997. Modelling Association Football Scores and Inefficiencies in the Football Betting Market.Journal of the Royal Statistical Society: Series C46, 2 (1997), 265–280. doi:10.1111/1467-9876.00065

    arXiv 1997

Show all 46 references
  1. [9]

    Stefan Feuerriegel, Christopher Barrie, M. J. Crockett, Laura K. Globig, Killian L. McLoughlin, Dan-Mircea Mirea, Arthur Spirling, Diyi Yang, Tim Althoff, Maria Antoniak, Lisa P. Argyle, Ashwini Ashokkumar, Mohammad Atari, Hannah Bailey, Kevin Bauer, Umang Bhatt, Yidong Chai, ...

    2026

  2. [10]

    Robertson, Steve Rathje, Jochen Hartmann, Saif M

    Stefan Feuerriegel, Abdurahman Maarouf, Dominik Bär, Dominique Geissler, Jonas Schweisthal, Nicolas Pröllochs, Claire E. Robertson, Steve Rathje, Jochen Hartmann, Saif M. Mohammad, Oded Netzer, Alexandra A. Siegel, Barbara Plank, and Jay J. Van Bavel. 2025. Using Natural Langu...

    2025 doi

  3. [11]

    Dennis Frauen, Athiya Deviyani, Mihaela van der Schaar, and Stefan Feuerriegel

  4. [12]

    Tilmann Gneiting and Adrian E. Raftery. 2007. Strictly Proper Scoring Rules, Prediction, and Estimation.J. Amer. Statist. Assoc.102, 477 (2007), 359–378. doi:10.1198/016214506000001437

    2007 doi

  5. [13]

    Andreas Groll, Christophe Ley, Gunther Schauberger, and Hans Van Eetvelde

  6. [14]

    Andreas Groll, Gunther Schauberger, and Gerhard Tutz. 2015. Prediction of Major International Soccer Tournaments Based on Team-Specific Regularized Poisson Regression: An Application to the FIFA World Cup 2014.Journal of Quantitative Analysis in Sports11, 2 (2015), 97–115. doi...

    2015 doi

  7. [15]

    Danny Halawi, Fred Zhang, Yueh-Han Chen, and Jacob Steinhardt. 2024. Ap- proaching Human-Level Forecasting with Language Models. Advances in Neural Information Processing Systems 37. doi:10.52202/079017-1598

    2024 doi

  8. [16]

    Hogarth and Spyros Makridakis

    Robin M. Hogarth and Spyros Makridakis. 1981. Forecasting and Planning: An Evaluation.Management Science27, 2 (Feb. 1981), 115–138. doi:10.1287/mnsc.27. 2.115

    1981 doi

  9. [17]

    Sture Holm. 1979. A Simple Sequentially Rejective Multiple Test Procedure. Scandinavian Journal of Statistics6, 2 (1979), 65–70

    1979

  10. [18]

    Elvis Hsieh, Preston Fu, and Jonathan Chen. 2024. Reasoning and Tools for Human-Level Forecasting. InProceedings of the Workshop on the Future of Event Detection (FuturED). Association for Computational Linguistics, Miami, Florida, USA, 48–57. doi:10.18653/v1/2024.futured-1.5

    2024 doi

  11. [19]

    Hyndman and George Athanasopoulos

    Rob J. Hyndman and George Athanasopoulos. 2018.Forecasting: Principles and Practice. OTexts

    2018

  12. [20]

    Ezra Karger, Houtan Bastani, Yueh-Han Chen, Zachary Jacobs, Danny Halawi, Fred Zhang, and Philip E. Tetlock. 2025. ForecastBench: A Dynamic Bench- mark of AI Forecasting Capabilities. International Conference on Learning Representations. doi:10.48550/arXiv.2409.19839

    2025 doi arXiv

  13. [21]

    Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengx- uan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and A...

    2021

  14. [22]

    Frank Konietschke and Markus Pauly. 2014. Bootstrapping and Permuting Paired T-Test Type Statistics.Statistics and Computing24, 3 (May 2014), 283–296. doi:10.1007/s11222-012-9370-4

    2014 doi

  15. [23]

    Anders Krogh and Jesper Vedelsby. 1994. Neural Network Ensembles, Cross Validation, and Active Learning. InAdvances in Neural Information Process- ing Systems 7. MIT Press. https://proceedings.neurips.cc/paper/1994/hash/ b8c37e33defde51cf91e1e03e51657da-Abstract.html

    1994

  16. [24]

    Christoph Leitner, Achim Zeileis, and Kurt Hornik. 2010. Forecasting Sports Tournaments by Ratings of (Prob)abilities: A Comparison for the EURO 2008. International Journal of Forecasting26, 3 (2010), 471–481. doi:10.1016/j.ijforecast. 2009.10.001

    2010 doi

  17. [25]

    Christophe Ley, Tom Van de Wiele, and Hans Van Eetvelde. 2019. Ranking Soccer Teams on the Basis of Their Current Strength: A Comparison of Maximum Likelihood Approaches.Statistical Modelling19, 1 (2019), 55–73. doi:10.1177/ 1471082X18817650

    2019

  18. [26]

    Manning, Christopher Ré, Diana Acosta-Navas, Drew A

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michi- hiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Nav...

    2023 arXiv

  19. [27]

    Yachuan Liu, Xiaochun Wei, Lin Shi, Xinnuo Li, Bohan Zhang, Paramveer Dhillon, and Qiaozhu Mei. 2026. ExAnte: A Benchmark for Ex-Ante Inference in Large Language Models. InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguisti...

    2026 doi

  20. [28]

    Zhiyi Ma, Kawin Ethayarajh, Tristan Thrush, Somya Jain, Ledell Wu, Robin Jia, Christopher Potts, Adina Williams, and Douwe Kiela. 2021. Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking. Advances in Neural Information Processing Systems 3...

    2021

  21. [29]

    M. J. Maher. 1982. Modelling Association Football Scores.Statistica Neerlandica 36, 3 (1982), 109–118. doi:10.1111/j.1467-9574.1982.tb00782.x

    1982

  22. [30]

    Marcel Meyer, Sascha Kaltenpoth, Henrik Albers, Kevin Zalipski, and Oliver Müller. 2025. TS-Arena – A Live Forecast Pre-Registration Platform. arXiv preprint arXiv:2512.20761. doi:10.48550/arXiv.2512.20761

    2025 doi arXiv

  23. [31]

    Montgomery

    Douglas C. Montgomery. [n. d.]. Design and Analysis of Experiments, 10th Edi- tion | Wiley. https://www.wiley.com/en-us/design-and-analysis-of-experiments- 10th-edition-p-9781119492443

  24. [32]

    Barrow, Souhaib Ben Taieb, Christoph Bergmeir, Ricardo J

    Fotios Petropoulos, Daniele Apiletti, Vassilios Assimakopoulos, Mohamed Zied Babai, Devon K. Barrow, Souhaib Ben Taieb, Christoph Bergmeir, Ricardo J. Bessa, Jakub Bijak, John E. Boylan, Jethro Browell, Claudio Carnevale, Jennifer L. Castle, Pasquale Cirillo, Michael P. Clemen...

    2022

  25. [33]

    Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d’Alché-Buc, Emily Fox, and Hugo Larochelle

  26. [34]

    Daniel J. Power. 2002.Decision Support Systems: Concepts and Resources for Managers(1. publ ed.). Quorum Books, Westport, Conn

    2002

  27. [35]

    Sarah Pratt, Seth Blumberg, Pietro Kreitlon Carolino, and Meredith Ringel Mor- ris. 2024. Can Language Models Use Forecasting Strategies? arXiv preprint arXiv:2406.04446. doi:10.48550/arXiv.2406.04446

    2024 doi arXiv

  28. [36]

    Markel Rico-González, José Pino-Ortega, Amaia Méndez, Filipe Clemente, and Arnold Baca. 2022. Machine Learning Application in Soccer: A Systematic Review. Biology of Sport40, 1 (2022), 249–263. doi:10.5114/biolsport.2023.112970

    2022

  29. [37]

    Shim, Merrill Warkentin, James Courtney, Daniel Power, Ramesh Sharda, and Christer Carlsson

    J. Shim, Merrill Warkentin, James Courtney, Daniel Power, Ramesh Sharda, and Christer Carlsson. 2002. Past, Present, And Future Of Decision Support Technology.Decision Support Systems33, 2 (Jan. 2002), 111–126. doi:10.1016/ S0167-9236(01)00139-7

    2002

  30. [38]

    Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Sid- dhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, Shubh Agrawal, Sandeep Sandha, Siddartha Naidu, Chinmay Hegde, Yann Le- Cun, Tom Goldstein, Willie Neiswanger, and Micah Gold...

    2025

  31. [39]

    Wolfe, Karl E

    Richard A. Wolfe, Karl E. Weick, John M. Usher, James R. Terborg, Laura Poppo, A. J. Murrell, John M. Dukerich, Deborah C. Core, Kevin E. Dickson, and James S. Jourdan. 2005. Sport and Organizational Studies: Exploring Synergy.Journal of Management Inquiry14, 2 (2005), 182–210...

    2005 doi

  32. [40]

    Qi Yan, Raihan Seraj, Jiawei He, Lili Meng, and Tristan Sylvain. 2024. AutoCast++: Enhancing World Event Prediction with Zero-Shot Ranking- Based Context Retrieval. International Conference on Learning Rep- resentations. https://proceedings.iclr.cc/paper_files/paper/2024/hash/...

    2024

  33. [41]

    Qingchuan Yang, Simon Mahns, Sida Li, Anri Gu, Jibang Wu, and Haifeng Xu. 2026. LLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet Arena. International Conference on Learning Representations. https: //openreview.net/forum?id=VpiHkMSPqI

    2026

  34. [42]

    2018.Probabilistic Forecasts for the 2018 FIFA World Cup Based on the Bookmaker Consensus Model

    Achim Zeileis, Christoph Leitner, and Kurt Hornik. 2018.Probabilistic Forecasts for the 2018 FIFA World Cup Based on the Bookmaker Consensus Model. Working Papers in Economics and Statistics 2018-09. University of Innsbruck, Innsbruck, Austria. https://EconPapers.repec.org/ReP...

    2018

  35. [43]

    unverified

    Andy Zou, Tristan Xiao, Ryan Jia, Joe Kwon, Mantas Mazeika, Richard Li, Dawn Song, Jacob Steinhardt, Owain Evans, and Dan Hendrycks. 2022. Forecasting Future World Events with Neural Networks. Advances in Neural Informa- tion Processing Systems 35. https://proceedings.neurips....

    2022

  36. [2019]

    doi:10.1515/jqas-2018-0060

    A Hybrid Random Forest to Predict Soccer Matches in International Tournaments.Journal of Quantitative Analysis in Sports15, 4 (2019), 271–287. doi:10.1515/jqas-2018-0060

    2019 doi

  37. [2020]

    arXiv:2003.12206 [cs.LG] doi:10.48550/arXiv.2003.12206

    Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program). arXiv:2003.12206 [cs.LG] doi:10.48550/arXiv.2003.12206

    2019 doi arXiv

  38. [2026]

    arXiv:2601.21816

    Nonparametric LLM Evaluation from Preference Data.arXiv preprint arXiv:2601.21816(2026). arXiv:2601.21816

    2026 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.