Pith. sign in

REVIEW 3 major objections 5 minor 63 references

A leave-one-out wagering payout makes LLM aggregation weights emerge as each model's expected score advantage, with truthful prediction as a dominant strategy.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 19:30 UTC pith:MJKFQ5NW

load-bearing objection Clean leave-one-out wagering fix that actually delivers DSIC under mutable beliefs, continuous advantage-proportional weights, and decentralized LLM wager learning that matches centralized aggregators. the 3 major comments →

arxiv 2607.04389 v1 pith:MJKFQ5NW submitted 2026-07-05 cs.AI cs.GT

Decentralized Aggregation of LLM Predictions via Wagering Mechanisms

Yuhong Luo , David M. Pennock , Xintong Wang This is my paper
classification cs.AI cs.GT
keywords LLM aggregationwagering mechanismsincentive compatibilitydecentralized learningadvantage-aligned weightsproper scoring rulesforecastingleave-one-out baseline
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

When several large language models answer the same question, some may hold private context or domain strength that others lack. The hard problem in a decentralized setting is how much weight each model should get without seeing its private information, and without letting it game the weight by misreporting confidence. This paper designs a family of wagering mechanisms called WALLA. Each model reports a probability and a wager; the mechanism pays each model its own score minus a leave-one-out baseline built only from the others, then subtracts a quadratic tax on the wager itself. Under that payout, telling the truth about one's belief is always optimal no matter how beliefs are formed or what others do, and the best wager is proportional to the model's expected score advantage. Because the optimal wager is well-defined even for a fixed (possibly imperfect) prediction, each model can train a small network on its own hidden states and observed payouts alone. Experiments on QA and forecasting tasks with homogeneous and heterogeneous models show that the resulting wager-weighted aggregates match strong centralized routers and stacked ensembles while remaining fully decentralized and incentive-compatible.

Core claim

Inserting a leave-one-out baseline into a wagering net-payout function yields three properties at once: dominant-strategy incentive-compatible prediction under arbitrary beliefs, best-response wagers proportional to expected score advantage, and prediction-agnostic wager optimization that lets agents learn aggregation weights from their own hidden states and payouts without needing optimal predictions.

What carries the argument

The net payout π_i = w_i (s(p_i, y) − b_{−i} − c_3 w_i), where b_{−i} is any leave-one-out baseline independent of agent i. The quadratic regularization produces an interior optimum w*_i = (A_i / 2c_3)^+ with A_i the expected score advantage, while strict properness of s keeps truthful prediction dominant.

Load-bearing premise

Each agent is risk-neutral and maximises expected net payout question by question, so the simple quadratic tax on the wager is enough to make the best wager exactly proportional to advantage.

What would settle it

Train the same wager networks under risk-averse or multi-question budget constraints, or let agents revise beliefs after seeing others' reports in a repeated game; if the learned wagers cease to track realised score advantages or predictive performance falls below centralised baselines, the alignment claim fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Aggregation weights can be learned fully on-device from each model's own hidden states and payouts, without a central party that sees every model's internals.
  • Models that expect to underperform the pool automatically wager near zero and drop out of the weighted average, yielding soft per-question routing.
  • The same payout incentivises temperature calibration: better-calibrated models receive higher realised net payoffs and therefore higher equilibrium influence.
  • Worst-case mechanism deficit is bounded by a designer-chosen constant independent of the number of participants and total money wagered.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If proprietary models can already play simple games in context, the same leave-one-out payout might be elicited by natural-language instructions rather than a trained wager head, extending the method beyond open-weight models.
  • The design separates 'when am I better than the pool' from 'how do I generate better predictions,' suggesting a path for small specialised models to contribute only on their comparative-advantage slices of a large task mixture.
  • Because the mechanism remains DSIC under mutable beliefs, it could be composed with multi-agent debate or iterative refinement protocols without reopening prediction incentives.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes WALLA, a family of wagering mechanisms for decentralized aggregation of LLM probabilistic predictions. Each agent reports a prediction and a wager; aggregation uses wagers as weights. The key design is a leave-one-out baseline in the net payout π_i = w_i (s(p_i,y) − b_{-i} − c_3 w_i), which yields three properties: (1) dominant-strategy incentive compatibility of prediction under arbitrary (including mutable) beliefs, (2) best-response wager proportional to expected score advantage A_i, and (3) prediction-agnostic wager optimization, so wager policies can be learned without requiring optimal predictions. Two concrete baselines trade off normality (WALLA I) against no-arbitrage (WALLA II), with a worst-case deficit bound independent of the number of agents and total stake. Empirically, small wager heads trained on frozen LLM hidden states and realized payouts match centralized stacked generalization and routers on QA and forecasting benchmarks across homogeneous private-context, heterogeneous, and mixed settings, while providing advantage-aligned weights and IC guarantees.

Significance. Decentralized, incentive-compatible aggregation of heterogeneous LLM predictions is a timely and practically important problem. The contribution is clean: a single structural change to the classical weighted-score wagering payout simultaneously restores DSIC under mutable beliefs, produces interior advantage-aligned wagers rather than boundary bets, and decouples wager learning from prediction quality. Theorems 3.2–3.3 and Corollaries 3.5–3.6 follow directly from strict properness and the quadratic form of the payout; the normality/no-arbitrage trade-off and the O(1) deficit bound (Prop. 3.14) are proved carefully. Experiments cover three information regimes, multiple pool sizes, in- and out-of-distribution evaluation, calibration incentives, and both linear and log pooling, with code released. If the claims hold under the stated risk-neutrality assumptions, WALLA is a genuine advance over heuristic confidence weighting and centralized routers for multi-provider LLM ensembles.

major comments (3)
  1. Section 3.3 states that the wager network is trained by minimizing the negative empirical net payout L(θ_i) = −(1/T)∑[g_θ(h)(s−b)−c_3 g_θ(h)²], but Section 4.2 reports training by MSE to the hindsight target w* = ((s−b)/(2c_3))^{+}, with a footnote that MSE is less aggressive when w*=0. These objectives share the same optimum for fixed A_i > 0 but are not identical as training losses under non-stationary multi-agent play and estimation noise. Please reconcile the two descriptions (or report both) and state which objective was used for all tables, so that the prediction-agnostic learning claim is tied to a single, reproducible procedure.
  2. Proposition 3.14 bounds worst-case deficit by (s−s̄)²/(4c_3), independent of M and total stake, but the mechanism is not weakly budget-balanced: the platform can lose money. Table 6 reports small average net payouts that are often negative for some models. For the practical claim that WALLA is deployable as a decentralized aggregation service, a short analysis or empirical estimate of expected (not only worst-case) platform deficit under the learned wager policies—and how c_3 trades off deficit against participation/weight informativeness—would make the design choice load-bearing rather than only theoretical.
  3. The equilibrium and learning claims rest on risk-neutral, per-question expected-payout maximization (Section 3.1). That assumption is stated, but multi-question budget constraints, risk aversion, or reputation across rounds would generally destroy the closed-form w* = (A_i/(2c_3))^{+} and could re-couple prediction and wager incentives. A brief limitations paragraph quantifying or discussing when the alignment fails (e.g., fixed total budget across a batch of questions) would strengthen the paper without changing the theorems.
minor comments (5)
  1. Figure 1 and the appendix figures (2–5) are informative but dense; axis labels and the meaning of colored dots (context vs. best model) should be self-contained in each caption so panels can be read without the main text.
  2. Table 1’s “IC / Manipulation-Resistant” column is slightly ambiguous for pre-inference routers (marked “–”). A one-sentence clarification that routers are not eliciting reports from strategic models would help.
  3. Notation: Fi-measurability of strategies is clear for LLMs, but the switch between Qi (belief) and P (data-generating measure) in the learning section could be flagged once more when discussing calibration incentives (Section 4.6).
  4. Minor typos and polish: e.g., “Heterogenenous” in Appendix B.2 heading; ensure consistent use of WALLA I/II vs. “WALLA I (log-pool)” in tables.
  5. Related work on prediction markets is appropriately scoped to the static setting; a pointer to recent LLM-as-forecaster aggregation (e.g., Schoenegger et al.) is already present and sufficient.

Circularity Check

0 steps flagged

No significant circularity: the three claimed properties follow by elementary calculus from an explicitly designed leave-one-out payout, not from a fit or self-citation chain.

full rationale

The paper’s central derivation is mechanism design, not a hidden identity. Definition 3.1 introduces the net payout π_i = w_i (s(p_i,y) − b_{−i} − c_3 w_i) with an arbitrary leave-one-out baseline b_{−i}. Theorem 3.2 (DSIC) is immediate from the fact that b_{−i} is independent of p_i together with strict properness of s; Theorem 3.3 (advantage–wager alignment) is the unique maximizer of the strictly concave quadratic w A − c_3 w^{2}. Both proofs are self-contained and do not rely on external uniqueness theorems or fitted parameters. The two concrete baselines (Defs. 3.7–3.8) are design choices that trade normality for no-arbitrage; their properties are proved directly (Props. 3.10–3.14). Empirically, wager networks are trained on realized payouts and evaluated on held-out questions; the reported positive correlation with hindsight advantage is a consistency check, not a definitional identity. Citations to prior wagering work (including co-authored papers) supply background axioms and the WSWM baseline; they are not load-bearing for the new leave-one-out construction or its three claimed properties. No step reduces a claimed prediction or first-principles result to its own inputs by construction.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 1 invented entities

The central claims rest on standard proper-scoring-rule theory plus three design choices (leave-one-out baseline, quadratic wager tax, risk-neutral single-question optimization). The free parameters c1,c2,c3 are set by the designer and control score range and deficit bound; they are not fitted to the evaluation metrics. No new physical or mathematical entities are postulated beyond the mechanism itself.

free parameters (2)
  • c3 (wager regularization) = 1/2
    Designer-chosen positive constant that sets the scale of optimal wagers and the worst-case deficit bound (s−s)²/(4c3). Fixed to 1/2 in all experiments; not learned from data.
  • c1, c2 (Brier-score affine parameters) = 1, 1/2
    Affine transform of the quadratic score; chosen as 1 and 1/2 so that scores lie in a convenient range. Affect absolute payout magnitudes but not the ranking or the DSIC property.
axioms (4)
  • domain assumption Agents are risk-neutral and maximize expected net payout on each individual question.
    Stated in Section 3.1; required for the first-order condition that yields w* = (A_i/(2c3))⁺.
  • standard math The scoring rule s is strictly proper.
    Definition 2.1; used in Theorem 3.2 to obtain unique truthful prediction.
  • domain assumption Wagers and predictions are Fi-measurable (depend only on the question and the agent’s private signal).
    Section 3.1; automatic for fixed LLM weights at inference time.
  • ad hoc to paper Leave-one-out baseline b−i does not depend on (pi,wi).
    Definition 3.1; the structural modification that delivers all three claimed properties.
invented entities (1)
  • Advantage-aligned wagering mechanism family (WALLA) with leave-one-out baseline independent evidence
    purpose: Simultaneously achieve DSIC under arbitrary beliefs, continuous advantage-proportional wagers, and prediction-agnostic wager learning for LLM aggregation.
    The specific payout form and the two concrete baselines (weighted-score vs. weighted-prediction) are introduced in this paper; independent evidence is the empirical match to centralized methods and the formal proofs.

pith-pipeline@v1.1.0-grok45 · 41490 in / 2698 out tokens · 29725 ms · 2026-07-11T19:30:48.924189+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Decentralized Aggregation of LLM Predictions via Wagering Mechanisms." pith.science (2026). https://pith.science/paper/MJKFQ5NW

@misc{pith2026260704389,
  author       = {Pith},
  title        = {Pith review of: Decentralized Aggregation of LLM Predictions via Wagering Mechanisms},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MJKFQ5NW}},
  note         = {Machine review of arXiv:2607.04389}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

It is increasingly common to aggregate predictions from multiple LLMs, each with domain expertise or access to private tools and data, to improve collective prediction performance. In decentralized settings, aggregation weights need to be determined without access to models' private information and should remain robust to strategic reporting. We propose a family of advantage-aligned wagering mechanisms for LLM aggregation (WALLA), in which each model reports a prediction and a learned wager, and predictions are aggregated using wagers as weights. WALLA introduces a leave-one-out baseline into the net payout function, yielding three desirable properties: (1) dominant-strategy incentive compatibility of prediction under arbitrary belief structure, (2) advantage--wager alignment, where the optimal wager is proportional to the model's expected score advantage, and (3) prediction-agnostic wager optimization, enabling decentralized learning of wager policies without requiring optimal predictions. We further instantiate two mechanism variants that trade off normality and no-arbitrage while maintaining a bounded worst-case deficit for the mechanism. Experiments on question-answering and forecasting benchmarks across heterogeneous models and private-information settings show that WALLA matches centralized aggregation methods in predictive performance, while simultaneously achieving decentralized learning, advantage-aligned aggregation weights, uncertainty awareness, and incentive-compatible prediction.

Figures

Figures reproduced from arXiv: 2607.04389 by David M. Pennock, Xintong Wang, Yuhong Luo.

Figure 1
Figure 1. Figure 1: Advantage–wager alignment for WALLA I. Learned wager ( [PITH_FULL_IMAGE:figures/full_fig_p014_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The alignment between the actual Brier scores ( [PITH_FULL_IMAGE:figures/full_fig_p029_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The alignment between the actual baseline average scores ( [PITH_FULL_IMAGE:figures/full_fig_p030_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The alignment between the wagers (x-axis) and the differences between the predicted own scores and the predicted baseline average scores (y-axis), which can be interpreted as the best-response wager under one’s belief. The alignment is strong even though the wagers, the predicted own scores, and the predicted baseline average scores are trained with separate heads [PITH_FULL_IMAGE:figures/full_fig_p031_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Dynamic regret, mean reciprocal rank of the best expert (MRR), and accuracy (ACC) over time of [PITH_FULL_IMAGE:figures/full_fig_p031_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

63 extracted references · 12 linked inside Pith

  1. [1]

    Routing to the expert: Efficient reward-guided ensemble of large language models.arXiv preprint arXiv:2311.08692, 2023

    Keming Lu, Hongyi Yuan, Rui Zheng, Jianye Yin, Xiaotian Zhao, Xuanjing Huang, and Xipeng Qiu. Routing to the expert: Efficient reward-guided ensemble of large language models.arXiv preprint arXiv:2311.08692, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    RouterDC: Query-based router by dual contrastive learning for assembling large language models

    Shuhao Chen, Weisen Jiang, Baijiong Lin, James Kwok, and Yu Zhang. RouterDC: Query-based router by dual contrastive learning for assembling large language models. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  3. [3]

    Gonzalez, M Waleed Kadous, and Ion Stoica

    Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, and Ion Stoica. RouteLLM: Learning to route LLMs from preference data. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  4. [4]

    IRT-router: Effective and interpretable multi-LLM routing via item response theory

    Wei Song, Zhenya Huang, Cheng Cheng, Weibo Gao, Bihan Xu, GuanHao Zhao, Fei Wang, and Runze Wu. IRT-router: Effective and interpretable multi-LLM routing via item response theory. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025

    2025

  5. [5]

    David H. Wolpert. Stacked generalization.Neural Networks, 5(2):241–259, 1992

    1992

  6. [6]

    Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171, 2022

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171, 2022

    Pith/arXiv arXiv 2022

  7. [7]

    Improving factuality and reasoning in language models through multiagent debate.arXiv preprint arXiv:2305.14325, 2023

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate.arXiv preprint arXiv:2305.14325, 2023

    Pith/arXiv arXiv 2023

  8. [8]

    Debate or vote: Which yields better decisions in multi-agent large language models? InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    Hyeong Kyu Choi, Jerry Zhu, and Sharon Li. Debate or vote: Which yields better decisions in multi-agent large language models? InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    2025

  9. [9]

    Scalable best-of-n selection for large language models via self-certainty

    Zhewei Kang, Xuandong Zhao, and Dawn Song. Scalable best-of-n selection for large language models via self-certainty. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    2025

  10. [10]

    Pack of LLMs: Model fusion at test-time via perplexity optimization

    Costas Mavromatis, Petros Karypis, and George Karypis. Pack of LLMs: Model fusion at test-time via perplexity optimization. InFirst Conference on Language Modeling, 2024

    2024

  11. [11]

    Lambert, John Langford, Jennifer Wortman, Yiling Chen, Daniel Reeves, Yoav Shoham, and David M

    Nicolas S. Lambert, John Langford, Jennifer Wortman, Yiling Chen, Daniel Reeves, Yoav Shoham, and David M. Pennock. Self-financed wagering mechanisms for forecasting. InProceedings of the 9th ACM Conference on Electronic Commerce, page 170–179, 2008

    2008

  12. [12]

    Lambert, John Langford, Jennifer Wortman Vaughan, Yiling Chen, Daniel M

    Nicolas S. Lambert, John Langford, Jennifer Wortman Vaughan, Yiling Chen, Daniel M. Reeves, Yoav Shoham, and David M. Pennock. An axiomatic characterization of wagering mechanisms.Journal of Economic Theory, 156:389–416, 2015

    2015

  13. [13]

    Devanur, David M

    Yiling Chen, Nikhil R. Devanur, David M. Pennock, and Jennifer Wortman Vaughan. Removing arbitrage from wagering mechanisms. InProceedings of the Fifteenth ACM Conference on Economics and Computation, page 377–394, 2014

    2014

  14. [14]

    Pennock, and Jennifer Wortman Vaughan

    Rupert Freeman, David M. Pennock, and Jennifer Wortman Vaughan. The double clinching auction for wagering. InProceedings of the 2017 ACM Conference on Economics and Computation, page 43–60, 2017

    2017

  15. [15]

    Zhijun Chen, Jingzheng Li, Pengpeng Chen, Zhuoran Li, Kai Sun, Yuankai Luo, Qianren Mao, Ming Li, Likang Xiao, Dingqi Yang, Yikun Ban, Hailong Sun, and Philip S. Yu. Harnessing multiple large language models: A survey on LLM ensemble.arXiv preprint arXiv:2502.18036, 2025. 16

    Pith/arXiv arXiv 2025

  16. [16]

    More agents is all you need.Transactions on Machine Learning Research, 2024

    Junyou Li, Qin Zhang, Yangbin Yu, Qiang Fu, and Deheng Ye. More agents is all you need.Transactions on Machine Learning Research, 2024. ISSN 2835-8856

    2024

  17. [17]

    Breaking the ceiling of the LLM community by treating token generation as a classification for ensembling

    Yao-Ching Yu, Chun Chih Kuo, Ye Ziqi, Chang Yucheng, and Yueh-Se Li. Breaking the ceiling of the LLM community by treating token generation as a classification for ensembling. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 1826–1839, 2024

    2024

  18. [18]

    Language models (mostly) know what they know.arXiv preprint arXiv:2207.05221, 2022

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Nova Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know.arXiv preprint arXiv:2207.05221, 2022

    Pith/arXiv arXiv 2022

  19. [19]

    The internal state of an LLM knows when it’s lying

    Amos Azaria and Tom Mitchell. The internal state of an LLM knows when it’s lying. InFindings of the Association for Computational Linguistics: EMNLP 2023, pages 967–976, 2023

    2023

  20. [20]

    Enhancing uncertainty estimation in LLMs with expectation of aggregated internal belief.Proceedings of the AAAI Conference on Artificial Intelligence, 40(40):34043–34051, Mar

    Zeguan Xiao, Diyang Dou, Boya Xiong, Yun Chen, and Guanhua Chen. Enhancing uncertainty estimation in LLMs with expectation of aggregated internal belief.Proceedings of the AAAI Conference on Artificial Intelligence, 40(40):34043–34051, Mar. 2026

    2026

  21. [21]

    Large language model routing with benchmark datasets

    Tal Shnitzer, Anthony Ou, Mírian Silva, Kate Soule, Yuekai Sun, Justin Solomon, Neil Thompson, and Mikhail Yurochkin. Large language model routing with benchmark datasets. InFirst Conference on Language Modeling, 2024

    2024

  22. [22]

    Solving math word problems with process- and outcome- based feedback.arXiv preprint arXiv:2211.14275, 2022

    Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process- and outcome- based feedback.arXiv preprint arXiv:2211.14275, 2022

    Pith/arXiv arXiv 2022

  23. [23]

    LLM-blender: Ensembling large language models with pairwise ranking and generative fusion

    Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LLM-blender: Ensembling large language models with pairwise ranking and generative fusion. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14165–14178, 2023

    2023

  24. [24]

    Let’s verify step by step.arXiv preprint arXiv:2305.20050, 2023

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step.arXiv preprint arXiv:2305.20050, 2023

    Pith/arXiv arXiv 2023

  25. [25]

    Mechanism design for large language models

    Paul Dütting, Vahab Mirrokni, Renato Paes Leme, Haifeng Xu, and Song Zuo. Mechanism design for large language models. InProceedings of the ACM Web Conference 2024, page 144–155, 2024

    2024

  26. [26]

    Auctions with llm summaries

    Avinava Dubey, Zhe Feng, Rahul Kidambi, Aranyak Mehta, and Di Wang. Auctions with llm summaries. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, page 713–722, 2024

    2024

  27. [27]

    Data- driven mechanism design: Jointly eliciting preferences and information

    Dirk Bergemann, Marek Bojko, Paul Duetting, Renato Paes Leme, Haifeng Xu, and Song Zuo. Data- driven mechanism design: Jointly eliciting preferences and information. InProceedings of the 26th ACM Conference on Economics and Computation, page 507, 2025

    2025

  28. [28]

    Truthful aggregation of LLMs with an application to online advertising

    Ermis Soumalias, Michael Curry, and Sven Seuken. Truthful aggregation of LLMs with an application to online advertising. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026

    2026

  29. [29]

    Is your LLM outdated? a deep look at temporal generalization

    Chenghao Zhu, Nuo Chen, Yufei Gao, Yunyi Zhang, Prayag Tiwari, and Benyou Wang. Is your LLM outdated? a deep look at temporal generalization. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1:...

    2025

  30. [30]

    Are LLMs prescient? a continuous evaluation using daily news as the oracle

    Hui Dai, Ryan Teehan, and Mengye Ren. Are LLMs prescient? a continuous evaluation using daily news as the oracle. InForty-second International Conference on Machine Learning, 2025. 17

    2025

  31. [31]

    Forecasting future world events with neural networks

    Andy Zou, Tristan Xiao, Ryan Jia, Joe Kwon, Mantas Mazeika, Richard Li, Dawn Song, Jacob Steinhardt, Owain Evans, and Dan Hendrycks. Forecasting future world events with neural networks. InProceedings of the 36th International Conference on Neural Information Processing Systems, 2022

    2022

  32. [32]

    Approaching human-level forecasting with language models

    Danny Halawi, Fred Zhang, Chen Yueh-Han, and Jacob Steinhardt. Approaching human-level forecasting with language models. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  33. [33]

    ForecastQA: A question answering challenge for event forecasting with temporal text data

    Woojeong Jin, Rahul Khanna, Suji Kim, Dong-Ho Lee, Fred Morstatter, Aram Galstyan, and Xiang Ren. ForecastQA: A question answering challenge for event forecasting with temporal text data. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol...

    2021

  34. [34]

    Analyzing temporal complex events with large language models? a benchmark towards temporal, long context understanding

    Zhihan Zhang, Yixin Cao, Chenchen Ye, Yunshan Ma, Lizi Liao, and Tat-Seng Chua. Analyzing temporal complex events with large language models? a benchmark towards temporal, long context understanding. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1588–1606, 2024

    2024

  35. [35]

    Forecastbench: A dynamic benchmark of AI forecasting capabilities

    Ezra Karger, Houtan Bastani, Chen Yueh-Han, Zachary Jacobs, Danny Halawi, Fred Zhang, and Philip Tetlock. Forecastbench: A dynamic benchmark of AI forecasting capabilities. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  36. [36]

    OpenForecast: A large-scale open-ended event forecasting dataset

    Zhen Wang, Xi Zhou, Yating Yang, Bo Ma, Lei Wang, Rui Dong, and Azmat Anwar. OpenForecast: A large-scale open-ended event forecasting dataset. InProceedings of the 31st International Conference on Computational Linguistics, pages 5273–5294, 2025

    2025

  37. [37]

    Futurex: An advanced live benchmark for LLM agents in future prediction.arXiv preprint arXiv:2508.11987, 2025

    Zhiyuan Zeng et al. Futurex: An advanced live benchmark for LLM agents in future prediction.arXiv preprint arXiv:2508.11987, 2025

    Pith/arXiv arXiv 2025

  38. [38]

    OpenEP: Open-ended future event prediction.ACM Trans

    Yong Guan, Hao Peng, Xiaozhi Wang, Lei Hou, and Juanzi Li. OpenEP: Open-ended future event prediction.ACM Trans. Inf. Syst., 2026

    2026

  39. [39]

    LLM-as-a-prophet: Understanding predictive intelligence with prophet arena

    Qingchuan Yang, Simon Mahns, Sida Li, Anri Gu, Jibang Wu, and Haifeng Xu. LLM-as-a-prophet: Understanding predictive intelligence with prophet arena. InThe Fourteenth International Conference on Learning Representations, 2026

    2026

  40. [40]

    Park, Rafael Valdece Sousa Bastos, and Philip E

    Philipp Schoenegger, Indre Tuminauskaite, Peter S. Park, Rafael Valdece Sousa Bastos, and Philip E. Tetlock. Wisdom of the silicon crowd: Llm ensemble prediction capabilities rival human crowd accuracy. Science Advances, 10(45):eadp1528, 2024

    2024

  41. [41]

    Pennock, Chara Podimata, and Jennifer Wortman Vaughan

    Rupert Freeman, David M. Pennock, Chara Podimata, and Jennifer Wortman Vaughan. No-regret and incentive-compatible online learning. InProceedings of the 37th International Conference on Machine Learning, 2020

    2020

  42. [42]

    A market for trading forecasts: A wagering mechanism.International Journal of Forecasting, 40(1):142–159, 2024

    Aitazaz Ali Raja, Pierre Pinson, Jalal Kazempour, and Sergio Grammatico. A market for trading forecasts: A wagering mechanism.International Journal of Forecasting, 40(1):142–159, 2024

    2024

  43. [43]

    Frongillo

    Jacob Abernethy and Rafael M. Frongillo. A collaborative mechanism for crowdsourcing prediction problems. InAdvances in Neural Information Processing Systems, 2011

    2011

  44. [44]

    Alina Beygelzimer, John Langford, and David M. Pennock. Learning performance of prediction markets with kelly bettors. InProceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems - Volume 3, page 1317–1318, 2012

    2012

  45. [45]

    Interpreting prediction markets: a stochastic approach

    Rafael Frongillo, Nicholás Della Penna, and Mark D Reid. Interpreting prediction markets: a stochastic approach. InAdvances in Neural Information Processing Systems, volume 25, 2012

    2012

  46. [46]

    Efficient market making via convex optimization, and a connection to online learning.ACM Transactions on Economics and Computation, 1 (2), 2013

    Jacob Abernethy, Yiling Chen, and Jennifer Wortman Vaughan. Efficient market making via convex optimization, and a connection to online learning.ACM Transactions on Economics and Computation, 1 (2), 2013. 18

    2013

  47. [47]

    Multi-period trading prediction markets with connections to machine learning

    Jinli Hu and Amos Storkey. Multi-period trading prediction markets with connections to machine learning. InProceedings of the 31st International Conference on Machine Learning, pages 1773–1781, 2014

    2014

  48. [48]

    Glenn W. Brier. Verification of forecasts expressed in terms of probability.Monthly Weather Review, 78: 1–3, 1950

    1950

  49. [49]

    Strictly proper scoring rules, prediction, and estimation

    Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359–378, 2007

    2007

  50. [50]

    Bayesian model averaging: a tutorial.Statistical science, 14(4):382–417, 1999

    Jennifer A Hoeting, David Madigan, Adrian E Raftery, and Chris T Volinsky. Bayesian model averaging: a tutorial.Statistical science, 14(4):382–417, 1999

    1999

  51. [51]

    A Characterization Theorem for Externally Bayesian Groups.The Annals of Statistics, 12(3):1100 – 1105, 1984

    Christian Genest. A Characterization Theorem for Externally Bayesian Groups.The Annals of Statistics, 12(3):1100 – 1105, 1984

    1984

  52. [52]

    Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118, 2024

    Gemma Team. Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118, 2024

    Pith/arXiv arXiv 2024

  53. [53]

    The Llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori et al. The Llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  54. [54]

    Aloe: A family of fine-tuned open healthcare LLMs.arXiv preprint arXiv:2405.01886, 2024

    Ashwin Kumar Gururajan, Enrique Lopez-Cuena, Jordi Bayarri-Planas, Adrian Tormos, Daniel Hinjos, Pablo Bernabeu-Perez, Anna Arias-Duart, Pablo Agustin Martin-Torres, Lucia Urcelay-Ganzabal, Marta Gonzalez-Mallo, Sergio Alvarez-Napagao, Eduard Ayguadé-Parra, and Ulises Cortés Dario Garcia-Gasulla. Aloe: A family of fine-tuned open healthcare LLMs.arXiv pre...

    Pith/arXiv arXiv 2024

  55. [55]

    BioMistral: A collection of open-source pretrained large language models for medical domains

    Yanis Labrak, Adrien Bazoge, Emmanuel Morin, Pierre-Antoine Gourraud, Mickael Rouvier, and Richard Dufour. BioMistral: A collection of open-source pretrained large language models for medical domains. InFindings of the Association for Computational Linguistics: ACL 2024, pages 5848–5864, 2024

    2024

  56. [56]

    PubMedQA: A dataset for biomedical research question answering

    Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. PubMedQA: A dataset for biomedical research question answering. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2567–2577, 2019

    2019

  57. [57]

    MedMCQA: A large-scale multi- subject multi-choice dataset for medical domain question answering

    Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. MedMCQA: A large-scale multi- subject multi-choice dataset for medical domain question answering. InProceedings of the Conference on Health, Inference, and Learning, volume 174, pages 248–260, 2022

    2022

  58. [58]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. InInternational Conference on Learning Representations, 2021

    2021

  59. [59]

    Think you have solved question answering? try ARC, the AI2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    Pith/arXiv arXiv 2018

  60. [60]

    Calibrating language models with adaptive temperature scaling

    Johnathan Xie, Annie S Chen, Yoonho Lee, Eric Mitchell, and Chelsea Finn. Calibrating language models with adaptive temperature scaling. InEMNLP, 2024

    2024

  61. [61]

    Improving cooperation in language games with bayesian inference and the cognitive hierarchy

    Joseph Bills, Christopher Archibald, and Diego Blaylock. Improving cooperation in language games with bayesian inference and the cognitive hierarchy. InProceedings of the Thirty-Ninth AAAI Conference on Artificial Intelligence and Thirty-Seventh Conference on Innovative Applications of Artificial Intelligence and Fifteenth Symposium on Educational Advance...

    2025

  62. [62]

    The multiplicative weights update method: a meta- algorithm and applications.Theory of Computing, 8(6):121–164, 2012

    Sanjeev Arora, Elad Hazan, and Satyen Kale. The multiplicative weights update method: a meta- algorithm and applications.Theory of Computing, 8(6):121–164, 2012. doi: 10.4086/toc.2012.v008a006

    doi:10.4086/toc.2012.v008a006 2012

  63. [63]

    yes,” “no,

    PengchengHe, JianfengGao, andWeizhuChen. DeBERTav3: ImprovingdeBERTausingELECTRA-style pre-training with gradient-disentangled embedding sharing. InThe Eleventh International Conference on Learning Representations, 2023. 19 A Theoretical Properties of WSWM Using a Scaled Linear Score Consider the repeated setting where questionx(t) arrives at timet with o...

    2023