{"id":"b576ccb5-5eaa-415c-9874-68b37f11a10d","arxiv_id":"2507.19477","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A position paper advocating large-scale training of event forecasting LLMs, with proposals for label selection, counterfactual training data, auxiliary rewards, and multi-source datasets.","lead":"This position paper argues that recent advances in LLM reasoning, reinforcement learning, and dynamic benchmarks make large-scale training of superforecaster-level event forecasting models a timely research goal. It proposes training methods and dataset strategies, and discusses societal impacts and risks.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The agenda's data-scale expansion depends on untested transfer from pre-cutoff poorly-recalled/counterfactual training to general forecasting; §4.2.2 concedes this, making the 'time is ripe' claim conditional.","rationale":"The reader identified the transfer assumption as the weakest link, and the paper's own §4.2.2 caveat confirms it. I agree that this is the most load-bearing concern for the central agenda: without transfer, the pre-cutoff and counterfactual training proposals lose their justification, and the large-scale dataset expansion in Section 5 is reduced to post-cutoff market/public data. The paper deserves credit for flagging the limitation explicitly and for grounding parts of its argument in existing results (e.g., Turtel et al.'s RLVR gains, Wen et al.'s in-task transfer, Karger et al.'s benchmark). The Appendix A simulation is illustrative rather than empirical, but it is honestly presented as such and is secondary to the transfer issue. Because the paper is a position paper and its proposals are framed as research directions, the appropriate verdict remains conditional acceptance: the agenda is plausible and timely, but its central mechanism is unvalidated. No change to the reader's verdict is warranted; the conditional stamp already encodes this risk.","tokens_in":26800,"tokens_out":5336,"duration_ms":55474,"concrete_test":"Run a controlled transfer experiment on a dynamic benchmark: take a single base LLM (e.g., a 14B open-weight model) and train three variants—(A) on ~50k pre-cutoff poorly-recallable constructed events as in §4.2.2, (B) on ~50k post-cutoff standard event-forecasting questions, (C) no event-forecasting training. Evaluate all variants on 200+ fresh post-cutoff questions from ForecastBench or Metaculus, reporting Brier score and ECE. If (A) does not significantly outperform (B) and (C), the pre-cutoff data strategy in §4.2/§5 is unsupported. Release code and data splits for reproduction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that large-scale training can approach superforecaster-level—depends on the feasibility of training on data before the knowledge cutoff. Section 4.2.2 proposes using events LLMs poorly recall, but its only cited evidence, Wen et al. (2025), demonstrates transfer within a single task family (predicting which research idea performs better on benchmarks), not to general event forecasting. Section 4.2.3's counterfactual-document pipeline is supported only by analogy to QA counterfactual training (Neeman et al. 2023). The paper itself flags the gap: 'it remains an open empirical question whether training on such poorly-recalled historical events improves performance on general event forecasting tasks' (§4.2.2). This is load-bearing because Section 5's projected expansion from ~10^4 to ~10^5 training instances explicitly relies on pre-cutoff data (§4.2) and on intermediate-prediction labels (§4.1.5), both of which are unvalidated. If transfer fails, the dataset-scale argument collapses to post-cutoff market/public data, and the agenda loses its main new lever. The paper is honest about the caveat, but the caveat is exactly where the central claim is least secure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This position paper argues that recent developments in LLM-based event forecasting — steadily improving results on dynamic benchmarks (Karger et al., 2025), outcome-based reinforcement learning results (Turtel et al., 2025a;b), and the emergence of reasoning and Deep Research-style models — indicate that 'the time is ripe for research on large-scale training of superforecaster-level event forecasting LLMs.' The paper identifies three training difficulties specific to event forecasting: noisiness and sparsity, the knowledge cut-off, and the simple reward structure. It then proposes mitigations: a hypothetical-event Bayesian-network framework for choosing training labels (§4.1.2 and Appendix A), training on poorly recalled and counterfactual pre-cutoff events (§4.2), and auxiliary reward signals from reasoning evaluation and subquestions (§4.3). For data, it advocates aggressive use of market, public, and crawled datasets, projecting a scale-up from roughly 10^4 to over 10^5 training instances (§5). The final sections survey societal impacts and risks. The main technical artifact is a simulation in Appendix A showing that intermediate market predictions m1 can, under certain noise conditions, serve as a better training label for the hidden question-time probability Phidden than either question-time market predictions m0 or final outcomes o.","tokens_in":27050,"tokens_out":16471,"duration_ms":165276,"significance":"The paper has genuine strengths: it is unusually honest about the field's evaluation pitfalls, it explicitly names its own key open question (§4.2.2), and the appendix simulation is specified precisely enough to be checked and reproduced. The framing of the knowledge cut-off as the central data-scale bottleneck is insightful, and the market/public/crawling dataset taxonomy gives the agenda a concrete and actionable shape. If the underlying transfer assumptions hold, the program would be high-impact and would close a real gap between current work and superforecaster-level performance. However, both load-bearing premises — transfer from poorly recalled and counterfactual historical training to general event forecasting, and the empirical realizability of the noise regime in which m1 dominates — are unvalidated, and the first is conceded by the paper itself. The stress-test concern about §4.2.2 therefore lands: the scale-up argument in §5 depends on exactly the assumption that the paper flags as open.","major_comments":[{"comment":"The Abstract and §3 state the readiness claim ('the time is ripe for research on large-scale training') unconditionally, yet the agenda's main new lever — the projected scale-up from ~10^4 to ≥10^5 training instances in §5 — depends on pre-cutoff training data (§4.2) and on intermediate-prediction labels (§4.1.5 and §5.4), and §4.2.2 explicitly concedes that 'it remains an open empirical question whether training on such poorly-recalled historical events improves performance on general event forecasting tasks.' If that transfer fails, the scale-up argument loses its main new lever and reduces to the already-explored regime of post-cutoff data with outcome/market labels. I recommend that the Abstract and Conclusion either carry this conditionality explicitly or commit to a concrete falsifiable first milestone, such as a pre-registered study that trains on poorly recalled historical event pairs and evaluates on a dynamic benchmark (ForecastBench or the Metaculus AI Benchmarking series); the readiness claim should be tied to such a test rather than to extrapolation from the positive trends in §3.","section":"Abstract; §3; §4.2.2; §5"},{"comment":"In Appendix A (Fig. 2), the superiority of m1 over m0 and o is a property of the assumed generative model, not an empirical discovery: m1 is sampled from (1−t)·α̂ + t·β̂ conditional on the realized intermediate state t, and the six noise parameters (σ²_biased and σ²_unbiased for α, β, and π) are free, so the m1-superiority regime in the first panel is produced by construction. What is missing is the bridge to the real training setting. §4.1.2's closing recommendation maps the simulation's N (the number of i.i.d. trials whose estimates are averaged) onto the training-data N (the number of similar events in a training corpus), but an LLM trained on N individual labels is not an estimator that averages those labels; moreover, the averaged m1 is unbiased for Phidden in the simulation precisely because m1 conditions on the realized S1, and the analogous noise structure in real markets is unknown. I ask the authors to add, or explicitly mark as a prerequisite for the §4.1.5 and §5.4 proposals, an empirical label-quality analysis on resolved prediction-market questions (e.g., the empirical Brier scores of m0, m1, and o as targets for question-time predictions, stratified by market depth), so that the m1-based proposals are presented as conditional on that evidence.","section":"Appendix A; Fig. 2; §4.1.2; §4.1.5; §5.4"},{"comment":"§4.2.3's counterfactual-event pipeline is supported only by analogy to counterfactual QA training: Neeman et al. (2023) train models to suppress parametric knowledge in favor of retrieved context for answering factual counterfactual questions, whereas the proposed pipeline requires the model to produce calibrated probability estimates from fictional news documents during training and to transfer that probabilistic reasoning to real questions at inference. The paper does not address why probability-calibration behavior learned under generated counterfactual outcomes would transfer to a real distribution in which the model's prior knowledge is partly valid. In addition, the proposed divergence-date selection criterion of §4.2.3 (query the model's probability at past time points and sample dates where uncertainty is 'appropriately high') risks encoding the model's own miscalibration into the training distribution. I recommend specifying a validation loop — for example, measuring ECE and Brier score on a held-out post-cutoff set after counterfactual training — or relabeling this proposal as an explicitly speculative direction rather than one of the paper's solutions.","section":"§4.2.3; §4.2.2; §3"}],"minor_comments":[{"comment":"The caption refers to the 'average of m0, m1, and m2', but the model in §4.1.2 and Appendix A defines only m0, m1, and o; 'm2' should be 'o'.","section":"Fig. 2 caption"},{"comment":"The RetroSearch technique is attributed to 'Bosse et al. (2025)', but the cited Bosse et al. paper is the Deep Research Bench study; please verify the attribution, since the primary source for RetroSearch appears to be Wildman et al. (2025), cited in the same sentence.","section":"§2.4"},{"comment":"Notation is inconsistent within the section: 'm₁' appears alongside 'm1', 'q1', and 'm0'; the subscript style should be unified.","section":"§4.1.5"},{"comment":"The SpaceX example in Table 1 (referenced to Polymarket 2023) is a different question from the orbital-flight-by-June-2024 example used in §4.1 and §4.1.2; the running example should be aligned throughout the paper.","section":"Table 1; §4.1"},{"comment":"There is a spacing typo in 'counterfactualoutcomesandusethemasiftheyweresearchresults'; the paragraph should receive a copyedit pass.","section":"§4.2.3"},{"comment":"The claim that AI 'has likely already significantly surpassed average individual performance' would benefit from an explicit statement of the comparison baseline, since the numbers reported just before (median general-public prediction 0.121 vs. GPT-4o 0.133 and Claude-3.5-Sonnet 0.122) show the crowd median roughly matching the best models; clarifying that the comparison is against average individuals, not the crowd median, would avoid confusion.","section":"§3"},{"comment":"There is a tension between the conclusion that outcomes are the best label when the number of similar events is large (§4.1.2) and the warning that outcome-based training pushes models toward extreme 0/1 predictions and thereby hinders search and reasoning (§4.1.3); the paper should spell out how the label-quality analysis relates to the reward-structure critique.","section":"§4.1.2 and §4.1.3"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern lands, and my recommendation of major revision rests on the gap between the unconditional framing in the Abstract/§3 and the conditional evidence conceded in §4.2.2 and implicit in Appendix A. My expectation is that the authors can address this without new experiments by reframing the readiness claim as conditional on explicit empirical milestones and marking the m1-based proposals as contingent on an empirical label-quality study; if the venue holds position papers to a high evidentiary bar, they should also consider adding a small pilot experiment. The heavy reliance on blog and notebook sources is standard for this subfield and appropriate here since many cited results first appear in those venues; I saw no other disclosure or citation concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should read this as a position paper with a clear agenda, not as a paper with validated results. The central argument is that recent LLM and RL progress makes large-scale training of superforecaster-level forecasting models a feasible goal, and it proposes specific training and data directions to get there. The best pieces are the hypothetical event Bayesian network in Section 4.1.2 and the label-choice discussion around m0, m1, and outcomes. That gives a clean vocabulary for a real problem: what should you train on when outcomes are noisy and sparse? The counterfactual document pipeline in Section 4.2.3 is also concrete, and the dataset expansion argument in Section 5 (market, public, crawling) is sensible and honest about quality versus quantity trade-offs.\n\nThe soft spots are real, but the paper does not hide them. The load-bearing premise is that training on pre-cutoff, poorly-recalled historical events and on counterfactual events transfers to general event forecasting. Section 4.2.2 concedes this is an open empirical question, and the only cited support (Wen et al.) is a single task family, not general forecasting. If transfer fails, the scale-up story loses most of its leverage. The m1-superiority claim in Appendix A is a toy simulation that demonstrates properties baked into the assumed generative model; the noise parameters drive the qualitative conclusion, so it should be read as an illustrative example, not empirical evidence. The Bayesian network is a conceptual framework, not a theoretical result. The subquestion auxiliary reward idea is a fresh combination but untested.\n\nThat said, the paper is coherent on its own terms, well-structured, and unusually candid about its own gaps. It is not overselling. Who gets value from it: researchers entering LLM forecasting training, especially those deciding what to work on, will get a useful map of challenges and proposed directions. It deserves serious referee time at a venue with a position-paper track. As a full research submission it would need at least one validated lever—say, an experiment showing counterfactual or poorly-recalled training improves forecasting on a held-out dynamic benchmark. I would send it to peer review for a position-track venue, and otherwise would not desk-reject it outright: the agenda is specific, falsifiable, and the field needs this kind of synthesis.","headline":"A genuinely useful, honest position paper that frames the forecasting-training agenda well; its load-bearing transfer assumptions are openly unvalidated, so read it as a research map, not a result.","tokens_in":27552,"tokens_out":1534,"would_cite":true,"duration_ms":19893,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Recent advances make it timely to train large language models toward superforecaster-level event forecasting.","keywords":["event forecasting","large language models","prediction markets","superforecaster-level","Bayesian networks","training labels","reinforcement learning","knowledge cutoff"],"falsifier":"Use the same base model and the same set of unresolved market questions, train with $m_0$ labels on one run and $m_1$ labels on another, and compare Brier scores on a held-out dynamic benchmark of questions resolved after training; if $m_1$-trained models do not beat $m_0$-trained models in the regime the paper's simulation identifies, the central sub-claim fails.","tokens_in":26594,"feed_emoji":"🔮","tokens_out":7224,"duration_ms":73882,"temperature":0.7,"pith_summary":"This position paper argues that recent advances in large language models have made large-scale training of superforecaster-level event forecasting systems both timely and feasible. It identifies three obstacles that make event forecasting training unlike other tasks: noisy and sparse outcomes, the knowledge cut-off that devalues pre-cutoff data, and a simple reward structure that rewards overconfident guessing, and it proposes mitigations for each. For label choice, it introduces a hypothetical event Bayesian network showing that intermediate market predictions can be a better training signal than question-time market predictions or final outcomes in certain noise regimes. It also argues for aggressively expanding training data from prediction markets, public databases, and web crawling. If the argument is right, forecasting AI could move beyond matching expert human forecasters to answering questions markets do not cover and powering future simulation and agent reasoning.","feed_headline":"Now is the time to train LLMs as superforecasters","feed_subtitle":"A position paper maps the training problems and data sources that could close the gap to top human forecasters.","key_machinery":"The key analytic object is a hypothetical event Bayesian network with hidden probability $P_{\\mathrm{hidden}} = (1-\\pi)\\alpha + \\pi\\beta$, where $\\alpha$ and $\\beta$ are the final success probabilities conditional on negative and positive intermediate states and $\\pi$ is the probability of the positive intermediate state. Market predictions $m_0$ and $m_1$ and outcome $o$ are treated as noisy observations of this structure, and simulation shows a bias-variance trade-off: $m_0$ has lowest variance but highest bias, $o$ has highest variance but lowest bias, and $m_1$ can win in the middle regime. This machinery justifies label-selection strategies in Section 4.1 and extends to using intermediate model predictions $q_1$ as training signals for questions without clear resolution conditions.","core_discovery":"The paper's central claim is that the time is ripe for research on large-scale training of superforecaster-level event forecasting LLMs. The supporting sub-claim, developed through a hypothetical event Bayesian network and simulation, is that intermediate market predictions $m_1$ can be a superior training label to question-time market predictions $m_0$ and final outcomes $o$ under certain conditions—specifically when there is significant uncertainty about the transition between the initial and intermediate states and a moderate number of training observations. The paper also claims that three mitigations—using poorly-recalled historical events and counterfactual events, and adding auxiliary reward signals—can overcome the knowledge cut-off and simple reward structure problems, and that combining market, public, and crawling datasets can expand training scale from roughly ten thousand instances to more than one hundred thousand.","pith_inferences":["Editorial: The paper's simulation suggests a direct head-to-head experiment—training identical models with $m_0$, $m_1$, and $o$ labels on the same data—that would validate or refute the central sub-claim before committing to a large-scale training run.","Editorial: If transfer from counterfactual or pre-cutoff training fails, the data-scale agenda in Section 5 still stands, because market and public datasets alone can plausibly reach the 100,000-instance scale; the paper somewhat over-couples these two arguments.","Editorial: The self-fulfilling prophecy risk the paper describes could also corrupt its own training labels once AI forecasts influence markets and public outcomes, creating a distribution shift that the proposed data pipeline does not address.","Editorial: The claim that AI already beats average individual forecasters suggests that consumer-facing personalized predictions may arrive before superforecaster-level AI, making reliability communication rather than raw accuracy the near-term bottleneck."],"forward_implications":["Label choice becomes a tunable training hyperparameter: use outcomes where similar events are plentiful, market predictions where they are sparse, and intermediate market predictions in between.","Pre-cutoff training data can be made usable through poorly-recalled comparative events and counterfactual scenarios, unlocking a much larger pool of historical training instances.","Auxiliary rewards from reasoning evaluation and subquestions could keep models from collapsing to overconfident 0% or 100% predictions during RL training.","Aggressive use of market, public, and crawling datasets enables both larger-scale training and faster dynamic benchmark evaluation, accelerating development of proprietary forecasting systems.","If training succeeds, forecasting AI could answer personalized and ambiguous questions that prediction markets cannot, and its probabilistic reasoning could be integrated into general agents and AI scientists."],"supporting_citations":[{"why":"Supplies the rigorous dynamic benchmark (ForecastBench) showing LLM forecasting improves with model generation, the paper's key positive trend.","marker":"Karger et al., 2025"},{"why":"Shows outcome-based reinforcement learning improves Brier scores and generates backtest profits, the evidence that training works.","marker":"Turtel et al., 2025b"},{"why":"Shows RLVR self-training on Polymarket data improves accuracy and reduces ECE, supporting large-scale market data training.","marker":"Turtel et al., 2025a"},{"why":"Defines the RAG retrieval pipeline and market-outcome interval labels that the paper's training discussion builds on.","marker":"Halawi et al., 2024"},{"why":"Demonstrates that training on 7,000 poorly-recalled paper-idea pairs reaches 77% accuracy, grounding the poorly-recalled events strategy.","marker":"Wen et al., 2025"},{"why":"Supplies the aleatoric/epistemic uncertainty framing used to characterize the noisiness-sparsity training difficulty.","marker":"Kendall and Gal, 2017"},{"why":"Shows counterfactual-document training improves grounding and reduces hallucination, motivating the counterfactual events idea.","marker":"Neeman et al., 2023"}],"fun_headline_variants":["Why train LLMs for superforecasting now","Forecast like a superforecaster: scale training data","Intermediate market prices as training labels for LLMs","Scaling to 100k events for superforecaster LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Training on historical events that the model cannot recall, and on counterfactual events, transfers to improved performance on general event forecasting tasks; the paper itself flags this as an open empirical question.","fun_headline_variants_meta":{"raw":{"variants":["Why train LLMs for superforecasting now","Forecast like a superforecaster: scale training data","Intermediate market prices as training labels for LLMs","Scaling to 100k events for superforecaster LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001281,"raw_usage":{"total_tokens":5246,"prompt_tokens":967,"completion_tokens":4279,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":583,"completion_tokens_details":{"reasoning_tokens":4214}},"tokens_in":583,"tokens_out":4279,"duration_ms":33018,"temperature":1.0,"reasoning_tokens":4214,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:15:10.924400+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Use the same base model and the same set of unresolved market questions, train with $m_0$ labels on one run and $m_1$ labels on another, and compare Brier scores on a held-out dynamic benchmark of questions resolved after training; if $m_1$-trained models do not beat $m_0$-trained models in the regime the paper's simulation identifies, the central sub-claim fails.","supporting_citations":[{"cited_title":"What uncertainties do we need in bayesian deep learning for computer vision? NIPS, 2017","cited_arxiv_id":null,"evidence_quote":"Supplies the aleatoric/epistemic uncertainty framing used to characterize the noisiness-sparsity training difficulty."},{"cited_title":"Disentqa: Disentangling parametric and contextual knowledge with counterfactual question answering","cited_arxiv_id":null,"evidence_quote":"Shows counterfactual-document training improves grounding and reduces hallucination, motivating the counterfactual events idea."}],"review_version":1}