Pith. sign in

REVIEW 4 major objections 4 minor 12 references

Hindcast grades LLM forecasters at a frozen past date, sealing off both answer leaks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 02:55 UTC pith:E2NZR2MY

load-bearing objection The replay protocol and topic split are worth engaging, but the central claim that Hindcast closes both leaks is overbuilt; send it to review with the expectation of major revision. the 4 major comments →

arxiv 2607.14051 v1 pith:E2NZR2MY submitted 2026-07-15 cs.CL

Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters

Xiao Ye , Jacob Dineen , Evan Zhu , Shijie Lu , Kevin Song , Ben Zhou This is my paper
classification cs.CL
keywords hindcastingLLM forecastingprediction marketstemporal contaminationretrieval-augmented generationbenchmarkingBrier scorefrozen archive
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces Hindcast, an evaluation protocol that tests language-model forecasters on resolved prediction-market questions while holding the information available to the model fixed at a chosen past date t0. The model may read only content posted before t0 from an immutable archive of public forum discussions, and its forecast is scored against the actual outcome and against the market price that traders set at that same past moment. The authors claim this closes two leaks that make ordinary backtests grade recall rather than foresight: retrieval can surface post-event reports, and newer models may have been trained on text written after the event. Once the leaks are closed, they find retrieval still improves most models, but only on questions the archived discussions had already covered with concrete pre-event signal; where the archive only carried speculation, retrieval makes forecasts worse. If correct, this gives the field a reusable, non-stale way to benchmark forecasting ability and pinpoints when retrieval-grounded forecasting genuinely helps.

Core claim

The central claim is that a forecaster can be graded on resolved questions as if it stood at a past date t0, and that this counterfactual is measurable. Hindcast pins every retrieval call to a cutoff t0 before the outcome existed, using a frozen archive of public forum posts and a per-market market price at the same t0 as a human-comparable yardstick. With both leak channels sealed, retrieval-grounded forecasting still lowers Brier score on eight of nine open-weight models (up to 23%), but the improvement is concentrated in topics where the pre-t0 forum posts already discussed the deciding facts (sports, awards, trading). On topics where the archive held only speculative chatter (notably ent

What carries the argument

The load-bearing mechanism is the temporal cutoff t0 combined with an immutable document archive. The archive consists of monthly snapshots of public forum posts, stored once and indexed; every retrieval tool enforces a row-level predicate created_at < t0 and a partition selector that restricts search to months up to t0. Because the archive never changes and the cutoff is set per market, the test can be re-run on newly resolved markets without going stale. The matched-time market price at t0 supplies a second score, a human forecast made from the same information the model saw. This combination turns a resolved question into a replayable counterfactual experiment.

Load-bearing premise

The protocol assumes the chosen cutoff date t0 is genuinely before the outcome was knowable; if a market's event resolved before t0 (possible when t0 is pinned to the archive's final day), posts retrieved before t0 could already contain the answer, reopening the retrieval leak.

What would settle it

Take a market whose resolution date falls after the archive's final day and whose event outcome was publicly known before the pinned t0; run the Hindcast retrieval agent and check whether any retrieved document created before t0 states or strongly implies the resolved outcome. If such documents are found and the agent's forecast moves toward them, the leak is not fully closed.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Retrieval-grounded forecasting improves calibration on most open-weight models even when the future is sealed off, contradicting the possibility that earlier gains were purely leakage artifacts.
  • The benefit of retrieval is topic-dependent: it recovers markets where the pre-t0 corpus carried concrete, fact-anchored discussion, and regresses where the corpus carried only speculative enthusiasm.
  • The protocol can be re-run on new markets as they resolve, so a fixed frozen archive does not decay into a recall test as models are updated.
  • Sweeping t0 along a market's lifetime yields a calibration trajectory, and the observed flat profile suggests retrieval gains saturate before the full pre-resolution lookback.
  • A model with no retrieval can beat a retrieval agent on speculation-heavy questions because the base rate is a better guess than misread hype.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the topic-dependence result generalizes, it suggests a design rule for forecasting agents: gate retrieval by an estimate of whether the archive contains fact-anchored pre-event discussion, rather than always retrieving.
  • The matched-time market price is a stricter and more informative yardstick than the binary outcome; future benchmarks could report probability-space error against the market rather than only accuracy, which would separate calibration from decision flips.
  • The protocol's reliance on a single social platform limits the evidence environment; extending the frozen-archive idea to other corpora (news, structured data) could test whether the topic split is a property of the corpus or of retrieval-grounded LLMs.
  • The finding that retrieval hurts where speculation dominates suggests forecasting agents should be evaluated under an explicit 'no-signal' mode that falls back to a base-rate prior.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces HINDCAST, an evaluation protocol that replays resolved Polymarket binary markets against a frozen snapshot of public Reddit, pinning every retrieval call to a per-market cutoff t0 and scoring the resulting LLM forecasts against the resolved outcome. Two predictor families are compared: a zero-shot baseline and a retrieval agent that reads only pre-t0 Reddit documents. The authors claim this protocol closes both the retrieval leak (post-event documents) and the parametric-memory leak (post-t0 training data), and report that on a 216-market eval set retrieval lowers Brier for eight of nine open-weight models, with gains concentrated in topics where Reddit carried concrete pre-event discussion and losses in Entertainment where hype is read as evidence. They also describe a coverage-probe-based eval-set construction, a duration sweep, and a per-topic recovery/regression analysis.

Significance. If the central claims held, HINDCAST would be a valuable, reusable evaluation methodology: it combines a genuinely immutable, timestamped corpus with per-market temporal pinning, and it ships an unusually complete artifact set (corpus, eval-set regeneration code, per-run logs, nine models, two pipelines). The finding that retrieval helps only where pre-event discussion is fact-anchored is interesting and falsifiable. However, the paper's most load-bearing claim—that both leakage channels are closed—is not supported by the methods as written, and the empirical design partially builds the headline result into the selection criterion. The retrieval-side temporal control is well engineered and worth preserving, but the manuscript overstates what it establishes.

major comments (4)
  1. [Abstract, §2, §5.1] The abstract's central claim that HINDCAST 'closes both leaks' and grades the model 'before the outcome existed in either channel' is contradicted by §2, which states that HINDCAST attacks the retrieval channel while retraining strategies attack the parametric-memory channel, and by §5.1, where the zero-shot baseline merely instructs the model to 'use only knowledge that could be known at t0.' All nine models are standard open-weight checkpoints whose pretraining data include post-t0 text; the instruction does not remove outcome information from the weights. The paper therefore does not close the parametric leak, and the retrieval-vs-zero-shot comparison is measured on top of an uncontrolled parametric channel. This is load-bearing for the headline 'leak is closed' claim.
  2. [§4.2, Limitations] The cutoff rule t0 = day before close, capped at the archive final day 2026-01-31, is not verified to precede public availability of the outcome. The Limitations concedes that most markets have t0 pinned at the archive boundary, with median close–t0 gap of 23 days. A market whose close falls after the archive end may have its underlying event resolved—and discussed on Reddit—before t0; retrieval would then surface outcome-containing posts. Because the entire evaluation rests on this temporal-placement premise, the paper needs an ex-post check that for every included market no pre-t0 document contains the outcome, and must move t0 earlier or exclude markets that fail the check.
  3. [§4.1, §6, Table 1] The eval set is constructed by keeping only markets the coverage probe labels STRONG for the resolved side (§4.1). The paper then reports that retrieval helps most on STRONG markets and concludes that retrieval helps only where Reddit discussed the event beforehand (§6, §7.1). Since the set contains no WEAK/NONE markets, the comparison cannot establish that claim; the result is partly built into the selection. The within-set S/W/N stratification in Table 1 is per-run from the agent's own retrieval tally (caption), not the original probe labels, so the table header 'coverage probe's three labels' is misleading and the bins are not an independent test. A fair test requires an eval set spanning coverage strengths or an explicit analysis of selection effects.
  4. [Abstract, §2, §4.2] The paper repeatedly promises a matched-time market-price score: the abstract says forecasts are scored 'against both what happened and the market's own price at t0,' and §2 says the matched-time market-implied probability serves as a baseline. However, Eq. (1) defines only accuracy and Brier against the resolved outcome, and no table or figure reports agreement with the contemporaneous market price q_t0. The market price is invoked as a yardstick but never used empirically. Either include the promised comparison or remove the claim from the abstract and introduction.
minor comments (4)
  1. [§4.2, Limitations] 'Day before close' and 'median close–t0 gap of 23 days' are inconsistent unless most close dates are after the archive end; state this explicitly and report how many markets are affected, ideally in a histogram of close dates relative to 2026-01-31.
  2. [Table 1] The caption says S/W/N are 'the coverage probe's three labels' but later says they come from each run's own retrieval tally. This is inconsistent and should be resolved; the two labeling procedures are different and should not be conflated.
  3. [Figure 1] The caption appears garbled ('1 mo', '0 0.5 1', '1 wk 1 dTr') and the visual legend is unclear. Please redraw and rewrite the caption so the illustrative market and the plotted quantities are legible.
  4. [§7.3] The explanation that R1-Distill-Qwen-7B's degradation is a 'context-budget problem' is asserted without supporting evidence. Neither context-window usage nor retrieval-tool-call counts are reported for this model. Provide a check (e.g., token consumption per run or number of successful tool calls) or soften the explanation.

Circularity Check

1 steps flagged

The 'retrieval helps only where Reddit discussed it' contrast is stratified on S/W/N labels generated by the retrieval agent's own per-run tally, and the same probe family selects the eval set; other results give some independent support.

specific steps
  1. self definitional [§4.1 (Eval-set construction) and §6/Table 1 (main results, S/W/N bins)]
    "We keep only the markets the probe labels STRONG. ... Because the S/W/N bin assignment is per-run from the agent’s own retrieval tally, the per-bin std also reflects bin-membership churn."

    The paper's central empirical claim is that retrieval helps 'only where Reddit discussed the event beforehand.' That 'where' is operationalized by the S/W/N bins, but the bins are assigned per run from the retrieval agent's own evidence tally, not from an independent measure of Reddit's pre-t0 content. The eval set was additionally filtered to markets that the same probe labels STRONG. A market lands in 'strong' exactly when the agent's retrieval finds evidence it considers strong for the resolved side, which mechanically makes retrieval look effective; a 'None' market is one where the agent failed to find evidence, not one where the archive lacked discussion. The strong-vs-none contrast is therefore partly an artifact of evaluating the system with its own labels, rather than an independen

full rationale

The paper has genuine independent content: the matched-time market-price baseline, the per-topic net effects (especially the negative Entertainment result), and the lookback-fraction sweep are not forced by eval-set construction, and no load-bearing self-citation or uniqueness theorem is invoked. However, the headline finding that retrieval helps only where Reddit discussed the event beforehand is stratified on S/W/N labels that are generated per run by the retrieval agent's own tally (Table 1 note, Appendix C.2), while the same probe family was used to select the eval set; this makes part of the 'where' variable endogenous to the system being measured. Separately, the abstract's claim that Hindcast 'closes both leaks' is contradicted by §2, which says Hindcast attacks only the retrieval channel while retraining attacks the parametric-memory channel; the zero-shot baseline's instruction to use only pre-t0 knowledge is a prompt, not a weight edit. That is a correctness/validity risk rather than a further circularity, but it reinforces the moderate score. On balance, the circularity is partial, not total, so score 4.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The central empirical claims rest on a chain of hand-chosen construction choices: the coverage probe's labels decide which markets enter the eval set and are reused for stratified claims; the t0 rule is a wall-clock heuristic not verified against event times; and the models' parametric memory is not neutralized. None are fitted constants in the physics sense, but each is a free knob that can move the headline numbers.

free parameters (5)
  • Coverage-probe inclusion criterion (STRONG only) = STRONG label only; hand-set weights +2/+1/0/−1
    Only markets the Qwen3-32B probe labels STRONG are kept in the eval set; this hand-set weighting and threshold determines which markets are measured and likely inflates retrieval gains.
  • Cutoff t0 placement rule = day before market close, capped at 2026-01-31
    Hand-chosen placement of t0; the cap at the archive end means most markets are evaluated at the same boundary, and t0 may fall after the event outcome.
  • Corpus engagement/length floors = score ≥ 5, comments ≥ 3, ≥ 20 words, comment ≥ 5 non-URL chars
    Arbitrary filtering thresholds that define the archive; changing them would change the retrieved evidence and the measured retrieval benefit.
  • Retrieval-agent loop parameters = 3 rounds, 6 seed queries, top-20 hits, expand 20 submissions, confidence stop 0.7
    Hand-set agent configuration; results are likely sensitive to these budgets and stopping rules.
  • Prior-anchor ranges in forecast prompt = .05-.20 / .10-.30 / .40-.60 / .55-.85 / .70-.95
    Hand-specified prior ranges in the forecasting stage shape final probabilities and therefore Brier scores.
axioms (5)
  • domain assumption Reddit monthly snapshots from Pushshift exactly preserve what was visible at t0
    The whole retrieval cutoff guarantee rests on snapshot immutability and the absence of later edits/removals in the captured copy (§3.1).
  • ad hoc to paper Polymarket close date is a good proxy for the moment the outcome became knowable
    t0 is defined as one day before market close, but for markets that close after the underlying event resolves, Reddit before t0 can contain the outcome; the paper never checks t0 against event time (§4.2).
  • domain assumption The LLM coverage probe correctly labels evidence quality for the resolved side
    A single Qwen3-32B judge with hand-set weights decides which markets enter the eval set; no validation of judge agreement or accuracy is reported (§4.1).
  • domain assumption The benchmarked models' pretraining does not leak outcomes after t0
    Several models were released during or after portions of the 2024–2026 eval window, so their weights can contain post-t0 outcome information; §2 admits Hindcast only attacks the retrieval channel.
  • domain assumption Brier/accuracy over 216 balanced markets is a sufficient measure of forecasting skill
    Standard scoring assumption, but the eval set is selected on coverage rather than random, so aggregate numbers are not representative of all resolved markets.

pith-pipeline@v1.3.0-alltime-deepseek · 16306 in / 19346 out tokens · 176006 ms · 2026-08-02T02:55:36.657774+00:00 · methodology

0 comments
read the original abstract

Forecasters are evaluated by backtesting, which replays resolved questions and grades the probability the system would have assigned before the outcome was known. For LLMs, two channels leak the answer into this test. A model that retrieves can surface reports written after the event, turning forecasting into a lookup, and each new model is trained on data closer to the event, so a question that lay in the future for last year's models sits inside this year's training data. Either way, the test grades recall while claiming to grade foresight. We introduce Hindcast, which closes both leaks by grading a model as if it stood at a chosen past date $t_0$, before the outcome existed in either channel. Hindcast replays resolved Polymarket prediction markets against a frozen snapshot of public Reddit, lets the model read only posts written before $t_0$, and scores each forecast against both what happened and the market's own price at $t_0$, itself a human forecast made from the same past information. Because the cutoff is set per market and the snapshot never changes, the evaluation re-runs on new markets as models improve, without going stale. Once the leak is closed, retrieval still helps most models, but only where Reddit discussed the event beforehand. Where the archive carried only speculation, retrieval hurts.

Figures

Figures reproduced from arXiv: 2607.14051 by Ben Zhou, Evan Zhu, Jacob Dineen, Kevin Song, Shijie Lu, Xiao Ye.

Figure 1
Figure 1. Figure 1: HINDCAST on a resolved market (illustra￾tive). At a simulated query time t0 (a one-month look￾back here, pinned per market in our experiments, §4.2), the agent sees only Reddit content created before t0 and emits a forecast (■), scored against the market’s implied probability at the same t0 (•). Argentina’s group-stage loss to Saudi Arabia drops the market sharply, while the agent’s forecast tracks Reddit’… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of HINDCAST. Any resolved Polymarket market can be frozen at any past time t0 along its open-to-resolution lifecycle (top). At the chosen t0, the Reddit archive is restricted to documents created before t0, and a predictor (zero-shot baseline or retrieval-grounded agent) reads that frozen evidence and emits a probability pˆt0 . The forecast is scored against both the resolved outcome y (Brier, acc… view at source ↗
Figure 3
Figure 3. Figure 3: Topic distribution of the 216-market eval set. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The two predictor pipelines, side by side. The zero-shot baseline is a single LLM call. The Re￾trieval agent runs a bounded loop over the archive at t0, alternating queries to the archive with a short re￾flection that proposes the next query, then emits a final probability after up to N rounds. Inputs (q, t0) and the probability-emitting interface are shared, and only the body of the call differs. 5.2 Retr… view at source ↗
Figure 5
Figure 5. Figure 5: Brier vs. per-market cutoff fraction for the duration sweep. Each fraction places t0 at 25%, 50%, 75%, or 100% of an individual market’s open￾to-resolution span, so the per-market wall-clock cutoff scales with how long the market was actually open. The Retrieval agent (solid) lowers Brier over the zero-shot baseline (dashed) for both probed models, and the gap between them stays roughly constant across fra… view at source ↗
Figure 6
Figure 6. Figure 6: Per-topic net effect of the Retrieval agent. For each topic with at least four markets in the eval set, the bar shows the number of markets the agent recovers (agent right, zero-shot baseline wrong) minus the number it regresses (the reverse), summed across the eight models with per-run logs available. Blue bars right of zero are topics where retrieval helps on net, and red bars left of zero are topics whe… view at source ↗
Figure 7
Figure 7. Figure 7: Accuracy vs. per-market cutoff fraction for the duration sweep. Accuracy companion to the Brier plot in [PITH_FULL_IMAGE:figures/full_fig_p013_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

12 extracted references · 3 linked inside Pith

  1. [1]

    Form your own base-rate prior using world knowledge, the resolution criteria, and the timing/cutoff

  2. [2]

    Move away from the prior in proportion to that asymmetry; confident moves require asymmetric evidence, not absence of evidence

    Compare the strength and asymmetry of evidence on each side. Move away from the prior in proportion to that asymmetry; confident moves require asymmetric evidence, not absence of evidence

  3. [3]

    Return ONLY a single JSON object matching the requested schema

    Treat indirect/contextual signals as actionable when they consistently lean one way. Return ONLY a single JSON object matching the requested schema. Forecast user template (abridged).Step 1 sets a prior anchor based on a five-category classifi- cation of the question. Step 2 walks the model through tally-based updates from the evidence re- port. Step 3 em...

  4. [4]

    choice":

    MIRAI: evaluating LLM agents for event fore- casting.CoRR, abs/2407.01231. 10 Xiao Ye, Jacob Dineen, Zhaonan Li, Zhikun Xu, Weiyu Chen, Shijie Lu, Yuxi Huang, Ming Shen, Phu Tran, Ji-Eun Irene Yum, and 1 others. 2025a. Evaluating medical LLMs by levels of autonomy: A survey mov- ing from benchmarks to applications.arXiv preprint arXiv:2510.17764. Xiao Ye,...

    Pith/arXiv arXiv 2025

  5. [5]

    reasoning

    Broadly likely outcomes -> .70-.95 State this prior P(Yes) as the first sentence of "reasoning". Step 2 - ADJUST using the Reddit evidence: {report} 12 LLM call Temp. Max tokens Query generation 0.2 4096 Follow-up / reflection 0.3 4096 Evidence summarization 0.1 3500 Evidence tally 0.1 2200 Final forecast 0.1 3200 Forecast retry 0.1 1400/2400/900 Zero-sho...

  6. [8]

    Narrow numeric range -> .05-.20

  7. [9]

    Underdog / against-trend -> .10-.30

  8. [10]

    Structurally uncertain -> .40-.60

  9. [11]

    Favorite / with-trend -> .55-.85

  10. [2021]

    ForecastQA: A question answering challenge for event forecasting with temporal text data. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 4636–

    2021

  11. [2024]

    Pu Cheng, Juncheng Liu, and Yunshen Long

    Dated data: Tracing knowledge cutoffs in large language models.CoRR, abs/2403.12958. Pu Cheng, Juncheng Liu, and Yunshen Long. 2026. PolyBench: Benchmarking LLM forecasting and trading capabilities on live prediction market data. CoRR, abs/2604.14199. Sarkar Snigdha Sarathi Das, Palash Goyal, Mihir Par- mar, Nanyun Peng, Vishy Tirumalashetty, Chun- Liang ...

    Pith/arXiv arXiv 2026

  12. [4650]

    Ezra Karger, Houtan Bastani, Chen Yueh-Han, Zachary Jacobs, Danny Halawi, Fred Zhang, and Philip Tet- lock

    Association for Computational Linguistics. Ezra Karger, Houtan Bastani, Chen Yueh-Han, Zachary Jacobs, Danny Halawi, Fred Zhang, and Philip Tet- lock. 2025. ForecastBench: A dynamic benchmark of AI forecasting capabilities. InThe Thirteenth In- ternational Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenRe- view.net. N...

    Pith/arXiv arXiv 2025