Pith. sign in

REVIEW 4 major objections 9 references

An explicit trading layer—not better probability scores alone—is what turns prediction-market forecasts into positive returns.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 05:26 UTC pith:35SVHUT2

load-bearing objection Clean isolation of the trading layer from a frozen forecaster; the positive ROI is real on their archive but mostly comes from quarter-Kelly on a Raven-pre-filtered set, and the sample is small. the 4 major comments →

arxiv 2607.03015 v1 pith:35SVHUT2 submitted 2026-07-03 cs.AI

Beyond Forecasting: The Belief-to-Trade Layer in Prediction-Market Agents

Yishu Wang , Yuxuan Wang , Jiaqi Deng , Hanyang Tang This is my paper
classification cs.AI
keywords prediction marketstrading agentsKelly criterionforecastingrisk controlbelief-to-tradeLLM agentsposition sizing
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

AI forecasting work has improved probability estimates, yet live prediction-market benchmarks show that strong calibration still fails to produce profitable trades. This paper treats the step from belief to action as a designable object: a modular trading layer that selects, sizes, and risk-controls positions on top of any forecaster. Raven-Agent ranks candidates by a time-normalized expected return, sizes stakes with a quarter-Kelly rule, and enforces hard exposure, stop-loss, and drawdown limits outside the language-model prompt. On a controlled replay that freezes archived forecasts and prices, this layer is the only policy among five that earns a positive return on stake and a positive risk-adjusted return. The finding isolates trade execution as something that can be improved and evaluated independently of the forecaster itself.

Core claim

On a controlled replay over a fixed archive of forecasts and market prices, Raven-Agent’s trading layer is the only policy that achieves positive return on stake (+15.9%) and positive stake-weighted Sharpe (+0.42). Policies that trade every positive-edge forecast, size only by raw edge, or apply simple edge filters all lose money; pure edge-proportional sizing without selection collapses to −55.5% ROI.

What carries the argument

The belief-to-trade layer: rank candidates by the time-normalized monthly return score ρi = (pi/qi − 1) · 30 / max(Ti, 1), size surviving positions with one-quarter Kelly, then reject any order that violates deterministic stake, exposure, stop-loss, or drawdown constraints the language model cannot override.

Load-bearing premise

The archived decision set was produced by the authors’ own live agent and then filtered, so measured gains may partly reflect that original selection rather than the trading layer alone.

What would settle it

Re-run the same five policies on an independently collected archive of markets, prices, and probability estimates never filtered by Raven-Agent’s own candidate logic; if the full trading layer is no longer the only positive-ROI policy, the central claim fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Selection and risk filtering are necessary complements to sizing; sizing by edge alone without selection amplifies losses.
  • Forecasting quality and trade execution can be ablated and improved independently by holding one side fixed.
  • Hard risk constraints outside the prompt produce auditable rejections that prompt-level guidance does not reliably enforce.
  • Any stronger forecaster can be swapped into the same layer without redesigning selection or risk modules.
  • Replay protocols that freeze archived forecasts make the trading policy itself the unit of evaluation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the same layer is paired with a substantially better-calibrated forecaster, positive ROI should widen rather than reverse, because low-quality edges are already discarded before sizing.
  • Time-normalized ranking implies a general rule for any market with uneven capital lockup: raw-edge rankings systematically over-allocate to long-horizon bets.
  • Deterministic external risk modules may transfer across agent architectures more cleanly than learned risk policies that must be retrained when the forecaster changes.
  • Standardized public archives of market snapshots and probability estimates would let competing trading layers be compared without each team’s own selection bias.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 0 minor

Summary. The paper argues that prediction-market evaluation of AI forecasters has under-specified the trading layer (selection, sizing, and risk control), and that this layer should be treated as an explicit, deterministic, composable module rather than a fixed protocol or prompt instruction. It introduces Raven-Agent, with ranking by a time-normalized return rate ρi (Eq. 2), quarter-Kelly sizing (Eq. 3), and hard risk services outside the LLM. On a controlled replay that freezes archived (p, q, T) and varies only the policy, Raven-Agent (full) is the only of five policies with positive ROI (+15.9%) and positive stake-weighted Sharpe (+0.42) on 42 executable trades (Table 1); edge-proportional sizing without selection collapses to −55.5%. Code and redacted artifacts are released.

Significance. If the isolation claim holds, the work usefully reframes a documented gap between calibrated probabilities and trading returns as a design problem rather than a pure forecasting failure. The modular split (swappable forecaster + deterministic trading services), the replay protocol that holds forecasts fixed, the public code release, and the bootstrap/leave-one-out checks are concrete strengths that other prediction-market agent papers can reuse. The architectural contrast with end-to-end LLM traders (e.g., Prediction Arena) is also of practical interest. Significance is currently limited by sample size and by the fact that the decision archive is produced by the same system under test; those issues constrain how far the positive ROI can be generalized, but they do not erase the value of the framing or the open infrastructure.

major comments (4)
  1. §4.1 and Appendix C: the 44-row executable set is drawn from Raven-Agent’s own live deployment (59 open decisions → 44 after liquidity/resolution/overlap filters). All policies are then compared on that Raven-pre-filtered pool. This undercuts the claim that outcome differences are attributable solely to the trading layer on an independent decision set. At minimum the paper should (i) quantify how much of the candidate pool already embeds ρi ranking and Raven filters, (ii) report a parallel ablation on an unfiltered or externally sourced candidate list, or (iii) reframe the result as “re-allocation on Raven-selected trades” rather than a general trading-layer win.
  2. Table 1 and Eq. (3): Raven-Agent (fixed) already applies the full selection/risk stack at flat $10 stakes and still loses (−4.7% ROI, Sw −0.09). The sign flip to +15.9% occurs only when quarter-Kelly raises total stake from $420 to $820 on the same 42 trades. Edge-proportional is capital-normalized to Forecast-only’s $580, but Raven (full) is not. Because ROI is stake-normalized this is not fatal, yet the central “only positive return” claim is driven by unequal, ρi-correlated stake concentration on a self-selected set. A capital-matched or bankroll-path version of Raven (full), and an ablation that sizes the filtered set with edge-proportional or fixed stakes only, are needed to separate selection from sizing.
  3. Table 1 vs Table 5 (Appendix E): on the main archive, positive ROI and Sw rest on N=42 with Kelly sizing; the fused 132-row fixed-stake replay yields only +5.9% ROI with bootstrap CI [−4.5%, +16.1%] that includes zero, and Sw CI that also crosses zero. The paper’s headline (“only positive return and risk-adjusted return”) therefore depends on a small, self-generated sample plus a free Kelly fraction. Either enlarge the independent archive, pre-register the free parameters (Kelly fraction, top-K, ρi threshold, risk caps), or qualify the claim to match the fused-archive uncertainty.
  4. Abstract / §1 claim of “the first autonomous trading agent for prediction markets” sits uneasily with concurrent end-to-end agents in Prediction Arena, PolyBench, and related work the paper itself cites (§B). The architectural distinction (deterministic trading layer vs LLM-issued orders) is real and should be the novelty claim; the absolute “first” phrasing should be dropped or tightly scoped so the contribution is not overstated relative to concurrent systems.

Circularity Check

0 steps flagged

Empirical policy ablation with fixed forecasts; no derivation reduces to its inputs by construction.

full rationale

The paper is an empirical systems paper, not a first-principles derivation. It holds archived probabilities, prices, and timestamps fixed and varies only the trading policy (selection by ρi, quarter-Kelly sizing, deterministic risk filters). ROI and stake-weighted Sharpe are computed from realized or mark-to-market payoffs (Eq. 1) under those policies; nothing forces the sign of Raven-Agent (full) by algebraic identity. ρi (Eq. 2) and the quarter-Kelly stake (Eq. 3) are textbook formulas applied to the same inputs that every baseline receives; they are not redefined so that the reported +15.9 % ROI equals a fitted constant. The archive originates from the authors’ own live deployment, which raises a selection-bias soundness concern, but that concern does not make the ablation circular: Raven-Agent (fixed) still loses money on the same filtered set, and the sign flip appears only after sizing is added. No uniqueness theorem, self-citation chain, or ansatz is load-bearing for the central claim. The result is therefore self-contained against the stated replay protocol and scores 0 for circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

The central empirical claim rests on standard Kelly theory, a hand-chosen fractional multiplier, a hand-chosen ranking formula and top-K, and several hard risk thresholds. No new physical entities are postulated; the invented objects are architectural modules and the ρi score. Free parameters are few but load-bearing for the reported ROI numbers.

free parameters (4)
  • Kelly fraction = 1/4
    Fixed at 1/4 of full Kelly (Eq. 3) as a practitioner correction for noisy pi; not derived from data in the paper but chosen by hand and directly scales all stakes and therefore ROI.
  • top-K retained candidates = 4
    K=4 after ranking by ρi; controls portfolio breadth and is not optimized or justified beyond a design choice.
  • edge / ρi selection threshold
    Candidates below an (unspecified numeric) ρi threshold are dropped; the threshold is a free filter that changes the traded subset and Selected Brier.
  • risk caps (per-trade notional, aggregate exposure, per-event, stop-loss fraction, drawdown halt)
    Five hard constraints in the execution module; their numeric values are design choices that determine which proposals survive and therefore the reported PnL.
axioms (5)
  • domain assumption The Kelly criterion (Kelly 1956) gives the stake that maximizes long-run log wealth under a known win probability; fractional Kelly is the appropriate correction when pi is estimated.
    Invoked in §3.3 to justify Eq. 3; standard in betting theory but assumes the model probability is the true win probability up to noise.
  • ad hoc to paper Time-normalized score ρi = (pi/qi − 1) · 30 / max(Ti, 1) is a valid ranking device for capital locked until resolution.
    Defined in Eq. 2 (§2.1) and used as the primary selection score; presented as a ranking heuristic, not a claim of exact monthly compounding.
  • domain assumption Binary prediction-market contracts pay $1 on YES and $0 otherwise; profit is Πi = si (zi/qi − 1) (or mark-to-market).
    Problem setup §2.1; standard market microstructure assumption for the evaluation.
  • domain assumption Deterministic risk constraints enforced outside the LLM prompt remain effective even when the model is confidently wrong; prompt-level risk guidance is unreliable.
    Motivated by concurrent evidence from Zhang et al. 2026 and Jang et al. 2025 (§3.4, Related Work); load-bearing for the architectural claim.
  • ad hoc to paper Holding all positions to resolution (or fixed evaluation-date mark-to-market) is a fair evaluation of the trading layer.
    Stated in §2.2; dynamic exit is deferred, so reported ROI ignores intermediate exit skill.
invented entities (2)
  • Raven-Agent modular trading layer (ranking, sizing, execution/risk services) independent evidence
    purpose: Make selection, stake sizing, and risk control explicit, deterministic, and composable with any forecaster.
    Core architectural contribution; no independent existence outside the paper’s system, but falsifiable via the public replay and live profile.
  • Time-normalized return rate ρi no independent evidence
    purpose: Rank candidates so that short-horizon edges are preferred when raw edges are similar.
    Defined in Eq. 2; a paper-specific ranking score rather than a previously standard metric in the cited benchmarks.

pith-pipeline@v1.1.0-grok45 · 14775 in / 3649 out tokens · 35545 ms · 2026-07-12T05:26:18.709478+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Beyond Forecasting: The Belief-to-Trade Layer in Prediction-Market Agents." pith.science (2026). https://pith.science/paper/35SVHUT2

@misc{pith2026260703015,
  author       = {Pith},
  title        = {Pith review of: Beyond Forecasting: The Belief-to-Trade Layer in Prediction-Market Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/35SVHUT2}},
  note         = {Machine review of arXiv:2607.03015}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Forecasting future events has attracted growing attention as a testbed for general-purpose AI. A natural way to ground this evaluation is let the models trade in the prediction markets. Trading, however, requires more than forecasting. Moreover, recent benchmarks report a substantial gap between calibrated probability scores and the trading results. We propose Raven-Agent, to the best of our knowledge, the first autonomous trading agent for prediction markets. On a controlled replay over an archived decision set, our architecture achieves the only positive return and the only positive risk-adjusted return among all tested policies. We have released our code in https://github.com/Alchemist-X/predict-raven .

Figures

Figures reproduced from arXiv: 2607.03015 by Hanyang Tang, Jiaqi Deng, Yishu Wang, Yuxuan Wang.

Figure 1
Figure 1. Figure 1: Raven-Agent’s live Polymarket profile and all-time profit and loss curve. Across 20 predictions, the agent currently holds $251.01 in open positions and has accumulated $53.92 in cumulative profit. (a) Current open positions: seven NO-side contracts spanning politics, commodities, sports, entertainment, and public health. (b) Recent trade history: a mix of NO-side entries (crude oil, Eurovision, F1, NBA, V… view at source ↗
Figure 2
Figure 2. Figure 2: Polymarket dashboard snapshots of Raven-Agent’s open positions and recent trade history during live deployment. whose trading layer handles selection, sizing, and risk con￾trol on top of a swappable forecaster. On a controlled replay over a fixed forecast archive, Raven-Agent achieved the only positive return and risk-adjusted return among five policies. The edge-proportional baseline, which sizes by edge … view at source ↗
Figure 3
Figure 3. Figure 3: Example Raven-Agent market screening trace. The artifact records the daily scan, filtering rules, representative rejected candidates, and final selected markets from one archived market pulse. 9 [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Example Raven-Agent probability reasoning trace for a selected crude oil market. The artifact records the market-implied probability, Raven-Agent probability estimate, evidence adjustments, settlement-rule checks, and proposed action. 10 [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

9 extracted references · 2 linked inside Pith

  1. [1]

    arXiv preprint arXiv:2409.19839 , year=

    ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities , author=. arXiv preprint arXiv:2409.19839 , year=

    Pith/arXiv arXiv

  2. [2]

    2025 , url=

    Yang, Qingchuan and Mahns, Simon and Li, Sida and Gu, Anri and Wu, Jibang and Xu, Haifeng , journal=. 2025 , url=

    2025

  3. [3]

    FutureX: An Advanced Live Benchmark for

    Zeng, Zhiyuan and others , journal=. FutureX: An Advanced Live Benchmark for. 2025 , url=

    2025

  4. [4]

    The Losing Winner: An

    Jang, Youwon and Kim, Joochan and Zhang, Byoung-Tak , journal=. The Losing Winner: An. 2025 , url=

    2025

  5. [5]

    Prediction Arena: Benchmarking

    Zhang, Jaden and Liu, Gardenia and Johansson, Oliver and Yitayew, Hileamlak and Ohly, Kamryn and Li, Grace , journal=. Prediction Arena: Benchmarking. 2026 , url=

    2026

  6. [6]

    PolyBench: Benchmarking

    Cheng, Pu and Liu, Juncheng and Long, Yunshen , journal=. PolyBench: Benchmarking. 2026 , url=

    2026

  7. [7]

    arXiv preprint arXiv:2604.18576 , year=

    Agentic Forecasting using Sequential Bayesian Updating of Linguistic Beliefs , author=. arXiv preprint arXiv:2604.18576 , year=

    Pith/arXiv arXiv

  8. [8]

    Monthly Weather Review , volume=

    Verification of Forecasts Expressed in Terms of Probability , author=. Monthly Weather Review , volume=. 1950 , doi=

    1950

  9. [9]

    The Bell System Technical Journal , volume=

    A New Interpretation of Information Rate , author=. The Bell System Technical Journal , volume=. 1956 , doi=

    1956