{"id":"dd6af24f-1386-4136-8d9b-df4f3bcbd72d","arxiv_id":"2607.16229","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"FinBench is a time-gated benchmark specification for measuring whether financial LLMs report calibrated probabilities and intervals, with a tiny pilot run.","lead":"This paper introduces FinBench, a test for whether financial AI models know when they are unsure, by scoring their probabilities and prediction intervals under strict time limits on data. It matters because an overconfident model can lose money under standard bet sizing, and current benchmarks mostly ignore calibration.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pretraining contamination, not input timestamps, threatens the leak-safety claim of §3.","rationale":"The paper's central contribution is a leak-safe calibration benchmark. The §3 time-gating protocol is a necessary but not sufficient condition for leak-freedom: LLMs have parametric memory, so a fixed historical evaluation date can be memorized. The reader correctly identifies this as the weakest assumption. I agree, and this is the single most load-bearing concern because if it lands, the primary claim—strict time-gating avoids look-ahead—fails, and the benchmark cannot distinguish genuine forecasting skill from memorized outcomes. The pilot's small N and unspecified baselines are secondary and easier to fix. The proposed leakage audit is a concrete, minimal addition that would settle the concern. Since the reader already conditioned the verdict on this issue, no verdict adjustment is needed.","tokens_in":5336,"tokens_out":5344,"duration_ms":56968,"concrete_test":"As a leakage audit, run a fixed probe: for each (ticker, date) in the benchmark, generate two versions of the model prompt—one with the true date and one with a different date (e.g., 2024-06-17) but identical price/news context. If the model's forecast probabilities or intervals shift materially, that indicates it is using date-specific memorization rather than the provided inputs. To settle whether the time-gating claim holds for the published pilot, do this for all 33 rows on 2025-01-15; compare the distribution of absolute changes in ˆp and interval midpoints. Additionally, verify whether each evaluated model's training cutoff precedes 2025-01-15; if any does not, flag its scores as potentially contaminated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that FinBench is 'strictly time-gated to avoid look-ahead bias.' The gating protocol in §3 only filters input data with timestamps > T; it does nothing about information stored in the model's parameters during pretraining. Because the evaluation date is a fixed, historical date (2025-01-15), any LLM whose training data includes that date's prices or news can answer with memorized, post-T information without any timestamped input. This is not a hypothetical edge case: the benchmark is explicitly for LLM-based agents, and future models will have training cutoffs after the benchmark's static dates. The paper's own limitations (§8) list plans for broader tickers and stress slices but omit any contamination audit, and §3's 'blind to any data with timestamps later than T' is simply false for parametric memory. If any of the evaluated models has seen 2025-01-15 in training, the Brier score differences in Table 1 could reflect memorization rather than calibration skill, undermining the benchmark's validity as a measure of forecasting ability. The central leak-safety guarantee therefore rests on an assumption—training data cutoff before evaluation dates—that is neither stated nor tested.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FinBench, a benchmark for evaluating calibration and uncertainty quality of LLM-based financial forecasters. The benchmark defines a structured output schema (probability of positive return and an 80% prediction interval for log return) and evaluates forecasts with the Brier and Winkler scores. The central design claim is that it is 'strictly time-gated' to prevent look-ahead bias, meaning models only receive data with timestamps at or before the forecast time. The paper also reports a small pilot run (one trading day, three tickers, 33 forecasts) testing six LLMs, explicitly describing the pilot as a descriptive sanity check rather than a statistically conclusive leaderboard. The scoring-rule definitions are standard and correctly stated, and the paper is appropriately modest about the pilot's scope.","tokens_in":5575,"tokens_out":4500,"duration_ms":49908,"significance":"The motivation for FinBench is legitimate and timely: as LLMs become components of autonomous trading agents, miscalibrated confidence can be more harmful than simple inaccuracy. The choice of strictly proper scoring rules (Brier, Winkler) is methodologically sound for evaluating probabilistic and interval forecasts. The paper's explicit framing of the pilot as descriptive is commendable. However, the benchmark's central leak-safety guarantee is currently incomplete, and the absence of artifacts and exact baseline definitions prevents independent verification. If the contamination issue is addressed and the artifacts are provided, the benchmark could be a useful evaluation substrate for agentic financial forecasting. As it stands, the paper's claims outrun what the submitted materials support.","major_comments":[{"comment":"The claim that FinBench is 'strictly time-gated to avoid look-ahead bias' is not valid for LLMs as stated. The protocol in §3 filters inputs by timestamp, but it does nothing to prevent a model from using memorized information from pretraining. Because the evaluation date is fixed at 2025-01-15, any LLM whose pretraining data includes that date's market outcomes could produce forecasts informed by post-T information without any timestamped input. This is not a hypothetical concern for a benchmark explicitly aimed at LLM-based agents; future models will have training cutoffs after the static evaluation date. The paper's limitations (§8) mentions expanding the ticker universe and publishing baselines, but omits any contamination audit or a statement on model training cutoffs. The phrase 'blind to any data with timestamps later than T' (abstract and §3) is therefore overstated. The authors","section":"§3 and §8: Time-Gating Protocol and Limitations"},{"comment":"The baseline constructions used to compute Brier Skill Scores are not fully specified. §4.3 describes the base rate as 'a rolling mean of recent outcomes (e.g., 60-day rolling mean of Y)' and the momentum heuristic as 'a conditional-probability rule based on recent short-horizon returns (e.g., sign of the last 30 minutes),' but no exact formulas, lookback windows, data sources, or handling of non-trading hours are given. The reported BSS values in Table 1 depend in detail on these baselines; with different parameter choices, the skill scores could change materially. The paper itself acknowledges in §8 that baseline standardization is future work, but for a benchmark specification this is part of the core definition. Furthermore, the 'pilot CSV' mentioned in §4.3 and §6.1 is not provided, so even the coin-flip BSS values cannot be independently computed from per-row data. Please include t","section":"§4.3 and §6.1: Baseline Definitions"},{"comment":"The manuscript's Reproducibility section states that 'a reference implementation that regenerates summary tables and figures from the stored run artifacts will be released alongside an expanded benchmark release,' but no data, code, or prompts accompany the current submission. For a benchmark paper, the claim that FinBench is 'built to support reproducible and auditable evaluation' requires the artifacts to be available at submission time, at least for the pilot. Without the raw model outputs (probabilities, intervals, rationales) and the realized returns, the pilot results in Tables 1–3 are not verifiable. This is a load-bearing issue for a paper whose contribution is a benchmark, even if the pilot is framed as a sanity check. I ask that the pilot dataset and the exact scoring script be included as supplementary material.","section":"Reproducibility Section"}],"minor_comments":[{"comment":"Typo: 'agenticsystems' should be 'agentic systems'.","section":"Abstract"},{"comment":"The 'composite score' is reported in Table 1 and Figure 1 but never defined. If it is not a primary metric, either define it precisely or omit it from the displays.","section":"§4.3 and Table 1"},{"comment":"Notation table includes P_mid = (P_high + P_low)/2, but this quantity is not used anywhere in the text. If it is part of the benchmark specification, explain where it is used; otherwise remove it.","section":"Appendix B"},{"comment":"The text says 'two rows (MSFT MORNING) show negative skill vs. baselines for Llama 3.1 70B and DeepSeek-V3,' but Table 1 only includes the 'BSS coin' column. Clarify whether the skill scores referenced are vs. the base-rate and momentum baselines, and show them in the table or the CSV.","section":"§6.2"},{"comment":"The target price is defined as the last mid-price in the window [15:54:30, 15:55:00] ET. The choice of this specific 30-second window is not justified; include a brief rationale.","section":"§2.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is well-written and avoids overclaiming the pilot results, which is appreciated. The main concern is that the central leak-safety guarantee as stated is incomplete for LLMs, and the missing artifacts and baseline definitions make the reported numbers unverifiable. These are fixable within the scope of the paper: the authors can add a contamination audit or reframe the time-gating claim, and can release the pilot data and exact baselines. I do not see a need for rejection, but the current version needs substantive revision before it supports the claims made for FinBench."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper is a benchmark specification, not a results paper. The new thing is the package: strictly time-gated inputs, strictly proper Brier/Winkler scoring, and an agentic-finance forecasting target. The components are standard, but the combination isn't in Pixiu or FinGPT. That's a real, modest contribution.\n\nThe scoring formulas are correct, the pilot is honestly labeled descriptive, and the limitations section admits the pilot can't support model comparisons. The authors know what they're building.\n\nThe load-bearing problem is in §3. The gating protocol filters inputs by timestamp, but the paper claims the model is \"blind\" to data later than T. For an LLM, that's false: anything in the weights from pretraining is available. Because the evaluation date is fixed at 2025-01-15, any model trained after that can leak the answer without a single timestamped input. The paper's limitations list future work but never mention a contamination audit. That's not a minor omission; it undercuts the central leak-safety guarantee as stated. The fix is straightforward—rolling evaluation dates, check training cutoffs, or test memorization—but the current text overstates the guarantee.\n\nOther issues are smaller: no pilot data or code released, so the sanity check can't be verified; base-rate and momentum baselines are described only as examples, not specified; and the pilot N=33 is too small for anything but a pipeline check, which the paper itself says.\n\nIf the authors fix the contamination issue and release the artifacts, FinBench could be a useful evaluation harness for risk managers and LLM calibrators. As written, it's a decent proposal with a hole in its main claim. I'd send it to peer review because the idea is worth engaging and the hole is fixable. But don't cite it yet.","headline":"FinBench is a sensible benchmark spec with standard but correctly combined scoring metrics, yet its central leak-safety claim ignores pretraining contamination.","tokens_in":6045,"tokens_out":1887,"would_cite":false,"duration_ms":19599,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A leak-safe benchmark can measure whether AI market forecasts are honest about confidence.","keywords":["calibration","uncertainty quantification","financial forecasting","LLM agents","Brier score","Winkler score","time-gating","look-ahead bias"],"falsifier":"Run a FinBench task on a date outside a model's pretraining horizon, then randomize the realized direction label while keeping the time-gated inputs unchanged; if the model's Brier skill score remains significantly positive, the score is driven by memorized outcomes rather than the supplied inputs, refuting the leak-safe claim.","tokens_in":5230,"feed_emoji":"🎯","tokens_out":6193,"duration_ms":66549,"temperature":0.7,"pith_summary":"The paper introduces FinBench, a benchmark specification for evaluating whether large-language-model agents in finance report calibrated probabilities and intervals rather than merely accurate point forecasts. Its central design is a strict time-gating protocol that admits only information stamped at or before the forecast time, combined with strictly proper scoring rules (Brier for probabilities, Winkler for prediction intervals) that numerically penalize overconfident or hollow confidence. The authors argue that in agentic finance, uncertainty is the output: a model that is only slightly above chance but consistently overconfident will destroy capital under standard bet-sizing rules. To sanity-check the pipeline, they run a 33-forecast pilot across three liquid tickers and three intraday horizons; the pilot is explicitly descriptive, but it shows calibration-sensitive metrics separating models that pure accuracy does not.","feed_headline":"Rate AI market forecasts on confidence, not accuracy alone","feed_subtitle":"Time-gated Brier and Winkler scores expose LLMs that overstate their edge, key to safe risk sizing.","key_machinery":"The central object is the paired time-gating and proper-scoring evaluation loop. A timestamp filter admits only OHLCV bars, headlines, and derived signals with timestamps at or before the forecast time, eliminating end-of-day leakage. Each forecast must specify a probability of positive return and an 80% interval; these are scored with the Brier score and Winkler interval score, with skill scores against coin-flip, base-rate, and momentum baselines. The loop has two effects: it structurally blocks look-ahead bias, and it makes inflated confidence costly, so the ranking reflects honest uncertainty.","core_discovery":"FinBench's central claim is that probabilistic calibration and interval quality of agentic financial forecasts are measurable in a leak-safe way. The benchmark forces models to output a probability of positive return and an 80% prediction interval, then scores these with the Brier and Winkler rules—strictly proper scoring rules that penalize overconfident or uninformative forecasts. The pilot run on one day and three tickers is explicitly descriptive, but it demonstrates that the pipeline produces sensible calibration and interval scores and that accuracy and calibration rank differently.","pith_inferences":["A natural stress test is to compare a model's FinBench score on dates inside vs. outside its training cutoff; if scores match, pretraining memorization is unlikely, but if not, timestamp gating alone cannot guarantee clean evaluation.","The same calibration-first protocol could transfer to other non-stationary, high-stakes forecasting contexts, such as clinical risk scores or energy load forecasting.","With enough evaluation days, Brier/Winkler skill scores could be mapped to expected capital growth under position-sizing rules, making calibration a capital-weighted performance metric.","The paper's confidence–competence argument implies that improved accuracy without improved calibration can be harmful; a future extension might define a 'safe confidence' threshold for deployment."],"forward_implications":["Any FinBench run gives an apples-to-apples, leakage-aware comparison of LLMs on calibration and interval quality.","Models that report inflated probabilities or over-tight intervals will rank lower, aligning leaderboard incentives with safe deployment.","Skill scores vs. baselines give a floor: a model must beat a coin flip in a proper-scoring sense to claim genuine forecasting signal.","The protocol scales to more tickers and horizons without changing the evaluation engine."],"fun_headline_variants":["FinBench: punishing AI overconfidence in market forecasts","New benchmark flags overconfident AI market forecasters","To trust AI in finance, check its calibration, not just accuracy","Time-gated benchmark scores AI forecast confidence, not just points","Why AI's market confidence is risky without calibration checks"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The benchmark's leak-safety depends on timestamp filtering being sufficient to exclude future information; it does not address pretraining contamination, in which the LLM may have memorized the historical outcome of the forecast date.","fun_headline_variants_meta":{"raw":{"variants":["FinBench: punishing AI overconfidence in market forecasts","New benchmark flags overconfident AI market forecasters","To trust AI in finance, check its calibration, not just accuracy","Time-gated benchmark scores AI forecast confidence, not just points","Why AI's market confidence is risky without calibration checks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00027,"raw_usage":{"total_tokens":1464,"prompt_tokens":749,"completion_tokens":715,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":493,"completion_tokens_details":{"reasoning_tokens":634}},"tokens_in":493,"tokens_out":715,"duration_ms":7079,"temperature":1.0,"reasoning_tokens":634,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T10:13:14.753465+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a FinBench task on a date outside a model's pretraining horizon, then randomize the realized direction label while keeping the time-gated inputs unchanged; if the model's Brier skill score remains significantly positive, the score is driven by memorized outcomes rather than the supplied inputs, refuting the leak-safe claim.","supporting_citations":[],"review_version":1}