{"id":"96b1ac12-8254-4890-91ca-00f357e86913","arxiv_id":"2611.10414","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"LLM judgments of web news track the market's reaction to the reported event, but the association is largely driven by news that is published after the price move, so usefulness requires a publication-aligned criterion.","lead":"This paper builds a multi-agent trading system to test whether LLM judgments of web news track the market's reaction to the reported event or merely follow the price move. It finds that the association between judgments and returns mostly reflects news published after the price move, and proposes that evaluation should use a publication-aligned criterion instead of an announcement window.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"No significant objection identified.","rationale":"The paper's central claim (that LLM item judgments track announcement-window abnormal returns) is well-supported by a pre-registered confirmatory test, post-release estimates, blinding probes, and agreement with 8-K filings. The reader identified the trust partition (Assumption A.1) as the weakest assumption. While that assumption is important for the influence bound, it is not load-bearing for the core validity claim; the bound is a separate contribution. The trust partition is backed by single-writer stores, immutable records, taint tracking, property tests, and worst-case checks. The main qualification—that the publication-aligned correlation is very small (0.023)—is already prominently reported as an exploratory finding, and the paper explicitly recommends reporting both criteria. The reader's CONDITIONAL verdict is appropriate, and no additional concern warrants moving to REJECT or ACCEPT without revision. The agreement is 'partial' because the reader's weakest assumption (trust partition) is not the most critical limitation for the central claim; rather, the interpretation of the tracking result given the small publication-aligned association is more central, and the paper already addresses this honestly.","tokens_in":26235,"tokens_out":4320,"duration_ms":42904,"concrete_test":"Replicate the confirmatory H2 statistic (weekly Spearman correlation of E[effect] with AR[0,+1], Newey–West standard errors with 4 lags) on an independent sample: either a later time period (e.g., weeks 116–200) or a different universe (e.g., S&P 400 mid-caps) using the same pinned LLM and question bank. If the correlation is not significantly positive, the generalizability of the central claim would be limited. If it is positive and of comparable magnitude, the central claim is strengthened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper is methodologically rigorous and transparent. The central claim—that LLM judgments of Web news track the market's reaction to the reported event—is supported by the confirmatory test (weekly Spearman 0.119, 95% CI [0.098, 0.140], Holm-adjusted p<10⁻⁶), and the association holds after the model's release (0.115) and under multiple sensitivity checks. The exploratory analyses honestly reveal that the association comes from returns realized before the item appears: the publication-aligned correlation is only 0.023 [0.005, 0.041]. The confirmatory test of usefulness (H3) detects no rank information at the weekly horizon (−0.016 [−0.029, −0.002]). The paper's own proposed reporting standard (publication-aligned criterion) addresses this. The trust partition assumption (A.1) is load-bearing for the influence bound but is implemented with single-writer immutable stores, tested with property tests and worst-case checks (0 violations in 8,912 checks), and the bound is not central to the main validity claim. All acknowledged limitations are clearly stated. No significant unresolved objection remains.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper presents a multi-agent trading system designed to test whether LLM judgments of Web news about listed firms track the market's reaction to the reported event or follow it. The system features point-in-time stores, typed question programs, a trust-partitioned learner with a proved influence bound, and a hash-committed pre-registered evaluation protocol. The confirmatory test (H2) finds that item-level judgments from a pinned LLM (gpt-4.1-mini) correlate with announcement-window abnormal returns at a weekly Spearman of 0.119 (95% CI [0.098, 0.140]), and the association persists after the model's release (0.115). Event-type judgments agree with 8-K filings (AUC 0.833 for earnings). However, exploratory analyses show that the association is driven by returns realized before the item appears: the publication-aligned correlation is only 0.023 [0.005, 0.041]. The confirmatory test of usefulness (H3) detects no rank information over code-only features at the weekly horizon (−0.016 [−0.029, −0.002]). Attack experiments show that all attempts stay within the influence bound and that decision cards trace 88% of successful attacks to the planted item. The paper concludes that LLM judgments of Web news tend to follow rather than track the market and proposes that market-graded tests report the publication-aligned criterion alongside the announcement-window criterion.","tokens_in":26495,"tokens_out":3245,"duration_ms":27083,"significance":"If the findings hold, the paper makes several important contributions. First, it provides a rigorous, pre-registered evaluation framework that separates event-level criterion validity from publication-aligned usefulness—a distinction that many prior studies conflate. Second, it delivers concrete empirical evidence that the apparent 'tracking' ability of LLMs on headline news is largely an artifact of the announcement window including returns that precede the item; the true publication-aligned signal is small and economically insignificant at a weekly horizon. Third, the multi-agent system design—with its typed question interface, single-writer stores, trust-partitioned learning, and formal influence bound—is a methodological contribution that others can adopt or extend. Fourth, the paper is unusually transparent: it reports all limitations, exploratory results (including negative ones), and checks for look-ahead, price-reporting items, and re-identification. The proposed reporting standard (publication-aligned criterion) is actionable and well-motivated. The paper does not overclaim; it honestly labels its confirmatory, registered, exploratory, and descriptive estimates.","major_comments":[{"comment":"The influence bound (Proposition A.2) rests on Assumption A.1, which states that the trusted logit, trusted ranks, bound parameters, reference-path quantities, and exclusion anchors are functions of trusted inputs only. The paper argues that single-writer immutable stores enforce this, and it provides property tests and worst-case checks (0 violations in 8,912 checks). However, the assumption implicitly relies on the modeler's choice of which features are designated as 'trusted' versus 'untrusted'. A future attacker who can influence price data (e.g., via market manipulation) could indirectly affect the trusted logit, which would violate the assumption. The paper notes this limitation in the bound's tightness discussion but does not discuss the possibility of attacks on the trusted inputs themselves. Given that the bound is not central to the main validity claim (H2, H3), this is a minor limitation. Still, the paper should acknowledge this boundary condition explicitly.","section":"§3.5, Assumption A.1 (Trust partition), and Appendix A","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The date-only probe D1 tests whether the model's re-identification of the date exceeds chance. The registered test uses a start rule that reads the window centred on July 2024, and the paper reports that the AUC over the evaluation window is 0.488 (p=0.896). However, in an exploratory monthly view, 13 of 14 pre-release windows lie above 0.5, and two windows between the cutoff and the release (2024-11 to 2025-02) show p<0.05 before Holm correction. The paper attributes this to 'training data past the cutoff, a name prior, or chance'. While this does not undermine the post-release estimate (0.115), the evidence of potential look-ahead in the pre-release period is suggestive. The paper should discuss whether the buffer period of four decisions is sufficient given that some windows before the release differ from chance. The current treatment—acknowledging the pre-release deviations but not examining their impact on the confirmatory window—seems slightly dismissive.","section":"§4.3, Registered probe D1 (Figure A1)","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The phrase 'code sets every position under a proved bound on how far text can move it' could be misinterpreted as a bound on any trading system. Clarify that the bound holds for this system's decomposition and relies on Assumption A.1 (trust partition).","section":"Abstract, line 9","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The abbreviation 'NG' appears without explanation in the table (defined later in §3.1 as 'this-that-model-1.0'). Consider adding a brief footnote or parenthetical in the table caption.","section":"Table 1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The sentence 'Screening removes the opinion, promotion and instructions that Web text mixes with reports before any judgment is asked' is slightly unclear. The word 'before' might be read as a time ordering; consider rephrasing to 'before any judgment is elicited' or similar.","section":"§3.3, paragraph on screening","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The column 'Reports move (%)' is 7.1 for all items, but the exploratory audit suggests the true share is near 10.6%. The paper already notes this undercount in Appendix C. Consider adding a parenthetical note in the table or footnote to avoid confusion.","section":"Table 3, row 'All frame items'","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The entry 'ASR (%) 10.5' and the subsequent Δ column are presented with a reference to 'the 7 secondary contrasts'. The reader might benefit from a brief reminder of which contrasts these are (A1–A7 pooled and the three reader variants).","section":"§4.5, Table 5B, row 'Per-item reader'","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper references the typed-model checklist [40] and LLM trading reporting items [43, 47]. It might also mention the broader crypto/AI evaluation guidelines from [8] if applicable, but this is not required.","section":"Appendix D, Reporting Checklists","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The panel labels (a), (b), (c) are in the caption but not on the figure itself. Please add panel letters directly to the figure for clarity.","section":"Figure 2 caption","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"minor_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"Colleague,\n\nThis is a well-executed paper that makes a real methodological contribution, but its central empirical claim is more subtle than the headline suggests. The paper asks whether LLM judgments of web news 'track' the market's reaction to the reported event or 'follow' it (i.e., merely report a price move that already happened). The confirmatory test finds a Spearman correlation of 0.119 with announcement-window returns, and this is robust to many controls. But the exploratory analysis shows that the publication-aligned association—measured from the first trade after the news item appears—is only 0.023, and the confirmatory test of trading usefulness finds nothing (−0.016 IC). So the interpretation is: LLM judgments do correlate with the market's reaction, but almost entirely because the news items themselves report that reaction. The judgments don't add predictive power beyond code-only features at a weekly horizon.\n\nWhat the paper does well is considerable. The system design—point-in-time stores, typed questions, a proved influence bound—is careful and transparent. The pre-registration is thorough, with many controls for look-ahead (post-release estimates, blinding probes, placebo windows). The exploratory content analysis of news timing and authorship cleanly explains where the association comes from. The paper is honest about its limits and proposes a sensible reporting standard (publication-aligned criterion).\n\nThe soft spots are mostly those the paper acknowledges. The main empirical finding, while real, is not practically useful for trading (and the paper says so). Generalizability is limited to one LLM, one news feed, 100 large caps, and a weekly horizon. The trust-partition assumption for the bound is plausible but not empirically verified under adversarial pressure; the bound itself is a design feature but its practical tightness depends on chosen parameters (mu, K, J). The H1 result (attack comparison) went the wrong way, which the paper handles honestly.\n\nThis paper should go to peer review. It is a serious piece of work. The revisions needed are minor: make the publication-aligned criterion and the null usefulness result more prominent in the abstract and conclusion, so readers don't take the headline correlation at face value. The real contribution is in the evaluation methodology and the proposed standard, not in the specific correlation number.","headline":"Rigorous, honest, and important for evaluation methodology, but the headline correlation (0.119) is mostly driven by news that follows the market, and the paper's own exploratory results show the publication-aligned association is near zero (0.023) and no trading usefulness at the weekly horizon.","tokens_in":27114,"tokens_out":2133,"duration_ms":20827},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"LLM news judgments track returns that happen before the news, not after it","keywords":["Web news","LLM readers","multi-agent trading systems","criterion validity","point-in-time evaluation","look-ahead bias","typed decision models","influence bound"],"falsifier":"If a future LLM reader were tested on news items published strictly before any price change (e.g., using pre-opening news from after-hours filing to the next open), and its judgments showed a Spearman correlation above 0.05 with subsequent abnormal returns from that point, the paper's claim that the association is mostly realized before publication would be weakened. Alternatively, if a different news feed or a larger cap universe yielded a publication-aligned correlation above 0.10, the generality of the lag effect would be challenged.","tokens_in":27649,"feed_emoji":"📰","tokens_out":4361,"duration_ms":42209,"temperature":0.7,"pith_summary":"The paper asks whether LLM judgments of Web news items about listed firms track the market's reaction to the reported event or simply follow a price move that has already occurred. To answer, the authors built a multi-agent trading system with point-in-time stores, typed decision interfaces, and a proved bound on how far text can influence positions. In a pre-registered test, LLM option-probability judgments correlated with announcement-window abnormal returns at a weekly Spearman of 0.119, even after the model's release. However, exploratory analysis showed that 72% of earnings news appears after the issuer's filing and that the correlation fell to 0.023 when measured from the first trade after publication. The paper concludes that standard market-graded tests should report the criterion from publication, not just from the event, because Web news largely reports what has already happened.","feed_headline":"News-reading LLMs track returns that precede the news","feed_subtitle":"A pre-registered trading system shows Web financial news mostly reports events after the market reacts, so criterion tests must measure…","key_machinery":"The central mechanism is a trust-partitioned learning system with a proved influence bound (Proposition A.2) that limits how far any text can move a position, combined with a publication-aligned evaluation criterion that separates returns realized before an item appears from those that follow its publication. The band-guarded decision layer (Lemma A.3) admits text to adjust ranks only among names the trusted data already rates near the cut, and the pre-registered protocol (hash-committed tests H1–H3) fixes the evaluation in advance.","core_discovery":"The confirmatory test detects that LLM item judgments rank announcement-window abnormal returns (weekly Spearman 0.119, 95% CI [0.098, 0.140]), and the association holds after the LLM's release (0.115). Event-type judgments agree with issuers' 8-K filings (AUC 0.833 for earnings). Yet the association is mostly realized before the item appears: 72% of earnings news items follow the issuer's filing, contributor-written items frequently report the price move, and from the first trade after publication the correlation drops to 0.023 [0.005, 0.041]. At a weekly decision horizon, the test detects no rank information over code-only features (ΔIC −0.016). All attacks stayed within the proved influence bound, and decision cards traced 88% of successful attacks to the planted item. The paper proposes that market-graded tests of Web-text readers should report the criterion from publication, not only from the event.","pith_inferences":["The same point-in-time evaluation methodology could be applied to other domains where Web text reports time-stamped outcomes, such as clinical trial results graded against registry data or product announcements graded against subsequent sales figures.","If the finding generalizes beyond the single news feed and LLM tested, much of the touted market-prediction ability of LLM readers may be an artifact of reporting the price move rather than anticipating it, reducing their practical trading value.","The band-guarded decision layer is a promising pattern for deploying any untrusted content (not just text) in safety-critical systems: a tight bound on influence from untrusted sources, enforced by code, independent of the content's quality.","The paper's failure to detect rank information at a weekly horizon (H3) suggests that fast assimilation of news by the market leaves little for a learner to extract, consistent with the efficient-market view at weekly scales."],"forward_implications":["Future tests of LLM readers on Web financial news should report a publication-aligned criterion alongside the announcement-window criterion, as the paper's results show the two can diverge sharply.","A trading system's exposure to adversarial text is bounded by a code-level cap, not by the reader's vigilance, so security guarantees can be made reader-agnostic.","Planted content that stays inside the band (e.g., fabricated facts or press releases) can still move positions, meaning defense must address both the reader and the decision-layer bound.","The association between LLM judgments and returns is mostly pre-publication, so claims of predictive insight from Web news readers should be tempered by a timing analysis.","The multi-agent architecture with per-item clearances and write-once stores is a replicable testbed for any typed reader, enabling paired re-evaluation without look-ahead bias."],"supporting_citations":[{"why":"Provides the prior finding that GPT-4 headline scores track the initial market reaction after the model's cutoff, which the present work extends and probes for look-ahead.","marker":"[32]"},{"why":"Establishes that LLMs outperform crowd workers in text-annotation tasks, supplying a baseline for evaluating typed reader accuracy.","marker":"[13]"},{"why":"Documents that typed decision models follow the option name rather than the definition bound to it, motivating the paper's separation of form (typed interface) from meaning (validity test).","marker":"[39]"},{"why":"Audits typed-model papers and finds most lack a label-probability baseline, directly informing the paper's choice to pin emu as that baseline.","marker":"[40]"},{"why":"Analyses Jev's functionality and ecosystem, providing the schema that all readers in the paper speak.","marker":"[31]"},{"why":"A pre-registered test of a typed model on earnings-call remarks, whose protocol the present paper adopts and extends to Web news.","marker":"[23]"},{"why":"Demonstrates that supervised language-model probabilities of news predict the sign of excess returns at daily horizons, a direct point of comparison for the paper's weekly-horizon null result.","marker":"[26]"}],"fun_headline_variants":["LLMs track returns that arrive before the news","Web news lags market, LLMs follow the reaction","Market-graded tests of LLM news readers miss the lag","LLM agents track market even when news is late","News-reading LLMs: tracking the market, not the event"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The trust partition (Assumption A.1) assumes that the trusted logit, trusted ranks, the bound parameter kappa, and all reference-path quantities depend only on trusted inputs and are unaffected by any attack on text. If a later attack could indirectly affect trusted features (e.g., through correlated price movements or contamination of blackboard stores by earlier text outputs), the influence bound would not hold. The paper argues that single-writer stores and immutable records enforce this separation, but the assumption also depends on the modeler's choice of which features are trusted versus untrusted.","fun_headline_variants_meta":{"raw":{"variants":["LLMs track returns that arrive before the news","Web news lags market, LLMs follow the reaction","Market-graded tests of LLM news readers miss the lag","LLM agents track market even when news is late","News-reading LLMs: tracking the market, not the event"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":4e-05,"raw_usage":{"cost":3.9557532e-05,"is_byok":false,"cost_details":{"upstream_inference_cost":3.9557532e-05,"upstream_inference_prompt_cost":2.5821852e-05,"upstream_inference_completions_cost":1.373568e-05},"total_tokens":1271,"prompt_tokens":1047,"completion_tokens":224,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":143}},"tokens_in":1047,"tokens_out":224,"cost_source":"openrouter_usage","duration_ms":2836,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":143,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-11-10T23:06:22.308720+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"If a future LLM reader were tested on news items published strictly before any price change (e.g., using pre-opening news from after-hours filing to the next open), and its judgments showed a Spearman correlation above 0.05 with subsequent abnormal returns from that point, the paper's claim that the association is mostly realized before publication would be weakened. Alternatively, if a different news feed or a larger cap universe yielded a publication-aligned correlation above 0.10, the generality of the lag effect would be challenged.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the prior finding that GPT-4 headline scores track the initial market reaction after the model's cutoff, which the present work extends and probes for look-ahead."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes that LLMs outperform crowd workers in text-annotation tasks, supplying a baseline for evaluating typed reader accuracy."},{"cited_title":"Type-Safe Is Not Error-Free: Typed Decision Models Follow the Option Name, Not the Definition Bound to It","cited_arxiv_id":"2609.26758","evidence_quote":"Documents that typed decision models follow the option name rather than the definition bound to it, motivating the paper's separation of form (typed interface) from meaning (validity test)."},{"cited_title":"Typed Decision Models: An Early Evidence Audit and Evaluation Checklist","cited_arxiv_id":"2609.32160","evidence_quote":"Audits typed-model papers and finds most lack a label-probability baseline, directly informing the paper's choice to pin emu as that baseline."},{"cited_title":"Jev in the Wild: A Data-Driven Analysis of the Jev Model's Functionality, Applications and Ecosystem","cited_arxiv_id":"2609.30216","evidence_quote":"Analyses Jev's functionality and ecosystem, providing the schema that all readers in the paper speak."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"A pre-registered test of a typed model on earnings-call remarks, whose protocol the present paper adopts and extends to Web news."},{"cited_title":"Financial Language Models as Applied Artificial Intelligence Systems for News-Based Trading under Market Frictions","cited_arxiv_id":"2609.23703","evidence_quote":"Demonstrates that supervised language-model probabilities of news predict the sign of excess returns at daily horizons, a direct point of comparison for the paper's weekly-horizon null result."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}