{"id":"ddb56e55-551f-4676-aec8-8a3499dc5c9e","arxiv_id":"2505.23826","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"FinRipple aligns LLMs with financial markets via knowledge-graph adapters and PPO using CAPM residuals as reward, claiming strong ripple-effect prediction, but the evaluation is circular and artifacts are unavailable.","lead":"A new framework, FinRipple, teaches large language models to predict how a financial event ripples through related companies by adding a time-varying knowledge graph and training the model with a reward based on asset-pricing residuals. The paper reports strong out-of-sample explanatory power and a profitable daily trading strategy, but the evaluation is closely tied to the training reward, and no code or data is released.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Monthly KG snapshots may admit lookahead: if a snapshot for month m contains relationships observable only after a news event on day d, the predictor contains the label's future, invalidating Tables 1 and 3.","rationale":"The reader's weakest assumption is the same as mine; I agree. The monthly KG is the single most load-bearing component because it sits directly between the event and the predicted residual; a leak there contaminates every headline number, while other flaws are secondary or affect only the strength of evidence. The 'evaluation metric equals training reward' observation is not by itself an error: training to maximize correlation with a target and then measuring R-squared against that target is standard practice, so I do not treat it as the decisive flaw. The Appendix A generalization bound is, as the reader notes, invalid (the claimed bound grows with m, and the Rademacher complexity term is asserted without a valid derivation), but the paper's empirical claim does not rest on that bound. The omitted transaction costs in the backtest are a real limitation but would not erase the cross-sectional R-squared. Thus the decisive issue remains temporal integrity. Because the paper neither specifies nor releases the data needed to verify the cutoff, the rejection stands. If a point-in-time rebuild reproduced the numbers, the verdict would need revisiting, but as written the leakage risk is unresolved.","tokens_in":24296,"tokens_out":7470,"duration_ms":84433,"concrete_test":"Rebuild the knowledge graph as point-in-time: for each news item with publication timestamp tau, retain only edges whose underlying record has an effective or disclosure date strictly before tau (13F holdings with the statutory 45-day lag, patents by grant/publication date, supply-chain transactions by filing date, leadership changes by announcement date), and compute CAPM betas on trailing windows only. Re-run FinRipple's full pipeline and recompute Table 1's R-squared and Table 3's Sharpe ratio. If the R-squared drops to the w/o-alignment level or the Sharpe no longer exceeds the Table 3 benchmarks, the monthly snapshots leak future structure. A decisive preliminary check is to sample test-month KGs and verify whether any edge is attributable to a record dated after the first day of that month; if any is, leakage is already established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"FinRipple's out-of-sample claim requires that the model's inputs be knowable at prediction time. Section 3.2.2 states 'for each news item, we retrieve the corresponding KG for the relevant time,' and Section 3.2.3 injects 'the KG corresponding to the specific month into the adapter.' Appendix B.2, however, reports only month ranges, not a per-record disclosure cutoff. The constituent data sources (USPTO patents, Thomson/Refinitiv mutual fund holdings, Compustat-Capital IQ supply-chain filings, BoardEx leadership records) have publication and filing lags; a monthly snapshot can therefore contain edges and weights that became observable only after the news date. If an event on March 15 is paired with the full March KG, the signed interaction measure mu_t and the next-day CAPM residual are no longer informationally separated: the model can see structure realized later in March, some of which is the same information that produces the residual. Table 1's R2 = 0.34 and Table 3's Sharpe = 1.153 would then measure lookahead, not ripple-effect prediction. The same point-in-time ambiguity applies to the 'rolling windows' used for CAPM beta in Section 3.2.2, which must be strictly trailing. The paper's Limitations section discusses delayed KG updates but not early incorporation, and no code or data release allows the cutoff to be audited. This unresolved temporal-integrity issue is the load-bearing concern: if it lands, the central empirical claims are uninterpretable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FinRipple, a three-stage framework for predicting financial market ripple effects from news events. Stage one constructs a time-varying knowledge graph of S&P 500 firms from patent, supply-chain, leadership, and mutual-fund-holding data. Stage two injects each monthly KG snapshot into a frozen LoRA adapter through instruction tuning. Stage three aligns the LLM with the market by fine-tuning the backbone with PPO, using a reward that combines the cosine similarity between the propagator-aggregated prediction Z and the CAPM residual epsilon with a magnitude-coverage term. The main empirical claims are cross-sectional R2 values up to 0.34 for explaining CAPM residuals (Table 1) and a daily long-short portfolio with Sharpe ratio 1.153 (Table 3) over January 2020 to June 2022.","tokens_in":24631,"tokens_out":8274,"duration_ms":83336,"significance":"The paper introduces a clearly defined task, a multi-channel time-varying KG, and a plausible adapter-based mechanism for injecting structured market knowledge into an LLM. The breadth of baselines across open- and closed-source models is substantial, and the case study in Figure 5 is a useful illustration of the intended reasoning behavior. If the empirical claims were valid, the work would be a meaningful step toward using LLMs for event-driven asset pricing and portfolio construction. However, the reported evidence is not currently interpretable as out-of-sample prediction: the point-in-time integrity of the monthly KG snapshots is not established, the main evaluation metric coincides with the training reward, and the theoretical appendix contains an inconsistent generalization bound. These are load-bearing issues for the central claim, not presentation concerns.","major_comments":[{"comment":"The point-in-time integrity of the knowledge graph is not established. The paper states that 'for each news item, we retrieve the corresponding KG for the relevant time' (Section 3.2.2) and that the model injects 'the KG corresponding to the specific month' (Section 3.2.3), but Appendix B.2 reports only month ranges for the training and testing sets, not per-record disclosure cutoffs. The KG sources (USPTO patents, Thomson/Refinitiv mutual fund holdings, Compustat-Capital IQ supply-chain filings, BoardEx leadership records) are subject to publication and filing lags. If a monthly snapshot paired with a news item dated day d contains edges that became observable only after day d, then the next-day CAPM residual epsilon is informationally contaminated by the input, and Tables 1 and 3 would measure lookahead rather than ripple-effect prediction. The paper must specify the exact information cutoff for every edge, verify that no edge uses data posterior to the news timestamp, and confirm that the CAPM beta rolling windows are strictly trailing. Without this, the central empirical claims are uninterpretable.","section":"Section 3.2.2–3.2.3 and Appendix B.2"},{"comment":"The main evaluation metric is the same statistical quantity that the training reward maximizes. The first reward term in Section 3.2.2 is R(Z, epsilon) = Z·epsilon / (||Z|| ||epsilon||), the cosine similarity between the aggregated prediction and the CAPM residual, and Table 1 reports the R2 from regressing those residuals on the predictions. For a simple regression, R2 is a monotone transformation of the squared correlation between Z and epsilon, so PPO directly optimizes the statistic used as headline evidence. The comparison against untrained baselines therefore shows that the model can optimize this objective, but it does not provide an independent test of ripple-effect prediction. An evaluation metric not used in training, such as event-time cumulative abnormal returns computed from raw returns with a point-in-time factor model, or decile portfolio spreads net of transaction costs, is needed to support the claim.","section":"Section 3.2.2 (reward) and Table 1"},{"comment":"The generalization bound is not a reliable basis for the paper's theoretical claims. The theorem states an additive term of order C·kl·sqrt(m), while the proof concludes with a term of order C·kl/sqrt(m); these differ by a factor of m, and the statement that a constant 'absorbs all logarithmic factors' cannot repair a mismatch in the dependence on m. In addition, the derivation of the inequality sum_i sum_j a_ij^2 <= H sum_i (sum_j a_ij)^2 from Assumption 3 requires the errors a_ij to be nonnegative, which is not assumed anywhere. The appendix should be corrected or removed; as written, it does not support the claimed theoretical grounding.","section":"Appendix A, Theorem 1"}],"minor_comments":[{"comment":"The ablation study removes 'Technical Relevance,' 'Supply Chain,' and 'Shared Leadership,' but Section 3.2.1 introduces four relationship types; there is no ablation removing mutual fund holdings, so the claim that all relationship types are useful is incomplete.","section":"Appendix E.3"},{"comment":"The magnitude-coverage term is described as maximizing 'recall rate,' but the expression min(|Z_j|,|epsilon_j|)/||epsilon||_1 is not a standard recall or coverage statistic; its interpretation as a recall measure should be clarified.","section":"Section 3.2.2 (reward)"},{"comment":"The text describes the strategy as 'intraday' in one place and as a daily portfolio that rebalances at the end of each trading day in the steps; this terminology should be made consistent, and the backtest should state whether transaction costs and trading frictions are included.","section":"Section 4.3"},{"comment":"The reproducibility statement says that a training framework and benchmark dataset 'will be released in the future,' but no code, data, or benchmark link is provided. Given that the point-in-time construction is the central validity concern, the absence of an auditable artifact is a serious practical obstacle for verification.","section":"Appendix G.2"},{"comment":"The abstract and contributions describe an 'open-source benchmark,' but the paper only promises future release; the wording should be corrected to avoid overstating the current availability of the resource.","section":"Section 1 and Appendix G.2"}],"recommendation":"reject","confidential_remarks":"The temporal-integrity concern is the decisive issue: if the monthly KG snapshots contain information posterior to the news date, every reported R2 and Sharpe ratio is invalid. The evaluation metric being identical to the training reward further weakens the evidence. These are fixable in principle with strict point-in-time data and an independent evaluation metric, but as presented the central claims cannot be trusted. Given the paper's reliance on these results, rejection is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: if you need a model that predicts CAPM residuals from news plus a time-varying knowledge graph, the FinRipple pipeline is a sensible thing to try. The combination of monthly KG adapters, instruction tuning, and PPO with a CAPM-residual correlation reward is not something I've seen before, and the paper does a decent job laying out the four-channel KG construction, the instruction generation, and ablations showing that supply-chain edges matter most. The case study in Figure 5 is a nice interpretability illustration.\n\nThe reported numbers, though, cannot be trusted as-is. The first and biggest problem is temporal integrity. The paper says it uses monthly KG snapshots and injects \"the KG corresponding to the specific month\" for each news item, but it never states a point-in-time cutoff. A March snapshot built from USPTO patents, mutual fund holdings, Compustat supply-chain filings, and BoardEx records will contain relationships that were only observable after a March 15 news event. If that is the case, the model is seeing the label's future, and Table 1's R2 of 0.34 and Table 3's Sharpe of 1.153 measure leakage, not ripple effects. This is the load-bearing flaw.\n\nSecond, the evaluation metric is the training reward. The PPO reward is cosine similarity between aggregated predictions and CAPM residuals, and Table 1 reports R2 from regressing residuals on those predictions. That is not automatically invalid — it is a supervised prediction setup with a held-out test period — but it means the \"alignment\" numbers simply say the model got better at the thing it was trained to do. The baselines were not trained on that objective, so the comparison is apples-to-oranges.\n\nThird, the Appendix A generalization bound is dimensionally wrong: the final term grows with m, C kl sqrt(m), while the Rademacher bound it comes from decreases as 1/sqrt(m). That appendix should be removed or fixed.\n\nFourth, the portfolio backtest has no transaction costs and reports a daily 0.52% return with Sharpe 1.15. That is implausible; it needs costs, capacity limits, and a benchmark that trades the same universe on the same days.\n\nFinally, the paper claims an open-source benchmark, but the reproducibility statement says code and data will be released \"in the future.\" That is a mismatch.\n\nOverall: the framework is a reasonable direction, but the empirical core is unverifiable in current form. I would not cite it yet, and I would not send it to peer review without a revision that (a) specifies exact information cutoffs per news item, (b) re-evaluates with a metric not identical to the reward, (c) fixes the theory appendix, and (d) releases the artifacts. If the authors can rule out leakage, the idea deserves a real look.","headline":"A plausible pipeline undermined by unresolved temporal leakage and self-referential evaluation; get the cutoffs right and re-evaluate before trusting any number.","tokens_in":25185,"tokens_out":5142,"would_cite":false,"duration_ms":60836,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An LLM aligned to market structure predicts event ripple effects: R² up to 0.34 on next-day residuals, Sharpe 1.153 on a daily portfolio.","keywords":["ripple effect prediction","financial knowledge graph","large language models","reinforcement learning","CAPM residuals","asset pricing","portfolio management","event study"],"falsifier":"Rebuild the monthly knowledge graphs as strictly point-in-time, deleting any edge (a mutual-fund holding, a supply-chain transaction, a patent, or a leadership appointment) dated after the prediction day, then re-run the Table 1 regression and the Table 3 backtest; if the $R^2$ collapses from 0.34 toward the 0.05–0.12 baseline band, or the Sharpe falls toward the 0.82–1.02 benchmark range, the reported gains came from look-ahead in the graphs rather than from ripple-effect reasoning. A complementary check is to permute the news-to-graph pairings and verify that the alignment reward loses its specificity.","tokens_in":24076,"feed_emoji":"📈","tokens_out":15715,"duration_ms":128512,"temperature":0.7,"pith_summary":"The paper is trying to establish that the ripple effect of a financial event — the way one company's news moves the stock prices of connected companies — is a definable and predictable quantity, not just a narrative. Its claim is that a large language model can be aligned to the market by training it, through reinforcement learning, to predict the part of next-day returns that the Capital Asset Pricing Model (CAPM) cannot explain, using a time-varying knowledge graph of four relationship channels as the market context. If the claim holds, event-driven residual returns are substantially forecastable from public news plus structured market relationships, and the route to that capability is domain alignment through a financial reward signal rather than larger models. The paper also introduces ripple-effect prediction as a standardized task with an open benchmark, so future work can be measured against a common target.","feed_headline":"Event ripple returns explained to 34 percent by an aligned LLM","feed_subtitle":"Trained to match CAPM residuals, the model yields a Sharpe 1.153 long-short portfolio on S&P 500 stocks.","key_machinery":"The load-bearing mechanism is the reward-driven alignment loop between predicted and realized shocks. Formally, the reward is $R(Z, \\epsilon) = \\frac{Z \\cdot \\epsilon}{\\|Z\\| \\|\\epsilon\\|} + \\lambda \\frac{\\sum_j \\min(|Z_j|, |\\epsilon_j|)}{\\|\\epsilon\\|_1}$: the first term rewards the direction match between the predicted impact vector $Z$ and the realized CAPM residual vector $\\epsilon$, while the second, scaled by $\\lambda = 0.1$, rewards magnitude coverage so that the model names as many genuinely affected firms as possible. This scalar signal is passed to PPO to fine-tune the LLM backbone while the adapter layers — the parameters that encode the time-varying knowledge graph — remain frozen. The graph $G_t = (C_t, R_t)$ with four edge types (leadership overlaps, mutual-fund cross-holdings, patent-based technical similarity, supply-chain links) supplies the market structure, and the CAPM regression supplies the residual that defines what 'unexplained by the market' means. The conjunction of these three pieces — time-varying graph, frozen adapters, residual-matching reward — is what the paper argues converts a general LLM into an event-ripple analyst.","core_discovery":"The paper's central claim is that the ripple effect of a market event is a learnable prediction target: for a news event $e_t$ and the current market structure $G_t$, the propagator $\\Phi_{e_t,\\theta}$ outputs a cross-sectional vector of next-day shocks, and its value is validated against CAPM residuals $\\epsilon_j^{t+\\Delta t}$, the return component unexplained by market $\\beta$. FinRipple achieves this by injecting a time-varying knowledge graph into the LLM through frozen adapters and fine-tuning the backbone with PPO, using a reward that scores both the direction and the magnitude coverage of predicted shocks against realized residuals. The reported results are strongest on the vicuna-13b backbone: $R^2 = 0.340$ for explaining the level of next-day CAPM residuals ($p < 0.001$), with $R^2$ rising to 0.55 and 0.59 for Fama-French three- and five-factor residuals, and a long-short portfolio formed on predicted impact achieving Sharpe 1.153 over the January 2020 to June 2022 test window. The authors state that alignment with the market, not graph injection alone, is the component that produces the gain, and that the framework constitutes the first standardized definition of ripple-effect prediction.","pith_inferences":["If the temporal-integrity condition holds, the framework generalizes the event-study paradigm: the unit of analysis shifts from one firm's abnormal return to a full cross-sectional residual vector, and any factor model's residual could serve as the alignment target.","The reported Sharpe of 1.153 is pre-cost and pre-slippage, and the test window (2020–2022) includes unusual volatility; a natural stress test is to rerun the backtest on a longer, calmer window with transaction costs, where the edge may narrow.","The paper's own ablation ranking — supply-chain edges matter most — suggests a testable implication the authors do not draw: predictive power should concentrate in episodes of supply-chain stress, so performance could be decomposed by news category to expose where the ripple mechanism actually operates.","Casting 'alignment' as agreement with a stochastic environmental target rather than human preference offers a template for other domains — energy grids, epidemiological spread, logistics networks — where a structured model of the environment defines the residual to be explained."],"forward_implications":["If the central claim holds, next-day CAPM residuals for S&P 500 firms are substantially cross-sectionally predictable from news plus a four-channel knowledge graph, implying that a large share of 'abnormal' returns is event propagation rather than noise.","A daily long-short portfolio built from predicted event impact (top decile long, bottom decile short) is claimed to beat equal-weight, volatility-weight, Markowitz, and min-variance benchmarks on return, Sharpe, and drawdown over the 2020–2022 test window.","Market alignment via PPO is the necessary ingredient: the 'FinRipple/w-o alignment' comparison shows that injecting the graph without the residual-matching reward yields only marginal gains.","The same reward design transfers across 7B–13B base models and explains Fama-French three- and five-factor residuals better than CAPM residuals, indicating the mechanism is not tuned to a single pricing model.","Model scale is secondary to domain alignment: after alignment, smaller open models exceed the unaligned performance of larger closed models, repositioning domain alignment as the binding constraint for financial LLMs."],"supporting_citations":[{"why":"Supplies the CPC-code-correlation methodology used to build the patent-based technical-relevance edges in the knowledge graph.","marker":"Lee et al. 2019"},{"why":"Prior work embedding financial knowledge graphs for quantitative investing, the lineage the time-varying KG design extends.","marker":"Cheng et al. 2020"},{"why":"LoRA is the parameter-efficient adaptation method behind the frozen time-varying adapters that inject the knowledge graph.","marker":"Hu et al. 2021"},{"why":"Defines the RAG baseline that FinRipple must beat; its retrieval failures motivate parameterized KG injection.","marker":"Lewis et al. 2020"},{"why":"Defines the in-context-learning baseline used in comparisons against the fine-tuned FinRipple variants.","marker":"Brown et al. 2020"},{"why":"Chain-of-thought prompting used at inference to expose the reasoning pathways claimed to show causal ripple analysis.","marker":"Wei et al. 2022a"},{"why":"The event-study methodology review that defines the limitations FinRipple's ripple-effect task is designed to overcome.","marker":"Sorescu et al. 2017"},{"why":"Provides the portfolio benchmarks (equal weighting, volatility weighting, Markowitz, min-variance) used in the backtest comparison.","marker":"Xu et al. 2024"}],"fun_headline_variants":["LLM aligned to market explains 34% of ripple variance","Event ripple prediction from LLM: R²=0.34, Sharpe 1.153","FinRipple: LLM learns ripple effects from market graph","Market-aligned LLM predicts next-day shock residuals","LLM ripple effect prediction: 34% variance explained"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"For every daily prediction, that month's knowledge-graph snapshot must contain only relationships knowable before the prediction day; the paper never specifies the exact cutoff inside each month, and its own limitations section concedes that stale or delayed graphs degrade the system.","fun_headline_variants_meta":{"raw":{"variants":["LLM aligned to market explains 34% of ripple variance","Event ripple prediction from LLM: R²=0.34, Sharpe 1.153","FinRipple: LLM learns ripple effects from market graph","Market-aligned LLM predicts next-day shock residuals","LLM ripple effect prediction: 34% variance explained"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000787,"raw_usage":{"total_tokens":3494,"prompt_tokens":987,"completion_tokens":2507,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":2417}},"tokens_in":603,"tokens_out":2507,"duration_ms":20180,"temperature":1.0,"reasoning_tokens":2417,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:17:17.976482+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rebuild the monthly knowledge graphs as strictly point-in-time, deleting any edge (a mutual-fund holding, a supply-chain transaction, a patent, or a leadership appointment) dated after the prediction day, then re-run the Table 1 regression and the Table 3 backtest; if the $R^2$ collapses from 0.34 toward the 0.05–0.12 baseline band, or the Sharpe falls toward the 0.82–1.02 benchmark range, the reported gains came from look-ahead in the graphs rather than from ripple-effect reasoning. A complementary check is to permute the news-to-graph pairings and verify that the alignment reward loses its specificity.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the CPC-code-correlation methodology used to build the patent-based technical-relevance edges in the knowledge graph."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior work embedding financial knowledge graphs for quantitative investing, the lineage the time-varying KG design extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the RAG baseline that FinRipple must beat; its retrieval failures motivate parameterized KG injection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the in-context-learning baseline used in comparisons against the fine-tuned FinRipple variants."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The event-study methodology review that defines the limitations FinRipple's ripple-effect task is designed to overcome."}],"review_version":1}