{"id":"3f59a40f-7547-4382-a2e9-2f61b63b2481","arxiv_id":"2412.17293","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"An RL agent fed price, SEC, and news sentiment data slightly beats the S&P 500 in a 2018-2019 backtest, but trails a simple equal-weight buy-and-hold.","lead":"This paper combines stock prices, SEC filing sentiment, and news headline sentiment in a reinforcement learning agent that chooses S&P 100 portfolio weights. The authors report that adding alternative data improves performance under a profit reward, though a simple equal-weight buy-and-hold benchmark still earns more over the test period.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own reported numbers contradict its central claim: the price-only CNN with Profit reward (net profit 0.1667, Sharpe 0.5615) beats the 'best' combined-data CNN (0.1661, 0.5536), and both underperform Equal Buy-and-Hold (0.1811, 0.5952).","rationale":"The reader's rejection is well-founded, but the sharpest load-bearing problem is not the imputation concern from Section 2.2.3; it is an internal inconsistency in the reported results. The paper selects the combined-data CNN Profit model as best, yet its own Table 7 reports a price-only CNN Profit model with higher net profit, Sharpe ratio, and Sortino ratio. Table 12 then omits that price-only configuration, which changes the comparison and makes the headline claim appear stronger than the data support. Even the selected 'best' model fails to beat a simple Equal Buy-and-Hold benchmark, further undermining the claim of superior RL portfolio performance. The imputation concern is real and is explicitly flagged by the authors, but it is secondary: even if the sentiment imputation were perfect, the reported tables do not demonstrate the claimed multimodal advantage. The abstract also promises topic embeddings that are never described or evaluated in the methodology, which weakens the stated contribution. Given the absence of error bars, selection of the best model on test-period metrics, and no released code or hyperparameters, the central claim is not established. The reader's REJECT verdict should stand unchanged.","tokens_in":13729,"tokens_out":4710,"duration_ms":47037,"concrete_test":"Reproduce the evaluation protocol that generated Tables 7, 10, and 11 (same S&P100 price data, same 2010-2017 train / 2018-2019 test split, same 1% transaction costs) and compute a paired bootstrap or t-test on daily portfolio returns across at least 5 seeds for two configurations: Hist Prices CNN (Profit) and SEC+News CNN (Profit). If the multimodal model does not show a positive, statistically distinguishable improvement over the price-only model, the paper's headline claim fails. As a secondary check, search the methodology for any implementation of the 'topic embeddings' promised in the abstract; none appears in Sections 2-3.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.4 states that 'our best-performing model across all three sets of experiments is the CNN EIIE model on the combined dataset using Profit reward,' and Section 4.5 claims this policy has the highest net profit, Sharpe, and Sortino ratio excluding Equal Buy-and-Hold. The paper's own Table 7 shows the price-only CNN with Profit reward achieves net profit 0.166717, Sharpe 0.561471, Sortino 0.676669, all higher than the combined SEC+News CNN Profit figures in Table 11 (0.166081, 0.553644, 0.664706). Table 12, which is used to support the headline comparison, omits the price-only CNN configuration even though it outperforms the selected 'best' model. In addition, every trained RL policy in Tables 6-11 has lower net profit and Sharpe than the Equal Buy-and-Hold benchmark (0.181138, 0.595225). The central claim that adding multimodal sentiment channels improves portfolio performance is therefore not just unproven; it is contradicted by the numbers the paper reports. No error bars or multiple-seed statistics are given, so even the small apparent advantage of SEC+News over SEC-only is not established as signal rather than noise.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a deep reinforcement learning framework for S&P100 portfolio optimization that augments price history with sentiment features derived from SEC filings and news headlines, using EIIE-style CNN, RNN, and MLP policies under profit and Differential Sharpe rewards. The authors report backtests over a 2018–2019 test period and claim that the CNN EIIE model trained on combined price+SEC+news data with a profit reward is the best-performing model across all experiments, and that it exceeds OLMAR and WMAMR benchmarks. The paper also describes data processing, an exponential-decay forward-filling rule for sparse sentiment observations, and a transaction-cost-aware environment.","tokens_in":13948,"tokens_out":2529,"duration_ms":24614,"significance":"If the central claim were supported, the paper would provide a moderately useful incremental demonstration that multimodal sentiment channels can be integrated into EIIE-style RL portfolio policies. The manuscript has several strengths: it carefully describes the data pipeline, implements a nontrivial transaction-cost model from prior work, and explicitly acknowledges data sparsity limitations. However, the reported evidence does not support the headline claim. The paper's own tables show that a price-only CNN with profit reward outperforms the combined-data CNN on net profit, Sharpe, and Sortino ratios, and that the simple Equal Buy-and-Hold benchmark beats every trained RL policy. The selected comparison table omits the price-only configuration that contradicts the claim. Without multiple seeds or error bars, the small apparent differences among RL variants cannot be distinguished from noise. The forward-filling rule for news sentiment, which supplies the majority of the multimodal state entries for most tickers, is a load-bearing assumption that is not validated. These issues undermine the paper's central contribution as presented.","major_comments":[{"comment":"The claim in §4.4 that 'our best-performing model across all three sets of experiments is the CNN EIIE model on the combined dataset using Profit reward' is contradicted by the paper's own numbers. Table 7 reports that the price-only CNN with Profit reward achieves net profit 0.166717, Sharpe ratio 0.561471, and Sortino ratio 0.676669, whereas Table 11 reports for the SEC+News CNN with Profit reward net profit 0.166081, Sharpe ratio 0.553644, and Sortino ratio 0.664706. The price-only model outperforms the combined model on all three metrics, so the stated 'best-performing model' is not the best according to the reported results.","section":"§4.4, Table 11 vs. Table 7"},{"comment":"Table 12, which is used to support the headline comparison of 'best strategies,' omits the price-only CNN with Profit reward from Table 7. Including that row would show that the selected SEC+News CNN is not the best trained policy, and the text's statement that the SEC+News CNN Profit policy has the highest net profit, Sharpe, and Sortino excluding Equal Buy-and-Hold is factually incorrect given the paper's own Table 7 values. The comparison must include all configurations, especially those that outperform the configuration being highlighted.","section":"§4.5, Table 12"},{"comment":"Every trained RL policy reported in Tables 6–11 has lower net profit and Sharpe ratio than the simple Equal Buy-and-Hold benchmark, which achieves net profit 0.181138 and Sharpe 0.595225. The abstract's claim that the approach 'delivers superior portfolio performance' and 'outperform standard benchmarks' is therefore not supported; the trained agents only outperform OLMAR and WMAMR, which are themselves weak in this test period. The benchmark comparison should be presented in full, and the conclusion should be revised to acknowledge that the method does not beat the equal-weight buy-and-hold baseline.","section":"§4.2–§4.5, Tables 5–12"},{"comment":"The multimodal state tensor is largely constructed by exponential-decay forward-filling of sparse sentiment observations. News coverage exists for only 84 of 100 tickers, and the median ticker has reports on only 905 trading dates out of the roughly 2,500 trading days in the full period. Eq. (2) with a tuned parameter γ≈0.8 imputes sentiment scores for the vast majority of state entries, and the paper itself notes in §2.2.3 that 'coverage is still incomplete.' Because the imputation rule determines most of the sentiment channel, the comparisons between price-only, SEC-only, and combined configurations reflect properties of the filling rule as much as the information content of the underlying news data. The paper should provide sensitivity analysis over the imputation rule (e.g., different decay factors, no imputation) before attributing performance differences to the data sources.","section":"§2.2.3 and Eq. (2)"},{"comment":"No error bars, confidence intervals, or multiple-seed results are reported for any RL policy. Differences between configurations—for example, SEC+News CNN (0.166081) versus SEC-only CNN (0.162534) under Profit reward—are small and may be within run-to-run variance, especially given that policy-gradient training is stochastic. The paper should report mean and standard deviation over at least several independent training runs, or otherwise provide statistical evidence that the claimed improvement is not noise.","section":"§4, all tables"}],"minor_comments":[{"comment":"The caption of Table 11 says 'Strategies with Combined Data (DiffSharpe Reward)' but the surrounding text in §4.4 states that the table reports results for the Profit reward. The caption should be corrected to 'Profit Reward' to match the content.","section":"Table 11 caption"},{"comment":"The statistic 'Minimum Observations 1' is ambiguous: it could mean one reporting date or one headline. Please clarify whether the count is of reporting dates or individual headlines.","section":"§2.2.3, Table 3"},{"comment":"The reference contains a typo: 'finaincial' should be 'financial.'","section":"Reference [10]"},{"comment":"The text states 'The CNN and RNN policies using the Differential Sharpe ratio are among the strongest contenders,' but Table 9 shows Profit-reward results for SEC data. The intended reference is likely Table 8; please correct the cross-reference.","section":"§4.3, Table 9"},{"comment":"The notation for the EIIE functions is inconsistent: the text uses fα, gβ, and γ, but also refers to fθ1 and trainable parameters α, β, and γ without defining θ1. Please harmonize the notation.","section":"§3.6"}],"recommendation":"reject","confidential_remarks":"The paper is likely a student project preprint, and the writing is clear in places, but the central empirical claim is contradicted by the authors' own tables. This is not a matter of taste or framing; the numbers reported in Tables 7 and 11 directly refute the statement that the combined-data CNN is the best model. Re-running experiments to correct this would likely also require addressing the lack of multiple seeds and the heavy reliance on an unvalidated sentiment imputation rule. Given the scope of the manuscript, these are load-bearing issues that cannot be fixed by minor edits. The paper would need a substantially revised experimental design and a reinterpretation of its results before it could be considered for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things up front. First, the paper does real work: it re-implements the EIIE architecture, adds SEC-filing and news-headline sentiment as extra state channels, and introduces a specific sentiment embedding function (their Eq. 1). The authors are also transparent about data gaps, especially the sparse news coverage. Second, the central claim—that the combined-data CNN with the profit reward is the best model—is directly contradicted by their own tables. Table 7 shows the price-only CNN achieving net profit 0.1667 and Sharpe 0.5615, while Table 11 shows the combined CNN at 0.1661 and 0.5536. The combined model loses. The comparison table that supposedly supports the headline omits the price-only CNN entirely, which looks like cherry-picking the table rather than the model.\n\nThe paper is a legitimate extension of an established line of work, not a breakthrough. The sentiment embedding is novel to this paper, and the empirical setup is reasonable in scope: S&P100 stocks, 2010-2020, multiple reward functions and policy architectures. But the extension is incremental, and the claimed edge is small even if it existed. The best RL policy still underperforms a trivial equal-weight buy-and-hold (0.1811 net profit, 0.5952 Sharpe). There are no error bars, no multiple-seed runs, and the best model is selected on test-period metrics, which inflates the risk of overfitting to the test window.\n\nThe biggest soft spot is the mismatch between the prose and the numbers. Section 4.4 says the combined-data CNN with profit reward is the best across all experiments; Section 4.5 repeats that claim and adds that it has the highest Sharpe and Sortino. Both statements are false for the price-only CNN, which beats it on all three metrics. That is not a subtle issue. It is the load-bearing result of the paper, and it fails on the paper's own evidence. The news data issue is secondary: with only 84 of 100 tickers covered and a median of 905 reporting dates, the imputed sentiment tensor is mostly a product of the decay rule, and the paper acknowledges this. But even if the data were perfect, the headline claim would still be unsupported.\n\nWho should read this? Someone working on RL portfolio allocation might look at it as a cautionary example of how to report baselines and why single-seed results are not enough. It is not a result to build on. My recommendation: do not send this to peer review in its current form. The internal contradiction alone warrants a reject or a major-revision request before any referee spends time on it. The authors should fix the comparison, add error bars, and decide whether the multimodal claim survives contact with their own tables.","headline":"A careful EIIE re-implementation with sentiment channels whose own tables contradict the paper's headline claim that combining data sources improves performance.","tokens_in":14570,"tokens_out":1593,"would_cite":false,"duration_ms":17636,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["91G10","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"A deep RL trader using SEC and news sentiment beats most S&P100 benchmarks in backtests.","keywords":["reinforcement learning","portfolio optimization","multimodal data","sentiment analysis","SEC filings","news headlines","EIIE policy","Differential Sharpe Ratio"],"falsifier":"Retrain the best configuration (CNN EIIE, combined data, profit reward) with the news and SEC channels replaced by constant zeros, and separately with the same channels filled with random noise matched to the observed sentiment distribution; if test-period net profit, Sharpe ratio, and Sortino ratio stay within noise of the reported 0.166, 0.554, and 0.665, then the sentiment channels are not carrying the claimed signal.","tokens_in":13443,"feed_emoji":"📈","tokens_out":6720,"duration_ms":61497,"temperature":0.7,"pith_summary":"This paper tries to show that feeding a deep reinforcement learning portfolio agent both price data and sentiment signals from SEC filings and news headlines improves trading performance on S&P100 stocks. Its central claim is that the best configuration — a CNN-based EIIE policy trained on the combined price-plus-sentiment state with a simple profit reward — posts the highest net profit, Sharpe ratio, and Sortino ratio among all trained agents and reference benchmarks in the 2018–2019 test period, with only Equal Buy-and-Hold doing better. The authors also argue that the profit reward is far easier for these agents to learn than the Differential Sharpe Ratio, and that compact CNN/RNN policies overfit less than a full-tensor MLP. A sympathetic reader would care because the result suggests a practical recipe: regular alternative data, a compact policy, and a simple reward can outperform established online portfolio selection methods on a large-cap universe.","feed_headline":"News-fed AI trader beats most S&P100 benchmarks","feed_subtitle":"A CNN policy using price, SEC, and news sentiment posts the top risk-adjusted returns in 2018–19 backtests.","key_machinery":"The central object is the state tensor $S=(S^*,\\delta)$ that adds alternative-data channels to historical prices: for each asset and each day in the lookback window, the tensor stores high, low, and close prices together with a SEC sentiment score (Loughran–McDonald word proportions) and a news sentiment score from FinBERT mapped through the $\\tanh(\\text{positive}/\\text{negative}/\\text{neutral})$ embedding of Equation (1). The policy is the Ensemble of Identical Independent Evaluators (EIIE) from the reference framework, where one CNN or RNN feature extractor scores each asset independently and a softmax converts scores into portfolio weights; the MLP baseline applies the same tensor to a single fully connected network. Rewards are either period profit or the Differential Sharpe Ratio, and transaction costs enter through the fixed-point update factor $\\mu_t$. Sparse sentiment dates are forward-filled by exponential decay $y=a(1-\\gamma)^t$ with $\\gamma\\approx0.8$.","core_discovery":"The paper's central discovery, stated on its own terms, is that the best-performing model across all three experiment sets is the CNN EIIE model trained on the combined dataset (price, SEC sentiment, and news sentiment) with the Profit reward. In the 2018–2019 out-of-sample test this agent reports a net profit of 0.166, Sharpe ratio of 0.554, Sortino ratio of 0.665, and max drawdown of 0.195, which beats the S&P500 index as well as the OLMAR and WMAMR benchmarks; only the Equal Buy-and-Hold portfolio has higher net profit and risk-adjusted ratios. The paper interprets this as evidence that multimodal company sentiment enriches the state representation, that SEC data in particular is regular and reliable enough to help learning, and that the choice of reward function matters more than model complexity once alternative data is present.","pith_inferences":["Beyond the paper's claims: the reported gains could be an artifact of the forward-filling rule, since the median ticker has actual news on only 905 trading dates; retraining with unfilled or shuffled sentiment would isolate the true signal.","The sentiment embedding in Equation (1) is an ad hoc ratio that ignores the absolute level of neutral probability; a calibrated alternative such as the raw FinBERT positive-minus-negative probability would test how much the embedding form matters.","The conclusions are drawn from one 2018–2019 test window with 1% transaction costs; extending the same architecture to other periods, universes, or cost levels would show whether the recipe generalizes.","A testable extension is to use the same state tensor with a learned attention or gating layer over the sentiment channels; if the model learns to ignore sentiment on days after long gaps, that would corroborate the paper's interpretation that data regularity drives the improvement."],"forward_implications":["A practical takeaway is that adding regularly reported alternative data (SEC sentiment) to a compact EIIE policy improves returns under both profit and Differential Sharpe rewards.","The profit reward is a more reliable training signal than the Differential Sharpe Ratio for these RL agents, so practitioners can expect better out-of-sample results from the simpler reward.","CNN and RNN EIIE policies are preferable to a full-tensor MLP because their smaller parameter counts reduce overfitting on the price-plus-sentiment state.","Combining news with SEC data gives the strongest configuration, while news alone is too sparse and irregular to help, especially under the harder reward.","All trained agents in the comparison outperform the OLMAR and WMAMR online portfolio benchmarks over the test period."],"supporting_citations":[{"why":"Introduces the Differential Sharpe Ratio reward that the paper trains against.","marker":"[1]"},{"why":"Supplies the state-augmentation architecture that adds news embeddings to price tensors and is the closest prior structure.","marker":"[3]"},{"why":"Provides the EIIE CNN/RNN policy framework and the transaction-cost fixed-point update used in every agent.","marker":"[4]"},{"why":"Supplies CRSP price data and the S&P500 index benchmark.","marker":"[7]"},{"why":"Defines the 10-K/10-Q MD&A sections whose text drives the SEC sentiment scores.","marker":"[8]"},{"why":"Provides EDGAR access to the SEC filings used for the SEC sentiment channel.","marker":"[9]"},{"why":"Supplies the daily financial headline dataset used for news sentiment and embeddings.","marker":"[10]"},{"why":"Defines the OLMAR benchmark the agents are compared against.","marker":"[13]"},{"why":"Defines the WMAMR benchmark the agents are compared against.","marker":"[14]"}],"fun_headline_variants":["News and SEC sentiment help RL agent beat S&P500","Profit-rewarded CNN tops S&P500 and online baselines","Multimodal RL approach outperforms index and classic tactics","Combined sentiment data boosts portfolio RL backtests","CNN EIIE with multimodal data beats S&P500 in 2018-19"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that exponential-decay forward-filling turns sparse SEC filings and news headlines into a faithful daily sentiment signal, because most of the sentiment values the agent sees are imputed rather than measured.","fun_headline_variants_meta":{"raw":{"variants":["News and SEC sentiment help RL agent beat S&P500","Profit-rewarded CNN tops S&P500 and online baselines","Multimodal RL approach outperforms index and classic tactics","Combined sentiment data boosts portfolio RL backtests","CNN EIIE with multimodal data beats S&P500 in 2018-19"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000244,"raw_usage":{"total_tokens":1487,"prompt_tokens":857,"completion_tokens":630,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":473,"completion_tokens_details":{"reasoning_tokens":553}},"tokens_in":473,"tokens_out":630,"duration_ms":6415,"temperature":1.0,"reasoning_tokens":553,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:37:16.962492+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the best configuration (CNN EIIE, combined data, profit reward) with the news and SEC channels replaced by constant zeros, and separately with the same channels filled with random noise matched to the observed sentiment distribution; if test-period net profit, Sharpe ratio, and Sortino ratio stay within noise of the reported 0.166, 0.554, and 0.665, then the sentiment channels are not carrying the claimed signal.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides EDGAR access to the SEC filings used for the SEC sentiment channel."},{"cited_title":"Deep Reinforce- ment Learning for Optimal Portfolio Allocation: A Comparative Study with Mean-Variance Optimization","cited_arxiv_id":null,"evidence_quote":"Introduces the Differential Sharpe Ratio reward that the paper trains against."},{"cited_title":"CRSP daily stocks, 2010-2024","cited_arxiv_id":null,"evidence_quote":"Supplies CRSP price data and the S&P500 index benchmark."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the 10-K/10-Q MD&A sections whose text drives the SEC sentiment scores."},{"cited_title":"Daily finaincial news for 6000+ stocks, 2020","cited_arxiv_id":null,"evidence_quote":"Supplies the daily financial headline dataset used for news sentiment and embeddings."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the OLMAR benchmark the agents are compared against."},{"cited_title":"∞X t=0 γtRt s0 = s # Qπ(s, a) = E π","cited_arxiv_id":null,"evidence_quote":"Defines the WMAMR benchmark the agents are compared against."}],"review_version":1}