{"id":"50ca7688-f91a-4f06-a62c-391b3d0ff4df","arxiv_id":"2507.18417","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"A preference-optimized LLM for financial sentiment beats supervised fine-tuning on a combined benchmark and, via a softmax-based score, appears to drive a profitable long-short portfolio in a backtest.","lead":"The paper presents FinDPO, a language model tuned with preference optimization to classify financial news sentiment as positive, negative, or neutral. It claims the model beats older sentiment models on average and, after converting its confidence scores into trading signals, would earn 67% per year in a backtest even after trading costs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The trading backtest aligns day-t news sentiment with day-t returns (Eqs. 3–5) without any stated publication-time lag; if articles arrive after the close, those same-day returns cannot be traded, so the 67% annual return and Sharpe 2.0 are not realizable.","rationale":"The reader identified the same load-bearing assumption I find: the backtest aligns day-t news sentiment with day-t returns without any evidence that the signal is available before the close. This is not a minor implementation detail; it is the difference between a realizable trading strategy and a look-ahead artifact. The headline claim in the abstract is explicitly framed as the first sentiment-based approach to maintain 67% annual returns and Sharpe 2.0 under 5 bps costs, so the timing assumption is central. The underlying DPO construction and the FinSFT-versus-FinDPO classification comparison are plausible and potentially useful, but they do not support the trading claim. No code, data, timestamps, or error bars are provided to verify the portfolio simulation. I see no need to manufacture additional objections; the timing concern alone is decisive for the paper's central claim. If the authors can supply article timestamps and rerun the backtest with a one-day lag, the claim could be reassessed, but as presented the result is unsupported.","tokens_in":12567,"tokens_out":3661,"duration_ms":41161,"concrete_test":"Re-run the portfolio construction exactly as in §4.2, but replace the same-day returns in Eq. (5) with next-day returns: for sentiment signal S_t, use r_Long(t+1) and r_Short(t+1) from the market data (equivalently, lag entry by one trading day). Apply the same 35% long/short ranking, equal weights, and the 5 bps turnover cost model of Eqs. (11)–(12). If the annualized return at 5 bps falls materially below 67% or the Sharpe ratio below 2.0, the same-day alignment is the source of the headline result and the trading claim is not supported. Also report the distribution of article publication timestamps relative to US market hours; if a material fraction of articles are timestamped after 16:00 ET, same-day execution is impossible.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim is the simulated trading result, and its weakest link is timing. In §4.2, daily sentiment S_t is computed from articles published on day t (Eq. 2), companies are ranked by S_t, and Eq. (5) sets the portfolio return for day t to r_Long(t) - r_Short(t), with r_Long and r_Short taken from the same day's market data (Eqs. 3–4). The paper nowhere reports article publication timestamps, processing latency, or a lag between signal generation and execution. For the backtest to be realizable, every article used in S_t must be published and processed before the market close on day t, and the position must be established before the close-to-close return is determined. If any articles are published after the close or during non-trading hours, their sentiment can only be acted upon the next trading day; using same-day returns then credits the strategy with price moves that occurred before the signal existed. Since the paper provides no timestamp statistics and no code or data, the most natural reading is a look-ahead alignment. This concern directly invalidates the abstract's headline numbers if it lands; the classification F1 of 0.846 is a separate claim and does not by itself establish tradeable returns.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FinDPO, a financial sentiment analysis model based on Direct Preference Optimization (DPO) applied to Llama-3-8B-Instruct, and a 'logit-to-score' converter that maps discrete sentiment labels to continuous scores for long-short portfolio construction. The authors report a weighted F1 of 0.846 on three sentiment benchmarks, claiming an 11% improvement over FinGPT v3.3, and a backtest on S&P 500 news from 2015–2021 that yields an annualized return of 66.64% with a Sharpe ratio of 2.03 at 5 bps transaction costs. The paper also emphasizes parameter-efficient training on a single A100 GPU.","tokens_in":12837,"tokens_out":5197,"duration_ms":49162,"significance":"If the claims hold, FinDPO would be a useful contribution to financial NLP by demonstrating that DPO-based alignment can improve sentiment classification over supervised fine-tuning, and the logit-to-score converter offers a practical way to use causal LLM probabilities in portfolio ranking. The low-resource training setup is also a strength. However, the central trading result rests on a timing assumption that is not stated, and the classification comparison with published baselines is not controlled, so the headline numbers are not currently supported.","major_comments":[{"comment":"The backtest aligns day-t news sentiment (Eq. 2) with day-t stock returns (Eqs. 3–5) without specifying any lag between article publication and portfolio execution. If any article used in S_t is published after the market close on day t, the same-day return cannot be realized, and the reported 66.64% annualized return and Sharpe ratio of 2.03 at 5 bps (Table 4) are not achievable. The paper must either provide timestamp statistics demonstrating that all signals are available before the close, or re-run the backtest with a one-day lag and report whether the results survive; without this, the abstract's primary trading claim is unsupported.","section":"§4.2, Eqs. (3)–(5)"},{"comment":"The comparison with FinGPT v3.3 and other published baselines is not apples-to-apples. FinDPO is trained on 80% of the same three datasets and evaluated on the remaining 20%, while FinGPT v3.3's scores are taken from prior work with unknown training and evaluation splits. Moreover, FinDPO trails FinGPT v3.3 on FPB (0.865 vs 0.879) and TFNS (0.872 vs 0.903), and the claimed 11% average improvement is driven entirely by the NWGI column (0.833 vs 0.643). The 'state-of-the-art by 11%' claim requires a controlled comparison in which all baselines are retrained on the same splits under the same protocol.","section":"§5.1, Table 2"}],"minor_comments":[{"comment":"The header 'No. of articles pre processing' contains a typo; it should read 'pre-processing'.","section":"Table 1"},{"comment":"The text states 'the top 35% of companies in terms of performance were allocated to long positions', but the ranking is based on sentiment scores, not historical performance; this wording should be corrected to avoid confusion.","section":"§4.2, Portfolio Construction"},{"comment":"The definition of annualized return R_p as (1/N) Σ r_log(i) · 252 is the annualized log return, not the simple annualized return; the authors should clarify whether the reported values are log or simple returns, as this affects comparability with standard finance metrics.","section":"§4.2, Eq. (7)"},{"comment":"The paper reports results for a single historical backtest period without statistical significance tests or confidence intervals for the Sharpe ratios; given the autocorrelation of daily returns, a bootstrap or subsample analysis would strengthen the claim of robustness.","section":"§5.3"}],"recommendation":"major_revision","confidential_remarks":"The timing issue in §4.2 is the most serious concern; if the authors cannot verify that all news used in daily sentiment was published before the market close, the trading results are likely invalid. The uncontrolled comparison with FinGPT v3.3 further weakens the classification claim. I recommend major revision, but if the timing issue cannot be resolved, rejection may be the appropriate outcome."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: the classification half of this paper is coherent and the FinSFT/FinDPO comparison is genuinely informative, but the 67% annualized return and Sharpe 2.0 come from a backtest that aligns day-t news sentiment with day-t returns and never states a lag. That is a look-ahead design unless every article was published and processed before the close, which the paper gives no evidence for. As written, the headline financial claim should not be believed.\n\nWhat is actually new: this is the first time DPO has been applied to financial sentiment classification, and the idea of turning ground-truth labels and model mistakes into preference pairs is reasonable. The logit-to-score converter is just a temperature-scaled softmax over first-token logits, but applying it to rank assets in a long-short portfolio is a sensible way to make a generative LLM usable for trading. The training cost story (LoRA on a single A100, 4.5 hours) is also a useful practical datapoint. The FinSFT versus FinDPO comparison, both on Llama-3-8B, suggests the DPO step adds something beyond instruction tuning, and that result is the most trustworthy piece of evidence in the paper.\n\nWhere it falls down: first, the backtest timing. Equations 3 through 5 compute daily sentiment from articles published that day and then set the portfolio return to the same day's long-minus-short return. No publication timestamps, no processing latency, no one-day lag. If any article arrives after the close, the strategy is using returns that happened before the signal existed. Since the authors provide no code and no data, the natural reading is look-ahead. This flaw directly invalidates the abstract's 67% and Sharpe 2.0. Second, the state-of-the-art comparison is not controlled. FinDPO is evaluated on a 20% split of the same datasets it trained on, while FinGPT v3.3 and FinLlama are cited numbers from prior papers whose training overlap is unknown. The claimed 11% average improvement is therefore not an apples-to-apples comparison. Third, there are no standard deviations, no repeated runs, and no code release, so we cannot assess whether any of the differences are meaningful. The paper also frames label-derived pairs as \"human preference alignment,\" which oversells what was done, but that is more spin than substance.\n\nWho this is for: someone working on financial LLMs or sentiment-based trading would want to see the FinSFT/FinDPO result, but only after the authors fix the timing and redo the comparison against matched baselines. The core idea is worth a serious look, which is why I would not desk-reject it. But my recommendation to the editor would be reject in current form: the central financial claim is unsupported, and the classification claim needs a more honest baseline. If the authors release code, add a lag, and rerun, a revised version could be a solid paper.","headline":"Plausible DPO-for-sentiment idea, but the headline trading returns rely on a same-day backtest that looks look-ahead; the numbers are not credible as written.","tokens_in":13379,"tokens_out":3142,"would_cite":false,"duration_ms":32832,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A preference-tuned financial LLM beats supervised fine-tuning and stays profitable after realistic trading costs.","keywords":["financial sentiment analysis","direct preference optimization","large language models","algorithmic trading","portfolio construction","long-short portfolio","transaction costs","logit-to-score conversion"],"falsifier":"Re-run the long-short backtest using day t sentiment to trade at day t+1's open or close, or restrict the news corpus to articles timestamped before the market's close; if the 5 bps annualized return and Sharpe ratio of 2.0 are no longer achieved, the real-time tradeability claim is falsified.","tokens_in":12348,"feed_emoji":"📈","tokens_out":9947,"duration_ms":91018,"temperature":0.7,"pith_summary":"This paper argues that supervised fine-tuning, the standard way to adapt large language models to financial sentiment analysis, can be replaced by Direct Preference Optimization (DPO) with better results. The authors introduce FinDPO, a DPO-aligned model, and report that it beats existing fine-tuned models by 11% on average weighted F1 across three financial news benchmarks. They also convert the model's discrete sentiment labels into continuous scores by applying a softmax to the first-token logits and calibrating with temperature scaling, which lets a causal LLM be ranked and traded in a long-short portfolio for the first time. In backtests from 2015 to 2021, FinDPO is the only sentiment method they test that keeps substantial positive performance under 5 basis-point transaction costs: 67% annualized returns and a Sharpe ratio of 2.0. The paper's claim matters because it suggests preference alignment, rather than a larger base model, is what improves generalization in financial language, and that the improvement is large enough to survive real trading frictions.","feed_headline":"Keeps 67% yearly returns after 5 bps trading costs","feed_subtitle":"A preference-tuned finance LLM stays profitable in a long-short backtest where all other sentiment methods fail.","key_machinery":"The load-bearing mechanism is the pairing of DPO alignment with a 'logit-to-score' converter. DPO replaces the cross-entropy objective of SFT with the preference loss in Equation 1, which increases the probability of the correct sentiment label and decreases the probability of a sampled wrong label relative to the frozen reference model; this explicit penalty is what the paper credits for better generalization to unseen financial text. The logit-to-score converter then takes the logits of the first generated token, applies a softmax over the three sentiment classes, and applies temperature scaling optimized on the training set, because preference-aligned models tend to output degenerate probabilities of 1.0 and 0.0. The resulting continuous, calibrated score is what lets the model rank stocks daily for the long-short portfolio, connecting generative LLM outputs to portfolio weights.","core_discovery":"The paper's central claim is that post-training a finance LLM with DPO rather than SFT produces a sentiment classifier that is both more accurate and more tradeable. On the classification side, FinDPO reports an average weighted F1 of 0.846 over the FPB, TFNS, and NWGI benchmarks, 11% above FinGPT v3.3, and it also outperforms its own instruction-tuned counterpart at 0.771, which the paper takes as evidence that the gain comes from DPO rather than from the base model. On the trading side, the paper claims that by converting the first generated token's logits into a temperature-calibrated probability score, a causal LLM can rank 417 S&P 500 stocks daily, placing the top 35% long and the bottom 35% short. In simulated backtests with daily rebalancing, FinDPO achieves an annualized return of 111.78% with a Sharpe ratio of 3.41 at zero cost, and remains at 66.64% annualized with a Sharpe ratio of 2.03 after 5 bps transaction costs, where all other tested methods turn negative or near-zero. The paper presents this as the first demonstration that a sentiment-based strategy built on a causal LLM can remain profitable under realistic costs.","pith_inferences":["The paper leaves the same-day timing assumption untested; a natural next test is to trade day t sentiment at day t+1's open, which would show whether the scores predict future returns or merely track same-day price moves.","Because the preference pairs are synthesized from ground-truth labels rather than elicited from human annotators, the recipe could extend to preference data built from model rationales or analyst commentary.","The portfolio experiment covers only 2015-2021 S&P 500 constituents; testing the ranking signal on post-2021 data, smaller caps, or international markets would reveal whether the performance is period-specific.","Applying the same logit-to-score conversion to other causal LLMs would separate the contribution of DPO alignment from the contribution of score calibration."],"forward_implications":["If FinDPO's classification result holds, DPO becomes a practical substitute for SFT in financial NLP, promising better accuracy and generalization without a larger base model.","If the backtest holds up, causal LLMs do not need a classification head to drive portfolios; their generative logits are enough to produce ranking scores.","The 5 bps result implies that sentiment signals can be strong enough to pay for realistic trading costs, making LLM-based sentiment a plausible input to systematic strategies.","The reported training budget, about 42 million trainable parameters on a single GPU, suggests preference alignment is cheap enough for finance teams to run on their own data.","The logit-to-score converter plus temperature scaling gives a reusable recipe for turning any causal LLM's discrete sentiment output into a continuous signal."],"supporting_citations":[{"why":"Supplies the DPO loss and preference-optimization framework that FinDPO is built on.","marker":"[20]"},{"why":"Provides the FinLlama baseline and the long-short portfolio construction protocol, including the 35% long/short equal-weight allocation.","marker":"[10]"},{"why":"Provides the FinGPT v3.3 benchmark that FinDPO compares against on weighted F1.","marker":"[24]"},{"why":"Supplies the Financial PhraseBank labeled news dataset used to build preference pairs for training and evaluation.","marker":"[17]"},{"why":"Supplies the Twitter Financial News dataset used to build preference pairs for training and evaluation.","marker":"[25]"},{"why":"Supplies the GPT-labeled financial news dataset used to build preference pairs for training and evaluation.","marker":"[16]"},{"why":"Supplies temperature scaling, the calibration method that converts overconfident logits into usable sentiment scores.","marker":"[7]"},{"why":"Supplies LoRA, which reduces trainable parameters to 0.52% and makes single-GPU DPO training feasible.","marker":"[8]"}],"fun_headline_variants":["Preference-tuned finance AI outearns SFT, keeps 67% after costs","DPO beats SFT for finance sentiment: 67% returns, Sharpe 2.0","First sentiment model to stay green in backtest: DPO-tuned LLM, 67% annual","Logit-to-score trick turns sentiment LLM into 67% annual strategy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The backtest assumes that a company's news on day t is available in time to trade that company's day-t return, with no delay between publication, sentiment scoring, and execution.","fun_headline_variants_meta":{"raw":{"variants":["Preference-tuned finance AI outearns SFT, keeps 67% after costs","DPO beats SFT for finance sentiment: 67% returns, Sharpe 2.0","First sentiment model to stay green in backtest: DPO-tuned LLM, 67% annual","Logit-to-score trick turns sentiment LLM into 67% annual strategy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000779,"raw_usage":{"total_tokens":3519,"prompt_tokens":1097,"completion_tokens":2422,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":713,"completion_tokens_details":{"reasoning_tokens":2326}},"tokens_in":713,"tokens_out":2422,"duration_ms":18088,"temperature":1.0,"reasoning_tokens":2326,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:12:40.665496+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the long-short backtest using day t sentiment to trade at day t+1's open or close, or restrict the news corpus to articles timestamped before the market's close; if the 5 bps annualized return and Sharpe ratio of 2.0 are no longer achieved, the real-time tradeability claim is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the FinLlama baseline and the long-short portfolio construction protocol, including the 35% long/short equal-weight allocation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the FinGPT v3.3 benchmark that FinDPO compares against on weighted F1."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Twitter Financial News dataset used to build preference pairs for training and evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the GPT-labeled financial news dataset used to build preference pairs for training and evaluation."},{"cited_title":"Weinberger","cited_arxiv_id":null,"evidence_quote":"Supplies temperature scaling, the calibration method that converts overconfident logits into usable sentiment scores."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies LoRA, which reduces trainable parameters to 0.52% and makes single-GPU DPO training feasible."}],"review_version":2}