{"id":"4bba4b55-66b1-41c2-a305-38b0a31a209b","arxiv_id":"2501.09636","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"An LLM router dynamically selects between two feedforward experts for stock movement prediction, reporting improved backtested returns on MSFT and AAPL.","lead":"This paper proposes replacing the learned router in a mixture-of-experts stock trading model with a large language model that reads five days of price features and news headlines to choose between an optimistic and a pessimistic expert network. On Microsoft and Apple data, the authors report higher total return and Sharpe ratio than the simple baselines they tested.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No ablation isolates the LLM router's contribution; the reported gains could arise from the two-expert split or test-period dynamics, so the central 'LLM routing causes outperformance' claim is unsupported.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing gap: the LLM router is never ablated or validated, so the causal attribution of the performance gain to LLM-based routing is unsupported. My reading of the methodology confirms that the router's only output is a binary label used to split training data between two identical-structure FNN experts, and the LLM itself does not make the final trading prediction. This makes the missing router ablation more central than the zero-cost execution issue, since the relative claim over baselines depends on the routing mechanism, not on absolute returns. I agree with the REJECT verdict: the headline claim is not established by the evidence presented. A concrete ablation and router-accuracy check could settle the matter, so the verdict should remain REJECT unless such evidence is added. The unrelated inserted heading and incomplete appendices are secondary but reinforce that the manuscript is not carefully verified.","tokens_in":7509,"tokens_out":3450,"duration_ms":40672,"concrete_test":"Retrain LLMoE with four router variants on the same 80/20 split: (1) random labels with the same class frequencies as the LLM router; (2) a constant 'optimistic' label; (3) a lexicon-based sentiment rule applied to the same news headlines; (4) oracle labels using the realized next-day direction. Recompute Table 1 for each variant. If random or trivial routers achieve Sharpe Ratio and Total Return within the LLMoE confidence intervals, or if the LLM router's test-period label accuracy is not significantly above 50%, the central claim fails. Additionally, compute per-trade turnover and apply a conservative 10 basis point round-trip transaction cost to all models to check whether relative rankings persist.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that replacing a neural router with an LLM router drives the large improvements in Table 1 (e.g., MSFT TR 65.44 vs MLP 33.92; SR 2.14 vs 1.21). But the LLM router only assigns one of two labels, Optimistic or Pessimistic, and those labels are used to train two feedforward experts on 55 numerical features. The final prediction is just the selected expert's output; the LLM's textual understanding enters only through the binary routing decision. The paper never validates whether these labels are accurate, informative, or better than a trivial rule. It reports no router accuracy against realized next-day direction, no comparison against random labels, always-optimistic routing, or a simple sentiment lexicon, and no ablation that removes the LLM while keeping the two-expert architecture. Without such an ablation, the observed outperformance cannot be attributed to LLM-based routing: it could come from the two-expert architecture, from the all-in-all-out backtest, from the particular 2014-2016 test window, or from spurious correlations memorized by the pretrained Llama3.2 model. The zero-transaction-cost assumption is also unrealistic, though it is secondary because it affects baselines similarly. The inserted unrelated heading 'Temporal Graph Networks for Graph Anomaly Detection in Financial Networks' and the missing detailed appendices further indicate the manuscript is not complete enough to support the claimed state-of-the-art result.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LLMoE, a mixture-of-experts framework in which a pretrained LLM (Llama3.2) serves as a router. The router classifies a five-day rolling window of 55 numerical features plus news headlines into Optimistic or Pessimistic, and the corresponding label selects one of two feedforward expert networks trained on those labels. The selected expert's binary prediction drives an All-in All-out trading strategy. The authors evaluate on MSFT and AAPL over 2006-2016, with a test period of roughly 2014-2016, comparing against LightGBM, MLP, LSTM, DNNE, and MoE with 2 and 10 experts. They report that LLMoE achieves the highest Total Return, Sharpe Ratio, and Calmar Ratio on both datasets, with over 25% improvements claimed in the conclusion.","tokens_in":7798,"tokens_out":4573,"duration_ms":50507,"significance":"The idea of replacing a learned neural router with an LLM router to combine numerical and textual data is timely and, if properly validated, would be a useful contribution to the growing literature on LLM-based trading and interpretable MoE. The paper gives a concrete architecture and reports positive results on two real-world datasets, and the explicit reasoning output of the router is a promising step toward interpretability. However, the evaluation does not isolate the router's contribution, omits the closest published baselines, uses a cost-free All-in All-out backtest, and contains clear signs of incompleteness. These issues prevent the current manuscript from substantiating its state-of-the-art claim.","major_comments":[{"comment":"The central claim that LLMoE outperforms baselines because of LLM-based routing is not supported by any ablation isolating the router. The router's Optimistic/Pessimistic labels are never validated: there is no accuracy of the labels against realized next-day direction, no comparison with an always-optimistic router, random labels, or a simple sentiment-lexicon rule, and no variant that keeps the two-expert architecture but replaces the LLM router with a static or random router. Without such an ablation, the large gains in Table 1 (e.g., MSFT TR 65.44 vs. 33.92 and SR 2.14 vs. 1.21) cannot be attributed to the LLM router; they could equally arise from the two-expert split, the particular test window, or the all-in-all-out execution.","section":"Experiment Results, Table 1"},{"comment":"The paper explicitly cites AlphaMix, TradExpert, and MIGA as related state-of-the-art MoE works, yet none of these is included in Table 1. Since the conclusion claims superiority over 'state-of-the-art MoE models,' the comparison set (LightGBM, MLP, LSTM, DNNE, MoE-2, MoE-10) is insufficient to support that claim. The absence of AlphaMix is especially consequential because the Related Work describes it as the closest competitor and the manuscript's own MoE baselines are simple static routers without the three-stage design discussed for AlphaMix.","section":"Baseline Models and Appendix"},{"comment":"The All-in All-out backtest assumes zero transaction costs, slippage, and market impact, and the paper does not report turnover or a cost-sensitivity analysis. For a strategy that moves between fully invested and fully liquidated on daily predictions (TR 65.44% on MSFT and 31.43% on AAPL over roughly two years), realistic costs can materially reduce or erase the reported return advantage. Because the headline result is a return comparison, this is a load-bearing limitation rather than a minor execution detail.","section":"Trading Algorithm Generation and Experiment Results"},{"comment":"The paper states that experiments were repeated with ten random seeds and confidence intervals were computed, but no statistical significance test is reported. The phrase 'significantly outperformed' in the results section is not backed by a pairwise test, a bootstrap on daily returns, or any multiple-comparison correction. Given that the paper's main claim is outperformance, a significance test is necessary to interpret the reported mean and standard-deviation intervals.","section":"Other Experimental Details"},{"comment":"The manuscript appears incomplete. The text 'Temporal Graph Networks for Graph Anomaly Detection in Financial Networks' is inserted in the middle of the Expert Prediction subsection, and several details promised in the appendix ('Appendix - Experiment Setup Details', 'Appendix - Expert Model Architecture') are either missing or under-specified, including baseline hyperparameters, the exact LLM prompt and decoding settings, and the handling of days with missing news. These omissions prevent reproduction and further undermine the completeness needed for a state-of-the-art claim.","section":"Expert Prediction and Appendix"}],"minor_comments":[{"comment":"Table 1 and the surrounding text contain a typo in the metric name: 'V olatility' should be 'Volatility'.","section":"Experiment Setup"},{"comment":"The reference list contains duplicate entries for Yoo et al. (2021a and 2021b), which appear to describe the same paper.","section":"References"},{"comment":"The related-work section refers to AlphaMix by name, but the citation is given only as Sun et al. 2023; the text should make the AlphaMix attribution explicit.","section":"Related Work"},{"comment":"The feature table defines zd5 explicitly, but the formulas for zd10, zd15, zd20, zd25, and zd30 are only implied; a generic formula for zd_n would be clearer.","section":"Table 3"},{"comment":"The router prompt and decoding settings are not described; without these, the router component is not reproducible.","section":"Implementation and Evaluation, Router"},{"comment":"The training period is given as 2006-12-07 to 2014-12-02 in the text, while Table 2 lists the start date as 06/12/01; these dates should be reconciled.","section":"Appendix, Dataset Splits"}],"recommendation":"reject","confidential_remarks":"The manuscript reads like an early draft: the inserted unrelated heading, the placeholder references to missing appendices, and the complete absence of a router ablation make it unsuitable for the stated venue in its current form. I would not rule out a substantially revised submission that adds an ablation, validates the router labels, includes the closest baselines, and performs a cost-sensitivity analysis, but as submitted the central claim is unsupported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The paper's idea is straightforward: use an LLM instead of a learned router in a MoE, with two experts for optimistic and pessimistic regimes. That's a reasonable extension of both MoE and LLM work, and the example of the router giving a balanced 'cautiously optimistic' read is a genuinely nice illustration of the interpretability selling point. The paper also reports ten seeds and several risk-adjusted metrics, which is more careful than a lot of trading papers.\n\nThe problem is that the central claim—that the LLM router causes the outperformance—has no direct evidence. The router assigns a binary label, and the two feedforward experts are trained on those labels. There's no ablation that keeps the two-expert architecture but replaces the LLM with something trivial: always-optimistic routing, random labels, a simple sentiment rule, or the original static router. Without that, the 65% vs 34% total return on MSFT could just as easily come from the two-expert split, the particular 2014-2016 test window, or the zero-transaction-cost all-in/all-out backtest. The missing comparison to the cited MoE trading baselines—TradExpert, MIGA, AlphaMix—makes it worse; those are the papers you'd need to beat to claim SOTA.\n\nThe secondary issues are real but less load-bearing. Zero transaction costs are unrealistic but they affect all models roughly equally. Two stocks and a two-year test period limit generalizability. And the stray 'Temporal Graph Networks...' heading in the middle of the method section suggests the manuscript wasn't carefully assembled.\n\nSo: the direction is worth pursuing, and I'd send it to review with a strong request for ablation studies and proper baselines. But the current evidence doesn't support the state-of-the-art claim. I wouldn't cite it yet, and I'd use it as a reading-group example of why router attribution requires an ablation.","headline":"Promising idea, unsupported central claim: no ablation isolates the LLM router's contribution, and closest baselines are missing.","tokens_in":8322,"tokens_out":2457,"would_cite":false,"duration_ms":25985,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLMoE replaces the learned router with an LLM, claiming gains of 25%+ in risk-adjusted trading returns.","keywords":["mixture of experts","LLM routing","stock movement prediction","multimodal trading","all-in all-out trading","risk-adjusted returns"],"falsifier":"Train the same two experts but route with a constant 'optimistic' label, or with a simple news-sentiment rule, and compare returns to LLMoE; if either matches the reported Sharpe and total return, the LLM router is not the cause. Also recompute the all-in/all-out backtest with realistic transaction costs and slippage: if the 31–65% total returns collapse to baseline levels, the trading rule, not the routing, explains the result.","tokens_in":7313,"feed_emoji":"📈","tokens_out":5329,"duration_ms":53490,"temperature":0.7,"pith_summary":"The paper proposes LLMoE, a mixture-of-experts architecture in which a large language model, rather than a trained neural router, decides each day which of two specialist predictors—one for optimistic periods and one for pessimistic periods—should forecast the next day's price direction. It argues that the LLM's ability to read both engineered price features and news headlines gives it a global, interpretable view that static routers lack. On ten-year MSFT and AAPL backtests using an all-in/all-out trading rule, LLMoE reports the highest total return, Sharpe ratio, and Calmar ratio among all compared models, including conventional two- and ten-expert MoE baselines, with gains over 25% in key risk-adjusted metrics. A sympathetic reader would take the paper's intended contribution as evidence that LLM-driven routing is a viable and flexible replacement for learned routers in financial MoE systems.","feed_headline":"LLM router picks the trading expert and beats static MoE by 25%+","feed_subtitle":"Using prices plus news headlines, an LLM decides which specialist model to trust each day, lifting Sharpe and total return.","key_machinery":"The load-bearing mechanism is the LLM router: a pretrained large language model turns each five-day window into a descriptive string of numeric features and news text, then classifies it as Optimistic or Pessimistic. The label routes the window to one of two feedforward networks, and the choice is made by the argmax of the LLM's predicted probabilities, so no gating network is trained. The LLM's natural-language reasoning is produced as a byproduct for interpretability, while the all-in/all-out trading rule converts the chosen expert's binary prediction into a position.","core_discovery":"LLMoE's central claim is that expert selection in a trading MoE should be performed by a large language model that reads a rolling five-day window of engineered price features plus the corresponding news headline, and emits an Optimistic or Pessimistic label with a natural-language explanation. Two feedforward experts are trained separately on the windows labeled by the router, and the selected expert's next-day direction prediction drives a binary all-in/all-out position. On MSFT and AAPL datasets spanning 2006–2016, the paper reports that LLMoE achieves the best total return (65.44% and 31.43%), Sharpe ratio (2.14 and 1.17), and Calmar ratio among every baseline, exceeding the best static MoE by more than 25% on the headline risk-adjusted metrics. The intended discovery is that a pretrained LLM router with world knowledge yields better and more interpretable expert allocation than a neural router trained only on numerical data.","pith_inferences":["Editorial: The paper does not ablate the router, so the cleanest test of its central claim is to freeze the two experts and swap the LLM router for an always-optimistic label or a simple lexicon sentiment rule; if either matches LLMoE's returns, the LLM's reasoning is not the source of the gain.","Editorial: Because the backtest assumes zero transaction costs and full liquidation, the reported 31–65% total returns are upper bounds; a cost-aware replay of the same all-in/all-out signals would show how much of the edge survives real trading frictions.","Editorial: If the routing labels prove reliable, they could be distilled into a small classifier to make the system cheaper at inference time, or reused as a regime indicator for risk management.","Editorial: The 80/20 chronological split may place the test mostly in a single market regime; extending the evaluation across multiple bull and bear periods would separate routing skill from a favorable upswing."],"forward_implications":["Routing decisions can be made without training a gating network, potentially avoiding router collapse in data-scarce financial settings.","An MoE can exploit textual news directly through the router's context window, removing the need for a separate sentiment-analysis module.","The reported reduction in maximum drawdown on MSFT, from 17.82% for the two-expert MoE to 11.32% for LLMoE, implies improved downside control in the tested period.","The router's natural-language reasoning gives a human-readable audit trail for each trade, which static neural routers do not provide.","The same router-plus-experts structure can be repurposed for other prediction tasks by changing the prompt and retraining the expert heads."],"supporting_citations":[{"why":"Supplies the AlphaMix MoE baseline and the existing static-routing approach that LLMoE is designed to replace, as well as the methodological inspiration for the problem formulation.","marker":"(Sun et al. 2023)"},{"why":"Provides a recent LLM-MoE trading system that the paper contrasts with, showing the line of work this framework extends.","marker":"(Ding, Shi, and Liu 2024)"},{"why":"Motivates the use of news text for stock trend prediction, backing the router's multimodal input design.","marker":"(Hu et al. 2018)"},{"why":"Represents the mixture-of-experts with group aggregation approach in stock prediction, used as context for the MoE family the paper compares against.","marker":"(Yu et al. 2024)"},{"why":"Cited to support the use of LLMs for automated strategy generation, the final stage of the LLMoE pipeline.","marker":"(Kou et al. 2024)"}],"fun_headline_variants":["LLM router picks trading expert, beats static MoE","LLM as MoE router lifts Sharpe and total return","LLMoE: LLM decides which trading expert to trust","World-knowledge LLM router improves trading MoE","LLM-directed MoE tops baselines in stock trading"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The performance gain is credited to the LLM router, but the paper never validates the router's Optimistic/Pessimistic labels or compares it against a trivial router, so the improvement could come from the expert architecture or the data split rather than from the routing mechanism.","fun_headline_variants_meta":{"raw":{"variants":["LLM router picks trading expert, beats static MoE","LLM as MoE router lifts Sharpe and total return","LLMoE: LLM decides which trading expert to trust","World-knowledge LLM router improves trading MoE","LLM-directed MoE tops baselines in stock trading"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000727,"raw_usage":{"total_tokens":3248,"prompt_tokens":927,"completion_tokens":2321,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":543,"completion_tokens_details":{"reasoning_tokens":2240}},"tokens_in":543,"tokens_out":2321,"duration_ms":17826,"temperature":1.0,"reasoning_tokens":2240,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:48:17.182595+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same two experts but route with a constant 'optimistic' label, or with a simple news-sentiment rule, and compare returns to LLMoE; if either matches the reported Sharpe and total return, the LLM router is not the cause. Also recompute the all-in/all-out backtest with realistic transaction costs and slippage: if the 31–65% total returns collapse to baseline levels, the trading rule, not the routing, explains the result.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the AlphaMix MoE baseline and the existing static-routing approach that LLMoE is designed to replace, as well as the methodological inspiration for the problem formulation."}],"review_version":1}