{"id":"f45fbc55-c34a-42b0-bf3f-7e3c74fb58fd","arxiv_id":"2509.05080","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":13,"one_line_summary":"A multimodal LLM router dynamically weights four technical trading experts and, in cost-free backtests, outperforms 15 baselines across stocks, futures, and crypto.","lead":"MM-DREX is a trading system where a vision-language model reads charts and time series, then dynamically allocates capital among four rule-based trading experts. The authors report backtested outperformance over 15 baselines, but the evaluation omits transaction costs and statistical significance tests, leaving the headline claim unverified.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Baseline backtest is internally inconsistent: Appendix D defines B&H as a look-ahead sign(P_{t+1}-P_t) rule, while Table 2's reported crypto B&H MDD of 8.92% is incompatible with a chronological 20% test split on BTC/ETH 2021-2025; without reproducible baselines the outperformance claim is ungrounde","rationale":"The reader's verdict is REJECT, and the reader's stated weakest assumption is the absence of transaction costs, slippage, and liquidity constraints. That is a legitimate concern and is explicitly acknowledged in the paper's future-work section. However, an even more load-bearing problem is that the baseline results themselves appear internally inconsistent. Appendix D.4 defines B&H in a way that is not buy-and-hold and is look-ahead; Table 1 and the chronological split make the reported crypto B&H MDD of 8.92% implausible. If the baseline numbers cannot be reproduced from the paper's own definitions and data ranges, then the claimed outperformance over those baselines is not interpretable. This does not change the reader's rejection verdict, but it identifies a different, more fundamental correctness risk. The proposed test is a direct recomputation of one baseline row, plus a re-implementation of the appendix's formula, which would settle whether the inconsistency is a typo or an evaluation defect.","tokens_in":26809,"tokens_out":4558,"duration_ms":53343,"concrete_test":"Recompute a strict buy-and-hold baseline (long on first day, rebalance at asset weights, no trading) on BTC and ETH from 2021-01-01 to 2025-03-23 using the paper's 60/20/20 chronological split, and compute TR and MDD on the final 20%. Then re-implement the Appendix D.4 sign(P_{t+1}-P_t) rule on the same split. If the former yields MDD above 15% (or the latter yields TR far above 11.5%), Table 2's B&H row is not reproducible; then recompute all 15 baselines under the paper's stated definitions and re-run the comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is relative outperformance over 15 baselines, so the baselines must be correctly realized. Appendix D defines the B&H baseline (Eq. D.4) as h_B&H = sign(P_{t+1} - P_t), i.e., a one-step look-ahead perfect-foresight rule, not a buy-and-hold strategy. If this rule were actually backtested, cumulative returns would compound daily absolute moves and would be far larger than the values in Table 2; if it was instead implemented as a long-only hold, the appendix is not describing the experiment. Independently, Table 1 states crypto data spans 2021-01-01 to 2025-03-23 for BTC/ETH, and the paper's split is chronological 60/20/20, so the test window is roughly 2024-2025. A long-only B&H portfolio in that window had drawdowns well exceeding 8.92% (BTC fell about 25% in mid-2024 and again in early 2025). The reported crypto B&H MDD of 8.92% cannot be reproduced under either the stated data range or the stated split. Because every relative improvement in Table 2 is measured against these baselines, the empirical core of the paper is not currently reproducible. The zero-transaction-cost issue is real, but it is downstream: even a cost-aware backtest is meaningless if the baseline numbers themselves are not reproducible.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MM-DREX, an LLM/VLM-based trading framework that decouples market-state perception from strategy execution. A dynamic router fuses candlestick images, technical time series, and textual summaries to allocate weights among four hand-designed trading experts (trend, reversal, breakout, positioning), and the system is trained with a two-stage SFT-RL procedure. The authors evaluate MM-DREX on a self-constructed multi-asset dataset covering US equities, A-shares, ETFs, futures, and cryptocurrencies, and claim that it significantly outperforms 15 baselines on total return, Sharpe ratio, and maximum drawdown. The paper also reports ablations for routing mechanisms and visual/temporal modalities, plus crisis-event comparisons against the S&P 500. The architectural idea is coherent and the cross-market dataset is a useful contribution, but the empirical evidence as presented is not robust enough to support the central outperformance claim.","tokens_in":27304,"tokens_out":3575,"duration_ms":44091,"significance":"If the reported results were reliable, the paper would make a meaningful contribution: the explicit separation of a VLM-based router from heterogeneous, interpretable trading experts is a plausible way to address regime shifts, and the hybrid SFT-RL training scheme is a reasonable solution to jointly optimizing modules with different objectives. The dataset spanning five asset classes across multiple markets is also valuable as a multimodal benchmark. However, the significance is currently conditional. The central claim rests on a backtest whose baselines are not reproducibly defined, whose hyperparameters are selected using the final test data, and whose returns ignore transaction costs and slippage. In addition, no statistical significance tests or multiple-seed results are provided. These are not cosmetic issues: they directly affect whether Table 2 can be read as evidence of outperformance. The paper should be credited for including explicit interpretability traces and for a detailed appendix with algorithmic descriptions, but these strengths do not compensate for the empirical validity gaps.","major_comments":[{"comment":"The B&H baseline is defined as h_B&H = sign(P_{t+1} - P_t), i.e., a one-step look-ahead perfect-foresight rule. If this rule were actually backtested, the cumulative returns would compound daily directional moves and would be far larger than the values reported in Table 2. If it was instead implemented as a long-only buy-and-hold strategy, the appendix does not describe the tested strategy. Independently, the reported crypto B&H MDD of 8.92% is inconsistent with a long-only BTC/ETH portfolio over the stated chronological 60/20/20 split, since the 2024-2025 test window includes drawdowns well above that level. Because every relative improvement in Table 2 is measured against baselines, this reproducibility failure undermines the empirical core of the paper.","section":"Appendix D, Eq. (D.4); Table 2"},{"comment":"The choice of input window (T_in=100), prediction horizon (T_pred=90), and teacher model (GPT-o3) is made by evaluating 12 multimodal LLMs on 37 assets randomly selected from the same dataset, with no indication that this evaluation is restricted to a separate validation partition. Since the final test set is then used for MM-DREX's evaluation, the test set has been used for hyperparameter/model selection. This is a form of test-set leakage that makes the reported test performance optimistically biased and invalidates the 'out-of-sample' framing.","section":"Appendix C; Section 'Dataset Construction'"},{"comment":"The backtest assumes zero transaction costs, no slippage, and no liquidity constraints; the paper explicitly lists incorporating these as future work. For a system that rebalances according to daily router weights and switches among expert rules, these frictions are first-order rather than second-order. Without at least a sensitivity analysis covering conservative cost/slippage assumptions, the reported returns are not financially realizable and the outperformance claim cannot be interpreted as evidence for practical trading.","section":"Section 'Experiments'; Section 'Conclusion and Future Work'"},{"comment":"All performance numbers are single point estimates. No standard errors, confidence intervals, significance tests, or multiple random seeds are reported for MM-DREX or the RL baselines. Given the stochasticity of RL training and VLM inference, the claim that MM-DREX 'significantly outperforms' 15 baselines is unsupported by any inferential statistic. In addition, the expert strategies in Appendix A contain numerous hand-set thresholds and parameters, and no sensitivity analysis is provided to show that the reported advantage is robust to these choices.","section":"Table 2 and Ablation Studies"}],"minor_comments":[{"comment":"The discounted return G_t is written as an infinite sum with an unexplained index variable K; the finite-horizon POMDP would be clearer if the horizon and discounting were stated precisely.","section":"Section 'Problem Formulation'"},{"comment":"The model name is spelled 'MMD-DREX' in the first sentence; should be 'MM-DREX'.","section":"Appendix E, first paragraph"},{"comment":"The crisis benchmark is described as 'S&P 500 index' in the text and table, but the dataset specification in Table 1 does not include an index asset; the relationship between the crisis-evaluation assets and the main dataset should be clarified.","section":"Figure 3 and Table 5"},{"comment":"The pseudo-code is useful but some algorithmic details are underspecified, e.g., 'recent breakout, MA divergence, pullback' and 'strong buy signals' are not formally defined. This limits reproducibility of the expert strategies.","section":"Appendix A"}],"recommendation":"reject","confidential_remarks":"The architectural proposal is interesting and the dataset construction is a genuine asset. My recommendation is driven by the reproducibility and validity issues in the empirical evaluation: the baseline definition in Appendix D is internally inconsistent, the test set is used for model selection in Appendix C, and no cost modeling or statistical testing is provided. These are load-bearing problems for a paper whose central claim is empirical outperformance, and they would require substantial new experiments rather than local revisions to fix."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, this paper has a genuinely interesting architecture and a deeply flawed evaluation. The idea—a VLM router that looks at candlestick charts and time series, then weights four heterogeneous rule-based experts—is a real combination I haven't seen in FinAgent/FinMem/TradExpert. The SFT-then-RL training for the router is a nice touch, and the ablations on routing and visual modality are the right experiments. The dataset is large and multi-market. If the results were trustworthy, this would be a solid system paper.\n\nBut the results are not trustworthy as reported. The biggest problem is the B&H baseline. I checked Appendix D myself: Eq. D.4 defines h_B&H = sign(P_{t+1} - P_t), which is a one-step look-ahead perfect-foresight rule, not buy-and-hold. If that were actually executed, cumulative returns would be far larger than the table shows. The table's crypto B&H MDD of 8.92% is also implausible for a long-only hold over the 2024-2025 test window. Either the appendix is describing a different experiment, or the baseline was implemented incorrectly. Either way, every relative improvement in Table 2 is measured against a baseline that is not reproducible. That's load-bearing.\n\nThere are other problems. No transaction costs or slippage—the authors list that as future work, but in trading it changes the answer. No significance tests or multiple seeds. And the hyperparameter selection is circular: Appendix C shows they picked the teacher model (o3) and the 100/90 window by testing on assets that later appear in the test set. That's leakage.\n\nWhat the paper does well: the architecture is clearly described, the appendices are thorough, and the ablations isolate the router and the visual modality. The authors are not hand-waving about the system; they're showing real engineering.\n\nMy bottom line: this deserves a serious referee, but only with the understanding that the empirical claims need major revision. The architectural idea is worth engaging with; the numbers are not. I'd want the authors to fix the baseline definition, re-run the comparison with proper B&H, add transaction costs, and report statistical tests. Until then, I wouldn't cite the performance claims, but I'd keep the architecture in mind.","headline":"Interesting architecture, untrustworthy numbers—the B&H baseline is defined as a look-ahead rule and the reported crypto MDD doesn't match the data.","tokens_in":27752,"tokens_out":3203,"would_cite":false,"duration_ms":35811,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A vision-language model reads candlestick charts and time-series data to route capital between four specialized trading strategies, and the authors report it outperforms 15 baselines across stocks, futures, and cryptocurrency markets.","keywords":["LLM trading agents","dynamic expert routing","multimodal financial data","mixture of experts","reinforcement learning","candlestick chart analysis","POMDP","drawdown control"],"falsifier":"Re-run the reported experiments with a per-trade cost of 10 basis points, a realistic slippage model, and position-size caps on illiquid futures and crypto, then compare total return and Sharpe ratio against FinAgent, PPO, and the technical baselines. If the gap over the best baseline on each market shrinks to within one standard error, the claimed outperformance fails.","tokens_in":26787,"feed_emoji":"📈","tokens_out":6769,"duration_ms":65397,"temperature":0.7,"pith_summary":"MM-DREX is an attempt to make LLM-based trading agents adaptive in markets that keep changing regime. The paper's central claim is that a system which separates “what is the market doing” from “what should we trade” will beat methods that freeze one strategy or rely on only numeric inputs. To show this, the authors build a vision-language router that looks at candlestick charts plus long-term price features and outputs a portfolio weight for each of four heterogeneous experts: trend, reversal, breakout, and positioning. They report that the resulting system outperforms 15 baselines—technical rules, machine learning, deep reinforcement learning, and financial LLMs—on total return, Sharpe ratio, and maximum drawdown across U.S. equities, Chinese A-shares, ETFs, futures, and cryptocurrencies. The reason this matters: if the result holds, it gives a practical architecture for sequential trading decisions under non-stationarity, with an audit trail.","feed_headline":"VLM-routed experts beat 15 baselines on five markets","feed_subtitle":"A vision-language router reads charts and price series to shift weight among four trading specialists, cutting drawdowns.","key_machinery":"The dynamic router: a vision-language model (a 72-billion-parameter VLM adapted with low-rank adapters) that fuses a candlestick chart image with a standardized window of OHLCV and technical-indicator time series, then outputs a softmax weight vector over four experts. The four heterogeneous experts are specialized heads—trend, reversal, breakout, positioning—each with its own policy and value head but sharing the same multimodal representation. The SFT-RL hybrid training is the mechanism that makes the two parts work together: supervised fine-tuning gives the router market-classification priors, and a PPO-style clipped objective with a reward combining excess return, Sharpe ratio, drawdown","core_discovery":"MM-DREX's central claim is that dynamic, multimodal routing is what makes an expert ensemble work in trading. The router, initialized by supervised fine-tuning on a three-way market classification task (uptrend, downtrend, consolidation) and then refined by reinforcement learning, allocates real-time weights to four independently trained experts. Each expert emits a sub-strategy from a small action space (e.g., moving-average crossover, Bollinger reversion, volume breakout, long-only or cash), and the final decision is a weighted aggregation. A group-relative reward pushes experts away from each other so they do not collapse into the same policy. In the paper's experiments, this design beats","pith_inferences":["If realistic per-trade costs and liquidity caps were applied, the absolute returns would shrink; whether MM-DREX keeps its edge is an open empirical question the paper does not test.","The 60.21% regime-classification accuracy is close to the 60% directional accuracy the authors measured for top multimodal LLMs, suggesting the system's advantage may come from diversification across experts rather than from superhuman forecasting.","A natural extension the paper leaves implicit is to apply the same routing mechanism at the portfolio level—selecting assets and position sizes—rather than only weighting strategies for a single asset.","The dual-annotation regime labels rely on three human traders plus technical indicators; a stress test would be to see whether routing degrades when labels are generated by noisy or adversarial annotations."],"forward_implications":["If MM-DREX's reported results hold, the router-plus-experts division becomes a template for adaptive LLM agents in any domain where states change regimes, not just finance.","Dynamic routing is the load-bearing component: the ablation claims a 45.3% larger maximum drawdown under uniform weighting, so any deployment should preserve adaptive weights rather than a fixed mixture.","Visual modality is not decorative in this architecture: dropping chart images is reported to cut directional accuracy by about 6.5 percentage points and to worsen drawdown substantially.","The group-relative reward gives a concrete way to keep specialized policies diverse, directly addressing the homogeneous-expert failure of earlier mixture-of-experts trading models.","The multimodal dataset spanning 62 assets across five markets could serve as a shared benchmark for evaluating LLM trading agents, provided its regime labels are trusted."],"supporting_citations":[{"why":"FinAgent, the multimodal foundation-agent baseline whose U.S. equity return is the strongest LLM comparator MM-DREX must beat.","marker":"Zhang et al. 2024"},{"why":"FinMem, the LLM trading agent with layered memory that serves as the second LLM baseline and the cross-market comparison point.","marker":"Yu et al. 2024"},{"why":"PIXIU, the financial LLM and benchmark whose dataset coverage is compared against the new MM-DREX multimodal dataset.","marker":"Xie et al. 2023"},{"why":"PPO, the clipped policy-gradient algorithm whose objective and hyperparameters underlie the expert-policy training in the RL stage.","marker":"Schulman et al. 2017"},{"why":"SAC, an off-policy maximum-entropy RL baseline used in the 15-model comparison.","marker":"Haarnoja et al. 2018"},{"why":"DQN, the deep Q-learning baseline used in the 15-model comparison.","marker":"Mnih et al. 2015"},{"why":"LightGBM, the gradient-boosting machine-learning baseline that posts the strongest classical competitor on several markets.","marker":"Ke et al. 2017"},{"why":"LSTM, the recurrent-network baseline for sequence-based trading prediction.","marker":"Fischer and Krauss 2018"},{"why":"Transformer, the self-attention sequence-model baseline included among the deep-learning comparators.","marker":"Vaswani et al. 2017"}],"fun_headline_variants":["VLM router picks trading experts dynamically, beats 15 baselines","Multimodal trading router adapts to market shifts","Vision-language router routes four trading experts in real time","Dynamic expert routing in trading outperforms 15 baselines","Chart-aware router optimizes expert ensemble, cuts drawdowns"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that the backtest results, which assume zero transaction costs, no slippage, and no liquidity constraints, represent the strategy's real-world edge; the paper lists adding costs, liquidity constraints, and latency as future work.","fun_headline_variants_meta":{"raw":{"variants":["VLM router picks trading experts dynamically, beats 15 baselines","Multimodal trading router adapts to market shifts","Vision-language router routes four trading experts in real time","Dynamic expert routing in trading outperforms 15 baselines","Chart-aware router optimizes expert ensemble, cuts drawdowns"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000234,"raw_usage":{"total_tokens":1372,"prompt_tokens":821,"completion_tokens":551,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":469}},"tokens_in":565,"tokens_out":551,"duration_ms":6136,"temperature":1.0,"reasoning_tokens":469,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T05:36:15.301012+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the reported experiments with a per-trade cost of 10 basis points, a realistic slippage model, and position-size caps on illiquid futures and crypto, then compare total return and Sharpe ratio against FinAgent, PPO, and the technical baselines. If the gap over the best baseline on each market shrinks to within one standard error, the claimed outperformance fails.","supporting_citations":[],"review_version":1}