{"id":"729a583b-5e6f-4de3-9106-2e09101b948d","arxiv_id":"2505.12759","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A bilevel reinforcement learning method with data transformations and minimum-of-ensemble TD targets improves offline-to-online stock trading performance on CSI-300 and NASDAQ-100.","lead":"MetaTrader is a reinforcement learning system for stock portfolio selection that trains on both raw market data and artificially distorted versions of it, then uses the worst of several predicted outcomes to make trading decisions. It reports higher cumulative returns and Sharpe ratios than existing RL trading methods on two public stock datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (5) defines the 'worst-case' TD target as a min over three fixed hand-crafted transformations, with no uncertainty set or bound linking these states to the actual next-market-state distribution, so the value-overestimation claim is unsupported.","rationale":"I agree with the reader's CONDITIONAL verdict, but I identify a different load-bearing weakness. The reader's weakest assumption is that market transitions are action-free; this supports the partial-offline reward computation. However, even granting that assumption entirely, Eq. (5) still requires that min over F1:F3 approximates the worst-case next state. The paper provides no formal uncertainty set, no distributional assumption, and no calibration showing that these three deterministic transformations cover plausible adverse market states. This matters because the abstract and Algorithm 1 present transformation-based worst-case TD as a core contribution, and Figure 10's evidence is an aggregate error plot, not a test of conservatism. The empirical superiority of MetaTrader could therefore be explained by bilevel data augmentation alone, independent of the claimed worst-case mechanism. The concern is concrete and testable: one can compare the min-over-transformations target against an empirical quantile of Bellman targets over actual next states on held-out data. Such a test would settle whether the method is conservative or merely pessimistic, and whether the central conceptual claim is supportable. The reader's action-free-transition concern is related but secondary, since even a non-action-free market would only strengthen the need for a defined worst-case set, not fix the missing bound in Eq. (5).","tokens_in":42326,"tokens_out":3651,"duration_ms":43960,"concrete_test":"Specify a high-probability uncertainty set U(s_t) such that the true next state satisfies P(h_{t+1} in U) >= 1 - delta, and prove that the min over transformations constrained to U is a lower bound on the expected TD target; then, on held-out CSI-300 data, compute the empirical 5th percentile of Bellman targets over realized next states and compare it with the Eq. (5) min over F1:F3. If the min-transformation target exceeds the empirical 5th percentile in more than delta fraction of states, the worst-case claim fails; if it is far below, it is not an approximation but a pessimistic heuristic. Recomputing Figure 10 with this empirical reference would settle whether the method is genuinely conservative or merely pessimistic on test states.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central technical contribution is Eq. (5), which replaces the one-step TD target with min_{n=0:N} [R(s_t, s^(n)_{t+1}) + gamma(-lambda log pi(a^(n)_{t+1}|s^(n)_{t+1}) + min_{k=1,2} \\bar{Q}_{\\phi_k}(s^(n)_{t+1}, a^(n)_{t+1}))]. The justification is that Monte Carlo sampling over transformations approximates the worst-case future payoff (Section 2, Figure 5). But F1-F3 are not samples from any distribution over plausible next market states; they are three fixed perturbations: invert top-gaining assets, reverse time, and downsample. There is no theorem, uncertainty set, or empirical calibration showing that the min over these three states is a lower bound on, or even an approximation to, the Bellman target under the market's actual transition distribution. By construction, increasing the number of transformations or making them more aggressive can drive the target arbitrarily low, so the method is not a conservative estimator in any defined sense; it is a heuristic pessimism penalty. Figure 10 only reports mean absolute Q-error on training trajectories; it does not verify that the min-target is conservative on held-out states. Thus the claimed mechanism for mitigating value overestimation is not established, and the reported gains could stem from generic data augmentation or bilevel regularization rather than from the proposed worst-case TD principle.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MetaTrader, a reinforcement learning method for sequential portfolio optimization framed as a 'partial-offline' RL problem in which the market state evolves action-free while the balance state evolves according to the agent's trades. The method combines a bilevel optimization scheme over subsets of raw and transformed market data with a modified temporal-difference target that takes the minimum over TD targets computed from several hand-crafted data transformations (Eq. 5). The empirical section reports that MetaTrader outperforms existing RL-based trading methods and stock-prediction baselines on CSI-300 and NASDAQ-100, including under streaming finetuning, and provides ablations of the transformation and TD components.","tokens_in":42690,"tokens_out":3944,"duration_ms":46798,"significance":"If the claims hold, the paper would offer a practical recipe for improving offline-to-online generalization in non-stationary financial decision-making, and the bilevel-plus-transformation scheme could be useful beyond finance. The paper provides a credible empirical comparison across many baselines, includes ablations of data transformations and TD variants, reports standard deviations over seeds, and tests on an expanded 587-stock setting; these are strengths. However, the central methodological claim that Eq. (5) approximates a worst-case or conservative TD target is not theoretically supported by the evidence presented, and the time-reversal transformation creates an internally inconsistent notion of 'next' state. The empirical results are suggestive but require either a proper justification of the worst-case interpretation or a reframing of the method as heuristic data augmentation.","major_comments":[{"comment":"The claim that the min over the three transformations in Eq. (5) approximates the 'worst-case TD target' is not supported by any uncertainty set, distributional assumption, or bound linking the transformed states to the true next-state distribution. Since F1-F3 are fixed deterministic maps (with only a deterministic top-alpha selection in F1), the 'Monte Carlo sampling' language in the text and Figure 5 is inaccurate, and increasing N or making the transformations more aggressive would drive the min target arbitrarily low. The method is a heuristic pessimism penalty, not a conservative estimator in a defined sense. The authors should either provide a formal statement with assumptions under which the min target lower-bounds the Bellman target, or revise the claims to describe the mechanism as data augmentation with a pessimistic heuristic.","section":"Section 3, Eq. (5)"},{"comment":"The second transformation reverses the temporal order of a T-length price sequence to form F2. Since Eq. (5) evaluates a 'next-step' state s_{t+1}^{(n)} from a reversed sequence, that state is temporally prior to the current state in the original data, so it is not a plausible future market state. This breaks the causal structure of the TD target and undermines the interpretation of the min as approximating worst-case future payoffs. The authors should either exclude F2 from the TD-target computation or provide an argument for why a reversed-time state is a meaningful adversarial future.","section":"Supplementary S1, F2 and Eq. (5)"},{"comment":"The partial-offline formulation relies on the assertion that 'individual buying and selling actions typically have minimal impact on market dynamics,' making market transitions action-free. No evidence is provided that the trading amounts used in the experiments (e.g., lots of 100 or 200 shares across an 88-stock portfolio) do not themselves affect market state transitions, especially for less liquid securities. If actions do influence prices, the rewards and TD targets computed from transformed market states are not realizable, and the bilevel training does not evaluate actual outcomes. This assumption should be tested empirically or at least discussed as a limiting condition with a concrete validity check.","section":"Section 2, 'Decoupled state space'"},{"comment":"The claim that transformation-based TD reduces value overestimation is supported only by Figure 10, which reports the discrepancy between learned Q-values and a single-trajectory discounted return on training trajectories. This does not establish that the min-target is conservative on held-out or out-of-distribution states, which is the setting the paper emphasizes. A proper evaluation would compare Q-value errors on test-period states or against a calibrated uncertainty interval; without such evidence, the value-overestimation argument remains unverified.","section":"Figure 10"}],"minor_comments":[{"comment":"The text states that RL-based results are 'from 10 random training seeds' in one place and 'averaged across three random training seeds' later; the number of seeds and the reporting convention should be made consistent.","section":"Section 4, Table 1"},{"comment":"In Eq. (5), the notation {s_{t+1}^{(n)}}_{n=1}^N is used while the min ranges over n=0:N, so the definition of the n=0 term and the indexing for the transformed states should be stated explicitly.","section":"Eq. (5) notation"},{"comment":"The caption reports mean results over 3 seeds but no error bars are shown; adding variability measures would strengthen the ablation claims.","section":"Figure 8 caption"},{"comment":"The comparison in Figure 9 against 'minimum value of ensemble Q' and 'mean value of ensemble Q' baselines uses multi-Q-networks with real future data, while MetaTrader uses transformed data with a single pair of target networks; the text should clarify which factors differ and how the comparison isolates the effect of the transformed-data min target.","section":"Section 4, 'Impact of the transformation-based conservative TD ensembles'"},{"comment":"No code or data is released, which limits reproducibility of the reported standard deviations and ablations; the authors should consider providing an implementation and configuration details sufficient to reproduce Table 1.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper leans heavily on StockFormer (the authors' own prior work) as both the network backbone and the strongest baseline, which is acceptable but should be acknowledged in the novelty positioning. The lack of code/data release is a concern for a paper whose main evidence is empirical. The central 'worst-case TD' claim needs to be either formally justified or reworded; the current formulation is not a conservative estimator in the sense of the offline-RL literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. What should you know: MetaTrader is a reasonably competent empirical paper on RL for stock trading. It takes MAML-style bilevel training, applies three fixed data transformations (invert top gainers, reverse time, downsample), and uses the minimum TD target across transformed next states as a conservative target. On two standard datasets, it reports 16-33% relative cumulative return gains over StockFormer, with ablations supporting that transformations and bilevel updates help. That's the core of it.\n\nWhat it does well: the empirical work is more complete than most in this area. It compares against a sensible set of baselines, uses standard metrics, provides ablations for each transformation and for bilevel updates, tests on a larger 587-stock universe, includes a more recent test period beyond 2022, and reports runtime. It also acknowledges training instability as a limitation. Given the fragile nature of backtest results in finance, that is decent practice.\n\nWhere it is soft: the central technical claim, Eq. (5), is not established. The min over F1-F3 is not a lower bound or an approximation to any defined worst-case distribution. F2 is chronological reversal, which produces the past, not a plausible future, and adding more aggressive transformations can drive the target arbitrarily low. So the claim that this 'worst-case TD' mitigates value overestimation is unsupported; Figure 10 measures Q-error on training trajectories, not conservatism on held-out states. The gains are likely coming from data augmentation and bilevel regularization, which are fine but should be the actual claims. There is also a seed-count inconsistency: the text says three random seeds, but Table 1 reports ten. And no code or data is released, which limits verification. The partial-offline formulation is essentially online rollouts on a fixed historical dataset; the action-free market state assumption is plausible for daily stock data but untested.\n\nOverall: the paper deserves a serious referee because the empirical comparison is substantial and the idea of transformation-based pessimism is worth discussing. But it needs major revision: either prove a bound linking the transformations to a plausible next-state set, or explicitly reframe the method as heuristic data augmentation with pessimistic targets. I'd also ask for code and a fix to the seed reporting. Not a desk reject; a conditional accept after significant work.","headline":"Solid empirical paper on RL for trading whose central 'worst-case TD' claim is not supported; still worth peer review with major revisions.","tokens_in":43172,"tokens_out":2979,"would_cite":false,"duration_ms":31304,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that offline RL trading policies memorize fixed-dataset behaviors and fail under market shifts, and proposes MetaTrader, a bilevel trading agent that trains on transformed market data and conservative worst-case TD…","keywords":["sequential portfolio optimization","offline reinforcement learning","bilevel optimization","temporal difference learning","out-of-distribution generalization","stock trading","value overestimation","data transformation"],"falsifier":"Backtest the same training procedure on a historical period with thin trading volume or with large institutional trades, using a simulator that applies price impact to the next market state; if the bilevel agent's returns or Q-values degrade relative to a baseline that accounts for impact, the action-free assumption fails.","tokens_in":42172,"feed_emoji":"📈","tokens_out":4447,"duration_ms":46489,"temperature":0.7,"pith_summary":"This paper argues that standard offline RL policies for stock trading memorize profitable actions within a fixed dataset and fail when the market shifts. It proposes MetaTrader, which treats portfolio optimization as a partial-offline RL problem in which market states come from the dataset but the agent can still act and receive reward feedback. The method trains policy and critics with bilevel optimization across original and transformed market data, and learns value estimates from worst-case TD targets over those transformations. On CSI-300 and NASDAQ-100, the authors report higher cumulative returns and Sharpe ratios than existing RL and stock-prediction baselines.","feed_headline":"Trading agent beats offline RL on transformed markets","feed_subtitle":"Bilevel training and worst-case TD targets lift returns and cut value overestimation on two stock datasets.","key_machinery":"The load-bearing mechanism is the partial-offline MDP with decoupled state branches: market state transitions are assumed action-free, while balance transitions are deterministic given the action. This makes Eq. (5) feasible, a Monte Carlo worst-case TD target obtained by taking the minimum over transformed next states. The other load-bearing component is bilevel learning across data subsets, where inner-loop gradients on one subset are evaluated on another subset, including transformed data, to discourage memorization of the offline dataset.","core_discovery":"The central claim is that decoupling trading states into action-free market states and action-dependent balance states makes out-of-distribution evaluation tractable: the agent can compute rewards and next balance states for any action, and only the next market state remains unknown. MetaTrader uses this to construct worst-case TD targets by transforming the next market state and taking the minimum Q-value over the transformations. Bilevel training then optimizes inner-loop parameters on in-domain data and evaluates them on transformed OOD data. The paper reports cumulative returns of 1.44 versus 1.24 for StockFormer on CSI-300 and 1.30 versus 0.98 on NASDAQ-100, with improved Sharpe ratios, and lower Q-estimation error than using the original TD target.","pith_inferences":["Extension: the decoupled-state idea may transfer to any domain where an agent's actions do not affect part of the environment state, such as traffic states in autonomous driving or weather in energy management, provided those action-free components can be transformed plausibly.","Extension: the min-over-transformations TD target could be interpreted as a learned robust Bellman backup, suggesting a possible theoretical connection to robust MDPs that the paper does not develop.","Extension: a testable next step would be to replace the handcrafted transformations with learned generative data augmentations and check whether the worst-case TD target remains a valid lower bound on future returns."],"forward_implications":["Policies trained only on a static historical dataset can be made to generalize to non-stationary markets by evaluating them on transformed versions of the data, not just the original trajectories.","A worst-case TD target formed by taking the minimum over data transformations reduces value overestimation compared with standard SAC targets and with ensemble Q-network targets.","Bilevel finetuning on recent in-domain data improves adaptation, whereas the paper reports naive finetuning of StockFormer gives only a 0.81% cumulative-return gain versus 13.39% for MetaTrader.","The method scales to a 587-stock pool and retains inference-time cost comparable to StockFormer, making daily-level trading feasible.","Existing conservative offline RL methods like CQL and IQL underperform the partial-offline bilevel approach on financial data, suggesting that standard offline RL assumptions need revisiting for finance."],"supporting_citations":[{"why":"Supplies the base StockFormer trading architecture, the CSI-300 and NASDAQ-100 datasets, and the main RL baseline that MetaTrader improves upon.","marker":"[Gao et al., 2023a]"},{"why":"Supplies the Soft Actor-Critic algorithm whose actor-critic structure and entropy term MetaTrader inherits and modifies.","marker":"[Haarnoja et al., 2018]"},{"why":"Provides the CQL conservative offline RL baseline that the paper compares against and finds inadequate for financial data.","marker":"[Kumar et al., 2020]"},{"why":"Provides the IQL offline RL baseline used to show that existing offline RL methods underperform the proposed partial-offline bilevel approach.","marker":"[Kostrikov et al., 2021]"},{"why":"Provides the FinRL framework and FinRL-SAC and FinRL-DDPG trading baselines used in the empirical comparisons.","marker":"[Liu et al., 2021]"},{"why":"Provides the SARL portfolio-management baseline that is compared in both offline and online adaptation setups.","marker":"[Ye et al., 2020]"},{"why":"Supplies the diversified Q-ensemble baseline used to compare ensemble-based TD targets against MetaTrader's transformation-based worst-case TD target.","marker":"[An et al., 2021]"},{"why":"Supplies the pessimistic Q-ensemble baseline for offline-to-online RL that the transformation-based TD method is compared against in ablations.","marker":"[Lee et al., 2022]"}],"fun_headline_variants":["MetaTrader: bilevel RL beats offline policies on stocks","Worst-case TD targets fix offline RL for trading","Bilevel RL for portfolios: OOD gains without overestimation","Trading with partial offline data: MetaTrader's dual-loop win","Offline policies fail; bilevel training trades better"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The assumption that a trader's buy or sell orders do not noticeably change the market state, so next market prices can be treated as independent of the action.","fun_headline_variants_meta":{"raw":{"variants":["MetaTrader: bilevel RL beats offline policies on stocks","Worst-case TD targets fix offline RL for trading","Bilevel RL for portfolios: OOD gains without overestimation","Trading with partial offline data: MetaTrader's dual-loop win","Offline policies fail; bilevel training trades better"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000526,"raw_usage":{"total_tokens":2513,"prompt_tokens":892,"completion_tokens":1621,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":508,"completion_tokens_details":{"reasoning_tokens":1536}},"tokens_in":508,"tokens_out":1621,"duration_ms":13416,"temperature":1.0,"reasoning_tokens":1536,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:26:43.911707+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Backtest the same training procedure on a historical period with thin trading volume or with large institutional trades, using a simulator that applies price impact to the next market state; if the bilevel agent's returns or Q-values degrade relative to a baseline that accounts for impact, the action-free assumption fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the CQL conservative offline RL baseline that the paper compares against and finds inadequate for financial data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the FinRL framework and FinRL-SAC and FinRL-DDPG trading baselines used in the empirical comparisons."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the pessimistic Q-ensemble baseline for offline-to-online RL that the transformation-based TD method is compared against in ablations."}],"review_version":1}