{"id":"0c49fd72-f978-4073-aec5-df2f34c95c96","arxiv_id":"2411.17900","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A GPT-2-initialized Decision Transformer with LoRA matches or beats CQL, IQL, and BC on offline DJIA trading and generally beats a randomly initialized twin, though with weak statistical support.","lead":"This paper tests whether a trading agent built with a Decision Transformer works better when it starts from pretrained GPT-2 language model weights than from random weights. It reports modest gains in cumulative return on simulated Dow Jones stock trading, but the evidence rests on a short single test period, one trajectory per expert, and no significance tests.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RTG deployment protocol is unspecified; if test-time returns-to-go are ground-truth future returns, the empirical comparison has look-ahead bias and the central claim collapses.","rationale":"The reader's weakest assumption (single deterministic trajectory) is a valid generalization concern, but it does not by itself invalidate the narrow claim that pretrained weights help on this task. The RTG deployment question is more fundamental: the Decision Transformer's output depends directly on the RTG input, and the paper is silent on how that input is chosen during the test period. If ground-truth future returns are used, every entry in Table 2 and every comparison in Figure 3 is contaminated by look-ahead, making the central claim unfounded regardless of trajectory diversity. The check is cheap and definitive because the authors provide code. I therefore disagree with the reader's identification of the weakest assumption, while agreeing that the paper is CONDITIONAL pending resolution. I am not alleging misconduct; the omission may be an oversight in the write-up, and the code may show a clean protocol.","tokens_in":11359,"tokens_out":4815,"duration_ms":45920,"concrete_test":"Inspect the evaluation script in https://github.com/syyunn/finrl-dt (search for 'rtg', 'return', 'eval'). Determine how the initial RTG is set and how it is updated at each step. If RTG is initialized from the expert's actual test-period cumulative return or if future rewards are used to construct inputs, re-run the A2C and TD3 experiments with information-free initial RTG (e.g., 0 or a fixed constant) and update RTG only from realized rewards. If the reported cumulative return and pretrained-vs-random differences change materially (e.g., >1 percentage point or significance reversal), the central claim is not supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 3.2 defines the model's conditioning input as return-to-go (RTG), the sum of future rewards, and training minimizes action MSE given RTG. The paper never states how RTG is set at deployment. Section 4.2/4.4 and Appendix A.11 describe evaluation only as 'same environment settings' with unseen data. In a Decision Transformer, the policy is conditional on RTG, so the evaluation must specify an initial target and an update rule. If the code initializes RTG using the expert's realized test-period cumulative return, or feeds the ground-truth future-return sequence as input, then the model is given look-ahead information about the test period. Table 2's cumulative returns and, crucially, the pretrained-vs-random gap in Figure 3 would then reflect leakage rather than transferable prior knowledge. This is the single most load-bearing assumption because it affects every reported number, not just generalizability across regimes. The public repository (github.com/syyunn/finrl-dt) can resolve this directly.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a Decision Transformer (DT) initialized with pretrained GPT-2 weights and fine-tuned with Low-Rank Adaptation (LoRA) for offline reinforcement learning in quantitative trading. The method is trained on expert trajectories from five RL agents (A2C, PPO, SAC, TD3, DDPG) in a FinRL DJIA trading environment and evaluated on a single test window (July 2020 to October 2021). The authors compare against CQL, IQL, BC, and a randomly initialized DT with LoRA, reporting cumulative return, maximum drawdown, and Sharpe ratio. The central claim is that the GPT-2-initialized DT performs competitively with established offline RL methods and consistently outperforms its randomly initialized counterpart.","tokens_in":49,"tokens_out":4305,"duration_ms":90587,"significance":"If validated, the result would be a meaningful empirical contribution: it would suggest that pretrained language model weights provide a transferable prior for learning trading policies from expert trajectories, and the random-initialization control is a sensible experimental design for isolating the effect of pretrained weights. The public replication code is a concrete strength. However, the significance is contingent on the deployment-time return-to-go protocol being causal, on the single-trajectory dataset supporting the offline RL claim, and on the statistical robustness of the reported comparisons. As written, those conditions are not established, so the empirical contribution is currently unverified.","major_comments":[{"comment":"The deployment-time return-to-go (RTG) protocol is never specified. The model is trained with RTG defined as the sum of future rewards (Eq. 4) and the action prediction is conditioned on this quantity. At test time, a Decision Transformer requires an initial RTG target and an update rule based on realized rewards; the manuscript only states that evaluation uses 'the same environment settings' with unseen data (A.11). If the test-time RTG is initialized with the expert's realized test-period cumulative return, or if ground-truth future returns for the test window are fed as inputs, then the model has look-ahead information about the evaluation period. This would make every number in Table 2 and Figure 3, including the pretrained-versus-random gap, reflect leakage rather than learned trading skill. Please specify the exact RTG initialization and update protocol used at deployment, and if the code uses realized test-period returns, rerun the evaluation with a causal RTG schedule.","section":"Section 3.2, Eq. (4); Section 4.2 and Appendix A.11"},{"comment":"Only one deterministic trajectory per expert is sampled for the offline dataset. This means the training data is a single path through the state-action space, with no coverage of alternative states or actions. The models are therefore trained to reproduce one policy rollout, which is closer to memorizing a single trajectory than to learning an offline RL policy from a diverse dataset. The pretrained-versus-random comparison may then reflect which initialization memorizes that one path better, not a transferable prior for trading. Please collect multiple stochastic (or otherwise diverse) trajectories per expert and report results, or explicitly reframe the claims as behavior cloning from a single expert rollout.","section":"Appendix A.10"},{"comment":"No significance tests, confidence intervals, or per-seed breakdowns are reported. Appendix A.7 lists five random seeds but does not state how they are used or how many runs underlie each mean and standard deviation. In the PPO row, the pretrained model's cumulative return is 34.99 ± 8.03 versus 28.76 ± 0.27 for the random initialization, so the reported variability is large enough to undermine the claim of a 'consistent performance boost.' Report the number of runs, per-seed results, and pairwise significance tests or bootstrap confidence intervals for all three metrics across all five expert agents.","section":"Table 2 and Figure 3"},{"comment":"The evaluation uses a single test window, July 2020 to October 2021, which is one post-COVID market regime. The abstract and Section 4.5 claim generalization to unseen market conditions, but one test period cannot establish robustness across market regimes. Add at least one additional out-of-sample period (for example 2022-2023) or temper the generalization claims to this specific test window.","section":"Section 4.2"}],"minor_comments":[{"comment":"The text contains a literal formatting artifact, 'vbnet Copy code', interrupting the experimental design subsection. This should be removed.","section":"Section 4.3"},{"comment":"References [2] and [3] cite the same Decision Transformer paper (arXiv:2106.01345 and the NeurIPS version), and references [6] and [7] cite the same LoRA paper. These duplicates should be consolidated.","section":"References"},{"comment":"The bar charts report only point estimates, while Table 2 provides standard deviations. Add error bars to Figure 3 so the visual comparison reflects the underlying variability.","section":"Figure 3"},{"comment":"Five random seeds are listed, but the manuscript does not explain whether each experiment was run once per seed or whether these seeds correspond to different training runs for each expert agent and method. Clarify the mapping of seeds to runs.","section":"Appendix A.7"},{"comment":"The phrase 'reflecting a significant performance gain' for the DDPG case (47.98% versus 42.88%) uses 'significant' in a statistical sense that is not supported by any test; rephrase to avoid implying formal significance.","section":"Section 4.5.2"}],"recommendation":"major_revision","confidential_remarks":"The central unresolved issue is the return-to-go deployment protocol. If the public repository initializes test-time RTG with ground-truth future returns or uses realized test-period returns to update RTG, the paper's empirical comparison is invalid and should be rejected. If a causal protocol exists, the authors need to state it precisely and may need to rerun experiments. The single-deterministic-trajectory design is also a fundamental limitation that should be addressed before publication, even in a workshop setting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: this paper ports LaMo's pretrained-LM-plus-LoRA Decision Transformer to offline stock trading, and the random-init control is the right experiment. But the paper never says how return-to-go (RTG) is set at test time, and if the model sees ground-truth future returns, every number in Table 2 collapses. That is not a corner-case worry; it is load-bearing.\n\nWhat is genuinely here: a clean application of an existing recipe to DJIA trading, with residual MLP embeddings and parameter-matched baselines. The pretrained-versus-random comparison is a sensible way to isolate the value of GPT-2 initialization. The prose is clear, the code is public, and the paper honestly credits LaMo as the framework it adopts. That is all real.\n\nNow the soft spots, in proportion. The RTG issue is the big one. In a Decision Transformer, the policy is conditioned on RTG; at deployment you need an initial target and an update rule. Section 3.2 defines RTG as the sum of future rewards; the evaluation sections (4.2, 4.4, A.11) describe only \"same environment settings\" with unseen data. No inference-time RTG protocol is given. If the code feeds the expert's realized test-period returns or the ground-truth future sequence, the model has look-ahead information and the pretrained-vs-random gap in Figure 3 could reflect leakage rather than transferable prior knowledge. The repository can settle this.\n\nThe single deterministic trajectory per expert (Appendix A.10) and the single July 2020-October 2021 test window make the generalization claims thin. The paper also overstates: \"significantly outperforms\" is used without significance tests, and several comparisons have overlapping error bars (e.g., SAC returns: 39.59±3.18 vs. 37.07±0.55). The prose claims \"consistently outperforms\" when the evidence is suggestive, not conclusive.\n\nThe paper is not muddled. It acknowledges the LaMo lineage, uses a fair control, and releases code. It is a credible workshop-level application, not a new mechanism. Who gets value: fin-RL practitioners wanting a starting point for pretrained-LM priors. It deserves peer review because the RTG ambiguity is fixable and the empirical question is worth answering rigorously. My recommendation: send to review, but require the authors to specify the deployment-time RTG rule, add a non-leakage baseline (e.g., a fixed target), and ideally more trajectories and test windows. Without that, the headline result is unverifiable.","headline":"Plausible LaMo-to-finance port, but the never-specified return-to-go deployment protocol could be look-ahead leakage and sink the central claim.","tokens_in":12074,"tokens_out":2405,"would_cite":false,"duration_ms":22483,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A Decision Transformer initialized with pretrained GPT-2 weights and fine-tuned with LoRA learns trading policies from expert trajectories and consistently beats the same model with random initialization, while matching established…","keywords":["quantitative trading","offline reinforcement learning","Decision Transformer","Low-Rank Adaptation (LoRA)","pretrained language models","GPT-2","transfer learning","DJIA trading"],"falsifier":"Train the same models on multiple trajectories per expert (for example, different random seeds or stochastic rollouts) and re-evaluate on several non-overlapping test periods; if the pretrained-initialized model no longer consistently beats random initialization, the reported advantage is an artifact of the single-trajectory setup.","tokens_in":11158,"feed_emoji":"📈","tokens_out":7768,"duration_ms":57997,"temperature":0.7,"pith_summary":"The paper argues that a Decision Transformer—a Transformer trained to predict actions from sequences of returns, states, and past actions—can learn effective trading policies for the 29 stocks of the Dow Jones Industrial Average from a single expert trajectory, provided it starts from pretrained GPT-2 language-model weights and is fine-tuned with Low-Rank Adaptation (LoRA). Across five expert agents (A2C, PPO, SAC, TD3, DDPG), the pretrained-initialized model matches or beats established offline RL baselines (CQL, IQL, BC) on cumulative return, Sharpe ratio, and maximum drawdown, and it consistently outperforms the same architecture with randomly initialized GPT-2 weights. The central empirical claim is that language-model pretraining transfers to financial decision-making, and that LoRA makes this transfer parameter-efficient, with roughly 0.7% of the model's parameters trainable.","feed_headline":"Pretrained GPT-2 beats random start in trading Decision Transformers","feed_subtitle":"Language-model pretraining improves trading agents and matches standard offline RL on DJIA data.","key_machinery":"The central mechanism is the Decision Transformer, which reframes reinforcement learning as conditional sequence modeling: each timestep contributes three tokens—return-to-go, state, and action—and the model predicts the next action by minimizing mean squared error over a sliding context window. The paper adapts this architecture to GPT-2 by aligning the input format with GPT-2 token structure, replacing linear embeddings with residual MLPs, freezing the pretrained Transformer weights, and injecting low-rank adapters of rank $r=16$ into the attention layers. The language-pretrained trunk combined with a small set of trainable LoRA parameters (about 900,000, under 1% of GPT-2 small) is what carries the transfer claim: the pretrained attention layers are reused nearly unchanged, so any performance gain over random initialization is attributed to the pretrained representations.","core_discovery":"The paper's central claim is that a Decision Transformer initialized with pretrained GPT-2 weights and fine-tuned using Low-Rank Adaptation (LoRA) learns trading policies from expert trajectories more effectively than the same model with random initialization, and competitively with established offline RL methods. The evidence comes from a simulated DJIA trading environment where five expert RL policies each supply one deterministic trajectory; on the unseen 2020-2021 test window, the pretrained DT-LoRA variant achieves higher cumulative returns than its randomly initialized counterpart in all five cases (for example, 47.98% versus 42.88% under the DDPG expert) and ranks among the top performers on Sharpe ratio and maximum drawdown across baselines. The paper interprets this as evidence that pretrained language representations provide a transferable prior for capturing temporal dependencies in financial time series, while LoRA keeps adaptation computationally feasible.","pith_inferences":["Because the paper trains on only one deterministic trajectory per expert and evaluates on a single post-COVID test window, the observed pretraining advantage could be specific to that market regime; a natural extension is to test across multiple trajectory samples and multiple non-overlapping test periods.","The paper does not isolate what the pretrained weights contribute; ablating which layers are frozen versus adapted, or comparing LoRA against full fine-tuning, would test whether the gain comes from the pretrained trunk or from the embedding architecture.","The same recipe may transfer to other sequential decision problems such as portfolio rebalancing or trade execution, but the paper does not test this.","The small trainable-parameter count raises the possibility that the pretrained initialization acts mainly as a regularizer; comparing against a randomly initialized model with stronger weight-decay or dropout could clarify the mechanism."],"forward_implications":["If the central claim is correct, pretrained language-model weights provide a reusable initialization for offline RL trading agents, so new trading tasks could be bootstrapped from a language-model prior rather than trained from scratch.","The parameter efficiency of LoRA (under 1% of parameters trainable) suggests that large pretrained models can be adapted to financial domains with limited computational budgets.","The consistent advantage over random initialization indicates that the benefit comes specifically from the pretrained representations, not merely from the Decision Transformer architecture or the LoRA training procedure.","The model's competitive performance against CQL, IQL, and BC, despite training on a single deterministic trajectory, implies that sequence-modeling approaches can extract usable policies from very limited offline data."],"supporting_citations":[{"why":"Supplies the Decision Transformer architecture that reframes RL as sequence modeling, which the paper initializes with pretrained weights.","marker":"[3]"},{"why":"Provides the low-rank adaptation method used to fine-tune only a small fraction of the pretrained model's parameters.","marker":"[6]"},{"why":"Supplies the pretrained GPT-2 weights and architecture that the paper adapts to trading.","marker":"[11]"},{"why":"The approach the paper explicitly adapts: leveraging pretrained language models for offline RL, here applied to finance.","marker":"[12]"},{"why":"Provides the DJIA trading environment and the expert RL agents used to generate trajectories.","marker":"[9]"},{"why":"Frames the offline RL problem that motivates training from fixed historical trajectories.","marker":"[8]"}],"fun_headline_variants":["GPT-2 prior lifts offline RL trading agent","Pretrained GPT-2 enhances trading DT with LoRA","Transfer from language model improves offline RL trading","LoRA fine-tuned GPT-2 trumps random start in trading","Pretrained LM prior beats random init in trading DT"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a single deterministic expert trajectory is enough to train and compare offline RL methods; if that one path does not cover the state-action distribution well, the test-period rankings, including the pretrained-versus-random gap, may reflect memorization of one policy path rather than general skill.","fun_headline_variants_meta":{"raw":{"variants":["GPT-2 prior lifts offline RL trading agent","Pretrained GPT-2 enhances trading DT with LoRA","Transfer from language model improves offline RL trading","LoRA fine-tuned GPT-2 trumps random start in trading","Pretrained LM prior beats random init in trading DT"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000458,"raw_usage":{"total_tokens":2298,"prompt_tokens":949,"completion_tokens":1349,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":1271}},"tokens_in":565,"tokens_out":1349,"duration_ms":9693,"temperature":1.0,"reasoning_tokens":1271,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:43:08.488523+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same models on multiple trajectories per expert (for example, different random seeds or stochastic rollouts) and re-evaluate on several non-overlapping test periods; if the pretrained-initialized model no longer consistently beats random initialization, the reported advantage is an artifact of the single-trajectory setup.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Decision Transformer architecture that reframes RL as sequence modeling, which the paper initializes with pretrained weights."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the DJIA trading environment and the expert RL agents used to generate trajectories."}],"review_version":1}