Pith. sign in

REVIEW 4 major objections 5 minor 46 references

Your Offline Policy is Not Trustworthy: Bilevel Reinforcement Learning for Sequential Portfolio Optimization

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that offline RL trading policies memorize fixed-dataset behaviors and fail under market shifts, and proposes MetaTrader, a bilevel trading agent that trains on transformed market data and conservative worst-case TD…

desk verdict Solid empirical paper on RL for trading whose central 'worst-case TD' claim is not supported; still worth peer review with major revisions. read the letter →

arxiv 2505.12759 v1 pith:UNCOJKTK submitted 2025-05-19 cs.LG

Haochen Yuan , Minting Pan , Yunbo Wang , Siyu Gao , Philip S.Yu , Xiaokang Yang This is my paper · ORCID
classification cs.LG
keywords sequentialportfoliooptimizationofflinereinforcementlearningbileveltemporaldifferenceout-of-distributiongeneralizationstocktradingvalueoverestimationdatatransformation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that standard offline RL policies for stock trading memorize profitable actions within a fixed dataset and fail when the market shifts. It proposes MetaTrader, which treats portfolio optimization as a partial-offline RL problem in which market states come from the dataset but the agent can still act and receive reward feedback. The method trains policy and critics with bilevel optimization across original and transformed market data, and learns value estimates from worst-case TD targets over those transformations. On CSI-300 and NASDAQ-100, the authors report higher cumulative returns and Sharpe ratios than existing RL and stock-prediction baselines.

What carries the argument

The load-bearing mechanism is the partial-offline MDP with decoupled state branches: market state transitions are assumed action-free, while balance transitions are deterministic given the action. This makes Eq. (5) feasible, a Monte Carlo worst-case TD target obtained by taking the minimum over transformed next states. The other load-bearing component is bilevel learning across data subsets, where inner-loop gradients on one subset are evaluated on another subset, including transformed data, to discourage memorization of the offline dataset.

What would settle it

Backtest the same training procedure on a historical period with thin trading volume or with large institutional trades, using a simulator that applies price impact to the next market state; if the bilevel agent's returns or Q-values degrade relative to a baseline that accounts for impact, the action-free assumption fails.

Watch

Extended reading notes

Core claim

The central claim is that decoupling trading states into action-free market states and action-dependent balance states makes out-of-distribution evaluation tractable: the agent can compute rewards and next balance states for any action, and only the next market state remains unknown. MetaTrader uses this to construct worst-case TD targets by transforming the next market state and taking the minimum Q-value over the transformations. Bilevel training then optimizes inner-loop parameters on in-domain data and evaluates them on transformed OOD data. The paper reports cumulative returns of 1.44 versus 1.24 for StockFormer on CSI-300 and 1.30 versus 0.98 on NASDAQ-100, with improved Sharpe ratios, and lower Q-estimation error than using the original TD target.

Load-bearing premise

The assumption that a trader's buy or sell orders do not noticeably change the market state, so next market prices can be treated as independent of the action.

Editorial extensions

If this is right

  • Policies trained only on a static historical dataset can be made to generalize to non-stationary markets by evaluating them on transformed versions of the data, not just the original trajectories.
  • A worst-case TD target formed by taking the minimum over data transformations reduces value overestimation compared with standard SAC targets and with ensemble Q-network targets.
  • Bilevel finetuning on recent in-domain data improves adaptation, whereas the paper reports naive finetuning of StockFormer gives only a 0.81% cumulative-return gain versus 13.39% for MetaTrader.
  • The method scales to a 587-stock pool and retains inference-time cost comparable to StockFormer, making daily-level trading feasible.
  • Existing conservative offline RL methods like CQL and IQL underperform the partial-offline bilevel approach on financial data, suggesting that standard offline RL assumptions need revisiting for finance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extension: the decoupled-state idea may transfer to any domain where an agent's actions do not affect part of the environment state, such as traffic states in autonomous driving or weather in energy management, provided those action-free components can be transformed plausibly.
  • Extension: the min-over-transformations TD target could be interpreted as a learned robust Bellman backup, suggesting a possible theoretical connection to robust MDPs that the paper does not develop.
  • Extension: a testable next step would be to replace the handcrafted transformations with learned generative data augmentations and check whether the worst-case TD target remains a valid lower bound on future returns.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes MetaTrader, a reinforcement learning method for sequential portfolio optimization framed as a 'partial-offline' RL problem in which the market state evolves action-free while the balance state evolves according to the agent's trades. The method combines a bilevel optimization scheme over subsets of raw and transformed market data with a modified temporal-difference target that takes the minimum over TD targets computed from several hand-crafted data transformations (Eq. 5). The empirical section reports that MetaTrader outperforms existing RL-based trading methods and stock-prediction baselines on CSI-300 and NASDAQ-100, including under streaming finetuning, and provides ablations of the transformation and TD components.

Significance. If the claims hold, the paper would offer a practical recipe for improving offline-to-online generalization in non-stationary financial decision-making, and the bilevel-plus-transformation scheme could be useful beyond finance. The paper provides a credible empirical comparison across many baselines, includes ablations of data transformations and TD variants, reports standard deviations over seeds, and tests on an expanded 587-stock setting; these are strengths. However, the central methodological claim that Eq. (5) approximates a worst-case or conservative TD target is not theoretically supported by the evidence presented, and the time-reversal transformation creates an internally inconsistent notion of 'next' state. The empirical results are suggestive but require either a proper justification of the worst-case interpretation or a reframing of the method as heuristic data augmentation.

major comments (4)
  1. [Section 3, Eq. (5)] The claim that the min over the three transformations in Eq. (5) approximates the 'worst-case TD target' is not supported by any uncertainty set, distributional assumption, or bound linking the transformed states to the true next-state distribution. Since F1-F3 are fixed deterministic maps (with only a deterministic top-alpha selection in F1), the 'Monte Carlo sampling' language in the text and Figure 5 is inaccurate, and increasing N or making the transformations more aggressive would drive the min target arbitrarily low. The method is a heuristic pessimism penalty, not a conservative estimator in a defined sense. The authors should either provide a formal statement with assumptions under which the min target lower-bounds the Bellman target, or revise the claims to describe the mechanism as data augmentation with a pessimistic heuristic.
  2. [Supplementary S1, F2 and Eq. (5)] The second transformation reverses the temporal order of a T-length price sequence to form F2. Since Eq. (5) evaluates a 'next-step' state s_{t+1}^{(n)} from a reversed sequence, that state is temporally prior to the current state in the original data, so it is not a plausible future market state. This breaks the causal structure of the TD target and undermines the interpretation of the min as approximating worst-case future payoffs. The authors should either exclude F2 from the TD-target computation or provide an argument for why a reversed-time state is a meaningful adversarial future.
  3. [Section 2, 'Decoupled state space'] The partial-offline formulation relies on the assertion that 'individual buying and selling actions typically have minimal impact on market dynamics,' making market transitions action-free. No evidence is provided that the trading amounts used in the experiments (e.g., lots of 100 or 200 shares across an 88-stock portfolio) do not themselves affect market state transitions, especially for less liquid securities. If actions do influence prices, the rewards and TD targets computed from transformed market states are not realizable, and the bilevel training does not evaluate actual outcomes. This assumption should be tested empirically or at least discussed as a limiting condition with a concrete validity check.
  4. [Figure 10] The claim that transformation-based TD reduces value overestimation is supported only by Figure 10, which reports the discrepancy between learned Q-values and a single-trajectory discounted return on training trajectories. This does not establish that the min-target is conservative on held-out or out-of-distribution states, which is the setting the paper emphasizes. A proper evaluation would compare Q-value errors on test-period states or against a calibrated uncertainty interval; without such evidence, the value-overestimation argument remains unverified.
minor comments (5)
  1. [Section 4, Table 1] The text states that RL-based results are 'from 10 random training seeds' in one place and 'averaged across three random training seeds' later; the number of seeds and the reporting convention should be made consistent.
  2. [Eq. (5) notation] In Eq. (5), the notation {s_{t+1}^{(n)}}_{n=1}^N is used while the min ranges over n=0:N, so the definition of the n=0 term and the indexing for the transformed states should be stated explicitly.
  3. [Figure 8 caption] The caption reports mean results over 3 seeds but no error bars are shown; adding variability measures would strengthen the ablation claims.
  4. [Section 4, 'Impact of the transformation-based conservative TD ensembles'] The comparison in Figure 9 against 'minimum value of ensemble Q' and 'mean value of ensemble Q' baselines uses multi-Q-networks with real future data, while MetaTrader uses transformed data with a single pair of target networks; the text should clarify which factors differ and how the comparison isolates the effect of the transformed-data min target.
  5. [General] No code or data is released, which limits reproducibility of the reported standard deviations and ablations; the authors should consider providing an implementation and configuration details sufficient to reproduce Table 1.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central performance claims are external empirical comparisons against held-out market data and independent baselines.

full rationale

The paper's central claim is empirical: MetaTrader outperforms baselines on the CSI-300 and NASDAQ-100 test sets (Table 1) and in the online adaptation setup (Figures 6-7). These results are evaluated against actual future market trajectories and external methods, not against the transformed data used in training, so the outcome is not defined by the method's own equations. The bilevel objective and Eq. (5) specify a training loss; the reported returns, Sharpe ratios, and drawdowns are measured on held-out test periods. The only notable self-citation is StockFormer [Gao et al., 2023a], which supplies the backbone architecture, pretrained feature extractor, datasets, and one comparison baseline. That citation is used as a building block and baseline, not as an authority for the paper's novelty, and the comparison against StockFormer is an independent empirical measurement on public data. The skeptic concern that Eq. (5)'s min over F1-F3 lacks a formal worst-case guarantee is a correctness or validity issue, not circularity: the method does not define the quantity of interest in terms of its own predictions, and no fitted parameter is renamed as a prediction. No uniqueness theorem is imported from the authors' prior work, and no derived result reduces to its own input by construction. The derivation chain is self-contained with respect to circularity.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The central method depends on the decoupled-state assumption, the heuristic worst-case TD target, and the recency assumption for finetuning. These are domain assumptions or ad hoc heuristics rather than derived results. Several hyperparameters (alpha, T, Delta, N, M') are chosen without sensitivity analysis, and M' is not reported.

free parameters (5)
  • F1 top percentile alpha = 10%
    Hyperparameter for F1 transformation; chosen without sensitivity analysis, affects the augmented data distribution and the TD target ensemble.
  • Sequence length T = 64
    Set to approximate one quarter of trading days; affects subset construction and transformations F2 and F3.
  • Downsampling step Delta = 4
    Chosen ad hoc for F3; no sensitivity analysis is reported.
  • Number of transformations N = 3
    The paper says N is scalable but uses 3 in all experiments.
  • M' recent subsets for finetuning = unspecified
    Algorithm 2 uses the most recent M' subsets, but M' is never reported, making the finetuning setup hard to reproduce.
assumptions (3)
  • domain assumption Market state transitions are action-free and balance transitions are deterministic given the action
    Section 2 states this decoupling and uses it to evaluate rewards for arbitrary actions and to define Eq. (5). If trading actions move prices, the partial-offline reward evaluation is invalid.
  • ad hoc to paper F1-F3 generate plausible out-of-distribution market states whose minimum TD target is a conservative estimate
    Eq. (5) takes the minimum over transformed next states; no argument is given that these transformations cover the true distribution shift or bound the true Q-value.
  • domain assumption Recent training data better match the test distribution
    Section 3: 'training data closer to the test set may better capture trend patterns that align with those in the test set'. This motivates the two-stage training design.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Your Offline Policy is Not Trustworthy: Bilevel Reinforcement Learning for Sequential Portfolio Optimization." pith.science (2026). https://pith.science/paper/UNCOJKTK

@misc{pith2026250512759,
  author       = {Pith},
  title        = {Pith review of: Your Offline Policy is Not Trustworthy: Bilevel Reinforcement Learning for Sequential Portfolio Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UNCOJKTK}},
  note         = {Machine review of arXiv:2505.12759}
}
read the original abstract

Reinforcement learning (RL) has shown significant promise for sequential portfolio optimization tasks, such as stock trading, where the objective is to maximize cumulative returns while minimizing risks using historical data. However, traditional RL approaches often produce policies that merely memorize the optimal yet impractical buying and selling behaviors within the fixed dataset. These offline policies are less generalizable as they fail to account for the non-stationary nature of the market. Our approach, MetaTrader, frames portfolio optimization as a new type of partial-offline RL problem and makes two technical contributions. First, MetaTrader employs a bilevel learning framework that explicitly trains the RL agent to improve both in-domain profits on the original dataset and out-of-domain performance across diverse transformations of the raw financial data. Second, our approach incorporates a new temporal difference (TD) method that approximates worst-case TD estimates from a batch of transformed TD targets, addressing the value overestimation issue that is particularly challenging in scenarios with limited offline data. Our empirical results on two public stock datasets show that MetaTrader outperforms existing methods, including both RL-based approaches and traditional stock prediction models.

Figures

Figures reproduced from arXiv: 2505.12759 by the authors.

Figure 1
Figure 1. A comparison of MetaTrader and existing RL-based trading methods. a, Existing RL-for-finance methods typically adopt an offline training setup rather than online RL, causing them to struggle with the generalization-optimality dilemma, a common challenge in the inherently non-stationary financial market. b, MetaTrader tackles this paradox through: (1) specialized data transformations to simulate OOD financial data, (… view at source ↗
Figure 2
Figure 2. The MDP in the partial-offline RL setup for sequential portfolio optimization. The MDP consists of decoupled pairs of action-free market states and action-dependent balance states, with market states restricted to the offline training set. Unlike standard offline RL, where no new rewards are accessible during policy optimization, the partial-offline setup allows the agent to interact with the fixed training set, exp… view at source ↗
Figure 3
Figure 3. The bilevel learning scheme of MetaTrader based on transformed market data. In the inner optimization loop (blue arrows), we optimize the model parameters on a batch of data subsets. In the outer optimization loop (red arrows), we perform bilevel gradient updates by explicitly evaluating the inner-loop parameters against another batch of data subsets. This process leads to a more generalizable agent and prevents ove… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: An example of market data transformations. F1 selects the top α% of assets with the highest price gains and inverses their original growth rate to declines to simulate unexpected short-term disruptions. F2 reverses the temporal order of a T-length sequence to simulate …
Figure 5
Figure 5. Figure 5: Transformation-based TD learning with worst-case bootstrapping. The left part represents the TD estimate, and the right part corresponds to the TD target. By approximating worst-case future payoffs through a Monte Carlo method over a batch of data transformations, our …
Figure 6
Figure 6. Figure 6: The cumulative returns under the online adaptation setup. We divide the entire test set into three equal-length splits and progressively finetune the models over the streaming data. All results are obtained from models trained with 10 random seeds. We compare MetaTrade…
Figure 7
Figure 7. Figure 7: Full comparisons in all metrics under the online adaptation setup. The online adaptation setup more effectively demonstrates the advantages of bilevel policy learning and finetuning for efficient domain adaptation, enabling MetaTrader to outperform StockFormer by signi…
Figure 8
Figure 8. Figure 8: Analyses of data transformation techniques for OOD policy learning (Algorithm 1). We report the mean results on the CSI dataset over 3 seeds. DT: Data Transformation. and by 44.4% on the NASDAQ dataset (1.30 vs. 0.90). In finance, the Sharpe ratio (also known as the re…
Figure 9
Figure 9. Figure 9: Ablation studies of transformation-based TD ensembles. We compare our approach with the following: (1) the use of the original TD method in bilevel learning, and (2-3) baseline models that compute TD targets based on real future market data using an ensemble of target …
Figure 10
Figure 10. Figure 10: The disparities between the predicted values by the critic and the true discounted future rewards. A larger disparity signifies a more pronounced value overestimation issue in offline RL. The results are obtained under the offline evaluation setup on the CSI dataset. …
Figure 11
Figure 11. Figure 11: Experiments on the expanded dataset with 587 stocks. We follow the same offline evaluation and online adaptation setups and demonstrate that MetaTrader achieves more significant performance gains over existing RL-based trading methods than those on small sets. Lee et …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 39 canonical work pages

  1. [1]

    An, G., Moon, S., Kim, J.-H., and Song, H. O. (2021). Uncertainty-based offline reinforcement learning with diversified q-ensemble. In NeurIPS , volume 34, pages 7436--7447

    work page 2021

  2. [2]

    Antoniou, A., Edwards, H., and Storkey, A. (2019). How to train your maml. In ICLR

    work page 2019

  3. [3]

    Briola, A., Turiel, J., Marcaccioli, R., Cauderan, A., and Aste, T. (2021). Deep reinforcement learning for active high frequency trading. arXiv preprint arXiv:2101.07107

    arXiv 2021

  4. [4]

    Cheng, C., Song, L., Xue, R., Wang, H., Sun, H., Ge, Y., and Shan, Y. (2023). Meta-adapter: An online few-shot learner for vision-language model. In NeurIPS

    work page 2023

  5. [5]

    J., Torn \'e , R

    Day, B. J., Torn \'e , R. V., Simidjievski, N., and Lio, P. (2022). Attentional meta-learners for few-shot polythetic classification. In ICML

    work page 2022

  6. [6]

    Deng, Y., Bao, F., Kong, Y., Ren, Z., and Dai, Q. (2016). Deep direct reinforcement learning for financial signal representation and trading. IEEE transactions on neural networks and learning systems , 28(3):653--664

    work page 2016

  7. [7]

    L., Sutskever, I., and Abbeel, P

    Duan, Y., Schulman, J., Chen, X., Bartlett, P. L., Sutskever, I., and Abbeel, P. (2017). Rl ^2 : Fast reinforcement learning via slow reinforcement learning. In ICLR

    work page 2017

  8. [8]

    Duan, Y., Wang, L., Zhang, Q., and Li, J. (2022). Factorvae: A probabilistic dynamic factor model based on variational autoencoder for predicting cross-sectional stock returns. In AAAI

    work page 2022

Show all 46 references
  1. [9]

    Feng, F., He, X., Wang, X., Luo, C., Liu, Y., and Chua, T.-S. (2019). Temporal relational ranking for stock prediction. ACM Transactions on Information Systems (TOIS) , 37(2):1--30

    2019

  2. [10]

    Finn, C. (2018). Learning to Learn with Gradients . PhD thesis, University of California, Berkeley, USA

    2018

  3. [11]

    Finn, C., Abbeel, P., and Levine, S. (2017). Model-agnostic meta-learning for fast adaptation of deep networks. In ICML

    2017

  4. [12]

    Gao, S., Wang, Y., and Yang, X. (2023a). Stockformer: learning hybrid trading machines with predictive coding. In IJCAI

    2023

  5. [13]

    Gao, Y., Zhang, R., Guo, J., Wu, F., Yi, Q., Peng, S., Lan, S., Chen, R., Du, Z., Hu, X., et al. (2023b). Context shift reduction for offline meta-reinforcement learning. In NeurIPS

    2023

  6. [14]

    Greenberg, I., Mannor, S., Chechik, G., and Meirom, E. (2023). Train hard, fight easy: Robust meta reinforcement learning. In NeurIPS

    2023

  7. [15]

    Gupta, A., Mendonca, R., Liu, Y., Abbeel, P., and Levine, S. (2018). Meta-reinforcement learning of structured exploration strategies. In NeurIPS

    2018

  8. [16]

    Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In ICML

    2018

  9. [17]

    Hospedales, T., Antoniou, A., Micaelli, P., and Storkey, A. (2021). Meta-learning in neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence , 44(9):5149--5169

    2021

  10. [18]

    A., Teh, Y

    Humplik, J., Galashov, A., Hasenclever, L., Ortega, P. A., Teh, Y. W., and Heess, N. (2019). Meta reinforcement learning as task inference. arXiv preprint arXiv:1905.06424

    2019 arXiv

  11. [19]

    and Kim, H

    Jeong, G. and Kim, H. Y. (2019). Improving financial trading decisions using deep q-learning: Predicting the number of shares, action strategies, and transfer learning. Expert Systems with Applications , 117:125--138

    2019

  12. [20]

    Kostrikov, I., Nair, A., and Levine, S. (2021). Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169

    2021 arXiv

  13. [21]

    Kumar, A., Zhou, A., Tucker, G., and Levine, S. (2020). Conservative q-learning for offline reinforcement learning. In NeurIPS , volume 33, pages 1179--1191

    2020

  14. [22]

    Kumar, P. (2023). Deep reinforcement learning for high-frequency market making. In ACML

    2023

  15. [23]

    Lee, S., Seo, Y., Lee, K., Abbeel, P., and Shin, J. (2022). Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In CoRL , pages 1702--1712. PMLR

    2022

  16. [24]

    Li, H., Shen, Y., and Zhu, Y. (2018). Stock price prediction using attention-based multi-input lstm. In ACML

    2018

  17. [25]

    Li, W., Wang, L., Xu, J., Huo, J., Gao, Y., and Luo, J. (2019). Revisiting local descriptor based image-to-class measure for few-shot learning. In CVPR

    2019

  18. [26]

    Liu, X.-Y., Xia, Z., Rui, J., Gao, J., Yang, H., Zhu, M., Wang, C., Wang, Z., and Guo, J. (2022). Finrl-meta: Market environments and benchmarks for data-driven financial reinforcement learning. In NeurIPS

    2022

  19. [27]

    Liu, X.-Y., Yang, H., Gao, J., and Wang, C. D. (2021). Finrl: Deep reinforcement learning framework to automate trading in quantitative finance. In ICAIF

    2021

  20. [28]

    Ma, Z., Guo, H., Chen, J., Li, Z., Peng, G., Gong, Y.-J., Ma, Y., and Cao, Z. (2023). Metabox: A benchmark platform for meta-black-box optimization with reinforcement learning. In NeurIPS

    2023

  21. [29]

    Mishra, N., Rohaninejad, M., Chen, X., and Abbeel, P. (2018). A simple neural attentive meta-learner. In ICLR

    2018

  22. [30]

    B., Levine, S., and Finn, C

    Mitchell, E., Rafailov, R., Peng, X. B., Levine, S., and Finn, C. (2021). Offline meta-reinforcement learning with advantage weighting. In ICML

    2021

  23. [31]

    S., Abbeel, P., Levine, S., and Finn, C

    Nagabandi, A., Clavera, I., Liu, S., Fearing, R. S., Abbeel, P., Levine, S., and Finn, C. (2019). Learning to adapt in dynamic, real-world environments through meta-reinforcement learning. In ICLR

    2019

  24. [32]

    H., Nair, A

    Pong, V. H., Nair, A. V., Smith, L. M., Huang, C., and Levine, S. (2022). Offline meta-reinforcement learning with online self-supervision. In ICML

    2022

  25. [33]

    S mundsson, S., Hofmann, K., and Deisenroth, M. P. (2018). Meta reinforcement learning with latent variable gaussian processes. arXiv preprint arXiv:1803.07551

    2018 arXiv

  26. [34]

    Saito, Y., Yao, J., and Joachims, T. (2024). Potec: Off-policy learning for large action spaces via two-stage policy decomposition. ICML

    2024

  27. [35]

    Schmidhuber, J. (1987). Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-... hook . PhD thesis, Technische Universit \"a t M \"u nchen

    1987

  28. [36]

    Tang, Y. (2022). Biased gradient estimate with drastic variance reduction for meta reinforcement learning. In ICML

    2022

  29. [37]

    Tavakoli, A., Pardo, F., and Kormushev, P. (2018). Action branching architectures for deep reinforcement learning. In AAAI , volume 32

    2018

  30. [38]

    Triantafillou, E., Zhu, T., Dumoulin, V., Lamblin, P., Evci, U., Xu, K., Goroshin, R., Gelada, C., Swersky, K., Manzagol, P.-A., et al. (2020). Meta-dataset: A dataset of datasets for learning to learn from few examples. In ICLR

    2020

  31. [39]

    Wang, H., Li, S., Wang, T., and Zheng, J. (2021). Hierarchical adaptive temporal-relational modeling for stock trend prediction. In IJCAI

    2021

  32. [40]

    Wang, J., Zhang, J., Jiang, H., Zhang, J., Wang, L., and Zhang, C. (2023). Offline meta reinforcement learning with in-distribution online adaptation. In ICML

    2023

  33. [41]

    Wu, H., Xu, J., Wang, J., and Long, M. (2021). Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. In NeurIPS

    2021

  34. [42]

    Wu, Y., Chen, X., Wang, C., Zhang, Y., and Ross, K. W. (2022). Aggressive q-learning with ensembles: Achieving both high sample efficiency and high asymptotic performance. In NeurIPS

    2022

  35. [43]

    and Cohen, S

    Xu, Y. and Cohen, S. B. (2018). Stock movement prediction from tweets and historical prices. In ACL

    2018

  36. [44]

    Ye, Y., Pei, H., Wang, B., Chen, P.-Y., Zhu, Y., Xiao, J., and Li, B. (2020). Reinforcement-learning based portfolio management with augmented asset movement prediction states. In AAAI

    2020

  37. [45]

    Zhao, K., Ma, Y., Liu, J., Zheng, Y., and Meng, Z. (2023). Ensemble-based offline-to-online reinforcement learning: From pessimistic learning to optimistic exploration. arXiv preprint arXiv:2306.06871

    2023 arXiv

  38. [46]

    Zheng, X., Liu, M., and Zhu, M. (2023). Deep hashing-based dynamic stock correlation estimation via normalizing flow. In IJCAI

    2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.