Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Learning from Expert Factors: Trajectory-level Reward Shaping for Formulaic Alpha Mining

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Trajectory-level Reward Shaping (TLRS) claims that dense rewards from exact subsequence matching against expert formulas speed up and stabilize RL-based alpha factor mining, improving Rank IC by 9.29% over potential-based shaping while…

desk verdict A plausible, practically useful reward-shaping idea with an honest empirical write-up, but its headline number is unsupported and its main theoretical guarantee (Prop. 3) is wrong; still worth refereeing. read the letter →

arxiv 2507.20263 v1 pith:M43TSJOA submitted 2025-07-27 cs.LG cs.AIq-fin.PM

Junjie Zhao , Chengxi Zhang , Chenkai Wang , Peng Yang This is my paper
classification cs.LGcs.AIq-fin.PM
keywords alphafactorminingtrajectory-levelrewardshapingreinforcementlearningreversePolishnotationexactsubsequencematchingrankinformationcoefficientquantitativefinance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Reinforcement-learning-based $\alpha$ factor mining suffers from sparse rewards: the agent gets a score only after generating an entire formulaic expression, so exploration of the symbolic space is slow and unstable. The paper asks whether expert-designed formulas can provide dense intermediate credit during generation, and proposes Trajectory-level Reward Shaping (TLRS) as an answer. TLRS rewards each step by the change in the fraction of expert formula sub-expressions exactly matched by the partially generated expression, using a temporal-difference signal of the same form as potential-based shaping. This gives a syntax-respecting, cheap shaping signal, reducing the shaping computation from $O(N \cdot L \cdot d)$ to $O(N \cdot L)$ relative to distance-based baselines. The paper reports a 9.29% Rank IC improvement over potential-based shaping baselines and faster, more stable convergence across six U.S. and Chinese equity indices, with final out-of-sample factor quality comparable to the strongest baseline.

What carries the argument

The load-bearing object is the exact subsequence matching ratio. For a partial Reverse Polish Notation expression of length $t$, $n_{1,t}$ counts how many expert-demonstration subsequences of length $t$ exactly match it, and $N_t$ is the total number of length-$t$ expert subsequences; their ratio $\Phi(s_t)=n_{1,t}/N_t$ is the state potential. The shaping reward is the temporal difference $f_t = \Phi(s_{t+1}) - \Phi(s_t)$ (with $\gamma=1$), so a step that moves the partial expression onto an exact expert sub-expression path receives positive credit. Because matching is done on token sequences directly, no vector embedding or distance metric is involved, which is what removes the $O(d)$ feature-dimension factor from the shaping cost. A second mechanism, reward centering, tracks $\bar r_{t+1} = \bar r_t + \beta(r_{t+1}-\bar r_t)$ and subtracts it from the value-target update, so the value network estimates the differential value rather than an ever-growing constant offset.

What would settle it

Replace the handcrafted expert formula set with an equal number of random, syntactically valid RPN expressions and rerun TLRS on the same six indices. If final Rank IC and convergence speed remain at the same level as with expert formulas, the advertised gain comes from the shaping mechanism's density rather than from expert knowledge; if they drop, the expert set is load-bearing.

Watch

Extended reading notes

Core claim

The central claim is that exact subsequence matching against a fixed set of handcrafted expert $\alpha$ factors is a reliable and efficient way to shape rewards during RPN formula generation. Defining the potential $\Phi(s_t)=n_{1,t}/N_t$ as the ratio of expert sub-expressions of length $t$ that exactly match the current partial sequence, the shaping reward $f_t=\Phi(s_{t+1})-\Phi(s_t)$ is added to the terminal information-coefficient reward, with $\gamma=1$ so the factor-mining objective retains no length bias. The paper argues this preserves the optimal policy, and that the ratio measures how close the current policy is to the expert policy that produced the formulas. TLRS also subtracts an online estimate of average reward, so the value learner fits a differential value function instead of a growing offset, which the paper claims stabilizes training. The empirical claim is that TLRS converges faster and more stably than no-shaping, PBRS, and DPBA baselines on six indices, improves Rank IC by 9.29% over potential-based shaping, and produces test-phase factors whose IC and RankIC are comparable to the strongest baseline while remaining interpretable formulas.

Load-bearing premise

The method rests on the assumption that the fraction of handpicked expert sub-expressions exactly matching the current partial expression is a trustworthy measure of partial-expression quality, so that pushing that fraction up guides the policy toward genuinely predictive formulas rather than merely toward expert-lookalike ones.

Editorial extensions

If this is right

  • Dense shaping rewards should let RL alpha miners escape the cold-start problem of random search, reaching useful factors with fewer environment evaluations.
  • The $O(N \cdot L)$ shaping cost makes the method practical as the feature dictionary grows, since distance-based shaping scales linearly in feature dimension.
  • Because shaping is additive and potential-based with $\gamma=1$, it can be dropped into existing policy-gradient alpha-mining pipelines without changing the policy network or adding a separate network.
  • Setting the discount factor to 1 removes the length bias documented by the paper, so the learned policy does not prematurely emit short, under-expressive formulas.
  • On the six tested indices, the method's training-phase Rank IC improves by 9.29% over potential-based shaping while its out-of-sample factors remain interpretable formulas competitive with the strongest baseline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the same ratio-based shaping idea transfers to other symbolic generation tasks with a library of exemplars, such as program synthesis or unit-test generation; the transferable quantity is the change in the exact subsequence match ratio along the generation path.
  • Editorial extension: the paper's own out-of-sample table shows TLRS roughly matching, not beating, the strongest REINFORCE-style baseline in final IC; if that pattern holds, the practical value of TLRS is sample efficiency and stability rather than higher final factor quality.
  • Editorial extension: a direct test of whether expert knowledge is doing the work would be to replace the expert formulas with random syntactically valid RPN expressions; TLRS's convergence advantage should vanish if the matching signal is genuinely expert-driven.
  • Editorial extension: an additional stress test would vary the expert set's composition, for example momentum versus mean-reversion families, and check whether mined factors shift correspondingly; such a result would confirm the shaping signal is transferring financial inductive bias rather than generic structure.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Trajectory-level Reward Shaping (TLRS) for reinforcement-learning-based formulaic alpha mining. TLRS defines a potential function as the fraction of expert formula subsequences exactly matching the current partially generated RPN expression, and uses its temporal difference as a dense shaping reward; it also introduces a reward centering mechanism to reduce variance. The authors claim a 9.29% improvement in Rank IC over potential-based shaping baselines, faster and more stable convergence on six stock indices, and a reduction in time complexity from O(N·L·d) to O(N·L). The paper includes a formal MDP formulation, three theoretical propositions (discount-factor bias, policy invariance, syntax-semantics error bound), experiments against several baselines, and an ablation study.

Significance. TLRS addresses a real and important problem: sparse rewards in RL-based symbolic factor mining. The idea of using exact subsequence matching against expert formulas as a discrete, non-distance-based potential is simple and computationally appealing, and the empirical training curves in Fig. 3 do suggest a convergence speedup over the no-shaping baseline. The paper is also careful to include an analysis of the discount-factor issue (Prop. 1) and an ablation separating the two contributions. However, the central theoretical guarantee for the matching-based shaping signal (Prop. 3) is flawed, and the headline quantitative claim is not supported by the reported tables. The paper's own Table III indicates that TLRS is comparable to, not significantly better than, the strongest baseline, QFR, which tempers the advertised advantage.

major comments (4)
  1. [§IV-C, Proposition 3] Proposition 3 is internally inconsistent and does not establish the stated bound. The proof sets n_{1,t} = m^t, arguing that 'each of these m^t sequences is uniquely paired with an identical copy of itself.' This would only hold if the expert demonstration set contained every possible length-t token sequence. In the actual method (Section IV-A) and in the experiments (Section V-A), the expert set is a handcrafted collection of roughly 130 Alpha101-inspired formulas, so N_t is the number of length-t subsequences in that finite set, which is far smaller than m^t. Consequently, the derivation of ε_t < (t/m)^2 k, and hence the claim that the error caused by ignoring semantic equivalence is bounded, does not follow. Because this proposition is the only theoretical justification for using exact syntactic subsequence matching in place of a semantic similarity measure (Section III-B and Section IV-A), the central shaping mechanism lacks the advertised theoretical safety net.
  2. [§V-B and Abstract] The headline claim that TLRS 'boosting the Rank Information Coefficient by 9.29% over existing potential-based shaping algorithms' is not supported by any tabulated result or statistical test. Figure 3 shows learning curves averaged over five seeds, but no numeric Rank IC values are reported, and Table III evaluates only CSI300 and CSI500 and compares against MLP, XGBoost, LightGBM, GP, AlphaGen, and QFR—not against PBRS or DPBA. Moreover, the text in Section V-E acknowledges that TLRS 'does not demonstrate a significant statistical superiority' over QFR, which is the strongest baseline. The 9.29% figure therefore appears to be a single point on a training curve, and the reader cannot assess its variability or significance.
  3. [§V-B, Complexity Claim] The claimed computational advantage of TLRS—O(N·L) versus O(N·L·d) for PBRS—is not substantiated. Computing the shaping reward in Eq. (9) requires, for each step t, counting the number n_{1,t} of expert demonstration subsequences of length t that exactly match the current partial sequence. With N expert formulas of length L, a direct implementation of this count is O(N·L) per step, and over a trajectory of length L it is O(N·L^2). The paper does not describe a data structure (e.g., a trie or hashing scheme) that would reduce this to O(N·L) over the whole trajectory. Without such details, the complexity comparison is not verifiable.
  4. [§IV-C, Proposition 4] The reward centering derivation in Proposition 4 starts from the infinite-horizon discounted decomposition of Blackwell and then asserts that for the finite-horizon episodic MDP with γ=1 the constant term 'should be redefined' as E[Σ r(πθ)]. This is a redefinition rather than a derivation; in particular, the average reward r(πθ) in Eq. (13) is defined as a limit over T→∞ for a process that terminates, so its meaning and existence for the factor-mining MDP are unclear. The claimed variance-reduction benefit of reward centering may hold empirically, but the theoretical support provided is insufficient.
minor comments (5)
  1. [§V-A] The description of SPX conflates the S&P 500 with the Dow Jones Industrial Average; the parenthetical defines SPX as 'the Dow Jones Industrial Average, which tracks 30 major US blue-chip companies' immediately before listing DJI as a separate index. This should be corrected.
  2. [§V-B] The sentence 'QFR maintains superior performance across these scenarios by removing critic-network, employing a subtractive baseline, and applying targeted reward shaping' is inconsistent with the surrounding comparison of reward shaping algorithms and with Table III; it appears to be an editorial leftover and should be removed or reworded.
  3. [§II-C, §IV-A] The state indexing is inconsistent: Section II-C defines s_t = a_{1:t-1}, while Section IV-A refers to 'the current generated partial sequence s_t' of length t and uses n_{1,t}/N_t for length t. The off-by-one notation should be harmonized.
  4. [Throughout] There are numerous typos: 'noval similarity' (Section IV-A), 'performence' (Section V-A), 'facor-mining' (Section V-A), and 'Specially' (Proof of Prop. 4). A careful proofread is needed.
  5. [§V-F] The sentence 'The performance degradation in TLRS no RS highlights that reward shaping is essential for guiding exploration effectively with sparse rewards' is fine, but 'TLRS no RC demonstrate' is a subject-verb agreement error; also, the ablation curves are not quantified in the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the shaping signal is computed from external expert formulas, and the performance claims are evaluated out-of-sample against market data rather than against the shaping reward itself.

full rationale

TLRS's central mechanism (Eq. 9) defines the shaping potential as the exact subsequence match ratio n1,t/Nt between the current RPN prefix and a handcrafted Alpha101-inspired expert set. This is an externally specified input, not a fitted parameter renamed as a prediction, and the reported Rank IC and IC results are computed on a test period distinct from the training period, so the empirical claim does not reduce to the shaping construction. Proposition 2 reproduces the standard potential-based shaping invariance argument, and Proposition 4 applies standard reward centering; neither depends on a self-citation. The only self-reference is the QFR baseline [16], which is used as a comparison method rather than as a load-bearing premise. The main theoretical weakness is Proposition 3, whose proof sets n1,t = m^t, implicitly assuming the expert demonstration set contains every possible length-t sequence, whereas the actual algorithm uses a finite set of roughly 130 handcrafted formulas. That assumption makes the claimed error bound inapplicable to the implemented method. This is an unsupported proof step and a correctness risk, but it is not a circular reduction: the bound is not derived from a fitted parameter and then re-reported as an independent result, and the experimental comparisons retain content independent of the shaping signal.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method's central mechanism is a potential function built from expert-subsequence match ratios. This introduces a strong dependency on the choice and quality of the expert set and on the validity of applying average-reward centering to an episodic MDP. No new physical entities are introduced.

free parameters (2)
  • Number of expert demonstrations N = 130
    Selected on validation; performance peaked at N=130 (Section V-D).
  • Reward centering learning rate beta = 2e-3
    Tuned on validation; non-monotonic sensitivity (Section V-D).
assumptions (4)
  • domain assumption Factor-mining MDP and RPN token representation from AlphaGen are adopted unmodified.
    Section II-C; state transitions and syntax constraints are not re-specified in this paper.
  • ad hoc to paper The exact subsequence matching ratio n1,t/Nt is a valid potential function for reward shaping.
    Section IV-A, Eq. (9); the paper justifies via Proposition 2 (PBRS form) but the semantic-equivalence error bound in Proposition 3 is not rigorous.
  • ad hoc to paper Reward centering applies to a finite-horizon episodic MDP with gamma=1.
    Section IV-C Proposition 4; the proof starts from ergodic infinite-horizon average-reward theory and then redefines the constant term for finite steps, which is heuristic.
  • domain assumption The handcrafted expert formulas inspired by Alpha101 are a sufficiently representative guide for successful alpha mining.
    Section V-A 'expert demonstrations used in the experiment are handcrafted, inspired by Alpha101'; no evidence that these formulas are optimal or unbiased guides.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning from Expert Factors: Trajectory-level Reward Shaping for Formulaic Alpha Mining." pith.science (2026). https://pith.science/paper/M43TSJOA

@misc{pith2026250720263,
  author       = {Pith},
  title        = {Pith review of: Learning from Expert Factors: Trajectory-level Reward Shaping for Formulaic Alpha Mining},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M43TSJOA}},
  note         = {Machine review of arXiv:2507.20263}
}
read the original abstract

Reinforcement learning (RL) has successfully automated the complex process of mining formulaic alpha factors, for creating interpretable and profitable investment strategies. However, existing methods are hampered by the sparse rewards given the underlying Markov Decision Process. This inefficiency limits the exploration of the vast symbolic search space and destabilizes the training process. To address this, Trajectory-level Reward Shaping (TLRS), a novel reward shaping method, is proposed. TLRS provides dense, intermediate rewards by measuring the subsequence-level similarity between partially generated expressions and a set of expert-designed formulas. Furthermore, a reward centering mechanism is introduced to reduce training variance. Extensive experiments on six major Chinese and U.S. stock indices show that TLRS significantly improves the predictive power of mined factors, boosting the Rank Information Coefficient by 9.29% over existing potential-based shaping algorithms. Notably, TLRS achieves a major leap in computational efficiency by reducing its time complexity with respect to the feature dimension from linear to constant, which is a significant improvement over distance-based baselines.

Figures

Figures reproduced from arXiv: 2507.20263 by the authors.

Figure 1
Figure 1. Three interchangeable forms of an alpha factor: (a) [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The detailed pipeline for the proposed TLRS algorithm. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Training-phase correlation between mined factor values and the prices of the six index constituents for all investigated [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Training-phase correlation between mined factor values [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: Sensitivity analysis of TLRS to number of expert [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AlphaEval: A Comprehensive and Efficient Evaluation Framework for Formula Alpha Mining

    cs.AI 2025-08 conditional novelty 6.0 of 10

    AlphaEval scores alpha mining models on prediction, stability, robustness, logic, and diversity, replacing backtests with fast parallel metrics that the paper claims align with backtest outcomes.

Reference graph

Works this paper leans on

36 extracted references · 25 canonical work pages · cited by 1 Pith paper

  1. [1]

    Saoftrl: A novel adaptive algorithmic framework for enhancing online portfolio selection,

    R. Shi and D. P. Palomar, “Saoftrl: A novel adaptive algorithmic framework for enhancing online portfolio selection,” IEEE Transactions on Signal Processing , 2024

    work page 2024

  2. [2]

    Optimal mean-reverting port- folio with leverage constraint for statistical arbitrage in finance,

    Z. Zhao, R. Zhou, and D. P. Palomar, “Optimal mean-reverting port- folio with leverage constraint for statistical arbitrage in finance,” IEEE Transactions on Signal Processing, vol. 67, no. 7, pp. 1681–1695, 2019

    work page 2019

  3. [3]

    Mean-reverting portfolio with budget constraint,

    Z. Zhao and D. P. Palomar, “Mean-reverting portfolio with budget constraint,” IEEE Transactions on Signal Processing , vol. 66, no. 9, pp. 2342–2357, 2018

    work page 2018

  4. [4]

    101 formulaic alphas,

    Z. Kakushadze, “101 formulaic alphas,” Wilmott, vol. 2016, no. 84, pp. 72–81, 2016

    2016

  5. [5]

    Openfe: automated feature generation with expert-level per- formance,

    T. Zhang, Z. A. Zhang, Z. Fan, H. Luo, F. Liu, Q. Liu, W. Cao, and L. Jian, “Openfe: automated feature generation with expert-level per- formance,” in International Conference on Machine Learning . PMLR, 2023, pp. 41 880–41 901

    work page 2023

  6. [6]

    Application research of the xgboost-svm combina- tion model in quantitative investment strategy,

    H. Zhu and A. Zhu, “Application research of the xgboost-svm combina- tion model in quantitative investment strategy,” in2022 8th International Conference on Systems and Informatics (ICSAI) . IEEE, 2022, pp. 1–7

    work page 2022

  7. [7]

    Research on multi factor stock selection model based on lightgbm and bayesian optimization,

    Z. Li, W. Xu, and A. Li, “Research on multi factor stock selection model based on lightgbm and bayesian optimization,” Procedia Computer Science, vol. 214, pp. 1234–1240, 2022

    work page 2022

  8. [8]

    Accurate label refinement from multiannotator of remote sensing data,

    X. Wang, L. Chen, T. Ban, D. Lyu, Y . Guan, X. Wu, X. Zhou, and H. Chen, “Accurate label refinement from multiannotator of remote sensing data,” IEEE Transactions on Geoscience and Remote Sensing , vol. 61, pp. 1–13, 2023

    work page 2023

Show all 36 references
  1. [9]

    Multiscale temporal dynamic learning for time series classification,

    S. Liu, X. Zhou, and H. Chen, “Multiscale temporal dynamic learning for time series classification,” IEEE Transactions on Knowledge and Data Engineering, 2025

    2025

  2. [10]

    Q-learning-based spec- trum access for multimedia transmission over cognitive radio networks,

    X.-L. Huang, Y .-X. Li, Y . Gao, and X.-W. Tang, “Q-learning-based spec- trum access for multimedia transmission over cognitive radio networks,” IEEE Transactions on Cognitive Communications and Networking , vol. 7, no. 1, pp. 110–119, 2020

    2020

  3. [11]

    Long short-term memory,

    S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Computation, vol. 9, no. 8, pp. 1735–1780, 1997

    1997

  4. [12]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017

    2017

  5. [13]

    Prediction of small-molecule compound solu- bility in organic solvents by machine learning algorithms,

    Z. Ye and D. Ouyang, “Prediction of small-molecule compound solu- bility in organic solvents by machine learning algorithms,” Journal of cheminformatics, vol. 13, no. 1, p. 98, 2021

    2021

  6. [14]

    Autoalpha: an efficient hierarchical evolutionary algorithm for mining alpha factors in quantitative invest- ment,

    T. Zhang, Y . Li, Y . Jin, and J. Li, “Autoalpha: an efficient hierarchical evolutionary algorithm for mining alpha factors in quantitative invest- ment,” arXiv preprint arXiv:2002.08245 , 2020

    2002 arXiv

  7. [15]

    Generating synergistic formulaic alpha collections via reinforcement learning,

    S. Yu, H. Xue, X. Ao, F. Pan, J. He, D. Tu, and Q. He, “Generating synergistic formulaic alpha collections via reinforcement learning,” in Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , 2023, pp. 5476–5486

    2023

  8. [16]

    Quantfactor reinforce: Mining steady formulaic alpha factors with variance-bounded reinforce,

    J. Zhao, C. Zhang, M. Qin, and P. Yang, “Quantfactor reinforce: Mining steady formulaic alpha factors with variance-bounded reinforce,” IEEE Transactions on Signal Processing , vol. 73, pp. 2448–2463, 2025

    2025

  9. [17]

    Policy invariance under reward transformations: Theory and application to reward shaping,

    A. Y . Ng, D. Harada, and S. Russell, “Policy invariance under reward transformations: Theory and application to reward shaping,” in Icml, vol. 99, 1999, pp. 278–287

    1999

  10. [18]

    Principled methods for advising reinforcement learning agents,

    E. Wiewiora, G. W. Cottrell, and C. Elkan, “Principled methods for advising reinforcement learning agents,” in Proceedings of the 20th international conference on machine learning (ICML-03) , 2003, pp. 792–799

    2003

  11. [19]

    Reinforcement learning from demonstration through shaping,

    T. Brys, A. Harutyunyan, H. B. Suay, S. Chernova, M. E. Taylor, and A. Now ´e, “Reinforcement learning from demonstration through shaping,” in Twenty-fourth international joint conference on artificial intelligence, 2015

    2015

  12. [20]

    Artificial intelligence stock selection: Generalized linear model,

    X. Lin and Y . Chen, “Artificial intelligence stock selection: Generalized linear model,” Huatai Securities, Tech. Rep., 6 2017

    2017

  13. [21]

    Prox- imal policy optimization algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Prox- imal policy optimization algorithms,” arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  14. [22]

    Robot shaping: Developing autonomous agents through learning,

    M. Dorigo and M. Colombetti, “Robot shaping: Developing autonomous agents through learning,” Artificial intelligence, vol. 71, no. 2, pp. 321– 370, 1994

    1994

  15. [23]

    A cold-start-free reinforce- ment learning approach for traffic signal control,

    N. Xiao, L. Yu, J. Yu, P. Chen, and Y . Liu, “A cold-start-free reinforce- ment learning approach for traffic signal control,” Journal of Intelligent Transportation Systems, vol. 26, no. 4, pp. 476–485, 2022

    2022

  16. [24]

    R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction . MIT press, 2018

    2018

  17. [25]

    The arcade learning environment: An evaluation platform for general agents,

    M. G. Bellemare, Y . Naddaf, J. Veness, and M. Bowling, “The arcade learning environment: An evaluation platform for general agents,” Jour- nal of Artificial Intelligence Research , vol. 47, pp. 253–279, 2013

    2013

  18. [26]

    A survey of robot learning from demonstration,

    B. D. Argall, S. Chernova, M. Veloso, and B. Browning, “A survey of robot learning from demonstration,” Robotics and autonomous systems , vol. 57, no. 5, pp. 469–483, 2009

    2009

  19. [27]

    Apprenticeship learning via inverse rein- forcement learning,

    P. Abbeel and A. Y . Ng, “Apprenticeship learning via inverse rein- forcement learning,” in Proceedings of the twenty-first international conference on Machine learning , 2004, p. 1

    2004

  20. [28]

    Multi-resolution exploration in continuous spaces,

    A. Nouri and M. Littman, “Multi-resolution exploration in continuous spaces,” Advances in neural information processing systems , vol. 21, 2008

    2008

  21. [29]

    Maximum entropy inverse reinforcement learning

    B. D. Ziebart, A. L. Maas, J. A. Bagnell, A. K. Dey et al., “Maximum entropy inverse reinforcement learning.” in Aaai, vol. 8. Chicago, IL, USA, 2008, pp. 1433–1438

    2008

  22. [30]

    From mimic to counteract: a two-stage reinforcement learning algorithm for google research football,

    J. Zhao, J. Lin, X. Zhang, Y . Li, X. Zhou, and Y . Sun, “From mimic to counteract: a two-stage reinforcement learning algorithm for google research football,” Neural Computing and Applications , vol. 36, no. 13, pp. 7203–7219, 2024

    2024

  23. [31]

    Learning continuous semantic representations of symbolic expressions,

    M. Allamanis, P. Chanthirasegaran, P. Kohli, and C. Sutton, “Learning continuous semantic representations of symbolic expressions,” in Inter- national Conference on Machine Learning . PMLR, 2017, pp. 80–88

    2017

  24. [32]

    Reward centering,

    A. Naik, Y . Wan, M. Tomar, and R. S. Sutton, “Reward centering,”arXiv preprint arXiv:2405.09999, 2024

    2024 arXiv

  25. [33]

    Stephens

    T. Stephens. (2015) gplearn: Genetic programming in python, with a scikit-learn inspired api. [Online]. Available: https://github.com/ trevorstephens/gplearn

    2015

  26. [34]

    Stable-baselines3: Reliable reinforcement learning implementa- tions,

    A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dor- mann, “Stable-baselines3: Reliable reinforcement learning implementa- tions,” Journal of Machine Learning Research, vol. 22, no. 268, pp. 1–8, 2021

    2021

  27. [35]

    Qlib: An ai-oriented quantitative investment platform,

    X. Yang, W. Liu, D. Zhou, J. Bian, and T.-Y . Liu, “Qlib: An ai-oriented quantitative investment platform,” arXiv preprint arXiv:2009.11189 , 2020

    2009 arXiv

  28. [36]

    Learning repre- sentations by back-propagating errors,

    D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning repre- sentations by back-propagating errors,” nature, vol. 323, no. 6088, pp. 533–536, 1986

    1986

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.