Pith. sign in

REVIEW 4 major objections 4 minor 8 references

Reinforcement Learning for Stock Transactions

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper reports that a Q-learning agent with linear function approximation outperforms a fixed-drop baseline in more than half of the buy-timing windows for all four companies examined.

desk verdict A clear, reproducible course project undermined by lookahead bias: the state includes the close price used for scoring, so the main '>50%' claim is an artifact. read the letter →

arxiv 2505.16099 v2 pith:36EIT4TW submitted 2025-05-22 cs.LG

Ziyi Zhou , Nicholas Stern , Julien Laasri This is my paper · ORCID
classification cs.LG
keywords reinforcementlearningQ-learninglinearfunctionapproximationdeepstocktradingbuytimingMarkovdecisionprocesspricemovementprediction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a simple reinforcement-learning agent can learn when to buy a stock inside a fixed time window using daily open, high, low, and close prices. The authors define a Markov decision process in which the agent must buy within each window, receives a reward based on how far the purchase price is below the window's starting price, and can use recent price history as its state. They compare exact Q-learning, Q-learning with linear function approximation, and deep Q-learning against a baseline that buys only after a fixed price drop. The paper's central claim is that the linear approximation agent outperforms this baseline in more than half of the test windows for all four companies studied, which the authors take as evidence that approximate Q-learning lets the agent exploit more information and is a step toward useful trading policies. If that claim holds, it suggests even a low-dimensional linear model can extract a small, consistent timing signal from daily stock data.

What carries the argument

The load-bearing object is a custom Markov decision process with time windows: each window of w days is one episode, the agent chooses 'buy' or 'wait' on each day, buying ends the episode, and the reward for a buy is the negative of the difference between the purchase-day price and the first-day price of the window, so positive rewards mean buying low. States are vectors of recent daily price data for the approximate agents or signs of the day's close-minus-open movement for exact Q-learning. The linear approximation agent represents the Q-value as a weighted sum of binary indicator features for state-action pairs and updates the weights with the standard Q-learning temporal-difference rule, which is what lets it handle the continuous price state space. This machinery is what the paper credits for the more-than-50% win rate: the continuous state allows the agent to use actual price levels rather than only up/down signs.

What would settle it

Re-run the four-company evaluation on daily data from 2019 to 2024, computing for each time window whether the linear agent's profit exceeds the baseline's, and count the fraction across all windows; if the pooled win rate is at or below 50%, or if the edge appears on only one or two tickers, the paper's central claim would be falsified.

Watch

Extended reading notes

Core claim

The core discovery, as the paper states it, is empirical: over unseen test data, the agent using Q-learning with linear function approximation beats the baseline in more than 50% of the time windows for Apple, Amazon, Microsoft, and Google. Because the baseline is deterministic, the comparison hinges on the linear agent's average profit over 51 randomized runs of its epsilon-greedy policy. The authors interpret this directional edge as evidence that the approximate Q-learning framework 'allows the agents to capitalize on more information' than the sign-only states of exact Q-learning. They are careful to note that the agents' policies have not fully converged and that all confidence intervals overlap, so the paper does not claim a reliable winner among agents; the claim is the more-than-half win rate and the promise of the approximate framework.

Load-bearing premise

The central claim rests on the assumption that the recent price history shown to the agent contains enough information about near-future price direction, and that stock behavior after 2005 is stable enough that an 80/10/10 time split gives training, validation, and test periods that look alike.

Editorial extensions

If this is right

  • If the more-than-50% win rate generalizes, a simple linear Q-learning agent can extract a small but real timing advantage from daily open-high-low-close data without deep networks or high-frequency data.
  • The approximate Q-learning framework should extend directly to a symmetric 'sell or wait' agent, since the authors define buy timing as easily generalizable to selling.
  • State spaces enriched with volume, momentum, or external news data would let the same linear update rule test whether the edge grows, because the framework already supports arbitrary features.
  • The near-chance supervised classification results imply that direct price-movement prediction is not the bottleneck; the RL framing itself, with its delayed rewards, is the proposed route to trading signals.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension would be to replace the single 80/10/10 temporal split with rolling-window backtests across multiple market regimes; the paper's split alone does not establish that the edge is robust to bull, bear, and sideways periods.
  • The linear agent's edge could come largely from the reward's benchmark to the window-start price rather than from learned price dynamics; a controlled variant that rewards buying before the next-day move would separate these two effects.
  • One could test the more-than-50% claim statistically by pairing agent and baseline profits window by window and running a sign test across all four companies; the paper does not report such a test, and overlapping confidence intervals leave room for the edge to be noise.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper defines a discrete-time MDP in which an agent must choose a day to buy a stock within a fixed window of daily OHLC data. Four agents are compared: a threshold baseline, tabular Q-learning on discretized up/down movement states, linear function approximation with continuous price-vector states, and a neural-network Q-function. The authors also test regression and classification price predictors, which they ultimately abandon as uninformative. Evaluation is performed on held-out test windows; because policies do not converge, each agent is run 51 times and the distributions of average profit are compared. The paper concludes that approximate Q-learning is a promising direction and that the linear approximation agent outperforms the baseline more than 50% of the time in all four companies.

Significance. If the central empirical claim were valid, it would be a modest but useful existence proof that a simple linear RL agent can exploit short-window price regularities to beat a naive threshold rule on daily stock data. The paper has real strengths: the code is publicly available, the evaluation pipeline is described in enough detail to reproduce, and the authors honestly disclose that their agents do not converge and that their price predictors fail to beat randomness. However, the significance is limited by a target-leakage confound in the state/reward design and by the absence of proper inferential statistics for the headline claim. As it stands, the results do not establish the advertised advantage of the approximate Q-learning framework.

major comments (4)
  1. [§4.1.3, §5.2.1, §6] The central comparison is contaminated by target leakage. The linear agent's state is defined as (p_{-3}, p_{-2}, p_{-1}, p_0), where p_0 is the current day's price vector, and its reward for buying is -(p_0 - p_i), with p_i the first day of the window. Section 5.2.1 then scores every agent by exactly this close-price difference. Thus the quantity being maximized is a deterministic function of a feature already contained in the state. Depending on whether the day's close is known before the buy decision, this is either lookahead (if the decision is made before the close) or direct target leakage (if it is made after the close); either way, a policy that simply buys when p_0 is low relative to p_i can achieve high scores without learning any temporal trend. The conclusion in Section 6 that approximate Q-learning 'allows the agents to capitalize on more information' is therefore not supported by the reported comparison. The authors should remove p_0 from the state, or score purchases at a price not observed at decision time, and rerun the experiments.
  2. [§4.1.1 and §5.2.1] The baseline is not evaluated under the same execution assumption as the approximate agents. The baseline buys when the price decreases to $d less than the initial price, which is naturally a condition on an intraday price, but Section 5.2.1 scores all agents using the close-price difference on the day of the buy. If the baseline triggers on an intraday low and is filled at the close, it can systematically buy at a price above its trigger, putting it at a disadvantage relative to the linear agent, for which the state price and the execution price coincide. The authors should specify exactly what price the baseline receives and make the trigger and execution-price assumptions consistent across all agents, since the linear-agent-versus-baseline claim is the main result.
  3. [§6 and Tables 2–5] The headline claim that the linear agent outperforms the baseline more than 50% of the time in all four companies is not documented. No per-company win rates, paired sign-test results, or effect sizes are reported; the histograms and tables show only distributions of average profit. Moreover, the surrounding text says that 'all of their confidence intervals overlap', which is contradicted by the tables: for Apple the baseline CI is degenerate at -0.2704 while the linear agent's CI is [-0.2213, -0.0845], and for Microsoft the baseline point -0.089 lies outside the Q-learning CI [-0.1960, -0.1243]. Because the same 51 runs are also used to choose which agent to highlight, a multiple-comparison-corrected pairwise test is needed before any directional claim is made.
  4. [§5.2.1, §5.2.2, §6] The paper acknowledges that the agents do not converge to a stable policy and that validation scores vary substantially under small epsilon. Given this, statements that one agent 'learned the best policy' or that approximate Q-learning is 'a step in the right direction' go beyond what the data show. The 51-run averaging estimates the performance of an epsilon-random decision process; it does not establish that the learned component of the policy is responsible for any advantage. The authors should report the performance of the greedy policies after training, or explicitly reframe all conclusions as describing the behavior of the epsilon-greedy training process rather than a converged policy.
minor comments (4)
  1. [§3.2 and §4.1] The values of the key hyperparameters—time-window size w, baseline threshold d, learning rate alpha, discount factor gamma, exploration epsilon, and the neural-network architecture—are never stated in the main text. The appendix points to code, but the paper should report the exact settings used to produce Tables 2–5.
  2. [Appendix A] The repository URL 'https://github.com/TeamAI-2018/final project' contains a space and is not a valid clickable link; it should be URL-encoded or replaced with the actual repository address.
  3. [Figures 3 and 4] Figures 3 and 4 have identical captions, which makes it hard to tell what each figure is intended to show; the captions should distinguish the time-series plot from the zoomed-in view.
  4. [Abstract and §7] The abstract says the agents are compared 'to see if they converge on a policy, and if so, which one learned the best policy,' but Section 6 reports that the policies have not converged; the abstract should be phrased to match the actual, more limited outcome.

Circularity Check

1 steps flagged · score 6.0 of 10

Approximate agents' claimed edge is largely a lookahead artifact: the state contains p0, the same price used to compute the reward and test score.

  1. self definitional [Section 4.1.3 and Section 5.2.1]
    "State= (⃗p−3, ⃗p−2, ⃗p−1, ⃗p0) ... Reward= { −(p0 − pi), if buy; 0, if wait } ... We compute this in the same way we computed the reward for the approximate Q-learning agents, by taking the negative of the difference in close price on the day they decide to buy vs. the first day of the time window."

    For the linear and deep approximate agents, the state includes p0, the vector of prices on the current day, and the buy reward is −(p0 − pi). The test-time score is exactly the same −(p0 − pi) close-price difference. Thus the success metric is an input feature at the moment the action is chosen. A policy that buys when p0 is low relative to pi, or that learns a threshold on p0, can score well without learning any temporal trend. The Section 6 claim that the linear agent 'outperforms the baseline more than 50% of the time' is therefore substantially an artifact of this construction, not evidence that approximate Q-learning 'capitalizes on more information' about future price movements.

full rationale

The paper is an empirical class project, and most of its analysis is not circular: the price-prediction section notices that its regression merely predicts today's price and explicitly compares against a trivial baseline; the exact Q-learning agent uses a different, non-lookahead reward based on next-day movement; and the authors openly state that all confidence intervals overlap and that they 'cannot say for sure whether our agents improve over one another.' The load-bearing positive claim in Section 6, however, is weakened by a self-definitional leakage: the approximate agents' state includes p0 while both the training reward and the test score are −(p0 − pi). This makes the reported >50% win rate partly a constructed consequence of the MDP definition rather than an independent empirical discovery. There is no reliance on self-citation or imported uniqueness theorems; the cited works [5] and [6] are external and not load-bearing. The score is 6 because one central 'prediction' (the linear agent's consistent edge) reduces substantially by construction, but the paper remains an honest, hedged empirical study rather than a forced derivation.

Assumptions & free parameters 8 free parameters · 5 assumptions · 0 invented entities

The central claim depends on several hand-chosen hyperparameters (window size, history length, learning rates, rewards, baseline threshold, neural network architecture) whose values are not reported. It also relies on domain assumptions that daily price movements are Markovian and stationary enough for temporal splits. No new physical or conceptual entities are introduced.

free parameters (8)
  • time_window_size_w = not reported
    Number of days in each time window; chosen by the authors, controls the length of the trading episode and the reward horizon.
  • history_size_h = not reported
    Number of previous days' movements or prices included in the state; kept small to limit state space, exact value not reported for each agent.
  • learning_rate_alpha = not reported
    Learning rate for Q-learning updates; not reported in paper.
  • discount_factor_gamma = not reported
    Discount factor for future rewards; not reported.
  • exploration_epsilon = not reported
    Epsilon-greedy exploration probability; small but not reported.
  • reward_parameters_r_and_c = not reported
    For the exact Q-learning agent: positive reward for correct buy, penalty for waiting; values not reported.
  • baseline_threshold_d = not reported
    Price drop threshold for the baseline agent; constant parameter, value not reported.
  • nn_hidden_layers_and_units = not reported
    Neural network architecture for deep Q-learning; tuned on validation, final values not reported.
assumptions (5)
  • domain assumption Daily stock prices can be modeled as a Markov process with states consisting of recent price movements.
    Invoked in the MDP definition in Section 3.2.
  • domain assumption The reward function (negative price difference from initial day) is a valid proxy for trading profit.
    Used in Section 4.1.3 and 5.2.1; ignores transaction costs and market impact.
  • domain assumption Past price information is sufficient for optimal buy timing without external data.
    This underlies the state design in Section 3.2 and is acknowledged as a limitation in the Discussion.
  • domain assumption The 80/10/10 temporal split ensures stationarity of trends.
    Stated in Section 3.1 to justify comparing agents and splits.
  • standard math Standard Q-learning convergence theory applies to the finite state space.
    Assumed in Section 4.1.2; however, the agents are reported not to converge.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reinforcement Learning for Stock Transactions." pith.science (2026). https://pith.science/paper/36EIT4TW

@misc{pith2026250516099,
  author       = {Pith},
  title        = {Pith review of: Reinforcement Learning for Stock Transactions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/36EIT4TW}},
  note         = {Machine review of arXiv:2505.16099}
}
read the original abstract

Much research has been done to analyze the stock market. After all, if one can determine a pattern in the chaotic frenzy of transactions, then they could make a hefty profit from capitalizing on these insights. As such, the goal of our project was to apply reinforcement learning (RL) to determine the best time to buy a stock within a given time frame. With only a few adjustments, our model can be extended to identify the best time to sell a stock as well. In order to use the format of free, real-world data to train the model, we define our own Markov Decision Process (MDP) problem. These two papers [5] [6] helped us in formulating the state space and the reward system of our MDP problem. We train a series of agents using Q-Learning, Q-Learning with linear function approximation, and deep Q-Learning. In addition, we try to predict the stock prices using machine learning regression and classification models. We then compare our agents to see if they converge on a policy, and if so, which one learned the best policy to maximize profit on the stock market.

Figures

Figures reproduced from arXiv: 2505.16099 by the authors.

Figure 1
Figure 1. A visualization of the stock history data for the four companies we examined. The grey section is [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. A drawing of our Markov Decision Process. The initial state is marked with an I, while terminal [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Accuracy of our predictions on the test set for the regression formulation of the problem. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Accuracy of our predictions on the test set for the regression formulation of the problem. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Histograms of the results for each company. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Example output after running ”run.py” four agents’ performance on one particular company, open the file in the main directory called ’agent evaluation.py’. Locate the comment line that says ’# Parameter Initializations’. Under this line, change the company name to be t…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

8 extracted references · 8 canonical work pages

  1. [1]

    Xgboost: A scalable tree boosting system

    Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. InProceedings of the 22Nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, pages 785–794, New York, NY, USA, 2016. ACM

    work page 2016

  2. [2]

    Deep learning for event-driven stock prediction

    Xiao Ding, Yue Zhang, Ting Liu, and Junwen Duan. Deep learning for event-driven stock prediction. InIjcai, pages 2327–2333, 2015

    work page 2015

  3. [3]

    Garman and Michael J

    Mark B. Garman and Michael J. Klass. On the estimation of security price volatilities from historical data.The Journal of Business, 53(1):67–78, 1980

    work page 1980

  4. [4]

    Lightgbm: A highly efficient gradient boosting decision tree

    Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. InNIPS, 2017

    work page 2017

  5. [5]

    Stock price prediction using reinforcement learning

    Jae Won Lee. Stock price prediction using reinforcement learning. InIndustrial Electronics,

    work page

  6. [6]

    Reinforcement learning for optimized trade execution

    Yuriy Nevmyvaka, Yi Feng, and Michael Kearns. Reinforcement learning for optimized trade execution. InProceedings of the 23rd International Conference on Machine Learning, ICML ’06, pages 673–680, New York, NY, USA, 2006. ACM

    work page 2006

  7. [7]

    Stock price forecasting by hybrid machine learning techniques

    CF Tsai and SP Wang. Stock price forecasting by hybrid machine learning techniques. In Proceedings of the International MultiConference of Engineers and Computer Scientists, volume 1, page 60, 2009. 14

    work page 2009

  8. [2001]

    ISIE 2001

    Proceedings. ISIE 2001. IEEE International Symposium on, volume 1, pages 690–695. IEEE, 2001

    work page 2001

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.