REVIEW 4 major objections 4 minor 2 references
Deep Reinforcement Learning for Foreign Exchange Trading
T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read This paper claims that a PPO reinforcement-learning agent outperforms a DQN agent at optimizing a Martingale-style foreign-exchange trading strategy on three currency pairs.
desk verdict A routine RL-for-trading application whose central PPO-vs-DQN claim rests on a backtest that checks stop triggers only at 4-hour closes, which the authors themselves admit; the feasibility result is therefore not established. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is the pairing of a Gramian Angular Field state encoder with a reward that discounts profit by the number of additional buy-ins. GAF turns a 12-bar sliding window of four-hour opening, high, low, and closing prices into a $12 \times 12 \times 4$ heat-map tensor, so a CNN can treat price history as an image. The reward is $\text{Profit} \times (1.0 - 0.1 \times \text{number of additional buy-ins})$, which tells the agent that every extra Martingale layer costs expected return and should be avoided unless necessary. The action set, covering the cap on additional buy-ins, the initial buy/sell direction, and the stop-gain level, defines the strategy space the agent is optimizing. This design lets the PPO/DQN comparison isolate the effect of the learning algorithm itself.
What would settle it
Use the same trained agents and the same December 2018 window, but trigger stop-loss and stop-gain orders with each four-hour bar's high and low instead of its close; if PPO's net profit falls below the constant baseline or its drawdown exceeds the reported values, the feasibility claim fails.
Extended reading notes
Core claim
The paper's central claim is that a PPO-based agent can reliably optimize the Sure-Fire Martingale trading policy, and that PPO outperforms DQN when both are given the same Gramian Angular Field encoded state. In the December 2018 evaluation, the PPO models earned net profits of 615 on EUR/USD, 763 on GBP/USD, and 597 on AUD/USD, compared with DQN's 486, 717, and 402; on GBP/USD and AUD/USD, PPO also beat the constant baseline, while on EUR/USD it earned less but with a much lower drawdown. The authors attribute the gap not to higher risk but to better timing: PPO made profitable decisions earlier and with lower drawdown than DQN. The intended conclusion is that reinforcement learning is feasible for forex strategy development as long as the state representation captures the environment and the reward is shaped to discourage excessive Martingale buy-ins.
Load-bearing premise
The load-bearing assumption is that checking stop-loss and stop-gain triggers against each four-hour bar's closing price tells the same story as checking them against the bar's high and low; if intra-bar moves would have fired the stops, the reported profits and drawdowns could be materially wrong.
Editorial extensions
If this is right
- A trader can treat the optimization of a Martingale-style strategy as a reinforcement-learning problem and expect the learned policy to be at least as profitable as a fixed baseline while carrying lower drawdown.
- Proximal Policy Optimization should be preferred over Deep Q-Network for discrete-action trading tasks with GAF-encoded price states, because it captured profit more consistently on the currency pairs tested.
- The GAF image representation is sufficient for a CNN-based agent to make profitable trading decisions from four-hour OHLC data.
- Penalizing each additional buy-in in the reward is a workable way to encode risk aversion into an RL trading agent.
Reading between the lines
- The reported results depend on a trigger-detection shortcut: using each bar's closing price to decide whether stops were hit. A natural next experiment is to re-run the backtest with intra-bar high/low trigger detection; the one-month profits would likely shrink, though the PPO-versus-DQN ranking might survive.
- Because the test period is a single month, the claimed superiority of PPO may be regime-specific; extending the same state and reward design across years and market conditions would test whether the finding generalizes.
- The same GAF-plus-discounted-reward recipe could be applied to other assets or to Martingale variants beyond the three-step ladder described here.
- If this feasibility result holds, it suggests that for strategies with known structural rules, RL can tune parameters such as ladder depth and stop levels that are otherwise set by hand.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a deep reinforcement learning system for foreign exchange trading. Price series over 4-hour bars are encoded as Gramian Angular Field (GAF) heat maps and fed to a CNN, which selects among three discrete actions that parameterize a Sure-Fire/Martingale-style strategy: the maximum number of additional buy-ins (1, 2, or 3), the initial trade direction (buy or sell), and the stop-gain distance (20, 25, or 30 pips). The reward is net profit multiplied by a discount factor that penalizes additional buy-ins. The system is trained on EUR/USD, GBP/USD, and AUD/USD from 1 August 2018 to 30 November 2018 and evaluated on December 2018. The paper compares DQN, PPO, and a constant baseline on net profit, profit factor, and max drawdown, concluding that PPO-based trading is feasible and outperforms DQN.
Significance. If the results were reliable, the paper would demonstrate a practical way to use deep RL with GAF-encoded states to tune a Martingale variant, with PPO outperforming DQN. The paper has some strengths: it includes a constant baseline, it is transparent about the execution approximation in the backtest, and it performs a simple out-of-sample split. However, the central claims rest on a single month of test data and on a closing-price-only trigger simulation that is not appropriate for stop-loss/stop-gain orders, so the significance of the findings is currently not established.
major comments (4)
- [III.2 (Trade Environment Settings)] The backtest triggers stop-loss and stop-gain by checking only the 4-hour closing price, explicitly ignoring intra-bar high and low points. With stop distances of only 20–30 pips, intra-bar price extremes routinely cross these thresholds even when the bar closes inside them. The assertion that 'time scales smaller than one day rarely contained stop-loss and stop-gain points' is unsupported and likely false for 4-hour bars. This approximation can both miss real stop-outs and falsely keep positions open, which changes the number of additional buy-ins, the realized profit, the drawdown, and the reward signal. Since the paper's central claim is that PPO reliably optimizes the strategy, Table II and the feasibility conclusion are not established by the reported experiments.
- [IV, Table II] The evaluation period is a single month (1–31 December 2018) with no error bars, no multiple random seeds, and no statistical significance tests. Observed differences such as CGU (753) versus PGU (763) are well within the noise expected for a single month of trading data, so the conclusion that PPO is better than DQN is not supported. Moreover, the paper claims 'quite feasible' performance while its own data show that on EUR/USD the constant baseline (CEU, net profit 753) outperforms both DQN (DEU, 486) and PPO (PEU, 615). A feasibility claim should address this direct counterexample rather than only emphasizing the profit factor and drawdown.
- [III.3 (Reward Design) and III.2] The reward is defined as Profit × Discount, where Discount = 1.0 − 0.1 × (number of additional buy-ins). The number of additional buy-ins is determined by the same closing-price-only trigger simulator that is used in the backtest. Therefore, the reward signal that trains DQN and PPO is biased by the flawed execution model: the agent is optimized to maximize profit under a simulator that both misses real stop-outs and postpones buy-in triggers. The reported PPO-over-DQN advantage could be an artifact of this simulator rather than a genuine property of the reinforcement learning algorithms.
- [III (Experiment Model Design)] The experimental setup is not reproducible: the paper does not specify the CNN architecture, the number of layers, learning rates, discount factors, exploration schedules, minibatch sizes, or any hyperparameters for DQN or PPO. The only architecture description is 'a CNN is used as the policy network or Q-network.' Without these details, another researcher cannot re-run the comparison, and the claimed performance difference between PPO and DQN cannot be independently verified.
minor comments (4)
- [Table I] In the rightmost column, DAU and PAU are labeled as 'EUR/USD'; based on the surrounding text and the CAU entry, these should be 'AUD/USD'.
- [II.A and II.C] The citations appear as '[ ?]' in the DQN and GAF subsections; these placeholder references must be completed before publication.
- [III (opening line)] There is a typo 'Oue experiment environment designs' that should read 'Our experiment environment designs.'
- [Abstract and Conclusion] The phrase 'favorable investment performance is achieved as long as the model is able to handle complex and random processes and the state is able to describe the environment' is too vague to be falsifiable; it should be replaced by a concrete operational criterion tied to the reported metrics.
Circularity Check
No significant circularity: PPO-over-DQN claim rests on a held-out backtest, not on an equation or self-citation that reproduces its own conclusion.
full rationale
The paper claims that PPO performs better than DQN for the optimized Sure-Fire strategy, based on a train/test split: 1 August to 30 November 2018 for training and 1 December to 31 December 2018 for evaluation (Section III.1). The reward is defined as Profit times a discount factor depending on the number of additional buy-ins, while the reported Table II metrics are net profit, profit factor, and max drawdown on the held-out month. These are distinct quantities, and the policy is not fitted to the test-period performance metric by construction. There is no equation in which the target conclusion is inserted as an input, no fitted parameter that is renamed as a prediction, and no load-bearing self-citation; the only named citations are standard RL references (Sutton and Barto, and Sutton et al.). The one self-identified limitation, Section III.2's statement that order triggering is checked on 4-hour closing prices rather than intra-bar high/low values ('the closing price is used as the data point rather than considering whether the transactions are triggered at a high point or low point in the data'), is a potential execution-modeling flaw that could affect which policy appears best, but it is not circular: the simulator's approximation is an input to the experiment, not the output being derived. The central feasibility conclusion is not derived from itself, so the circularity score is 0.
Assumptions & free parameters
free parameters (6)
- Reward discount coefficient =
0.1
- Stop-gain levels =
20, 25, 30 pips
- Maximum additional buy-ins =
1, 2, 3
- Sliding window length =
12 bars
- Number of training episodes =
1300
- Minimum earnings threshold =
20 pips
assumptions (4)
- domain assumption Sure-Fire strategy (Martingale variant) will eventually profit if price fluctuates within increasing margins
- domain assumption Transaction costs and slippage are negligible relative to the 20 pip target
- domain assumption Closing-price testing adequately approximates intra-bar order triggers
- standard math The environment is a Markov decision process for the agent
Cite this review
Pith. "Pith review of Deep Reinforcement Learning for Foreign Exchange Trading." pith.science (2026). https://pith.science/paper/CJ7XL2PE
@misc{pith2026190808036,
author = {Pith},
title = {Pith review of: Deep Reinforcement Learning for Foreign Exchange Trading},
year = {2026},
howpublished = {\url{https://pith.science/paper/CJ7XL2PE}},
note = {Machine review of arXiv:1908.08036}
}
read the original abstract
Reinforcement learning can interact with the environment and is suitable for applications in decision control systems. Therefore, we used the reinforcement learning method to establish a foreign exchange transaction, avoiding the long-standing problem of unstable trends in deep learning predictions. In the system design, we optimized the Sure-Fire statistical arbitrage policy, set three different actions, encoded the continuous price over a period of time into a heat-map view of the Gramian Angular Field (GAF) and compared the Deep Q Learning (DQN) and Proximal Policy Optimization (PPO) algorithms. To test feasibility, we analyzed three currency pairs, namely EUR/USD, GBP/USD, and AUD/USD. We trained the data in units of four hours from 1 August 2018 to 30 November 2018 and tested model performance using data between 1 December 2018 and 31 December 2018. The test results of the various models indicated that favorable investment performance was achieved as long as the model was able to handle complex and random processes and the state was able to describe the environment, validating the feasibility of reinforcement learning in the development of trading strategies.
Figures
Reference graph
Works this paper leans on
-
[1]
Reinforcement learning: An introduction
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018
2018
-
[2]
Policy gradient methods for reinforcement learning with function approximation
Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in neural information processing systems, pages 1057–1063, 2000
2000
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.