REVIEW 4 major objections 5 minor 13 references
Improving DeFi Accessibility through Efficient Liquidity Provisioning with Deep Reinforcement Learning
T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read A deep reinforcement learning agent that actively rebalances a Uniswap v3 liquidity position earns higher cumulative reward than a passive periodic strategy in 7 of 11 out-of-sample windows.
desk verdict A competent DRL-for-DeFi application whose headline 7-of-11 claim is undermined by best-of-50 selection on the test reward, plus a suspicious appendix table. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Markov decision process the paper builds for a single LP position. Its reward is Eq. (17), $R_{t+1}=f_t-\ell_t(\sigma,p)-\mathbb{I}[a_t\neq0]g$, where $f_t$ is the trading fee accumulated from price moves inside the active range, $\ell_t=L\sigma_t^2\sqrt{p_t}/4$ is the instantaneous Loss-Versus-Rebalancing penalty with exponentially weighted volatility, and $g$ is the fixed gas charge for withdrawing and redeploying. The PPO agent observes a state of current price, tick index, prior interval width, liquidity level, volatility, moving averages, and technical indicators, and outputs a discrete choice of symmetric interval width in ticks. This MDP formulation lets the agent learn a rebalancing policy that trades fee capture against redeployment costs instead of rebalancing on a fixed clock.
What would settle it
Recompute the 11 out-of-sample comparisons using observed Ethereum gas fees at every withdrawal and redeployment and mark the position to market at each hourly step; if the active LP does not earn higher cumulative realized value in at least 7 of the 11 windows, the central claim fails.
Extended reading notes
Core claim
The central claim, stated in Section 6, is that the active LP trained with Proximal Policy Optimization earns higher cumulative reward than the passive LP in 7 of the 11 out-of-sample testing windows on the WETH/USDC pool at the 0.05% fee tier over May 2021 to January 2024. The reward being maximized is $R_{t+1}=f_t-\ell_t(\sigma,p)-\mathbb{I}[a_t\neq 0]g$: collected fees minus an instantaneous LVR penalty minus a fixed gas cost applied whenever the agent changes its range. The agent's edge, as the paper describes it, is adaptive timing and interval width: it narrows or widens its liquidity band around the current price, and sometimes deliberately leaves the position inactive during expected mean reversion, while the passive baseline rebalances on a fixed 500-hour schedule regardless of the price path.
Load-bearing premise
The load-bearing premise is that the simulated reward, with a flat $5 gas fee and an LVR estimate standing in for opportunity cost, faithfully represents what a liquidity provider actually earns, so the active policy's advantage is measured inside the model rather than in realized dollars.
Editorial extensions
If this is right
- The active policy can learn when not to rebalance: in the reported windows it sometimes keeps a position that has fallen out of range, betting on mean reversion and saving gas and LVR costs.
- The best action space differs across windows, so a single fixed tick width would not reproduce the strategy's edge; interval width itself is a learned choice.
- Raising initial liquidity from $x_0=2$ to $x_0=10$ leaves the relative result unchanged, with the active LP still winning 7 of 11 windows.
- Because the reward function charges gas only when the agent changes its range, the agent internalizes the fee-versus-rebalancing trade-off as part of its long-run objective.
Reading between the lines
- An inference beyond the paper: the fixed $5 gas charge is favorable to an active strategy over a period when Ethereum gas prices varied by more than an order of magnitude, so a time-varying gas series is the natural robustness test.
- An inference beyond the paper: the comparison is stated in terms of the model reward, not realized mark-to-market profit; a backtest that tracks actual token balances and real gas payments would show whether the edge is dollars, not just reward units.
- An inference beyond the paper: the same MDP design is portable to other Uniswap v3 pools, fee tiers, and lower-cost networks, where the learned threshold for rebalancing should shift as gas prices fall.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper formulates liquidity provisioning in Uniswap v3 as a Markov decision process and trains a PPO-based active LP agent that periodically decides whether to maintain or redeploy its liquidity range. The reward in Eq. (17) is trading fees minus an LVR opportunity cost minus a fixed $5 gas fee. Using rolling windows of 7,500 training hours and 1,500 out-of-sample hours on WETH/USDC 0.05% pool data from May 2021 to January 2024, the authors report that the active LP outperforms a passive 500-hour/50-tick rebalancing heuristic in 7 of 11 out-of-sample windows. The appendix provides hyperparameter tables and additional results for x0=10.
Significance. If the empirical claim holds, the paper would provide a practical demonstration that a DRL-based rebalancing policy can beat a simple periodic heuristic for retail LPs in Uniswap v3, without off-chain hedging. The work is transparent about its reward decomposition and hyperparameter choices, and it targets a population (small retail LPs) that is often underserved in the DeFi literature. However, the central contribution is empirical, and the current evidence is undermined by an ambiguous model-selection protocol, a lack of statistical uncertainty quantification, and an internal inconsistency in the appendix. These issues must be resolved before the claim can be considered reliable.
major comments (4)
- [Section 6, 'Empirical Studies'] The selection protocol is ambiguous and, on the natural reading, invalidates the out-of-sample claim. The text states that 'for each out-of-sample window, we train 50 different active LP agents, selecting the one that performs best according to the cumulative sum of the reward function described in Eq. (17).' If this cumulative reward is evaluated over the same 1,500-hour test window used for the reported active-vs-passive comparison, then the 7/11 result is not an out-of-sample evaluation but a best-of-50 selection on the test objective. With 50 candidates, selection noise alone can produce a 7/11 win count even if the average PPO policy is no better than the passive baseline. The manuscript must state explicitly that selection is performed on the training window only, and should report the distribution of the 50 candidates (mean, median, standard deviation) or, at minimum, results for multiple seeds.
- [Appendix B, Fig. 5] The table in Figure 5 reports the x0=10 results with exactly the same numerical entries as the x0=2 table in Figure 2, and it adds a window ending 2024-04-01 even though the dataset ends on 2024-01-29 and the rolling-window design produces only 11 test windows. Since fees and the LVR penalty in Eq. (17) scale with the liquidity parameter L, which changes when x0 is increased from 2 to 10, identical rewards are not possible unless the results are normalized (which is not stated). This internal inconsistency makes the supporting evidence in Appendix B unreliable and must be corrected or removed before the paper can be considered further.
- [Section 5, Eq. (17)] The reward model fixes the gas fee at $5 for every withdrawal and redeployment and uses the LVR penalty ℓ_t = L σ_t^2 sqrt(p_t)/4 with an exponentially weighted volatility. Ethereum gas prices varied by more than an order of magnitude over 2021-2024, and LVR is a model-based estimate rather than observed PnL. Because the active agent rebalances more frequently than the passive 500-hour baseline, these two modeling choices directly determine whether 'higher cumulative reward' survives under realistic costs. The authors should provide a sensitivity analysis over gas fee levels and LVR calibration, or at least justify the constant $5 with a documented source.
- [Section 6, Fig. 2] The reported comparison contains only point estimates for cumulative reward. No confidence intervals, standard errors, or repeated-seed statistics are given for the active agent, whose policy is stochastic and whose reward trajectory depends on initialization. The claim of superiority in 7 of 11 windows is therefore not accompanied by any measure of uncertainty; the win count could be within sampling noise. The authors should report seed statistics (e.g., mean ± std over at least 10 seeds per window) or a permutation test against the passive baseline.
minor comments (5)
- [Abstract and Section 6] The phrase "consistently outperforms" is stronger than "outperforms in 7 of 11 windows"; please rephrase to match the reported result.
- [Equation (14)] The notation for the PPO objective is inconsistent: the entropy coefficient c2 appears only in Table 2 and is not defined in the objective, and the critic loss term L_AF is not defined in the text.
- [Appendix A] Tables 1 and 2 are not referenced in the main text; please add a cross-reference near the hyperparameter discussion in Section 6.
- [Figure 3] The layout of the four panels is described only in the caption; the figure itself should label the active and passive rows/columns clearly.
- [Section 4, 'Deep Reinforcement Learning with PPO'] The statement that the agent "lacks prior knowledge of the exact functional form of the reward function" is confusing, because the reward is specified by Eq. (17) and used for training; please rephrase to describe the model-free RL setup accurately.
Circularity Check
Best-of-50 selection on the test reward makes the 7/11 'out-of-sample' win count a selected maximum, not a prediction.
-
fitted input called prediction
[Section 6, Empirical Studies (agent selection and headline result; Fig. 2 and Fig. 4)]
"For each out-of-sample window, we train 50 different active LP agents, selecting the one that performs best according to the cumulative sum of the reward function described in Eq. (17). ... Our results demonstrate that the active LP consistently outperforms the passive LP strategy in 7 out of the 11 out-of-sample windows."
The active agent is chosen per window as the argmax of the cumulative Eq. (17) reward, and the reported comparison is the same cumulative Eq. (17) reward over the same out-of-sample window. Therefore the headline 7/11 count is the performance of the best of 50 draws on the test objective, not the out-of-sample performance of a single pre-specified policy. With 50 candidates, selection noise alone can produce such a count even when the average PPO policy does not beat the passive baseline, and the paper reports no per-seed distribution or confidence interval to separate selection from skill. The 'out-of-sample prediction' is thus statistically forced by the selection protocol: fitted input (best-of-50 agent choice) is renamed as an out-of-sample result.
full rationale
The paper's construction of the MDP and reward is self-contained: Eq. (17) combines fee formulas from the cited external literature, an LVR penalty, and a fixed gas fee, and the passive baseline is an independent periodic heuristic. There is no definitional equivalence or self-citation chain in the reward derivation itself. The load-bearing circularity is in the evaluation protocol: the active agent is selected using the same cumulative Eq. (17) reward on which the 7/11 out-of-sample win count is reported, which makes the central empirical claim a best-of-50 in-sample statistic rather than an unbiased out-of-sample prediction; this matches the fitted-input-called-prediction pattern. The Appendix B table repeats the x0=2 numbers verbatim and adds a 2024-04-01 row despite the dataset ending 2024-01-29; that is a data-integrity/correctness concern, not a circularity step, but it reinforces caution about the reported results. No other circular steps were found.
Assumptions & free parameters
free parameters (6)
- Volatility smoothing factor alpha =
0.05
- Gas fee g =
$5
- Passive LP interval half-width =
50 ticks
- Passive LP rebalancing period =
500 hours
- Initial risky token amount x0 =
2 (and 10 in Appendix B)
- Per-window PPO hyperparameters =
Learning rate, clip, entropy coefficient, discount factor, action space, activation, and hidden sizes per Appendix A…
assumptions (5)
- standard math Policy gradient theorem and PPO objective (Eqs. 10-14) are valid optimization tools.
- domain assumption Equations (2)-(6) correctly describe Uniswap v3 positions and fee accrual.
- domain assumption The LVR formula in Eq. (16), with exponentially weighted volatility, is a valid opportunity cost for an unhedged LP.
- domain assumption Hourly resampling of subgraph data preserves the fee and price dynamics relevant to LP decisions.
- domain assumption The agent is a price taker and its rebalancing does not affect pool price or other LPs.
Cite this review
Pith. "Pith review of Improving DeFi Accessibility through Efficient Liquidity Provisioning with Deep Reinforcement Learning." pith.science (2026). https://pith.science/paper/5YGQY4NG
@misc{pith2026250107508,
author = {Pith},
title = {Pith review of: Improving DeFi Accessibility through Efficient Liquidity Provisioning with Deep Reinforcement Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/5YGQY4NG}},
note = {Machine review of arXiv:2501.07508}
}
read the original abstract
This paper applies deep reinforcement learning (DRL) to optimize liquidity provisioning in Uniswap v3, a decentralized finance (DeFi) protocol implementing an automated market maker (AMM) model with concentrated liquidity. We model the liquidity provision task as a Markov Decision Process (MDP) and train an active liquidity provider (LP) agent using the Proximal Policy Optimization (PPO) algorithm. The agent dynamically adjusts liquidity positions by using information about price dynamics to balance fee maximization and impermanent loss mitigation. We use a rolling window approach for training and testing, reflecting realistic market conditions and regime shifts. This study compares the data-driven performance of the DRL-based strategy against common heuristics adopted by small retail LP actors that do not systematically modify their liquidity positions. By promoting more efficient liquidity management, this work aims to make DeFi markets more accessible and inclusive for a broader range of participants. Through a data-driven approach to liquidity management, this work seeks to contribute to the ongoing development of more efficient and user-friendly DeFi markets.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[2]
End of Test A Activation Hidden Layers 2022-05-14 [0, 20, 50] sigmoid [6, 4] 2022-07-16 [0, 10, 20] sigmoid [4, 2] 2022-09-16 [0, 40, 50, 60] sigmoid [4, 2] 2022-11-18 [0, 50, 100] relu [6, 6, 6] 2023-01-19 [0, 10, 20, 30] relu [6, 4] 2023-03-23 [0, 40, 50, 60] relu [4, 2] 2023-05-24 [0, 10, 20, 30] tanh [8, 2] 2023-07-26 [0, 40, 50, 60] sigmoid [8, 2] 20...
work page 2022
-
[3]
arXiv preprint arXiv:2106.12033
Strategic liquidity provi- sion in uniswap v3. arXiv preprint arXiv:2106.12033. Fan, Z.; Marmolejo-Coss ´ıo, F. J.; Altschuler, B.; Sun, H.; Wang, X.; and Parkes, D
-
[4]
In Proceedings of the 2021 ACM CCS Workshop on Decentralized Finance and Security , 15–20
Concentrated liquidity in automated mar- ket makers. In Proceedings of the 2021 ACM CCS Workshop on Decentralized Finance and Security , 15–20. Heimbach, L.; Schertenleib, E.; and Wattenhofer, R
work page 2021
-
[6]
arXiv preprint arXiv:2111.09192
Impermanent loss in uniswap v3. arXiv preprint arXiv:2111.09192. Marbach, P.; and Tsitsiklis, J
-
[7]
ArXiv preprint arXiv:2208.06046
Automated Market Making and Loss-Versus- Rebalancing. ArXiv preprint arXiv:2208.06046. Ottina, M.; Steffensen, P. J.; and Kristensen, J
-
[9]
In Companion Proceedings of the Web Conference 2022 , 12–19
Cyclic arbitrage in decentralized exchanges. In Companion Proceedings of the Web Conference 2022 , 12–19. Werner, S.; Perez, D.; Gudgeon, L.; Klages-Mundt, A.; Harz, D.; and Knottenbelt, W
work page 2022
-
[10]
arXiv preprint arXiv:2309.10129
Adaptive Liq- uidity Provision in Uniswap V3 with Deep Reinforcement Learning. arXiv preprint arXiv:2309.10129. Zheng, R.; Dou, S.; Gao, S.; Hua, Y .; Shen, W.; Wang, B.; Liu, Y .; Jin, S.; Zhou, Y .; Xiong, L.; et al
-
[11]
In NeurIPS 2023 Workshop on Instruction Tuning and Instruction F ol- lowing
Delve into PPO: Implementation Matters for Stable RLHF. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction F ol- lowing. A Hyperparameter Optimization and Training Setup8 During the training of the PPO algorithm, we use an early stopping procedure to prevent overfitting by halting train- ing when performance improvements stagnated beyond a set...
work page 2023
Show all 13 references
-
[13]
The results align with those for x0 = 2, as the active LP strategy outperforms the passive LP in 7 out of 11 testing windows
This section provides results similar to those discussed earlier, illustrat- ing the performance of both active and passive LP strategies with higher initial liquidity. The results align with those for x0 = 2, as the active LP strategy outperforms the passive LP in 7 out of 11...
2022
-
[2017]
arXiv preprint arXiv:1707.06347
Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Sutton, R.; Mcallester, D.; Singh, S.; and Mansour, Y
-
[2021]
arXiv preprint arXiv:2106.14404
Uniswap: Imper- manent loss and risk profile of a liquidity provider. arXiv preprint arXiv:2106.14404. Ali, K.; Shahzad, A.; and Chaudhary, H. K
-
[2022]
arXiv preprint arXiv:2205.08904
Risks and returns of uniswap v3 liquidity providers. arXiv preprint arXiv:2205.08904. Juliano, A
-
[2023]
arXiv preprint arXiv:2309.13648
The costs of swapping on the uniswap protocol. arXiv preprint arXiv:2309.13648. Adams, H.; Zinsmeister, N.; Salem, M.; Keefer, R.; and Robinson, D
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.