Pith. sign in

REVIEW 2 major objections 5 minor 25 references

In a closed-loop DEX simulator with dynamic fees, a small DQN cuts implementation shortfall by 13.3 bps over tuned one-step routing, and the edge is learned only when fees reprice.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 08:02 UTC pith:U7I653AE

load-bearing objection Solid within-model RL-for-execution result with unusually clean evaluation hygiene; the novelty is the closed-loop dynamic-fee simulator and protocol, not a general market claim. the 2 major comments →

arxiv 2607.10960 v1 pith:U7I653AE submitted 2026-07-12 cs.LG q-fin.CPstat.ML

Reinforcement Learning for Execution under Dynamic Fees in a Closed-Loop DEX Simulator

Wen-Ting Wang This is my paper
classification cs.LG q-fin.CPstat.ML
keywords automated market makersdecentralized exchangesdynamic feesoptimal executionreinforcement learningmarket simulationimplementation shortfallclosed-loop simulator
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Historical on-chain tapes cannot answer how traders would respond to fees that reprice as they trade: trader-facing fees do not vary, trader types are latent, and a replayed tape is not a sequential decision environment. This paper builds a minimal closed-loop market in which that missing signal exists by construction: two constant-product pools whose fees are set by an equilibrium-inspired linear rule, fee-sensitive noise flow, and closed-form CEX–AMM arbitrage. Against a ladder of schedule, planning, lookahead, and tabular policies, a small DQN is the only valid evaluated policy whose paired improvement over tuned one-step routing excludes zero. On a reserved final block of 1,000 seeds with completion forced for every policy, it lowers shortfall under every tested intra-step ordering, by 13.3 bps of order notional under the pre-specified agent-last ordering. The edge is concentrated in, and learned from, dynamic-fee environments; under constant fees the paired difference is indistinguishable from zero. The result is model-conditioned counterfactual evidence about execution control, not a claim about historical traders, equilibrium play, or deployable profit.

Core claim

Under an equilibrium-inspired dynamic-fee closure, a small model-free DQN is the only evaluated valid policy whose paired improvement over validation-tuned one-step routing excludes zero. On a reserved final block of 1,000 seeds with completion forced to 1.0 for every policy, it reduces implementation shortfall under every tested intra-step ordering, by 13.3 bps of order notional under the pre-specified agent-last ordering, and the advantage is concentrated in and learned from dynamic-fee environments rather than from generic closed-loop timing.

What carries the argument

The closed-loop transition that composes the agent’s action with market-response operators (noise routing, arbitrage, oracle update, and the clipped linear dynamic-fee rule) in a declared intra-step order, together with a frozen reserved-seed evaluation protocol against a tuned policy ladder under forced completion.

Load-bearing premise

The market’s defensive response is adequately represented by the researcher-chosen clipped linear fee rule and the accompanying noise-routing and arbitrage operators; if that specific closure does not create the same fee-state timing structure as real dynamic-fee AMMs, the measured edge is an artifact of this model.

What would settle it

Retrain and re-evaluate the same DQN and lookahead ladder under a different fee closure (for example a nonlinear or solved Nash fee process, or constant fees already tested) on a held-out seed block; if the paired DQN–lookahead edge disappears or reverses under every dynamic-fee variant while remaining zero under constant fees, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper constructs a minimal closed-loop DEX simulator with two constant-product pools, an equilibrium-inspired clipped linear dynamic-fee rule (Eq. 11), fee-sensitive noise routing, and closed-form CEX–AMM arbitrage, then evaluates execution policies under a strict protocol (common forced-terminal completion, tuned ladder, reserved final seed block). Against schedule, planning, lookahead, and tabular baselines, a small DQN is the only evaluated valid policy whose paired shortfall improvement over tuned one-step routing excludes zero; on the reserved 1,000-seed block with completion forced to 1.0 it reduces implementation shortfall under every tested intra-step ordering (13.3 bps under the pre-specified agent-last ordering). Fee-mode attribution shows the edge is concentrated in, and learned from, dynamic-fee environments; under constant fees the paired difference is indistinguishable from zero. The claim is explicitly scoped as model-conditioned counterfactual evidence about execution control, not historical identification, equilibrium play, or deployable profit.

Significance. If the result holds under the stated scope, the paper makes two useful contributions. First, it supplies a reusable evaluation discipline for closed-loop execution (action-responsive transitions with declared operator order, common completion semantics, a multi-rung tuned ladder, a frozen reserved seed block, and the transfer-versus-re-solving distinction) that is more careful than typical RL-for-execution claims. Second, it produces a clean within-model finding that model-free control can extract fee-state timing value that shallow planners and tuned one-step heuristics do not recover, with the advantage vanishing under constant fees. Strengths that should be credited include the reserved final block first evaluated after design freeze, paired seed-level bootstrap CIs, forced-terminal completion so shortfall is not traded against residual inventory, the full ladder including stochastic rollout and a hindsight reference, fee-mode and priority re-solving, and a public bit-reproducible implementation with content-hashed manifests. The external-validity premise (the specific fee closure) is already scoped and partially stress-tested; it does not overturn the internal claim.

major comments (2)
  1. §3.1–3.2 and Table 1: the fee coefficients (a_own, a_riv, a_orc) and clips are researcher-chosen design parameters, not estimated or solved from the cited Nash system. The paper already treats this as closure rather than equilibrium and reports coefficient perturbations that do not flip rankings, but a short additional sensitivity that replaces the linear rule with a qualitatively different (e.g., threshold or nonlinear) fee response would better separate “dynamic fees create timing value” from “this particular linearization creates timing value.” This is a scope-strengthening request, not a claim of internal inconsistency.
  2. §3.4 and Appendix A: the multi-step planners (deterministic K=2,3 and stochastic rollout with N=16) fail to close the gap to the DQN, yet deeper search, learned-value-guided search, and expectimax (rather than certainty-equivalent collapse) are not evaluated. The manuscript already notes this limitation; a brief discussion or one deeper-search ablation would make the “planning does not substitute for learning here” claim more robust, since limited depth/budget remains a competing explanation for the planner shortfall.
minor comments (5)
  1. Figure 2 and the abstract report 13.3 bps for agent-last; the body text sometimes uses −13.29. Standardize the rounding and sign convention (negative favors DQN) across abstract, figures, and text.
  2. §2.2 Eq. (3) and §3.3: the training–evaluation completion-rule switch is disclosed and audited, but a one-sentence reminder in the Figure 2 caption that every policy (not only the DQN) is evaluated under forced-terminal would reduce reader confusion.
  3. §3.5: the fee-cash vs. quoted-fee comparison (33.5 vs. 45 bps) is carefully caveated as “not a direct estimate of fee savings”; consider moving that caveat into the figure or table that first reports those numbers so it is not missed.
  4. Appendix C: the noise-routing logistic and arbitrage closed forms are clear; a single line stating that the deterministic planner’s forward model uses the exact mean noise volume (λ sinh(η)/η) would help readers who implement the ladder.
  5. References: several arXiv preprints are cited for concurrent fee-design work; ensure the final version updates any that have since appeared in venues, and keep the companion event-study citation consistent.

Circularity Check

0 steps flagged

No significant circularity: the headline is a held-out paired shortfall comparison inside a declared simulator, not a first-principles prediction forced by its inputs.

full rationale

The paper’s load-bearing claim is empirical and model-conditioned: under a researcher-specified closed-loop DEX (Eq. 11 fee closure, noise routing, closed-form arb), a small DQN is the only evaluated valid policy whose paired IS improvement over validation-tuned one-step lookahead excludes zero on a reserved final block of 1,000 seeds with forced completion 1.0 for every policy, with the edge concentrated in dynamic-fee modes. That estimand is not derived from, or equivalent to, the fee coefficients or the equilibrium literature by construction; it is a seed-averaged policy comparison after freeze (Algorithm 1). Equilibrium enters only as an explicit closure principle for the environment’s fee rule (Abstract; §3.1), not as a solved object the trader learns or a Nash claim the paper re-labels as prediction. The fee form is literature-inspired from Baggiani et al. (different authors); coefficients are disclosed researcher-chosen design parameters with robustness axes, not fits that force the DQN edge. Companion self-citation [25] motivates why historical tape cannot identify the counterfactual and is not used to certify the simulator result. Validation-tuned κ and checkpoint selection on disjoint seed blocks, then evaluation on untouched S_final, is standard ML hygiene rather than fitted-input-called-prediction. No uniqueness theorem, ansatz smuggled as theorem, or renaming of a known empirical law is load-bearing. The derivation chain is therefore self-contained simulation evidence with explicit scope limits (§4), not circular reduction of outputs to inputs.

Axiom & Free-Parameter Ledger

7 free parameters · 7 axioms · 2 invented entities

The central claim is a within-simulator paired performance result. It rests on standard MDP/RL and CPMM mathematics, domain modeling choices for noise/arbitrage/oracle, and a large set of researcher-chosen free parameters (fee coefficients, clips, noise scales, gas, κ, architecture) that define the environment in which the edge appears. The main invented construct is the minimal closed-loop dynamic-fee market used as the evaluation world; equilibrium theory is used only as fee-rule inspiration, not as a solved object. No new physical entity is postulated. Counts of free parameters dominate the ledger because the quantitative bps edge is model-conditioned on those choices.

free parameters (7)
  • fee response coefficients (a_own, a_riv, a_orc) = (0.01, -0.005, 0.5)
    Researcher-chosen values (0.01, −0.005, 0.5) that set how strongly inventory and oracle gaps reprice fees; load-bearing for the dynamic-fee edge the DQN learns.
  • fee clip bounds [f_min, f_max] = [1, 500] bps
    Design bounds [1, 500] bps that truncate the linear fee rule; shape the feasible fee-state variation.
  • lookahead/planner urgency coefficients κ = mode-specific 8 or 16
    Validation-selected per market mode (κ=16 dynamic/constant duopoly; κ=8 monopoly; planner κ similarly tuned); define the strength of the main baseline the DQN must beat.
  • noise median scales, dispersion η, routing sensitivity β = λ±=2.0 Y; η=0.5; β=0.05/bps
    Researcher-chosen flow primitives (λ±=2.0 Y, η=0.5, β=0.05 per bps) that determine fee-sensitive rerouting and the residual timing structure planners may miss.
  • terminal penalty rate ρ and agent gas g = ρ=0.02; g=2 X
    ρ=0.02 and g=2 X per pool set training completion pressure and per-touch costs; audited but still free design choices affecting learned behavior.
  • order size, horizon, pool reserves, base fee, oracle vol = Q=50; H=50; f̄=30 bps; σ=0.5
    Task and scale anchors (Q=50 Y, H=50, reserves 10^6 X / 10^3 Y, f̄=30 bps, σ=0.5) chosen to match common on-chain magnitudes without formal estimation; fix the economic regime of the result.
  • DQN architecture and training hyperparameters = 16-64-64-8; E=12000; Adam 1e-3
    16–64–64–8 MLP, replay 2e5, Adam 1e-3, ε anneal, E=12,000, etc., researcher-chosen and frozen; checkpoint selected on 50 validation seeds.
axioms (7)
  • standard math Finite-horizon MDP with declared operator composition for market response (agent, noise, arb, oracle, fee) and no future-information leakage in observations.
    Section 2.1–2.3; standard sequential decision formalism plus explicit anti-leakage schema.
  • domain assumption Constant-product pool invariant with fees accruing outside reserves; venue cash cost as in Eq. (13).
    Section 3.1; standard Uniswap-lineage CPMM mechanics matching fee treatment in cited theory.
  • domain assumption External mid follows zero-drift GBM; arbitrage closes to fee-adjusted no-arbitrage ratio in closed form when profit exceeds gas.
    Section 3.1 and Appendix C; stylized CEX–AMM structure used as market closure.
  • ad hoc to paper Noise traders route by softmax/logistic over effective execution cost of a 1Y probe (researcher-chosen reduced form).
    Section 3.1 and Appendix C; motivated by cost-sensitive flow literature but functional form is author-chosen.
  • ad hoc to paper Clipped linear dynamic fee rule is a valid equilibrium-inspired closure for competing CPMMs (form from Baggiani et al., not a solved Nash system).
    Eq. (11), Section 3.1; paper explicitly does not solve the coupled PDE and uses the rule only for environment defense.
  • domain assumption Forced-terminal completion makes episode score equal realized implementation shortfall and is applied identically to every policy.
    Section 2.2; evaluation semantics that define the headline metric IS(π; ξ).
  • domain assumption Baseline pool depths are fixed over the horizon in the main environment.
    Section 3.2, assumption inherited from [4]; relaxed only as Appendix B sensitivity.
invented entities (2)
  • Minimal closed-loop dynamic-fee DEX simulator (two CPMM pools + fee rule + noise + arb + execution agent) no independent evidence
    purpose: Create by construction the action-dependent sequential signal that historical tapes lack, so execution policies can be compared under dynamic fees.
    The market is a research construct, not an estimated digital twin; all headline numbers are counterfactual outputs of this model.
  • Closed-loop execution evaluation discipline (tuned ladder, reserved seed block, transfer vs re-solving, common completion) no independent evidence
    purpose: Turn an easy-to-overstate RL-beats-baseline claim into a claim with explicit conditions and artifact checks.
    Method contribution of Section 2; reusable as protocol but not independently validated outside this paper's instantiation.

pith-pipeline@v1.1.0-grok45 · 22010 in / 4890 out tokens · 47829 ms · 2026-07-14T08:02:49.158305+00:00 · methodology

0 comments
read the original abstract

Trader-facing dynamic fees are increasingly proposed for automated market makers (AMMs), but historical data do not identify how order flow would respond: trader-facing fees do not vary, trader types are latent, and a replayed tape is not a sequential decision environment. We therefore construct a minimal closed-loop simulator in which the missing signal exists by construction: two constant-product pools repriced by an equilibrium-inspired dynamic-fee rule, fee-sensitive noise flow, and closed-form CEX--AMM arbitrage. Equilibrium is used as a closure principle, not as an object the trader learns. Against a tuned benchmark ladder of schedule, planning, lookahead, and tabular policies, a small DQN is the only evaluated valid policy whose paired improvement over tuned one-step routing excludes zero. On a reserved final block of 1{,}000 seeds with completion forced to 1.0 for every policy, it reduces implementation shortfall under every tested intra-step ordering, by $13.3\bps$ of order notional under the pre-specified agent-last ordering, and the edge is concentrated in, and learned from, dynamic-fee environments: under constant fees the paired difference is indistinguishable from zero. The result is model-conditioned counterfactual evidence about execution control in AMMs, not evidence about historical traders, equilibrium play, or deployable profit.

Figures

Figures reproduced from arXiv: 2607.10960 by Wen-Ting Wang.

Figure 1
Figure 1. Figure 1: The closed-loop market. The execution agent trades a target order across two constant [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Final reserved seed block (90,000–90,999, [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Representative episodes: remaining inventory for the DQN and tuned lookahead, with [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: State-conditional behavior on development test seeds, DQN vs. tuned lookahead. [PITH_FULL_IMAGE:figures/full_fig_p012_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Priority retraining on 300 development test seeds, forced-terminal completion. (a) DQN [PITH_FULL_IMAGE:figures/full_fig_p013_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Agent-first benchmark ladder on the final reserved seed block (90,000–90,999, [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Sensitivity layers: paired DQN − lookahead edge (points) with pointwise 95% bootstrap CIs (whiskers), 500-seed reserved blocks, frozen agent-first checkpoint, no retraining. Every interval lies below zero. Depth troughs under LP adaptation are 0.97 (weak) and 0.58 (aggressive); under the aggressive searcher, absolute costs rise by roughly 70 bps for every policy. Thinner books penalize larger clips more, s… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

25 extracted references · 6 linked inside Pith

  1. [1]

    Almgren and N

    R. Almgren and N. Chriss. Optimal execution of portfolio transactions.Journal of Risk, 3(2):5–39, 2001. 14

    2001

  2. [2]

    Angeris and T

    G. Angeris and T. Chitra. Improved price oracles: Constant function market makers. In Proceedings of the 2nd ACM Conference on Advances in Financial Technologies (AFT), pages 80–91, 2020

    2020

  3. [3]

    Angeris, H.-T

    G. Angeris, H.-T. Kao, R. Chiang, C. Noyes, and T. Chitra. An analysis of Uniswap markets. Cryptoeconomic Systems, 1(1), 2021

    2021

  4. [4]

    Baggiani, M

    L. Baggiani, M. Herdegen, and L. S´ anchez-Betancourt. Competition between DEXs through dynamic fees. arXiv:2603.09669, 2026

    arXiv 2026

  5. [5]

    Baggiani, M

    L. Baggiani, M. Herdegen, and L. S´ anchez-Betancourt. Optimal dynamic fees in automated market makers. arXiv:2506.02869, 2025

    Pith/arXiv arXiv 2025

  6. [6]

    Bayraktar, A

    E. Bayraktar, A. Cohen, and A. Nellis. DEX Specs: a mean field approach to DeFi currency exchanges. arXiv:2404.09090, 2024

    Pith/arXiv arXiv 2024

  7. [7]

    Bergault, L

    P. Bergault, L. Bertucci, D. Bouba, and O. Gu´ eant. Automated market makers: Mean- variance analysis of LPs payoffs and design of pricing functions.Digital Finance, 6(2):225–247, 2024

    2024

  8. [8]

    Bertsimas and A

    D. Bertsimas and A. W. Lo. Optimal control of execution costs.Journal of Financial Markets, 1(1):1–50, 1998

    1998

  9. [9]

    Campbell, P

    S. Campbell, P. Bergault, J. Milionis, and M. Nutz. Optimal fees for liquidity provision in automated market makers. arXiv:2508.08152, 2025

    Pith/arXiv arXiv 2025

  10. [10]

    Capponi and R

    A. Capponi and R. Jia. The adoption of blockchain-based decentralized exchanges. arXiv:2103.08842, 2021

    Pith/arXiv arXiv 2021

  11. [11]

    Cartea, F

    ´A. Cartea, F. Drissi, and M. Monga. Decentralised finance and automated market making: Execution and speculation.Journal of Economic Dynamics and Control, 177:105134, 2025

    2025

  12. [12]

    Cartea, F

    ´A. Cartea, F. Drissi, and M. Monga. Decentralized finance and automated market making: Predictable loss and optimal liquidity provision.SIAM Journal on Financial Mathematics, 15(3):931–959, 2024

    2024

  13. [13]

    Daian, S

    P. Daian, S. Goldfeder, T. Kell, Y. Li, X. Zhao, I. Bentov, L. Breidenbach, and A. Juels. Flash Boys 2.0: Frontrunning in decentralized exchanges, miner extractable value, and consensus instability. InIEEE Symposium on Security and Privacy (S&P), pages 910–927, 2020

    2020

  14. [14]

    Hasbrouck, T

    J. Hasbrouck, T. J. Rivera, and F. Saleh. The need for fees at a DEX: How increases in fees can increase DEX trading volume.Management Science, Articles in Advance, 2026. doi:10.1287/mnsc.2023.00726

    doi:10.1287/mnsc.2023.00726 2026

  15. [15]

    van Hasselt

    H. van Hasselt. Double Q-learning. InAdvances in Neural Information Processing Systems 23 (NeurIPS), pages 2613–2621, 2010

    2010

  16. [16]

    Henderson, R

    P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger. Deep reinforce- ment learning that matters. InProceedings of the 32nd AAAI Conference on Artificial Intelligence, pages 3207–3214, 2018

    2018

  17. [17]

    Hendricks and D

    D. Hendricks and D. Wilcox. A reinforcement learning extension to the Almgren–Chriss framework for optimal trade execution. InIEEE Conference on Computational Intelligence for Financial Engineering & Economics (CIFEr), pages 457–464, 2014

    2014

  18. [18]

    Lehar and C

    A. Lehar and C. A. Parlour. Decentralized exchange: The Uniswap automated market maker.Journal of Finance, 80(1):321–374, 2025. 15

    2025

  19. [19]

    Milionis, C

    J. Milionis, C. C. Moallemi, T. Roughgarden, and A. L. Zhang. Automated market making and loss-versus-rebalancing. arXiv:2208.06046, 2022

    Pith/arXiv arXiv 2022

  20. [20]

    V. Mnih, K. Kavukcuoglu, D. Silver, et al. Human-level control through deep reinforcement learning.Nature, 518(7540):529–533, 2015

    2015

  21. [21]

    Nevmyvaka, Y

    Y. Nevmyvaka, Y. Feng, and M. Kearns. Reinforcement learning for optimized trade execution. InProceedings of the 23rd International Conference on Machine Learning (ICML), pages 673–680, 2006

    2006

  22. [22]

    B. Ning, F. H. T. Lin, and S. Jaimungal. Double deep Q-learning for optimal execution. Applied Mathematical Finance, 28(4):361–380, 2021

    2021

  23. [23]

    A. F. Perold. The implementation shortfall: Paper versus reality.Journal of Portfolio Management, 14(3):4–9, 1988

    1988

  24. [24]

    R. S. Sutton and A. G. Barto.Reinforcement Learning: An Introduction. MIT Press, 2nd edition, 2018

    2018

  25. [25]

    W.-T. Wang. Causal effects of protocol-fee changes on liquidity provision in automated market makers. arXiv:2607.08525, 2026. A Agent-first benchmark ladder Figure 6: Agent-first benchmark ladder on the final reserved seed block (90,000–90,999, n = 1,000; DynamicDuopoly, forced-terminal completion, agent-first ordering; 95% bootstrap CIs; completion is 1 ...

    Pith/arXiv arXiv 2026