REVIEW 4 major objections 4 minor 20 references
Where to Intervene: Action Selection in Deep Reinforcement Learning
T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Knockoff sampling identifies the minimal sufficient action set in online deep RL and controls the modified false discovery rate.
desk verdict Neat knockoff-sampling idea for online RL action selection, but the FDR guarantee does not cover the majority-voted set the agent actually uses. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing device is the knockoff pair $(A_t, \tilde A_t)$: for each action $A_t$ sampled from the policy $\pi_\theta$, the algorithm independently resamples $\tilde A_t \sim \pi_\theta(\cdot \mid S_t)$. Because the policy supplies the exact conditional distribution, the swap property $(A,\tilde A)_{\mathrm{swap}(\Omega)} \stackrel{d}{=} (A,\tilde A)$ holds by construction, and Lemma I.2 extends this exchangeability to the joint law of states and responses for null actions. The resulting anti-symmetric knockoff statistics $W_j = f(Z_j,\tilde Z_j)$ with the threshold of equation (2) give mFDR-controlled selection, while sample splitting into $K$ folds plus a majority vote converts temporally dependent RL trajectories into approximately independent blocks. The other piece is a hard binary mask that blocks gradients for non-selected actions in both the $Q$-function and the policy log-probability, allowing selection to be applied mid-training without reinitializing the network.
What would settle it
Run the KS procedure on a synthetic MDP with a known minimal action set, a near-uniform initial policy, and a single influential action with a small transition or reward effect; if the selected mask frequently drops the true action (TPR below 1) or the empirical FDR exceeds the specified $\alpha$ once the sample is large, the claim of reliable mFDR-controlled identification in this regime is falsified.
Extended reading notes
Core claim
The paper's central claim is that action selection in online RL can be reduced to a knockoff variable selection problem whose knockoff features cost almost nothing: because the policy is a known conditional distribution of actions given states, one can resample a second action from the same policy as an exact model-X knockoff. Coupled with sample splitting and a majority vote, this yields a selected action set $\hat G$ that is consistent for the minimal sufficient action set and satisfies $\mathrm{mFDR} \le \alpha + O\{K^{-1}(NT)^{-c}\}$ under stationarity and exponential $\beta$-mixing. The masking trick then converts the selected indices into a hard binary mask on the policy and value functions, so no new network needs to be reinitialized after selection.
Load-bearing premise
The load-bearing premise is that by the time selection runs, the policy network has already learned enough about which actions matter that importance scores computed from buffered transitions can separate essential from redundant actions; if the early policy is nearly random or the true actions have weak effects, the mask can be wrong and the agent can be permanently degraded.
Editorial extensions
If this is right
- KS-selected agents should outperform full-action agents across continuous-control tasks, and the gap should widen as the number of redundant actions grows; the paper reports this pattern for Ant, HalfCheetah, and Hopper with $p = 20$ and $p = 50$.
- The selection step is lightweight, completing in under about 20 seconds on a few thousand transitions, so inserting it during online training adds negligible computational overhead compared with RL training itself.
- Mid-training masking can rescue an agent that has spent the first half of its budget exploring the full action space: after the mask is applied, learning curves improve relative to continuing on all actions.
- When the mask is correct, excluding redundant actions does not add bias because those actions are conditionally independent of reward and next state, and shrinking the action input improves sample efficiency for value and policy estimation.
- The selected set is interpretable: in the sepsis treatment environment, KS selects vaso_dose and iv_input while avoiding non-essential medications such as beta_blocker and diuretic.
Reading between the lines
- If the main mechanism is right, the same 'resample from the acting distribution as a knockoff' trick should transfer to off-policy or batch settings wherever a known proposal distribution is available, giving FDR-controlled action subset selection without a learned generative model.
- A natural extension the authors flag but do not pursue is repeated adaptive selection: scheduling KS at multiple checkpoints would likely correct an initial miss, at the cost of using stale buffers, and the procedure's sensitivity to the selection time $T_{vs}$ is directly testable.
- The theory controls the mFDR of the selection step, not the reward distribution of the masked agent; a sharper question is how mask errors, both false positives and false negatives, translate into value-function bias, since the paper's bias argument assumes the mask is correct.
- The unstated practical bottleneck is the early policy's informativeness: a sweep over initial policy entropy and action-effect strength would reveal where the method transitions from reliable selection to degraded masks.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Knockoff Sampling (KS), a data-driven action selection method for online deep reinforcement learning with continuous actions. The method samples a knockoff copy of the action from the current policy network at each step, stores it alongside the real transition, then at a predefined time performs sample-splitting, per-split model-X knockoff selection, and a majority vote to produce a final selected action set. Non-selected actions are neutralized by a hard mask in the policy and value networks, allowing the selection to be integrated into PPO and SAC without reinitializing the model. The authors claim theoretical mFDR control for the selection and report experiments in synthetic MuJoCo tasks and a sepsis treatment environment showing improved rewards and low FDR compared with using all actions.
Significance. The core idea of using the policy network's own conditional action distribution to generate exact knockoff copies is a genuinely neat contribution that sidesteps the main practical difficulty of model-X knockoffs. If the theoretical guarantee were established for the final set produced by Algorithm 2, the paper would provide a broadly applicable, model-free, computationally light action-selection method for continuous-action deep RL. The experimental study is substantial: multiple MuJoCo tasks, two RL algorithms, two selection timings, an EHR-calibrated treatment environment, and clear reporting of TPR/FDR/FPR in addition to rewards. The paper also states its limitation about weak-signal actions explicitly. However, as detailed below, the advertised FDR guarantee is not currently proven for the final majority-voted selected set, and the base theoretical lemmas are asserted rather than proved.
major comments (4)
- [§4, Theorem 4.4; Algorithm 2] Theorem 4.4 controls mFDR only for each per-split selection set G_k, as its statement reads 'Then G_k obtained by Algorithm 2 ... controls the modified FDR'. However, Algorithm 2 returns the majority-voted set G = { j : sum_{k=1}^K I(j in G_k) >= K Gamma }, and no lemma or corollary transfers the mFDR bound to this final set. This gap is load-bearing because the abstract and Section 1 claim that the method 'controls the false discovery rate' and 'selects minimal sufficient actions.' The issue is not merely cosmetic: with Gamma = 0.5 and K = 2, G is the union of G_1 and G_2, and a null action selected with probability near alpha in each split can appear in the union with probability roughly 2 alpha, so even perfect per-split control does not imply alpha-level control for the returned set. The authors should either prove an mFDR bound for the majority-voted G under the stated conditions or explicitly restate the paper's claims to apply only to per-split sets rather than to the set used for masking.
- [Appendix I.1, Lemma I.2] Lemma I.2 is not proved in the manuscript. The proof says 'we can directly utilize the proof of Lemma 3.2 in Candès et al. (2018)' and then 'We omit further details of the proof.' Since Lemma I.2 establishes the joint exchangeability of (A, A_tilde, S, Y) under swaps of null actions, it is the foundation of the independent-data mFDR argument in Lemma I.4. The reduction to Candès et al. (2018) is nontrivial here because the action distribution is conditional on the state S and the actions are sampled from the current policy network rather than from a fixed covariate distribution; one must verify that conditioning on S and swapping null action coordinates preserves the joint law of the response and next state. The authors should provide a complete, self-contained proof of Lemma I.2 (and of Lemma I.1, which is also asserted in a single sentence).
- [Appendix I.2, Proof of Theorem 4.4] The dependent-data part of Theorem 4.4 is delegated: the proof states that it is 'essentially the same as Theorem 1 in Ma et al. (2023)' and omits the steps. This is a central theoretical claim of the paper, and the dependence adjustment, the choice K = k0 log(NT), and the beta-mixing argument are part of the paper's stated contribution. Relying on an unpublished arXiv preprint for the main proof means the guarantee is not established within the manuscript. The authors should either provide the full dependent-data proof or clearly mark Theorem 4.4 as a corollary of Ma et al. (2023) with all additional assumptions and constants stated and verified for this setting.
- [Appendix G, flip-sign property] The paper claims in Appendix G that the fairness requirement on the feature importance scores is 'typically met by standard tabular machine learning algorithms,' and the experiments use LASSO as the base algorithm. This is not generally true for all algorithms listed in Algorithm 2: for example, impurity-based random forest importance and neural-network weight/gradient scores are not guaranteed to be anti-symmetric under swapping an original action with its knockoff when predictors are correlated, so Lemma I.3 can fail. The theoretical results should be stated for a class of algorithms for which the flip-sign property is proven, and the experiments should either use such an algorithm or the practical discrepancy should be explicitly acknowledged and tested.
minor comments (4)
- [Section 5] In the experiment setup paragraph, '2× 105 and 106 steps' should read '2×10^5 and 10^6 steps'; the superscripts appear to have been lost in typesetting.
- [Table 1] The header of Table 1 appears corrupted: the column 'Env RL Algo. p Selection Ant' and the duplicated 'Ant' entries make the table structure difficult to read. Please reformat the table so that environment names and algorithm names are aligned with their rows.
- [Figure 2 caption] The right panel of Figure 2 is labeled 'Ant-v4 SAC' but the legend inside the panel shows 'PPO + True Actions', 'PPO + KS', and 'PPO + VS'. If the panel uses SAC, the labels should say SAC; if it uses PPO, the caption should say so. This inconsistency is confusing.
- [Algorithm 1 and Section 3.2] Algorithm 1 stores the tuple (s_t, a_t, a_tilde_t, r_t, s_{t+1}) in the replay buffer, while Section 3.2 describes the data as N trajectories of length T_j and then says the data buffer has size M. The relationship between the transition-tuple view and the trajectory view should be clarified so that the reader can see how the sample-splitting step in Algorithm 2 maps to the beta-mixing argument.
Circularity Check
Dependent-data half of the FDR theorem is delegated to a same-author preprint; the remainder of the derivation is self-contained.
-
self citation load bearing
[Appendix I.2, Proof of Theorem 4.4; see also Section 3.2]
"Using Lemma I.4, we can show that if the data points in Dk are independent, then mFDR can be controlled. Now we want to weaken the independence assumption to stationarity and exponential β-mixing assumption in 4.3. Based on Lemma I.4, the following proof is essentially the same as Theorem 1 in Ma et al. (2023). We will omit those steps for brevity."
Theorem 4.4 is the paper's central advertised guarantee of false discovery rate control. The independent-data case is handled by Lemma I.4, but the dependent beta-mixing case is not derived in this paper: the proof simply asserts it is 'essentially the same as Theorem 1 in Ma et al. (2023)' and omits the steps. Ma et al. (2023) is an arXiv preprint whose author list includes present co-author Hengrui Cai; it is not machine-checked, code-reproduced, or independently verified here, and the paper supplies no argument bridging Candes et al. (2018) to the beta-mixing bound other than this same-author citation. The same dependency is visible in Section 3.2, where sample splitting and majority vote are adopted 'following Ma et al. (2023)'.
full rationale
The knockoff-sampling construction itself is not circular: the knockoff copy is drawn from the current policy network π_θ(·|s_t), so exchangeability of A and tilde A given the state holds by construction, and the importance-score/threshold machinery is a direct application of the external model-X knockoff theory of Candes et al. (2018). The empirical reward comparisons against all-actions and true-actions agents are externally falsifiable, not derived from the fitted mask. The main circularity concern is narrow but real: Theorem 4.4's dependent-data extension is explicitly omitted and attributed to Ma et al. (2023), an author-overlapping preprint, making the central theoretical guarantee partly load-bearing on a self-citation. I do not count the absence of an FDR guarantee for the final majority-voted set Ĝ as circularity—it is a missing justification (Algorithm 2 returns Ĝ = {j : Σ_k I(j ∈ Ĝ_k) ≥ KΓ} while Theorem 4.4 only covers per-split Ĝ_k), which I weigh as an additional correctness risk without inflating the circularity score. Since the paper contains genuine independent content (action-knockoff swap lemmas, independent-data mFDR control, and self-contained experiments), the appropriate score is 4 rather than 6 or 8.
Assumptions & free parameters
free parameters (3)
- FDR level alpha =
0.1
- majority voting ratio Gamma =
0.5
- variable selection step T_vs =
4000 samples (initial stage), 1000 samples (EHR)
assumptions (4)
- domain assumption The process {(S_t, A_t, R_t)} is stationary and exponentially beta-mixing (Definition 4.3).
- domain assumption The actions are conditionally independent given the state under the diagonal Gaussian policy.
- domain assumption The state S_t is sufficient (no useless states), so redundant actions are not needed.
- standard math The feature importance method satisfies the fairness constraint that swapping a variable and its knockoff swaps their importance scores.
Cite this review
Pith. "Pith review of Where to Intervene: Action Selection in Deep Reinforcement Learning." pith.science (2026). https://pith.science/paper/ZHOSHIXY
@misc{pith2026250704187,
author = {Pith},
title = {Pith review of: Where to Intervene: Action Selection in Deep Reinforcement Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZHOSHIXY}},
note = {Machine review of arXiv:2507.04187}
}
read the original abstract
Deep reinforcement learning (RL) has gained widespread adoption in recent years but faces significant challenges, particularly in unknown and complex environments. Among these, high-dimensional action selection stands out as a critical problem. Existing works often require a sophisticated prior design to eliminate redundancy in the action space, relying heavily on domain expert experience or involving high computational complexity, which limits their generalizability across different RL tasks. In this paper, we address these challenges by proposing a general data-driven action selection approach with model-free and computationally friendly properties. Our method not only selects minimal sufficient actions but also controls the false discovery rate via knockoff sampling. More importantly, we seamlessly integrate the action selection into deep RL methods during online training. Empirical experiments validate the established theoretical guarantees, demonstrating that our method surpasses various alternative techniques in terms of both performance in variable selection and overall achieved rewards.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
It is shown that KS is efficient in selecting only the minimal sufficient action set in almost all scenarios, which also empirically validates our theory of FDR control under the proposed method. C.2 Treatment Allocation for Sepsis Patients We utilize the MIMIC-III Clinical Database to construct our environment for Sepsis patients. We filter and clean the...
work page 2025
-
[2]
This condition is typically met by standard tabular machine learning algorithms
The only requirement for the machine learning method is that it satisfies a fairness constraint, ensuring that exchanging an original feature with its knockoff counterpart results solely in the corresponding exchange of the model’s importance score for those features. This condition is typically met by standard tabular machine learning algorithms. H Exten...
work page 2018
-
[4]
Model-based reinforcement learning for atari
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model-based reinforcement learning for atari. arXiv preprint arXiv:1903.00374,
arXiv 1903
-
[8]
Quasi-optimal Reinforcement Learning with Continuous Actions
Yuhan Li, Wenzhuo Zhou, and Ruoqing Zhu. Quasi-optimal reinforcement learning with continuous actions. arXiv preprint arXiv:2301.08940,
-
[10]
Sequential Knockoffs for Variable Selection in Reinforcement Learning
Tao Ma, Hengrui Cai, Zhengling Qi, Chengchun Shi, and Eric B Laber. Sequential knockoffs for variable selection in reinforcement learning.arXiv preprint arXiv:2303.14281,
-
[12]
Proximal policy optimization algorithms
15 Published in Transactions on Machine Learning Research (7/2025) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,
arXiv 2025
-
[15]
FRESH: Interactive Reward Shaping in High-Dimensional State Spaces using Human Feedback
Baicen Xiao, Qifan Lu, Bhaskar Ramasubramanian, Andrew Clark, Linda Bushnell, and Radha Poovendran. Fresh: Interactive reward shaping in high-dimensional state spaces using human feedback.arXiv preprint arXiv:2001.06781,
work page Pith review arXiv 2001
-
[18]
or the patient’s mortality rate reaching the maximum (SOFA score being24). We also observe that Weight-kg and cumulated-balance have minimal influence on Sepsis; therefore, we treat them as non-essential state variables and exclude them when constructing the responseyt for variable selection. 21 Published in Transactions on Machine Learning Research (7/20...
work page 2025
Show all 20 references
-
[20]
Then for suchϵ, denote Ω :={i :ϵi =−1}, which is a subset ofH0 by the assumption (and recall thatH0 is the collection of all null variables)
= 1/2 are independent for all null state variables. Then for suchϵ, denote Ω :={i :ϵi =−1}, which is a subset ofH0 by the assumption (and recall thatH0 is the collection of all null variables). By Lemma I.3 we know (W1·ϵ1,...,W p·ϵp) =Wswap(Ω). For convenience, we also useh to...
2015
-
[2009]
Generalized fisher score for feature selection.arXiv preprint arXiv:1202.3725,
Quanquan Gu, Zhenhui Li, and Jiawei Han. Generalized fisher score for feature selection.arXiv preprint arXiv:1202.3725,
-
[2011]
Deep reinforcement learning with attention for slate markov decision processes with high-dimensional states and actions
Peter Sunehag, Richard Evans, Gabriel Dulac-Arnold, Yori Zwols, Daniel Visentin, and Ben Coppin. Deep reinforcement learning with attention for slate markov decision processes with high-dimensional states and actions. arXiv preprint arXiv:1512.01124,
-
[2012]
Sample efficient feature selection for factored mdps.arXiv preprint arXiv:1703.03454,
Zhaohan Daniel Guo and Emma Brunskill. Sample efficient feature selection for factored mdps.arXiv preprint arXiv:1703.03454,
-
[2013]
Modern perspectives on reinforcement learning in finance.Modern Perspectiveson ReinforcementLearning in Finance (September 6, 2019)
Petter N Kolm and Gordon Ritter. Modern perspectives on reinforcement learning in finance.Modern Perspectiveson ReinforcementLearning in Finance (September 6, 2019). The Journal of MachineLearning in Finance, 1(1),
2019
-
[2018]
Growing action spaces
13 Published in Transactions on Machine Learning Research (7/2025) Gregory Farquhar, Laura Gustafson, Zeming Lin, Shimon Whiteson, Nicolas Usunier, and Gabriel Synnaeve. Growing action spaces. InInternational Conference on Machine Learning, pp. 3040–3051. PMLR,
2025
-
[2019]
Action space shaping in deep reinforcement learning
Anssi Kanervisto, Christian Scheller, and Ville Hautamäki. Action space shaping in deep reinforcement learning. In 2020 IEEE conference on games (CoG), pp. 479–486. IEEE,
2020
-
[2020]
Playing atari with deep reinforcement learning.arXiv preprint arXiv:1312.5602,
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning.arXiv preprint arXiv:1312.5602,
-
[2021]
Deep reinforcement learning in continuous action spaces: a case study in the game of simulated curling
14 Published in Transactions on Machine Learning Research (7/2025) Kyowoon Lee, Sol-A Kim, Jaesik Choi, and Seong-Whan Lee. Deep reinforcement learning in continuous action spaces: a case study in the game of simulated curling. InInternational conference on machine learning, p...
2025
-
[2022]
Auto-encoding knockoff generator for fdr controlled variable selection.arXiv preprint arXiv:1809.10765,
Ying Liu and Cheng Zheng. Auto-encoding knockoff generator for fdr controlled variable selection.arXiv preprint arXiv:1809.10765,
-
[2023]
Matteo Sesia, Chiara Sabatti, and Emmanuel J Candès
URLhttps://openreview.net/forum?id= C-xa_D3oTj6. Matteo Sesia, Chiara Sabatti, and Emmanuel J Candès. Gene hunting with knockoffs for hidden markov models. arXiv preprint arXiv:1706.04677,
-
[2024]
(2023) adopted a two-stage framework, performing variable selection offline before applying reinforcement learning
16 Published in Transactions on Machine Learning Research (7/2025) A Comparison with Closely Related Work Ma et al. (2023) adopted a two-stage framework, performing variable selection offline before applying reinforcement learning. This design is ill-suited for online settings...
2023
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.