REVIEW 3 major objections 5 minor 15 references
Select before Act: Spatially Decoupled Action Repetition for Continuous Control
T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read Per-dimension action repetition outperforms whole-action repetition in continuous control.
desk verdict Useful, genuinely per-dimension action repetition with solid experiments, but the selection-policy update in Eq. (9) has a load-bearing mismatch that needs a fix or clarification before the high-dimensional claims are fully credible. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the per-dimension selection mask $b\in\{0,1\}^{|A|}$ produced by the selection policy $\beta(b|s,a^-)$, together with the Mix operation $a=(1-b)\odot a^-+b\odot\hat a$. The mask turns action repetition into a differentiable, state-dependent masking problem: repetition coordinates copy the previous action, while 'act' coordinates take fresh samples from the action policy. The two policies are trained by entropy-regularized critic maximization, with an exact summation over masks for small action spaces and an importance-sampled surrogate for high-dimensional ones.
What would settle it
Compare Humanoid training with Equation (9) exactly as printed against a version with the entropy term corrected to $\log \beta(b|s,a^-)$ inside the expectation; differing curves would show the published update rule is not what produced the reported results. As a complementary check, run SDAR on a locomotion task with tightly synchronized joints and show that per-dimension masks still beat whole-vector repetition when the selection-action loop is otherwise identical.
Extended reading notes
Core claim
SDAR replaces the single act-or-repeat decision over the whole action vector with per-dimension Bernoulli decisions. Given state $s$ and previous action $a^-$, the selection policy $\beta(b|s,a^-)$ samples a binary mask $b\in\{0,1\}^{|A|}$; the action policy $\pi(\hat a|s,a^-,b)$ then generates candidates only for coordinates with $b_i=1$, and the final action is $a=(1-b)\odot a^-+b\odot\hat a$. This 'select first, act second' pipeline lets some joints persist while others update. The paper's experiments report that SDAR obtains higher normalized AUC (area under the learning curves) than SAC, N-Rep, TempoRL, UTE, and TAAC across classic control, locomotion, and manipulation, together with higher episode return and lower action fluctuation on representative tasks.
Load-bearing premise
The load-bearing premise is that Equation (9) optimizes the same entropy-regularized objective as Equation (7); as printed, the entropy term uses the old policy's log-probability rather than the new one's, so if that is not a typo the high-dimensional training objective differs from what the derivation claims.
Editorial extensions
If this is right
- If SDAR's reported gains hold, closed-loop repetition should be implemented at actuator granularity rather than on the whole action vector, especially on agents with many heterogeneous joints.
- The two-stage policy is a drop-in modification for off-policy actor-critic methods: it only needs the previous action stored in the replay buffer, so sample efficiency can come without environment-specific repetition schedules.
- On tasks like Humanoid where whole-vector methods slow down late in training, per-dimension masks preserve final-policy agility instead of trading it away for persistence.
- The higher action persistence rate and lower action fluctuation rate reported together suggest that persistence and smoothness can improve simultaneously once repetition is spatially decoupled.
Reading between the lines
- Extending the paper's logic, grouping correlated action dimensions (for example, all joints of one leg) could shrink the $2^{|A|}$ selection space while keeping most of the flexibility; the conclusion points to this as future work.
- A per-dimension value or advantage estimate could replace the importance-sampled surrogate in Equation (9), scaling to high-dimensional action spaces while sidestepping the old-policy entropy term in the published update.
- The decoupling mechanism transfers naturally to real robots whose actuators run at different control frequencies; the LunarLander visualization suggests SDAR would learn a different repeat cadence per actuator.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Spatially Decoupled Action Repetition (SDAR), a two-stage policy framework for continuous control. A selection policy β decides for each action dimension whether to repeat the previous action or act anew, and an action policy π generates new actions for the "act" dimensions. The authors derive an entropy-regularized objective (Eq. 7), give an exact but computationally expensive update for small action spaces (Eq. 8), and an importance-sampling approximation for larger action spaces (Eq. 9). They evaluate SDAR against SAC, N-Rep, TempoRL, UTE, and TAAC on classic control, locomotion, and manipulation tasks, reporting improved area-under-curve sample efficiency, higher episode returns, higher action persistence, and lower action fluctuation.
Significance. If the method performs as claimed, the idea of per-dimension act-or-repeat decisions is a useful and intuitive extension of closed-loop action repetition, and the paper provides a clean two-stage formulation with a correct construction-level proof in Appendix A that repeated dimensions keep their previous values. The experimental comparison covers many standard benchmarks and baselines. However, the high-dimensional training update in Eq. (9) as printed does not optimize the stated objective, and the central sample-efficiency and final-return claims for tasks with |A|>3 rest on this equation. Since no code is released, the discrepancy cannot be resolved by inspection, and the paper requires a corrected derivation or a clarification of the actual algorithm used in the experiments.
major comments (3)
- [Sec. 5.3, Table 2] Equation (9) is not an unbiased importance-weighted estimator of the objective in Eq. (7). The entropy term is written as -αβ log β_old(b|s,a−) rather than -αβ log β(b|s,a−), and since β_old is fixed when optimizing θβ, this term does not provide the intended entropy regularization. The gradient of the whole contribution involving β_old is nonzero because of the IS weight β/β_old, but it is not the gradient of E_{b~β}[-αβ log β(b)]; it differs by a factor involving log β_old and misses the (1+log β) structure. This matters because Appendix B.1 states that all tasks with |A|>3 (Humanoid, HalfCheetah, Walker2d, Ant, etc.) use Eq. (9). The authors should either correct the formula to use log β(b) or explicitly state that the reported results were obtained with a different objective; if the formula is a typo, the empirical claims need to be re-derived under the corrected update.
- [Sec. 5.1 / Appendix C] The APR and AFR metrics are not neutral measurements for SDAR: the method is designed to force some dimensions to keep their previous action values, so the higher APR and lower AFR are partly mechanical consequences of the algorithm rather than independent evidence of improved smoothness. The contribution bullet in the abstract explicitly lists "reduced action fluctuation," and the conclusion repeats this claim, but the current metric cannot distinguish between a policy that produces smooth actions because it is good and one that does so because it is forced to repeat. A fairer comparison would measure smoothness at matched action-change budgets or reward-equivalent settings.
- [Sec. 5.1 / Appendix C] The normalization of the AUC scores in Table 1 is inconsistent with Appendix C. Table 1 states that scores are normalized to [0,1] where 1.0 denotes the best method, while Appendix C defines the n-score as (Z−Z0)/(Z1−Z0) with Z0 from a random policy and Z1 from vanilla DRL. These are different normalizations and would generally produce different averages across tasks. Since the AUC table is the main quantitative support for the sample-efficiency claim, the authors should clarify the exact procedure used to compute Table 1 and the associated standard errors.
minor comments (5)
- [Abstract and Sec. 1] There is a typo "exsiting" in the introductory paragraph that should be "existing."
- [Sec. 5.3] The bullet list in Sec. 5.3 states that SDAR achieves "a lower AFR than the vanilla DRL," but Table 2 shows that in LunarLander SDAR has AFR 0.10 versus SAC's 0.09, so the statement is not universally true; it should be qualified to the tasks where it holds.
- [Sec. 5.3] The text says "SDAR also selects a lot of repeat during interaction with a high APR (3.69)", but Table 2 reports the average APR for SDAR as 3.75; please correct the inconsistency.
- [Appendix B.1] The hyperparameter table lists "Sample number (b) 10", but the main text in Sec. 4.3 only says "several b ∈ B"; specifying the exact number in the main text would help reproducibility.
- [Table 2] The "Average" row mixes raw episode returns and normalized scores without explanation; since rewards have different scales across tasks, the reader cannot interpret this average without knowing the normalization used.
Circularity Check
No significant circularity: SDAR's claims rest on independent empirical comparisons, and the only self-citation is non-load-bearing.
full rationale
The paper's central claims - higher sample efficiency, higher final return, and reduced action fluctuation relative to prior repetition frameworks - are established by empirical comparison against independent baselines (SAC, N-Rep, TempoRL, UTE, TAAC) in Section 5, with learning curves, AUC scores, and episode-return tables. The derivation of the policy updates (Eqs. 7-9) is a standard entropy-augmented actor-critic objective; the apparent substitution of log beta_old for log beta in the entropy term of Eq. (9) is a likely notational or typographical issue that affects whether the printed objective is exactly optimized, but it does not make the result follow from its own inputs or from a fitted parameter. The reported reduction in AFR is an intended consequence of the algorithm (Eq. (4) forces output action components to equal the previous action where repetition is selected), but the paper does not define success solely by AFR; episode return and sample efficiency are measured independently. The only self-citation (Zhang et al., 2024, an earlier locomotion paper by overlapping authors) is cited in the introduction as an example of RL success in locomotion and is not load-bearing for any derivation or comparison. No fitted parameter is relabeled as a prediction, no uniqueness theorem is imported from the authors' prior work, and no known result is merely renamed. Thus the derivation chain is self-contained rather than circular.
Assumptions & free parameters
free parameters (3)
- lambda (target entropy scale for beta) =
0.4 to 0.6
- action mask xi =
-2
- importance-sampling sample count =
10
assumptions (4)
- standard math The interaction is a Markov Decision Process with the standard Bellman operator
- domain assumption The two-stage policy pi_all in Eq. (5) is a valid policy for the MDP and can be trained by off-policy Q-learning
- domain assumption Per-dimension act-or-repeat decisions are independent; the selection policy factorizes as Bernoulli per dimension
- ad hoc to paper Eq. (9) is an unbiased importance-sampling estimator of Eq. (7)
Cite this review
Pith. "Pith review of Select before Act: Spatially Decoupled Action Repetition for Continuous Control." pith.science (2026). https://pith.science/paper/EZCJHEWS
@misc{pith2026250206919,
author = {Pith},
title = {Pith review of: Select before Act: Spatially Decoupled Action Repetition for Continuous Control},
year = {2026},
howpublished = {\url{https://pith.science/paper/EZCJHEWS}},
note = {Machine review of arXiv:2502.06919}
}
read the original abstract
Reinforcement Learning (RL) has achieved remarkable success in various continuous control tasks, such as robot manipulation and locomotion. Different to mainstream RL which makes decisions at individual steps, recent studies have incorporated action repetition into RL, achieving enhanced action persistence with improved sample efficiency and superior performance. However, existing methods treat all action dimensions as a whole during repetition, ignoring variations among them. This constraint leads to inflexibility in decisions, which reduces policy agility with inferior effectiveness. In this work, we propose a novel repetition framework called SDAR, which implements Spatially Decoupled Action Repetition through performing closed-loop act-or-repeat selection for each action dimension individually. SDAR achieves more flexible repetition strategies, leading to an improved balance between action persistence and diversity. Compared to existing repetition frameworks, SDAR is more sample efficient with higher policy performance and reduced action fluctuation. Experiments are conducted on various continuous control scenarios, demonstrating the effectiveness of spatially decoupled repetition design proposed in this work.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[4]
In addition, we need to tune the target entropies Hβ and Hπ to improve the efficiency of the entropy-based exploration described in Eq. (10). In this work, we utilize Hπ = −|A|, where |A| denotes the size of the action space, corresponding to the recommendation given in (Haarnoja et al., 2018). Besides, Hβ = λ · |A| log 2, where λ is a hyper-parameter tun...
work page 2018
-
[6]
Multi-goal reinforce- ment learning: Challenging robotics environments and request for research
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Pow- ell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, et al. Multi-goal reinforce- ment learning: Challenging robotics environments and request for research. arXiv preprint arXiv:1802.09464,
-
[9]
Between mdps and semi-mdps: A frame- work for temporal abstraction in reinforcement learning
12 Published as a conference paper at ICLR 2025 Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A frame- work for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181– 211,
work page 2025
-
[12]
Given ∀s ∈ S, a− ∈ A, b∈ {0, 1}|A|, based on the two-stage policy described in Sec
13 Published as a conference paper at ICLR 2025 A P ROOF OF THE STATEMENT IN SECTION 4.1 Statement. Given ∀s ∈ S, a− ∈ A, b∈ {0, 1}|A|, based on the two-stage policy described in Sec. 4.1, the output action a replicates the same actions as a− in repetition dimensions {i|bi = 0, 1 ≤ i ≤ |A|}. Proof. As described in Sec. 4.1, a = (1− b) ⊙ a− + b ⊙ ˆa. Thus,...
work page 2025
-
[14]
is a famous model-free RL in continuous control domains, which trains policies efficiently with entropy-based exploration strategies. In this work, we utilize SAC implementation and hyper-parameter settings proposed in CleanRL (Huang et al., 2022)1. (2) N-Rep forces the agent to repeat the actions output by the policy for n times, where n is a hyper-param...
work page 2022
-
[15]
As shown in Table 6, our method requires less training times than vanilla SAC to achieve the same policy performance, because of the higher sample efficiency of SDAR, requiring less training steps in this experiment. In addition, there exist two policy networks β and π in the SDAR, both of which need to be optimized through gradient descent. However, β an...
work page 2017
-
[1999]
Mujoco: A physics engine for model-based control
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , pp. 5026–5033. IEEE,
2012
-
[2012]
doi: 10.1109/IROS.2012.6386109. Alexander Vezhnevets, V olodymyr Mnih, Simon Osindero, Alex Graves, Oriol Vinyals, John Aga- piou, et al. Strategic attentive writer for learning macro-actions. Advances in neural information processing systems, 29,
arXiv 2012
Show all 15 references
-
[2014]
Learning options in reinforcement learning
Martin Stolle and Doina Precup. Learning options in reinforcement learning. In Abstraction, Re- formulation, and Approximation: 5th International Symposium, SARA 2002 Kananaskis, Alberta, Canada August 2–4, 2002 Proceedings 5, pp. 212–223. Springer,
2002
-
[2015]
Juewu-mc: Playing minecraft with sample-efficient hierarchical reinforcement learning
Zichuan Lin, Junyou Li, Jianing Shi, Deheng Ye, Qiang Fu, and Wei Yang. Juewu-mc: Playing minecraft with sample-efficient hierarchical reinforcement learning. arXiv preprint arXiv:2112.04907,
-
[2017]
Proximal policy optimization algorithms
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,
-
[2018]
Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates
Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In 2017 IEEE international confer- ence on robotics and automation (ICRA), pp. 3389–3396. IEEE,
2017
-
[2021]
Deep reinforcement learning for autonomous driving: A survey
11 Published as a conference paper at ICLR 2025 B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yoga- mani, and Patrick P ´erez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportati...
2025
-
[2022]
An analysis of frame-skipping in rein- forcement learning
Shivaram Kalyanakrishnan, Siddharth Aravindan, Vishwajeet Bagdawat, Varun Bhatt, Harshith Goka, Archit Gupta, Kalpesh Krishna, and Vihari Piratla. An analysis of frame-skipping in rein- forcement learning. arXiv preprint arXiv:2102.03718,
-
[2024]
Learning quadrupedal locomotion over challenging terrain
Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science robotics, 5(47):eabc5986, 2020b. TP Lillicrap. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971,
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.