{"id":"bfb7addc-d67c-45c4-8031-dbaef555a6a5","arxiv_id":"2502.06919","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SDAR performs closed-loop act-or-repeat selection separately for each action dimension, improving sample efficiency and reducing action fluctuation in continuous control.","lead":"This paper introduces SDAR, a reinforcement learning method in which each action dimension can independently decide to repeat its previous value or act anew, rather than repeating all dimensions together. It reports better sample efficiency, higher returns, and smoother actions on continuous control benchmarks, which matters for robot controllers where different joints run at different rates.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (9) as printed uses log β_old in the entropy term, so the selection-policy update does not optimize the stated objective Eq. (7); since all |A|>3 tasks train via Eq. (9), this casts doubt on the high-dimensional results until the formula is clarified or corrected.","rationale":"I read the paper in good faith and find that the central claim—SDAR improves sample efficiency, policy performance, and action fluctuation over existing repetition frameworks—is supported by extensive experiments with multiple seeds, and the spatial decoupling idea is clearly presented and visualized. However, the derivation of the selection-policy update for high-dimensional action spaces is mathematically inconsistent as printed: Eq. (9) does not optimize Eq. (7) because the entropy term uses the old policy's log-probability, making its gradient zero. This is precisely the reader's weakest assumption, and I agree it is load-bearing because the majority of experiments, including the flagship Humanoid task, rely on Eq. (9). An honest non-finding would be inappropriate here. I considered the omitted baseline FiGAR as an alternative concern, since FiGAR may itself perform per-dimension repetition, but that would weaken the novelty claim rather than the correctness of the reported results; the Eq. (9) issue directly affects the reproducibility and theoretical grounding of the method. The paper's empirical evidence is otherwise credible, and the hyperparameter choices, ablation-style visualizations, and computational cost comparisons add support. Given the ambiguity, the appropriate verdict remains conditional, as the reader concluded; no change to the reader's verdict is needed.","tokens_in":15196,"tokens_out":14392,"duration_ms":117177,"concrete_test":"Check the official implementation, if available, for the selection-policy loss: if the entropy term uses log_prob of the newly sampled β(b|s,a-) multiplied by the importance weight β/β_old, then Eq. (9) is a typo and the concern is resolved. If the code matches the printed equation, re-derive the gradient of Eq. (7) under importance sampling; the correct estimator must contain −αβ log β(b) inside the bracket. As a further check, run an ablation on Humanoid with the corrected entropy term and compare AUC against the printed objective; a significant change would confirm the term is consequential.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper claims to optimize the entropy-regularized objective Eq. (7), but Eq. (9), used for all tasks with action dimension larger than 3, replaces log β(b) with log β_old(b) in the entropy term. Because β_old is fixed during the update, the gradient of the term −αβ log β_old(b) is zero, so the selection policy β is trained without any entropy regularization. This means the method actually optimizes a different objective from the one stated, potentially allowing β to collapse toward deterministic decisions. The central claim of improved sample efficiency, especially on Humanoid (|A|=17), HalfCheetah, Walker2d, and Ant, rests on this equation. If this is a typographical error, it is trivially corrected; if not, the empirical results are obtained by an algorithm that differs from the paper's description, and the role of the β entropy term in exploration is unclear. The paper provides no formal verification or code release to disambiguate.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Spatially Decoupled Action Repetition (SDAR), a two-stage policy framework for continuous control. A selection policy β decides for each action dimension whether to repeat the previous action or act anew, and an action policy π generates new actions for the \"act\" dimensions. The authors derive an entropy-regularized objective (Eq. 7), give an exact but computationally expensive update for small action spaces (Eq. 8), and an importance-sampling approximation for larger action spaces (Eq. 9). They evaluate SDAR against SAC, N-Rep, TempoRL, UTE, and TAAC on classic control, locomotion, and manipulation tasks, reporting improved area-under-curve sample efficiency, higher episode returns, higher action persistence, and lower action fluctuation.","tokens_in":15455,"tokens_out":6057,"duration_ms":51584,"significance":"If the method performs as claimed, the idea of per-dimension act-or-repeat decisions is a useful and intuitive extension of closed-loop action repetition, and the paper provides a clean two-stage formulation with a correct construction-level proof in Appendix A that repeated dimensions keep their previous values. The experimental comparison covers many standard benchmarks and baselines. However, the high-dimensional training update in Eq. (9) as printed does not optimize the stated objective, and the central sample-efficiency and final-return claims for tasks with |A|>3 rest on this equation. Since no code is released, the discrepancy cannot be resolved by inspection, and the paper requires a corrected derivation or a clarification of the actual algorithm used in the experiments.","major_comments":[{"comment":"Equation (9) is not an unbiased importance-weighted estimator of the objective in Eq. (7). The entropy term is written as -αβ log β_old(b|s,a−) rather than -αβ log β(b|s,a−), and since β_old is fixed when optimizing θβ, this term does not provide the intended entropy regularization. The gradient of the whole contribution involving β_old is nonzero because of the IS weight β/β_old, but it is not the gradient of E_{b~β}[-αβ log β(b)]; it differs by a factor involving log β_old and misses the (1+log β) structure. This matters because Appendix B.1 states that all tasks with |A|>3 (Humanoid, HalfCheetah, Walker2d, Ant, etc.) use Eq. (9). The authors should either correct the formula to use log β(b) or explicitly state that the reported results were obtained with a different objective; if the formula is a typo, the empirical claims need to be re-derived under the corrected update.","section":"Sec. 5.3, Table 2"},{"comment":"The APR and AFR metrics are not neutral measurements for SDAR: the method is designed to force some dimensions to keep their previous action values, so the higher APR and lower AFR are partly mechanical consequences of the algorithm rather than independent evidence of improved smoothness. The contribution bullet in the abstract explicitly lists \"reduced action fluctuation,\" and the conclusion repeats this claim, but the current metric cannot distinguish between a policy that produces smooth actions because it is good and one that does so because it is forced to repeat. A fairer comparison would measure smoothness at matched action-change budgets or reward-equivalent settings.","section":"Sec. 5.1 / Appendix C"},{"comment":"The normalization of the AUC scores in Table 1 is inconsistent with Appendix C. Table 1 states that scores are normalized to [0,1] where 1.0 denotes the best method, while Appendix C defines the n-score as (Z−Z0)/(Z1−Z0) with Z0 from a random policy and Z1 from vanilla DRL. These are different normalizations and would generally produce different averages across tasks. Since the AUC table is the main quantitative support for the sample-efficiency claim, the authors should clarify the exact procedure used to compute Table 1 and the associated standard errors.","section":"Sec. 5.1 / Appendix C"}],"minor_comments":[{"comment":"There is a typo \"exsiting\" in the introductory paragraph that should be \"existing.\"","section":"Abstract and Sec. 1"},{"comment":"The bullet list in Sec. 5.3 states that SDAR achieves \"a lower AFR than the vanilla DRL,\" but Table 2 shows that in LunarLander SDAR has AFR 0.10 versus SAC's 0.09, so the statement is not universally true; it should be qualified to the tasks where it holds.","section":"Sec. 5.3"},{"comment":"The text says \"SDAR also selects a lot of repeat during interaction with a high APR (3.69)\", but Table 2 reports the average APR for SDAR as 3.75; please correct the inconsistency.","section":"Sec. 5.3"},{"comment":"The hyperparameter table lists \"Sample number (b) 10\", but the main text in Sec. 4.3 only says \"several b ∈ B\"; specifying the exact number in the main text would help reproducibility.","section":"Appendix B.1"},{"comment":"The \"Average\" row mixes raw episode returns and normalized scores without explanation; since rewards have different scales across tasks, the reader cannot interpret this average without knowing the normalization used.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The key issue is the incorrect importance-weighted objective in Eq. (9), which is the algorithm actually used for most benchmarks. This is a technical error that is correctable if it is a typo, but the paper as submitted does not provide the corrected form and the experimental claims are contingent on it. The AUC and return comparisons are otherwise plausible for an ICLR paper, and the decoupling idea is a reasonable incremental contribution. I would not reject outright, but the authors must fix Eq. (9) and either re-run experiments with the corrected algorithm or verify that the published results were obtained with the corrected version. The AFR/APR circularity is a secondary concern but should be addressed in the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the core idea is genuinely per-dimension act-or-repeat selection, which none of the cited baselines have; it's a natural extension of TAAC's whole-vector switch but it is new. Second, Eq. (9), which trains the selection policy for every task with more than 3 action dimensions, does not match the stated objective Eq. (7): the entropy term uses log β_old(b) instead of log β(b), so that term has zero gradient and the selection policy receives no entropy regularization. This is load-bearing because all the high-dimensional results—Humanoid, HalfCheetah, Ant, Walker2d—depend on Eq. (9).\n\nWhat the paper does well: the two-stage formulation is clean, the Mix operation is sensible, and the experiments are extensive—at least 10 seeds, AUC normalization across categories, and a nice per-dimension visualization in Walker2d showing that different joints do want different repeat frequencies. The Appendix A proof is correct. The AFR/APR metrics are informative, though the AFR reduction is partly mechanical since forcing repeats trivially lowers successive action differences.\n\nThe soft spots: Eq. (9) is the main one. If it's a typo, it's a one-line fix and the paper is mostly fine; if it isn't, the algorithm described is not the algorithm evaluated, and the role of the β entropy term in exploration is unclear. The paper also omits a closely related prior method from the baselines (per the reader's note; I didn't independently verify), and there is no code release to disambiguate either issue. Those are worth asking for.\n\nOverall, the central empirical claim is plausible and the mechanism is worth knowing about, but the current text is internally inconsistent at a load-bearing point. I would send this to reviewers and ask for the correction; if the authors confirm the typo, I'd be comfortable citing it for the per-dimension idea.","headline":"Useful, genuinely per-dimension action repetition with solid experiments, but the selection-policy update in Eq. (9) has a load-bearing mismatch that needs a fix or clarification before the high-dimensional claims are fully credible.","tokens_in":15932,"tokens_out":2566,"would_cite":true,"duration_ms":21254,"reading_group":"yes","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Per-dimension action repetition outperforms whole-action repetition in continuous control.","keywords":["action repetition","continuous control","sample efficiency","temporal abstraction","reinforcement learning","spatially decoupled","closed-loop policy","action fluctuation"],"falsifier":"Compare Humanoid training with Equation (9) exactly as printed against a version with the entropy term corrected to $\\log \\beta(b|s,a^-)$ inside the expectation; differing curves would show the published update rule is not what produced the reported results. As a complementary check, run SDAR on a locomotion task with tightly synchronized joints and show that per-dimension masks still beat whole-vector repetition when the selection-action loop is otherwise identical.","tokens_in":15027,"feed_emoji":"🤖","tokens_out":8462,"duration_ms":65946,"temperature":0.7,"pith_summary":"Reinforcement learning agents often benefit from repeating an action for several steps, but existing repetition methods tie every actuator to the same repeat-or-act decision. This paper argues that the tie is the problem: in continuous control, joints need different decision cadences. SDAR performs a closed-loop act-or-repeat selection for each action dimension independently, then draws new actions only for the dimensions flagged 'act'. The reported result is that this decoupling improves sample efficiency, final policy return, and action smoothness on standard control benchmarks, including a Humanoid with 17 action dimensions.","feed_headline":"Per-joint repeat-or-act choice beats whole-action repetition","feed_subtitle":"A two-stage policy selects repeat or act for every joint separately, raising sample efficiency and cutting action fluctuation.","key_machinery":"The central object is the per-dimension selection mask $b\\in\\{0,1\\}^{|A|}$ produced by the selection policy $\\beta(b|s,a^-)$, together with the Mix operation $a=(1-b)\\odot a^-+b\\odot\\hat a$. The mask turns action repetition into a differentiable, state-dependent masking problem: repetition coordinates copy the previous action, while 'act' coordinates take fresh samples from the action policy. The two policies are trained by entropy-regularized critic maximization, with an exact summation over masks for small action spaces and an importance-sampled surrogate for high-dimensional ones.","core_discovery":"SDAR replaces the single act-or-repeat decision over the whole action vector with per-dimension Bernoulli decisions. Given state $s$ and previous action $a^-$, the selection policy $\\beta(b|s,a^-)$ samples a binary mask $b\\in\\{0,1\\}^{|A|}$; the action policy $\\pi(\\hat a|s,a^-,b)$ then generates candidates only for coordinates with $b_i=1$, and the final action is $a=(1-b)\\odot a^-+b\\odot\\hat a$. This 'select first, act second' pipeline lets some joints persist while others update. The paper's experiments report that SDAR obtains higher normalized AUC (area under the learning curves) than SAC, N-Rep, TempoRL, UTE, and TAAC across classic control, locomotion, and manipulation, together with higher episode return and lower action fluctuation on representative tasks.","pith_inferences":["Extending the paper's logic, grouping correlated action dimensions (for example, all joints of one leg) could shrink the $2^{|A|}$ selection space while keeping most of the flexibility; the conclusion points to this as future work.","A per-dimension value or advantage estimate could replace the importance-sampled surrogate in Equation (9), scaling to high-dimensional action spaces while sidestepping the old-policy entropy term in the published update.","The decoupling mechanism transfers naturally to real robots whose actuators run at different control frequencies; the LunarLander visualization suggests SDAR would learn a different repeat cadence per actuator."],"forward_implications":["If SDAR's reported gains hold, closed-loop repetition should be implemented at actuator granularity rather than on the whole action vector, especially on agents with many heterogeneous joints.","The two-stage policy is a drop-in modification for off-policy actor-critic methods: it only needs the previous action stored in the replay buffer, so sample efficiency can come without environment-specific repetition schedules.","On tasks like Humanoid where whole-vector methods slow down late in training, per-dimension masks preserve final-policy agility instead of trading it away for persistence.","The higher action persistence rate and lower action fluctuation rate reported together suggest that persistence and smoothness can improve simultaneously once repetition is spatially decoupled."],"supporting_citations":[{"why":"Supplies the base SAC objective and automatic temperature tuning that the entropy terms in Eq. (7) and Eq. (10) build on.","marker":"(Haarnoja et al., 2018)"},{"why":"Defines TAAC, the closed-loop whole-action repetition baseline whose aggregate decision SDAR is designed to outperform.","marker":"(Yu et al., 2021)"},{"why":"Defines TempoRL, an open-loop skip-policy baseline representing inflexible repetition in the comparison.","marker":"(Biedenkapp et al., 2021)"},{"why":"Defines UTE, an uncertainty-aware open-loop repetition baseline SDAR compares against.","marker":"(Lee et al., 2024)"},{"why":"Introduces policy inertia to reduce action oscillations, the metric family SDAR's AFR claim extends.","marker":"(Chen et al., 2021)"},{"why":"Introduces dynamic action repetition, the repetition lineage SDAR generalizes to spatially decoupled masks.","marker":"(Lakshminarayanan et al., 2017)"}],"fun_headline_variants":["Per-axis repeat-or-act beats whole-vector repetition","Spatially decoupled repetition: each joint decides to persist or update","Select before act: per-joint repetition beats whole-action","Per-dimension act-or-repeat: more flexible, more efficient","Each joint picks repeat or act: SDAR boosts sample efficiency"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that Equation (9) optimizes the same entropy-regularized objective as Equation (7); as printed, the entropy term uses the old policy's log-probability rather than the new one's, so if that is not a typo the high-dimensional training objective differs from what the derivation claims.","fun_headline_variants_meta":{"raw":{"variants":["Per-axis repeat-or-act beats whole-vector repetition","Spatially decoupled repetition: each joint decides to persist or update","Select before act: per-joint repetition beats whole-action","Per-dimension act-or-repeat: more flexible, more efficient","Each joint picks repeat or act: SDAR boosts sample efficiency"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001032,"raw_usage":{"total_tokens":4319,"prompt_tokens":892,"completion_tokens":3427,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":508,"completion_tokens_details":{"reasoning_tokens":3356}},"tokens_in":508,"tokens_out":3427,"duration_ms":21075,"temperature":1.0,"reasoning_tokens":3356,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T14:53:54.655049+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare Humanoid training with Equation (9) exactly as printed against a version with the entropy term corrected to $\\log \\beta(b|s,a^-)$ inside the expectation; differing curves would show the published update rule is not what produced the reported results. As a complementary check, run SDAR on a locomotion task with tightly synchronized joints and show that per-dimension masks still beat whole-vector repetition when the selection-action loop is otherwise identical.","supporting_citations":[],"review_version":1}