ZAPS-DA: Zero-Phase Action Policy Smoothing with Decoupled Actor for Continuous Control in Reinforcement Learning

Faiq Shamass

REVIEW 4 major objections 3 minor 13 references

ZAPS-DA deploys a separate actor that imitates zero-phase-filtered actions, cutting control jitter by an order of magnitude with negligible phase lag and no inference filter.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A decoupled actor imitating zero-phase-filtered replay targets cuts continuous-control action jitter 14–21× with negligible phase lag and no inference-time filter.

T0 review reviewed 2026-07-12 challenge →

load-bearing objection Abstract-only package for a practical RL smoothing method with strong reported effect sizes; full ZAPS-DA manuscript is still missing, so the core claim is un-auditable. the 4 major comments →

arxiv 2605.30612 v2 pith:HH5IIORU submitted 2026-05-28 cs.RO cs.LGcs.SYeess.SY

ZAPS-DA: Zero-Phase Action Policy Smoothing with Decoupled Actor for Continuous Control in Reinforcement Learning

Faiq Shamass This is my paper

classification cs.RO cs.LGcs.SYeess.SY

keywords reinforcement learningcontinuous controlaction smoothingzero-phase filterdecoupled actorSoft Actor-Criticpolicy distillationrobotics

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Off-policy continuous-control policies often output high-frequency action jitter that physical actuators cannot use. Post-hoc filters add phase lag, and smoothness penalties inside the actor loss mix reward learning with over-smoothing. ZAPS-DA keeps the main RL actor untouched and trains a second, decoupled actor by supervised imitation of zero-phase-filtered action targets stored in the replay buffer. Only that decoupled actor is deployed: a plain feed-forward map from observation to smooth action, with no filter and no action history at inference time. On two driving simulators the method reduces steering jitter by roughly an order of magnitude while preserving task success, and it beats a carefully re-tuned penalty baseline without any per-environment smoothness tuning.

Core claim

A feed-forward policy trained only to imitate non-causal zero-phase-filtered action targets from the replay buffer yields a deployable controller that is dramatically smoother than the original RL actor, yet retains task performance and incurs essentially no phase lag, because the non-causal filter is distilled into a causal observation-to-action map.

What carries the argument

The decoupled actor: a second network trained by magnitude-matched MSE imitation of zero-phase (Savitzky–Golay) filtered actions stored in the replay buffer, while the main actor continues to optimize the ordinary RL loss; only the decoupled actor is used at deployment.

Load-bearing premise

A pure observation-to-action network can faithfully copy the smoothness of a non-causal filter without needing past actions and without materially harming the task policy learned by the main actor.

What would settle it

On the same MetaDrive and Webots setups, measure whether the decoupled actor’s action spectrum and closed-loop phase lag match a true zero-phase filter while task success stays statistically indistinguishable from the unfiltered RL baseline; any large drop in success or reappearance of high-frequency content would refute the claim.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

Physical robots can run the learned policy directly without online low-pass filters or action-history inputs.
Smoothness can be added after an RL policy is already trained, because the decoupled actor is trained from the existing replay buffer.
The same magnitude-matched imitation loss works across optimizers without re-tuning a smoothness coefficient per environment.
Penalty-based methods such as CAPS remain substantially less effective even when their hyperparameters are re-tuned per task.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same distillation idea could be applied to other non-causal post-processors (e.g., bidirectional smoothers or offline trajectory optimizers) to obtain causal deployable policies.
If the observation distribution of the main actor drifts far from the buffer used for imitation, the decoupled actor may silently degrade; a lightweight online monitor of imitation residual would be a natural safeguard.
Hardware trials on real steering and throttle actuators would test whether the measured jitter reduction survives actuator dynamics and sensor delay.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Abstract-only package for a practical RL smoothing method with strong reported effect sizes; full ZAPS-DA manuscript is still missing, so the core claim is un-auditable.

read the letter

The one thing you need to know: we still only have the ZAPS-DA abstract. The body that was supplied is CacheProbe (prompt-cache isolation), not this paper. So I cannot check methods, filter construction, training schedule, ablations, or the CAPS re-tuning protocol. Everything below is abstract-level.

What looks new and useful is the recipe itself. Keep the main SAC actor on the usual RL loss; train a second, decoupled actor by supervised imitation of zero-phase (Savitzky–Golay) filtered actions already sitting in the replay buffer; deploy only the decoupled actor as a pure observation→action map. No inference-time filter, no action history, and a magnitude-matched MSE that is meant to be portable without per-optimizer retuning. That is a clean separation of reward learning from smoothness, and it is a real alternative to post-hoc filtering (phase lag) and to CAPS-style penalties (coupled gradients, environment-specific weights).

The reported numbers, if they hold, are large: MetaDrive paired n=150, 14–21× steering and 3–5× throttle jitter cuts with task-completion parity and ~6% reward cost; Webots ACC reward parity plus 8–45× steering jitter and lower failure rate; versus re-tuned CAPS a ~4.6× gap on the best steering-jitter reduction, with no per-environment smoothness-signal tuning and post-hoc applicability. The abstract also flags Bonferroni and multiple p-values, which is the right posture for that many comparisons.

Soft spots are almost all verification blockers, not demonstrated technical holes. The load-bearing assumption—that causal distillation of non-causal filtered targets from observation alone stays smooth and task-competent under the same observation distribution—cannot be checked without the body. Free parameters (SG window/order, any residual MSE scale, base SAC/env settings) are also invisible. Circularity risk looks ordinary for an empirical methods paper, not structural.

Who it is for: people who ship continuous-control policies on actuators or in driving sims and who are tired of either laggy filters or CAPS-style retuning. If the full paper matches the abstract’s care, it deserves a serious referee. Right now I would not cite it or bring it to reading group until the correct manuscript appears. Send it to peer review only after the real PDF is in hand; the abstract alone is not enough to desk-accept, but the claimed contribution is important enough that a complete version should not be desk-rejected.

Referee Report

4 major / 3 minor

Summary. The submission claims that ZAPS-DA reduces high-frequency action jitter in continuous-control off-policy RL by pairing an unmodified main actor (base RL loss) with a separate decoupled actor trained by supervised imitation of zero-phase-filtered action targets stored in the replay buffer. The deployed policy is the feed-forward decoupled actor alone (observation→smooth action), with no inference-time filter and no action-history input—i.e., causal distillation of a non-causal filter—using a magnitude-matched MSE that is presented as zero-hyperparameter portable. With SAC and a Savitzky–Golay filter, the abstract reports large jitter reductions on MetaDrive (steering 14–21×, throttle 3–5×; paired n=150, Bonferroni p<10^{-4}) at matched task completion and ~6.3% reward cost, a Pareto improvement on a Webots ACC task, and a ~4.6× gap versus re-tuned CAPS on steering jitter with no per-environment smoothness-signal tuning and post-hoc applicability.

Significance. If the mechanism and statistics hold under full methods scrutiny, the work would be a practically useful contribution to continuous-control RL: it targets a known deployment barrier (actuator-infeasible jitter), separates smoothness from the RL gradient (unlike penalty methods such as CAPS), claims negligible phase lag without post-processing, and reports strong paired statistics plus a re-tuned baseline comparison. The post-hoc applicability claim and zero-hyperparameter portability of the imitation loss would further increase engineering value. Those strengths cannot be credited as established until the architecture, filter/target construction, training schedule, and CAPS re-tuning protocol are inspectable.

major comments (4)

The full manuscript body supplied for this review is not ZAPS-DA (arXiv:2605.30612). It is an unrelated security paper (CacheProbe / prompt-cache isolation). Only the ZAPS-DA abstract is available. Without methods, equations, filter/target construction, decoupled-actor training schedule, observation/action spaces, seed protocol, CAPS re-tuning details, or ablations, the central mechanism claim—that a feed-forward observation→action map trained by MSE on non-causal zero-phase-filtered replay targets yields smooth, low-lag actions at deployment without action history or material task degradation—cannot be audited. This is a verification blocker for every quantitative claim (n=150 paired tests, Bonferroni p-values, 14–21× / 8–45× jitter factors, CAPS gap).
Abstract core mechanism: the weakest load-bearing assumption is that causal distillation of a non-causal (zero-phase) filter generalizes under the same observation distribution as the main actor, without reintroducing phase lag or needing action-history input. That claim requires explicit construction of the filtered targets in the buffer, the imitation objective (including how magnitude-matched MSE is defined and scaled), any freeze/update schedule relative to the main SAC actor, and evidence that the deployed map does not merely average or lag. None of this is present in the material available for review.
Abstract CAPS comparison: the reported ~4.6× gap (14.7× vs CAPS best 3.2× steering-jitter reduction) is load-bearing for superiority over the standard penalty baseline. It depends on per-environment re-tuning of CAPS penalty weight, spatial noise, and entropy coefficient at both auto-entropy and fixed-entropy operating points, with matched seeds. Without the re-tuning grid, selected hyperparameters, and full metric tables, the gap cannot be verified or attributed to the decoupling design rather than under-tuned CAPS or environment-specific filter settings.
Abstract free parameters: Savitzky–Golay window and polynomial order, residual imitation scaling (if any), and base SAC/environment hyperparameters remain free. The claim of “zero-hyperparameter portability across optimizer classes” for the magnitude-matched MSE needs a precise definition and cross-optimizer evidence; without it, portability is an untested assertion rather than a demonstrated property.

minor comments (3)

Abstract only: “anchor protocol” on MetaDrive and the custom Webots ACC task definition are undefined; they should be specified with observation/action dimensions and success/failure criteria once the full methods are restored.
Abstract only: jitter metric definition (e.g., high-frequency energy, successive-difference variance) is not stated; it is needed to interpret the 14–21× and 8–45× factors.
Abstract only: “post-hoc applicability to trained policies” is an important claim and should be isolated as an experiment (freeze main actor, train only decoupled actor on existing buffer) with the same paired statistics.

Circularity Check

0 steps flagged

No derivation-level circularity: ZAPS-DA is an empirical methods claim measured by jitter/reward/task metrics, not a first-principles prediction forced by its own inputs.

full rationale

Only the ZAPS-DA abstract is available for the claimed paper (arXiv:2605.30612); the cached full-text body is a different manuscript (CacheProbe). On the available material, the central claim is experimental: a decoupled actor trained by supervised imitation of zero-phase-filtered replay targets is deployed as a feed-forward observation→action map and evaluated by measured steering/throttle jitter, reward, and task-completion rates against SAC and re-tuned CAPS. That is a training-and-measurement procedure, not a derivation chain in which a fitted parameter or self-defined quantity is renamed as a prediction. There is no equation that reduces a claimed first-principles result to its own inputs by construction, no uniqueness theorem imported from the authors, and no ansatz smuggled in as a forced mathematical fact. Residual experimental risks (same-env tuning, stationarity of filtered targets, sufficiency of observation alone) are ordinary validity concerns, not circularity of the kind this pass audits. Score 0; steps empty.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 1 invented entities

Abstract-only review of an empirical RL method. Load-bearing background is standard off-policy actor-critic practice, zero-phase filtering (Savitzky–Golay), and supervised imitation. Free parameters of the smoothness pipeline (filter window/order, imitation loss scaling) are claimed to be largely eliminated by magnitude-matched MSE, but filter choice and RL hyperparameters remain. No new physical entities are postulated.

free parameters (3)

Savitzky–Golay filter hyperparameters (window, polynomial order)
Abstract fixes SG as the zero-phase filter but does not state window/order; these shape the imitation targets and thus the deployed policy.
Imitation / magnitude-matched MSE scaling (if any residual scale)
Paper claims zero-hyperparameter portability via magnitude-matched MSE; any residual scale or target normalization still acts as a free choice if present in the full method.
Base SAC / environment hyperparameters
Main actor is trained by the base RL loss; entropy, learning rates, network size, and env reward scales affect both reward and the actions being filtered.

axioms (3)

domain assumption Zero-phase (non-causal) filtering of stored actions yields desirable smooth targets that a feed-forward policy can imitate from observations alone.
Core mechanism of ZAPS-DA; if imitation from o_t cannot recover the filtered a_t distribution, jitter reduction or task performance fails.
ad hoc to paper Decoupling smoothness into a supervised actor does not re-couple reward regression with over-aggressive smoothing the way an actor-loss penalty does.
Stated motivation versus CAPS-style penalties; empirical claim rather than theorem.
domain assumption Standard off-policy actor-critic (SAC) training and replay-buffer storage are valid for continuous control in the tested driving simulators.
Background RL practice assumed throughout.

invented entities (1)

Decoupled actor (ZAPS-DA) no independent evidence
purpose: Separate policy trained only by imitation of zero-phase-filtered actions; becomes the deployed map o→a.
Architectural construct of the method; not a physical entity. Independent evidence would be third-party replications or hardware deployment results (not available in abstract).

reviewed 2026-07-12 · how reviews work

0 comments

Cite this review

Pith. "Pith review of ZAPS-DA: Zero-Phase Action Policy Smoothing with Decoupled Actor for Continuous Control in Reinforcement Learning." pith.science (2026). https://pith.science/paper/HH5IIORU

@misc{pith2026260530612,
  author       = {Pith},
  title        = {Pith review of: ZAPS-DA: Zero-Phase Action Policy Smoothing with Decoupled Actor for Continuous Control in Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HH5IIORU}},
  note         = {Machine review of arXiv:2605.30612}
}

Share X Bluesky LinkedIn Reddit HN

abstract

Continuous control policies trained with off-policy reinforcement learning frequently exhibit high-frequency action jitter, impractical for direct deployment on physical actuators. Post-hoc filtering attenuates jitter but adds phase lag; embedding smoothness penalties in the actor's loss couples them with the RL gradient and conflates reward regression with over-aggressive smoothing. We present ZAPS-DA, which reduces action jitter at deployment with negligible phase lag and no post-processing. ZAPS-DA pairs an unmodified main actor (trained by the base RL loss) with a separate decoupled actor trained via supervised imitation of zero-phase filtered targets stored in the replay buffer. The deployed policy is the decoupled actor: a feed-forward map from observation to smooth action, with no inference-time filter and no action-history input -- causal distillation of a non-causal filter. A magnitude-matched MSE loss gives zero-hyperparameter portability across optimizer classes. Validated with Soft Actor-Critic and a Savitzky-Golay filter in two driving simulators (paired n=150): on MetaDrive (anchor protocol), ZAPS-DA cuts steering jitter 14-21x and throttle jitter 3-5x (all $p<10^{-4}$, Bonferroni) while matching task-completion (p=0.28 success, p=0.31 crash) at 6.3% reward cost; on a custom Webots adaptive cruise control task, the same configuration yields a Pareto improvement -- reward parity (p=0.121), 8-45x steering-jitter reduction, task-failure rate 2.0% to 0.7%. Against CAPS, the standard penalty-based baseline -- at both its auto-entropy and native fixed-entropy operating points, with penalty weight, spatial noise, and entropy coefficient re-tuned per environment -- ZAPS-DA reaches 14.7x steering-jitter reduction versus CAPS's best 3.2x at matched seeds, a ~4.6x gap, with no per-environment tuning of the smoothness signal and post-hoc applicability to trained policies.

Figures

Figures reproduced from arXiv: 2605.30612 by Faiq Shamass.

**Figure 1.** Figure 1: Steering command trace over a 60-step (6 s) window [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Reference graph

Works this paper leans on

13 extracted references · 1 linked inside Pith

[1]

11em plus .33em minus .07em @technote 4000 4000 100 4000 4000 500 `\.=1000 = #1 #1 #1 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEauthorblockAstyle \@IEEEauthordefaulttextstyle \@IEEEauthorblockconfadjspace -0.25em \@IEEEauthorblockNtopspace 0.0ex \@IEEEauthorblockAtopspace 0.0ex \@IEEEauthorblockNinterlinespace 2.6ex \@IEEEauthorblockAinte...

2008
[2]

Mysore, B

S. Mysore, B. Mabsout, R. Mancuso, and K. Saenko, ``Regularizing Action Policies for Smooth Control with Reinforcement Learning,'' IEEE ICRA, 2021

2021
[3]

S. Ross, G. Gordon, and D. Bagnell, ``A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning,'' AISTATS, 2011

2011
[4]

Ho and S

J. Ho and S. Ermon, ``Generative Adversarial Imitation Learning,'' NeurIPS, 2016

2016
[5]

Haarnoja, A

T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, and S. Levine, ``Soft Actor-Critic Algorithms and Applications,'' arXiv:1812.05905, 2018

Pith/arXiv arXiv 2018
[6]

Savitzky and M

A. Savitzky and M. J. E. Golay, ``Smoothing and Differentiation of Data by Simplified Least Squares Procedures,'' Analytical Chemistry, vol. 36, no. 8, pp. 1627--1639, 1964

1964
[7]

Hinton, O

G. Hinton, O. Vinyals, and J. Dean, ``Distilling the Knowledge in a Neural Network,'' NeurIPS Workshop, 2015

2015
[8]

Stooke, J

A. Stooke, J. Achiam, and P. Abbeel, ``Responsive Safety in Reinforcement Learning by PID Lagrangian Methods,'' ICML, 2020

2020
[9]

Q. Li, Z. Peng, L. Feng, Q. Zhang, Z. Xue, and B. Zhou, ``MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning,'' IEEE T-PAMI, vol. 45, no. 3, pp. 3461--3475, 2023

2023
[10]

Michel, ``Cyberbotics Ltd.\ Webots: Professional Mobile Robot Simulation,'' Journal of Advanced Robotics Systems, vol

O. Michel, ``Cyberbotics Ltd.\ Webots: Professional Mobile Robot Simulation,'' Journal of Advanced Robotics Systems, vol. 1, no. 1, pp. 39--42, 2004

2004
[11]

Hausknecht and P

M. Hausknecht and P. Stone, ``Deep Recurrent Q-Learning for Partially Observable MDPs,'' AAAI, 2015

2015
[12]

Andrychowicz et al., ``Learning Dexterous In-Hand Manipulation,'' IJRR, vol

M. Andrychowicz et al., ``Learning Dexterous In-Hand Manipulation,'' IJRR, vol. 39, no. 1, pp. 3--20, 2020

2020
[13]

Siciliano, L

B. Siciliano, L. Sciavicco, L. Villani, and G. Oriolo, Robotics: Modelling, Planning and Control. Springer, 2009

2009

This paper was first reviewed by grok-4.5 on July 12, 2026.

[1] [1]

11em plus .33em minus .07em @technote 4000 4000 100 4000 4000 500 `\.=1000 = #1 #1 #1 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEauthorblockAstyle \@IEEEauthordefaulttextstyle \@IEEEauthorblockconfadjspace -0.25em \@IEEEauthorblockNtopspace 0.0ex \@IEEEauthorblockAtopspace 0.0ex \@IEEEauthorblockNinterlinespace 2.6ex \@IEEEauthorblockAinte...

2008

[2] [2]

Mysore, B

S. Mysore, B. Mabsout, R. Mancuso, and K. Saenko, ``Regularizing Action Policies for Smooth Control with Reinforcement Learning,'' IEEE ICRA, 2021

2021

[3] [3]

S. Ross, G. Gordon, and D. Bagnell, ``A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning,'' AISTATS, 2011

2011

[4] [4]

Ho and S

J. Ho and S. Ermon, ``Generative Adversarial Imitation Learning,'' NeurIPS, 2016

2016

[5] [5]

Haarnoja, A

T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, and S. Levine, ``Soft Actor-Critic Algorithms and Applications,'' arXiv:1812.05905, 2018

Pith/arXiv arXiv 2018

[6] [6]

Savitzky and M

A. Savitzky and M. J. E. Golay, ``Smoothing and Differentiation of Data by Simplified Least Squares Procedures,'' Analytical Chemistry, vol. 36, no. 8, pp. 1627--1639, 1964

1964

[7] [7]

Hinton, O

G. Hinton, O. Vinyals, and J. Dean, ``Distilling the Knowledge in a Neural Network,'' NeurIPS Workshop, 2015

2015

[8] [8]

Stooke, J

A. Stooke, J. Achiam, and P. Abbeel, ``Responsive Safety in Reinforcement Learning by PID Lagrangian Methods,'' ICML, 2020

2020

[9] [9]

Q. Li, Z. Peng, L. Feng, Q. Zhang, Z. Xue, and B. Zhou, ``MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning,'' IEEE T-PAMI, vol. 45, no. 3, pp. 3461--3475, 2023

2023

[10] [10]

Michel, ``Cyberbotics Ltd.\ Webots: Professional Mobile Robot Simulation,'' Journal of Advanced Robotics Systems, vol

O. Michel, ``Cyberbotics Ltd.\ Webots: Professional Mobile Robot Simulation,'' Journal of Advanced Robotics Systems, vol. 1, no. 1, pp. 39--42, 2004

2004

[11] [11]

Hausknecht and P

M. Hausknecht and P. Stone, ``Deep Recurrent Q-Learning for Partially Observable MDPs,'' AAAI, 2015

2015

[12] [12]

Andrychowicz et al., ``Learning Dexterous In-Hand Manipulation,'' IJRR, vol

M. Andrychowicz et al., ``Learning Dexterous In-Hand Manipulation,'' IJRR, vol. 39, no. 1, pp. 3--20, 2020

2020

[13] [13]

Siciliano, L

B. Siciliano, L. Sciavicco, L. Villani, and G. Oriolo, Robotics: Modelling, Planning and Control. Springer, 2009

2009

ZAPS-DA: Zero-Phase Action Policy Smoothing with Decoupled Actor for Continuous Control in Reinforcement Learning

The reading

Core claim §

What carries the argument

Load-bearing premise §

What would settle it §

If this is right

Where Pith is reading between the lines

discussion (0)

Core claim

Load-bearing premise

What would settle it