{"id":"b944573e-2a8c-46e7-9f2d-37c3c54fb56b","arxiv_id":"2502.02265","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"AAC wraps Soft Actor-Critic with a hand-tuned PID 'Adviser' that emits a fake error signal, claiming to eliminate steady-state error, but the proof is flawed and experiments lack statistics.","lead":"This paper adds a PID controller, called an Adviser, on top of a reinforcement learning agent to reduce steady-state error in goal-reaching control tasks. The authors report improved precision on simulated robotic arms and a real quadcopter, but the supporting theory and experiments are not strong enough to back the 'eliminating error' claim.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Appendix A's proof that fake goals eliminate steady-state error is invalid: Eq. 15 requires an unstated A(0)=I assumption and Eq. 18 reverses the spectral-radius inequality, so the title's guarantee is unsupported.","rationale":"The central claim of the paper is that the Adviser mechanism eliminates steady-state error in RL control, as stated in the title and abstract. The only theoretical argument for this claim is Appendix A's derivation of Eqs. 15–19. That derivation is invalid: Eq. 15 silently assumes A(0)=I, and Eq. 18 reverses the standard spectral-radius inequality. These are not cosmetic typos; they are exactly the steps that turn 'the actor has a steady-state error' into 'the fake goal reduces that error.' Without them, the paper has no proof of the title's guarantee. The experimental sections provide qualitative evidence that the adviser helps in three specific tasks, including a real quadcopter roll-angle trial, but the figures show single trajectories or aggregated curves without repeated-seed error bars, so they cannot independently establish a general 'elimination' result. The method may be salvageable—for example, by proving contraction in a norm tailored to ρ(I-B), or by explicitly acknowledging that the adviser implements integral feedback and relying on classical PID analysis—but as written the central claim is unsupported. This is precisely the reader's main objection, and I agree with the rejection verdict.","tokens_in":12600,"tokens_out":8862,"duration_ms":78065,"concrete_test":"Symbolically re-derive Eq. 15 for A(s)=s^2 - a1 s - a0 (the scalar version of Eq. 6) with general B, and verify that ess,now=(I-B)ess,prev requires A(0)=I. Separately, test the contraction claim numerically with a matrix I-B whose spectral radius is <1 but whose 2-norm is >1 (e.g., [[0.9,10],[0,0.9]], so B=[[0.1,-10],[0,0.1]]): compute ||(I-B)e||_2 for a random unit e and compare to ||e||_2; the proof's Eq. 19 predicts a decrease, but the inequality in Eq. 18 does not guarantee it. If either check fails, the Appendix A guarantee is invalid.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central guarantee—that the Adviser's fake goals eliminate steady-state error—rests entirely on Appendix A, and that proof is mathematically invalid in two independent places.\n\nFirst, Eq. 15 defines ess,now = lim_{s→0} s A(s)^{-1}[B(U(s) - ess,prev/s) + D(s)]. Carrying out linearity gives ess,now = ess,prev - lim_{s→0} A(s)^{-1} B ess,prev. The paper silently replaces lim_{s→0} A(s)^{-1}B by B and concludes ess,now = (I-B) ess,prev. This replacement is valid only if A(0)=I (or at least A(0)^{-1}B=B). But from Eq. 11, A(s) has entries s^k + Σ_{l=0}^{k-1} a_{ijl} s^l, so A(0) is the matrix of zeroth-derivative coefficients—generally not I, and the paper never states or argues this. For the decoupled second-order model of Eq. 6, the scalar transfer function is A(s)=s^2 - a1 s - a0, so A(0)=-a0, which is 1 only for a0=-1. Thus the identity ess,now=(I-B)ess,prev does not follow.\n\nSecond, Eq. 18 asserts ρ(I-B) ≥ ||I-B||_2 for any compatible matrix norm. The standard property is reversed: the spectral radius is a lower bound for every matrix norm, ρ(A) ≤ ||A|| for all norms. The inequality in the paper is backwards, and Eq. 19's chain ||·||_2 ≤ ρ(I-B) requires the opposite. Even with ρ(I-B)<1, the 2-norm can exceed 1 (e.g., B=[[0.1,-10],[0,0.1]] gives ρ(I-B)=0.9 but ||I-B||_2≈10.1). The claimed contraction in the 2-norm is therefore not established.\n\nBecause these two steps are the only theoretical support for the title's 'eliminating' rather than 'reducing,' the central claim is unsupported. The empirical plots may show error reduction in specific tasks, but without corrected theory or repeated-seed statistics, they cannot carry the general guarantee.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Adviser-Actor-Critic (AAC), a framework that augments an actor-critic RL agent with a PID-based 'adviser' that generates fake goals and synthetic error signals. The authors claim this eliminates steady-state error in goal-conditioned control tasks, and support the claim with a Routh-Hurwitz stability analysis and an appendix proof that iterating fake-goal updates contracts the steady-state error. They evaluate AAC against SAC and SAC+HER on three environments—a mass-spring-damper system, a 3-DoF robotic arm, and a quadcopter—and also deploy the trained policy on a physical quadcopter.","tokens_in":13132,"tokens_out":7159,"duration_ms":62380,"significance":"If the central guarantee were valid, combining PID-based guidance with RL would be a practically useful way to improve precision in goal-conditioned control. The paper's strengths include a real-world quadcopter experiment and released code. However, the theoretical proof that fake goals eliminate steady-state error is invalid: it relies on an unstated A(0)=I assumption and reverses the standard spectral-radius-norm inequality. As a result, the title claim of 'eliminating' steady-state error is unsupported, and the contribution reduces to an empirical observation that a PID-style signal can improve RL performance—already expected from classical control theory.","major_comments":[{"comment":"The derivation of ess,now = (I - B) ess,prev equates lim_{s->0} A(s)^{-1} B with B. This is valid only if A(0) = I or B has special structure; neither is stated. For the second-order model of Eq. (6), the scalar A(s) is s^2 - a1 s - a0, so A(0) = -a0, which is not 1 in general. Consequently the claimed contraction does not follow from the stated assumptions.","section":"Appendix A, Eq. (15)"},{"comment":"The inequality rho(I - B) >= ||I - B||_2 is reversed; the standard property is rho(M) <= ||M|| for every matrix norm. The chain in Eq. (19) therefore does not establish ||ess,now||_2 < ||ess,prev||_2, even under rho(I - B) < 1. A concrete counterexample is B = [[0.1, -10], [0, 0.1]], for which rho(I - B) = 0.9 but ||I - B||_2 is approximately 10.1.","section":"Appendix A, Eq. (18)"},{"comment":"The stability analysis and the fake-goal contraction argument assume the system dynamics are decoupled into independent second-order per-dimension equations. The paper asserts this decoupling but provides no justification for the MIMO environments (robotic arm, quadcopter) in which it is evaluated. Without a validated decoupling assumption, the theoretical result cannot be claimed to apply to the experimental systems.","section":"Sec. 4.4 and Appendix A"},{"comment":"The algorithmic mechanism is underspecified. The paper defines the adviser's synthetic error epsilon(t) in Eq. (5) but never states how this signal enters the actor's policy or how the fake goal is constructed from epsilon. The experimental section provides only parameter tables, not the policy update rule. This makes the method difficult to reproduce from the text and prevents the reader from determining whether the reported gains are due to the adviser mechanism or to other implementation details.","section":"Sec. 4.3 and Figure 2"}],"minor_comments":[{"comment":"The phrase 'Code are available' should be 'Code is available'.","section":"Abstract"},{"comment":"Equation (19) uses ess,new while Eq. (15) defines ess,now; the notation should be consistent.","section":"Eqs. (15) and (19)"},{"comment":"The table provides adviser hyperparameters only for the robotic arm; the corresponding settings for the mass-spring-damper and quadcopter experiments are missing.","section":"Table 4"},{"comment":"The figure reports cumulative reward and steady-state error without error bars or multiple-seed statistics; the paper does not state how many random seeds were run.","section":"Figure 7"},{"comment":"The phrase 'command tracking' is not formally defined beyond a heuristic description; the paper should give a precise mathematical condition.","section":"Appendix A"},{"comment":"The relation between the gains K'_p and K'_d used in Eq. (7) and the actual PID gains K_p, K_d, K_i in Eq. (5) is explained only in the caption of Figure 3 and not in the main text; this connection should be made explicit in the body of the paper.","section":"Figure 3 caption"}],"recommendation":"reject","confidential_remarks":"The central proof errors are not local typos; they invalidate the paper's main theoretical claim. I would not consider a minor revision. The empirical study might be suitable for a different venue that does not require a theoretical guarantee, but as submitted the contribution rests on an unproven 'elimination' claim. I have no concerns about citation patterns or novelty disclosure beyond the overstatement inherent in the title."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Name],\n\nQuick take: the idea is plausible but the paper oversells it. The theoretical proof that fake goals eliminate steady-state error is broken in two places, so the title's guarantee is unsupported. Still, there is a real empirical contribution here, and I would not desk-reject it.\n\nWhat's actually new: a PID-based 'adviser' that outputs a fake error signal to guide the actor, applied to goal-conditioned RL control. The paper is honest that this is 'a sophisticated alternative to integral feedback' (Tracey et al. 2024), and it does not pretend to be a completely novel mechanism. The Routh-Hurwitz stability analysis of the closed-loop third-order system is textbook and correct. The real-quadcopter experiment is a positive sign—we get actual hardware results with and without the adviser, and the curves show a meaningful difference.\n\nThe soft spots are serious. Appendix A is the only theoretical support for the 'eliminating' claim, and it fails twice. First, the move from lim_{s→0} A(s)^{-1} B ess,prev to B ess,prev silently assumes A(0)=I, which is not stated and does not hold for the second-order model in Eq. 6 (A(0)=-a0). Second, Eq. 18 asserts ρ(I-B) ≥ ||I-B||_2, but the standard inequality is ρ(A) ≤ ||A||. So the 2-norm contraction does not follow. The derivation also assumes decoupled per-dimension second-order dynamics, which is a big restriction for the MIMO systems the paper targets. On the experimental side, there are no error bars or repeated seeds, and no baseline with plain integral error in the observation—so we don't know if the adviser is better than the cheaper alternative it is meant to replace.\n\nNet: the core recipe is salvageable, but the paper as written does not prove 'eliminating' steady-state error. The experiments are suggestive, not conclusive. I'd send this to review because the problem is important and the hardware demo is real, but the authors should be told to fix or remove the proof, add statistics, and include the integrator-feedback baseline. It could become a decent paper after major revision.\n\nRecommendation: engage with it, but don't take the theory at face value.","headline":"The paper's central proof is broken in two places, so the 'eliminating' claim is unsupported, but the empirical recipe and real quadcopter demo make it a salvageable submission.","tokens_in":13642,"tokens_out":2657,"would_cite":false,"duration_ms":23758,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A PID controller acting as an 'adviser' can erase steady-state error in reinforcement learning control, letting actors hit precise goals.","keywords":["reinforcement learning","steady-state error","PID control","goal-conditioned control","actor-critic","fake goals","adviser mechanism","quadcopter control"],"falsifier":"Take the mass-spring-damper or quadcopter plant, measure its DC-gain matrix $A(0)$, and apply the adviser's fake-goal update while holding a constant setpoint. If $A(0)$ is not the identity or $\\rho(I - B)$ is not below 1, the steady-state error should fail to contract on each iteration, directly contradicting the paper's Eq. (19) guarantee.","tokens_in":1538,"feed_emoji":"🎯","tokens_out":2114,"duration_ms":65894,"temperature":0.7,"pith_summary":"This paper argues that reinforcement learning agents can achieve high-precision goal tracking by having a PID controller act as an 'adviser' that feeds a synthetic error signal into the actor's observations. The central claim is that this fake-goal mechanism reduces steady-state error far below what standard actor-critic methods achieve, without requiring a system model. The authors support the claim with experiments on a mass-spring-damper, a robotic arm, and a real quadcopter, where adviser-augmented SAC and SAC+HER show lower steady-state error and higher cumulative reward. If the claim holds, the framework offers a simple retrofit for existing actor-critic implementations to reach precise setpoints in robot control.","feed_headline":"PID adviser erases steady-state error in RL control","feed_subtitle":"An added PID adviser cuts persistent error in goal-conditioned tasks, from simulation to a real quadcopter.","key_machinery":"The Adviser is the load-bearing component: it is a PID controller that takes the tracking error $e = g_d - g_a$ and outputs a fake error $\\epsilon = -K_p e - K_i \\int e \\, d\\tau - K_d \\frac{de}{dt}$, appended to the actor's observation. The actor learns to act on this augmented input, which biases the policy toward the fake goal; the RL loop then corrects the bias online. Stability of the adviser is checked with the Routh–Hurwitz criterion on a per-dimension second-order model, giving the conditions $K'_d > 0$, $K_i > 0$, and $K'_p K'_d > K_i$. The error-reduction property is derived from the Laplace-domain equation $A(s)E(s) = B U(s) + D(s)$, leading to the contraction condition $\\rho(I - B) < 1$ for the steady-state error.","core_discovery":"The central discovery is that a PID controller used as an adviser can eliminate the residual steady-state error that actor-critic policies leave in goal-conditioned tasks. The adviser computes an error signal $\\epsilon$ from the current goal error plus proportional, integral, and derivative terms, and feeds $\\epsilon$ to the actor; the actor is thereby steered toward a 'fake goal' that compensates for its systematic bias. The paper formalizes the error dynamics of this setup and derives that, under stability and command-tracking conditions, the new steady-state error equals $(I - B)$ times the previous one, so the error contracts when the spectral radius $\\rho(I - B)$ is less than 1. In experiments, the adviser lowers steady-state error and raises cumulative reward for both SAC and SAC+HER across three environments, and a real quadcopter reaches a 5° roll target that the unaided policy misses.","pith_inferences":["The paper's proof of contraction requires $A(0)=I$, so for plants with a non-identity DC gain the same 'fake goal' update could fail or even increase error; a direct test is to measure the plant's DC-gain matrix before applying AAC.","Because the adviser acts only on the error, AAC should also reduce steady-state error caused by unmodeled disturbances, not just actor bias, as long as the disturbance enters the error dynamics the same way.","A natural extension the authors leave implicit is to make the adviser gains learnable: a meta-controller could tune $K_p, K_i, K_d$ online, which would generalize the framework beyond the fixed-gain experiments reported here."],"forward_implications":["With the adviser active, both SAC and SAC+HER show lower steady-state error and higher cumulative reward across all three test environments.","Training with the adviser active and evaluating with different gains still helps, meaning the adviser's parameters can be readjusted at deployment without retraining.","The method transfers to a physical quadcopter: adviser-guided roll control reaches the 5° target that the unaided policy misses.","Because the RL side of the framework is model-free, the adviser does not require a dynamics model of the plant, unlike MPC.","The adviser can be combined with replay strategies such as HER, so it is an additive modification rather than a replacement for existing goal-conditioned RL tools."],"supporting_citations":[{"why":"Provides the SAC algorithm that all experiments wrap with the adviser.","marker":"[Haarnoja et al., 2018]"},{"why":"Provides Hindsight Experience Replay, the goal-replay strategy combined with the adviser in the SAC+HER variants.","marker":"[Andrychowicz et al., 2017]"},{"why":"Defines the PID controller structure and gain roles (proportional, integral, derivative) that the adviser is built on.","marker":"[Li et al., 2006]"},{"why":"The integrator-feedback approach that AAC positions itself against for reducing steady-state error in RL control.","marker":"[Tracey et al., 2024]"}],"fun_headline_variants":["AAC's PID adviser cuts goal error to zero","PID mentor steers RL actor to exact goals","RL control precision fixed by PID adviser","Adviser-Actor-Critic zeroes steady-state RL error","PID adviser erases RL steady-state error"],"cache_read_input_tokens":15488,"weakest_assumption_plain":"For the claimed error reduction to hold, the plant must have no steady-state distortion at zero frequency and each axis must act independently; real coupled robots satisfy neither automatically.","fun_headline_variants_meta":{"raw":{"variants":["AAC's PID adviser cuts goal error to zero","PID mentor steers RL actor to exact goals","RL control precision fixed by PID adviser","Adviser-Actor-Critic zeroes steady-state RL error","PID adviser erases RL steady-state error"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000502,"raw_usage":{"total_tokens":2416,"prompt_tokens":872,"completion_tokens":1544,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":488,"completion_tokens_details":{"reasoning_tokens":1472}},"tokens_in":488,"tokens_out":1544,"duration_ms":9992,"temperature":1.0,"reasoning_tokens":1472,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T12:44:00.787646+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the mass-spring-damper or quadcopter plant, measure its DC-gain matrix $A(0)$, and apply the adviser's fake-goal update while holding a constant setpoint. If $A(0)$ is not the identity or $\\rho(I - B)$ is not below 1, the steady-state error should fail to contract on each iteration, directly contradicting the paper's Eq. (19) guarantee.","supporting_citations":[{"cited_title":"Hindsight experience replay","cited_arxiv_id":null,"evidence_quote":"Provides Hindsight Experience Replay, the goal-replay strategy combined with the adviser in the SAC+HER variants."},{"cited_title":"D., Michi, A., Chervonyi, Y., Davies, I., Paduraru, C., Lazic, N., Felici, F., Ewalds, T., Donner, C., Galperti, C., et al","cited_arxiv_id":null,"evidence_quote":"The integrator-feedback approach that AAC positions itself against for reducing steady-state error in RL control."}],"review_version":1}