{"id":"5d325092-8c65-45f9-8027-0d4fab7e9a47","arxiv_id":"2608.08598","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A two-stage, teacher-student reinforcement learning controller with privileged information and reward shaping outperforms a cascaded P-PID controller in simulated ROV station-keeping under time-varying currents.","lead":"The paper trains a reinforcement-learning controller that lets a remotely operated underwater vehicle hold its position and orientation under changing ocean currents, and tests it in simulation against a standard PID-based controller. It reports faster settling, smaller errors, less energy use, and smoother thruster commands, but only in simulation with no physical experiments.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline superiority claim rests on a single undocumented P-PID baseline; better-tuned or feedforward-enabled baseline could erase the reported margins.","rationale":"The paper is a competent simulation study: the two-stage distillation architecture is clearly described, the ablations support the role of privileged information and the multi-encoder design, and the reported inference latency (4.4 ms) is compatible with the 16 ms control period. These positives are real. The central claim, however, is comparative: TSRCA-PPO 'consistently outperforms' cascaded P-PID on all metrics. For that claim to hold, the baseline must be a fair representative of conventional practice. That condition is the least secure link in the argument. The baseline gains, tuning procedure, anti-windup, and feedforward terms are absent from Section 5.2, and the reported P-PID settling time of 28.55 s with 72.8% success in a 32 s window suggests many episodes never reach the settling threshold. A better-tuned baseline or one with current feedforward would probably reduce the margins, especially in settling time and force smoothness. A secondary concern is that the training reward uses the same energy proxy (Eq. 37/43) and smoothness-related penalties as the evaluation metrics, so the RL method is partly optimizing the evaluation instrument itself; this makes the absolute magnitude of the energy/smoothness advantages less informative, though it does not invalidate the comparison. The reader's weakest_assumption identified the same load-bearing concern, and I agree with the CONDITIONAL verdict. The condition should include baseline transparency and a systematic recalibration test, plus ideally code/data release. I do not see a reason to move the verdict to REJECT, because the central claim is conditional on the baseline and can be substantiated or narrowed by a concrete rerun.","tokens_in":16030,"tokens_out":2818,"duration_ms":34000,"concrete_test":"Run a controlled baseline-recalibration experiment: use the same 1,000-episode protocol and Table 8 metrics, but replace the single P-PID configuration with a systematically tuned cascade (e.g., grid/random search over outer-loop proportional gain and inner-loop PID gains, with anti-windup and derivative filtering), and add an optional current-velocity feedforward using the known Gauss-Markov current model or an estimator. Report the tuned gains and rerun; if the margins in settling time and J_FS shrink materially (e.g., settling-time ratio rises above 50% or smoothness margin falls below 50%), the central claim should be restated as 'outperforms the as-tuned baseline' rather than 'conventional P-PID.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.2 (Table 8) supports the central claim with a single comparator: a 'conventional cascaded P-PID' whose controller gains, tuning procedure, anti-windup, and feedforward terms are not reported. Under the stated settling criterion of 0.02 m / 2 deg, the P-PID mean settling time is 28.55 s with only 72.8% success in a 32 s test, meaning a large fraction of baseline episodes never settle within the horizon. The 10.6% settling-time ratio and 84.1% smoothness margin are therefore computed against a baseline that is plausibly under-tuned. The P-PID also has no current feedforward, while the evaluation injects stepwise current changes every 10 s (Fig. 9); a feedforward or better-integrated baseline would be expected to reduce both settling time and steady-state error. Because every claimed percentage reduction is relative to this single baseline, the 'consistently outperforms' statement is only as strong as the baseline tuning. This is a correctness risk, not an internal inconsistency: the RL pipeline, ablations, and real-time latency evidence support the method's functionality, but they do not establish superiority over conventional practice.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TSRCA-PPO, a two-stage reinforcement-learning controller for 6-DOF station keeping of an overactuated ROV. Stage I trains a PPO teacher policy with privileged information split into static, dynamic, and current-flow encoders; Stage II distills the resulting latent representation into a deployable student policy using a GRU-based adaptation module trained with DAgger-style data aggregation. The reward includes terminal pose terms, velocity-guidance terms, an energy proxy, and an action-smoothness term. Evaluation is performed in a MarineGym-style simulator with domain randomization and Gauss-Markov ocean currents. The central claim is that TSRCA-PPO consistently outperforms a conventional cascaded P-PID controller on steady-state position error, steady-state attitude error, settling time, success rate, energy index, and thrust-smoothness index, with the specific reductions listed in the abstract and conclusions.","tokens_in":16398,"tokens_out":5265,"duration_ms":55513,"significance":"The paper addresses a relevant problem and contains a substantial simulation study: 1,000 evaluation episodes per configuration, multiple metrics, ablations over the architecture components, and a latency measurement (4.407 ms per inference step) showing real-time feasibility relative to the 16 ms control period. The idea of structuring privileged information into separate encoders and distilling it with a temporal adaptation module is sensible, and the ablations give useful insight, such as the dynamic-state branch being the main contributor to action smoothness. If the comparative claims are supported, the method would be a useful contribution to RL-based ROV control. However, the headline comparison is weakened by the undocumented single baseline and by the overlap between some evaluation metrics and the training objective.","major_comments":[{"comment":"The single comparator is a 'conventional cascaded P-PID' whose gains, tuning procedure, anti-windup, and feedforward terms are not reported. Every headline percentage in the abstract and conclusion is relative to this controller. Because the P-PID has no current feedforward while the evaluation injects stepwise current changes (Fig. 9), and because its success rate is only 72.8%, the reported margins (e.g., settling time 28.55 s vs. 3.04 s) may reflect an under-tuned baseline rather than an inherent advantage of the proposed method. I ask the authors to report the controller parameters and tuning procedure, and to add at least one better-tuned or current-feedforward baseline to substantiate the superiority claim.","section":"Section 5.2, Table 8"},{"comment":"The text states that 'the settling time is computed only over successful episodes.' P-PID succeeds in only 72.8% of episodes while TSRCA-PPO succeeds in 100%. The reported mean settling time of 28.55 s for P-PID is therefore a conditional mean over the successful subset, whereas 3.04 s is effectively unconditional; comparing these two numbers is not apples-to-apples. Please report the settling-time statistic over all episodes, treating unsuccessful episodes as censored at the horizon, or report quantiles in addition to the conditional mean.","section":"Section 5.1, settling-time definition; Table 8"},{"comment":"The energy index J_E in Eq. (43) is exactly the accumulated per-step energy proxy E_t that enters the reward through Eq. (38), and the thrust-smoothness metric J_FS, although formally a second-order force difference, is directly driven by the action-smoothness reward in Eq. (39) because the policy is trained to minimize consecutive action changes. Consequently, the reported reductions in energy and smoothness are partly a direct consequence of the training objective and are not independent measurements of the controller's intrinsic efficiency or smoothness. Please state this overlap explicitly and, if possible, compare against a baseline that is tuned or optimized on the same objective, or use genuinely independent metrics.","section":"Equations (37)-(39) and (43)-(44)"},{"comment":"The conclusion states that 'Ablation studies validate the effectiveness and rationality of the proposed reward function,' but Section 5.1 explicitly ablates only the architecture (static/dynamic/current branches, single encoder, uncompressed latent) while keeping the reward function fixed. No ablation of the reward terms (e.g., velocity guidance, energy, smoothness, terminal terms) is presented. Either add such an ablation or temper the conclusion to refer only to the architecture and privileged-information design.","section":"Section 6, conclusion"}],"minor_comments":[{"comment":"The expression '180/π 2 arccos(|⟨q_t,q_d⟩|)' appears to be missing a multiplication sign; it should presumably be (180/π) × 2 arccos(|⟨q_t,q_d⟩|) or similar.","section":"Equation (32)"},{"comment":"The notation 'sgn(q_e,w,t) q_e,v,t' is ambiguous: it is unclear whether the sign is applied to the scalar quaternion component, to the vector part, or to each component of the vector part. Please define this notation precisely.","section":"Equation (36)"},{"comment":"The single-trajectory force-smoothness values in Fig. 10 (0.23 for TSRCA-PPO, 0.92 for P-PID) differ from the mean values in Table 8 (0.27 and 1.70); the caption and text should state clearly that the figure shows one representative example, not the average over the 1,000 evaluation episodes.","section":"Figure 10 and Table 8"},{"comment":"The PPO and TQC baselines have inference latencies of 0.86 ms and 0.36 ms, respectively, which are substantially lower than the 4.41 ms of the proposed student policy. The paper should comment on this latency increase as a cost of the adaptation module, even though 4.41 ms is still below the 16 ms control period.","section":"Tables 6 and 7, inference latency"},{"comment":"The phrase 'quantified by theZmean squared error' appears to be a typesetting artifact; please rephrase it as 'quantified by the mean squared error of the latent feature' or similar.","section":"Section 5.1, latent-feature MSE"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely salvageable: the method, ablations, and real-time latency evidence are coherent, and the internal arithmetic of the reported percentages checks out. The main weakness is the experimental comparison: the single P-PID baseline is not documented well enough to support the strong 'consistently outperforms' claim, and two of the headline metrics overlap with the training objective. I would encourage the editor to request a revised version that adds a properly specified and fairly tuned baseline, clarifies the settling-time censoring, and either adds reward ablations or tempers the corresponding conclusion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Take: this is a solid, honest RL-for-ROV paper, but the headline claim is weaker than the abstract suggests. The genuinely new pieces are the three-branch privileged encoder (static/dynamic/current) and the velocity-guidance reward terms that encourage deceleration near the target. The two-stage distillation follows RMA and learning-by-cheating, so novelty is incremental, but the ablations are thorough: 1000 evaluation episodes, multiple metrics, and a clear finding that the dynamic-state branch matters most for smoothness while the current branch matters most for settling time. The paper also reports training time and 4.4 ms inference latency, which is useful evidence for deployability.\n\nSoft spots, in proportion. The biggest one is the P-PID baseline. Its gains and tuning procedure are undocumented, there is no anti-windup or feedforward, and it fails to settle in 27.2% of episodes with a 28.55 s mean settling time in a 32 s test. A better-tuned PID with current feedforward could plausibly shrink those margins, so the 'consistently outperforms' claim is only as strong as that baseline. The stress-test note is correct here. Also correct: the energy metric JE is literally the same per-step proxy used in the reward (Eq. 37 vs Eq. 43), so the 6% energy reduction is partly a training objective, not an independent measurement. The smoothness metric is better separated (second-order force difference vs action difference), but it is still close to the reward's smoothness term.\n\nOne correction to the reader's take: the paper does benchmark other RL methods. Tables 6 and 7 include PPO and TQC baselines, and the proposed method clearly beats them on success rate and smoothness. That improves my confidence in the architecture, even if the P-PID comparison remains the weak pillar.\n\nWho this is for: researchers working on end-to-end or privileged-learning control for underwater vehicles. It deserves a serious referee. The method is clearly explained, the ablation logic holds, and the latency data are real. To move from conditional to accept, the authors should disclose the P-PID tuning, release code/data, and add at least one stronger baseline (e.g., PID with feedforward or better-tuned gains). A pool experiment would be a bonus, not a dealbreaker for a simulation paper.\n\nRecommendation: send it to peer review, but require baseline transparency and reward-metric independence before publication.","headline":"A competent RL-for-ROV paper with a narrow but real architectural novelty, whose headline superiority claim is held hostage by an undocumented P-PID baseline and reward-overlapping energy metrics.","tokens_in":16797,"tokens_out":2735,"would_cite":false,"duration_ms":30521,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An end-to-end reinforcement learning controller can hold a six-degree-of-freedom remotely operated vehicle on station under time-varying ocean currents more accurately, faster, more energy-efficiently, and with smoother thrust than a…","keywords":["deep reinforcement learning","remotely operated vehicles","6-DOF control","thrust smoothing","ocean current adaptation","knowledge distillation","proximal policy optimization","station-keeping"],"falsifier":"Run the same 1,000-episode station-keeping evaluation against a P-PID baseline whose gains are re-tuned under the same settling criterion and augmented with ocean-current feedforward and anti-windup; if its settling time and thrust-smoothness index approach TSRCA-PPO's values, the claimed superiority would be shown to depend on the baseline's construction rather than on the learned controller.","tokens_in":15813,"feed_emoji":"🌊","tokens_out":8968,"duration_ms":82192,"temperature":0.7,"pith_summary":"TSRCA-PPO is a proximal-policy-optimization controller that maps onboard observations directly to eight thruster commands, bundling motion control and thrust allocation into one learned policy. The paper's central claim is that this end-to-end approach meets a combination of requirements that earlier reinforcement-learning ROV controllers have not met at once: low steady-state position and attitude error, fast settling, low energy use, and smooth thrust under time-varying ocean currents. The method first trains a teacher policy with privileged information—true physical parameters, true dynamic states, motor speeds, and current conditions—then distills the teacher's latent current-awareness into a student policy that uses only a history of ordinary onboard observations. In simulation, the student policy reduces the position error, attitude error, settling time, energy index, and thrust-smoothness index to 42.7%, 76.5%, 10.6%, 93.5%, and 15.9% of the corresponding cascaded P-PID values, and raises the success rate from 72.8% to 100.0%.","feed_headline":"Learned ROV controller cuts settling time to a tenth of P-PID's","feed_subtitle":"A two-stage distillation policy keeps station with 57% lower position error and 84% smoother thrust than P-PID.","key_machinery":"The load-bearing object is the two-stage distillation pipeline with a privileged multi-encoder. Stage I trains a PPO teacher whose actor encodes the action-observation pair and three separately encoded privileged groups—true static parameters, true dynamic states including actual motor rotational speeds, and true current parameters—into a low-dimensional latent feature $z^{\\mathrm{priv}}_t$; the critic receives the full concatenated privileged state. Stage II freezes the base policy and trains a GRU-based adaptation module, using DAgger-style on-policy data collection, to predict the latent feature from a history buffer $H_t = [(\\tilde{o}_{t-L}, a_{t-L}), \\ldots, (\\tilde{o}_{t-1}, a_{t-1})]$ by minimizing a three-branch regression loss. This mechanism decouples optimal-policy learning from implicit state estimation: the deployable student infers current conditions and internal parameters from proprioceptive history rather than requiring a current sensor.","core_discovery":"On the paper's own terms, the discovery is that a carefully shaped reward plus a privileged multi-encoder lets reinforcement learning produce near-optimal station-keeping for an overactuated 6-DOF ROV under currents the policy has not been designed around. The reward's velocity-guidance terms project the vehicle's velocity onto the direction of its position and attitude error, acting as an implicit braking signal that suppresses overshoot and oscillation near the target. The privileged multi-encoder splits static parameters, dynamic states, and current flow into separate latent branches; ablation shows the dynamic-state branch is the main contributor to thrust smoothness, while the current-flow branch mainly shortens settling time. After distillation, the student estimates the teacher's latent feature from a 30-step history of observations and actions, and runs in 4.407 ms per inference step, well below the 16 ms control period.","pith_inferences":["A testable extension would be to benchmark the same student policy against a P-PID baseline augmented with ocean-current feedforward and anti-windup; the paper does not report the baseline's gains, tuning procedure, or such augmentations, so it is an open question which margins survive against a stronger conventional controller.","If dynamic-state latent information is the main driver of smoothness, then replacing the GRU adaptation module with a lightweight recursive estimator of motor RPM and acceleration could bring similar smoothness to platforms without the full distillation pipeline.","The energy comparison uses a cubic-RPM proxy rather than measured electrical power; instrumenting thrusters to log true power in a hardware test would show whether the 6.4% energy saving is real or an artifact of the proxy.","The method's success in simulation suggests a natural next step of zero-shot transfer to a physical ROV in a current basin, where the student's reliance on observation history could be stress-tested under sensor dropout and unmodeled waves."],"forward_implications":["The student policy's 4.407 ms inference latency fits inside the 16 ms control period, so the method can run in real time rather than only as an offline benchmark.","Because removing the dynamic-state branch raises the force-smoothness index from 0.26 to 0.61 in Stage I and roughly doubles it in Stage II, accurate onboard state estimation is the main lever that buys thrust smoothness.","Because removing the current-flow branch primarily lengthens settling time, rapid current adaptation is concentrated in the flow-latent branch rather than spread across the whole policy.","The distilled student achieves 100.0% success under the 0.02 m and 2-degree settling criterion while using only observation histories, so the privileged information does not need to be measured at deployment.","A stepwise current-reversal test shows the policy recovers attitude shortly after each 90-degree current-direction change, indicating the flow latent is being tracked online rather than merely fitting a static offset."],"supporting_citations":[{"why":"Supplies the Isaac-based simulation platform, Gauss-Markov current setup, and T200 thruster model used for training and evaluation.","marker":"Chu et al., 2025"},{"why":"Provides the standard 6-DOF marine vehicle model and the thrust configuration matrix that maps thruster forces to generalized forces.","marker":"Fossen, 2002"},{"why":"Provides benchmark vehicle parameters and motivates the wider domain-randomization range for rotational added mass.","marker":"von Benzon et al., 2022"},{"why":"Introduces the learning-by-cheating paradigm that licenses training with privileged information and distilling it to a student.","marker":"Chen et al., 2020"},{"why":"Demonstrates rapid motor adaptation through privileged distillation, the template for the two-stage teacher-student design.","marker":"Kumar et al., 2021"},{"why":"Provides DAgger, the on-policy data aggregation method used to train the student adaptation module.","marker":"Ross et al., 2011"},{"why":"Supplies the proximal policy optimization algorithm used to train the Stage-I teacher policy.","marker":"Schulman et al., 2017"}],"fun_headline_variants":["RL policy trims ROV settling time by 89% vs P-PID","Two-stage distillation tames ROV thrust, cuts errors and energy","TSRCA-PPO: RL for smooth ROV control under ocean currents","Reinforcement learning cuts ROV position error by 57% and thrust ripple by 84%","Distilled RL policy beats P-PID on all metrics for ROV station-keeping"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole comparison rests on the assumption that the cascaded P-PID controller is a well-tuned representative of conventional practice; the paper does not report its gains, tuning procedure, or whether it compensates for currents, so the reported margins could shrink if the baseline were made stronger.","fun_headline_variants_meta":{"raw":{"variants":["RL policy trims ROV settling time by 89% vs P-PID","Two-stage distillation tames ROV thrust, cuts errors and energy","TSRCA-PPO: RL for smooth ROV control under ocean currents","Reinforcement learning cuts ROV position error by 57% and thrust ripple by 84%","Distilled RL policy beats P-PID on all metrics for ROV station-keeping"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001164,"raw_usage":{"total_tokens":4830,"prompt_tokens":971,"completion_tokens":3859,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":3752}},"tokens_in":587,"tokens_out":3859,"duration_ms":25440,"temperature":1.0,"reasoning_tokens":3752,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:30:05.270220+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 1,000-episode station-keeping evaluation against a P-PID baseline whose gains are re-tuned under the same settling criterion and augmented with ocean-current feedforward and anti-windup; if its settling time and thrust-smoothness index approach TSRCA-PPO's values, the claimed superiority would be shown to depend on the baseline's construction rather than on the learned controller.","supporting_citations":[{"cited_title":"2025 , publisher=","cited_arxiv_id":null,"evidence_quote":"Supplies the Isaac-based simulation platform, Gauss-Markov current setup, and T200 thruster model used for training and evaluation."},{"cited_title":"2021 , doi=","cited_arxiv_id":null,"evidence_quote":"Demonstrates rapid motor adaptation through privileged distillation, the template for the two-stage teacher-student design."}],"review_version":1}