{"id":"9413029e-e95b-4e6a-aabb-82c9bbd412b6","arxiv_id":"2602.20220","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Sim-to-online RL on three real robots is stabilized by retaining data, warm-starting the replay buffer, and using asymmetric actor-critic updates with a low actor learning rate.","lead":"A large real-robot study tests which standard RL design choices make simulation-pretrained policies learn stably online. The recipe: keep previous data, warm-start the replay buffer, and update the actor less often than the critic.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Real-world evidence for 'asymmetric updates are critical' is confounded: Figure 10 varies both update delay (M) and actor learning rate simultaneously, so the effect could be due to lower LR alone.","rationale":"The reader's weakest assumption is that the hyperparameters M=20 and actor LR=1e-5 are selected from simulated mismatch sweeps and assumed to transfer to real robots, and that Figure 10 tests only the selected point against one baseline. My concern sharpens this: the baseline differs in two variables simultaneously, so even the sim-to-sim sweeps do not establish that the delayed update frequency is the causal factor. This is the most load-bearing issue because the paper's headline 'asymmetric updates are critical' rests on it. The proposed 2x2 factorial would settle whether the delay matters independently of the learning rate. If the delay is not separately necessary, the paper's recipe still works but the specific mechanism is misattributed, and the guidance to use M=20 would need to be reframed as a consequence of using a low actor LR. This does not change the overall assessment of the paper — the empirical study is valuable and the off-policy finetuning recipe is plausible — but it does mean the current verdict of CONDITIONAL remains appropriate, with the condition being the isolation of the asymmetric-update effect.","tokens_in":16369,"tokens_out":4909,"duration_ms":45527,"concrete_test":"Run a 2x2 factorial on the sim-to-sim mismatch task (e.g., Franka with soft ball) and, if feasible, on one real robot: M ∈ {1,20} × actor LR ∈ {3e-4, 1e-5}, with all other hyperparameters fixed (critic LR 3e-4, same annealing schedule, same seeds). Compare mean ± SE over at least 3 seeds. If M=1 with LR=1e-5 performs as well as M=20 with LR=1e-5, the update delay is not necessary; if M=20 with LR=3e-4 fails while M=1 with LR=1e-5 succeeds, the LR reduction is the active ingredient. This isolates whether 'asymmetric updates' or simply a small actor step size drives stability.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's recipe (Section 4) highlights 'asymmetric updates' — updating the actor every M=20 critic steps with actor LR 1e-5 — and claims they are 'critical' for stable transfer (Section 5.2, Figure 10). The baseline in Figure 10 changes two hyperparameters at once: it updates the actor every critic step (M=1) and uses a shared actor/critic LR (3e-4), while the proposed method uses M=20 and actor LR 1e-5. Thus any observed improvement could be caused by the 30x lower actor learning rate rather than by the delayed actor updates. The sim-to-sim ablation in Figure 4 is described as 'as we increase M and reduce learning rate from 3e-4 to 1e-5', which suggests the two factors were varied jointly rather than independently; if so, that ablation also fails to isolate which factor stabilizes learning. No real-world experiment fixes LR while varying M, or fixes M while varying LR. Consequently, the mechanistic claim that 'asymmetric updates are critical' is underdetermined. This is load-bearing because the asymmetric-update component is highlighted as a key 'what matters' finding in the introduction and conclusion. If the actual cause is merely a smaller actor step size, the specific M=20 recommendation is overfit to the simulated mismatch and may not transfer to other real-robot settings. The broader claim that SAC can fine-tune effectively may still hold, but the paper's specific attribution is not established by the presented experiments.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the sim-to-online RL setting: policies pretrained in simulation are fine-tuned online on physical robots. On three platforms — a Franka Panda manipulator, a Unitree Go1 quadruped, and a remote-controlled race car — the authors run a large number of real-world training episodes and ablate design choices including retaining simulation/previous-experiment data via annealed replay mixing (α from 0.5 to 1), warm-starting the online replay buffer, using asymmetric actor-critic updates (actor LR 1e-5, actor updated once every 20 critic updates), and scaling the update-to-data ratio with the number of parallel simulation environments. The paper reports that Soft Actor-Critic with these ingredients fine-tunes successfully and stably, while baselines without them (no data retention, symmetric updates with a shared 3e-4 learning rate) often unlearn the prior policy. The authors also release an open-source training pipeline for the Panda task.","tokens_in":16753,"tokens_out":8846,"duration_ms":83707,"significance":"If the recipe is viewed as a package, this is a valuable and timely contribution: it provides one of the largest real-robot comparisons of sim-to-online RL design choices, covers three very different hardware platforms, reports standard errors, and ships an open-source pipeline for the Franka Panda. The work also extends prior replay-mixing ideas to real robots with annealed mixing, which is a meaningful step. However, the paper's headline attribution — that 'asymmetric updates are critical' — is not established by the experiments as presented, because the key ablation varies the update delay and the actor learning rate simultaneously. The broader conclusion that standard off-policy algorithms can be fine-tuned on real robots with modest changes is credible, but the specific mechanistic claim needs additional isolation experiments or a careful reframing.","major_comments":[{"comment":"The claim that 'asymmetric updates are critical' (Section 5.2, Figure 10) is confounded. In Figure 4, the ablations change both the update delay M and the actor learning rate jointly (caption: 'as we increase M and reduce learning rate from 3e-4 to 1e-5'). In Figure 10, the baseline differs from the proposed method in two dimensions: M=1 instead of M=20 and a shared actor/critic LR of 3e-4 instead of an actor LR of 1e-5. The observed instability of the baseline could therefore be due entirely to the 30x smaller actor learning rate, and there is no experiment in either the sim-to-sim sweep or the real-robot study that fixes LR while varying M, or fixes M while varying LR. This is load-bearing because the introduction and conclusion highlight asymmetric updates as a key 'what matters' finding. Please add independent sweeps (e.g., M in {1,5,20} with LR fixed at 1e-5, and LR in {3e-4,1e-5} w","section":"Section 4 / Figure 4 / Section 5.2 / Figure 10"},{"comment":"The paper's broad quantitative claims — 'significant performance gains' (Section 5.2), 'critical for effective transfer across all robots' (Section 5.2), and 'large-sample empirical study' (Abstract) — rest on three seeds per condition and no statistical tests or raw data release. The plotted standard errors in Figures 8-10 are sizable, so several differences appear qualitative rather than statistically robust. This is not a demand for a full hypothesis-test battery in a real-robot study, but the wording should match the evidence: either add effect sizes, paired comparisons, or confidence intervals where feasible, or hedge the categorical language as observations on three platforms with limited replication.","section":"Section 5.2 / Figures 8-10 / Abstract"},{"comment":"The specific recommended values M=20 and actor LR=1e-5 are selected from simulated mismatch sweeps (Figure 4) and then tested on real robots only at that single operating point against one symmetric baseline. No real-robot sensitivity analysis is provided around M or LR, so it remains possible that the ranking of these settings changes under the sim-to-real gap, and that a different combination (e.g., M=5 or LR=5e-6) would be equally or more stable. The paper should either report a small sensitivity analysis on real hardware or in a more faithful sim-to-sim mismatch setup, or explicitly describe these hyperparameters as task-specific choices rather than universal 'what matters' findings.","section":"Section 4 / Section F / Figure 10"}],"minor_comments":[{"comment":"'simply delaying critic updates (akin to Fujimoto et al. [10])' is inaccurate: TD3 delays policy/actor updates, not critic updates. Section 4 uses the correct terminology; please fix the bullet to avoid confusion.","section":"Introduction, contribution bullet"},{"comment":"'each experiment is composed of four trials that only share the same random seed' is confusing. It appears you mean each seed is run across four trials with shared initial randomness; please rephrase for clarity.","section":"Section 5.2, 'Recycling data' paragraph"},{"comment":"The caption says 'showing significant stability improvements' but no statistical significance test is reported; consider replacing 'significant' with 'clear' or 'consistent' to avoid implying formal tests.","section":"Figure 4 caption"},{"comment":"Figure 8 labels trials P/Q/R/S while the text refers to trials 0/1/2/3; align the labels.","section":"Figure 8 and text"},{"comment":"Typo: 'need seen' should be 'never seen'.","section":"Appendix D, Figure 17 caption"},{"comment":"The code release is appreciated, but the paper does not release raw reward trajectories or hyperparameter logs. Given the 'large-sample' claim and the small per-condition seed count, releasing the raw data would substantially strengthen reproducibility and allow readers to evaluate the statistical strength of the claims.","section":"Data availability"}],"recommendation":"major_revision","confidential_remarks":"This is a solid empirical systems paper with real-world value, and the code release is a clear strength. The main risk is that the headline attribution of stability to 'asymmetric updates' is underdetermined by the current ablation design; this is fixable with additional sim-to-sim isolation sweeps and, ideally, one real-robot control, or by softening the claims. I would be willing to accept after such a revision. I would also encourage the editor to look at whether the 'large-sample' and 'significant' language is adequately supported by three-seed statistics."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this is a genuinely useful empirical paper, but the specific claim that “asymmetric updates are critical” is not actually established by the experiments. The real-world comparison in Figure 10 changes two things at once: the actor update interval M (1 to 20) and the actor learning rate (shared 3e-4 to 1e-5). The sim-to-sim sweep in Figure 4 does the same. So the observed stability gain could be the low LR alone, or the delay, or the combination. That undermines a load-bearing conclusion.\n\nWhat the paper does well: 100+ real-run dataset across three platforms (Panda, Go1, race car), open-source pipeline, and careful documentation of pitfalls like optimizer-state restores. The data retention result with annealed alpha is well supported and extends [20] to hardware. Warm starts as a proxy is also reasonable. The zero-shot transfer measurements and the Ne=8192 vs 128 result are valuable. The paper is clearly written and straightforwardly honest about what was tested.\n\nSoft spots beyond the confound: three seeds per condition, no statistical tests, no raw data release. The claim “across tasks and hardware” is a stretch from three platforms. The hyperparameters M=20 and LR=1e-5 are selected from simulated mismatch sweeps, so there is a tuning-selection risk; but the real validation does at least test the chosen point. The confound is the more serious issue because it directly weakens the paper’s stated mechanism.\n\nAlso note: the paper’s own text in Section 4 says “increasing M ... and reduce learning rate”, so they know they varied both. That makes the overclaim in Section 5.2 harder to excuse. A simple fix: hold LR fixed and vary M, and vice versa, on at least one real robot; or soften the claim to “conservative actor updates” rather than “asymmetric updates are critical.”\n\nWho this is for: practitioners working on sim-to-real online RL, and researchers benchmarking off-policy algorithms. It deserves a serious referee – the empirical contribution and open pipeline are worth engaging with even if the attribution issue needs fixing. I’d send it to review, and I’d ask the authors to address the confound before publication.\n\nRecommendation: engage with the work; cite the data retention and pipeline findings, but not the asymmetric-update claim as stated.","headline":"Solid empirical study of sim-to-online RL on three real robots, but the headline claim that asymmetric updates are critical is undercut by a two-factor confound.","tokens_in":17226,"tokens_out":2127,"would_cite":true,"duration_ms":20744,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that standard off-policy reinforcement learning can be finetuned on real robots without major algorithmic changes, provided three design choices—retaining simulation or prior data with annealed mixing, warm-starting the rep","keywords":["sim-to-real transfer","online reinforcement learning","real robot learning","off-policy RL","soft actor-critic","replay buffer mixing","asymmetric actor-critic updates","domain randomization"],"falsifier":"Run the same three task deployments with data retention and warm starts fixed, but sweep the actor update delay M in {1,5,10,20} and actor learning rate in {1e-5,3e-4} on hardware; if a configuration close to vanilla off-policy RL matches or beats the chosen recipe across all robots, the central claim collapses.","tokens_in":16293,"feed_emoji":"🤖","tokens_out":6389,"duration_ms":57788,"temperature":0.7,"pith_summary":"This paper tries to establish that sim-to-online reinforcement learning on physical robots does not need bespoke algorithms. Across more than 100 real-world training runs on three very different robot types—a manipulation arm, a quadruped, and a race car—the authors show that a standard off-policy method (soft actor-critic, SAC) stays stable when three practices are combined: keep the simulation or prior-experiment data in a separate buffer and anneal the mixing weight from 0.5 to 1; warm-start the online buffer with the pretrained policy if prior data cannot be kept; and update the actor only every 20 critic updates at a reduced learning rate. The payoff, if the claim is right, is that deploying online RL on real hardware takes less engineering effort, and the same recipe works for vision-based, sparse-reward tasks and high-frequency control.","feed_headline":"Three simple tweaks stabilize RL finetuning on real robots","feed_subtitle":"A 100-run study across three robot types finds retaining data and delaying actor updates let off-policy RL keep learning online.","key_machinery":"The load-bearing mechanism is the two-buffer replay mixture, (1-alpha)Unif(D0) + alpha Unif(Donline), combined with an asymmetric actor-critic update schedule (actor learning rate 1e-5, one actor update per 20 critic updates). The paper's theoretical anchor is the approximate policy improvement bound: cumulative improvement is guaranteed only while the Q-value error term stays smaller than the greedy improvement term; retaining low-error prior data and slowing the actor keep that error term in check. The same asymmetry is grounded in two-timescale stochastic approximation, where the critic is allowed to track the value function before the policy changes.","core_discovery":"The central claim is that the instability seen when finetuning a simulation-trained policy on a real robot is a predictable 'downward spiral,' and that three within-standard-practice choices stop it without modifying the algorithm. The authors document that vanilla SAC overestimates the action-value function in state-action pairs visited after deployment, and that this bias compounds across actor-critic updates. Their recipe combines (1) sampling minibatches as a mixture of the prior buffer and the online buffer, with the mixture weight annealed from 0.5 to 1.0 so the prior regularizes early updates and is dropped later; (2) warming the online buffer with transitions from the pretrained poli","pith_inferences":["If the mixture-annealing logic is the active ingredient, the same recipe should apply to any off-policy algorithm that uses replay, not only soft actor-critic—the TD3 experiments in the paper already point this way.","The fixed five-episode annealing schedule for alpha is a free hyperparameter; a reasonable extension is to tie alpha to measured critic error, annealing only when the online critic becomes trustworthy, which could make the recipe robust to larger sim-to-real gaps.","The delayed actor update (M=20) and low actor learning rate likely trade off against each other: a lower learning rate might allow more frequent updates, and higher rates might require longer delays. Mapping this trade-off per task is a cheap, testable next step.","The paper's manual-reset episodic setting limits immediate use in fully autonomous deployments; combining this recipe with automatic reset mechanisms is the natural next extension rather than a change to the learning algorithm itself."],"forward_implications":["Practitioners can finetune simulation-trained policies on real robots with an unmodified off-policy algorithm by adopting three settings: annealed data mixing, warm-started replay, and delayed low-rate actor updates.","Retaining the simulation replay buffer (or data from a few prior real trials) stabilizes online finetuning because critic updates stay concentrated in state-action regions where the critic is already accurate; annealing to alpha=1 ensures the final policy is optimized purely on real-world data.","When offline data cannot be retained, warming the online buffer with transitions from the pretrained policy is a fallback that preserves most of the stability benefit, though it is weaker than full data retention.","Massively parallel simulation training of off-policy RL works if the update-to-data ratio is raised in proportion to the number of parallel environments; a sufficiently large number of domain-randomized environments (around a thousand) is needed for robust sim-to-real transfer.","The recipe transfers across manipulation, locomotion, and navigation, including vision-based sparse-reward manipulation, suggesting the choices are task-agnostic rather than platform-specific."],"fun_headline_variants":["Retain data, warm buffer: stable robot RL in 100 runs","Three tweaks stop RL spiral on real robots, 100-run study","Off-policy RL on robots: simple fixes from 100 real runs","Why robot RL fails and how to fix it without new algorithms","Mixture sampling and buffer warm-up tame real-robot RL"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The recipe's exact settings—actor learning rate 1e-5 and one actor update per 20 critic updates—were selected from simulated mismatch sweeps and assumed to transfer to the real robots; if the sim-to-real gap changes which settings are best, the claim that asymmetric updates are critical would be an artifact of tuning.","fun_headline_variants_meta":{"raw":{"variants":["Retain data, warm buffer: stable robot RL in 100 runs","Three tweaks stop RL spiral on real robots, 100-run study","Off-policy RL on robots: simple fixes from 100 real runs","Why robot RL fails and how to fix it without new algorithms","Mixture sampling and buffer warm-up tame real-robot RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0002,"raw_usage":{"total_tokens":1143,"prompt_tokens":607,"completion_tokens":536,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":351,"completion_tokens_details":{"reasoning_tokens":444}},"tokens_in":351,"tokens_out":536,"duration_ms":5503,"temperature":1.0,"reasoning_tokens":444,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T21:32:16.757926+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same three task deployments with data retention and warm starts fixed, but sweep the actor update delay M in {1,5,10,20} and actor learning rate in {1e-5,3e-4} on hardware; if a configuration close to vanilla off-policy RL matches or beats the chosen recipe across all robots, the central claim collapses.","supporting_citations":[],"review_version":1}