{"id":"e98f468c-8767-4e47-91aa-d2c3cc55cdfc","arxiv_id":"2507.10030","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Hybrid SAC+SNES training improves swing-up and competition scores for underactuated robots over RL-only baselines.","lead":"This paper proposes a two-step recipe for robot control: train a reinforcement learning policy on a smooth approximate reward, then use evolutionary search to fine-tune it on the real, sparse competition score. On cartpole and the 2nd AI Olympics underactuated double-pendulum tasks, the hybrid controller beats RL-only and classical baselines while keeping robustness.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pendubot headline gain (0.596 vs 0.567) appears to be a best-of-5 selection effect: the mean EvolSAC score (0.544±0.029) does not significantly exceed the SAC mean (0.521±0.024).","rationale":"The reader's stated weakest assumption is that the method requires the SAC policy to be near-optimal so that SNES performs a local refinement. That is a real scope limitation, but it does not undermine the specific reported gains; the method is explicitly a finetuning step, and the experiments start from SAC policies with reasonable scores. The more load-bearing concern is statistical: the strongest quantitative claim for pendubot (0.596 vs 0.567) is based on a single best-of-5 agent, while the mean scores on the same seeds show an overlapping, non-significant difference. This is not a question of internal consistency or correctness of the algorithm, but of whether the evidence supports the claimed improvement. The reader's verdict of CONDITIONAL already flags 'small samples without code' and 'mean results rather than best-of-N' in the rationale, so our concern partially overlaps with the reader's overall assessment, even though it is not the reader's chosen weakest_assumption. We therefore keep the verdict unchanged: conditional acceptance is appropriate, but the conditions should explicitly require significance testing and mean-based reporting, especially for the pendubot configuration. The acrobot result appears robust, so a wholesale rejection is not warranted, and the method itself is neither internally inconsistent nor contradicted by the data for at least one system.","tokens_in":8450,"tokens_out":5732,"duration_ms":63479,"concrete_test":"Obtain the raw per-seed performance scores for the 5 SAC and 5 EvolSAC agents used for Tables 2–4. Compute a paired or two-sample t-test (or Wilcoxon signed-rank) for acrobot and pendubot separately. Also report the mean and 95% confidence interval of the selected 'optimal' agents and compare the 0.596 pendubot value against the distribution of the 5 seeds. If the pendubot comparison is non-significant at p<0.05, revise the headline claim to specify that the improvement is statistically supported only for acrobot, and report the mean scores (and CIs) as the primary evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline numbers for the double-pendulum benchmark come from Tables 3 and 4, which report the 'optimal agents' selected as the highest average of performance and robustness among the 5 trained seeds. This is best-of-N selection, not the central tendency. Table 2 gives the mean over the same 5 seeds: for pendubot, SAC is 0.5207±0.0235 and EvolSAC is 0.5441±0.0288. The mean improvement is 0.0234, with overlapping standard deviations; a two-sample t-test on the reported statistics yields p≈0.19, not significant. The cited 0.596 is the maximum of a small sample, and selection over 5 seeds inflates the expected maximum, so the single best case does not provide evidence of a reliable improvement. For acrobot the mean improvement (0.4972→0.5249) is statistically significant and the claim holds there, but the broad statement 'significantly improves agent performance' across both systems rests on the pendubot result, which is not supported by the mean comparison. The paper does not report per-seed raw scores, confidence intervals, or any significance test, so the reader cannot distinguish a real effect from sampling noise. This is load-bearing because the strongest claim explicitly cites the 0.596 number, and the central contribution is the claimed improvement, not the method's existence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Evolutionary SAC (EvolSAC), a two-phase method for control of underactuated systems: a SAC policy is first trained on a dense surrogate reward, and its parameters are then fine-tuned with Separable Natural Evolution Strategies (SNES) directly optimizing the sparse task score (swing-up time on cartpole; the RealAIGym AI Olympics performance score on acrobot and pendubot). On cartpole, SAC+SNES reduces average swing-up time from 2.085 s to 1.135 s, versus 1.565 s for continued SAC training. On the double-pendulum benchmark, EvolSAC is reported to raise the acrobot performance score from 0.504 to 0.524 and the pendubot score from 0.567 to 0.596, with small reductions in robustness (0.700 to 0.692 and 0.800 to 0.796), and to exceed listed baselines. The authors claim that evolutionary fine-tuning significantly improves performance while maintaining high robustness, and they explicitly assume that the initial SAC policy is near-optimal.","tokens_in":8838,"tokens_out":15526,"duration_ms":142191,"significance":"If the claims hold, the paper is a useful empirical contribution: the two-phase surrogate-RL/ES scheme is simple, plausible, and directly addresses the mismatch between dense surrogate rewards and sparse trajectory-level evaluation metrics. The cartpole comparison against continued SAC training is a well-chosen control, the acrobot mean improvement (0.4972 ± 0.0088 to 0.5249 ± 0.0085) is statistically credible, and the final competition scores are competitive with official baselines. The authors also deserve credit for explicitly acknowledging the near-optimal-initialization assumption and for reporting robustness even though it is not part of the optimized objective. However, the central statistical evidence on the double-pendulum benchmark is weaker than the abstract suggests: the headline pendubot gain is a best-of-5 selection effect, no significance tests or per-seed data are reported, and the robustness comparison is also selection-based. With per-seed reporting and appropriately tempered claims, this would be a sound empirical paper; as written, the claims outpace the evidence. No code repository is mentioned.","major_comments":[{"comment":"The headline pendubot improvement (0.596 vs 0.567) is a best-of-5 selection effect. Table 2 reports means over five trained agents: SAC 0.5207 ± 0.0235 and EvolSAC 0.5441 ± 0.0288; a two-sample t-test on these statistics gives t ≈ 1.4 and p ≈ 0.19, i.e., the mean improvement is not significant. Table 4, by contrast, reports the 'optimal agent' selected as the highest average of performance and robustness over the five seeds, so 0.596 is the maximum of a small sample rather than the central tendency; selecting the maximum inflates it more for the higher-variance EvolSAC distribution. Because the abstract claims that evolutionary fine-tuning 'significantly improves agent performance' for the competition tasks, and the pendubot result is one of the two supporting cases, this is load-bearing. Please report per-seed scores and paired differences with confidence intervals or a significance test, and if the pendubot effect is not significant, temper the claim accordingly. For acrobot the mean improvement (0.4972 to 0.5249) is significant, so the issue is specific to pendubot but affects the abstract's general statement.","section":"Section 4.2, Tables 2 and 4"},{"comment":"The robustness-maintenance claim ('minimal reduction in robustness scores, which drop from 0.700 to 0.692 for the acrobot and from 0.800 to 0.796 for the pendubot') is also based on the selected optimal agents, not on the mean over seeds, and no per-seed robustness values or tests are given. In addition, the SAC rows of Tables 3 and 4 (0.504 and 0.567) appear to be the same 'top performance scores' reported in the SAC training paragraph, i.e., also the best of five runs; comparing two selection maxima is a valid protocol only if stated as such and analyzed accordingly. Please supplement Tables 3 and 4 with the mean ± standard deviation robustness results, state explicitly how each entry was selected, and confirm whether the SAC baselines in these tables are best-of-5 values.","section":"Section 4.2, Tables 3 and 4"},{"comment":"The sentence introducing Table 2 — '5 agents, each trained via SAC with a specific set of hyperparameters' — is ambiguous. If the five agents use different hyperparameter settings, then the mean and standard deviation in Table 2 do not estimate the performance of a fixed configuration, and the comparison with SAC would mix hyperparameter effects with the effect of SNES fine-tuning. If all five use the identical configuration from Table 1, the sentence should say so. This ambiguity matters because the table is the only non-selected evidence for the double-pendulum claim.","section":"Section 4.2, Table 2"},{"comment":"The claim that injecting noise of variance σ² during SNES evaluation 'prevents the posterior from collapsing to a Dirac delta distribution' and thereby preserves robustness is asserted but never tested: there is no ablation comparing SNES fine-tuning with and without this noise injection, and σ² itself is measured from a trained SAC trajectory, so it is a data-dependent design choice. Since robustness preservation is a central part of the paper's claim ('while maintaining high robustness'), please add an ablation or, at minimum, per-seed robustness statistics for the variant without noise injection.","section":"Section 3, noise-injection procedure"},{"comment":"The paper correctly acknowledges that the method is contingent on the starting point: 'we assume that the initial SAC agent has approximated a near-optimal policy, with a discrepancy amenable to further optimization via SNES.' This assumption is load-bearing because it limits the approach to local refinement, yet the abstract and introduction frame the method more generally ('in some cases, policies may require refinement'). Please state this limitation explicitly in the abstract and conclusions, and if feasible report one experiment that quantifies how the fine-tuning gain degrades when SNES is initialized from an earlier, less-converged checkpoint.","section":"Section 4.2, SNES agent training"}],"minor_comments":[{"comment":"The left-column y-axis label is 'returns' while the right column is labeled 'score [s]', but the text reports swing-up times in seconds; please make the axes consistent and state whether the plotted quantity is a rolling window of swing-up time or reward.","section":"Section 4.1, Figure 2"},{"comment":"The cartpole comparison reports average swing-up times (2.085 s to 1.135 s vs continued SAC 1.565 s) for 10 agents but gives no variance, confidence intervals, or test; the effect is large, but a paired or two-sample summary statistic would substantiate the word 'significant'.","section":"Section 4.1"},{"comment":"Several standard training details are missing: network architecture, discount factor, replay buffer size, entropy target for SAC, and episode length for the double-pendulum tasks; the learning rate 0.001 appears only in the text.","section":"Section 4.2, Table 1"},{"comment":"Clarify whether the σ² noise injection is applied only during SNES candidate rollouts or also at deployment of the final EvolSAC controller.","section":"Section 3"},{"comment":"Clarify how the normalized action at ∈ [−1, 1] maps to the physical torque limit τmax = 3 N·m (for example, through a scaling factor).","section":"Section 4.2"},{"comment":"The statement that 'SNES alone does not discover viable solutions within this time frame' would benefit from reporting how many generations and environment evaluations were used in that ablation.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the paper's claims as written outpace the evidence. The pendubot headline is a selection effect, and the abstract's 'significantly improves' is not supported by the mean comparison; I would condition acceptance on per-seed data, significance testing, and a clear statement of the selection protocol for the competition tables. The circularity concern (SNES optimizes precisely the reported metric) is real but not disqualifying; it means the continued-training control on cartpole and the robustness results carry the argument, and the revision should foreground those. The ambiguity about whether the five seeds share hyperparameters also needs to be resolved. This is a solid empirical short paper if the statistics are fixed; the near-optimal-initialization limitation should be stated in the abstract."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is straightforward and worth taking seriously: train SAC on a surrogate reward, then finetune the policy with SNES directly on the sparse competition score. The paper does this cleanly and reports more than most empirical RL papers, including mean and standard deviation over five seeds in Table 2, a comparison against continued SAC/TD3 training, and a robustness metric that is not directly optimized. That alone puts it ahead of the typical competition-paper baseline. What is actually new is the application to the AI Olympics tasks: SAC+SNES beats SAC and the listed baselines on acrobot both in mean performance (0.5249 vs 0.4972) and in the selected best case (0.524 vs 0.504). The cartpole result is also meaningful: average swing-up time drops from 2.085 s to 1.135 s, and the comparison against continued training shows the gain is not just diminishing-returns noise. The robustness scores barely drop, which is a genuinely non-obvious finding given that SNES has no explicit robustness mechanism. The soft spot is the pendubot headline. The paper reports 0.596 for EvolSAC in Table 4, but that is the best of five seeds, selected by highest average of performance and robustness. The mean from Table 2 is 0.5441 ± 0.0288 versus SAC's 0.5207 ± 0.0235. Those overlap heavily; a quick t-test on the reported numbers gives p ≈ 0.19. So the broad claim that evolutionary finetuning significantly improves performance on both double-pendulum configurations rests on the acrobot result plus a selected best case for pendubot. That is not enough. The paper should report per-seed raw scores, confidence intervals, or at least be explicit that the pendubot gain is a selection effect, not a reliable mean improvement. The circularity concern is real but not fatal. SNES directly optimizes the competition score, so higher scores on that metric are expected. What saves the paper is the acrobot mean improvement, the robustness maintenance, and the comparison against continued SAC training, which shows the finetuning does something beyond just extra optimization time. The method is also clearly limited by the near-optimality assumption: if the SAC policy is poor, SNES local search will not escape the basin. The authors state this assumption honestly. Who is this for? Anyone working on RL-to-ES hybrid pipelines or on the AI Olympics benchmark specifically. It deserves a serious referee. The experiments are reproducible in principle, but no code or raw per-seed data is provided, which should be required before acceptance. My recommendation: send it to review, require the authors to report means rather than best-of-N for all headline claims, add significance testing, and release code and per-seed results. The core method is sound and the acrobot result is solid; the pendubot claim needs to be reframed or supported.","headline":"Useful hybrid RL+ES finetuning study with a real result for acrobot and cartpole, but the pendubot headline number is a best-of-five selection effect; report the means.","tokens_in":873,"tokens_out":913,"would_cite":true,"duration_ms":23131,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning a Soft Actor-Critic policy with Separable Natural Evolution Strategies directly on the true task score improves underactuated robot control beyond RL alone.","keywords":["Control Systems","Deep Learning","Evolutionary algorithms","Reinforcement learning","Underactuated systems","Soft Actor-Critic","Separable Natural Evolution Strategy","Swing-up control"],"falsifier":"Run the same SNES fine-tuning from SAC checkpoints taken at 500, 1000, and 2000 episodes; if the improvement over the un-finetuned SAC policy shrinks or vanishes as the starting policy worsens, the near-optimal warm start is doing the work.","tokens_in":8294,"feed_emoji":"🤖","tokens_out":6887,"duration_ms":66911,"temperature":0.7,"pith_summary":"This paper proposes a two-stage recipe for control tasks whose true objective is a sparse, trajectory-level score that is hard to optimize directly with reinforcement learning. First a Soft Actor-Critic (SAC) agent is trained on a dense surrogate reward; then the policy is fine-tuned with the Separable Natural Evolution Strategy (SNES) directly on the true score. On the 2nd AI Olympics RealAIGym benchmark, the resulting EvolSAC controllers reach performance scores of 0.524 (acrobot) and 0.596 (pendubot), above SAC alone (0.504 and 0.567) and above the listed baselines, with only a small drop in robustness. On cartpole, the same fine-tuning cuts average swing-up time from 2.085 s to 1.135 s. The paper's point is that a cheap black-box optimization step can align an RL-trained policy with the objective that actually matters, without sacrificing robustness.","feed_headline":"Evolutionary fine-tuning lifts underactuated robot scores","feed_subtitle":"Adding SNES to SAC policies improves acrobot and pendubot scores while keeping robustness.","key_machinery":"The load-bearing object is the SNES fine-tuning step applied to all parameters of the SAC policy network. Separable Natural Evolution Strategy is a zero-order optimizer that keeps a separate step size per parameter dimension and updates the parameter vector by sampling Gaussian perturbations; the step sizes adapt through a log-normal rule. Because SNES evaluates whole trajectories, it can optimize the sparse competition score directly, including trajectory-level terms like success flags, which a per-transition reward cannot see. A second mechanism is the noise-injection procedure used during SNES evaluation: the agent samples its greedy action, undoes the tanh transform, adds Gaussian noise, and re-squashes, which keeps the policy stochastic and prevents SNES from overfitting to a single brittle trajectory.","core_discovery":"The central claim is that evolutionary fine-tuning directly on the evaluation score repairs the bias introduced by surrogate rewards in deep RL. Starting from a SAC policy trained on a dense reward, the paper applies SNES to the policy parameters and optimizes the raw competition score, with noise added before the tanh squashing to prevent the stochastic policy from collapsing to a deterministic one. In the AI Olympics double-pendulum benchmark this raises the best performance score from 0.504 to 0.524 in the acrobot and from 0.567 to 0.596 in the pendubot, while final score (performance plus robustness) also improves; robustness scores decline only from 0.700 to 0.692 and 0.800 to 0.796. On cartpole, SAC+SNES reduces average swing-up time from 2.085 s to 1.135 s, a 45.6 percent reduction, and TD3+SNES improves similarly, while additional SAC or TD3 training alone gives much smaller gains. The paper's stated assumption is that the SAC policy is already near-optimal, so SNES only has to make local adjustments.","pith_inferences":["A natural extension the paper does not run is to use the same two-phase recipe on other trajectory-level metrics, such as maximum energy, maximum velocity, or maximum action, where surrogate rewards are equally biased.","The local-search assumption implies the method will shine when RL already solves the task; for harder tasks, one could test restarting SNES from several SAC checkpoints or interleaving ES with continued RL to escape poor basins.","Treating the added noise during ES evaluation as an explicit robustness regularizer suggests a testable variant that optimizes the competition score plus a robustness penalty in the same ES loop.","The cartpole improvement comes from a low-cost ES refinement; a hardware test would show whether the gains survive real-world dynamics mismatch, since evaluation here is in simulation."],"forward_implications":["On the AI Olympics double pendulum, EvolSAC outperforms SAC and all listed baselines on performance score for both acrobot (0.524 vs 0.504) and pendubot (0.596 vs 0.567).","On cartpole, SAC+SNES cuts average swing-up time from 2.085 s to 1.135 s, and TD3+SNES from 2.525 s to 1.345 s, while continuing SAC or TD3 training alone yields smaller gains.","Robustness is largely preserved: the acrobot robustness score drops from 0.700 to 0.692 and the pendubot from 0.800 to 0.796, so final competition scores still improve.","Because the fine-tuning phase only needs the policy as a black box, the same recipe should transfer to other model-free and model-based RL algorithms."],"supporting_citations":[{"why":"Supplies the Soft Actor-Critic algorithm whose trained policy is the starting point for evolutionary fine-tuning.","marker":"Haarnoja et al. (2018)"},{"why":"Supplies the Separable Natural Evolution Strategy used as the zero-order fine-tuning optimizer.","marker":"Schaul et al. (2011)"},{"why":"Defines the AI Olympics performance score, robustness score, and the baseline controllers that EvolSAC is compared against.","marker":"Wiebe et al. (2025)"},{"why":"Supplies TD3, the second RL algorithm used to show the fine-tuning step generalizes beyond SAC.","marker":"Fujimoto et al. (2018)"},{"why":"Supplies a prior surrogate-reward approach for the underactuated double pendulum that the paper extends.","marker":"Turcato et al. (2024)"}],"fun_headline_variants":["SNES fine-tuning lifts SAC scores on underactuated robots","Evolutionary fine-tuning boosts acrobot and pendubot control","SAC plus SNES reduces swing-up time by 45% in AI Olympics","Fine-tune RL policies with evolutionary strategies for robust control"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method depends on the SAC-trained policy already being close to optimal, because SNES fine-tuning only makes local parameter adjustments and cannot escape a poor region of policy space.","fun_headline_variants_meta":{"raw":{"variants":["SNES fine-tuning lifts SAC scores on underactuated robots","Evolutionary fine-tuning boosts acrobot and pendubot control","SAC plus SNES reduces swing-up time by 45% in AI Olympics","Fine-tune RL policies with evolutionary strategies for robust control"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00021,"raw_usage":{"total_tokens":1411,"prompt_tokens":944,"completion_tokens":467,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":392}},"tokens_in":560,"tokens_out":467,"duration_ms":5437,"temperature":1.0,"reasoning_tokens":392,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:40:33.926244+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same SNES fine-tuning from SAC checkpoints taken at 500, 1000, and 2000 episodes; if the improvement over the un-finetuned SAC policy shrinks or vanishes as the starting policy worsens, the near-optimal warm start is doing the work.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Soft Actor-Critic algorithm whose trained policy is the starting point for evolutionary fine-tuning."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Separable Natural Evolution Strategy used as the zero-order fine-tuning optimizer."},{"cited_title":"Reinforcement Learning for Robust Athletic Intelligence: Lessons from the 2nd 'AI Olympics with RealAIGym' Competition","cited_arxiv_id":"2503.15290","evidence_quote":"Defines the AI Olympics performance score, robustness score, and the baseline controllers that EvolSAC is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies TD3, the second RL algorithm used to show the fine-tuning step generalizes beyond SAC."},{"cited_title":"Learning control of underactuated double pendulum with Model-Based Reinforcement Learning","cited_arxiv_id":"2409.05811","evidence_quote":"Supplies a prior surrogate-reward approach for the underactuated double pendulum that the paper extends."}],"review_version":1}