{"id":"3dd4bc5a-9d2c-4dce-8270-19b8df9e2521","arxiv_id":"2607.17326","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Under wall-clock budgets, sample-inefficient PPO on 2,048 parallel environments often reached a usable policy faster than sample-efficient SAC and TD-MPC2, while domain randomization offered no paradigm-specific robustness advantage.","lead":"A controlled benchmark of three RL paradigms — on-policy PPO, off-policy SAC, and model-based TD-MPC2 — shows that algorithm rankings flip between wall-clock time and interaction counts, and that domain randomization helps and hurts all three depending on task and coverage. For practitioners choosing transfer algorithms, it argues for judging methods by time-to-policy and robustness, not just sample efficiency.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Wall-clock axis is derived from an unspecified per-interaction time conversion (Appendix A), never validated against direct timing; if biased across PPO/SAC/TD-MPC2, Insight 1's ranking could flip.","rationale":"I read the paper as making two empirical claims: (1) under wall-clock, parallel PPO can reach a usable policy faster than single-env SAC/TD-MPC2; (2) DR effects are similar across paradigms. The first is the more distinctive and is directly tied to the wall-clock construction. The paper is honest about the configuration mismatch and hedges in §4.1 and the conclusion, so I do not treat that as a hidden flaw. The transfer claim rests on best-of-seed selection and fixed evaluation contexts, which is a secondary validity concern, but the primary measurement risk is the wall-clock conversion: the appendix gives one sentence, with no window definition, no overhead breakdown, no direct timing validation. If the conversion is biased, the headline ranking could be an artifact. The reader's weakest assumption identifies this same point; I agree. Because this is an addressable methodological gap and the paper is otherwise carefully hedged, the verdict should remain CONDITIONAL. The concrete test is a direct timing re-run on a representative subset.","tokens_in":16978,"tokens_out":6869,"duration_ms":77679,"concrete_test":"Instrument the training scripts to record wall-clock timestamps at each evaluation point for PPO, SAC, and TD-MPC2 on at least cheetah-run, finger-spin, walker-stand, and reacher-hard, including all overheads (startup, replay warmup, planning, evaluation, checkpointing). Plot return against actual elapsed time and compute time-to-threshold (e.g., first time a run exceeds 80% of the final best return). If PPO's wall-clock advantage over SAC/TD-MPC2 disappears or reverses on any task, the Appendix A conversion is biased and Insight 1 must be re-qualified; if it survives, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central time-to-policy claim (Insight 1) rests entirely on Appendix A's conversion: 'we averaged the elapsed time per environment interaction over a representative training window and used it to convert training steps into wall-clock time.' This is the only basis for the wall-clock curves in Figs. 1/B.1/B.3. The window is not specified; per-algorithm overheads are not itemized (startup, replay warmup, planning, periodic evaluation, checkpointing); and the conversion is never checked against directly measured timestamps. If per-interaction time is not stationary over training or if fixed overheads are non-negligible, the linear conversion distorts the time axis differently for each algorithm. For example, SAC's early random-exploration phase (if present) has lower per-step cost than the averaged window, so converting early steps with the average overstates SAC's early wall-clock time; TD-MPC2's planning cost and PPO's periodic evaluation add overheads not captured by per-step averages. Since the ranking of time-to-policy is the paper's headline, this unvalidated measurement assumption is load-bearing. The paper's own limitation section discloses the parallel-configuration confound but not this conversion.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares PPO, SAC, and TD-MPC2 on 18 continuous-control tasks under two evaluation axes — interaction-based sample efficiency and wall-clock training time — and across five domain-randomization (DR) training distributions with zero-shot transfer to nominal-centered and shifted evaluation regimes. The central claims are (i) that sample-inefficient PPO can nevertheless produce a performant policy faster in wall-clock time than SAC and TD-MPC2 under its massively parallel configuration, and (ii) that DR does not systematically favor any of the three paradigms, its effect being task-, algorithm-, coverage-, and regime-dependent. The body carefully hedges the wall-clock claim to the configurations studied, and the supplementary material provides full learning curves and transfer box plots.","tokens_in":17193,"tokens_out":5082,"duration_ms":56786,"significance":"If the results hold, the paper is a valuable methodological contribution: it demonstrates that algorithm rankings can invert under time-based budgets and that transfer-oriented evaluation should jointly consider sample efficiency, wall-clock cost, and robustness under dynamics mismatch. The study's scale is a strength: 18 tasks, five DR levels, two evaluation regimes, and standard reference implementations for all three algorithms. The reproduction of the conventional sample-efficiency ranking (SAC and TD-MPC2 ahead of PPO) under interaction-based evaluation is clean and gives credibility to the experimental setup. However, the headline wall-clock result rests on a wall-clock conversion procedure that is underspecified and unvalidated, and the transfer conclusions are based on best-of-seed summaries without variance accounting. These issues are load-bearing for the paper's two main insights.","major_comments":[{"comment":"The wall-clock axis for all time-based curves is not directly measured. Appendix A states only that 'we averaged the elapsed time per environment interaction over a representative training window and used it to convert training steps into wall-clock time.' The window is not defined, no direct timing validation is reported, and per-algorithm fixed costs (replay warmup, TD-MPC2 planning, periodic evaluation, checkpointing) and nonstationarity of per-step time are not accounted for. Since the ranking in Fig. 1 and the paper's first insight are built entirely on this conversion, this is a load-bearing measurement assumption. The authors should either report direct wall-clock timestamps or provide itemized per-algorithm timing and a sensitivity analysis of the chosen window.","section":"Appendix A / §4.1 / Fig. 1"},{"comment":"The zero-shot transfer results and the wall-clock curves are reported using a post-hoc best-of-seed selection. Appendix A says 'we select the best-performing run among the training seeds,' and Fig. 1's dark curves are similarly obtained by ranking runs by the sum of returns over the full recorded curve. Best-of-seed selection inflates expected performance and can favor algorithms with higher run-to-run variance; no standard-error or individual-seed summary is provided for the transfer comparisons. This weakens the claim that DR 'affects all three algorithms in a similar way,' because the apparent similarity may partly reflect selection across highly variable runs. Please report all seeds or a summary statistic that does not depend on hindsight selection.","section":"§4.3 / Appendix A / Figs. 3–4"},{"comment":"The claim that 'domain randomization affects all three algorithms in a similar way' is not operationalized. The body itself concludes more cautiously that the effect is 'task-, algorithm-, coverage-, and regime-dependent' and that broader coverage does not consistently improve transfer. The box plots in Figs. 3 and 4 show overlapping distributions with no statistical comparisons, effect sizes, or hypothesis tests. As stated, the abstract overstates the evidence. The authors should either provide a quantitative definition of 'similar way' (e.g., interaction effects in a regression, paired comparisons across tasks) or soften the claim to match the descriptive evidence.","section":"Abstract / §4.3"}],"minor_comments":[{"comment":"Typo: 'an insufficient criteria' should be 'an insufficient criterion.' Also, the abstract's statement that 'the sample-inefficient PPO algorithm can produce a performant policy faster' should carry the configuration qualifier used in §4.1 ('under its commonly adopted massively parallel configuration relative to the SAC and TD-MPC2 default configurations considered in this study').","section":"Abstract"},{"comment":"The term 'representative training window' is undefined. Even if the conversion method is retained, the authors must specify the window (e.g., which training steps, how many interactions, how the average was computed) and justify that it is representative for all three algorithms.","section":"Appendix A"},{"comment":"The caption says 'log-scale' and 'for an extended period of time,' but the x-axis appears to be linear seconds ranging from 0 to 30000. This inconsistency should be corrected.","section":"Fig. B.3 caption"},{"comment":"Minor grammatical issue: 'Narendra et al. (2025) noted diminishing stability of TD-MPC2 when utilized with a higher-degree of parallelism' — 'utilized with' should likely be 'utilized at' or 'under.'","section":"Related Work"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and practically important question, and the experimental breadth is commendable. My main concern is that the headline wall-clock insight is not yet established because the time axis is constructed by an underspecified and unvalidated conversion. This is fixable by re-analyzing or re-running with direct timing, but it is central enough that I cannot recommend acceptance without it. The best-of-seed issue also needs a more careful treatment, especially for the transfer claims. I would encourage the editor to seek a revision rather than reject, since the core setup and the interaction-based results are solid."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The controlled DR-coverage comparison is the real contribution here. A systematic sweep across 18 DMC tasks, five randomization levels, and two fixed evaluation regimes, with the same transfer protocol applied to PPO, SAC, and TD-MPC2, is something I haven't seen done cleanly before. The interaction-based results reproduce the conventional sample-efficiency ranking, and the body is mostly careful to hedge the wall-clock claim to the specific configurations used. The authors also disclose the parallel-configuration confound in their limitations, which is more than many benchmarks do.\n\nThe soft spot is the wall-clock axis, and it is load-bearing. Appendix A says the authors averaged elapsed time per environment interaction over a \"representative training window\" and used that to convert steps into seconds. The window is unspecified, per-algorithm overheads (replay warmup, planning, periodic evaluation, checkpointing) are not itemized, and the conversion is never checked against directly measured timestamps. If per-step cost is not stationary—SAC's early random exploration is cheaper than its later replay updates, TD-MPC2's planning cost varies, PPO's periodic evaluation adds fixed overhead—then a linear conversion distorts the time axis differently for each algorithm. That could flip the very ranking the abstract leads with. The body is more careful than the abstract here, but the abstract's unqualified \"PPO can produce a performant policy faster\" needs the Section 4.1 qualifier, and the paper should either report direct wall-clock times or at least run a sensitivity analysis over the conversion window.\n\nThe transfer results have a secondary weakness: best-of-six seed selection with no significance tests or confidence intervals. The claim that DR effects are task-, algorithm-, coverage-, and regime-dependent is a plausible negative result, and the box plots support it as a tendency, but \"no consistent advantage\" is a stronger statement than the evidence strictly licenses. This is a minor issue relative to the wall-clock conversion, since the DR comparison is descriptive and the trend is visible in the figures.\n\nOverall the paper is honest, the empirical work is broad, and the central DR finding is new. The wall-clock insight is confirmatory of prior work, which the authors acknowledge. For a practitioner audience this is still useful: it consolidates known wisdom into a systematic benchmark.\n\nI would send this to peer review. The fix is concrete—measure wall-clock directly or validate the conversion—and the rest of the paper is solid enough to be worth referee time.","headline":"Solid empirical benchmark with a genuinely new DR-coverage comparison, but the wall-clock headline rests on an unvalidated per-interaction time conversion that needs direct timing before it is trusted.","tokens_in":17775,"tokens_out":1121,"would_cite":true,"duration_ms":14841,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Sample efficiency alone can mislead RL algorithm choice for transfer; wall-clock time and domain-randomization robustness matter.","keywords":["reinforcement learning","transfer","domain randomization","wall-clock evaluation","sample efficiency","PPO","SAC","TD-MPC2"],"falsifier":"Directly measure end-to-end wall-clock training time for PPO, SAC, and TD-MPC2 on the same tasks and hardware, then compare time-to-threshold (e.g., reaching 80% of final return) on cheetah-run and walker-walk; if SAC or TD-MPC2 reaches the threshold before PPO in the majority of tasks, the claimed wall-clock advantage fails. Alternatively, training SAC and TD-MPC2 with 2,048 parallel environments and observing their time-to-policy drop below PPO's would show that the result stems from configuration rather than algorithm family.","tokens_in":16760,"feed_emoji":"🕒","tokens_out":3442,"duration_ms":36134,"temperature":0.7,"pith_summary":"This paper argues that for transfer-oriented reinforcement learning, choosing an algorithm by sample efficiency alone can mislead practitioners. It compares PPO, SAC, and TD-MPC2 under two evaluation axes—interactions and wall-clock time—and finds that massively parallel PPO often produces a usable policy sooner in wall-clock terms even though it consumes more interactions. The paper also varies domain randomization coverage across five levels and reports that all three algorithm families benefit similarly; none is systematically more robust. The authors conclude that algorithm suitability depends on the evaluation axis and on how the training distribution is constructed, not just on sample efficiency.","feed_headline":"Wall-clock time overturns RL sample-efficiency rankings","feed_subtitle":"Massively parallel PPO reaches usable policies sooner, and domain randomization helps all three algorithm families alike.","key_machinery":"The paper's central device is the separation of evaluation axes: interaction-based evaluation (returns versus environment samples) versus time-based evaluation (returns versus wall-clock time), with wall-clock time reconstructed by converting training steps using an averaged elapsed time per interaction. Transfer is formalized with a contextual Markov decision process in which training and evaluation draw context parameters (mass, friction, damping, etc.) from different distributions, and domain randomization is treated as shaping the training context distribution. This setup lets the authors study how the support of the training distribution relates to the evaluation contexts, and how that","core_discovery":"Under the configurations studied—PPO with 2,048 parallel environments versus SAC and TD-MPC2 in their standard single-environment setups—the sample-inefficient PPO frequently reaches strong performance earlier on a wall-clock axis, even though SAC and TD-MPC2 remain more sample-efficient on an interaction axis. This wall-clock advantage comes from PPO's ability to generate interactions rapidly through massive parallelism, not from more efficient use of each interaction. In transfer experiments with domain randomization, the paper finds no consistent advantage for any algorithmic paradigm: PPO, SAC, and TD-MPC2 can all benefit from randomized training, and the effect depends on the task, the","pith_inferences":["The wall-clock ranking is tied to PPO's 2048-environment configuration; running SAC and TD-MPC2 with comparable parallelism—an active research direction the authors cite—could narrow or reverse the gap, so the result may be configuration-dependent rather than paradigm-dependent.","The conversion from training steps to wall-clock time averages per-interaction elapsed time over an unspecified window and may miss fixed costs such as model-based planning overhead, replay sampling, or logging; direct end-to-end timing would strengthen or weaken the practical-efficiency claim.","If domain randomization effects are truly paradigm-agnostic, then the choice of randomization schedule and coverage may be a more important design lever for sim-to-real robustness than the choice among these algorithm families.","The zero-shot sim-to-sim protocol could be extended to sim-to-real, where wall-clock includes real-environment interaction and per-step time is far higher; such an extension could change the relative ranking more dramatically than the sim-to-sim results suggest."],"forward_implications":["Transfer-oriented RL evaluations should report time-to-policy alongside sample efficiency, because the algorithm ranking can reverse between the two axes.","The common use of PPO in sim-to-real pipelines may be explained by its ability to exploit massive parallel simulation, not by superior sample efficiency.","Domain randomization can be applied to off-policy and model-based algorithms (SAC and TD-MPC2) without expecting a systematic incompatibility.","Increasing domain randomization coverage does not guarantee better zero-shot transfer; the relationship between training support and evaluation support matters more than coverage alone.","Standard interaction-based benchmark rankings may not predict which algorithm performs best under realistic wall-clock constraints in transfer workflows."],"fun_headline_variants":["Wall-clock time flips RL algorithm rankings","PPO wins via parallel speed, not sample efficiency","Domain randomization aids all RL paradigms equally","Sample efficiency isn't enough: wall-clock matters","Parallel PPO overtakes sample-efficient SAC and TD-MPC2"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The wall-clock rankings rest on an assumed conversion from interaction counts to time using an averaged per-interaction elapsed time over an unspecified training window, rather than on direct wall-clock measurement; if that conversion is biased across algorithms with very different per-step compute, the paper's headline result is not established.","fun_headline_variants_meta":{"raw":{"variants":["Wall-clock time flips RL algorithm rankings","PPO wins via parallel speed, not sample efficiency","Domain randomization aids all RL paradigms equally","Sample efficiency isn't enough: wall-clock matters","Parallel PPO overtakes sample-efficient SAC and TD-MPC2"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000171,"raw_usage":{"total_tokens":1126,"prompt_tokens":783,"completion_tokens":343,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":270}},"tokens_in":527,"tokens_out":343,"duration_ms":4227,"temperature":1.0,"reasoning_tokens":270,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T18:20:29.115603+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Directly measure end-to-end wall-clock training time for PPO, SAC, and TD-MPC2 on the same tasks and hardware, then compare time-to-threshold (e.g., reaching 80% of final return) on cheetah-run and walker-walk; if SAC or TD-MPC2 reaches the threshold before PPO in the majority of tasks, the claimed wall-clock advantage fails. Alternatively, training SAC and TD-MPC2 with 2,048 parallel environments and observing their time-to-policy drop below PPO's would show that the result stems from configuration rather than algorithm family.","supporting_citations":[],"review_version":1}