{"id":"a417eac4-78b3-433e-ae20-19151d3e2a4a","arxiv_id":"2509.06714","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"RT-HCP combines a physics-informed model, multi-step planning, and an actor-critic policy to learn a swing-up controller on a real Furuta pendulum under strict time and sample limits.","lead":"This paper presents RT-HCP, a reinforcement learning method that lets a robot keep control while the AI computes slowly, by planning and replaying a short sequence of actions. It combines physics-based models with learned corrections to train on a real pendulum with far fewer interactions, reaching the control task in about 20 minutes.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-training-seed comparison undermines the reported sample-efficiency advantage","rationale":"The reader's weakest_assumption focuses on the residual neural network's out-of-distribution generalization (Section IV-B). That is a plausible mechanism-level concern, and Figure 7 provides some supporting evidence, but the paper's central empirical claim is the sample-efficiency comparison. The single-seed issue is more directly load-bearing: if a different seed changes the ordering of threshold-reaching steps, the headline result collapses, regardless of whether the residual network generalizes. The reader's rationale already mentions the single-seed limitation, but the formal weakest_assumption field does not elevate it to the primary concern. I choose it because it is unaddressed in the paper and threatens the strongest quantitative claim. The verdict remains CONDITIONAL: the method is plausible and the real-robot experiment is valuable, but multi-seed validation is required before accepting the specific 60k/100k/160k numbers. My proposed test is feasible on the same platform, albeit expensive; a smaller number of seeds (e.g., 3) would still provide evidence about variance. I do not see a basis to reject the paper outright, as the framework and preliminary results are meaningful, but the current evidence is insufficient for unconditional acceptance.","tokens_in":9484,"tokens_out":6132,"duration_ms":63987,"concrete_test":"Run each method (RT-HCP, TD-MPC, TD3) with at least 5 independent training seeds on the same physical platform, interleaved over time. For each seed, record the first training step at which the mean evaluation reward (over 10 episodes) exceeds 300. Report the median and IQR of these threshold steps per method, and perform a permutation or Mann-Whitney U test comparing RT-HCP against each baseline. If the threshold-step distributions overlap substantially (e.g., p > 0.05 or IQRs overlap), the 60k-vs-100k-vs-160k claim is not supported. Additionally, report the run order and any hardware state measurements to rule out time-of-day effects.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that RT-HCP reaches reward 300 in 60k steps vs 100k/160k for TD-MPC/TD3 rests on a single training run per method (Section V-B, Fig. 4). The 95% CIs in Fig. 4 are computed over 10 evaluation episodes of a single policy, not over independent training seeds, so they do not reflect training stochasticity. Real-robot RL is high-variance: different initializations, exploration noise, and hardware drift can shift the threshold-reaching step by tens of thousands of steps. The Welch t-test only indicates that evaluation returns differ at certain training steps; it does not establish that the threshold step is significantly smaller. Moreover, the paper does not report whether the runs were interleaved or controlled for hardware state (e.g., motor temperature). If baselines were trained at different times, systematic hardware drift could bias the comparison. Thus the quantitative sample-efficiency advantage is not statistically supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses two obstacles to learning RL controllers directly on real robots: sample inefficiency and inference delays. It introduces a delay-MDP formalization in which d-step MPC plans sequences of d actions and the state is augmented with missed states and buffered actions to (allegedly) restore the Markov property. On top of this framework, the authors propose RT-HCP, a hybrid MBRL/MFRL controller that combines a physics-informed residual model, CEM planning seeded by a learned policy, and an actor-critic trained on real and imagined data. The method is evaluated on a real Furuta pendulum under a 20 ms control period, comparing against TD3, TD-MPC, and PETS adapted with the same delay-management framework. The paper claims RT-HCP reaches a reward threshold of 300 after 60k training steps, versus 100k for TD-MPC and 160k for TD3, and also reports faster, more stable swing-up and better trajectory prediction accuracy.","tokens_in":9741,"tokens_out":8795,"duration_ms":95021,"significance":"If the results hold, this is a useful contribution to real-world MBRL: it tackles a practical constraint (inference delay) that is often ignored, combines physical priors with learned residuals in a way that is plausible for high-frequency control, and reports real-hardware experiments rather than simulation-only results. The paper ships code, which aids reproducibility. The strongest aspects are the measured inference-time table (Table I), the explicit integration of delay handling into the learning loop, and the real-robot evaluation. However, the central quantitative claim rests on a single training seed, and the framework's benefit is not isolated against simple action-repetition baselines, so the current evidence is not yet at the level needed for a strong acceptance.","major_comments":[{"comment":"The central sample-efficiency claim (RT-HCP reaches reward 300 at 60k steps vs 100k/160k for TD-MPC/TD3) is based on a single training seed per method. The 95% confidence intervals are computed over 10 evaluation episodes of one policy, not over independent training runs. The Welch t-test at specific training steps does not establish that the threshold-crossing step is significantly smaller for RT-HCP. Run interleaving and hardware state (e.g., motor temperature) are also not reported. This is load-bearing: the quantitative advantage is not statistically supported. Please add multiple seeds or a seed-sensitivity study, report the distribution of threshold-crossing steps, and clarify how hardware conditions were controlled.","section":"Section V-B, Fig. 4"},{"comment":"The first contribution is a delay-MDP/d-step MPC framework, but the experiments never compare against the simplest existing remedy for inference delays: holding/repeating the last action while waiting. Since d-step MPC also fills time with a precomputed action buffer, an action-repetition baseline is needed to isolate the benefit of the augmented state and the planning-based action sequence. Without this baseline, it is unclear whether the observed gains come from the proposed delay-MDP construction or merely from having any action buffer instead of execution gaps.","section":"Section IV-A, Section V-B"},{"comment":"The claim that the physics-informed residual model mitigates compounding errors is central to the paper's explanation of why RT-HCP tolerates delays well. The residual MLP is trained on up to 200k real transitions, and Fig. 7 shows one qualitative trajectory. This is insufficient support for the out-of-distribution generalization on which the long-horizon prediction argument depends. Please add quantitative multi-step prediction error as a function of horizon, and evaluate the residual model on states/action sequences outside the training distribution. If such generalization is not guaranteed, the limitation should be stated explicitly.","section":"Section IV-B, Fig. 7"},{"comment":"The delay-MDP is described informally. The augmented state s'_t = {s_{d·t}, s_{d·t-(d-1)}, ..., s_{d·t-1}, a_{d·t}, ..., a_{d·t+(d-1)}} mixes previous missed states with the current buffer actions, and the transition function T' is only written as T^d without a formal definition. The key assertion that this restores the Markov property is not proven. Since this is contribution 1, please give a precise definition of the state/action spaces and the transition function, and state under what assumptions the augmented process is an MDP.","section":"Section IV-A"}],"minor_comments":[{"comment":"The assumption that inference time grows less than linearly with the planning horizon is not verified; only one H_p per model-based method is reported. Reporting T_i for at least two horizons per method would support the feasibility criterion in Section IV-A.","section":"Table I"},{"comment":"The reward threshold of 300 is used to define success but is not defined in the text. Please state what this threshold represents and how it was chosen.","section":"Section V-B, Fig. 4"},{"comment":"The notation is inconsistent: the maximization is over a_{t0:t0+H}, but the sum goes to t0+H and the text uses H_p elsewhere. Clarify the horizon indexing and the relationship between H and H_p.","section":"Eq. (1)"},{"comment":"There are typographical issues in the formalization: 'the reward function is also augmented to (R' = R^d)' and 'the transition function is updated to T = T d' need proper definitions of R' and T'. Also, the apostrophe notation for a'_{t+1} is confusing with the derivative notation.","section":"Section IV-A"},{"comment":"For 'rotor deviation from the center', clarify whether this is the absolute angular error and what units are used. This would make the control-performance comparison easier to interpret.","section":"Table II"}],"recommendation":"major_revision","confidential_remarks":"The manuscript fits the scope of the journal and the real-robot results are valuable. The main risk is the single-seed comparison; if additional seeds or a suitably softened claim are provided, the paper may be acceptable. The delay-MDP formalization also needs tightening. Self-citation of PHiHP is reasonable here, as the new contribution is the delay framework and real-time adaptation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before you read it. First, this is real-robot RL, not simulation: the experiments are on a physical FURUTA pendulum with a 200k-step budget. Second, the framework half is stronger than the evaluation half. The authors formalize multi-step inference delay — the controller takes several control periods to produce an action — as a delay-MDP with augmented state (current state, missed states, buffered actions) and d-step MPC, and they are right that prior delay work (Ramstedt, Xiao) handled only sub-step delays. The formalization is simple and coherent, and it directly motivates the method: RT-HCP plans short action sequences, seeds CEM with an actor-critic policy, and uses a physics-informed residual model so the open-loop execution horizon stays short.\n\nThe empirical section is the real payload. Table I's measured inference times on the actual hardware (TD3 16ms, TD-MPC 47ms, PETS 156ms, RT-HCP 36ms) ground the whole discussion. Adapting the model-based baselines to the same d-step MPC delay management is the right experimental choice: it isolates the algorithm. PETS failing outright under a 7.8-step delay, while hybrid methods succeed, is a genuine data point. The physics prior is well motivated — the paper is candid that it ignores friction and cable effects — and the trajectory-prediction plots support the residual model. Code is provided, and the link to their PHiHP prior is not circular: the central result is on external hardware with measured delays.\n\nThe soft spot is the single training seed. Section V-B reports the learning curves from 'a single policy trained with a fixed random seed,' and the 95% confidence intervals come from ten evaluation episodes of that one policy, not from independent training runs. So the 60k vs 100k vs 160k threshold-step counts are not statistically supported, and the Welch-test stars do not fix that: they compare returns at given steps, not the crossing point. The qualitative ordering is likely robust — the effects are large and mechanistically plausible — but the paper states the quantitative advantage without hedging. Either a few seeds or a reframed claim is needed. The stress-test also notes the runs' interleaving and hardware state (e.g., motor temperature) are not reported; those are open questions, not demonstrated biases. Two smaller gaps: no ablation against the natural default of repeating the last action during inference, and the physics-parameter values for the prior are not listed, which hurts reproducibility.\n\nVerdict: a solid, honest paper with one over-claimed statistic. Send it to a serious referee; the main asks are multi-seed or hedged results, an action-repetition ablation, and a paragraph on hardware-state control. I would bring it to our reading group as a useful case study in what real-hardware RL evidence can and cannot support.","headline":"Real-hardware RL paper with a clean multi-step delay framework and a genuine robot comparison; the sample-efficiency headline, however, rests on one training seed per method and needs re-running or hedging before it can carry that weight.","tokens_in":10165,"tokens_out":8042,"would_cite":true,"duration_ms":82813,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RT-HCP learns to control a real FURUTA pendulum in about 20 minutes of direct training, reaching a reward threshold in 60k steps versus 100k for TD-MPC and 160k for TD3.","keywords":["inference delay","model-based reinforcement learning","real-time control","physics-informed model","sample efficiency","Furuta pendulum","hybrid control","model predictive control"],"falsifier":"Replace RT-HCP's physics-informed model with a purely data-driven model (same d-step MPC and actor-critic, same inference time) on the same FURUTA setup; if the data-driven variant matches or beats RT-HCP on sample efficiency and swing-up stability, the paper's central claim that the physics prior is the source of the improvement is refuted. Alternatively, measure open-loop prediction error on rollouts starting from states far outside the 200k-transition training distribution: if the error grows faster than a data-driven ensemble's, the robustness claim fails.","tokens_in":9443,"feed_emoji":"🤖","tokens_out":5467,"duration_ms":54277,"temperature":0.7,"pith_summary":"RT-HCP is a reinforcement learning algorithm designed for learning controllers directly on real robots, where both training data and computation time are scarce. The paper's central claim is that by redefining the control problem around inference delay—the time the controller needs to compute an action—and by combining a physics-based model of the robot with a small residual neural network, a d-step model predictive control planner, and an actor-critic policy, a real FURUTA pendulum can be brought from rest to stable upright swing-up within 20 minutes of training. This matters because model-based RL is sample-efficient but usually too slow to meet control frequencies, while model-free RL is fast enough but needs far more interactions. The experiments on a high-frequency physical platform show RT-HCP reaching a task-completion reward in 60k training steps, versus 100k for TD-MPC and 160k for TD3, and with more stable swing-up.","feed_headline":"Real robot learns pendulum swing-up in 20 minutes","feed_subtitle":"RT-HCP blends a physics model with fast planning, beating TD-MPC and TD3 on the live FURUTA platform.","key_machinery":"The load-bearing object is the delay-MDP with d-step MPC: instead of computing one action per control period, the agent computes a sequence of d actions, buffers them, and augments its observation with the d-1 states it missed while computing and the d future actions not yet executed. This restores the Markov property under slow inference. RT-HCP's second key component is the physics-informed model: an analytical two-link rotary inverted pendulum model that captures the bulk of the dynamics, plus a small residual neural network that corrects for friction, cable effects, and parameter error, making open-loop trajectory prediction accurate enough for longer execution horizons.","core_discovery":"The paper proposes that inference delay should be treated as part of the MDP rather than as an engineering nuisance. It defines a delay-MDP in which the state is augmented with the d-1 missed states and the buffered d actions, restoring the Markov property, and uses d-step MPC to fill the execution gap with a precomputed action sequence. On top of this, RT-HCP uses a physics-informed model—an analytical Euler-Lagrange model of the FURUTA pendulum with a 4-layer, 16-neuron residual MLP trained on 200k real transitions—together with a hybrid planner that seeds CEM with actor-critic action candidates and uses a Q-value to estimate long-term return. The empirical claim is that this combination a","pith_inferences":["The framework's advantage should grow as the ratio of inference time to the control period grows, so a natural extension is to test it on platforms with tighter real-time constraints or more complex dynamics (e.g., a quadrotor) where the delay is larger relative to the control frequency.","Because the residual network is small and trained on only 200k transitions, the approach could be extended to image-based control by replacing the state input with a learned encoder, a direction the authors explicitly mention.","The delay-MDP state augmentation assumes the missed states are observable or predictable; a testable variant would let the residual model also predict missed states, which would matter when sensors drop or the action buffer timing is jittery."],"forward_implications":["Learning a controller directly on a physical, unstable, high-frequency system can be done within tens of minutes rather than hours, without needing sim-to-real transfer.","The delay-MDP and d-step MPC framework gives a general recipe for making any model-based RL algorithm real-time capable: measure the inference time, precompute an action sequence that covers it, and augment the state with missed states and buffered actions.","Hybrid planning that seeds CEM with actor-critic action candidates and a terminal Q-value reduces the population and iteration count needed, cutting inference delay without sacrificing performance.","A physics-informed model trained on real data reduces compounding prediction errors compared with purely data-driven models, which is especially important for longer open-loop execution horizons used to bridge inference delays."],"supporting_citations":[{"why":"Provides the TD-MPC hybrid model-based/model-free baseline that RT-HCP is compared against and whose Q-value-plus-planning structure RT-HCP builds on.","marker":"[6]"},{"why":"Introduces the physics-informed hybrid planning idea that RT-HCP adapts for real-time learning with real data instead of simulated rollouts alone.","marker":"[8]"},{"why":"Supplies the PETS sample-efficient model-based baseline whose accurate predictions still fail under large inference delay.","marker":"[4]"},{"why":"Provides the TD3 model-free baseline that has very short inference time but requires many more samples.","marker":"[19]"},{"why":"Names the FURUTA rotary inverted pendulum used as the unstable, high-frequency real-robot benchmark.","marker":"[9]"},{"why":"Defines the cross-entropy method used for planning action sequences, whose population and iteration settings directly determine inference time.","marker":"[7]"},{"why":"Supplies the background on delayed MDPs and state augmentation for restoring the Markov property, which the delay-MDP formulation extends.","marker":"[12]"}],"fun_headline_variants":["Delay as state: RT-HCP learns real robot in 20 minutes","Inference delay part of MDP: RT-HCP learns pendulum in 20 min","RT-HCP: delay-aware MDP learns pendulum in 20 minutes","Delay-aware RL: real pendulum learned in 20 minutes"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The small residual neural network, trained on 200k real transitions, must generalize to states outside its training distribution well enough that the physics-informed model stays accurate over the full planning horizon; if it does not, the claimed mitigation of compounding errors and the benefit of shorter horizons collapse.","fun_headline_variants_meta":{"raw":{"variants":["Delay as state: RT-HCP learns real robot in 20 minutes","Inference delay part of MDP: RT-HCP learns pendulum in 20 min","RT-HCP: delay-aware MDP learns pendulum in 20 minutes","Delay-aware RL: real pendulum learned in 20 minutes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001641,"raw_usage":{"total_tokens":6332,"prompt_tokens":695,"completion_tokens":5637,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":439,"completion_tokens_details":{"reasoning_tokens":5558}},"tokens_in":439,"tokens_out":5637,"duration_ms":38888,"temperature":1.0,"reasoning_tokens":5558,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T23:11:39.607961+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace RT-HCP's physics-informed model with a purely data-driven model (same d-step MPC and actor-critic, same inference time) on the same FURUTA setup; if the data-driven variant matches or beats RT-HCP on sample efficiency and swing-up stability, the paper's central claim that the physics prior is the source of the improvement is refuted. Alternatively, measure open-loop prediction error on rollouts starting from states far outside the 200k-transition training distribution: if the error grows faster than a data-driven ensemble's, the robustness claim fails.","supporting_citations":[{"cited_title":"Temporal difference learning for model predictive control,","cited_arxiv_id":null,"evidence_quote":"Provides the TD-MPC hybrid model-based/model-free baseline that RT-HCP is compared against and whose Q-value-plus-planning structure RT-HCP builds on."},{"cited_title":"Physics-informed model and hybrid planning for efficient dyna-style reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Introduces the physics-informed hybrid planning idea that RT-HCP adapts for real-time learning with real data instead of simulated rollouts alone."},{"cited_title":"Swing-up control of inverted pendulum using pseudo-state feedback,","cited_arxiv_id":null,"evidence_quote":"Names the FURUTA rotary inverted pendulum used as the unstable, high-frequency real-robot benchmark."},{"cited_title":"A tutorial on the cross-entropy method,","cited_arxiv_id":null,"evidence_quote":"Defines the cross-entropy method used for planning action sequences, whose population and iteration settings directly determine inference time."},{"cited_title":"Markov decision processes with delays and asynchronous cost collection,","cited_arxiv_id":null,"evidence_quote":"Supplies the background on delayed MDPs and state augmentation for restoring the Markov property, which the delay-MDP formulation extends."}],"review_version":1}