Pith. sign in

REVIEW 4 major objections 5 minor 19 references

RT-HCP learns to control a real FURUTA pendulum in about 20 minutes of direct training, reaching a reward threshold in 60k steps versus 100k for TD-MPC and 160k for TD3.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

RT-HCP combines a physics-informed model, multi-step planning, and an actor-critic policy to learn a swing-up controller on a real Furuta pendulum under strict time and sample limits.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection Real-hardware RL paper with a clean multi-step delay framework and a genuine robot comparison; the sample-efficiency headline, however, rests on one training seed per method and needs re-running or hedging before it can carry that weight. the 4 major comments →

arxiv 2509.06714 v1 pith:FI2EYHBQ submitted 2025-09-08 cs.LG

RT-HCP: Dealing with Inference Delays and Sample Efficiency to Learn Directly on Robotic Platforms

Zakariae El Asri , Ibrahim Laiche , Cl\'ement Rambour , Olivier Sigaud , Nicolas Thome This is my paper
classification cs.LG
keywords inference delaymodel-based reinforcement learningreal-time controlphysics-informed modelsample efficiencyFuruta pendulumhybrid controlmodel predictive control
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RT-HCP is a reinforcement learning algorithm designed for learning controllers directly on real robots, where both training data and computation time are scarce. The paper's central claim is that by redefining the control problem around inference delay—the time the controller needs to compute an action—and by combining a physics-based model of the robot with a small residual neural network, a d-step model predictive control planner, and an actor-critic policy, a real FURUTA pendulum can be brought from rest to stable upright swing-up within 20 minutes of training. This matters because model-based RL is sample-efficient but usually too slow to meet control frequencies, while model-free RL is fast enough but needs far more interactions. The experiments on a high-frequency physical platform show RT-HCP reaching a task-completion reward in 60k training steps, versus 100k for TD-MPC and 160k for TD3, and with more stable swing-up.

Core claim

The paper proposes that inference delay should be treated as part of the MDP rather than as an engineering nuisance. It defines a delay-MDP in which the state is augmented with the d-1 missed states and the buffered d actions, restoring the Markov property, and uses d-step MPC to fill the execution gap with a precomputed action sequence. On top of this, RT-HCP uses a physics-informed model—an analytical Euler-Lagrange model of the FURUTA pendulum with a 4-layer, 16-neuron residual MLP trained on 200k real transitions—together with a hybrid planner that seeds CEM with actor-critic action candidates and uses a Q-value to estimate long-term return. The empirical claim is that this combination a

What carries the argument

The load-bearing object is the delay-MDP with d-step MPC: instead of computing one action per control period, the agent computes a sequence of d actions, buffers them, and augments its observation with the d-1 states it missed while computing and the d future actions not yet executed. This restores the Markov property under slow inference. RT-HCP's second key component is the physics-informed model: an analytical two-link rotary inverted pendulum model that captures the bulk of the dynamics, plus a small residual neural network that corrects for friction, cable effects, and parameter error, making open-loop trajectory prediction accurate enough for longer execution horizons.

Load-bearing premise

The small residual neural network, trained on 200k real transitions, must generalize to states outside its training distribution well enough that the physics-informed model stays accurate over the full planning horizon; if it does not, the claimed mitigation of compounding errors and the benefit of shorter horizons collapse.

What would settle it

Replace RT-HCP's physics-informed model with a purely data-driven model (same d-step MPC and actor-critic, same inference time) on the same FURUTA setup; if the data-driven variant matches or beats RT-HCP on sample efficiency and swing-up stability, the paper's central claim that the physics prior is the source of the improvement is refuted. Alternatively, measure open-loop prediction error on rollouts starting from states far outside the 200k-transition training distribution: if the error grows faster than a data-driven ensemble's, the robustness claim fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Learning a controller directly on a physical, unstable, high-frequency system can be done within tens of minutes rather than hours, without needing sim-to-real transfer.
  • The delay-MDP and d-step MPC framework gives a general recipe for making any model-based RL algorithm real-time capable: measure the inference time, precompute an action sequence that covers it, and augment the state with missed states and buffered actions.
  • Hybrid planning that seeds CEM with actor-critic action candidates and a terminal Q-value reduces the population and iteration count needed, cutting inference delay without sacrificing performance.
  • A physics-informed model trained on real data reduces compounding prediction errors compared with purely data-driven models, which is especially important for longer open-loop execution horizons used to bridge inference delays.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The framework's advantage should grow as the ratio of inference time to the control period grows, so a natural extension is to test it on platforms with tighter real-time constraints or more complex dynamics (e.g., a quadrotor) where the delay is larger relative to the control frequency.
  • Because the residual network is small and trained on only 200k transitions, the approach could be extended to image-based control by replacing the state input with a learned encoder, a direction the authors explicitly mention.
  • The delay-MDP state augmentation assumes the missed states are observable or predictable; a testable variant would let the residual model also predict missed states, which would matter when sensors drop or the action buffer timing is jittery.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper addresses two obstacles to learning RL controllers directly on real robots: sample inefficiency and inference delays. It introduces a delay-MDP formalization in which d-step MPC plans sequences of d actions and the state is augmented with missed states and buffered actions to (allegedly) restore the Markov property. On top of this framework, the authors propose RT-HCP, a hybrid MBRL/MFRL controller that combines a physics-informed residual model, CEM planning seeded by a learned policy, and an actor-critic trained on real and imagined data. The method is evaluated on a real Furuta pendulum under a 20 ms control period, comparing against TD3, TD-MPC, and PETS adapted with the same delay-management framework. The paper claims RT-HCP reaches a reward threshold of 300 after 60k training steps, versus 100k for TD-MPC and 160k for TD3, and also reports faster, more stable swing-up and better trajectory prediction accuracy.

Significance. If the results hold, this is a useful contribution to real-world MBRL: it tackles a practical constraint (inference delay) that is often ignored, combines physical priors with learned residuals in a way that is plausible for high-frequency control, and reports real-hardware experiments rather than simulation-only results. The paper ships code, which aids reproducibility. The strongest aspects are the measured inference-time table (Table I), the explicit integration of delay handling into the learning loop, and the real-robot evaluation. However, the central quantitative claim rests on a single training seed, and the framework's benefit is not isolated against simple action-repetition baselines, so the current evidence is not yet at the level needed for a strong acceptance.

major comments (4)
  1. [Section V-B, Fig. 4] The central sample-efficiency claim (RT-HCP reaches reward 300 at 60k steps vs 100k/160k for TD-MPC/TD3) is based on a single training seed per method. The 95% confidence intervals are computed over 10 evaluation episodes of one policy, not over independent training runs. The Welch t-test at specific training steps does not establish that the threshold-crossing step is significantly smaller for RT-HCP. Run interleaving and hardware state (e.g., motor temperature) are also not reported. This is load-bearing: the quantitative advantage is not statistically supported. Please add multiple seeds or a seed-sensitivity study, report the distribution of threshold-crossing steps, and clarify how hardware conditions were controlled.
  2. [Section IV-A, Section V-B] The first contribution is a delay-MDP/d-step MPC framework, but the experiments never compare against the simplest existing remedy for inference delays: holding/repeating the last action while waiting. Since d-step MPC also fills time with a precomputed action buffer, an action-repetition baseline is needed to isolate the benefit of the augmented state and the planning-based action sequence. Without this baseline, it is unclear whether the observed gains come from the proposed delay-MDP construction or merely from having any action buffer instead of execution gaps.
  3. [Section IV-B, Fig. 7] The claim that the physics-informed residual model mitigates compounding errors is central to the paper's explanation of why RT-HCP tolerates delays well. The residual MLP is trained on up to 200k real transitions, and Fig. 7 shows one qualitative trajectory. This is insufficient support for the out-of-distribution generalization on which the long-horizon prediction argument depends. Please add quantitative multi-step prediction error as a function of horizon, and evaluate the residual model on states/action sequences outside the training distribution. If such generalization is not guaranteed, the limitation should be stated explicitly.
  4. [Section IV-A] The delay-MDP is described informally. The augmented state s'_t = {s_{d·t}, s_{d·t-(d-1)}, ..., s_{d·t-1}, a_{d·t}, ..., a_{d·t+(d-1)}} mixes previous missed states with the current buffer actions, and the transition function T' is only written as T^d without a formal definition. The key assertion that this restores the Markov property is not proven. Since this is contribution 1, please give a precise definition of the state/action spaces and the transition function, and state under what assumptions the augmented process is an MDP.
minor comments (5)
  1. [Table I] The assumption that inference time grows less than linearly with the planning horizon is not verified; only one H_p per model-based method is reported. Reporting T_i for at least two horizons per method would support the feasibility criterion in Section IV-A.
  2. [Section V-B, Fig. 4] The reward threshold of 300 is used to define success but is not defined in the text. Please state what this threshold represents and how it was chosen.
  3. [Eq. (1)] The notation is inconsistent: the maximization is over a_{t0:t0+H}, but the sum goes to t0+H and the text uses H_p elsewhere. Clarify the horizon indexing and the relationship between H and H_p.
  4. [Section IV-A] There are typographical issues in the formalization: 'the reward function is also augmented to (R' = R^d)' and 'the transition function is updated to T = T d' need proper definitions of R' and T'. Also, the apostrophe notation for a'_{t+1} is confusing with the derivative notation.
  5. [Table II] For 'rotor deviation from the center', clarify whether this is the absolute angular error and what units are used. This would make the control-performance comparison easier to interpret.

Circularity Check

0 steps flagged

No significant circularity: the central real-robot results are measured against external baselines; the only self-citation is motivational and non-load-bearing.

full rationale

The paper's central claims are validated by direct experiments on a physical FURUTA pendulum against established external baselines (TD3, TD-MPC, PETS). The d-step MPC delay framework calibrates the execution horizon H_e from measured inference times (Table I), which is an experimental setup choice, not a fitted parameter later renamed as a prediction. The physics-informed model, while building on the authors' prior PHiHP work [8], is restated with explicit equations in Section V-A, so the method does not depend on an unverified self-cited result. The self-citation to [8] appears in related work and in the description of the approach, but it is not used as a load-bearing theorem or uniqueness argument. No equation in the paper reduces by construction to an input, and no fitted value is reported as a predicted outcome. The single-seed training comparison noted by the skeptic is a statistical robustness concern, not a circularity concern.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The central claim depends on several hand-set hyperparameters (horizon, CEM budget, network architecture) and on modeling assumptions about the physics prior and the state augmentation restoring the Markov property. No new physical entities are introduced.

free parameters (5)
  • Planning horizon H_p = 5 for RT-HCP (5 for RT-TDMPC, 15 for PETS)
    Tuned to balance inference time and performance; directly sets the inference delay d and the execution horizon.
  • CEM population size P = 500
    Chosen by hand for RT-HCP; larger populations improve performance but increase inference time.
  • CEM iterations I = 3
    Chosen to keep inference time within the real-time budget.
  • Number of policy-seeded CEM samples = 50
    Part of the 500 total action candidates; chosen by hand to accelerate convergence of CEM.
  • Execution horizon H_e = 2 for RT-HCP
    Derived as H_e = int(T_i / Delta_t) + 1 from the measured inference time; determines the action buffer length.
axioms (4)
  • domain assumption The Euler-Lagrange frictionless two-link model approximates the FURUTA pendulum, and a residual neural network compensates for unmodeled friction, cable effects, and parameter error.
    Stated in Section V-A and used in Section IV-B; the accuracy of this hybrid model is central to planning and to the claimed mitigation of compounding errors.
  • domain assumption State augmentation with missed states and pending actions restores the Markov property of the delay-MDP.
    Section IV-A; this assumes the augmented state and action sequence are sufficient statistics for the transition under inference delay.
  • domain assumption Inference time grows less than linearly with planning horizon, i.e., there exists H such that T_i^H < H * Delta_t.
    Section IV-A; this feasibility assumption is stated but not proven for the real hardware.
  • domain assumption CEM with a small population and few iterations, seeded by the actor policy, finds a sufficiently good action sequence.
    Section IV-B; relies on the policy providing informative candidates so that the reduced CEM budget remains effective.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of RT-HCP: Dealing with Inference Delays and Sample Efficiency to Learn Directly on Robotic Platforms." pith.science (2026). https://pith.science/paper/FI2EYHBQ

@misc{pith2026250906714,
  author       = {Pith},
  title        = {Pith review of: RT-HCP: Dealing with Inference Delays and Sample Efficiency to Learn Directly on Robotic Platforms},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FI2EYHBQ}},
  note         = {Machine review of arXiv:2509.06714}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Learning a controller directly on the robot requires extreme sample efficiency. Model-based reinforcement learning (RL) methods are the most sample efficient, but they often suffer from a too long inference time to meet the robot control frequency requirements. In this paper, we address the sample efficiency and inference time challenges with two contributions. First, we define a general framework to deal with inference delays where the slow inference robot controller provides a sequence of actions to feed the control-hungry robotic platform without execution gaps. Then, we compare several RL algorithms in the light of this framework and propose RT-HCP, an algorithm that offers an excellent trade-off between performance, sample efficiency and inference time. We validate the superiority of RT-HCP with experiments where we learn a controller directly on a simple but high frequency FURUTA pendulum platform. Code: github.com/elasriz/RTHCP

Figures

Figures reproduced from arXiv: 2509.06714 by Cl\'ement Rambour, Ibrahim Laiche, Nicolas Thome, Olivier Sigaud, Zakariae El Asri.

Figure 1
Figure 1. Figure 1: In the case of inference delay, the agent requires [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Schematic description of RT-HCP 3) Learning in imagination: The updated model periodi￾cally generates synthetic transitions into Dim to further refine π. This real-time learning loop ensures that π is continu￾ously refined using both real-world and model-based data, achieving higher sample efficiency while maintaining real￾time execution constraints. RT-HCP is particularly well suited for real-time applica… view at source ↗
Figure 3
Figure 3. Figure 3: The Furuta pendulum (schematic on the left, experi [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Representative trajectory for each agent, showing the [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 4
Figure 4. Figure 4: presents the evolution of episodic reward through￾out training. The results are averaged over 10 evaluation episodes using a single policy trained with a fixed random seed. The results show that RT-HCP achieves higher sample efficiency and better overall performance than TD-MPC and TD3. Specifically, RT-HCP reaches a threshold reward of 300, which is sufficient to successfully complete the task, after only… view at source ↗
Figure 6
Figure 6. Figure 6: Evolution of the average return as a function of the [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Trajectory predictions vs. ground truth from the same [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

19 extracted references · 10 canonical work pages

  1. [1]

    Mastering atari games with limited data,

    W. Ye, S.-W. Liu, T. Kurutach, P. Abbeel, and Y . Gao, “Mastering atari games with limited data,”ArXiv, vol. abs/2111.00210, 2021

    Pith/arXiv arXiv 2021

  2. [2]

    Benchmarking deep reinforcement learning for continuous control,

    Y . Duan, X. Chen, R. Houthooft, J. Schulman, and P. Abbeel, “Benchmarking deep reinforcement learning for continuous control,” inInternational Conference on Machine Learning, 2016

    work page 2016

  3. [3]

    Challenges of real-world reinforcement learning,

    G. Dulac-Arnold, D. Mankowitz, and T. Hester, “Challenges of real-world reinforcement learning,”arXiv preprint arXiv:1904.12901, 2019

    Pith/arXiv arXiv 1904

  4. [4]

    Deep rein- forcement learning in a handful of trials using probabilistic dynamics models,

    K. Chua, R. Calandra, R. McAllister, and S. Levine, “Deep rein- forcement learning in a handful of trials using probabilistic dynamics models,”Advances in neural information processing systems, vol. 31, 2018

    2018

  5. [5]

    pytorch implementation of PETS,

    “pytorch implementation of PETS,” 2019. [Online]. Available: https://github.com/quanvuong/handful-of-trials-pytorch

    work page 2019

  6. [6]

    Temporal difference learning for model predictive control,

    N. Hansen, X. Wang, and H. Su, “Temporal difference learning for model predictive control,” inICML, 2022

    work page 2022

  7. [7]

    A tutorial on the cross-entropy method,

    P. T. de Boer, D. P. Kroese, S. Mannor, and R. Y . Rubinstein, “A tutorial on the cross-entropy method,”Annals of Operations Research, vol. 134, pp. 19–67, 2005

    work page 2005

  8. [8]

    Physics-informed model and hybrid planning for efficient dyna-style reinforcement learning,

    Z. El asri, O. Sigaud, and N. Thome, “Physics-informed model and hybrid planning for efficient dyna-style reinforcement learning,” Reinforcement Learning Journal, vol. 1, 2024

    work page 2024

  9. [9]

    Swing-up control of inverted pendulum using pseudo-state feedback,

    K. Furuta, M. Yamakita, and S. Kobayashi, “Swing-up control of inverted pendulum using pseudo-state feedback,”Proceedings of the Institution of Mechanical Engineers, Part I: Journal of Systems and Control Engineering, vol. 206, pp. 263 – 269, 1992

    work page 1992

  10. [10]

    Exploring model-based planning with policy networks,

    T. Wang and J. Ba, “Exploring model-based planning with policy networks,”arXiv preprint arXiv:1906.08649, 2019

    Pith/arXiv arXiv 1906

  11. [11]

    Blending mpc & value function approximation for efficient reinforcement learning,

    M. Bhardwaj, S. Choudhury, and B. Boots, “Blending mpc & value function approximation for efficient reinforcement learning,”arXiv preprint arXiv:2012.05909, 2020

    Pith/arXiv arXiv 2012

  12. [12]

    Markov decision processes with delays and asynchronous cost collection,

    K. Katsikopoulos and S. Engelbrecht, “Markov decision processes with delays and asynchronous cost collection,”IEEE Transactions on Automatic Control, vol. 48, no. 4, pp. 568–574, 2003

    work page 2003

  13. [13]

    Delay-aware model-based reinforcement learning for continuous control,

    B. Chen, M. Xu, L. Li, and D. Zhao, “Delay-aware model-based reinforcement learning for continuous control,”Neurocomputing, vol. 450, pp. 119–128, 2020

    work page 2020

  14. [14]

    Rein- forcement learning with random delays,

    S. Ramstedt, Y . Bouteiller, G. Beltrame, C. J. Pal, and J. Binas, “Rein- forcement learning with random delays,”ArXiv, vol. abs/2010.02966, 2020

    Pith/arXiv arXiv 2010

  15. [15]

    Real-time reinforcement learning,

    S. Ramstedt and C. J. Pal, “Real-time reinforcement learning,” in Neural Information Processing Systems, 2019

    work page 2019

  16. [16]

    Thinking while moving: Deep reinforcement learning with concurrent control,

    T. Xiao, E. Jang, D. Kalashnikov, S. Levine, J. Ibarz, K. Hausman, and A. Herzog, “Thinking while moving: Deep reinforcement learning with concurrent control,”ArXiv, vol. abs/2004.06089, 2020

    Pith/arXiv arXiv 2004

  17. [17]

    Asynchronous reinforcement learning for real-time control of physical robots,

    Y . Yuan and R. Mahmood, “Asynchronous reinforcement learning for real-time control of physical robots,”2022 International Conference on Robotics and Automation (ICRA), pp. 5546–5552, 2022

    work page 2022

  18. [18]

    Enabling realtime reinforcement learning at scale with staggered asynchronous inference,

    M. Riemer, G. R. Subbaraj, G. Berseth, and I. Rish, “Enabling realtime reinforcement learning at scale with staggered asynchronous inference,”ArXiv, vol. abs/2412.14355, 2024

    Pith/arXiv arXiv 2024

  19. [19]

    Addressing function approxi- mation error in actor-critic methods,

    S. Fujimoto, H. Hoof, and D. Meger, “Addressing function approxi- mation error in actor-critic methods,” inInternational conference on machine learning. PMLR, 2018, pp. 1587–1596

    2018

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.