Pith. sign in

REVIEW 4 major objections 7 minor 54 references

A continual RL policy trained only on real tracks can beat classical racing controllers after 15 minutes on an unseen lower-friction track.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 18:13 UTC pith:BLVS72NN

load-bearing objection Solid real-world continual-RL systems result on RoboRacer: multi-track SAC+CBP pretrain then ~15 min fine-tune beats lightly tuned MAP/MPC, with honest from-scratch and plasticity controls. the 4 major comments →

arxiv 2607.24320 v1 pith:BLVS72NN submitted 2026-07-27 cs.RO cs.SYeess.SY

Continual-RL for Generalization in Autonomous Racing on the RoboRacer Platform

Joel Siegert , Edoardo Ghignone , Michele Magno This is my paper
classification cs.RO cs.SYeess.SY
keywords continual reinforcement learningautonomous racingRoboRacerContinual Backpropagationreal-world RLplasticitySoft Actor-Criticoffline RL
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that autonomous racing on physical cars is a natural continual-learning problem: a new track and floor arrive with almost no time for new data, and simulators cannot cover every layout and grip condition. The authors show that Soft Actor-Critic, modified with Continual Backpropagation, weight-initialization regularization, and recent-experience sampling, can be pre-trained on four real tracks and then fine-tuned on a fifth, lower-friction track-floor combination in about 15 minutes. After that brief adaptation the learned policy beats established model-based controllers. An offline RL alternative pre-trains a more plastic network but adapts more slowly. The result matters because it offers a practical route to generalist real-world policies when simulation is incomplete and sample budgets are measured in minutes.

Core claim

With only real-world data, a SAC policy trained continually with Continual Backpropagation (plus L2 Init and recent-experience sampling) on four tracks can be fine-tuned on an unseen lower-friction track-floor combination within 15 minutes and outperform classical MAP and MPC controllers.

What carries the argument

Continual Backpropagation applied to every layer of the SAC actor and critic, combined with L2 Init regularization and a priority curve that emphasizes recent replay samples while still replaying older ones; this package is what keeps the network plastic enough to adapt in minutes.

Load-bearing premise

The claim that beating MAP and MPC after fifteen minutes is a meaningful racing advance assumes those classical controllers, after the paper’s system identification and basic tuning, are strong fair baselines rather than under-tuned comparators.

What would settle it

On the same polished-concrete test track, run the identical 15-minute fine-tuning protocol and check whether the continual policy’s mean lap time still undercuts a carefully retuned MAP controller that receives the same tire-model identification; if it does not, the outperformance claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A generalist driving policy can be built from sequential real-track experience and adapted to a new layout and floor in minutes rather than hours.
  • Continual network modifications (CBP and L2 Init) improve fine-tuning gains beyond buffer-management alone, as shown by lower dead-unit counts and faster lap-time reduction.
  • Offline RL pre-training can produce highly plastic networks yet still lag continual online pre-training on final racing speed at equal fine-tuning steps.
  • The same pre-train-then-fine-tune pattern is proposed for other robots where accurate full simulation is impractical, such as quadrupeds on novel terrain or manipulators with novel materials.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If dead-unit percentage reliably predicts fine-tuning speed on hardware, teams could monitor that metric online and trigger reinitialization before a race weekend rather than waiting for lap-time collapse.
  • The large gap between simulation dead-unit rates (~60%) and real-world rates (~18%) suggests physical non-stationarity forces richer representations; methods validated only in sim may understate the value of plasticity-preserving updates.
  • Privileged tire-force observations re-identified per floor may be doing part of the generalization work; ablating them would test how much of the 15-minute success is representation versus continual optimization.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper presents a continual reinforcement learning framework for the RoboRacer (F1TENTH) autonomous racing platform, built on an asynchronous SAC implementation extended with Continual Backpropagation (CBP), L2 Init regularization, and an ERE-inspired recency-weighted replay sampling scheme adapted to a growing buffer. A policy is pre-trained sequentially on four physical tracks (two floor types, 100k gradient steps) and then fine-tuned on a fifth, unseen track with 30–39% lower friction (polished concrete). The headline result (Table III, §IV-C) is that after 15 minutes of on-car fine-tuning the continual policy achieves a mean lap time of 10.44 s, versus 10.97 s for MAP and 11.50 s for MPC classical controllers, and 10.62 s for a from-scratch RL policy given ~50 minutes. An offline-RL (IQL) pre-training alternative is implemented and shown to be highly plastic but slower, and a simulation study (5 seeds) analyzes dead-unit dynamics across methods. Code, tracks, and models are open-sourced.

Significance. If the results hold, this is a useful empirical contribution to real-world continual RL: physical-robot demonstrations of plasticity-preserving methods (CBP, L2 Init) are rare, and the RoboRacer competition setting — minutes, not hours, of adaptation budget on an unseen track-floor combination — is a natural and practically motivated continual-learning benchmark. Strengths that deserve explicit credit: real-hardware experiments with three seeds reporting mean/min/max/σ; simulation experiments with five seeds tracking dead units through both pre-training and fine-tuning; inclusion of both a from-scratch RL control and two established classical controllers; an honest negative result on offline RL (IQL) rather than a curated success story; and full open-sourcing of tracks, simulation models, and frameworks, which makes the work reproducible in principle. The adaptation-speed improvement over the authors' prior result (15 min vs. 82 min in [12]) is a meaningful increment if the comparator tuning question (below) is resolved.

major comments (4)
  1. [§IV-B, Table III, Fig. 7] The central claim — that the continual policy 'outperforms classical controllers' — rests on MAP/MPC baselines whose preparation is described in a single sentence ('system identification following the procedure of [50] was used, and basic tuning was conducted'). This is load-bearing for three reasons. (i) The margin over MAP is 0.53 s (4.8%) with σ_MAP ≈ 0.09 s; a modest re-tune of MAP's speed-profile aggressiveness on the polished-concrete floor could plausibly close a gap of this size. (ii) The paper's own trajectory analysis (Fig. 7) states that MAP 'commits to a too high velocity before the highlighted corners and needs to overcorrect' — i.e., its velocity profile is mismatched to the new floor, which is precisely the symptom one would expect from an under-tuned speed scaling rather than a controller at the friction limit. (iii) There is an adaptation-budget asymmetry: the RL policy
  2. [§V (Conclusions), cf. Fig. 5/6 and §IV-A] The conclusion states that the findings 'strengthen the dead units metric proposed in [14] and further the correlation between it and improved network adaptation capabilities.' This is not supported by the paper's own IQL result: IQL-n converges to 0% dead units (full activation) during fine-tuning — the best plasticity score of any method — yet produces the slowest lap times at all three evaluation points (Fig. 5 right; Table III row 'Offline pre-training', 12.13 s mean). The data therefore show that low dead-unit count is at best insufficient for good adaptation, and the claimed correlation is contradicted in one of the three conditions. Please temper this claim to what the evidence supports (dead units discriminate Baseline-C-n from CBP-n, but do not predict fine-tuning performance across all methods) and discuss why high plasticity did not translate into performance for IQL (e.g., va
  3. [§III-C, §IV-A (Figs. 3–5)] The framework bundles three modifications (CBP with a novel multi-head utility extension, L2 Init, and the ERE-style priority curve), but no ablation separates their contributions. All comparisons are bundle-vs-baseline. Because the multi-head instantaneous-utility extension for the SAC actor's mean/SD heads is itself a claimed methodological novelty, and because L2 Init and recency-weighted sampling could each plausibly account for much of the gain, the attribution of the improvement to 'CBP' specifically (as the title and framing suggest) is not established. A minimal ablation in simulation (CBP only, L2 Init only, ERE-curve only, and pairs) over the existing 5-seed protocol would substantially strengthen the paper; if compute or car time prohibits this, the limitation should be stated and the framing adjusted from 'CBP-based' to 'a combination of continual-learning techniques'.
  4. [§IV-C, Table III, Fig. 6] The performance advantage of the continual policy over the from-scratch reference is small and likely within noise: 10.44 s vs. 10.62 s mean, with σ of 0.199 s and 0.431 s respectively over only three seeds, and overlapping min/max ranges. The genuine contribution is time-to-performance (15 min vs. ~50 min / 100k steps), which Fig. 6 does support, but the text in §IV-C ('Almost the same performance... was reached by CBP-FS, albeit with a much longer time at disposal') buries this. Please state explicitly that the continual method's benefit is adaptation speed rather than asymptotic lap time, and report a statistical comparison (e.g., confidence intervals or a paired test across the three seeds) for the 15-minute continual vs. from-scratch lap times. Also clarify whether the three seeds share the same pre-trained CBP-4 checkpoint (i.e., whether seed variance reflects only fine-tuning) — t
minor comments (7)
  1. [§IV-A, paragraph after Fig. 3] Typo: 'CPB-4 exhibits a marginally higher average reward' — CPB should be CBP. Same paragraph: please quantify 'marginally higher' and 'significantly reduced plasticity' with numbers, since these are comparative claims.
  2. [Table II] Several unit/description errors: tire forces F are given range '[−40,40] m' (should be N); slip angles αf, αr given '[−π,π] m' (should be rad or dimensionless); p_in and p_out are described as 'Lateral velocity in car frame' but are presumably points along the inner/outer track boundaries (analogous to p_race). Also, the observation vector uses s_{t−s} while the table lists s_{t−1}; and the reward r_sa uses s_t − s_{t−s}. Please make the notation consistent.
  3. [§III-C, Fig. 2] The priority-curve definition mixes index conventions: the figure caption defines i=1 as the newest sample and i=0 as the oldest, but the smooth-step polynomial is defined for i ∈ [c_low, c_high] with c_low=0.5, c_high=0.95 as 'percentages'. Please clarify the direction of the curve (which end receives P_high) and the normalization of t in the quintic.
  4. [§III-A and §IV-C] The role of human supervision needs one clarifying sentence: were the timed evaluation laps in Table III / Fig. 6 fully autonomous (no drive-button releases or interventions)? Since r_drive penalizes releases during training, readers need to know interventions did not occur during evaluation.
  5. [§IV-B/Table III] The number of timed laps underlying each row of Table III is not reported; please add it. Also the naming is inconsistent: 'IQL-4-CBP' in the text vs. 'Offline pre-training' in Table III.
  6. [§III-B] The UTD ratio is reported as varying between 1 and 1.7 'primarily influenced by the sampling time for the increasing replay buffer size.' Since UTD strongly affects SAC sample efficiency, please note whether UTD was matched across the compared methods (Baseline-C-n, CBP-n, IQL-n) in simulation, or report it per method.
  7. [General] The footnote 'This work was not supported by any organization' is unusual for a camera-ready; presumably a placeholder — please fix. Figure 5 and Figure 6 have nearly identical captions differing only in 'Simulation'/'Physical'; consider making the distinguishing conditions (friction values, seeds) explicit in each caption.

Circularity Check

0 steps flagged

Empirical robotics paper with measured lap times against external baselines; no derivation that reduces claims to fitted inputs or self-citation by construction.

full rationale

The paper’s load-bearing claims are experimental outcomes (continual SAC+CBP pre-training on four real tracks, then 15-minute fine-tuning on an unseen lower-friction track outperforming MAP/MPC and a from-scratch RL baseline; Table III, Figs. 5–7), not first-principles derivations. Lap times, dead-unit counts, and reward curves are measured on held-out track-floor combinations and compared to classical controllers and alternative RL setups (Baseline-C, IQL, CBP-FS). Citations to the authors’ prior stack ([12], [49]) and system-ID procedure ([50]) supply platform, observation features, and baseline implementations—normal engineering dependence—but do not define or force the fine-tune speedup or the ranking versus MAP/MPC. Continual methods (CBP, L2 Init, ERE-style sampling) and offline IQL are taken from external literature and evaluated empirically. There is no self-definitional identity, no fitted parameter renamed as a prediction, no uniqueness theorem imported from the authors, and no ansatz smuggled in as a derivation. Score 0 is appropriate.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 2 invented entities

The claim is experimental, not axiomatic. It rests on standard RL/control modeling choices, hardware and safety-protocol assumptions, and several hand-set training hyperparameters and buffer rules that shape sample efficiency. No new physical entities are postulated; ‘generalistic policy’ is an operational label for the multi-track pretrained network.

free parameters (6)
  • SAC learning rate / network width / batch / n-step / gSDE rate = lr=0.003; net 2x256; batch 128; n-step 3; gSDE 20
    Table I values (lr 0.003, 2×256 ReLU, batch 128, n-step 3, gSDE every 20 steps) are chosen design knobs that govern sample efficiency and final lap times.
  • Reward weights w_sa, w_collision, w_drive = 20, 16, 10
    Empirically chosen weights (20, 16, 10) directly shape what ‘good driving’ means during human-monitored training (§III-B).
  • ERE-style priority curve parameters = c_low=0.5, c_high=0.95, η=0.001, P_low,min=0.001
    c_low=0.5, c_high=0.95, η=0.001, P_low,min=0.001 control how strongly recent experience is emphasized (§III-C).
  • Replay diversity prune (PCA 10 comps, k-NN k=5) and fine-tune seed buffer size = 10 PCs (~95% var); k=5; 5000 seed samples
    Trajectory priority via PCA+kNN and starting fine-tune with 5k diverse prior samples are hand-designed continual-buffer choices that the authors report as necessary for stable adaptation (§III-A).
  • IQL expectile and temperature = expectile 0.7; temperature 0.1
    Offline comparator uses expectile 0.7 and advantage temperature 0.1 (§III-D), affecting the offline baseline quality.
  • Pretrain budget 100k gradient steps; fine-tune horizons 5/10/15 min = 100000 pretrain steps; eval at 5/10/15 min
    Fixed compute/time budgets define the headline ‘within 15 minutes’ claim and the comparison points.
axioms (5)
  • domain assumption Off-policy maximum-entropy actor-critic (SAC) with the stated asynchronous UTD regime is an appropriate base learner for scarce real driving data.
    Method builds on SB3 SAC + Async-SAC-1 without claiming a new convergence theorem (§III-B).
  • domain assumption Continual Backpropagation’s utility-based reinitialization plus L2-Init sufficiently preserves plasticity under sequential track/floor shifts for this control task.
    CBP is applied to all actor/critic layers with a multi-head utility extension; justification is empirical dead-unit and lap-time trends (§III-C, §IV).
  • domain assumption Pacejka tire parameters identified on each floor yield observation features (tire forces, slip angles) that are valid inputs for both pretrain and fine-tune.
    Observation space includes F and α from a model identified per floor via [50] (Table II, §III-B).
  • ad hoc to paper Human drive-button supervision and occasional interventions during early on-car learning do not invalidate the reported autonomy/lap-time comparisons.
    Safety protocol adds r_drive and operator monitoring; authors argue interventions become sporadic after minutes (§III-A).
  • ad hoc to paper The four pretrain tracks and single held-out polished-concrete track are a fair test of ‘unseen track-floor’ generalization under 30–39% friction drop.
    Experimental design in Figure 1 and §IV-B; generalization scope is defined by this split only.
invented entities (2)
  • CBP multi-head instantaneous utility extension for SAC actor mean/SD heads no independent evidence
    purpose: Extend CBP’s unit utility to SAC’s two output heads so continual reinitialization applies to the stochastic actor.
    Defined in §III-C as a summation over outgoing head weights; engineering extension of [14], not a new physical object.
  • Async replay priority curve (high/transient/low) replacing fixed-buffer ERE shrinkage no independent evidence
    purpose: Emphasize recent experience while the async replay buffer grows continuously.
    Figure 2 and §III-C adapt ERE to unbounded async buffers; method component specific to this pipeline.

pith-pipeline@v1.2.0-grok45-kimik3 · 17933 in / 4129 out tokens · 80890 ms · 2026-07-31T18:13:30.441720+00:00 · methodology

0 comments
read the original abstract

A key challenge in modern robotics is to adapt to changing environments, a challenge that is exacerbated when simulations cannot encompass every possible real-world configuration, and therefore Reinforcement Learning (RL) in the physical world becomes necessary. Continual Reinforcement Learning provides the tools to address this challenge; however, both the frameworks and the methods remain underexplored. Autonomous Racing and in particular the RoboRacer competition provide a testing ground for such methods, as learning to drive on a new track-floor combination with the least amount of new experience naturally frames a continual learning problem. This work tries to address this gap by proposing a continual RL framework based on Continual Backpropagation that is able, with only real-world data, to train a generalistic policy on a set of tracks and then fine- tune it within 15 minutes to outperform classical controllers. Furthermore, a comparison method based on offline RL is proposed, and a simulation analysis of the plasticity properties of the methods is conducted.

Figures

Figures reproduced from arXiv: 2607.24320 by Edoardo Ghignone, Joel Siegert, Michele Magno.

Figure 1
Figure 1. Figure 1: Set of tracks used for this study, subdivided into four pre-training and one test track. A Newton-meter was used to [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Priority curve for sampling from the replay buffer with [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Percentage of dead units in the actor network during [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Simulation results on the test track. Left: Percentage [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Physical results on the test track. Left: Percentage [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Minimum lap time trajectories on Track 5. Arrows [PITH_FULL_IMAGE:figures/full_fig_p006_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

54 extracted references · 16 linked inside Pith

  1. [1]

    ELF opengo: an analysis and open reimplementation of alphazero,

    Y . Tianet al., “ELF opengo: an analysis and open reimplementation of alphazero,” inProceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, 2019, pp. 6244–6253. [Online]. Available: http://proceedings.mlr.press/ v97/tian19a.html

    2019

  2. [2]

    Circus anymal: A quadruped learning dexterous manipulation with its limbs,

    F. Shiet al., “Circus anymal: A quadruped learning dexterous manipulation with its limbs,” in2021 IEEE International Conference on Robotics and Automation (ICRA), 2021, pp. 2316–2323

    2021

  3. [3]

    Adaptive Interactive Navigation of Quadruped Robots using Large Language Models,

    K. Zhouet al., “Adaptive Interactive Navigation of Quadruped Robots using Large Language Models,” Mar. 2025, arXiv:2503.22942 [cs]. [Online]. Available: http://arxiv.org/abs/2503.22942

    arXiv 2025

  4. [4]

    Champion-level drone racing using deep reinforcement learning,

    E. Kaufmann, L. Bauersfeld, A. Loquercio, M. M ¨uller, V . Koltun, and D. Scaramuzza, “Champion-level drone racing using deep reinforcement learning,”Nature, vol. 620, no. 7976, pp. 982–987, Aug. 2023. [Online]. Available: https://www.nature.com/articles/s41586-023-06419-4

    2023

  5. [5]

    Learning Quadrupedal Locomotion over Challenging Terrain,

    J. Lee, J. Hwangbo, L. Wellhausen, V . Koltun, and M. Hutter, “Learning Quadrupedal Locomotion over Challenging Terrain,”Science Robotics, vol. 5, no. 47, p. eabc5986, Oct. 2020, arXiv:2010.11251 [cs]. [Online]. Available: http://arxiv.org/abs/2010.11251

    Pith/arXiv arXiv 2020

  6. [6]

    Dexpoint: Generalizable point cloud reinforcement learning for sim-to-real dexterous manipulation,

    Y . Qin, B. Huang, Z.-H. Yin, H. Su, and X. Wang, “Dexpoint: Generalizable point cloud reinforcement learning for sim-to-real dexterous manipulation,” in6th Annual Conference on Robot Learning, 2022. [Online]. Available: https://openreview.net/forum?id=tJE1Yyi8fUX

    2022

  7. [7]

    Simulation for robotics test automation: Developer perspec- tives,

    A. Afzal, D. S. Katz, C. Le Goues, and C. S. Timperley, “Simulation for robotics test automation: Developer perspec- tives,” in2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST), 2021, pp. 263–274

    2021

  8. [8]

    Continual learning as computationally constrained reinforcement learning,

    S. Kumaret al., “Continual learning as computationally constrained reinforcement learning,”Found. Trends Mach. Learn., vol. 18, no. 5, p. 913–1053, Aug. 2025. [Online]. Available: https://doi.org/10.1561/2200000116

    doi:10.1561/2200000116 2025

  9. [9]

    End-to-end Lidar-Driven Reinforce- ment Learning for Autonomous Racing,

    M. Mammadov, “End-to-end Lidar-Driven Reinforce- ment Learning for Autonomous Racing,” Sep. 2023, arXiv:2309.00296 [cs]. [Online]. Available: http://arxiv.org/ abs/2309.00296

    Pith/arXiv arXiv 2023

  10. [10]

    FastRLAP: A system for learning high-speed driving via deep RL and autonomous practicing,

    K. Stachowicz, D. Shah, A. Bhorkar, I. Kostrikov, and S. Levine, “FastRLAP: A system for learning high-speed driving via deep RL and autonomous practicing,” in7th Annual Conference on Robot Learning, 2023. [Online]. Available: https://openreview.net/forum?id=FRKBdXhkQE0

    2023

  11. [11]

    TinyLidarNet: 2D LiDAR-based End-to-End Deep Learning Model for F1TENTH Autonomous Racing,

    M. M. Zarrar, Q. Weng, B. Yerjan, A. Soyyigit, and H. Yun, “TinyLidarNet: 2D LiDAR-based End-to-End Deep Learning Model for F1TENTH Autonomous Racing,” in2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Abu Dhabi, United Arab Emirates: IEEE, Oct. 2024, pp. 2878–2884. [Online]. Available: https://ieeexplore.ieee.org/document...

    arXiv 2024

  12. [12]

    Drive fast, learn faster: On-board RL for high performance autonomous racing,

    B. Hildisch, E. Ghignone, N. Baumann, C. Hu, A. Carron, and M. Magno, “Drive fast, learn faster: On-board RL for high performance autonomous racing,” inReinforcement Learning Conference, 2025. [Online]. Available: https: //openreview.net/forum?id=4naF7E5J6n

    2025

  13. [13]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,

    T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,”International Conference on Machine Learning (ICML), 2018

    2018

  14. [14]

    Loss of plasticity in deep continual learning,

    S. Dohare, J. F. Hernandez-Garcia, Q. Lan, P. Rahman, A. R. Mahmood, and R. S. Sutton, “Loss of plasticity in deep continual learning,”Nature, vol. 632, no. 8026, pp. 768–774, Aug. 2024. [Online]. Available: https://www.nature. com/articles/s41586-024-07711-7

    2024

  15. [15]

    Offline Reinforcement Learning with Implicit Q-Learning,

    I. Kostrikov, A. Nair, and S. Levine, “Offline Reinforcement Learning with Implicit Q-Learning,” Oct. 2021, arXiv:2110.06169 [cs]. [Online]. Available: http://arxiv.org/abs/2110.06169

    Pith/arXiv arXiv 2021

  16. [16]

    Maintaining Plasticity in Continual Learning via Regenerative Regularization,

    S. Kumar, H. Marklund, and B. V . Roy, “Maintaining Plasticity in Continual Learning via Regenerative Regularization,” Oct. 2024, arXiv:2308.11958 [cs]. [Online]. Available: http://arxiv.org/abs/2308.11958

    Pith/arXiv arXiv 2024

  17. [17]

    Parkour in the wild: Learning a general and extensible agile locomotion policy using multi-expert distillation and rl fine-tuning,

    N. Rudin, J. He, J. Aurand, and M. Hutter, “Parkour in the wild: Learning a general and extensible agile locomotion policy using multi-expert distillation and rl fine-tuning,” 2025. [Online]. Available: https://arxiv.org/abs/2505.11164

    Pith/arXiv arXiv 2025

  18. [18]

    Magnetic control of tokamak plasmas through deep reinforcement learning,

    J. Degraveet al., “Magnetic control of tokamak plasmas through deep reinforcement learning,”Nat., vol. 602, no. 7897, pp. 414–419, 2022. [Online]. Available: https: //doi.org/10.1038/s41586-021-04301-9

    doi:10.1038/s41586-021-04301-9 2022

  19. [19]

    Deep reinforcement learning for robotics: A survey of real-world successes,

    C. Tang, B. Abbatematteo, J. Hu, R. Chandra, R. Mart ´ın- Mart´ın, and P. Stone, “Deep reinforcement learning for robotics: A survey of real-world successes,”Annual Review of Control, Robotics, and Autonomous Systems, vol. 8, no. V olume 8, 2025, pp. 153–188, 2025. [Online]. Available: https://www.annualreviews.org/content/journals/10. 1146/annurev-contro...

    2025

  20. [20]

    Reference-free formula drift with reinforcement learning: From driving data to tire energy-inspired, real-world policies,

    F. Djeumou, M. Thompson, M. Suminaka, and J. Subosits, “Reference-free formula drift with reinforcement learning: From driving data to tire energy-inspired, real-world policies,”

  21. [21]

    A Survey of Sim-to-Real Methods in RL: Progress, Prospects and Challenges with Foundation Models,

    L. Da, J. Turnau, T. P. Kutralingam, A. Velasquez, P. Shakarian, and H. Wei, “A Survey of Sim-to-Real Methods in RL: Progress, Prospects and Challenges with Foundation Models,” Mar. 2025, arXiv:2502.13187 [cs]. [Online]. Available: http://arxiv.org/abs/2502.13187

    Pith/arXiv arXiv 2025

  22. [22]

    Learning to walk in minutes using massively parallel deep reinforcement learning,

    N. Rudin, D. Hoeller, P. Reist, and M. Hutter, “Learning to walk in minutes using massively parallel deep reinforcement learning,” in5th Annual Conference on Robot Learning,

  23. [23]

    A Definition of Continual Reinforcement Learning,

    D. Abel, A. Barreto, B. V . Roy, D. Precup, H. v. Hasselt, and S. Singh, “A Definition of Continual Reinforcement Learning,” Dec. 2023, arXiv:2307.11046 [cs]. [Online]. Available: http://arxiv.org/abs/2307.11046

    Pith/arXiv arXiv 2023

  24. [24]

    Crossq: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity,

    A. Bhattet al., “Crossq: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity,” inThe Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https: //openreview.net/forum?id=PczQtTsTIX

    2024

  25. [25]

    Simba: Simplicity bias for scaling up parameters in deep reinforcement learning,

    H. Leeet al., “Simba: Simplicity bias for scaling up parameters in deep reinforcement learning,” inThe Thirteenth International Conference on Learning Representations,

  26. [26]

    Stable-baselines3: Reliable reinforcement learning implementations,

    A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann, “Stable-baselines3: Reliable reinforcement learning implementations,”Journal of Machine Learning Research, vol. 22, no. 268, pp. 1–8, 2021. [Online]. Available: http://jmlr.org/papers/v22/20-1364.html

    2021

  27. [27]

    Asynchronous Reinforcement Learning for Real-Time Control of Physical Robots,

    Y . Yuan and A. R. Mahmood, “Asynchronous Reinforcement Learning for Real-Time Control of Physical Robots,” in2022 International Conference on Robotics and Automation (ICRA), 2022, pp. 5546–5552

    2022

  28. [28]

    Smooth exploration for robotic reinforcement learning,

    A. Raffin, J. Kober, and F. Stulp, “Smooth exploration for robotic reinforcement learning,” in5th Annual Conference on Robot Learning, 2021. [Online]. Available: https://openreview. net/forum?id=TSuSGVkjuXd

    2021

  29. [29]

    Bridging the sim-to-real gap with bayesian inference,

    J. Rothfuss, B. Sukhija, L. Treven, F. D ¨orfler, S. Coros, and A. Krause, “Bridging the sim-to-real gap with bayesian inference,” in2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2024, pp. 10 784– 10 791

    2024

  30. [30]

    Latent imagination facilitates zero- shot transfer in autonomous racing,

    A. Brunnbaueret al., “Latent imagination facilitates zero- shot transfer in autonomous racing,” in2022 International Conference on Robotics and Automation (ICRA), 2022, pp. 7513–7520

    2022

  31. [31]

    A Survey of Continual Reinforcement Learning,

    C. Panet al., “A Survey of Continual Reinforcement Learning,” Jun. 2025, arXiv:2506.21872 [cs]. [Online]. Available: http://arxiv.org/abs/2506.21872

    Pith/arXiv arXiv 2025

  32. [32]

    Rolnick, A

    D. Rolnick, A. Ahuja, J. Schwarz, T. P. Lillicrap, and G. Wayne,Experience replay for continual learning. Red Hook, NY , USA: Curran Associates Inc., 2019

    2019

  33. [33]

    Cora: Benchmarks, baselines, and metrics as a platform for continual reinforcement learning agents,

    S. Powers, E. Xing, E. Kolve, R. Mottaghi, and A. Gupta, “Cora: Benchmarks, baselines, and metrics as a platform for continual reinforcement learning agents,” inProceedings of The 1st Conference on Lifelong Learning Agents, ser. Proceedings of Machine Learning Research, vol. 199. PMLR, 22–24 Aug 2022, pp. 705–743. [Online]. Available: https://proceedings....

    2022

  34. [34]

    Boosting Soft Actor-Critic: Emphasizing Recent Experience without Forgetting the Past,

    C. Wang and K. Ross, “Boosting Soft Actor-Critic: Emphasizing Recent Experience without Forgetting the Past,” Jun. 2019, arXiv:1906.04009 [cs]. [Online]. Available: http://arxiv.org/abs/1906.04009

    Pith/arXiv arXiv 2019

  35. [35]

    Loss of Plasticity in Continual Deep Reinforcement Learning,

    Z. Abbas, R. Zhao, J. Modayil, A. White, and M. C. Machado, “Loss of Plasticity in Continual Deep Reinforcement Learning,” Mar. 2023, arXiv:2303.07507 [cs]. [Online]. Available: http://arxiv.org/abs/2303.07507

    Pith/arXiv arXiv 2023

  36. [36]

    The Dormant Neuron Phenomenon in Deep Reinforcement Learning,

    G. Sokar, R. Agarwal, P. S. Castro, and U. Evci, “The Dormant Neuron Phenomenon in Deep Reinforcement Learning,” Jun. 2023, arXiv:2302.12902 [cs]. [Online]. Available: http://arxiv.org/abs/2302.12902

    Pith/arXiv arXiv 2023

  37. [37]

    Autonomous overtaking in gran turismo sport using cur- riculum reinforcement learning,

    Y . Song, H. Lin, E. Kaufmann, P. D ¨urr, and D. Scaramuzza, “Autonomous overtaking in gran turismo sport using cur- riculum reinforcement learning,” in2021 IEEE International Conference on Robotics and Automation (ICRA), 2021, pp. 9403–9409

    2021

  38. [38]

    F1tenth: An open-source evaluation environment for continuous control and reinforcement learning,

    M. O’Kelly, H. Zheng, D. Karthik, and R. Mangharam, “F1tenth: An open-source evaluation environment for continuous control and reinforcement learning,” in Proceedings of the NeurIPS 2019 Competition and Demonstration Track, ser. Proceedings of Machine Learning Research, H. J. Escalante and R. Hadsell, Eds., vol. 123. PMLR, 08–14 Dec 2020, pp. 77–89. [Onli...

    2019

  39. [39]

    Implementation of the pure pursuit path tracking algorithm,

    R. Coulter, “Implementation of the pure pursuit path tracking algorithm,” Pittsburgh, Pennsylvania, Jan 1990

    1990

  40. [40]

    Optimization-Based Autonomous Racing of 1:43 Scale RC Cars,

    A. Liniger, A. Domahidi, and M. Morari, “Optimization-Based Autonomous Racing of 1:43 Scale RC Cars,”Optimal Control Applications and Methods, vol. 36, no. 5, pp. 628–647, Sep. 2015, arXiv:1711.07300 [math]. [Online]. Available: http://arxiv.org/abs/1711.07300

    Pith/arXiv arXiv 2015

  41. [41]

    Model- and Acceleration- based Pursuit Controller for High-Performance Autonomous Racing,

    J. Becker, N. Imholz, L. Schwarzenbach, E. Ghignone, N. Baumann, and M. Magno, “Model- and Acceleration- based Pursuit Controller for High-Performance Autonomous Racing,” in2023 IEEE International Conference on Robotics and Automation (ICRA), May 2023, pp. 5276–5283, arXiv:2209.04346 [cs]. [Online]. Available: http://arxiv.org/abs/2209.04346

    Pith/arXiv arXiv 2023

  42. [42]

    Unifying F1TENTH Autonomous Racing: Survey, Methods and Benchmarks,

    B. D. Evanset al., “Unifying F1TENTH Autonomous Racing: Survey, Methods and Benchmarks,” Apr. 2024, arXiv:2402.18558 [cs]. [Online]. Available: http://arxiv.org/ abs/2402.18558

    Pith/arXiv arXiv 2024

  43. [43]

    Tum autonomous motorsport: An autonomous racing software for the indy autonomous challenge,

    J. Betzet al., “Tum autonomous motorsport: An autonomous racing software for the indy autonomous challenge,”Journal of Field Robotics, vol. 40, no. 4, pp. 783–809, 2023. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/rob. 22153

    doi:10.1002/rob 2023

  44. [44]

    er.autopilot 1.1: A software stack for au- tonomous racing on oval and road course tracks,

    A. Rajiet al., “er.autopilot 1.1: A software stack for au- tonomous racing on oval and road course tracks,”IEEE Transactions on Field Robotics, vol. 1, pp. 332–359, 2024

    2024

  45. [45]

    Outracing champion Gran Turismo drivers with deep reinforcement learning,

    P. R. Wurmanet al., “Outracing champion Gran Turismo drivers with deep reinforcement learning,”Nature, vol. 602, no. 7896, pp. 223–228, Feb. 2022. [Online]. Available: https://www.nature.com/articles/s41586-021-04357-7

    2022

  46. [46]

    Train in Austria, Race in Montecarlo: Generalized RL for Cross-Track F1 tenth LIDAR-Based Races,

    M. Bosello, R. Tse, and G. Pau, “Train in Austria, Race in Montecarlo: Generalized RL for Cross-Track F1 tenth LIDAR-Based Races,” in2022 IEEE 19th Annual Consumer Communications & Networking Conference (CCNC). Las Vegas, NV , USA: IEEE, Jan. 2022, pp. 290–298. [Online]. Available: https://ieeexplore.ieee.org/document/9700730/

    arXiv 2022

  47. [47]

    Keep doing what worked: Behavior modelling priors for offline reinforcement learning,

    N. Siegelet al., “Keep doing what worked: Behavior modelling priors for offline reinforcement learning,” in International Conference on Learning Representations,

  48. [48]

    Offline- to-online reinforcement learning via balanced replay and pessimistic q-ensemble,

    S. Lee, Y . Seo, K. Lee, P. Abbeel, and J. Shin, “Offline- to-online reinforcement learning via balanced replay and pessimistic q-ensemble,” in5th Annual Conference on Robot Learning, 2021. [Online]. Available: https://openreview.net/ forum?id=AlJXhEI6J5W

    2021

  49. [49]

    ForzaETH Race Stack – Scaled Autonomous Head-to-Head Racing on Fully Commercial off-the-Shelf Hardware,

    N. Baumannet al., “ForzaETH Race Stack – Scaled Autonomous Head-to-Head Racing on Fully Commercial off-the-Shelf Hardware,”Journal of Field Robotics, p. rob.22429, Sep. 2024, arXiv:2403.11784 [cs]. [Online]. Available: http://arxiv.org/abs/2403.11784

    Pith/arXiv arXiv 2024

  50. [50]

    Learning-based on-track system identification for scaled autonomous racing in under a minute,

    O. Dikiciet al., “Learning-based on-track system identification for scaled autonomous racing in under a minute,”IEEE Robotics and Automation Letters, vol. 10, no. 2, pp. 1984– 1991, 2025

    1984

  51. [2020]

    Available: https://openreview.net/forum?id= rke7geHtwH

    [Online]. Available: https://openreview.net/forum?id= rke7geHtwH

  52. [2021]

    Available: https://openreview.net/forum?id= wK2fDDJ5VcF

    [Online]. Available: https://openreview.net/forum?id= wK2fDDJ5VcF

  53. [2024]

    Available: https://arxiv.org/abs/2410.20990

    [Online]. Available: https://arxiv.org/abs/2410.20990

    Pith/arXiv arXiv

  54. [2025]

    Available: https://openreview.net/forum?id= jXLiDKsuDo

    [Online]. Available: https://openreview.net/forum?id= jXLiDKsuDo