Pith. sign in

REVIEW 3 major objections 4 minor 27 references

CuRLA: Curriculum Learning Based Deep Reinforcement Learning for Autonomous Driving

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that ordering training from empty roads to traffic, while revising the speed reward to rise toward a target speed, lets a PPO-VAE driving agent reach higher average speed without losing distance traveled.

desk verdict A clear, honest write-up of an incremental RL-for-driving recipe; the speed gain comes mostly from the reward function, and the curriculum's own contribution rests on single runs. read the letter →

arxiv 2501.04982 v1 pith:2B23SYCI submitted 2025-01-09 cs.RO cs.AIcs.LG

Bhargava Uppuluri , Anjel Patel , Neil Mehta , Sridhar Kamath , Pratyush Chakraborty This is my paper · ORCID
classification cs.ROcs.AIcs.LG
keywords DeepReinforcementLearningCurriculumAutonomousDrivingProximalPolicyOptimizationVariationalAutoencoderCARLAsimulatorRewardfunctiondesignCollisionavoidance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes CuRLA, a self-driving agent built from Proximal Policy Optimization and a variational autoencoder, and asks whether curriculum learning can make it both safer and faster to train. The answer it presents is yes: by first letting the agent drive alone and then gradually adding traffic and a collision penalty, the agent learns lane-following before it must handle avoidance, while a revised speed reward pushes it toward the target speed instead of letting it cruise slowly. In CARLA Town 7, CuRLA and a one-fold curriculum variant reach average speeds of about 22 and 20 km/h during training, against 14 km/h for the baseline agent, with roughly the same 1.5-lap distance in training and 0.3-lap distance in evaluation. If the comparisons hold up, this suggests task ordering can relieve some of the tension between speed and safety in scalar reward design.

What carries the argument

The load-bearing components are the revised reward function and the two-fold curriculum schedule, working on top of a PPO-Clip policy that consumes a VAE-compressed latent state. The reward is a product of three terms—angle alignment, lane centering, and speed—plus a collision penalty. The baseline speed term is constant at 1 for all speeds between the minimum and the target, giving the agent no reason to prefer the fast end of that band; the revised speed term rises across the same band, so moving toward the target speed earns strictly more. The curriculum adds traffic density and the collision penalty only after 1,500 episodes, so the agent first learns stable lane-following on an empty highway and then learns collision avoidance under traffic. This staged schedule is the named contribution, CuRLA.

What would settle it

Train SCA, One-Fold CL, and CuRLA in CARLA Town 7 from at least five random seeds each, recording lap distance and average speed at fixed episode counts. If the per-seed ranges for average speed overlap between the baseline and CuRLA at evaluation, or if the 14 versus 22 km/h training gap does not separate, the paper's claim that curriculum and the revised reward improve speed would fail. A simpler version is to rerun just the evaluation episodes three times per agent; because each evaluation covers roughly one-third of a lap, three runs would at least reveal the spread within one trained policy.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a staged training schedule plus a monotone speed reward yields a driving policy that travels at higher average speed without covering less distance. The baseline Self-Centering Agent and CuRLA both complete about 1.5 laps in training and 0.3 laps in evaluation, but CuRLA's training average speed is 22 km/h versus 14 km/h for the baseline; in evaluation the gap is 6 versus 4 km/h. The authors attribute the gain to two design changes: the revised speed reward removes the flat plateau between minimum and target speed that lets the agent settle for slow cruising, and the curriculum teaches basic driving before collision avoidance is expected. One-Fold CL, which has the new reward and penalty but traffic from the start, lands between the two, supporting the role of staged traffic rather than the reward alone.

Load-bearing premise

The comparisons rest on a single training run per agent, with smoothed curves and no repeated-seed statistics; if run-to-run variation is as large as the observed speed gaps, the reported ordering of agents could change.

Editorial extensions

If this is right

  • If the reported gains are reproducible, curriculum ordering lets a PPO-VAE agent reach roughly 22 km/h in training while keeping the baseline's roughly 1.5-lap training distance.
  • The revised speed reward, which increases from minimum to target speed, provides a directly usable alternative to a flat speed reward wherever slow cruising is the failure mode.
  • Adding the collision penalty only after lane-following is established suggests a schedule for handling conflicting objectives in a single scalar reward.
  • The One-Fold CL result, with lower distance than both CuRLA and SCA, indicates that staged traffic is doing real work and is a candidate for further ablation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An editorial extension would be to run each agent across several random seeds and report intervals; the reported evaluation gap of 2 km/h between CuRLA and SCA is small enough that single-run variation could change the ordering.
  • An editorial extension would be to unconfound the two changes by testing the new reward function without curriculum and the curriculum with the old reward, isolating which component drives the speed gain.
  • The same two-fold schedule could be tested with a different continuous-control algorithm, such as a DDPG-based stack, to see whether the benefit is specific to PPO or transfers across policy-gradient methods.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes CuRLA, a PPO+VAE agent trained in the CARLA Town 7 environment with a two-fold curriculum that gradually introduces traffic and a collision penalty, alongside a revised speed reward function. The authors compare CuRLA against a Self-Centering Agent (SCA) baseline from prior work and against a One-Fold CL variant that has traffic from the start, reporting distance traveled and average speed over 3500 training episodes. The main claims are that CuRLA trains faster, reaches higher average speed (22 km/hr vs 14 km/hr for SCA), and maintains comparable lap distance. The central mechanism is plausible, but the evidence as presented is limited by single-run comparisons, confounded reward/curriculum changes, and short evaluation trajectories.

Significance. If the claims were fully supported, the paper would provide a useful incremental demonstration that curriculum learning and reward shaping can improve the speed-efficiency trade-off in vision-based autonomous driving with PPO+VAE. The authors give clear equations for all reward components, a well-defined experimental protocol, and a three-way comparison that separates the effect of curriculum structure (CuRLA vs One-Fold CL) from the effect of the revised reward (revised-reward agents vs SCA). However, the reported quantitative results rest on one training run per condition with heavy curve smoothing, and the speed advantage over SCA is substantially attributable to the reward functions in Eqs. (7)-(8) rather than to curriculum learning. The paper's contribution is therefore evidentiary rather than conceptual, and the current experiments do not yet substantiate the central claims.

major comments (3)
  1. [Section 3.3, Eqs. (7) and (8)] The comparison of CuRLA and One-Fold CL against SCA does not validate curriculum learning because the reward function differs between the conditions. Equation (8) explicitly increases the speed reward from 0 at vmin to 1 at vtarget, whereas Eq. (7) is flat at 1 over [vmin, vtarget]; an agent optimizing the revised reward is directly incentivized to drive faster. The observed speed gap (22 and 20 km/hr vs 14 km/hr) is therefore largely a consequence of the reward change, not of the two-fold curriculum. To isolate the curriculum effect, the authors should add an ablation in which SCA is trained with the revised reward function, or CuRLA/One-Fold CL are trained with the original reward function, while keeping all other factors fixed.
  2. [Section 3.3, Figs. 6-9] All reported performance comparisons are based on a single training run per agent, with no error bars, confidence intervals, or statistical significance tests. The learning curves are smoothed with factor 0.999, and the text quotes approximate values read from these smoothed curves. In particular, the curriculum-specific gain of CuRLA over One-Fold CL is only about 2 km/hr in training speed and 0.25 laps in distance; without seed variance estimates, these differences are within the typical run-to-run variability of PPO in a high-dimensional visuomotor task. The claim that CuRLA and One-Fold CL 'significantly outperform' SCA is not supported by the reported evidence. The authors should run multiple seeds per condition and report mean and standard deviation (or equivalent intervals) for training and evaluation metrics.
  3. [Section 3.1 and Section 3.3] The evaluation result is based on very short trajectories: the reported evaluation distance is about 0.3 laps for all agents. With such short episodes, average speed and distance are dominated by early termination behavior (e.g., leaving the lane or slow-speed stalls), and the paper does not report how often each termination condition occurs. Furthermore, despite introducing a collision penalty in Eq. (9), the manuscript reports no collision counts, collision rates, or any direct safety metric. Given that the stated motivation includes safe driving, the authors should report termination reasons and collision statistics, and preferably evaluate on longer or multiple episodes.
minor comments (4)
  1. [Section 3.3] There is a typo in the text: 'CurLA' should be 'CuRLA'.
  2. [Section 3.3] The smoothing factor of 0.999 is mentioned but the smoothing method is not specified; please describe whether it is exponential moving average or another filter, and consider showing at least one unsmoothed or lightly smoothed curve so the reader can assess variance.
  3. [Section 3.1] The curriculum schedule is described qualitatively ('after 1500 episodes, traffic and a collision penalty are introduced'), but the exact traffic density values, the rate of increase, and the maximum traffic density are not specified. Please provide the full schedule so that the experiments are reproducible.
  4. [Section 3.2] Equation (9) defines the collision penalty through collision intensity Ic, but the manuscript does not explain how Ic is obtained from CARLA or how it behaves for different collision severities; a short clarification would help.

Circularity Check

1 steps flagged · score 5.0 of 10

The average-speed advantage over SCA is built into the revised reward function (Eq. 8), so the headline speed result is partly circular; the curriculum comparison is underpowered but not circular.

  1. self definitional [Section 3.2 'A Reward Function Optimization', Eqs. (7)-(8); Section 3.3 'Metrics and Result Analysis'; Conclusions]
    "rv′ = { 0.5·v/vmin, v<vmin; 1−(vtarget−v)/(vtarget−vmin), vmin≤v≤vtarget; (vmax−v)/(vmax−vtarget), vtarget<v≤vmax } ... 'we have replaced the constant graph in [vmin, vtarget] with an increasing function ... to prioritize getting as close to the target speed as possible' ... 'This superior performance of CuRLA and One-Fold CL (on the revised reward function) compared to the SCA agent (using the original reward function) underscores our reward function’s efficiency in optimizing speed-related aspects.'"

    The paper's headline speed result is the direct consequence of the reward function it deliberately designed. Eq. (8) makes rv′ strictly increasing with speed on [vmin, vtarget], while the original rv in Eq. (7) is flat at 1 on that interval. A PPO agent maximizing cumulative reward is therefore explicitly incentivized to drive faster, so the observed speed gap (22/20 vs 14 km/hr in training, 6/5 vs 4 km/hr in evaluation) is not an independent empirical discovery; it is a by-construction property of the changed reward.

full rationale

The derivation chain has one genuinely circular link: the revised reward function rv′ in Eq. (8) is defined to increase with speed up to vtarget, and the paper then reports 'average speed' as the key success metric and claims the improvement as evidence for the method. Since an RL agent optimizes exactly this reward, the speed advantage over SCA is forced by the objective rather than being an independent prediction. The paper's own language admits the intent ('deliberate design choice'), confirming the reduction. However, the curriculum-learning component is not circular: CuRLA and One-Fold CL both use the revised reward, so the CuRLA-vs-One-Fold comparison isolates the timing of traffic introduction, and that comparison is an ordinary (though single-seed and statistically underpowered) empirical claim. There is no load-bearing self-citation, no imported uniqueness theorem, and no ansatz smuggled via citation; the SCA baseline is from an external thesis. The score of 5 reflects a partial circularity: the speed result is by construction, but the central curriculum claim retains independent empirical content, weakened only by lack of seeds and error bars rather than by circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The core results rest on hand-picked reward thresholds (vmin, vtarget, vmax, maximum angle, maximum lane deviation) and a hand-picked curriculum switch point (1500 episodes), plus the assumption that CARLA's collision intensity and the borrowed VAE latent space faithfully support the training signal. No new physical or conceptual entities are introduced.

free parameters (4)
  • Speed reward thresholds (vmin, vtarget, vmax) = 15, 60, 105 km/hr
    Chosen by hand to smooth the speed reward; these values directly bias the agent toward higher speeds.
  • Maximum angle deviation alpha_max = 20 degrees
    Hand-set threshold for the angle reward; affects how strictly the agent is penalized for misalignment.
  • Maximum lane deviation d_max = 3 meters
    Used in centering reward and episode termination; hand-set.
  • Curriculum switch point = 1500 episodes
    Traffic and collision penalty are introduced after 1500 of 3500 episodes; chosen by hand, not justified by data.
assumptions (4)
  • domain assumption CARLA's collision intensity Ic is an accurate and consistent safety signal.
    Used as the basis for the collision penalty in Eq. (9); if Ic is noisy or not comparable across collisions, the penalty is not well-calibrated.
  • domain assumption The pretrained VAE from Vergara (2019) provides a suitable latent state representation for this task.
    The paper uses the pretrained VAE without retraining; this assumes transferability to their environment setup.
  • domain assumption PPO-Clip with the given hyperparameters converges reliably for all three agents.
    The paper reports only single runs; this assumes the training algorithm is stable enough that one run is representative.
  • domain assumption Town 7 is a representative autonomous-driving benchmark for evaluating the claims.
    Results are only measured in one town; generalization to other environments is asserted without evidence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CuRLA: Curriculum Learning Based Deep Reinforcement Learning for Autonomous Driving." pith.science (2026). https://pith.science/paper/2B23SYCI

@misc{pith2026250104982,
  author       = {Pith},
  title        = {Pith review of: CuRLA: Curriculum Learning Based Deep Reinforcement Learning for Autonomous Driving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2B23SYCI}},
  note         = {Machine review of arXiv:2501.04982}
}
read the original abstract

In autonomous driving, traditional Computer Vision (CV) agents often struggle in unfamiliar situations due to biases in the training data. Deep Reinforcement Learning (DRL) agents address this by learning from experience and maximizing rewards, which helps them adapt to dynamic environments. However, ensuring their generalization remains challenging, especially with static training environments. Additionally, DRL models lack transparency, making it difficult to guarantee safety in all scenarios, particularly those not seen during training. To tackle these issues, we propose a method that combines DRL with Curriculum Learning for autonomous driving. Our approach uses a Proximal Policy Optimization (PPO) agent and a Variational Autoencoder (VAE) to learn safe driving in the CARLA simulator. The agent is trained using two-fold curriculum learning, progressively increasing environment difficulty and incorporating a collision penalty in the reward function to promote safety. This method improves the agent's adaptability and reliability in complex environments, and understand the nuances of balancing multiple reward components from different feedback signals in a single scalar reward function. Keywords: Computer Vision, Deep Reinforcement Learning, Variational Autoencoder, Proximal Policy Optimization, Curriculum Learning, Autonomous Driving.

Figures

Figures reproduced from arXiv: 2501.04982 by the authors.

Figure 1
Figure 1. Variational autoencoder architecture. Figure (1) shows a variational autoencoder archi￾tecture. The encoder operates on the input vector, yielding two vectors, zµ and zσ. Then, a sample z is drawn from the distribution N(zµ;zσ), which is fed into the decoder p, producing a reconstructed signal [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Top down view of the lap in Town 7. 3 EXPERIMENTAL STUDY AND RESULT ANALYSIS In this section, we present the experimental setup, methodology, and results of our study. We start by de￾scribing the experimental environment, followed by a detailed explanation of the evaluation metrics and the baseline methods for comparison. We then present the results of our experiments and finally discuss the implications of our find… view at source ↗
Figure 3
Figure 3. PPO+VAE training architecture. The models are trained with the same parameters mentioned in (Vergara, 2019) [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: SCA Reward Function. 15 60 105 0.5 1 Speed Reward [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: CuRLA & One-Fold CL Reward Function. 4. Collision Penalty: A collision penalty factor was introduced for both One-Fold CL and CuRLA to ensure the agent explicitly learns the behaviour of safe driving, avoiding collisions with other objects and vehicles in the environme…
Figure 6
Figure 6. Figure 6: Training Metric: Distance Traveled [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 8
Figure 8. Figure 8: Evaluation Metric: Distance Traveled [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 23 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...

  2. [2]

    Baldi, P. (2011). Autoencoders, unsupervised learning, and deep architectures. In ICML Unsupervised and Transfer Learning

    work page 2011

  3. [3]

    Bengio, Y., Louradour, J., Collobert, R., and Weston, J. (2009). Curriculum learning. volume 60, page 6

    work page 2009

  4. [4]

    Dickmanns, E. D. and Zapp, B. (1987). An integrated dynamic scene analysis system for autonomous road vehicles. In Intelligent Vehicles '87 , pages 157--164. IEEE

    work page 1987

  5. [5]

    Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., and Koltun, V. (2017). Carla: An open urban driving simulator

    2017

  6. [6]

    Grigorescu, S., Trasnea, B., Cocias, T., and Macesanu, G. (2019). A survey of deep learning techniques for autonomous driving. Journal of Field Robotics , 37(3):362–386

    work page 2019

  7. [7]

    a llstr \

    Hayes, C. F., Radulescu, R., Bargiacchi, E., K \" a llstr \" o m, J., Macfarlane, M., Reymond, M., Verstraeten, T., Zintgraf, L. M., Dazeley, R., Heintz, F., Howley, E., Irissappane, A. A., Mannion, P., Now \' e , A., de Oliveira Ramos, G., Restelli, M., Vamplew, P., and Roijers, D. M. (2021). A practical guide to multi-objective reinforcement learning an...

    arXiv 2021

  8. [8]

    Kendall, A., Hawke, J., Janz, D., Mazur, P., Reda, D., Allen, J.-M., Lam, V.-D., Bewley, A., and Shah, A. (2018). Learning to drive in a day

    work page 2018

Show all 27 references
  1. [9]

    Kingma, D. P. and Welling, M. (2022). Auto-encoding variational bayes

    2022

  2. [10]

    Krizhevsky, A., Sutskever, I., and Hinton, G. (2012). Imagenet classification with deep convolutional neural networks. Neural Information Processing Systems , 25

    2012

  3. [11]

    and Leibler, R

    Kullback, S. and Leibler, R. A. (1951). On information and sufficiency. The Annals of Mathematical Statistics , 22(1):79--86

    1951

  4. [12]

    and Ibanez-Guzman, J

    Li, Y. and Ibanez-Guzman, J. (2020). Lidar for autonomous driving: The principles, challenges, and trends for automotive lidar and perception systems. IEEE Signal Processing Magazine , 37(4):50--61

    2020

  5. [13]

    P., Hunt, J

    Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2019). Continuous control with deep reinforcement learning

    2019

  6. [14]

    Liu, C., Xu, X., and Hu, D. (2015). Multiobjective reinforcement learning: A comprehensive overview. IEEE Transactions on Systems, Man, and Cybernetics: Systems , 45(3):385--398

    2015

  7. [15]

    P., Mirza, M., Graves, A., Lillicrap, T

    Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., and Kavukcuoglu, K. (2016). Asynchronous methods for deep reinforcement learning

    2016

  8. [16]

    Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. (2013). Playing atari with deep reinforcement learning

    2013

  9. [17]

    Moravec, H. (1990). Sensor fusion in autonomous vehicles. In Sensor Fusion , pages 125--153. Springer, Boston, MA

    1990

  10. [18]

    E., and Stone, P

    Narvekar, S., Peng, B., Leonetti, M., Sinapov, J., Taylor, M. E., and Stone, P. (2020). Curriculum learning for reinforcement learning domains: A framework and survey

    2020

  11. [19]

    Pomerleau, D. A. (1988). Alvinn: An autonomous land vehicle in a neural network. In Touretzky, D., editor, Advances in Neural Information Processing Systems , volume 1. Morgan-Kaufmann

    1988

  12. [20]

    E., Hinton, G

    Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1986). Learning internal representations by error propagation

    1986

  13. [21]

    I., and Abbeel, P

    Schulman, J., Levine, S., Moritz, P., Jordan, M. I., and Abbeel, P. (2017a). Trust region policy optimization

    2017

  14. [22]

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017b). Proximal policy optimization algorithms

    2017

  15. [23]

    S., McAllester, D., Singh, S., and Mansour, Y

    Sutton, R. S., McAllester, D., Singh, S., and Mansour, Y. (1999). Policy gradient methods for reinforcement learning with function approximation. In Solla, S., Leen, T., and M\" u ller, K., editors, Advances in Neural Information Processing Systems , volume 12. MIT Press

    1999

  16. [24]

    Thrun, S., Burgard, W., and Fox, D. (2005). Probabilistic Robotics (Intelligent Robotics and Autonomous Agents)

    2005

  17. [25]

    and Now \'e , A

    Van Moffaert, K. and Now \'e , A. (2014). Multi-objective reinforcement learning using sets of pareto dominating policies. The Journal of Machine Learning Research , 15(1):3483--3512

    2014

  18. [26]

    Vergara, M. L. (2019). Accelerating training of deep reinforcement learning-based autonomous driving agents through comparative study of agent and environment designs. Master thesis, NTNU

    2019

  19. [27]

    Darpa grand challenge (2005) --- Wikipedia , the free encyclopedia

    Wikipedia contributors (2024). Darpa grand challenge (2005) --- Wikipedia , the free encyclopedia

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.