Learning to Drive from a World Model

Adeeb Shihadeh; Armand du Parc Locmaria; George Hotz; Greg Hogan; Harald Sch\"afer; Kacper Raczy; Mitchell Goff; Weixing Zhang; Yassine Yousfi

REVIEW 4 major objections 5 minor 1 cited by

Learning to Drive from a World Model

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Learned world model trains a vision-only policy that steers a real car with no hand-coded rules.

desk verdict A credible systems demonstration that on-policy training in a learned world model can produce a deployable steering policy, with real but non-fatal weaknesses in the real-world evidence and a mild circularity in the action supervisor. read the letter →

arxiv 2504.19077 v1 pith:5YGKYPCY submitted 2025-04-27 cs.CV cs.RO

Mitchell Goff , Greg Hogan , George Hotz , Armand du Parc Locmaria , Kacper Raczy , Harald Schäfer , Adeeb Shihadeh , Weixing Zhang

show 1 more author

Yassine Yousfi

This is my paper · ORCID

classification cs.CVcs.RO

keywords end-to-enddrivingworldmodelon-policylearningfutureanchoringdiffusiontransformerpolicylanekeepingADAS

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Most self-driving systems are built from hand-coded perception outputs and engineered driving rules. This paper proposes an alternative: train a vision-only driving policy entirely inside data-driven simulators that reproduce real driving scenes, then deploy it in a real car. The authors describe two simulators—one that reprojects camera images from depth maps, and one based on a learned diffusion world model—and use a Future Anchored Plan Model to generate action labels for the policy during on-policy training. In closed-loop tests, the on-policy policies pass nearly all lane-centering and lane-change scenarios, while an off-policy baseline fails most; the same policies then provided steering assistance in real-world driving through an open-source ADAS platform. The paper's claim is that end-to-end learning inside a learned world model is a viable route to production driver assistance without hand-coded rules.

What carries the argument

The central machinery is the Future Anchored World Model operating as a Plan Model: a latent diffusion transformer trained with a rectified flow objective that is conditioned on a future state anchor $F=(f_s, f_e)$ and on a history of images and poses. A Plan Head on the same model predicts a multi-hypothesis 10-second future trajectory, which the Vehicle Model inverts into action labels $\hat{a}^{\text{wp}}$ used to supervise the driving policy. The future anchor supplies 'recovery pressure' so that, even when the simulator has drifted out of the training distribution, the predicted trajectory converges to a goal state rather than compounding errors. Conditioning on poses rather than actions lets the World Model ignore the Vehicle Model, so vehicle-dynamics randomization can be applied without retraining the World Model, which is what makes policies trained in simulation transfer to the real vehicle.

What would settle it

Generate a set of counterfactual driving states by running the World Model with forced lateral commands, record the Plan Model's predicted trajectory at each state, and have several human drivers produce reference trajectories from the same states in a fixed-base simulator; if the Plan Model's trajectories deviate from the human reference by more than 0.5 m in lateral position after one second on more than a small fraction of cases, the action labels that supervise the policy are not validated.

Watch

Extended reading notes

Core claim

The paper's central claim is that a vision-only driving policy can be trained on-policy in a data-driven simulator and learn meaningful driving behavior without any hand-coded rules. The key mechanism is a Future Anchored World Model that acts as both environment and supervisor: conditioned on a future state anchor, it predicts the next image and a future trajectory that converges to that anchor, and the trajectory is converted into action labels through an invertible Vehicle Model. The policy is trained to imitate those labels while interacting with the simulator, so it experiences and recovers from its own errors rather than only imitating recorded demonstrations. The paper reports that policies trained in this way pass closed-loop lane-keeping and lane-change unit tests and, when deployed in a real-world ADAS, engaged for over a quarter of total driving time. To the authors' knowledge this is the first demonstration of an end-to-end policy trained without handcrafted features being used in a real-world driver assistance system.

Load-bearing premise

The Plan Model's predicted future trajectories are used as ground-truth actions for training, but those predictions are never verified against a real driver or a safety metric in the off-distribution states the simulator can create.

Editorial extensions

If this is right

If the central claim holds, an end-to-end driving policy can be trained for new vehicles and new behaviors simply by collecting more human driving data and scaling the world model, without re-engineering perception outputs or control rules.
The future-anchoring mechanism provides a general supervision signal for counterfactual states, so the policy can recover from errors that never appear in the original demonstrations.
The same training loop should extend to longitudinal control: the paper states that the methods generalize to acceleration and braking policies, which would make the entire driving stack learned end-to-end.
The world-model simulator's image quality and pose accuracy improve with dataset and model size, so policies trained within it should keep improving as the simulator scales, unlike hand-built simulators that require manual scene construction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same 'future anchoring + learned simulator + on-policy imitation' recipe could apply to other sequential-decision domains where a goal state is available, such as robot manipulation or pedestrian navigation, not just driving.
The paper never ablates the Plan Model against the simpler baseline of using the recorded human action at each simulation state; doing so would isolate how much of the trained policy's success comes from the future-anchored labels rather than from the on-policy interaction itself.
If the Plan Model's labels are valid, then the bottleneck to safer autonomous driving shifts from policy design to the fidelity of the world model and the breadth of the human data, implying that data collection and simulation fidelity, not rule engineering, are the scaling axes.
A direct test of the world-model path would be to train the policy with the reprojective simulator's labels but the world model's images (or vice versa), separating the effect of image realism from the effect of label quality.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The genuinely new thing here is integration: a production ADAS steering policy trained on-policy inside a learned world model with future-anchored supervision, then deployed to a few hundred real users. The building blocks are known, but combining them with the Plan Model as action supervisor and showing closed-loop unit tests pass is a real engineering contribution.

The MetaDrive unit tests are the strongest part. Off-policy training fails (5/24 lane-center, 8/20 lane-change), while both on-policy methods pass (24/24 and 20/20, 19/20). That is concrete evidence the on-policy loop does what it claims. The paper also honestly lists the reprojective simulator's limitations (counterfactual problem, artifacts, shortcut learning) and uses an information bottleneck to address shortcut learning. The architecture is described in enough detail to reproduce, though no code or data are released.

Soft spots, in proportion. First, the real-world claim rests on engagement percentages from a self-selected cohort with no baseline and no safety metrics. That is suggestive, not demonstrative, and the authors' phrasing ('meaningful driver assistance') is appropriately modest. Second, lane changes are triggered by a conditioning impulse added during training and again at inference. That is a hand-coded rule, which sits awkwardly beside the 'no hand-coded driving rules' claim. Third, the Plan Model circularity flagged in the stress-test is real: the policy imitates Plan Model actions, and the Plan Model was trained on the same data with future anchors from the same recordings. At the off-distribution states on-policy training visits, the Plan Model's labels are never audited. I do not think this is load-bearing, because the evaluation is behavioral (does the policy converge to lane center and complete lane changes?) rather than label-matching, and the policy could learn good behavior despite imperfect labels. But the absence of any check on Plan Model trajectory quality for perturbed states is a genuine gap.

This deserves peer review. It is a serious systems paper with a clear training-loop claim, decent closed-loop evidence, and a deployed real-world system. The field metrics should be treated as suggestive, and the hand-coded lane-change impulse should be acknowledged as a rule. For someone working on world models for control or end-to-end driving, this is worth reading and citing.

Referee Report

4 major / 5 minor

Summary. This paper proposes a vision-only, end-to-end driving policy trained via on-policy interaction in data-driven simulators built from real human driving data. Two simulator variants are presented: a reprojective novel-view synthesis simulator and a learned Diffusion Transformer world model with future anchoring plus a Plan Model that supplies action labels. The policy is trained to imitate those Plan Model actions during simulator rollouts. The paper reports closed-loop MetaDrive unit tests in which on-policy-trained policies outperform an off-policy baseline on lane centering and lane change tasks, together with real-world openpilot ADAS deployment metrics expressed as engagement percentages.

Significance. The paper has a clear and credible core: the closed-loop unit tests in Table 2 (24/24 lane centering for both on-policy methods, 20/20 and 19/20 lane changes) are a clean demonstration that on-policy training in data-driven simulators changes behavior relative to off-policy imitation. The authors also deserve credit for an honest enumeration of reprojective simulation limitations in Section 3.1, for the information-bottleneck regularization idea in Section 5.2, and for reporting scaling trends of LPIPS with model and dataset size in Figure 5. If the Plan Model supervision can be validated on off-distribution states, the approach is a meaningful step toward scaling end-to-end driving from data. However, the real-world claim rests on weak field metrics, and the learned-supervisor loop is not audited; as written, the evidence is not commensurate with the strength of the abstract and conclusion.

major comments (4)

[5.1, Eq. (7); 2.6] The action labels \hat{a}^{wp} used to train the policy are generated by the Plan Model, which is trained on the same human-driving data that also supplies the future anchors F; the policy is therefore trained to imitate a learned expert that is never validated on the off-distribution states that on-policy rollouts are designed to visit. Section 3.1 identifies the counterfactual problem for reprojected images, but the same counterfactual issue applies to the action labels: at a state displaced by the ±0.5 m noise model or during a lane-change impulse, no human ever drove the trajectory that the Plan Model proposes. No experiment audits whether Plan Model trajectories are lane-consistent, obstacle-free, or human-approved on those perturbed states. Please add such an audit, or an ablation using a filtered or otherwise validated supervisor; the 'no hand-coded driving rules' claim depends on the supervisor being a reliable expert rather than a simulator artifact.
[5.4, Table 3] The real-world ADAS claim is not supported by the reported metrics. Table 3 gives only engagement percentage (time and distance) for the two policies over roughly 40,000–47,000 trips, with no baseline, no error bars, no disengagement reasons, no safety events, and no comparison against a hand-coded or off-policy baseline on the same cohort. Because openpilot is a level-2 system, engagement percentage is affected by user trust, trip context, and system availability, not only by policy quality. The sentence 'both policies are capable of delivering meaningful driver assistance in real-world conditions' is not derivable from this table. Please report at least a matched baseline and disengagement or safety statistics, or soften the claim to usage acceptance rather than demonstrated capability.
[5.3.1] The lane-change test relies on an externally injected conditioning impulse both during training and at inference to trigger the maneuver. This is a hand-designed trigger, so the paper does not demonstrate that the policy learns the decision of when to change lanes; it demonstrates execution of a lane change once commanded. The abstract and conclusion should be qualified accordingly, and the paper should state explicitly that lane-change initiation remains externally specified.
[5.3, 5.4] The reported evaluation is limited to lateral control in relatively simple MetaDrive scenarios, and the paper states in Section 5.3 that longitudinal metrics are deferred to future work. This is a reasonable scope choice, but the claims about learning 'driving behavior' and 'useful ADAS products' should be tied to this lateral-only scope, especially given the production system in Section 5.3.3 still uses a classical ACC policy for longitudinal control.

minor comments (5)

[4.2.2, Eq. (5)] The value of the hyperparameter \alpha is missing; the text reads 'with a hyperparameter \alpha = as described in Equation 5.' Please provide the value or a reference to where it is set.
[4.6.1, Figure 6] The text says the next-frame prediction result is 'shown in left Figure 6' and the sequential rollout result is 'shown in right Figure 6,' but the caption associates the labels in the opposite way. Please correct the mismatch.
[5.2] The claim that the information bottleneck limits the feature extractor to 'roughly 700 bits' would benefit from the explicit capacity calculation for the described Gaussian channel with additive white Gaussian noise; as written, the reader cannot reproduce the number.
[Figure 5] The scaling plots show single-curve LPIPS values without error bars or multiple seeds; please indicate the variance across seeds or state that only one run was used.
[5.4, Table 3] The field deployment section would be easier to interpret if the cohort size, the definition of an 'engaged' trip, and the collection window were stated in the table caption rather than only in the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the policy imitates a learned Plan Model, but the chain is acyclic and the central claims are validated by external MetaDrive tests and real-world deployment.

full rationale

The paper's training loop is a supervised chain, not a circle: a Future Anchored World Model (Plan Model) is trained on human driving data (Section 4.2.2, Eq. 5) to predict future poses, and the driving policy is then trained to imitate the Plan Model's action labels (Section 5.1, Eq. 7: 'the policy learns to predict the actions that the World Model would take'). This is an imitation chain human-data -> Plan Model -> policy; the Plan Model does not depend on the policy, so there is no definitional equivalence between the prediction and its inputs. The potentially circular-sounding language of Section 2.6 ('the predicted trajectory can be mapped to ground-truth actions') is a labeling choice, not a derivation: the central claims are not validated solely by agreement with the Plan Model. The policy is evaluated closed-loop in the external MetaDrive simulator (Section 5.3.1, Table 2) and deployed in a real-world ADAS (Section 5.3.3, Table 3), which are independent of the training labels. The self-citations ([25], [26]) are background references to comma.ai's prior world-model and dataset work and are not load-bearing for the new architecture or its validation. Thus no circular step meets the evidence bar.

Assumptions & free parameters 7 free parameters · 6 assumptions · 0 invented entities

The paper contributes a training architecture, not a mathematical derivation. The central claim rests on empirical assumptions about pose accuracy, the validity of Plan Model labels, the absence of simulator shortcuts, and the meaningfulness of field engagement metrics. We list these as domain assumptions rather than standard math axioms, and we record the hand-set hyperparameters that the pipeline depends on. No new physical entities are introduced.

free parameters (7)

Future anchor window = fs < 9s, fe - fs = 1s, context T = 2s
Hand-set in Section 4.5; determines how much future information is used to generate action labels and how long world-model rollouts run.
Diffusion Euler steps = 15 steps, Delta tau = 1/15
Hand-set in Section 4.3; fewer steps would change the training signal and rollout quality.
Noise augmentation probability = p = 0.3
Hand-set in Section 4.4; the paper says this augmentation was essential for robustness to autoregressive drift.
MHP planning hypotheses = 5
Hand-set for both the Plan Model and the driving policy trajectory loss in Sections 2.6 and 5.
Information bottleneck capacity = about 700 bits
Hand-set in Section 5.2 to prevent the feature extractor from encoding simulator-specific artifacts.
Loss weighting alpha = not given (Eq. 5: 'alpha =')
Controls the trade-off between the rectified flow image loss and the plan loss in the world model objective; the value is missing from the text.
Data and model scales = 100k to 400k segments; 250M to 1B DiT
Chosen as a scaling experiment in Section 4.5; the deployed policy uses the 500M model trained on 400k segments.

assumptions (6)

domain assumption The global pose estimate from the tightly coupled GPS/Vision MSCKF filter is accurate enough to condition the world model and to invert the Vehicle Model into actions.
Used in Sections 2.1 and 2.4. All ground-truth actions and all world-model transitions depend on p_t. Pose errors propagate through training and supervision and are not characterized in the paper.
domain assumption The Future Anchored Plan Model produces correct action labels even when the policy drifts to states far from the recorded data.
Sections 2.6 and 5.1 train the policy to match a_wp from the Plan Model (Eq. 7). The paper calls future anchoring 'recovery pressure' but never validates those labels against a human driver or a safety metric in drifted states.
domain assumption The learned World Model images and poses contain no shortcut signal that the policy can exploit to fool the simulator.
Sections 4.4 and 5.2 add noise augmentation and a 700-bit bottleneck precisely to reduce simulator artifacts, but no direct measurement of residual shortcut signal is reported before deployment claims are made.
domain assumption MetaDrive unit tests are a valid proxy for real driving competence.
Section 5.3.1 treats 24/24 lane-center and 20/20 lane-change passes as evidence that policies 'learn driving behavior'. The link between these synthetic closed-loop tests and road behavior is asserted, not established.
domain assumption Engagement time and distance are meaningful proxies for real-world ADAS usefulness.
Section 5.3.3 and Table 3 use engaged percentage as the primary field metric, with no control group, no disengagement-cause analysis, and no safety metric.
domain assumption The inverted Vehicle Model maps desired pose trajectories to realizable curvature and acceleration actions with enough fidelity for sim2real transfer.
Section 2.3 says the Vehicle Model includes dynamics, delayed steering response, and wind, but its parameters and validation against real vehicles are not given.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Learning to Drive from a World Model." pith.science (2026). https://pith.science/paper/5YGKYPCY

@misc{pith2026250419077,
  author       = {Pith},
  title        = {Pith review of: Learning to Drive from a World Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5YGKYPCY}},
  note         = {Machine review of arXiv:2504.19077}
}

read the original abstract

Most self-driving systems rely on hand-coded perception outputs and engineered driving rules. Learning directly from human driving data with an end-to-end method can allow for a training architecture that is simpler and scales well with compute and data. In this work, we propose an end-to-end training architecture that uses real driving data to train a driving policy in an on-policy simulator. We show two different methods of simulation, one with reprojective simulation and one with a learned world model. We show that both methods can be used to train a policy that learns driving behavior without any hand-coded driving rules. We evaluate the performance of these policies in a closed-loop simulation and when deployed in a real-world advanced driver-assistance system.

Figures

Figures reproduced from arXiv: 2504.19077 by the authors.

**Figure 2.** Left: Top: Image at T. Bottom: Depth map at T. Right: Reprojected images at T using 4 different translation vectors [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Left: Image at T, Right: Reprojected Images at T. Notice the lighting artifacts in the reprojected images. To make it a Plan Model, the Transformer is equipped with a Plan Head, which is a stack of residual Feed Forward blocks. The Plan Head predicts the trajectory T . 4.2.2. Training objective We adopt the Rectified Flow (RF) objective [16] for training the Conditional Diffusion Transformer. For simplicity, we o… view at source ↗

Figures from the paper (8 more)

**Figure 4.** Figure 4: Five examples of World Model simulation. Blue bordered frames are the last frames of the past context, red bordered frames are [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: Left: LPIPS for different DiT model sizes, trained on [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]

**Figure 7.** Figure 7: Action teacher-forced sequential rollout pose errors. [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]

**Figure 6.** Figure 6: LPIPS for right: observation and action teacher-forced [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]

**Figure 8.** Figure 8: Action noise-forced sequential rollout. We force a [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]

**Figure 9.** Figure 9: Deviation from the lane center in noise forced simula [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]

**Figure 10.** Figure 10: Examples of the MetaDrive simulated unit test scenar [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]

**Figure 11.** Figure 11: Left: Convergence to lane-center test results. Right: [PITH_FULL_IMAGE:figures/full_fig_p008_11.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Interpretability and Generalization Bounds for Learning Spatial Physics
cs.LG 2025-06 conditional novelty 6.0 of 10

A learned linear solver converges only to the projection of the true solution operator onto the training-data function space, and richer polynomial training data can actually increase the finite-difference parameter bias.

Reference graph

Works this paper leans on

33 extracted references · 25 canonical work pages · cited by 1 Pith paper

[1]

A framework for be- havioural cloning

Michael Bain and Claude Sammut. A framework for be- havioural cloning. In Machine intelligence 15 , pages 103– 129, 1995. 1

work page 1995
[2]

Video pretraining (vpt): Learning to act by watching unlabeled online videos

Bowen Baker, Ilge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampe- dro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24639–24654, 2022. 2

work page 2022
[3]

Navigation world models, 2024

Amir Bar, Gaoyue Zhou, Danny Tran, Trevor Darrell, and Yann LeCun. Navigation world models, 2024. 3

work page 2024
[4]

End to end learning for self-driving cars

Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316, 2016. 1

arXiv 2016
[5]

Multimodal trajectory predictions for autonomous driving using deep convolutional networks

Henggang Cui, Vladan Radosavljevic, Fang-Chieh Chou, Tsung-Han Lin, Thi Nguyen, Tzu-Kuo Huang, Jeff Schnei- der, and Nemanja Djuric. Multimodal trajectory predictions for autonomous driving using deep convolutional networks. In 2019 international conference on robotics and automation (icra), pages 2090–2096. IEEE, 2019. 4

work page 2019
[6]

CARLA: An open urban driving simulator

Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In Proceedings of the 1st Annual Conference on Robot Learning, pages 1–16, 2017. 2

2017
[7]

Impala: Scalable dis- tributed deep-rl with importance weighted actor-learner ar- chitectures

Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Si- monyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al. Impala: Scalable dis- tributed deep-rl with importance weighted actor-learner ar- chitectures. In International conference on machine learn- ing, pages 1407–1416. PMLR, 2018. 7

work page 2018
[8]

Scaling rec- tified flow transformers for high-resolution image synthesis

Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. Scaling rec- tified flow transformers for high-resolution image synthesis. In Proceedings of the 41st International Conference on Ma- chine Learnin...

work page 2024

Show all 33 references

[9]

Shortcut learning in deep neural networks

Robert Geirhos, J ¨orn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Fe- lix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673, 2020. 3

2020
[10]

Recurrent world models facilitate policy evolution

David Ha and J ¨urgen Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neural Informa- tion Processing Systems 31, pages 2451–2463. Curran Asso- ciates, Inc., 2018. https://worldmodels.github. io. 1

2018
[11]

Gaia-1: A generative world model for au- tonomous driving, 2023

Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gian- luca Corrado. Gaia-1: A generative world model for au- tonomous driving, 2023. 1

2023
[12]

V ehicle dynamics

Reza N Jazar. V ehicle dynamics. Springer, 2008. 2

2008
[13]

Learning to drive in a day

Alex Kendall, Jeffrey Hawke, David Janz, Przemyslaw Mazur, Daniele Reda, John-Mark Allen, Vinh-Dieu Lam, Alex Bewley, and Amar Shah. Learning to drive in a day. In 2019 international conference on robotics and automation (ICRA), pages 8248–8254. IEEE, 2019. 1

2019
[14]

Mourikis

Mingyang Li and Anastasios I. Mourikis. High-precision, consistent ekf-based visual-inertial odometry. The Interna- tional Journal of Robotics Research , 32(6):690–711, 2013. 2

2013
[15]

Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning

Quanyi Li, Zhenghao Peng, Lan Feng, Qihang Zhang, Zhenghai Xue, and Bolei Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. IEEE Transactions on Pattern Analysis and Ma- chine Intelligence, 2022. 2, 7

2022
[16]

Flow straight and fast: Learning to generate and transfer data with rectified flow

Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003, 2022. 4

2022 arXiv
[17]

Mourikis and Stergios I

Anastasios I. Mourikis and Stergios I. Roumeliotis. A multi- state constraint kalman filter for vision-aided inertial naviga- tion. In Proceedings 2007 IEEE International Conference on Robotics and Automation, pages 3565–3572, 2007. 2

2007
[18]

Estimating the mean and variance of the target probability distribution

David A Nix and Andreas S Weigend. Estimating the mean and variance of the target probability distribution. In Pro- ceedings of 1994 ieee international conference on neural networks (ICNN’94), pages 55–60. IEEE, 1994. 4

1994
[19]

Scalable diffusion models with transformers

William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF inter- national conference on computer vision , pages 4195–4205,
[20]

Sim-to-real transfer of robotic con- trol with dynamics randomization

Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic con- trol with dynamics randomization. In 2018 IEEE Inter- national Conference on Robotics and Automation (ICRA) , pages 3803–3810, 2018. 2

2018
[21]

Language models are unsu- pervised multitask learners

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsu- pervised multitask learners. OpenAI blog, 1(8):9, 2019. 5

2019
[22]

Gaussian dropout as an information bottleneck layer

M ´elanie Rey and Andriy Mnih. Gaussian dropout as an information bottleneck layer. In NeurIPS Workshop on Bayesian Deep Learning, 2021. 7

2021
[23]

High-resolution image synthesis with latent diffusion models

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 3

2022
[24]

CAD2RL: real single- image flight without a single real image

Fereshteh Sadeghi and Sergey Levine. CAD2RL: real single- image flight without a single real image. In Robotics: Sci- ence and Systems XIII, Massachusetts Institute of Technol- ogy, Cambridge, Massachusetts, USA, July 12-16, 2017 ,

2017
[25]

Learning a driving simulator,

Eder Santana and George Hotz. Learning a driving simulator,
[26]

A commute in data: The comma2k19 dataset, 2018

Harald Schafer, Eder Santana, Andrew Haden, and Riccardo- Biasini. A commute in data: The comma2k19 dataset, 2018. 2

2018
[27]

View morphing

Steven M Seitz and Charles R Dyer. View morphing. In Pro- ceedings of the 23rd annual conference on Computer graph- ics and interactive techniques, pages 21–30, 1996. 1

1996
[28]

Domain randomization for transferring deep neural networks from simulation to the real world

Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Woj- ciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 2...

2017
[29]

Diffusion models are real-time game engines

Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines. In The Thirteenth International Conference on Learning Rep- resentations, 2025. 4

2025
[30]

Fastvit: A fast hybrid vision transformer using structural reparameterization

Pavan Kumar Anasosalu Vasu, James Gabriel, Jeff Zhu, On- cel Tuzel, and Anurag Ranjan. Fastvit: A fast hybrid vision transformer using structural reparameterization. In Proceed- ings of the IEEE/CVF international conference on computer vision, pages 5785–5795, 2023. 7

2023
[31]

Attention is all you need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 7

2017
[32]

Understanding and improving layer normaliza- tion

Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, and Junyang Lin. Understanding and improving layer normaliza- tion. Advances in neural information processing systems, 32,
[33]

The unreasonable effectiveness of deep features as a perceptual metric

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shecht- man, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 586–595, 2018. 5

2018

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

A framework for be- havioural cloning

Michael Bain and Claude Sammut. A framework for be- havioural cloning. In Machine intelligence 15 , pages 103– 129, 1995. 1

work page 1995

[2] [2]

Video pretraining (vpt): Learning to act by watching unlabeled online videos

Bowen Baker, Ilge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampe- dro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24639–24654, 2022. 2

work page 2022

[3] [3]

Navigation world models, 2024

Amir Bar, Gaoyue Zhou, Danny Tran, Trevor Darrell, and Yann LeCun. Navigation world models, 2024. 3

work page 2024

[4] [4]

End to end learning for self-driving cars

Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316, 2016. 1

arXiv 2016

[5] [5]

Multimodal trajectory predictions for autonomous driving using deep convolutional networks

Henggang Cui, Vladan Radosavljevic, Fang-Chieh Chou, Tsung-Han Lin, Thi Nguyen, Tzu-Kuo Huang, Jeff Schnei- der, and Nemanja Djuric. Multimodal trajectory predictions for autonomous driving using deep convolutional networks. In 2019 international conference on robotics and automation (icra), pages 2090–2096. IEEE, 2019. 4

work page 2019

[6] [6]

CARLA: An open urban driving simulator

Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In Proceedings of the 1st Annual Conference on Robot Learning, pages 1–16, 2017. 2

2017

[7] [7]

Impala: Scalable dis- tributed deep-rl with importance weighted actor-learner ar- chitectures

Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Si- monyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al. Impala: Scalable dis- tributed deep-rl with importance weighted actor-learner ar- chitectures. In International conference on machine learn- ing, pages 1407–1416. PMLR, 2018. 7

work page 2018

[8] [8]

Scaling rec- tified flow transformers for high-resolution image synthesis

Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. Scaling rec- tified flow transformers for high-resolution image synthesis. In Proceedings of the 41st International Conference on Ma- chine Learnin...

work page 2024