Pith. sign in

REVIEW 2 major objections 4 minor 60 references

Mirror learning turns third-person observation into first-person training data: watch a demonstrator, synthesize its viewpoint, infer the actions, and behavior-clone from the result.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 00:32 UTC pith:3KG5URAO

load-bearing objection A genuinely new geometry-free mirror-learning pipeline with careful splits, but the action-labeling step under distribution shift is unmeasured and Table 1 conflates data quantity with mirror value. the 2 major comments →

arxiv 2607.28737 v1 pith:3KG5URAO submitted 2026-07-30 cs.LG cs.CVcs.RO

Mirror Learning

Yunpeng Liu , Matthew Niedoba , Oluwanifemi A. Adekanye , Jason Yoo , Yingchen He , Berend Zwartsenberg , Frank Wood This is my paper
classification cs.LG cs.CVcs.RO
keywords mirror learningthird-person imitation learningbehavior cloning from observationinverse dynamics modelvideo diffusion modelviewpoint transformationpseudo first-person dataautonomous driving policy
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that a policy can be learned from purely third-person demonstrations: watch another agent, synthesize what that agent saw from its own first-person viewpoint, infer the actions the learner would need to take in that viewpoint, and behavior-clone from the resulting 'mirror data'. The central empirical claim is that mirror data alone can train an effective driving policy, and that mixing it with ground-truth first-person data improves open-loop performance while outperforming style-transfer augmentation in a new-town expansion setting. If true, imitation learning no longer depends on teleoperation-heavy first-person data collection: any observation of another agent, from another viewpoint, becomes usable training signal. The authors also report zero-shot transfer of the view-synthesis step from simulated training to real-world robotaxi footage, suggesting the pretrained generative model already encodes enough scene structure for the transformation.

Core claim

The central claim is that a fine-tuned video diffusion model can place a learner 'in the shoes' of a demonstrator: conditioned on the learner's own ego-view video plus a binary mask identifying the demonstrator, it generates a plausible first-person video from the demonstrator's viewpoint, without any known camera pose or explicit geometry. A separately trained inverse dynamics model then turns that generated video into action labels in the learner's control space, by predicting the actions the learner would take to produce those observations. The paper argues that these two stages compose into 'mirror data' — synthetic first-person expert demonstrations — and that this data is sufficient by

What carries the argument

The key machinery is the composition of two learned components. The mirror video model (MVM) is a pretrained video diffusion model, fine-tuned with a flow-matching objective, that generates the demonstrator's ego-view video from the learner's video and a per-frame binary attention mask identifying the demonstrator; the three temporally aligned input streams (mask, learner video, noised target) are encoded together and share aligned positional embeddings so the model can transform viewpoint while preserving scene motion. The inverse dynamics model (IDM) is a deterministic neural head trained on the learner's own observation-action pairs to predict the action between frames from a short observ

Load-bearing premise

The load-bearing premise is that the inverse dynamics model, trained on the learner's real first-person observation-action pairs, assigns correct action labels to the synthesized demonstrator-view videos; no evaluation in the paper measures IDM accuracy on MVM outputs, so if generation errors mislead the IDM, the mirror data's action labels are wrong and behavior cloning from it degrades.

What would settle it

Take the trained IDM and apply it to MVM-generated videos of held-out simulator scenarios where the demonstrator's true executed actions are known; compare predicted actions against those ground-truth actions. If the IDM's error on mirrored video is close to its reported validation error (about 0.42 ADE), the pipeline's label quality is confirmed; if it degrades toward or beyond the 0.67 ADE seen after only 1k-sample fine-tuning, the central claim that mirror data is trustworthy training signal is not supported.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the claim holds, a physical system can acquire new behaviors by watching other agents, eliminating the need to instrument or teleoperate the demonstrator; the paper estimates each mirror sample is worth roughly half a ground-truth first-person sample.
  • In geographic expansion, augmenting a small ground-truth set with mirror data improves open-loop policy metrics both in the training environment and in a zero-shot held-out town, and beats style-transfer augmentation at matched ratios.
  • A view-transformation model trained only in simulation transfers zero-shot to real-world footage, so synthetic paired data may suffice to elicit the capability in new domains.
  • Because the policy is trained entirely from observation, mirror learning offers a safer route to data collection for high-cost tasks where trial-and-error or teleoperation is dangerous.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural test the paper leaves open is measuring IDM error specifically on MVM-generated videos against simulator ground-truth actions; the reported validation error is on real learner observations, so label noise on synthetic video is unquantified and determines how far mirror data can be trusted.
  • The IDM also predicts an aleatoric variance, which is discarded when taking the mean action as the label; an extension is to use that uncertainty to weight or reject low-confidence mirror samples, potentially improving downstream behavior cloning.
  • The approach is demonstrated on low-dimensional action spaces (steering and acceleration); applying it to high-degree-of-freedom embodiments, where inverse dynamics ambiguity is much larger, is the most direct stress test of the watch-and-learn principle.
  • Because the MVM inpaints unobserved content such as occluded vehicles, buildings, and player appearance, policies may learn implicitly from imagined scene structure; distinguishing beneficial generalization under occlusion from harmful sensitivity to generative hallucinations is a targeted next experiment.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper introduces mirror learning, a pipeline for behavior cloning from third-person demonstrations. A fine-tuned video diffusion model (MVM), conditioned on a learner's egocentric video and an instance mask of a demonstrator, synthesizes the demonstrator's first-person video; an inverse dynamics model (IDM) trained on the learner's own observation-action pairs then labels the synthesized video with actions (Eq. 1). The resulting 'mirror data' is used to train BC policies. Experiments in CARLA show (i) BC from mirror data alone can produce non-trivial policies, and (ii) adding mirror data to ground-truth data improves open-loop minADE5, outperforming style-transfer augmentation in a geographic-expansion setting. Additional results include zero-shot MVM transfer to real May Mobility data and qualitative/quantitative MVM results in CARLA and Minecraft, together with a release plan.

Significance. If the central claim holds, the paper makes a valuable step: it demonstrates that pretrained video diffusion priors plus a mask can approximate cross-agent viewpoint transfer without camera geometry, and that the resulting synthetic first-person data can support policy learning. The paper is careful about data splits: scenario-level splits, disjoint mirror-generation split, held-out town, and separated evaluation conditions (App. A.1.1, A.5). It also ships quantitative MVM metrics and a reproducibility plan. The main reservation is that the composition in Eq. (1) is not validated where it matters: IDM accuracy on MVM-generated frames is never measured, so the policy-learning results rest on an untested distribution-shift assumption.

major comments (2)
  1. [§3.2, Eq. (1), App. A.5] The load-bearing step is p_phi(a | o_hat) with o_hat drawn from the MVM. The IDM is trained exclusively on real learner observation-action pairs and achieves ADE 0.42 on its real-frame validation split; after fine-tuning on 1k Town03 samples the ADE rises to 0.67 (App. A.5). MVM outputs are not real frames: they can contain hallucinated buildings, altered lane markings, invented traffic (Fig. 10), and arbitrary avatar appearance (Fig. 4). Because actions are regressed from pixels, systematic visual artifacts could bias the IDM labels. No experiment measures IDM label accuracy on MVM outputs, and open-loop minADE5 cannot separate label bias from label quality. I request a direct evaluation: run the MVM on held-out paired learner-demonstrator clips, apply the IDM to the MVM-generated video, and compare predicted actions with the true actions (or with IDM predictions on the ground-truth dem
  2. [Table 1 (caption and rows)] The caption states that the GT+mirror rows have approximately the same number of samples (33.6k) and were formed by replacing part of the ground-truth data with mirror data. The rows, however, show GT only at 23.5k and 27.9k and GT+mirror at 23.5+10.0=33.5k and 27.9+5.6=33.5k. Thus mirror data is added on top of the same GT amounts rather than replacing GT data. The observed improvement could therefore be due simply to increased training data, not to mirror data quality. Either re-run with fixed total budgets (e.g., 33.5k GT-only vs 23.5k GT + 10k mirror) or revise the caption and the 'replacing' claim.
minor comments (4)
  1. [§3.1 vs App. A.1.1] The main text says 'around twenty-two thousand paired demonstrator/learner videos (8.5 seconds each)' while the appendix says 37,000 paired videos at six seconds, and Table 2 totals 21,966 clips. Please reconcile these numbers and the frame counts.
  2. [Fig. 5(a)] The caption claims mirror-only data leads to 'equivalent performance' with GT data. At each data size the mirror curve is visibly worse (e.g., 1.23 vs 1.04 at 2k), so the wording overstates the result. Suggest 'approaches GT performance' or quantify the gap.
  3. [§3.1, App. A.4] The MVM evaluation reports absolute FVD/PSNR/SSIM but no quantitative comparison to existing third-to-first baselines such as EgoX or WorldWander. A baseline row would help contextualize the claims and support the related-work discussion.
  4. [App. A.1.1] There is a typographical inconsistency in the split rationale: the appendix at one point refers to 'MWM' instead of 'MVM'. Minor, but should be fixed.

Circularity Check

0 steps flagged

No material circularity: mirror-data labels are not the policy target by construction, and downstream policy evaluation is held out.

full rationale

The derivation chain is compositional, not self-referential. The MVM (Sec. 2.1) is trained to predict a demonstrator ego-video from a learner video and mask; the IDM is trained on real learner observation-action pairs with Eq. (2). Mirror data is then formed by Eq. (1), sampling an MVM output and labeling it with the IDM. The downstream BC policy is trained on these tuples and evaluated with minADE5 on disjoint test splits (App. A.1.1, A.5). No fitted parameter is renamed as a prediction: the IDM validation ADE (0.42, or 0.67 after Town03 fine-tuning) is a label-quality diagnostic, not the policy evaluation metric, and the policy itself is not used to generate the mirror labels. The paper's self-citations ([20], [23], [34], [60]) are dataset/tooling references, not load-bearing uniqueness theorems or ansatz sources; the central capability claim is supported by the paper's own held-out-town and real-world zero-shot experiments. The concern that the IDM is validated on real frames but applied to MVM-synthesized frames is a distribution-shift and correctness risk, not a circularity, because the downstream result is not equal to an input by construction.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 0 invented entities

The paper's claims rest on two domain assumptions about generalization: (1) that a binary mask plus a pretrained video diffusion model is enough to infer relative pose and render the demonstrator's view without explicit geometry; and (2) that an IDM trained on real learner observations labels synthetic MVM views correctly. Both are plausible but only indirectly supported. The CARLA evaluation also relies on a shared egocentric camera convention across vehicles, which narrows the setting to same-camera-type agents. No invented physical entities are introduced; 'mirror data' is a synthetic dataset artifact, not a new physical object.

free parameters (2)
  • CARLA pair visibility thresholds = 70° FOV, 24 m range, ≥30-timestep visible window, speed >0.1 m/s
    Hand-chosen filters that define which paired demonstrations enter MVM training and mirror-data generation; changing them changes the task distribution and could affect the measured mirror-learning gains.
  • Mirror augmentation ratio = 1× and 2× relative to ground truth
    Experimental design choice; reported benefits are specific to these ratios, with no sweep beyond 2× and no evidence about saturation or failure at higher ratios.
axioms (5)
  • domain assumption A binary instance segmentation mask of the demonstrator is sufficient conditioning for cross-view synthesis; no camera pose, intrinsics, or geometry is required.
    Central to MVM; the model must implicitly infer relative pose from mask and learner video. Stated in §2.2 and relied on throughout §3.1.
  • domain assumption The IDM trained on a learner's own first-person observation-action pairs generalizes to synthetic MVM-generated videos from the demonstrator's viewpoint.
    Powers Eq. (1); if IDM labels for hallucinated MVM frames are wrong, mirror data is mislabeled. IDM ADE 0.42 on validation is the only evidence, and that validation contains real observations, not MVM outputs.
  • domain assumption All vehicles in CARLA share an egocentric camera convention, so mirrored videos can be interpreted in a common action space.
    Acknowledged in App. A.1.1; this constrains the claim to near-coplanar, same-camera-type agents and limits generalization to arbitrary embodiments with different camera placements.
  • domain assumption Cosmos-Predict2.5's pretrained representations are rich enough to support mirror synthesis after fine-tuning on modest simulation data.
    Core premise tested in §3.1; supported only by qualitative samples and FVD/PSNR/SSIM without baselines, so the reader cannot calibrate whether these values indicate task success.
  • domain assumption Simulated CARLA data with a learned behavior model is representative enough for zero-shot transfer to real robotaxi scenes.
    Zero-shot May results are qualitative and on fewer than 20 scenes; no quantitative metric is reported, so sim-to-real generalization is asserted, not measured.

pith-pipeline@v1.3.0-alltime-deepseek · 19878 in / 16289 out tokens · 154598 ms · 2026-08-03T00:32:00.648563+00:00 · methodology

0 comments
read the original abstract

We investigate imitation learning through the lens of third-person observation and propose a framework for mirror learning: acquiring actionable policies from passive observation. While behavior cloning (BC) excels under dense, well-aligned first-person data, it fundamentally fails to leverage the rich observational signals arising from third-person demonstrations that humans and animals routinely exploit. We introduce a method that composes (i) a learned perspective transformation that places learners in demonstrators' shoes using a fine-tuned video diffusion model and (ii) an inverse dynamics model that infers action trajectories in the learners' control space. This enables the synthesis of mirror data, pseudo first-person expert data generated from third-person observations of demonstrator behavior. Empirically, we show that mirror data alone can train effective policies, and that augmenting first-person BC training with mirror data further improves downstream policy performance. Our results suggest that modern generative world models implicitly encode sufficient structure to enable a scalable and safe alternative to teleoperation-heavy data collection.

Figures

Figures reproduced from arXiv: 2607.28737 by Berend Zwartsenberg, Frank Wood, Jason Yoo, Matthew Niedoba, Oluwanifemi A. Adekanye, Yingchen He, Yunpeng Liu.

Figure 1
Figure 1. Figure 1: Mirror video model: test environment video (CARLA town excluded from training). Top [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Architecture of our mirror video model (MVM). Given a learner ego-view video and a binary attention-mask video that identifies a demonstrator, the MVM predicts the corresponding demonstrator ego-view video. During training, we fine-tune COSMOS-PREDICT 2.5-2B to denoise a noised latent encoding of the target demonstrator ego-view conditioned on the learner video and the demonstrator mask. The temporally ali… view at source ↗
Figure 3
Figure 3. Figure 3: Zero-shot mirror video model predictions on real-world data (MVM trained only on [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Minecraft mirror video model samples. Demonstrator instance segmentation mask indicated [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: (a) BC from mirror data alone suffices for policy learning. GT data is always best, but larger quantities of mirror-only data lead to equivalent performance. (b) Geographic expansion into a new town. Here we fine-tune a model (first row [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Data generation pipeline. Diverse multi-agent traffic scenarios are generated in Bird’s￾eye-view CARLA maps using a realistic scenario generator trained on real-world trajectories [34]. After task-specific filtering for our mirror learning setting, the selected scenarios are replayed in the simulator to record candidate paired observations and the agents’ executed trajectories, while demonstrator instance … view at source ↗
Figure 7
Figure 7. Figure 7: Bird’s-eye-view traffic scenarios generated by a multi-agent driving behavior model [ [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Spatial trajectory density comparison between the test split and the remaining data from [PITH_FULL_IMAGE:figures/full_fig_p016_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Qualitative mirroring results in the held-out town for MVM. In each example, the top row [PITH_FULL_IMAGE:figures/full_fig_p019_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Qualitative diversity results in the held-out town for MVM. In each example, the top row [PITH_FULL_IMAGE:figures/full_fig_p020_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Qualitative passing-diversity results in the held-out town. In these challenging pass-by [PITH_FULL_IMAGE:figures/full_fig_p021_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Qualitative zero-shot mirroring results on real-world data using our mirror video model [PITH_FULL_IMAGE:figures/full_fig_p022_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Qualitative zero-shot mirroring results on real-world data using our mirror video model [PITH_FULL_IMAGE:figures/full_fig_p023_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Validation set qualitative mirroring results for a scenario the mirror network has not [PITH_FULL_IMAGE:figures/full_fig_p025_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Validation set qualitative mirroring results for a scenario the mirror network has not [PITH_FULL_IMAGE:figures/full_fig_p026_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Style-transfer examples using text prompts and edge conditioning with Cosmos-Transfer [PITH_FULL_IMAGE:figures/full_fig_p028_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Demonstrator-specific view transformation in CARLA. In each example, MVM is [PITH_FULL_IMAGE:figures/full_fig_p029_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Additional demonstrator-specific view transformation results in CARLA Test locations [PITH_FULL_IMAGE:figures/full_fig_p030_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Training set qualitative mirroring results for a scenario the mirror network has been trained [PITH_FULL_IMAGE:figures/full_fig_p031_19.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

60 extracted references · 15 linked inside Pith

  1. [1]

    Learning to poke by poking: Experiential learning of intuitive physics.Advances in neural information processing systems, 29, 2016

    Pulkit Agrawal, Ashvin V Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Learning to poke by poking: Experiential learning of intuitive physics.Advances in neural information processing systems, 29, 2016

    2016

  2. [2]

    World simulation with video foundation models for physical ai.arXiv preprint arXiv:2511.00062, 2025

    Arslan Ali, Junjie Bai, Maciej Bala, Yogesh Balaji, Aaron Blakeman, Tiffany Cai, Jiaxin Cao, Tianshi Cao, Elizabeth Cha, Yu-Wei Chao, et al. World simulation with video foundation models for physical ai.arXiv preprint arXiv:2511.00062, 2025

    Pith/arXiv arXiv 2025

  3. [3]

    Cosmos-reason1: From physical common sense to embodied reasoning.arXiv preprint arXiv:2503.15558, 2025

    Alisson Azzolini, Junjie Bai, Hannah Brandon, Jiaxin Cao, Prithvijit Chattopadhyay, Huayu Chen, Jinju Chu, Yin Cui, Jenna Diamond, Yifan Ding, et al. Cosmos-reason1: From physical common sense to embodied reasoning.arXiv preprint arXiv:2503.15558, 2025

    Pith/arXiv arXiv 2025

  4. [4]

    Video pretraining (vpt): Learning to act by watching unlabeled online videos.Advances in Neural Information Processing Systems, 35:24639–24654, 2022

    Bowen Baker, Ilge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos.Advances in Neural Information Processing Systems, 35:24639–24654, 2022

    2022

  5. [5]

    Prentice-hall Englewood Cliffs, NJ, 1977

    Albert Bandura and Richard H Walters.Social learning theory, volume 1. Prentice-hall Englewood Cliffs, NJ, 1977

    1977

  6. [6]

    Vavim and vavam: Autonomous driving through video generative modeling.arXiv preprint arXiv:2502.15672, 2025

    Florent Bartoccioni, Elias Ramzi, Victor Besnier, Shashanka Venkataramanan, Tuan-Hung Vu, Yihong Xu, Loick Chambon, Spyros Gidaris, Serkan Odabas, David Hurych, et al. Vavim and vavam: Autonomous driving through video generative modeling.arXiv preprint arXiv:2502.15672, 2025

    Pith/arXiv arXiv 2025

  7. [7]

    Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023

    Pith/arXiv arXiv 2023

  8. [8]

    End to end learning for self-driving cars.arXiv preprint arXiv:1604.07316, 2016

    Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning for self-driving cars.arXiv preprint arXiv:1604.07316, 2016

    Pith/arXiv arXiv 2016

  9. [9]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, et al. Video generation models as world simulators. 2024. URL https://openai. com/research/video-generation-models-as-world-simulators, 3(1):3, 2024

    2024

  10. [10]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  11. [11]

    Sam 3: Segment anything with concepts, 2025

    Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, Jie Lei, Tengyu Ma, Baishan Guo, Arpit Kalla, Markus Marks, Joseph Greer, Meng Wang, Peize Sun, Roman Rädle, Triantafyllos Afouras, Effrosyni Mavroudi, Katherine Xu, Tsung-Han Wu, Yu Zhou, Liliane ...

    Pith/arXiv arXiv 2025

  12. [12]

    Learning from all vehicles

    Dian Chen and Philipp Krähenbühl. Learning from all vehicles. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17222–17231, 2022

    2022

  13. [13]

    CARLA: An open urban driving simulator

    Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. InProceedings of the 1st Annual Conference on Robot Learning, pages 1–16, 2017

    2017

  14. [14]

    Rap: 3d rasterization augmented end-to-end planning.arXiv preprint arXiv:2510.04333, 2025

    Lan Feng, Yang Gao, Eloi Zablocki, Quanyi Li, Wuyang Li, Sichao Liu, Matthieu Cord, and Alexandre Alahi. Rap: 3d rasterization augmented end-to-end planning.arXiv preprint arXiv:2510.04333, 2025. 10

    arXiv 2025

  15. [15]

    Panoptic nuscenes: A large-scale benchmark for lidar panoptic segmentation and tracking.arXiv preprint arXiv:2109.03805, 2021

    Whye Kit Fong, Rohit Mohan, Juana Valeria Hurtado, Lubing Zhou, Holger Caesar, Oscar Beijbom, and Abhinav Valada. Panoptic nuscenes: A large-scale benchmark for lidar panoptic segmentation and tracking.arXiv preprint arXiv:2109.03805, 2021

    Pith/arXiv arXiv 2021

  16. [16]

    Efros, and Xiaolong Wang

    Yang Fu, Sifei Liu, Amey Kulkarni, Jan Kautz, Alexei A. Efros, and Xiaolong Wang. Colmap- free 3d gaussian splatting. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20796–20805, June 2024

    2024

  17. [17]

    A survey for foundation models in autonomous driving

    Haoxiang Gao, Zhongruo Wang, Yaqian Li, Kaiwen Long, Ming Yang, and Yiqing Shen. A survey for foundation models in autonomous driving. In2025 6th International Conference on Computer Vision and Data Mining (ICCVDM), pages 63–71. IEEE, 2025

    2025

  18. [18]

    A comprehensive survey on safe reinforcement learning

    Javier Garcıa and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437–1480, 2015

    2015

  19. [19]

    Shortcut learning in deep neural networks.Nature Machine Intelligence, 2(11):665–673, 2020

    Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks.Nature Machine Intelligence, 2(11):665–673, 2020

    2020

  20. [20]

    Semantically consistent video inpainting with conditional diffusion models.arXiv preprint arXiv:2405.00251, 2024

    Dylan Green, William Harvey, Saeid Naderiparizi, Matthew Niedoba, Yunpeng Liu, Xiaoxuan Liang, Jonathan Lavington, Ke Zhang, Vasileios Lioutas, Setareh Dabiri, et al. Semantically consistent video inpainting with conditional diffusion models.arXiv preprint arXiv:2405.00251, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    Learning latent dynamics for planning from pixels

    Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. InInternational conference on machine learning, pages 2555–2565. PMLR, 2019

    2019

  22. [22]

    Flexible diffusion modeling of long videos.Advances in neural information processing systems, 35:27953–27965, 2022

    William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank Wood. Flexible diffusion modeling of long videos.Advances in neural information processing systems, 35:27953–27965, 2022

    2022

  23. [23]

    Plaicraft: Large-scale time-aligned vision-speech-action dataset for embodied ai.arXiv preprint arXiv:2505.12707, 2025

    Yingchen He, Christian D Weilbach, Martyna E Wojciechowska, Yuxuan Zhang, and Frank Wood. Plaicraft: Large-scale time-aligned vision-speech-action dataset for embodied ai.arXiv preprint arXiv:2505.12707, 2025

    arXiv 2025

  24. [24]

    Automatic imitation.Psychological bulletin, 137(3):463, 2011

    Cecilia Heyes. Automatic imitation.Psychological bulletin, 137(3):463, 2011

    2011

  25. [25]

    Eight problems for the mirror neuron theory of action understanding in monkeys and humans.Journal of cognitive neuroscience, 21(7):1229–1243, 2009

    Gregory Hickok. Eight problems for the mirror neuron theory of action understanding in monkeys and humans.Journal of cognitive neuroscience, 21(7):1229–1243, 2009

    2009

  26. [26]

    Video diffusion models.Advances in neural information processing systems, 35: 8633–8646, 2022

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models.Advances in neural information processing systems, 35: 8633–8646, 2022

    2022

  27. [27]

    Cortical mechanisms of human imitation.science, 286(5449):2526–2528, 1999

    Marco Iacoboni, Roger P Woods, Marcel Brass, Harold Bekkering, John C Mazziotta, and Giacomo Rizzolatti. Cortical mechanisms of human imitation.science, 286(5449):2526–2528, 1999

    1999

  28. [28]

    Egox: Egocentric video generation from a single exocentric video.arXiv preprint arXiv:2512.08269, 2025

    Taewoong Kang, Kinam Kim, Dohyeon Kim, Minho Park, Junha Hyung, and Jaegul Choo. Egox: Egocentric video generation from a single exocentric video.arXiv preprint arXiv:2512.08269, 2025

    arXiv 2025

  29. [29]

    Somatosensation in social perception

    Christian Keysers, Jon H Kaas, and Valeria Gazzola. Somatosensation in social perception. Nature Reviews Neuroscience, 11(6):417–428, 2010

    2010

  30. [30]

    Deep learning.nature, 521(7553):436–444, 2015

    Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning.nature, 521(7553):436–444, 2015

    2015

  31. [31]

    End-to-end training of deep visuomotor policies.Journal of Machine Learning Research, 17(39):1–40, 2016

    Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies.Journal of Machine Learning Research, 17(39):1–40, 2016

    2016

  32. [32]

    Diffueraser: A diffusion model for video inpainting.arXiv preprint arXiv:2501.10018, 2025

    Xiaowen Li, Haolan Xue, Peiran Ren, and Liefeng Bo. Diffueraser: A diffusion model for video inpainting.arXiv preprint arXiv:2501.10018, 2025. 11

    Pith/arXiv arXiv 2025

  33. [33]

    Multiagent multitraversal multimodal self-driving: Open mars dataset

    Yiming Li, Zhiheng Li, Nuo Chen, Moonjun Gong, Zonglin Lyu, Zehong Wang, Peili Jiang, and Chen Feng. Multiagent multitraversal multimodal self-driving: Open mars dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22041–22051, 2024

    2024

  34. [34]

    Control-itra: Controlling the behavior of a driving model.arXiv preprint arXiv:2501.12408, 2025

    Vasileios Lioutas, Adam Scibior, Matthew Niedoba, Berend Zwartsenberg, and Frank Wood. Control-itra: Controlling the behavior of a driving model.arXiv preprint arXiv:2501.12408, 2025

    Pith/arXiv arXiv 2025

  35. [35]

    Exocentric- to-egocentric video generation.Advances in Neural Information Processing Systems, 37: 136149–136172, 2024

    Jia-Wei Liu, Weijia Mao, Zhongcong Xu, Jussi Keppo, and Mike Zheng Shou. Exocentric- to-egocentric video generation.Advances in Neural Information Processing Systems, 37: 136149–136172, 2024

    2024

  36. [36]

    Imitation from observation: Learning to imitate behaviors from raw video via context translation

    YuXuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Imitation from observation: Learning to imitate behaviors from raw video via context translation. In2018 IEEE international conference on robotics and automation (ICRA), pages 1118–1125. IEEE, 2018

    2018

  37. [37]

    Repaint: Inpainting using denoising diffusion probabilistic models

    Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11461–11471, 2022

    2022

  38. [38]

    Exo2egosyn: Unlocking foundation video generation models for exocentric-to- egocentric video synthesis.arXiv preprint arXiv:2511.20186, 2025

    Mohammad Mahdi, Yuqian Fu, Nedko Savov, Jiancheng Pan, Danda Pani Paudel, and Luc Van Gool. Exo2egosyn: Unlocking foundation video generation models for exocentric-to- egocentric video synthesis.arXiv preprint arXiv:2511.20186, 2025

    arXiv 2025

  39. [39]

    Human-level control through deep reinforcement learning.nature, 518(7540):529–533, 2015

    V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning.nature, 518(7540):529–533, 2015

    2015

  40. [40]

    Policy invariance under reward transforma- tions: Theory and application to reward shaping

    Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transforma- tions: Theory and application to reward shaping. InIcml, volume 99, pages 278–287. Citeseer, 1999

    1999

  41. [41]

    Physicalai autonomous vehicles, 2025

    NVIDIA. Physicalai autonomous vehicles, 2025. URL https://huggingface.co/ datasets/nvidia/PhysicalAI-Autonomous-Vehicles . Dataset card, created October 28, 2025; accessed April 8, 2026

    2025

  42. [42]

    Curiosity-driven exploration by self-supervised prediction

    Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. InInternational conference on machine learning, pages 2778–

  43. [43]

    Alvinn: An autonomous land vehicle in a neural network.Advances in neural information processing systems, 1, 1988

    Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network.Advances in neural information processing systems, 1, 1988

    1988

  44. [44]

    Simlingo: Vision-only closed-loop autonomous driving with language-action alignment

    Katrin Renz, Long Chen, Elahe Arani, and Oleg Sinavski. Simlingo: Vision-only closed-loop autonomous driving with language-action alignment. InConference on Computer Vision and Pattern Recognition (CVPR), 2025

    2025

  45. [45]

    Premotor cortex and the recognition of motor actions.Cognitive brain research, 3(2):131–141, 1996

    Giacomo Rizzolatti, Luciano Fadiga, Vittorio Gallese, and Leonardo Fogassi. Premotor cortex and the recognition of motor actions.Cognitive brain research, 3(2):131–141, 1996

    1996

  46. [46]

    Time-contrastive networks: Self-supervised learning from video

    Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. In2018 IEEE international conference on robotics and automation (ICRA), pages 1134–1141. IEEE, 2018

    2018

  47. [47]

    Self-supervised disentangled representation learning for third-person imitation learning

    Jinghuan Shang and Michael S Ryoo. Self-supervised disentangled representation learning for third-person imitation learning. In2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 214–221. IEEE, 2021

    2021

  48. [48]

    Extranerf: Visibility-aware view extrapolation of neural radiance fields with diffusion models

    Meng-Li Shih, Wei-Chiu Ma, Lorenzo Boyice, Aleksander Holynski, Forrester Cole, Brian Curless, and Janne Kontkanen. Extranerf: Visibility-aware view extrapolation of neural radiance fields with diffusion models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20385–20395, 2024. 12

    2024

  49. [49]

    Worldwander: Bridg- ing egocentric and exocentric worlds in video generation.arXiv preprint arXiv:2511.22098, 2025

    Quanjian Song, Yiren Song, Kelly Peng, Yuan Gao, and Mike Zheng Shou. Worldwander: Bridg- ing egocentric and exocentric worlds in video generation.arXiv preprint arXiv:2511.22098, 2025

    Pith/arXiv arXiv 2025

  50. [50]

    Third person imitation learning

    Bradly C Stadie, Pieter Abbeel, and Ilya Sutskever. Third person imitation learning. In International Conference on Learning Representations, 2017

    2017

  51. [51]

    Behavioral cloning from observation.arXiv preprint arXiv:1805.01954, 2018

    Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation.arXiv preprint arXiv:1805.01954, 2018

    Pith/arXiv arXiv 2018

  52. [52]

    From image to video: An empirical study of diffusion representations

    Pedro Vélez, Luisa F Polanía, Yi Yang, Chuhan Zhang, Rishabh Kabra, Anurag Arnab, and Mehdi SM Sajjadi. From image to video: An empirical study of diffusion representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16948– 16958, 2025

    2025

  53. [53]

    Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    Pith/arXiv arXiv 2025

  54. [54]

    Alpamayo-r1: Bridging reasoning and action prediction for generalizable autonomous driving in the long tail.arXiv preprint arXiv:2511.00088, 2025

    Yan Wang, Wenjie Luo, Junjie Bai, Yulong Cao, Tong Che, Ke Chen, Yuxiao Chen, Jenna Dia- mond, Yifan Ding, Wenhao Ding, et al. Alpamayo-r1: Bridging reasoning and action prediction for generalizable autonomous driving in the long tail.arXiv preprint arXiv:2511.00088, 2025

    Pith/arXiv arXiv 2025

  55. [55]

    Knowledge of results and motor learning—implications for physical therapy

    Carolee J Winstein. Knowledge of results and motor learning—implications for physical therapy. Physical therapy, 71(2):140–149, 1991

    1991

  56. [56]

    Trajectory-guided control prediction for end-to-end autonomous driving: A simple yet strong baseline

    Penghao Wu, Xiaosong Jia, Li Chen, Junchi Yan, Hongyang Li, and Yu Qiao. Trajectory-guided control prediction for end-to-end autonomous driving: A simple yet strong baseline. InNeurIPS, 2022

    2022

  57. [57]

    Wod-e2e: Waymo open dataset for end-to-end driving in challenging long-tail scenarios.arXiv preprint arXiv:2510.26125, 2025

    Runsheng Xu, Hubert Lin, Wonseok Jeon, Hao Feng, Yuliang Zou, Liting Sun, John Gorman, Ekaterina Tolstaya, Sarah Tang, Brandyn White, et al. Wod-e2e: Waymo open dataset for end-to-end driving in challenging long-tail scenarios.arXiv preprint arXiv:2510.26125, 2025

    arXiv 2025

  58. [58]

    Learning by watching

    Jimuyang Zhang and Eshed Ohn-Bar. Learning by watching. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12711–12721, 2021

    2021

  59. [59]

    Unipc: A unified predictor- corrector framework for fast sampling of diffusion models.Advances in Neural Information Processing Systems, 36:49842–49869, 2023

    Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. Unipc: A unified predictor- corrector framework for fast sampling of diffusion models.Advances in Neural Information Processing Systems, 36:49842–49869, 2023

    2023

  60. [60]

    Split Rationale

    Berend Zwartsenberg, Adam ´Scibior, Matthew Niedoba, Vasileios Lioutas, Yunpeng Liu, Justice Sefas, Setareh Dabiri, Jonathan Wilder Lavington, Trevor Campbell, and Frank Wood. Conditional permutation invariant flows.arXiv preprint arXiv:2206.09021, 2022. 13 Multi-agent Behavior Model LearnerDemonstrator Replay with different weather and time of the day Mi...

    Pith/arXiv arXiv 2022