Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Video fine-tuning sharpens driving scenes but weakens object tracking.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Fine-tuning video generators on driving data can improve visual fidelity while degrading how accurately the model predicts the movement of cars and pedestrians.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Useful evaluation protocol, but the central claim about fine-tuning causing forgetting is confounded by using Vista as the only fine-tuned model. the 4 major comments →

arxiv 2508.16512 v1 pith:PW7LR5VH submitted 2025-08-22 cs.CV

Seeing Clearly, Forgetting Deeply: Revisiting Fine-Tuned Video Generators for Driving Simulation

Chun-Peng Chang , Chen-Yu Wang , Julian Schmidt , Holger Caesar , Alain Pagani This is my paper
classification cs.CV
keywords video predictionfine-tuningcatastrophic forgettingdriving world modelsdiffusion modelssemantic-critical actorscontinual learningobject tracking
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper investigates what happens when a general-purpose video generation model, Stable Video Diffusion, is fine-tuned on driving data. It claims that while this fine-tuning improves visual quality, it can degrade the model's spatial accuracy for dynamic, decision-critical objects such as pedestrians and vehicles. The explanation offered is that driving scenes are visually repetitive and structured, so the model can improve appearance by imitating dominant scene motion without tracking fine-grained actor behavior; the two goals, visual fidelity and dynamic understanding, become decoupled. The paper further shows that a simple continual-learning intervention, replaying diverse egocentric video during fine-tuning, preserves most of the visual improvement while restoring spatial accuracy.

Core claim

The paper's central claim is that fine-tuning a pretrained video diffusion model on driving data creates a trade-off: surface-level visual quality improves while the modeling of dynamic scene elements may stagnate or worsen. Comparing the pretrained SVD (baseline M0) with the driving-fine-tuned Vista (Mf.t), the paper reports that FID and FVD improve after fine-tuning, but object-level measures do not: SCA presence matching, recall, and centroid-distance distributions show no improvement and sometimes degrade, especially over longer horizons and for interaction reasoning. The paper attributes this to a shift in objective alignment: in general video, visual quality and dynamic accuracy reinfo

What carries the argument

The key machinery is a semantic decomposition of driving scenes into semantic-invariant background (SIB) and semantic-critical actors (SCA), where SCAs are compact, decision-relevant objects like pedestrians, vehicles, and animals. The paper measures their average displacement in nuScenes, finding that they move only about 10% of the image width over 2.5 seconds, which it uses to argue that driving data is temporally monotonous. The argument then relies on an objective-alignment shift: the optimal parameters for visual fidelity and dynamic understanding are approximately aligned for diverse egocentric video but diverge for structured driving data. This divergence is the mechanism said to all

Load-bearing premise

The paper's explanation assumes that Vista is a valid stand-in for 'SVD fine-tuned on driving data'; because no identical fine-tuning run without replay is compared, the observed dynamic degradation is attributed to fine-tuning rather than to differences in data, training setup, or conditioning.

What would settle it

A controlled experiment that fine-tunes the same SVD checkpoint on the same driving data with the same recipe, once with replay and once without, and then measures SCA positional accuracy; if the plain fine-tuned model matches or exceeds the pretrained baseline's spatial accuracy, the claimed forgetting trade-off would not be reproduced.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Visual-quality metrics such as FID and FVD are insufficient to establish that a driving world model is simulation-ready; object-level spatial and presence metrics are needed.
  • Fine-tuning a video generator for a constrained domain should be treated as a continual-learning problem, not simply as domain adaptation, because pretrained dynamic knowledge can be lost.
  • Replay from a diverse egocentric dataset is a practical intervention that keeps most visual quality gains while retaining better dynamic accuracy.
  • The finding that traffic-sign compliance is near chance for both pretrained and fine-tuned models indicates that rule understanding is a separate unsolved problem, independent of appearance quality.
  • Short-horizon evaluation windows of about 2.5 seconds are the appropriate setting for measuring SCA dynamics, since longer rollouts are dominated by unobservable future events and scene collapse.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same visual-quality/dynamic-accuracy decoupling could appear in any fine-tuning target with low temporal diversity, such as indoor robot navigation or industrial inspection; the paper's mechanism is testable outside driving.
  • If the mechanism is objective misalignment rather than data scarcity, dynamic-aware auxiliary losses or temporal-contrastive objectives might recover SCA accuracy without a replay buffer at all.
  • The fixed 1:1 offline replay mix is only one point in the design space; adaptive replay schedules or generative replay could preserve even more visual fidelity while avoiding the forgetting it aims to fix.
  • For safety-critical generation tasks, evaluation protocols should routinely pair global visual metrics with actor-level position and presence tracking, so appearance gains do not mask dynamic regressions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper studies what happens when a general-purpose video diffusion model (Stable Video Diffusion, M0) is fine-tuned for driving-scene prediction. Using Vista as the representative fine-tuned model (Mf.t) and a replay-augmented SVD fine-tune (Mr.p.f.t), it reports that fine-tuning improves visual quality (FID/FVD and GPT-4o judgments) but does not improve—and in some respects worsens—spatial/dynamic accuracy of semantic-critical actors, as measured by SAM2-based tracking, duration matching, centroid displacement, and a traffic-sign human evaluation. The authors attribute this to an objective-alignment shift and to catastrophic forgetting, and they propose offline replay from EgoExo4D as a mitigation strategy.

Significance. The question—whether fine-tuning for driving data trades appearance fidelity for dynamic correctness—is timely and important for learned world models. The paper contributes a useful SIB/SCA decomposition, a SAM2-based evaluation protocol, a traffic-sign compliance human evaluation, and a replay baseline, and it is unusually candid that the replay model is not a novel architecture. The visual-quality half of the story is supported by standard metrics and a judge evaluation, and the preprocessing for FID/FVD is described in enough detail to be reproducible. However, the dynamic-accuracy evidence is mixed and statistically thin, and the causal attribution to fine-tuning is confounded because Mf.t is a third-party model with a different recipe, data, and conditioning rather than a same-recipe fine-tune of M0.

major comments (4)
  1. [§6, Appendix A] The central causal claim—that fine-tuning induces catastrophic forgetting—is not identifiable from the reported comparisons. Appendix A defines Mf.t as Vista and states 'Since Vista is fine-tuned from Stable Video Diffusion (SVD), we denote the original SVD model as our baseline M0.' Vista differs from SVD in training data, conditioning, and recipe, and no same-recipe SVD fine-tune without replay is evaluated: Mr.p.f.t is trained from SVD with 1:1 EgoExo4D replay, but there is no M_f.t-no-replay. Consequently, Tables 3-4 and Figures 5-6 cannot separate fine-tuning effects from model/recipe differences. Please train a no-replay SVD fine-tune using exactly the nuScenes protocol used for Mr.p.f.t (minus replay) and report it. Without this, the catastrophic-forgetting interpretation in Section 5 is a hypothesis, not a supported conclusion.
  2. [Table 4, Figures 5-6] Quantitative support for 'dynamic degradation' is thin and mixed. On nuScenes validation, M0 vs Mf.t gives match rate 63.5 vs 62.8, FP 25.5 vs 22.9 (Mf.t better), FN 11.0 vs 14.3 (M0 better), precision 71.4 vs 73.3 (Mf.t better), recall 85.2 vs 81.5 (M0 better). No error bars, confidence intervals, or significance tests are reported. Figure 5 states 'higher standard deviation and longer tail' without numeric values, and Figure 6 has no error bars. Please report bootstrap CIs, paired tests (e.g., across clips or seeds), and quantify the standard deviation and tail mass for Fig. 5. If only recall/FN are significantly worse, the claim should be softened to 'mixed dynamic accuracy' rather than 'degradation.'
  3. [§5] The θ*_1 ≈ θ*_2 objective-alignment story is asserted post hoc. No operational definition of Lvis and Ldyn, no estimate of alignment, and no comparison between driving and diverse egocentric video is provided. As written, it is an untested explanation, not a result. Either derive a measurable proxy (e.g., gradient alignment between a perceptual loss and a tracking loss) or clearly label this as a hypothesis for future work and remove it from the abstract/contribution framing as an established mechanism.
  4. [§6.2, Appendix B.1] The traffic-sign human evaluation uses 200 clips and reports 5% vs 5% for left/right-turn-only signs and 12% vs 30% for red lights (Mf.t vs M0). No confidence intervals, inter-rater agreement, or statistical test is reported, and the explanatory claim that correct stops are due to preceding stopped cars or mid-turn motion is qualitative. With 200 clips, even the 30%-vs-12% difference may be within sampling error depending on clip selection. Report exact counts, a significance test, and clarify whether the same 200 clips are used for both models and how ties/ambiguous clips were handled.
minor comments (6)
  1. [Table 4] Column header 'nuscene Train' is a typo; use consistent notation for training/validation splits.
  2. [§6.2] Typo: 'Vistas' should be 'Vista'; there are also a few duplicated function words in the discussion of visual quality.
  3. [Table 5] The average pixel-wise differences are signed, and many early-frame entries are negative. The sign convention is unexplained; please state whether positive means predicted mask is larger than ground truth and why early frames are negative.
  4. [Appendix C.1] The base model name appears as '������ [3]' in the rendered text; it should be 'SVD_xt [3]' or similar.
  5. [Figure 6] The x-axis is 'Frame Number' with data at 2 FPS; please also indicate the corresponding time in seconds for readability.
  6. [Appendix C.2, Algorithm 1] Line 1 uses 'E-1_rotation' notation; define E and C as homogeneous transformation matrices and use explicit inverse notation to avoid ambiguity.

Circularity Check

0 steps flagged

No significant circularity: the trade-off is measured against external metrics and third-party models; the causal story is a hypothesis, not a derived result.

full rationale

The paper's empirical claims are operationalized with externally defined metrics (FID, FVD, SAM2-based tracking) and externally released models (SVD, Vista). No parameter is fitted to the target metrics in a way that would make the reported trade-off true by construction, and no equation equates a fitted input with a predicted output. The theta_1/theta_2 alignment discussion in Section 5 is a post hoc explanatory hypothesis rather than a result derived from the data, so it cannot be circular. The main validity concern is that Vista is used as a stand-in for 'SVD fine-tuned on driving data' without a same-recipe no-replay baseline, making the causal attribution to fine-tuning confounded; however, this is a correctness/confounding issue, not a circularity of the kind this review targets. Appendix A's statement is an externally checkable modeling assumption, not a self-referential reduction. Self-citations [7,8] appear only in support of LLM-judge agreement and are accompanied by external references [13,91], so they are not load-bearing. Overall, the paper is self-contained against external benchmarks and no derivation chain reduces to its own inputs.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

The list captures the measurement and attribution assumptions behind the central trade-off claim. The paper contains no parametric derivation, so free parameters are limited to a hand-chosen replay hyperparameter. SCA and SIB are analytic evaluation labels, not invented physical or model entities.

free parameters (1)
  • Offline replay sampling ratio (nuScenes:EgoExo4D = 1:1) = 1:1
    Hand-chosen training hyperparameter for the replay-augmented model M^{r.p.}_{f.t}; not fitted to data and does not enter the central SVD versus Vista comparison.
axioms (4)
  • domain assumption Vista is a representative and valid proxy for 'existing fine-tuning methods' applied to SVD.
    Invoked in Section 6 and Appendix A; the trade-off is inferred from a comparison of SVD and Vista despite unstated differences in training recipe, data, and conditioning.
  • domain assumption SAM2 with ground-truth bounding boxes as prompts yields faithful tracking across generated and real videos.
    Section 6.1 uses SAM2 masks to compute all SCA spatial metrics; a failure of tracking would directly affect the measured degradation.
  • domain assumption The 2.5 second, 25 frame horizon is sufficient to characterize dynamic understanding of SCAs.
    Section 6.2 and Table 4 use 25 frames at 10 FPS; longer horizons may change the trade-off, as the paper acknowledges.
  • ad hoc to paper Visual fidelity and dynamic accuracy objectives are approximately aligned for diverse egocentric video but misaligned for driving scenes.
    Section 5 treats this as an explanation of the observed trade-off, but theta_1/theta_2 are not measured directly; it is a post hoc mechanism, not a derived result.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Seeing Clearly, Forgetting Deeply: Revisiting Fine-Tuned Video Generators for Driving Simulation." pith.science (2026). https://pith.science/paper/PW7LR5VH

@misc{pith2026250816512,
  author       = {Pith},
  title        = {Pith review of: Seeing Clearly, Forgetting Deeply: Revisiting Fine-Tuned Video Generators for Driving Simulation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PW7LR5VH}},
  note         = {Machine review of arXiv:2508.16512}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Recent advancements in video generation have substantially improved visual quality and temporal coherence, making these models increasingly appealing for applications such as autonomous driving, particularly in the context of driving simulation and so-called "world models". In this work, we investigate the effects of existing fine-tuning video generation approaches on structured driving datasets and uncover a potential trade-off: although visual fidelity improves, spatial accuracy in modeling dynamic elements may degrade. We attribute this degradation to a shift in the alignment between visual quality and dynamic understanding objectives. In datasets with diverse scene structures within temporal space, where objects or perspective shift in varied ways, these objectives tend to highly correlated. However, the very regular and repetitive nature of driving scenes allows visual quality to improve by modeling dominant scene motion patterns, without necessarily preserving fine-grained dynamic behavior. As a result, fine-tuning encourages the model to prioritize surface-level realism over dynamic accuracy. To further examine this phenomenon, we show that simple continual learning strategies, such as replay from diverse domains, can offer a balanced alternative by preserving spatial accuracy while maintaining strong visual quality.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Invaria: Learning Scale and Density Invariance in Point Clouds via Next-Resolution Prediction

    cs.CV 2026-05 unverdicted novelty 6.0

    Invaria trains point cloud encoders with next-resolution prediction to learn scale and density invariant features, yielding higher mIoU on ScanNet under lower resolution and scaled objects while using a smaller model.

Reference graph

Works this paper leans on

93 extracted references · 37 canonical work pages · cited by 1 Pith paper · 1 internal anchor

  1. [1]

    Memory aware synapses: Learning what (not) to forget

    Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European conference on computer vision (ECCV), pages 139–154, 2018

    2018

  2. [2]

    Wasserstein gan, 2017

    Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan, 2017

    2017

  3. [3]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023

    Pith/arXiv arXiv 2023

  4. [4]

    Align your latents: High-resolution video synthesis with latent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22563–22575, 2023

    2023

  5. [5]

    Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom

    Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuScenes: A multimodal dataset for autonomous driving. In CVPR, 2020

    2020

  6. [6]

    Co2l: Contrastive continual learning

    Hyuntak Cha, Jaeho Lee, and Jinwoo Shin. Co2l: Contrastive continual learning. In Proceedings of the IEEE/CVF International conference on computer vision, pages 9516–9525, 2021

    2021

  7. [7]

    3D Spatial Understanding in MLLMs: Disambiguation and Evaluation

    Chun-Peng Chang, Alain Pagani, and Didier Stricker. 3d spatial understanding in mllms: Disambiguation and evaluation. arXiv preprint arXiv:2412.06613, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  8. [8]

    Mikasa: Multi-key-anchor & scene-aware transformer for 3d visual grounding

    Chun-Peng Chang, Shaoxiang Wang, Alain Pagani, and Didier Stricker. Mikasa: Multi-key-anchor & scene-aware transformer for 3d visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14131–14140, 2024

    2024

  9. [9]

    Riemannian walk for incremental learning: Understanding forgetting and intransigence

    Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European conference on computer vision (ECCV), pages 532–547, 2018

    2018

  10. [10]

    On tiny episodic memories in continual learning

    Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc’Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv:1902.10486, 2019

    Pith/arXiv arXiv 1902

  11. [11]

    Videocrafter2: Overcoming data limitations for high-quality video diffusion models

    Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7310–7320, 2024

    2024

  12. [12]

    Gentron: Diffusion transformers for image and video generation

    Shoufa Chen, Mengmeng Xu, Jiawei Ren, Yuren Cong, Sen He, Yanping Xie, Animesh Sinha, Ping Luo, Tao Xiang, and Juan-Manuel Perez-Rua. Gentron: Diffusion transformers for image and video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6441–6451, 2024

    2024

  13. [13]

    Can large language models be an alternative to human evaluations? arXiv preprint arXiv:2305.01937, 2023

    Cheng-Han Chiang and Hung-yi Lee. Can large language models be an alternative to human evaluations? arXiv preprint arXiv:2305.01937, 2023

    Pith/arXiv arXiv 2023

  14. [14]

    Autoregressive video generation without vector quantization

    Haoge Deng, Ting Pan, Haiwen Diao, Zhengxiong Luo, Yufeng Cui, Huchuan Lu, Shiguang Shan, Yonggang Qi, and Xinlong Wang. Autoregressive video generation without vector quantization. arXiv preprint arXiv:2412.14169, 2024

    Pith/arXiv arXiv 2024

  15. [15]

    Parameter-efficient fine-tuning of large-scale pre-trained language models

    Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220–235, 2023

    2023

  16. [16]

    Carla: An open urban driving simulator

    Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference on robot learning, pages 1–16. PMLR, 2017

    2017

  17. [17]

    Self-supervised models are continual learners

    Enrico Fini, Victor G Turrisi Da Costa, Xavier Alameda-Pineda, Elisa Ricci, Karteek Alahari, and Julien Mairal. Self-supervised models are continual learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9621–9630, 2022

    2022

  18. [18]

    Vista: A generalizable driving world model with high fidelity and versatile controllability

    Shenyuan Gao, Jiazhi Yang, Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li. Vista: A generalizable driving world model with high fidelity and versatile controllability. In Advances in Neural Information Processing Systems (NeurIPS), 2024. 10

    2024

  19. [19]

    Factorizing text-to-video generation by explicit image conditioning

    Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Factorizing text-to-video generation by explicit image conditioning. In European Conference on Computer Vision, pages 205–224. Springer, 2024

    2024

  20. [20]

    Generative adversarial networks.Communications of the ACM, 63(11):139– 144, 2020

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks.Communications of the ACM, 63(11):139– 144, 2020

    2020

  21. [21]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18995–19012, 2022

    2022

  22. [22]

    Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives

    Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19...

    work page 2024

  23. [23]

    Gem: A generalizable ego-vision multimodal world model for fine-grained ego-motion, object dynamics, and scene composition control

    Mariam Hassan, Sebastian Stapf, Ahmad Rahimi, Pedro Rezende, Yasaman Haghighi, David Brüggemann, Isinsu Katircioglu, Lin Zhang, Xiaoran Chen, Suman Saha, et al. Gem: A generalizable ego-vision multimodal world model for fine-grained ego-motion, object dynamics, and scene composition control. arXiv preprint arXiv:2412.11198, 2024

    Pith/arXiv arXiv 2024

  24. [24]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium.Advances in neural information processing systems, 30, 2017

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium.Advances in neural information processing systems, 30, 2017

    2017

  25. [25]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020

    2020

  26. [26]

    Video diffusion models

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633–8646, 2022

    2022

  27. [27]

    Cogvideo: Large-scale pretraining for text-to-video generation via transformers

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.15868, 2022

    Pith/arXiv arXiv 2022

  28. [28]

    Learning a unified classifier incrementally via rebalancing

    Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 831–839, 2019

    work page 2019

  29. [29]

    Re-evaluating continual learning scenarios: A categorization and case for strong baselines

    Yen-Chang Hsu, Yen-Cheng Liu, Anita Ramasamy, and Zsolt Kira. Re-evaluating continual learning scenarios: A categorization and case for strong baselines. arXiv preprint arXiv:1810.12488, 2018

    Pith/arXiv arXiv 2018

  30. [30]

    Gaia-1: A generative world model for autonomous driving

    Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. Gaia-1: A generative world model for autonomous driving. arXiv preprint arXiv:2309.17080, 2023

    Pith/arXiv arXiv 2023

  31. [31]

    St-p3: End-to- end vision-based autonomous driving via spatial-temporal feature learning

    Shengchao Hu, Li Chen, Penghao Wu, Hongyang Li, Junchi Yan, and Dacheng Tao. St-p3: End-to- end vision-based autonomous driving via spatial-temporal feature learning. In European Conference on Computer Vision, pages 533–549. Springer, 2022

    work page 2022

  32. [32]

    Make it move: controllable image-to-video generation with text descriptions

    Yaosi Hu, Chong Luo, and Zhenzhong Chen. Make it move: controllable image-to-video generation with text descriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18219–18228, 2022

    work page 2022

  33. [33]

    Planning-oriented autonomous driving

    Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. Planning-oriented autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17853–17862, 2023

    work page 2023

  34. [34]

    Self-supervised street gaussians for autonomous driving

    Nan Huang, Xiaobao Wei, Wenzhao Zheng, Pengju An, Ming Lu, Wei Zhan, Masayoshi Tomizuka, Kurt Keutzer, and Shanghang Zhang. Self-supervised street gaussians for autonomous driving. arXiv preprint arXiv:2405.20323, 2024

    Pith/arXiv arXiv 2024

  35. [35]

    Meta-learning representations for continual learning

    Khurram Javed and Martha White. Meta-learning representations for continual learning. Advances in neural information processing systems, 32, 2019

    work page 2019

  36. [36]

    Adriver-i: A general world model for autonomous driving

    Fan Jia, Weixin Mao, Yingfei Liu, Yucheng Zhao, Yuqing Wen, Chi Zhang, Xiangyu Zhang, and Tiancai Wang. Adriver-i: A general world model for autonomous driving. arXiv preprint arXiv:2311.13549, 2023. 11

    Pith/arXiv arXiv 2023

  37. [37]

    Yolov11: An overview of the key architectural enhancements

    Rahima Khanam and Muhammad Hussain. Yolov11: An overview of the key architectural enhancements. arXiv preprint arXiv:2410.17725, 2024

    Pith/arXiv arXiv 2024

  38. [38]

    Drivegan: Towards a controllable high-quality neural simulation

    Seung Wook Kim, Jonah Philion, Antonio Torralba, and Sanja Fidler. Drivegan: Towards a controllable high-quality neural simulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5820–5829, 2021

    work page 2021

  39. [39]

    Overcoming catastrophic forgetting in neural networks

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521–3526, 2017

    2017

  40. [40]

    Mixture of experts meets prompt-based continual learning

    Minh Le, Huy Nguyen, Trang Nguyen, Trang Pham, Linh Ngo, Nhat Ho, et al. Mixture of experts meets prompt-based continual learning. Advances in Neural Information Processing Systems, 37:119025–119062, 2024

    work page 2024

  41. [41]

    Theory on mixture-of-experts in continual learning

    Hongbo Li, Sen Lin, Lingjie Duan, Yingbin Liang, and Ness B Shroff. Theory on mixture-of-experts in continual learning. arXiv preprint arXiv:2406.16437, 2024

    Pith/arXiv arXiv 2024

  42. [42]

    Zhiqi Li, Zhiding Yu, Shiyi Lan, Jiahan Li, Jan Kautz, Tong Lu, and Jose M Alvarez. Is ego status all you need for open-loop end-to-end autonomous driving? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14864–14873, 2024

    work page 2024

  43. [43]

    Flowvid: Taming imperfect optical flows for consistent video-to-video synthesis

    Feng Liang, Bichen Wu, Jialiang Wang, Licheng Yu, Kunpeng Li, Yinan Zhao, Ishan Misra, Jia-Bin Huang, Peizhao Zhang, Peter Vajda, et al. Flowvid: Taming imperfect optical flows for consistent video-to-video synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8207–8216, 2024

    work page 2024

  44. [44]

    Are nerfs ready for autonomous driving? towards closing the real-to-simulation gap

    Carl Lindström, Georg Hess, Adam Lilja, Maryam Fatemi, Lars Hammarstrand, Christoffer Petersson, and Lennart Svensson. Are nerfs ready for autonomous driving? towards closing the real-to-simulation gap. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4461–4471, 2024

    work page 2024

  45. [45]

    Neuroncap: Photorealistic closed-loop safety testing for autonomous driving

    William Ljungbergh, Adam Tonderski, Joakim Johnander, Holger Caesar, Kalle Åström, Michael Felsberg, and Christoffer Petersson. Neuroncap: Photorealistic closed-loop safety testing for autonomous driving. In European Conference on Computer Vision, pages 161–177. Springer, 2024

    work page 2024

  46. [46]

    Wovogen: World volume-aware diffusion for controllable multi-camera driving scene generation

    Jiachen Lu, Ze Huang, Zeyu Yang, Jiahui Zhang, and Li Zhang. Wovogen: World volume-aware diffusion for controllable multi-camera driving scene generation. In European Conference on Computer Vision, pages 329–345. Springer, 2024

    work page 2024

  47. [47]

    Representational continuity for unsupervised continual learning

    Divyam Madaan, Jaehong Yoon, Yuanchun Li, Yunxin Liu, and Sung Ju Hwang. Representational continuity for unsupervised continual learning. arXiv preprint arXiv:2110.06976, 2021

    Pith/arXiv arXiv 2021

  48. [48]

    Fine-tuning can cripple your foundation model; preserving features may be the solution

    Jishnu Mukhoti, Yarin Gal, Philip HS Torr, and Puneet K Dokania. Fine-tuning can cripple your foundation model; preserving features may be the solution. arXiv preprint arXiv:2308.13320, 2023

    Pith/arXiv arXiv 2023

  49. [49]

    Conditional image- to-video generation with latent flow diffusion models

    Haomiao Ni, Changhao Shi, Kai Li, Sharon X Huang, and Martin Renqiang Min. Conditional image- to-video generation with latent flow diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18444–18455, 2023

    work page 2023

  50. [50]

    A review on deep learning techniques for video prediction

    Sergiu Oprea, Pablo Martinez-Gonzalez, Alberto Garcia-Garcia, John Alejandro Castro-Vargas, Sergio Orts-Escolano, Jose Garcia-Rodriguez, and Antonis Argyros. A review on deep learning techniques for video prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(6):2806–2826, 2020

    work page 2020

  51. [51]

    Continual lifelong learning with neural networks: A review

    German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural networks, 113:54–71, 2019

    work page 2019

  52. [52]

    Summarize the past to predict the future: Natural language descriptions of context boost multimodal object interaction anticipation

    Razvan Pasca, Alexey Gavryushin, Muhammad Hamza, Yen-Ling Kuo, Kaichun Mo, Luc Van Gool, Otmar Hilliges, and Xi Wang. Summarize the past to predict the future: Natural language descriptions of context boost multimodal object interaction anticipation. In Conference on Computer Vision and Pattern Recognition 2024, 2024

    work page 2024

  53. [53]

    Egovideo: Exploring egocentric foundation model and downstream adaptation

    Baoqi Pei, Guo Chen, Jilan Xu, Yuping He, Yicheng Liu, Kanghua Pan, Yifei Huang, Yali Wang, Tong Lu, Limin Wang, et al. Egovideo: Exploring egocentric foundation model and downstream adaptation. arXiv preprint arXiv:2406.18070, 2024. 12

    Pith/arXiv arXiv 2024

  54. [54]

    An outlook into the future of egocentric vision.International Journal of Computer Vision, 132(11):4880–4936, 2024

    Chiara Plizzari, Gabriele Goletto, Antonino Furnari, Siddhant Bansal, Francesco Ragusa, Giovanni Maria Farinella, Dima Damen, and Tatiana Tommasi. An outlook into the future of egocentric vision.International Journal of Computer Vision, 132(11):4880–4936, 2024

    work page 2024

  55. [55]

    Egovlpv2: Egocentric video-language pre-training with fusion in the backbone

    Shraman Pramanick, Yale Song, Sayan Nag, Kevin Qinghong Lin, Hardik Shah, Mike Zheng Shou, Rama Chellappa, and Pengchuan Zhang. Egovlpv2: Egocentric video-language pre-training with fusion in the backbone. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5285–5297, 2023

    work page 2023

  56. [56]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024

    Pith/arXiv arXiv 2024

  57. [57]

    icarl: Incremental classifier and representation learning

    Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001–2010, 2017

    2001

  58. [58]

    Consisti2v: Enhancing visual consistency for image-to-video generation

    Weiming Ren, Huan Yang, Ge Zhang, Cong Wei, Xinrun Du, Wenhao Huang, and Wenhu Chen. Consisti2v: Enhancing visual consistency for image-to-video generation. arXiv preprint arXiv:2402.04324, 2024

    Pith/arXiv arXiv 2024

  59. [59]

    Learning to learn without forgetting by maximizing transfer and minimizing interference

    Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint arXiv:1810.11910, 2018

    Pith/arXiv arXiv 2018

  60. [60]

    Gaia-2: A controllable multi-view generative world model for autonomous driving

    Lloyd Russell, Anthony Hu, Lorenzo Bertoni, George Fedoseev, Jamie Shotton, Elahe Arani, and Gianluca Corrado. Gaia-2: A controllable multi-view generative world model for autonomous driving. arXiv preprint arXiv:2503.20523, 2025

    Pith/arXiv arXiv 2025

  61. [61]

    Gradient projection memory for continual learning

    Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. arXiv preprint arXiv:2103.09762, 2021

    Pith/arXiv arXiv 2021

  62. [62]

    Temporal generative adversarial nets with singular value clipping

    Masaki Saito, Eiichi Matsumoto, and Shunta Saito. Temporal generative adversarial nets with singular value clipping. In Proceedings of the IEEE international conference on computer vision, pages 2830–2839, 2017

    2017

  63. [63]

    Motion-i2v: Consistent and controllable image-to-video generation with explicit motion modeling

    Xiaoyu Shi, Zhaoyang Huang, Fu-Yun Wang, Weikang Bian, Dasong Li, Yi Zhang, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, et al. Motion-i2v: Consistent and controllable image-to-video generation with explicit motion modeling. In ACM SIGGRAPH 2024 Conference Papers, pages 1–11, 2024

    work page 2024

  64. [64]

    Continual learning with deep generative replay

    Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. Advances in neural information processing systems, 30, 2017

    2017

  65. [65]

    Ernie 2.0: A continual pre-training framework for language understanding

    Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. Ernie 2.0: A continual pre-training framework for language understanding. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 8968–8975, 2020

    work page 2020

  66. [66]

    Drivingforward: Feed-forward 3d gaussian splatting for driving scene reconstruction from flexible surround-view input

    Qijian Tian, Xin Tan, Yuan Xie, and Lizhuang Ma. Drivingforward: Feed-forward 3d gaussian splatting for driving scene reconstruction from flexible surround-view input. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 7374–7382, 2025

    work page 2025

  67. [67]

    Videotetris: Towards compositional text-to-video generation

    Ye Tian, Ling Yang, Haotian Yang, Yuan Gao, Yufan Deng, Xintao Wang, Zhaochen Yu, Xin Tao, Pengfei Wan, Di ZHANG, et al. Videotetris: Towards compositional text-to-video generation. Advances in Neural Information Processing Systems, 37:29489–29513, 2024

    work page 2024

  68. [68]

    Mocogan: Decomposing motion and content for video generation

    Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1526–1535, 2018

    2018

  69. [69]

    Towards accurate generative models of video: A new metric & challenges

    Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717, 2018

    Pith/arXiv arXiv 2018

  70. [70]

    Fvd: A new metric for video generation

    Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. Fvd: A new metric for video generation. 2019

    2019

  71. [71]

    Three scenarios for continual learning

    Gido M Van de Ven and Andreas S Tolias. Three scenarios for continual learning. arXiv preprint arXiv:1904.07734, 2019. 13

    Pith/arXiv arXiv 1904

  72. [72]

    A comprehensive survey of continual learning: Theory, method and application

    Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  73. [73]

    Training networks in null space of feature covariance for continual learning

    Shipeng Wang, Xiaorong Li, Jian Sun, and Zongben Xu. Training networks in null space of feature covariance for continual learning. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pages 184–193, 2021

    work page 2021

  74. [74]

    Swap attention in spatiotemporal diffusions for text-to-video generation

    Wenjing Wang, Huan Yang, Zixi Tuo, Huiguo He, Junchen Zhu, Jianlong Fu, and Jiaying Liu. Swap attention in spatiotemporal diffusions for text-to-video generation. International Journal of Computer Vision, pages 1–19, 2025

    work page 2025

  75. [75]

    Drivedreamer: Towards real-world-drive world models for autonomous driving

    Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Jiagang Zhu, and Jiwen Lu. Drivedreamer: Towards real-world-drive world models for autonomous driving. In European Conference on Computer Vision, pages 55–72. Springer, 2024

    work page 2024

  76. [76]

    Emu3: Next-token prediction is all you need

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869, 2024

    Pith/arXiv arXiv 2024

  77. [77]

    Lavie: High-quality video generation with cascaded latent diffusion models

    Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, et al. Lavie: High-quality video generation with cascaded latent diffusion models. International Journal of Computer Vision, 133(5):3059–3078, 2025

    work page 2025

  78. [78]

    Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving

    Yuqi Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14749–14759, 2024

    work page 2024

  79. [79]

    Fairy: Fast parallelized instruction-guided video-to-video synthesis

    Bichen Wu, Ching-Yao Chuang, Xiaoyan Wang, Yichen Jia, Kapil Krishnakumar, Tong Xiao, Feng Liang, Licheng Yu, and Peter Vajda. Fairy: Fast parallelized instruction-guided video-to-video synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8261–8270, 2024

    work page 2024

  80. [80]

    Towards a better metric for text-to-video generation

    Jay Zhangjie Wu, Guian Fang, Haoning Wu, Xintao Wang, Yixiao Ge, Xiaodong Cun, David Junhao Zhang, Jia-Wei Liu, Yuchao Gu, Rui Zhao, et al. Towards a better metric for text-to-video generation. arXiv preprint arXiv:2401.07781, 2024

    Pith/arXiv arXiv 2024

Showing first 80 references.

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.