Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

PosePilot: Steering Camera Pose for Generative World Models with Self-supervised Depth

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read PosePilot claims that attaching self-supervised depth and ego-motion readouts with photometric warping losses to generative world models cuts camera pose error on driving and real-estate benchmarks.

desk verdict The pose-control idea is a genuine new combination, but the missing evaluation protocol leaves the headline gains potentially measuring the pose readout rather than the world model. read the letter →

arxiv 2505.01729 v2 pith:GKGL4RHM submitted 2025-05-03 cs.CV

Bu Jin , Weize Li , Baihan Yang , Zhenxin Zhu , Junpeng Jiang , Huan-ang Gao , Haiyang Sun , Kun Zhan
show 4 more authors
Hengtong Hu Xueyang Zhang Peng Jia Hao Zhao
This is my paper · ORCID
classification cs.CV
keywords cameraposecontrolgenerativeworldmodelsself-superviseddepthestimationego-motionphotometricwarpinglosscontrollablevideogenerationautonomousdrivingstructure-from-motion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PosePilot is a lightweight, plug-and-play addition to generative world models that aims to make camera pose controllable without retraining the base generator. It adds two small readout heads, one for depth and one for relative ego-motion, and uses them to warp frames according to a specified pose, then penalizes the mismatch between warped and generated frames with a photometric loss. The paper reports that attaching PosePilot to DiVE, Vista, and DrivingWorld lowers translation and rotation pose errors on nuScenes, and that adding it to CameraCtrl improves pose control on RealEstate10K. The point of the work is that geometric self-supervision from structure-from-motion can inject pose controllability into models that were not designed for it.

What carries the argument

The central mechanism is the differentiable projection $x_{i\to j}=K T_{i\to j} D_i K^{-1} x_i$, which maps a pixel from frame $i$ to frame $j$ using estimated depth $D_i$, relative pose $T_{i\to j}$, and camera intrinsics $K$. It appears in both directions: forward warping aligns a generated frame with the pose-conditioned warp of an input frame, and inverse warping maps the generated frame back to the original viewpoint. The depth and ego-motion readouts are small learnable heads attached to the generator, and the photometric control loss (L1 plus SSIM) together with the pose MSE loss pushes the generator to produce frames consistent with the specified camera trajectory. The load-bearing property is that the warping is fully differentiable, so gradients from the photometric losses flow into the generator without additional pose annotation; the separate pose regression term can use the reference pose when available.

What would settle it

An experiment that replaces the learned readouts with ground-truth depth and pose, or masks out moving objects when computing the photometric loss, would settle the mechanism: if pose-error gains on nuScenes disappear under either change, the improvement is not caused by the proposed self-supervised warping supervision itself.

Watch

Extended reading notes

Core claim

The paper's central claim is that camera pose controllability in generative world models can be established by a self-supervised geometric loop rather than by learning pose-conditioned attention. Using depth and ego-motion readouts, PosePilot builds the projection $x_{i\to j}=K T_{i\to j} D_i K^{-1} x_i$ to warp a source frame onto a target viewpoint, then applies forward and inverse photometric control losses (L1 plus SSIM) between the warped image and the generated frame, plus a pose regression loss against the reference pose. With these terms added to the generator's own loss, the reported TransErr and RotErr drop for diffusion-based (DiVE, Vista) and autoregressive (DrivingWorld) world models on nuScenes, and CameraCtrl's pose control improves on RealEstate10K. If correct, the implication is that explicit geometry readouts are enough to steer an existing world model's camera trajectory, with no architectural change to the generator.

Load-bearing premise

The reported gains rest on the assumption that the L1/SSIM photometric warping losses, computed from learned depth and pose, remain a trustworthy training signal when the generated frames contain imperfect, novel, or moving content; the paper does not describe a mechanism to mask dynamic objects or to prevent degenerate flat-depth and identity-pose solutions.

Editorial extensions

If this is right

  • On nuScenes, PosePilot reduces translation error from 13.07 to 6.37 and rotation error from 4.52 to 1.40 for DiVE, and from 6.83 to 6.52 and 1.74 to 1.53 for Vista.
  • For the autoregressive model DrivingWorld, PosePilot lowers translation error from 3.17 to 2.95 and rotation error from 1.64 to 1.48.
  • On RealEstate10K, CameraCtrl with PosePilot reaches 6.52 translation error and 0.70 rotation error, below CameraCtrl alone and the other camera-control baselines.
  • Ablation results show the inverse photometric loss improves temporal consistency and FID, so geometric supervision contributes to visual quality, not just pose metrics.
  • The module adds about 181 million parameters to base models exceeding one billion parameters, while leaving iteration and inference time roughly unchanged, supporting the plug-and-play claim.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper leaves implicit is using the same geometric loop to steer surround-view multi-camera generation, since the relative-pose readouts can apply between any camera pair rather than only consecutive frames.
  • An ablation that removes only the pose regression term while keeping both warping losses would reveal whether the photometric terms or the explicit regression term carry the pose-control gain.
  • A testable extension is to mask dynamic objects in the photometric loss; if the gains persist under such masking, the mechanism is robust to moving content, whereas if they vanish, the reported improvement mostly comes from static background geometry.
  • The same readout-and-warp loop could be applied at inference time to refine latents toward a user-specified trajectory without training, turning the method into a training-free controller that the paper does not explore.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. PosePilot attaches learnable depth and ego-motion readout heads to pretrained generative world models and trains them with a forward photometric warping loss, an inverse photometric warping loss, and an MSE pose-regression loss. The claimed contribution is that this plug-and-play module improves camera-pose controllability of diffusion-based and autoregressive world models without changing their architectures. The paper reports lower TransErr/RotErr for DiVE, Vista, and DrivingWorld on nuScenes, and for CameraCtrl on RealEstate10K, plus ablations of the three loss terms in terms of pose error, FID, parameter count, and runtime.

Significance. If the empirical case were complete, PosePilot would be a genuinely useful contribution: it reuses well-understood self-supervised depth/ego-motion machinery as a geometric prior for video generation, and the reported improvements are directionally consistent across several base models and two domains. The plug-and-play nature and the small parameter overhead are also attractive. However, the evaluation protocol is not specified, and the one load-bearing detail that is missing directly interacts with the design: Eq. (5) trains a pose readout against reference poses, while the paper's headline metrics are camera-trajectory errors. Unless the paper states exactly which pose estimator extracts trajectories from generated videos and shows that it is not the trained readout, the central claim that PosePilot improves pose controllability of the generator cannot be verified. A second concern is statistical: all tables are point estimates with no variance or seed information, and some reported gains are small (e.g., DrivingWorld 3.17 to 2.95). The central mechanism is plausible, but the evidence as written is incomplete.

major comments (4)
  1. [Section IV-A, Eq. (5)] The paper never states which pose estimator is used to extract camera trajectories from generated videos for TransErr/RotErr. Eq. (5) directly supervises the pose readout's output \hat T against the reference pose T_ref, and the gradient of L_MSE ends at the readout rather than at the generator. If the same readout (or a fine-tuned variant of it) is used as the evaluation-time pose estimator, the TransErr/RotErr gains could reflect the readout learning to emit poses near T_ref even if the generated frames are geometrically unchanged. The authors must name the external pose estimator used in evaluation, confirm that the identical protocol is applied to all rows of Tables I-III, and preferably add a control experiment that compares a pose readout trained with L_MSE against a same-capacity readout trained without it while keeping the photometric losses fixed.
  2. [Eq. (6), Tables I-III] The loss weights \alpha_p, \alpha_pinv, and \alpha_mse in Eq. (6) are never reported, and no experimental configuration (batch size, training iterations, learning rate, which world-model weights are frozen versus fine-tuned) is given. In addition, every number in Tables I-III appears to be a single run with no variance, and several improvements are small in absolute terms (Vista 6.83 to 6.52; DrivingWorld 3.17 to 2.95). As written, the improvements cannot be distinguished from run-to-run noise, and the ablation cannot be reproduced without the weight values.
  3. [Section III-A, Eq. (3)] The photometric control loss uses L1 plus SSIM over all valid pixels but provides no automasking for dynamic objects, no explainability mask, and no depth smoothness regularization. In standard self-supervised depth estimation these components are important because photometric losses can be trivially minimized by degenerate depth/pose predictions and are corrupted by moving objects and occlusions. Since the paper's reported gains are modest, the absence of these safeguards makes it unclear whether the observed TransErr/RotErr improvements are caused by genuine geometric control or by the readouts exploiting artifacts of the loss; the criterion defining the 'valid' pixel set N in Eq. (2) is also never specified.
  4. [Abstract, Section IV-A, Table II] The abstract claims that PosePilot 'significantly enhances structural understanding and motion reasoning,' but no experiment directly measures structural understanding or motion reasoning. TransErr/RotErr measure only the estimated camera trajectory, and FID is a distribution-level appearance metric; neither measures depth consistency, semantic structure, or object-motion reasoning. If these claims are kept, the paper should add a direct structural or motion metric, or the claims should be narrowed to pose alignment and image fidelity.
minor comments (6)
  1. [Section III-A] The sentence 'Once the extrinsic matrices (or pose) of each camera are avaiGiven two framesle' is corrupted and should be rewritten.
  2. [Section III-B, Eq. (4)] Eq. (4) uses p_j for a pixel in the generated frame while Eq. (1) uses x_i and x_{i->j}; the notation should be made consistent, and p_j should be defined.
  3. [Figure 2] The figure labels 'Wrapped Input' and 'Wrapped Prediction' should read 'Warped Input' and 'Warped Prediction'; the same typo appears near the end of Section III-B as 'warp usxing'.
  4. [Section IV-A, Table I] The statement that the nuScenes validation set is divided into four subsets is not followed by any description of how the subsets are constructed or how the averaged numbers are computed; this prevents reproduction of the reported scores.
  5. [Table II] The 'Parameters (M) ↓' column header is confusing because the reported entries are increases over the baseline model (+181.1 and +90.5); the table should state the baseline parameter count explicitly and clarify that lower is preferred but the values are deltas.
  6. [Section IV-D, Table III] For the RealEstate10K experiment it is not stated whether CameraCtrl is fine-tuned jointly with PosePilot or kept frozen, nor which camera-trajectory conditioning format is used; this matters because CameraCtrl's original trajectory representation already contains the pose information that PosePilot's readouts are trained to predict.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation found; the method is a set of geometric training objectives anchored to external baselines, with one evaluation-reporting gap that is not demonstrated circularity.

full rationale

PosePilot's claimed derivation chain is a set of differentiable geometric losses, not a fitted prediction. The forward photometric loss (Eqs. 2-3) warps an input frame using depth and pose readouts and compares the warped image against the world model's generated frame, providing a standard self-supervised warping supervisory signal; the inverse loss (Eq. 4) does the same in the reverse direction. The pose regression loss (Eq. 5) directly anchors the pose readout to a reference pose, but the readout is not the quantity the paper claims to improve; the headline claim is about camera pose controllability of the world model generator. The main results are evaluated against external baselines (DiVE, Vista, DrivingWorld, CameraCtrl) using TransErr and RotErr relative to specified camera poses, and no load-bearing premise is justified solely by a self-citation; self-cited works appear only as base models or related work. One reporting gap is worth flagging but does not amount to demonstrated circularity: Section IV-A does not name the pose estimator used to extract camera trajectories from generated videos for TransErr/RotErr, so a scenario in which the trained pose readout also serves as the evaluation estimator is not explicitly ruled out. However, the paper does not exhibit that reduction, and the photometric losses still provide independent gradient paths to the generator, so this is a reproducibility or correctness concern rather than a circular step.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The method reuses standard structure-from-motion assumptions. The main unstated choices are the loss weights, the readout network details, and the pose measurement protocol; no new physical entities are introduced.

free parameters (3)
  • alpha_p (photometric loss weight) = not reported
    Eq (6) combines L_p with L_inv and L_mse; alpha_p is not specified in the paper.
  • alpha_pinv (inverse photometric loss weight) = not reported
    Eq (6); value not given, likely tuned by hand.
  • alpha_mse (pose regression loss weight) = not reported
    Eq (6); value not given, affects the balance between geometric consistency and generation objective.
assumptions (5)
  • domain assumption Pinhole camera model with known intrinsics K is valid; Eq (1) correctly projects pixels between frames.
    Section III-A uses Eq (1) without discussing calibration error or distortion.
  • domain assumption Brightness constancy and Lambertian surfaces hold, so the photometric loss in Eq (3) is a valid alignment signal.
    Standard in self-supervised depth; the paper adds SSIM but reports no masking of dynamic objects or non-Lambertian regions.
  • domain assumption The world model's generated frames are in the same image space as inputs, so photometric comparison with warped inputs is meaningful.
    Section III-A uses predicted frames as reference frames; if generated frames have different statistics, the loss may be noisy.
  • standard math The 6D pose readout can be converted into the 4x4 transformation T used in Eq (1) and Eq (4).
    The paper writes T as a matrix but says the readout outputs a 6D pose; the conversion is not specified.
  • domain assumption The external pose metrology (TransErr/RotErr) reliably estimates the camera trajectory from generated videos.
    Section IV-A defines the metrics by reference to CameraCtrl but does not state which pose estimator is used.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PosePilot: Steering Camera Pose for Generative World Models with Self-supervised Depth." pith.science (2026). https://pith.science/paper/GKGL4RHM

@misc{pith2026250501729,
  author       = {Pith},
  title        = {Pith review of: PosePilot: Steering Camera Pose for Generative World Models with Self-supervised Depth},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GKGL4RHM}},
  note         = {Machine review of arXiv:2505.01729}
}
read the original abstract

Recent advancements in autonomous driving (AD) systems have highlighted the potential of world models in achieving robust and generalizable performance across both ordinary and challenging driving conditions. However, a key challenge remains: precise and flexible camera pose control, which is crucial for accurate viewpoint transformation and realistic simulation of scene dynamics. In this paper, we introduce PosePilot, a lightweight yet powerful framework that significantly enhances camera pose controllability in generative world models. Drawing inspiration from self-supervised depth estimation, PosePilot leverages structure-from-motion principles to establish a tight coupling between camera pose and video generation. Specifically, we incorporate self-supervised depth and pose readouts, allowing the model to infer depth and relative camera motion directly from video sequences. These outputs drive pose-aware frame warping, guided by a photometric warping loss that enforces geometric consistency across synthesized frames. To further refine camera pose estimation, we introduce a reverse warping step and a pose regression loss, improving viewpoint precision and adaptability. Extensive experiments on autonomous driving and general-domain video datasets demonstrate that PosePilot significantly enhances structural understanding and motion reasoning in both diffusion-based and auto-regressive world models. By steering camera pose with self-supervised depth, PosePilot sets a new benchmark for pose controllability, enabling physically consistent, reliable viewpoint synthesis in generative world models.

Figures

Figures reproduced from arXiv: 2505.01729 by the authors.

Figure 1
Figure 1. Illustration of PosePilot. We propose PosePilot, a lightweight pose controllability enhancer for world models in autonomous driving. PosePilotcan generate videos collaboratively aligned with the input camera pose. The red box indicates the reference objects for easy identification. hances the world model’s ability to reason about its location, orientation, and viewpoint, thereby improving pose control￾lability. Furt… view at source ↗
Figure 2
Figure 2. Overview of PosePilot. We utilize off-the-shelf generative world model as our base model and build our PosePilot on top of them. We introduce two auxiliary tasks: depth estimation and ego-motion estimation. The depth estimation predicts per-pixel depth maps, while ego-motion prediction estimates the relative camera pose between consecutive frames. We introduce a photometric control loss and an inverse photometric co… view at source ↗
Figure 3
Figure 3. Controllability of PosePilot. Our model can enhance the model’s pose controllability, which generates videos that collaboratively align with the conditional inputs. More examples can be found in the supplementary video. TABLE II ELEMENT-WISE ABLATION STUDY PosePilot TransErr↓ RotErr↓ FID↓ Parameters (M)↓ Iter Time (s)↓ Inference Time (s)↓ Lmse Lp Linv p ✔ ✔ ✔ 6.37 1.40 13.7 +181.1 8.83 47.8 ✔ ✔ 7.56 1.67 14.5 +90.5 … view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. 3D and 4D World Modeling: A Survey

    cs.CV 2025-09 conditional novelty 5.0 of 10

    A survey that defines 3D/4D world modeling, organizes methods into VideoGen, OccGen, and LiDARGen categories, and compiles datasets, metrics, and benchmark numbers.

Reference graph

Works this paper leans on

49 extracted references · 23 canonical work pages · cited by 1 Pith paper

  1. [1]

    Planning-oriented autonomous driving,

    Y . Hu, J. Yang, L. Chen, K. Li, C. Sima, X. Zhu, S. Chai, S. Du, T. Lin, W. Wanget al., “Planning-oriented autonomous driving,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 17 853–17 862

    2023

  2. [2]

    Unipad: A uni- versal pre-training paradigm for autonomous driving,

    H. Yang, S. Zhang, D. Huang, X. Wu, H. Zhu, T. He, S. Tang, H. Zhao, Q. Qiu, B. Lin et al., “Unipad: A uni- versal pre-training paradigm for autonomous driving,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 15 238–15 250

    work page 2024

  3. [3]

    P-mapnet: Far-seeing map gen- erator enhanced by both sdmap and hdmap priors,

    Z. Jiang, Z. Zhu, P. Li, H.-a. Gao, T. Yuan, Y . Shi, H. Zhao, and H. Zhao, “P-mapnet: Far-seeing map gen- erator enhanced by both sdmap and hdmap priors,” IEEE Robotics and Automation Letters , 2024

    work page 2024

  4. [4]

    Lode: Locally conditioned eikonal implicit scene completion from sparse lidar,

    P. Li, R. Zhao, Y . Shi, H. Zhao, J. Yuan, G. Zhou, and Y .-Q. Zhang, “Lode: Locally conditioned eikonal implicit scene completion from sparse lidar,” in 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023, pp. 8269–8276

    work page 2023

  5. [5]

    Monoocc: Digging into monocular semantic occupancy prediction,

    Y . Zheng, X. Li, P. Li, Y . Zheng, B. Jin, C. Zhong, X. Long, H. Zhao, and Q. Zhang, “Monoocc: Digging into monocular semantic occupancy prediction,” in 2024 IEEE International Conference on Robotics and Automa- tion (ICRA). IEEE, 2024, pp. 18 398–18 405

    work page 2024

  6. [6]

    Unsupervised road anomaly detection with language anchors,

    B. Tian, M. Liu, H.-a. Gao, P. Li, H. Zhao, and G. Zhou, “Unsupervised road anomaly detection with language anchors,” in 2023 IEEE international conference on robotics and automation (ICRA). IEEE, 2023, pp. 7778– 7785

    work page 2023

  7. [7]

    Tod3cap: Towards 3d dense captioning in outdoor scenes,

    B. Jin, Y . Zheng, P. Li, W. Li, Y . Zheng, S. Hu, X. Liu, J. Zhu, Z. Yan, H. Sun et al. , “Tod3cap: Towards 3d dense captioning in outdoor scenes,” in European Con- ference on Computer Vision . Springer, 2024, pp. 367– 384

    work page 2024

  8. [8]

    Uniscene: Unified occupancy-centric driving scene generation,

    B. Li, J. Guo, H. Liu, Y . Zou, Y . Ding, X. Chen, H. Zhu, F. Tan, C. Zhang, T. Wang et al. , “Uniscene: Unified occupancy-centric driving scene generation,” in Proceed- ings of the Computer Vision and Pattern Recognition Conference, 2025, pp. 11 971–11 981

    work page 2025

Show all 49 references
  1. [9]

    Hint-ad: Holis- tically aligned interpretability in end-to-end autonomous driving,

    K. Ding, B. Chen, Y . Su, H.-a. Gao, B. Jin, C. Sima, W. Zhang, X. Li, P. Barsch, H. Li et al., “Hint-ad: Holis- tically aligned interpretability in end-to-end autonomous driving,” arXiv preprint arXiv:2409.06702 , 2024

    2024 arXiv

  2. [10]

    Model- based imitation learning for urban driving,

    A. Hu, G. Corrado, N. Griffiths, Z. Murez, C. Gurau, H. Yeo, A. Kendall, R. Cipolla, and J. Shotton, “Model- based imitation learning for urban driving,” Advances in Neural Information Processing Systems , vol. 35, pp. 20 703–20 716, 2022

    2022

  3. [11]

    Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving,

    Y . Wang, J. He, L. Fan, H. Li, Y . Chen, and Z. Zhang, “Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 14 749–14 759

    2024

  4. [12]

    Drivedreamer: Towards real-world-driven world models for autonomous driving,

    X. Wang, Z. Zhu, G. Huang, X. Chen, J. Zhu, and J. Lu, “Drivedreamer: Towards real-world-driven world models for autonomous driving,” arXiv preprint arXiv:2309.09777, 2023

    2023 arXiv

  5. [13]

    Bevcontrol: Accurately controlling street-view elements with multi-perspective consistency via bev sketch lay- out,

    K. Yang, E. Ma, J. Peng, Q. Guo, D. Lin, and K. Yu, “Bevcontrol: Accurately controlling street-view elements with multi-perspective consistency via bev sketch lay- out,” arXiv preprint arXiv:2308.01661 , 2023

    2023 arXiv

  6. [14]

    Magicdrive: Street view genera- tion with diverse 3d geometry control,

    R. Gao, K. Chen, E. Xie, L. Hong, Z. Li, D.-Y . Yeung, and Q. Xu, “Magicdrive: Street view genera- tion with diverse 3d geometry control,” arXiv preprint arXiv:2310.02601, 2023

    2023 arXiv

  7. [15]

    Panacea: Panoramic and controllable video generation for au- tonomous driving,

    Y . Wen, Y . Zhao, Y . Liu, F. Jia, Y . Wang, C. Luo, C. Zhang, T. Wang, X. Sun, and X. Zhang, “Panacea: Panoramic and controllable video generation for au- tonomous driving,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 6902–6912

    2024

  8. [16]

    Subjectdrive: Scaling generative data in autonomous driving via subject control,

    B. Huang, Y . Wen, Y . Zhao, Y . Hu, Y . Liu, F. Jia, W. Mao, T. Wang, C. Zhang, C. W. Chen et al. , “Subjectdrive: Scaling generative data in autonomous driving via subject control,” arXiv preprint arXiv:2403.19438 , 2024

    2024 arXiv

  9. [17]

    Vista: A generalizable driving world model with high fidelity and versatile controlla- bility,

    S. Gao, J. Yang, L. Chen, K. Chitta, Y . Qiu, A. Geiger, J. Zhang, and H. Li, “Vista: A generalizable driving world model with high fidelity and versatile controlla- bility,” arXiv preprint arXiv:2405.17398 , 2024

    2024 arXiv

  10. [18]

    Motionctrl: A unified and flexible motion controller for video generation,

    Z. Wang, Z. Yuan, X. Wang, Y . Li, T. Chen, M. Xia, P. Luo, and Y . Shan, “Motionctrl: A unified and flexible motion controller for video generation,” in ACM SIG- GRAPH 2024 Conference Papers , 2024, pp. 1–11

    2024

  11. [19]

    Direct-a-video: Customized video generation with user-directed camera movement and object motion,

    S. Yang, L. Hou, H. Huang, C. Ma, P. Wan, D. Zhang, X. Chen, and J. Liao, “Direct-a-video: Customized video generation with user-directed camera movement and object motion,” in ACM SIGGRAPH 2024 Conference Papers, 2024, pp. 1–12

    2024

  12. [20]

    Cameractrl: Enabling camera control for text- to-video generation,

    H. He, Y . Xu, Y . Guo, G. Wetzstein, B. Dai, H. Li, and C. Yang, “Cameractrl: Enabling camera control for text- to-video generation,” arXiv preprint arXiv:2404.02101 , 2024

    2024 arXiv

  13. [21]

    Animatediff: Animate your personalized text-to-image diffusion models without specific tuning,

    Y . Guo, C. Yang, A. Rao, Z. Liang, Y . Wang, Y . Qiao, M. Agrawala, D. Lin, and B. Dai, “Animatediff: Animate your personalized text-to-image diffusion models without specific tuning,” arXiv preprint arXiv:2307.04725, 2023

    2023 arXiv

  14. [22]

    Unsu- pervised learning of depth and ego-motion from video,

    T. Zhou, M. Brown, N. Snavely, and D. G. Lowe, “Unsu- pervised learning of depth and ego-motion from video,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 1851–1858

    2017

  15. [23]

    Unsuper- vised learning of depth and ego-motion from monocular video using 3d geometric constraints,

    R. Mahjourian, M. Wicke, and A. Angelova, “Unsuper- vised learning of depth and ego-motion from monocular video using 3d geometric constraints,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 5667–5675

    2018

  16. [24]

    Unsupervised scale-consistent depth and ego-motion learning from monocular video,

    J. Bian, Z. Li, N. Wang, H. Zhan, C. Shen, M.-M. Cheng, and I. Reid, “Unsupervised scale-consistent depth and ego-motion learning from monocular video,” Advances in neural information processing systems , vol. 32, 2019

    2019

  17. [25]

    Unsuper- vised cnn for single view depth estimation: Geometry to the rescue,

    R. Garg, V . K. Bg, G. Carneiro, and I. Reid, “Unsuper- vised cnn for single view depth estimation: Geometry to the rescue,” in Computer Vision–ECCV 2016: 14th Euro- pean Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14. Springer, 2016, pp...

    2016

  18. [26]

    Unsupervised scale-consistent depth learning from video,

    J.-W. Bian, H. Zhan, N. Wang, Z. Li, L. Zhang, C. Shen, M.-M. Cheng, and I. Reid, “Unsupervised scale-consistent depth learning from video,” Interna- tional Journal of Computer Vision , vol. 129, no. 9, pp. 2548–2564, 2021

    2021

  19. [27]

    A survey of world models for autonomous driving,

    T. Feng, W. Wang, and Y . Yang, “A survey of world models for autonomous driving,” arXiv preprint arXiv:2501.11260, 2025

    2025 arXiv

  20. [28]

    End-to-end autonomous driving: Challenges and frontiers,

    L. Chen, P. Wu, K. Chitta, B. Jaeger, A. Geiger, and H. Li, “End-to-end autonomous driving: Challenges and frontiers,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  21. [29]

    Dome: Taming diffusion model into high-fidelity controllable occupancy world model,

    S. Gu, W. Yin, B. Jin, X. Guo, J. Wang, H. Li, Q. Zhang, and X. Long, “Dome: Taming diffusion model into high-fidelity controllable occupancy world model,” arXiv preprint arXiv:2410.10429, 2024

    2024 arXiv

  22. [30]

    World4drive: End- to-end autonomous driving via intention-aware physical latent world model,

    Y . Zheng, P. Yang, Z. Xing, Q. Zhang, Y . Zheng, Y . Gao, P. Li, T. Zhang, Z. Xia, P. Jia et al., “World4drive: End- to-end autonomous driving via intention-aware physical latent world model,” arXiv preprint arXiv:2507.00603 , 2025

    2025 arXiv

  23. [31]

    Int2: Interactive trajectory pre- diction at intersections,

    Z. Yan, P. Li, Z. Fu, S. Xu, Y . Shi, X. Chen, Y . Zheng, Y . Li, T. Liu, C. Liet al., “Int2: Interactive trajectory pre- diction at intersections,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 8536–8547

    2023

  24. [32]

    Transfusion: Robust lidar-camera fusion for 3d object detection with transformers,

    X. Bai, Z. Hu, X. Zhu, Q. Huang, Y . Chen, H. Fu, and C.-L. Tai, “Transfusion: Robust lidar-camera fusion for 3d object detection with transformers,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 1090–1099

    2022

  25. [33]

    Pivotnet: Vectorized pivot learning for end-to-end hd map con- struction,

    W. Ding, L. Qiao, X. Qiu, and C. Zhang, “Pivotnet: Vectorized pivot learning for end-to-end hd map con- struction,” 2023

    2023

  26. [34]

    Gaia-1: A generative world model for autonomous driving,

    A. Hu, L. Russell, H. Yeo, Z. Murez, G. Fedoseev, A. Kendall, J. Shotton, and G. Corrado, “Gaia-1: A generative world model for autonomous driving,” arXiv preprint arXiv:2309.17080, 2023

    2023 arXiv

  27. [35]

    Drivedreamer: Towards real-world-drive world models for autonomous driving,

    X. Wang, Z. Zhu, G. Huang, X. Chen, J. Zhu, and J. Lu, “Drivedreamer: Towards real-world-drive world models for autonomous driving,” in European Conference on Computer Vision. Springer, 2024, pp. 55–72

    2024

  28. [36]

    Worlddreamer: Towards general world models for video generation via predicting masked tokens,

    X. Wang, Z. Zhu, G. Huang, B. Wang, X. Chen, and J. Lu, “Worlddreamer: Towards general world models for video generation via predicting masked tokens,” arXiv preprint arXiv:2401.09985, 2024

    2024 arXiv

  29. [37]

    Muvo: A multimodal world model with spatial rep- resentations for autonomous driving,

    D. Bogdoll, Y . Yang, T. Joseph, and J. M. Z ¨ollner, “Muvo: A multimodal world model with spatial rep- resentations for autonomous driving,” arXiv preprint arXiv:2311.11762, 2023

    2023 arXiv

  30. [38]

    Occworld: Learning a 3d occupancy world model for autonomous driving,

    W. Zheng, W. Chen, Y . Huang, B. Zhang, Y . Duan, and J. Lu, “Occworld: Learning a 3d occupancy world model for autonomous driving,” in European conference on computer vision . Springer, 2024, pp. 55–72

    2024

  31. [39]

    Motionbooth: Motion-aware customized text-to-video generation,

    J. Wu, X. Li, Y . Zeng, J. Zhang, Q. Zhou, Y . Li, Y . Tong, and K. Chen, “Motionbooth: Motion-aware customized text-to-video generation,” Advances in Neural Informa- tion Processing Systems , vol. 37, pp. 34 322–34 348, 2025

    2025

  32. [40]

    Training- free camera control for video generation,

    C. Hou, G. Wei, Y . Zeng, and Z. Chen, “Training- free camera control for video generation,” arXiv preprint arXiv:2406.10126, 2024

    2024 arXiv

  33. [41]

    Camctrl3d: Single-image scene explo- ration with precise 3d camera control,

    S. Popov, A. Raj, M. Krainin, Y . Li, W. T. Freeman, and M. Rubinstein, “Camctrl3d: Single-image scene explo- ration with precise 3d camera control,” arXiv preprint arXiv:2501.06006, 2025

    2025 arXiv

  34. [42]

    Cami2v: Camera-controlled image-to-video diffusion model,

    G. Zheng, T. Li, R. Jiang, Y . Lu, T. Wu, and X. Li, “Cami2v: Camera-controlled image-to-video diffusion model,” arXiv preprint arXiv:2410.15957 , 2024

    2024 arXiv

  35. [43]

    I2vcontrol-camera: Precise video camera control with adjustable motion strength,

    W. Feng, J. Liu, P. Tu, T. Qi, M. Sun, T. Ma, S. Zhao, S. Zhou, and Q. He, “I2vcontrol-camera: Precise video camera control with adjustable motion strength,” arXiv preprint arXiv:2411.06525, 2024

    2024 arXiv

  36. [44]

    Dive: Dit-based video generation with enhanced control,

    J. Jiang, G. Hong, L. Zhou, E. Ma, H. Hu, X. Zhou, J. Xiang, F. Liu, K. Yu, H. Sun et al., “Dive: Dit-based video generation with enhanced control,” arXiv preprint arXiv:2409.01595, 2024

    2024 arXiv

  37. [45]

    Drivingworld: Constructingworld model for autonomous driving via video gpt,

    X. Hu, W. Yin, M. Jia, J. Deng, X. Guo, Q. Zhang, X. Long, and P. Tan, “Drivingworld: Constructingworld model for autonomous driving via video gpt,” arXiv preprint arXiv:2412.19505, 2024

    2024 arXiv

  38. [46]

    Image quality assessment: from error visibility to struc- tural similarity,

    Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to struc- tural similarity,” IEEE transactions on image processing, vol. 13, no. 4, pp. 600–612, 2004

    2004

  39. [47]

    nuscenes: A multimodal dataset for autonomous driving,

    H. Caesar, V . Bankiti, A. H. Lang, S. V ora, V . E. Liong, Q. Xu, A. Krishnan, Y . Pan, G. Baldan, and O. Bei- jbom, “nuscenes: A multimodal dataset for autonomous driving,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2020, pp. 11 621–11 631

    2020

  40. [48]

    Stereo magnification: Learning view synthesis using multiplane images,

    T. Zhou, R. Tucker, J. Flynn, G. Fyffe, and N. Snavely, “Stereo magnification: Learning view synthesis using multiplane images,” in SIGGRAPH, 2018

    2018

  41. [49]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium,

    M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,” Advances in neural information processing systems , vol. 30, 2017

    2017

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.