{"id":"6571a01c-21b1-4b19-a70b-205f275d6e7f","arxiv_id":"2411.19324","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"An auxiliary trajectory attention branch, added to temporal attention in video diffusion models, improves camera motion control precision while preserving generation quality.","lead":"This paper introduces a trajectory attention module that makes video diffusion models follow camera motion more precisely by attending along pixel paths. It reports large reductions in trajectory error for camera control on images and videos with about one day of fine-tuning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Tables 1–2 never state how camera trajectories are estimated from generated videos; the reported ATE/RPE gains may reflect estimator behavior rather than the auxiliary-branch design, so the central precision claim is not yet verifiable.","rationale":"I looked for a flaw in the architecture itself. The auxiliary-branch idea is coherent: temporal attention and trajectory attention are decoupled, QKV weights are inherited, the output projector is zero-initialized, and back-projection with normalization is a reasonable way to return trajectory-sampled features to the full grid. The ablation in Table 3 supports the branch design over simple retuning. So the most vulnerable point is not the mechanism but the empirical support for \"precise motion control.\" The paper's headline evidence is ATE/RPE, yet the measurement procedure is absent. ATE and RPE are defined for trajectories estimated from sensor data, and applying them to generated video requires choosing a pose estimator; this choice can dominate the numbers. Without it, an independent reader cannot tell whether the reported gains reflect true camera control or an artifact of the estimator. This concern is more fundamental than the reader's stated weakest assumption (accuracy of depth/point tracking/intrinsics): that assumption affects the conditioning signal but is acknowledged in Appendix A.1 and does not undermine the architectural claim if trajectories are assumed correct. The missing evaluation protocol blocks verification of the central claim as stated. I therefore keep the verdict conditional: the paper is promising, but the quantitative case needs a specified, reproducible evaluation before the claim of significant precision improvement is accepted.","tokens_in":15144,"tokens_out":6865,"duration_ms":64369,"concrete_test":"Ask the authors to release the exact ATE/RPE evaluation pipeline for Tables 1–2: the camera-pose estimator (e.g., COLMAP with specific matcher/SfM options), how generated frames are aligned to the ground-truth trajectory, and the intrinsics used. Then rerun the 230 image-case evaluations and the video evaluations with three random seeds per method, reporting mean and standard deviation for every metric. If the ATE/RPE advantage over CameraCtrl and NVS Solver collapses or falls within seed noise under a fixed, pre-registered estimator, the central precision claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.1 lists ATE and RPE as the headline metrics for camera-control precision but never states how the camera trajectory is recovered from the generated video. Appendix A.9 shows \"estimated trajectories from our generation\" aligned with ground truth, yet no estimator (COLMAP, monocular SLAM, or otherwise) is named. This is load-bearing because the central claim—that the auxiliary trajectory-attention branch yields precise camera control—is quantified entirely by these ATE/RPE values in Tables 1 and 2. If the estimator is sensitive to video smoothness or temporal consistency, a method that produces more static or less dynamic videos can appear to have lower pose error even when it follows the commanded camera motion less accurately; conversely, if the trajectories are recovered using the same depth/projection pipeline that generated the conditioning signal, the metric is partly circular. The paper also reports no error bars or significance tests, and some comparisons mix base models and frame lengths, so the 10–100x improvements in Table 1 cannot be attributed to the proposed design with confidence. This is an evaluation-protocol gap rather than a flaw in the mechanism itself.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces trajectory attention, an auxiliary attention branch that operates on features sampled along pixel trajectories, in parallel with the standard temporal attention of a video diffusion model. The branch is initialized from the temporal attention weights, its output is added as residuals, and only this branch is fine-tuned. The method is applied to camera motion control from a single image, camera motion control on videos, and first-frame-guided video editing. The experiments report ATE/RPE/FID improvements over several baselines, and the paper demonstrates the extension to full 3D attention models. The authors also provide ablations on the branch design, training data, and trajectory sparsity.","tokens_in":15374,"tokens_out":6746,"duration_ms":54660,"significance":"The core idea is elegant and practically valuable: rather than replacing or modifying temporal attention, trajectory attention is added as a lightweight, plug-and-play branch with weight inheritance and zero-initialized output, enabling efficient fine-tuning (about 24 GPU hours) without changing the base model parameters. The attention-map analysis (Fig. 2) provides a sensible motivation for the auxiliary-branch design. If the reported precision gains are reproducible, the method would be a significant step for controllable video synthesis. The paper is also transparent about its limitations (Appendix A.1) regarding dependence on external trajectory extractors and failure modes on fast or complex motions. However, the evaluation protocol has gaps that currently prevent full verification of the central precision claim.","major_comments":[{"comment":"The paper reports ATE and RPE as the headline metrics for camera-control precision, but it never states how camera trajectories are recovered from the generated videos. Appendix A.9 shows 'estimated trajectories from our generation' without naming the estimator (COLMAP, monocular SLAM, or a learned method) or the alignment procedure. This is load-bearing: if the estimator is sensitive to video smoothness or temporal consistency, a method that produces more static videos could appear to have lower pose error even when it follows the commanded camera motion less accurately. Please specify the full evaluation protocol, including the estimator, alignment (Sim(3)/SE(3)), and any outlier handling, and provide a sensitivity analysis.","section":"Sec. 5.1, Tables 1-2"},{"comment":"The ablation states that for the 'Vanilla' condition, 'some outputs exhibiting complete noise (we omit such invalid results during evaluation).' This selection biases the reported ATE/RPE/FID and makes the comparison with the proposed design difficult to interpret. Please report the fraction of invalid outputs for each ablation condition, and either include a failure-rate metric (e.g., percentage of runs that produce non-finite or degenerate outputs) or use a metric that handles failures. Without this, the large gap between Vanilla and the proposed design is not fully supported.","section":"Sec. 5.5, Table 3"},{"comment":"The quantitative comparison mixes base models and frame lengths: MotionI2V uses AnimateDiff while the other baselines and the proposed method use SVD, and the comparisons are at 14, 16, and 25 frames. The table footnote acknowledges this, but the text claims 'consistently achieve higher or comparable control precision' across settings. To attribute the improvements to the trajectory-attention design, please provide a same-base-model, same-frame-length comparison at least for the strongest baseline, or clearly state the confound. The 10-100x ATE improvements over MotionCtrl may partly reflect the different base models or frame lengths.","section":"Sec. 5.1 and Table 1"},{"comment":"Single-image camera control uses predefined camera intrinsics (fx=fy=260, principal point at image center) and a depth estimate from DepthAnythingV2 to compute the conditioning trajectories. The paper does not quantify how the reported ATE/RPE depend on these choices, and it is unclear whether the same depth/projection pipeline is used to extract the 'estimated trajectories' from the generated videos that enter the ATE/RPE computation. If the same pipeline is used, the evaluation is partially circular. Please clarify the evaluation trajectory extraction and provide a quantitative sensitivity analysis over focal lengths (or other intrinsic parameters), beyond the qualitative illustration in Fig. 21.","section":"Appendix A.3 and Sec. 4.1"}],"minor_comments":[{"comment":"Algorithm 1 does not specify how fractional 2D coordinates are sampled from the hidden states; please state whether nearest-neighbor or bilinear (or other) interpolation is used.","section":"Algorithm 1"},{"comment":"Equation (3) does not define the conditioning signal c or the noise schedule σ; for reproducibility, please specify what c is for each task (e.g., trajectories and first frame) and cite the schedule used by SVD.","section":"Sec. 3.4"},{"comment":"The citation 'Yang et al. (2023a)' for optical flow extraction points to a paper on semi-supervised semantic segmentation; the intended optical-flow reference appears to be missing or mis-cited. Please correct.","section":"Sec. 5.1 and A.10"},{"comment":"Tables 1 and 2 are titled 'Qualitative comparison' but contain quantitative metrics; please rename them 'Quantitative comparison' or similar.","section":"Tables 1 and 2"},{"comment":"In Algorithm 4, the combination Tr = Pt + P assumes that Pt and P are in the same pixel-coordinate frame; please state how the sampled camera translation Pt and the point trajectories P are aligned, and what happens when a point becomes occluded.","section":"Algorithm 4"},{"comment":"The paper claims 'long-range consistency' but no temporal consistency metric (e.g., per-video LPIPS or feature stability) is reported; the FID is a distribution-level metric and does not directly measure temporal consistency within a generated clip.","section":"Sec. 5 and throughout"},{"comment":"Figure 16 shows estimated trajectories from 'our generation' and from CameraCtrl, but the estimation method is not described; please name the estimator and state whether it is the same for all compared methods.","section":"Appendix A.9"},{"comment":"Minor typos: 'straighwarpward' (Sec. 3.2), 'lenght' (Algorithm 4), 'trajecories' (Algorithms 3 and 4), and 'wrapped frames' (Fig. 13) should be corrected.","section":"Various"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of a computer vision journal and the core idea has potential. The main risk is the missing trajectory-estimation protocol for the ATE/RPE metrics; this is fixable but essential. The authors should also consider releasing code or at least detailed implementation details for the sampling and back-projection steps. The paper would benefit from a careful revision addressing the evaluation concerns raised in the major comments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper has a genuinely plausible new mechanism for camera control in video diffusion—an auxiliary attention branch that operates along pixel trajectories—and the reported gains would matter if the evaluation held up. Right now the evaluation protocol has a big unstated gap: the authors never say how they recover camera trajectories from generated videos to compute ATE/RPE. Until that is pinned down, the headline numbers are hard to trust.\n\nWhat is actually new: the trajectory sampling and back-projection (Algorithms 1 and 2) with a separate attention branch initialized from temporal attention weights and a zero-initialized output projector. That design choice is well motivated: temporal attention focuses on short-range dynamics, trajectory attention enforces long-range consistency. The ablation in Table 3 supports the two-branch design with weight inheriting. Training is cheap (24 GPU-hours on 10k clips), and the extension to Open-Sora-Plan's 3D attention and to first-frame-guided editing is suggestive, if only qualitative.\n\nWhere it gets soft. Most importantly, the ATE/RPE numbers in Tables 1 and 2 require estimating a camera trajectory from generated frames. The paper never names the estimator. If it is a monocular SLAM or COLMAP pipeline, results depend on video smoothness; if it reuses the same depth/projection assumptions that generated the conditioning, the metric is partly circular. This is not a flaw in the mechanism, but it is a flaw in the evidence. On top of that: no error bars or significance tests; baseline comparisons mix base models (SVD vs AnimateDiff) and frame lengths (14/16/25); video editing has no quantitative evaluation; and the image-control results depend on hand-set intrinsics (fx=fy=260) that are acknowledged but not swept. The ablation also omits invalid 'Vanilla' outputs from the table because they were complete noise—omitting failure cases from a quantitative table is a dangerous practice even when the failure is catastrophic.\n\nThe limitations appendix is honest about fast-motion and complex-trajectory failures, which counts in the authors' favor. The mechanism itself is clear and reproducible enough to re-implement; no code or checkpoints are released, which makes the missing estimator detail worse.\n\nBottom line: this deserves a serious referee. The idea is worth engaging, but the paper should not be accepted until the evaluation protocol is specified and corrected—name the trajectory estimator, report variance, and align baselines. I would bring it to a reading group as a discussion piece on how to evaluate camera-control metrics.","headline":"Genuinely plausible new attention mechanism for camera control, but the reported precision gains rest on an unstated trajectory-estimation protocol that needs to be pinned down before the numbers can be believed.","tokens_in":15893,"tokens_out":3406,"would_cite":true,"duration_ms":26520,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that adding an auxiliary attention branch that matches features along camera-defined pixel trajectories gives pretrained video diffusion models precise, long-range motion control without retraining the base model.","keywords":["trajectory attention","camera motion control","video diffusion models","temporal attention","video generation","first-frame-guided video editing","motion control","attention mechanism"],"falsifier":"On a benchmark with known camera poses and ground-truth depth, generate the same scenes twice, once with trajectories computed from ground-truth depth and once from the paper's estimated depth with its default camera assumptions, then compare ATE and RPE. If pose error does not increase when the depth map is corrupted or replaced, trajectory accuracy is not what drives the control; if error rises sharply, the dependence on trajectory extraction is confirmed. A complementary test is to ablate the auxiliary branch itself by feeding the sampled trajectory features directly into the temporal attention's value stream; if control precision stays equally high, the auxiliary-branch design is not the source of the reported gains.","tokens_in":14956,"feed_emoji":"🎥","tokens_out":8261,"duration_ms":66756,"temperature":0.7,"pith_summary":"This paper is trying to establish that camera motion in video generation can be controlled precisely by attending along the pixel trajectories the camera would trace, rather than by feeding pose embeddings or warped frames into the model. The key idea is to keep the ordinary temporal attention untouched and add a second, auxiliary attention branch that samples hidden features at trajectory coordinates, runs attention with its own parameters, and adds the result back as a residual. The authors argue that temporal attention mostly locks onto nearby frames, so it cannot enforce long-range consistency, while trajectory attention is free to align features over the whole path and still lets the model generate new content where no trajectory is given. If the claim holds, the same training-light recipe should transfer across base models and tasks, which the paper supports with image camera control, video camera control, and first-frame-guided video editing experiments.","feed_headline":"Trajectory attention makes video camera motion precise","feed_subtitle":"Auxiliary attention branch follows pixel paths, cutting camera-pose error while keeping generation quality.","key_machinery":"Trajectory attention is an auxiliary multi-head attention module that operates on features sampled along user-specified pixel trajectories instead of on the full spatial grid. For each trajectory, Algorithm 1 reads the hidden states at the trajectory's 2D coordinates in every frame, masks invalid frames, and stacks them into a sequence; after attention, Algorithm 2 scatters the updated features back to their original locations and normalizes by how many trajectories covered each pixel. The QKV projectors are copied from the temporal-attention module, while the output projection is zero-initialized, so the branch starts as a no-op and grows gradually during fine-tuning. It is this sampled-along-trajectory, read-and-scatter attention that injects the camera motion as an explicit correspondence constraint while leaving the base model's parameters untouched.","core_discovery":"The central claim is that a dedicated trajectory-attention branch, initialized from the temporal-attention weights and added to it as a zero-initialized residual, is sufficient to make a pretrained video diffusion model follow a specified camera path with much lower error than pose-embedding or warping-based conditioning. Concretely, hidden states are sampled at each frame along the supplied 2D trajectories, attended to frame-wise, and written back and averaged at the sampled locations, so the attention mechanism itself carries the motion condition. The paper reports that this design reduces absolute trajectory error and relative pose error on single-image camera control (for example, an ATE of 0.0396 on the 25-frame setting versus 0.0411 for the best pose-conditioning baseline) and on video camera control, while keeping FID comparable. It also shows the branch extends from 12-frame training to 25-frame generation and to full 3D-attention models, and that it can drive first-frame-guided video editing. The role of the auxiliary branch is to enforce long-range consistency along the trajectory without forcing the original temporal attention to abandon its short-range motion synthesis.","pith_inferences":["The authors leave implicit that the same auxiliary-branch recipe could apply to other correspondence-based controls, such as object dragging, depth-guided motion, or user-sketched flow, since the branch never needs to know where the trajectories came from.","Because the fixed camera intrinsics are acknowledged as a source of error, a natural next step is to jointly predict or calibrate intrinsics from the image, which would likely extend the method to 360-degree and wide-angle camera paths.","The attention-window visualization suggests a quantitative diagnostic: tracking the effective window width of the trajectory branch during generation should correlate with long-range consistency, giving a cheap way to predict when control will fail on fast motion.","Since the branch is trained on only 10k clips, the method appears data-efficient, but the paper does not show how performance scales with much larger trajectory-conditioned training sets, so the ceiling of the approach remains unknown."],"forward_implications":["Single-image camera control becomes a fine-grained operation: reported absolute trajectory error drops to roughly 0.02-0.04 meters on the tested 14-, 16-, and 25-frame settings, close to the ground-truth camera path.","Because trajectory attention is a residual add-on, it combines with orthogonal techniques such as warping-based test-time optimization, improving both precision and fidelity over either approach alone.","The same branch transfers from 12-frame training to 25-frame generation, suggesting the learned trajectory prior is not tied to a specific sequence length.","Trajectory attention extends beyond camera motion to first-frame-guided video editing, preserving edited content across large spatial and temporal ranges where frame-warping methods drift.","The design adapts to full 3D-attention video architectures, so it is not limited to decomposed spatial-temporal attention models."],"supporting_citations":[{"why":"The pretrained video diffusion model whose temporal-attention weights initialize the trajectory branch and whose architecture the method fine-tunes.","marker":"(Blattmann et al., 2023)"},{"why":"Supplies the point trajectories that Algorithm 4 combines with camera motion for video control.","marker":"(Karaev et al., 2023)"},{"why":"Extracts optical flow and occlusion masks from training clips to serve as trajectory guidance.","marker":"(Yang et al., 2023a)"},{"why":"Provides the 10k-clip training dataset used to fine-tune the trajectory attention branch.","marker":"(Ju et al., 2024)"},{"why":"The zero-initialized add-on branch design that motivates modeling trajectory attention as an auxiliary residual branch.","marker":"(Zhang et al., 2023a)"},{"why":"The pose-embedding baseline for image camera control whose precision the method is compared against.","marker":"(Wang et al., 2024c)"},{"why":"The warping-based video baseline shown to be orthogonal and combinable with trajectory attention.","marker":"(You et al., 2024)"},{"why":"The pose-conditioning baseline used in the 25-frame image comparison and trajectory visualizations.","marker":"(He et al., 2024)"},{"why":"The flow-based attention baseline for image-to-video control that the method compares against.","marker":"(Shi et al., 2024)"},{"why":"Defines the ATE and RPE metrics in which all control-precision results are reported.","marker":"(Goel et al., 1999)"}],"fun_headline_variants":["Pixel paths guide video diffusion cameras","Trajectory attention tightens camera control in video generation","Video diffusion follows pixel trajectories for precise camera motion","New attention branch gives video cameras a precise path","Trajectory attention: precise camera motion via pixel paths"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the extracted pixel trajectories are accurate: they come from an estimated depth map, fixed assumptions about the camera's focal length and center, and, for videos, a point tracker, so any error in those estimates means the attention aligns features along paths that are not the true 3D trajectories.","fun_headline_variants_meta":{"raw":{"variants":["Pixel paths guide video diffusion cameras","Trajectory attention tightens camera control in video generation","Video diffusion follows pixel trajectories for precise camera motion","New attention branch gives video cameras a precise path","Trajectory attention: precise camera motion via pixel paths"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00048,"raw_usage":{"total_tokens":2375,"prompt_tokens":943,"completion_tokens":1432,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":1360}},"tokens_in":559,"tokens_out":1432,"duration_ms":8971,"temperature":1.0,"reasoning_tokens":1360,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:17:15.400354+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a benchmark with known camera poses and ground-truth depth, generate the same scenes twice, once with trajectories computed from ground-truth depth and once from the paper's estimated depth with its default camera assumptions, then compare ATE and RPE. If pose error does not increase when the depth map is corrupted or replaced, trajectory accuracy is not what drives the control; if error rises sharply, the dependence on trajectory extraction is confirmed. A complementary test is to ablate the auxiliary branch itself by feeding the sampled trajectory features directly into the temporal attention's value stream; if control precision stays equally high, the auxiliary-branch design is not the source of the reported gains.","supporting_citations":[{"cited_title":"Robust localization using relative and absolute position estimates","cited_arxiv_id":null,"evidence_quote":"Defines the ATE and RPE metrics in which all control-precision results are reported."}],"review_version":1}