{"id":"b006c4e4-889a-4c63-9ee9-8da039599db4","arxiv_id":"2506.17220","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Video diffusion transformers encode temporal correspondences primarily in query-key similarities of a few specific attention layers, which can be extracted for zero-shot point tracking and used for training-free motion-enhanced video generation.","lead":"This paper introduces DiffTrack, a framework that traces how video diffusion transformers match points across frames by reading their internal attention. It shows that a few attention layers do most of the temporal matching, uses this to track points in real videos, and adds a training-free guidance that improves motion consistency in generated videos.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central layer/timestep analysis is scored against CoTracker pseudo-GT trajectories; if CoTracker's biases shape those pseudo-labels, the claimed 'critical layers' (e.g., l=17) may reflect CoTracker rather than true temporal correspondence.","rationale":"The reader's conditional verdict focuses on CoTracker pseudo-GT; I agree this is the load-bearing assumption. The paper has real strengths: the framework is well specified, the zero-shot point tracking results on TAP-Vid are evaluated against human-annotated tracks and beat the listed baselines, and the CAG guidance is ablated across layers. None of this, however, independently verifies the intra-model layer/timestep rankings, because those rankings are the output of an evaluation whose reference labels come from CoTracker. The TAP-Vid numbers could remain high even if the rankings were CoTracker-specific, as long as CoTracker's behavior correlates with true motion on typical videos. Thus the practical application is more robust than the scientific attribution claim. Since the paper's headline contribution is precisely the attribution claim ('specific but not all layers'), the CoTracker dependency is not a minor footnote; it is the main thing that would have to be true for the analysis to measure what it says. The proposed check -- rerunning the same pipeline with an independent tracker or synthetic GT -- is inexpensive relative to the claim and would settle whether the rankings are an artifact. Because the concern is addressable and the existing external validation supports practical usefulness, the conditional verdict remains appropriate.","tokens_in":28492,"tokens_out":4679,"duration_ms":50509,"concrete_test":"Recompute the layer-wise and noise-level analysis of Sec. 3.4 on the same 100 generated videos (50 object + 50 scene) using an independent pseudo-GT source, e.g., TAPIR or CoTracker3 trajectories instead of CoTracker. If the top-20 harmonic-mean layer set changes or the selected layer/timestep (l=17, t=1) drops out of the top ranks, the central claim is tracker-specific and the verdict should be conditional on replacing or benchmarking the pseudo-GT oracle. A complementary check would be rendered synthetic videos with known ground-truth trajectories, but the independent-tracker rerun is the minimal decisive test.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The weakest load-bearing step is the evaluation substrate introduced in Sec. 3.1: CoTracker is used to produce the pseudo-GT tracks T for all generated videos, and the matching-accuracy metric in Sec. 3.3 then scores DiffTrack's correspondences against T. Every subsequent ranking -- representation selection, the layer-wise harmonic mean that isolates l=17 (and l=13,17,21 for CAG), and the timestep analysis that selects t=1 -- is computed against this same CoTracker reference. If CoTracker is systematically biased (e.g., it relies on appearance or positional priors, fails on large or deformable motion, or is itself shaped by attention-like correlations in video transformers), the analysis may be measuring agreement with CoTracker's inductive biases rather than with true physical correspondence. The confidence and attention scores are also computed only on points CoTracker labels visible, so occluded or difficult points are excluded from all three metrics. The real-video analysis in Fig. A.6 is likewise scored against CoTracker tracks, so it does not break the dependency. External validation on TAP-Vid (Table 1) uses human-annotated tracks and mitigates the practical claim, but it does not independently establish the layer/timestep rankings that drive the analysis, because those rankings were fixed using CoTracker pseudo-GT before Table 1 was computed. Therefore, the central scientific claim that 'specific, but not all, layers play a critical role in temporal matching' is currently contingent on CoTracker being a sufficiently unbiased oracle for synthesized videos.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DiffTrack, a framework for quantitatively analyzing how video diffusion transformers (DiTs) establish temporal correspondences during generation. The authors construct a dataset of prompt-generated videos with pseudo ground-truth trajectories from CoTracker, propose three metrics (matching accuracy, confidence score, and attention score) combined via a harmonic mean, and use these to analyze representation type (query-key vs. intermediate features), layer, and denoising timestep in CogVideoX-2B, CogVideoX-5B, HunyuanVideo, and CogVideoX-2B-I2V. They report that query-key similarities in a few specific layers dominate temporal matching and that matching strengthens as denoising progresses. The findings are applied to zero-shot point tracking on TAP-Vid, where the approach achieves high average accuracy, and to Cross-Attention Guidance (CAG), a training-free guidance method that improves motion consistency in generated videos.","tokens_in":28973,"tokens_out":5271,"duration_ms":55110,"significance":"If the findings hold, the work is a valuable step toward understanding where and when video DiTs encode motion information, and it demonstrates practical payoffs in zero-shot tracking and motion-enhanced generation. The paper is commendably concrete: it evaluates multiple open-source DiT backbones, provides ablations for the proposed tracking architecture (Tables A.2-A.4), and validates CAG with both automatic metrics and human evaluation. The external TAP-Vid benchmark provides a strong check on the practical utility of the selected layer and timestep. However, the central scientific claim about which layers and timesteps matter is currently tied to a single pseudo-label source, so the analysis needs independent validation before the conclusions can be regarded as robust.","major_comments":[{"comment":"The layer-wise and timestep rankings that drive the paper's central claims are computed against pseudo ground-truth tracks produced by CoTracker. The matching accuracy metric (Eqs. 5-7) is evaluated against T from CoTracker, and the confidence and attention scores are averaged only over points CoTracker marks visible. Consequently, the selection of l=17, t=1, and the CAG layers l=13,17,21 is a measurement of agreement with CoTracker's inductive biases rather than with ground-truth physical motion. The real-video analysis in Fig. A.6 also uses CoTracker as the reference, so it does not break this dependency. Because the paper's main scientific claim is about where temporal correspondences are established inside video DiTs, this is load-bearing. I recommend validating the rankings with an independent reference, such as human-annotated TAP-Vid tracks, a second point tracker, or synthetic sequences with known ground-truth motion, and showing that the layer/timestep selections are stable.","section":"Sec. 3.1 and Sec. 3.3"},{"comment":"The abstract and Section 3.4 state that temporal matching becomes increasingly prominent during the denoising process, but the authors' own analysis of CogVideoX-2B-I2V (Fig. A.4(c)) shows a sharp decline at later timesteps, and Table A.1 shows a corresponding drop in point accuracy. The claim in the abstract is therefore too strong and should be qualified as model-dependent, or the exception should be discussed in the main text.","section":"Abstract and Sec. 3.4 vs. Fig. A.4(c)"},{"comment":"The TAP-Vid benchmark validates the zero-shot tracking application but does not independently verify the internal analysis: the layer and timestep were selected using CoTracker-based pseudo-GT before Table 1 was computed. The paper states that the benchmark results 'highlight the in-depth analysis,' but the benchmark cannot serve as evidence for the rankings themselves. Please either provide an independent validation of the internal rankings or explicitly state that the benchmark only confirms the practical quality of the selected features.","section":"Sec. 4, Table 1"}],"minor_comments":[{"comment":"The normalization scheme used before computing the harmonic mean of matching accuracy, confidence score, and attention score is not described; please specify how each metric is normalized across layers and timesteps.","section":"Sec. 3.3"},{"comment":"For the fair-evaluation comparison, most baselines are resized to produce 30x45 feature maps, while ZeroCo uses a 14x14 feature map at 224x224 input; the potential effect of this resolution mismatch on the comparison should be discussed.","section":"Sec. 4.1 and Table 1"},{"comment":"The layer-wise analysis in Fig. 4(b) reports top-20/top-50 scores without error bars or statistical significance across the 50 prompts per dataset; adding variance or a significance test would strengthen the claim that a specific layer dominates.","section":"Fig. 4"},{"comment":"There is a typo in the implementation details: 'for for CogVideoX-5B' should read 'for CogVideoX-5B'.","section":"Sec. C.2"},{"comment":"The notation for the set of cross-frame indices F and the number of frames F in the preliminaries is used interchangeably; please disambiguate the symbol.","section":"Eq. (9)"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the journal and the central idea is interesting. My main concern is that the internal analysis is anchored to CoTracker pseudo-labels; the authors should be encouraged to add an independent reference validation. The manuscript also overgeneralizes the denoising trend in the abstract despite the authors' own I2V counterexample. These issues are fixable in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful, well-executed paper. The central finding—query-key similarities in a few layers carry most temporal matching, and matching sharpens during denoising—is new and plausible, and the zero-shot tracking application is a real, externally validated payoff. The main weakness is exactly what the stress test flags: the layer/timestep rankings are selected against CoTracker pseudo-labels, so the analysis measures agreement with CoTracker as much as with true physical correspondence. That doesn't sink the paper, but it should be said plainly.\n\nWhat's new: earlier work on diffusion-model correspondence focused on image U-Nets and two-frame matching. This is the first systematic look at full 3D attention in video DiTs across layers and timesteps, plus a training-free guidance method (CAG) that improves temporal consistency. The tracking ablations are useful, and the paper honestly documents where feature fusion does not help.\n\nWhat's solid: Table 1 on TAP-Vid is the strongest evidence—human-annotated tracks, consistent gains over DINOv2, DIFT, VFS, and SVD across Kinetics and DAVIS and across three backbones. The CogVideoX-2B-I2V result is a nice sanity check: I2V finetuning preserves the first frame at the expense of motion, and DiffTrack's matching accuracy drops accordingly. The PCA and attention visualizations support the qualitative story.\n\nSoft spots: the pseudo-label dependency is real. All rankings in Sec. 3 come from comparing against CoTracker on 100 generated videos; occluded points are excluded, so the metrics cover easier points, and the real-video analysis in Fig. A.6 still uses CoTracker. What would fix it: run the layer/timestep sweep directly on TAP-Vid human tracks, or validate on synthetic videos with known motion. Missing error bars and the exact CAG guidance scale are minor but should be added.\n\nOverall: this deserves peer review. I'd send it out, and ask referees to push for the TAP-Vid layer sweep and error bars. The citation pattern is fine—the group's earlier query-key work is genuinely the relevant prior art.\n\nRecommendation: engage with it, conditional on the layer-selection validation being strengthened.","headline":"A useful empirical map of where temporal correspondence lives in video DiTs, with a real zero-shot tracking payoff—but the layer rankings are selected against CoTracker pseudo-labels, so the mechanism claims are suggestive, not settled.","tokens_in":29344,"tokens_out":2971,"would_cite":true,"duration_ms":29863,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Temporal correspondences in video diffusion transformers emerge from query-key similarity in a few dominant attention layers, and can be extracted for zero-shot point tracking and motion-enhanced generation.","keywords":["video diffusion transformers","temporal correspondence","query-key matching","cross-frame attention","zero-shot point tracking","diffusion guidance","attention analysis","point tracking"],"falsifier":"Manually annotate point trajectories on a sample of the generated videos, or replace CoTracker with an independent tracker, and rerun the layer and timestep selection; if the top descriptors move away from $l=17$, $t=1$ for CogVideoX-2B or the layer ranking changes substantially, the claimed locus of temporal correspondence is an artifact of the pseudo-labeling.","tokens_in":28317,"feed_emoji":"🎯","tokens_out":6667,"duration_ms":64793,"temperature":0.7,"pith_summary":"This paper tries to establish where and when video diffusion transformers (DiTs) encode temporal correspondence during the denoising process. Using a curated dataset of prompt-generated videos with pseudo ground-truth trajectories and three complementary metrics, it finds that query-key similarities inside the full 3D attention mechanism, not intermediate features, carry the matching signal. The matching is concentrated in a small set of layers and becomes progressively sharper as denoising proceeds. The paper shows that reading query-key descriptors at the identified layer and timestep yields state-of-the-art zero-shot point tracking on real videos, and that perturbing cross-frame attention in those layers improves temporal consistency of generated videos without additional training.","feed_headline":"Video diffusion models already track moving points inside attention","feed_subtitle":"Reading one attention layer at the final denoising step beats specialized trackers on TAP-Vid, with no training.","key_machinery":"The central object is full 3D attention in DiT blocks, where each layer computes softmax($QK^\\top/\\sqrt{d}$) over a sequence containing all frame latents and text tokens. The relevant sub-block is cross-frame attention $A^{i,j}_{t,l}$, the attention from tokens of frame $i$ to tokens of frame $j$, whose per-pixel argmax gives a predicted match. DiffTrack's machinery is the query-key matching cost $C^{1,j}_{t,l} = \\mathrm{Softmax}(D^1_{t,l}(D^j_{t,l})^\\top/\\sqrt{d})$ with descriptors $D$ set to queries and keys rather than intermediate features, together with three metrics: matching accuracy (PCK against pseudo ground truth), confidence (maximum attention value), and attention score (sum of cross-frame attention). Their harmonic mean identifies the dominant layer and timestep, e.g. $l=17$, $t=1$ for CogVideoX-2B. The same cross-frame attention maps are the target of Cross-Attention Guidance, which zeros them out in chosen layers to create a degraded motion sample and guides the model away from it.","core_discovery":"The central discovery is that video DiTs establish temporal correspondences through query-key matching in full 3D attention: the softmax query-key product between latents of different frames acts as a matching cost that tracks corresponding points across frames. This matching is not uniform across the network; a handful of layers are responsible, and for CogVideoX-2B the single best descriptor location is layer 17 at the final denoising timestep $t=1$. Matching accuracy is highest late in denoising, with a slight degradation at the very end. The same pattern holds across CogVideoX-5B and HunyuanVideo, and an analysis on real videos using noised latents reproduces the layer and timestep conclusions. DiffTrack treats these attention-derived descriptors as a zero-shot tracker, achieving the best average point accuracy on TAP-Vid among the compared image foundation models and self-supervised video models, and it uses the identified dominant layers to design Cross-Attention Guidance, which improves motion consistency.","pith_inferences":["An independent test with human-labeled trajectories on a sample of the generated videos would tell whether the layer and timestep conclusions describe physical motion or CoTracker's bias; this is testable with relatively small annotation effort.","If the layer specialization is robust, practitioners could speed up video DiTs by restricting full cross-frame attention to the few dominant layers for motion-relevant steps, or reuse those layers as a training-free motion prior for tracking, flow-like estimation, and video editing.","The late-denoising sharpening suggests motion is committed near the end of sampling, which may explain why trajectory-conditioned control methods are effective and why early-step motion conditioning is insufficient.","The guidance result implies cross-frame attention is a source of temporal consistency rather than a purely descriptive byproduct; this predicts that selectively degrading cross-frame attention in dominant layers during sampling will produce characteristic motion errors such as frozen or duplicated frames."],"forward_implications":["Zero-shot point tracking: query-key descriptors from one selected layer and timestep outperform single-image foundation models and self-supervised video models on TAP-Vid average accuracy.","Feature selection matters: using the single dominant layer and timestep is much better than averaging over all layers or timesteps, and fusing multiple timesteps and layers does not improve temporal matching.","Cross-frame attention is causally involved in motion: zeroing it in the dominant layers and guiding away from the degraded sample improves subject consistency, background consistency, dynamic degree, and imaging quality without training.","The pattern generalizes across architectures: CogVideoX-5B and HunyuanVideo show the same query-key dominance, concentration in a few layers, and strengthening during denoising, while the image-to-video variant loses temporal matching as self-frame attention dominates."],"supporting_citations":[{"why":"Supplies the pseudo ground-truth trajectories for generated videos and serves as the reference for matching accuracy, so the analysis hinges on its reliability.","marker":"[42]"},{"why":"Provides the TAP-Vid benchmark and the real-video point tracks used to evaluate zero-shot point tracking.","marker":"[18]"},{"why":"CogVideoX-2B and CogVideoX-5B are the primary video DiT backbones analyzed and the baselines for Cross-Attention Guidance.","marker":"[80]"},{"why":"HunyuanVideo is the second video DiT used to show that the temporal-matching findings generalize.","marker":"[47]"},{"why":"DIFT establishes the prior paradigm of extracting correspondences from diffusion features and is a baseline for zero-shot tracking.","marker":"[71]"},{"why":"ZeroCo demonstrates that query-key matching in cross-attention yields geometric correspondence, the idea DiffTrack transfers to video DiTs.","marker":"[3]"},{"why":"PAG provides the perturbed-attention guidance mechanism that Cross-Attention Guidance adapts to cross-frame attention.","marker":"[2]"},{"why":"VBench supplies the automatic temporal-quality metrics used to evaluate Cross-Attention Guidance.","marker":"[37]"},{"why":"DDIM inversion is used to map real videos into latents so DiffTrack can extract descriptors for zero-shot tracking.","marker":"[68]"}],"fun_headline_variants":["Video diffusion models track points via attention layers","Zero-shot point tracking from video DiT attention","Attention query-key similarities are temporal trackers","Layer 17 at final step tracks points in video DiTs","Video DiTs hide a no-training point tracker in attention"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The analysis depends on the assumption that CoTracker's estimated trajectories, used as stand-in ground truth for videos the model itself generated, are accurate enough to rank layers and timesteps; if CoTracker systematically misses or biases certain motions, the identified dominant layer could be tuned to CoTracker rather than to true physical correspondence.","fun_headline_variants_meta":{"raw":{"variants":["Video diffusion models track points via attention layers","Zero-shot point tracking from video DiT attention","Attention query-key similarities are temporal trackers","Layer 17 at final step tracks points in video DiTs","Video DiTs hide a no-training point tracker in attention"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000268,"raw_usage":{"total_tokens":1626,"prompt_tokens":962,"completion_tokens":664,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":590}},"tokens_in":578,"tokens_out":664,"duration_ms":7178,"temperature":1.0,"reasoning_tokens":590,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:09:29.073712+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Manually annotate point trajectories on a sample of the generated videos, or replace CoTracker with an independent tracker, and rerun the layer and timestep selection; if the top descriptors move away from $l=17$, $t=1$ for CogVideoX-2B or the layer ranking changes substantially, the claimed locus of temporal correspondence is an artifact of the pseudo-labeling.","supporting_citations":[{"cited_title":"CoTracker: It is better to track together","cited_arxiv_id":null,"evidence_quote":"Supplies the pseudo ground-truth trajectories for generated videos and serves as the reference for matching accuracy, so the analysis hinges on its reliability."},{"cited_title":"TAP-Vid: A benchmark for tracking any point in a video.NeurIPS, 35:13610–13626, 2022","cited_arxiv_id":null,"evidence_quote":"Provides the TAP-Vid benchmark and the real-video point tracks used to evaluate zero-shot point tracking."},{"cited_title":"Emergent correspondence from image diffusion.NeurIPS, 36:1363–1389, 2023","cited_arxiv_id":null,"evidence_quote":"DIFT establishes the prior paradigm of extracting correspondences from diffusion features and is a baseline for zero-shot tracking."},{"cited_title":"VBench: Comprehensive benchmark suite for video generative models","cited_arxiv_id":null,"evidence_quote":"VBench supplies the automatic temporal-quality metrics used to evaluate Cross-Attention Guidance."}],"review_version":2}