{"id":"dd0a70ab-cc0a-4533-a55c-5398e65aa3f8","arxiv_id":"2412.06016","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Adding point-tracking supervision to video diffusion features reduces appearance drift in generated videos while preserving generation quality.","lead":"Track4Gen adds point-tracking supervision to a video diffusion model, training it to track objects across frames while generating video. The method reduces the appearance drift that makes generated objects morph or degrade over time.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported generation gains may come from the added zero-conv residual path rather than from the tracking loss; no ablation removes Lcorr while keeping the architecture.","rationale":"The reader's verdict of CONDITIONAL is appropriate, but for a sharper reason than the one stated. The pseudo-ground-truth concern is valid, yet the paper has a more immediate internal-validity gap: the benefit attributed to tracking supervision is confounded with the addition of a trainable residual injection path. Since zeta_psi is trained by Ldiff and can learn to modify features even if Rphi stays identity, a no-Lcorr version of the full architecture is necessary to support the claim that 'teaching video diffusion models to track points improves video generation.' The paper's ablations do not include this control, so the central mechanism is not established. This is addressable with one additional training run, so it does not warrant rejection, but it should be a condition for acceptance. I also note in good faith that the method is clearly described, the refiner design is sensible, and the qualitative results are plausible; the missing control is a testable gap rather than a fundamental flaw.","tokens_in":17118,"tokens_out":6865,"duration_ms":67434,"concrete_test":"Train a control variant with the exact Track4Gen architecture (identity-initialized refiner and zero-convolution injection) but with lambda = 0 so Lcorr is absent, keeping the same 567 clips, 20K steps, and evaluation protocol as Section 4.1 and Table 1. Run this control, finetuned SVD, and Track4Gen with 3 seeds each and report means and standard deviations for the five VBench metrics plus FID/FVD. If the lambda = 0 control matches Track4Gen within run-to-run noise (e.g., subject consistency within 0.003), the tracking loss is not the active ingredient; if it matches finetuned SVD instead, the causal claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central causal claim is that point-tracking supervision improves video generation by enhancing spatial correspondence in diffusion features. However, the experimental design never isolates Lcorr from the architectural change that carries it. Track4Gen differs from the finetuned-SVD baseline in two coupled ways: (1) the correspondence loss supervises the refiner Rphi, and (2) the decoder features are modified by a trainable zero-convolution zeta_psi that injects Rphi's output into the backbone. zeta_psi is optimized by Ldiff alone, initialized to zero, and receives stop-gradient features from Rphi, so it is a learnable residual adapter that can adapt the third-decoder features to the 567-video finetuning set even if the tracking loss contributes nothing. The ablations in Table 4 vary which transformer blocks are trained and whether the refiner uses 2D or 3D convolutions, but no variant trains the full Track4Gen architecture with Lcorr removed. The 'Track4Gen w/o refiner' baseline also removes the injection path, so it does not control for the adapter. Thus the observed improvements in Table 1 over finetuned SVD could be caused by the extra adapter capacity rather than by tracking supervision, leaving the paper's headline claim causally untested. The pseudo-label quality concern raised by the reader is relevant, but it is secondary: even perfect trajectories would not establish the mechanism without a no-Lcorr control.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Track4Gen, a method to add point-tracking supervision to a video diffusion model (Stable Video Diffusion) in order to reduce appearance drift. The method attaches a trainable refiner module to a selected U-Net block, supervises the refined features with a correspondence loss computed against RAFT-generated point tracks, and injects the refined features back into the backbone through a zero convolution initialized to identity. The authors claim that this joint training unifies video generation and point tracking, and report quantitative improvements on VBench metrics, FID/FVD, and user studies, as well as improved zero-shot feature-based tracking on TAP-Vid benchmarks.","tokens_in":17460,"tokens_out":3747,"duration_ms":36714,"significance":"If the central causal claim is correct, Track4Gen would be a valuable and simple recipe for improving spatial coherence of video generators by adding correspondence supervision. The paper is clearly written, the method is well motivated by a feature analysis (Sec. 3.2), and the evaluation is broad, including tracking benchmarks and a user study. The authors are honest about limitations (reduced camera motion, failure cases). However, the headline claim that tracking supervision specifically improves generation is not directly tested, because no ablation removes the correspondence loss while keeping the architecture identical. The quantitative results also lack error bars and significance tests, which is important given the small evaluation set and the modest metric gaps.","major_comments":[{"comment":"The experimental design does not isolate the correspondence loss Lcorr from the architectural change. Track4Gen differs from the finetuned-SVD baseline in two coupled ways: the refiner module Rphi is supervised by Lcorr, and the zero-convolution zeta_psi injects Rphi's output into the backbone. The ablation 'Track4Gen w/o refiner' removes both the refiner and the injection path, so the improved scores in Table 1 could come from the added residual-adapter capacity rather than from tracking supervision. Please add a variant trained with the full Track4Gen architecture (refiner Rphi plus zero-conv zeta_psi) but with Lcorr removed (lambda = 0 in the joint loss), so the effect of the correspondence loss is directly measurable. This control is necessary to support the paper's central claim.","section":"Sec. 4.2, Table 4 and Eq. (4)"},{"comment":"Quantitative generation results are reported without error bars, confidence intervals, or significance tests. Several VBench differences between Track4Gen and finetuned SVD are small (e.g., Temporal Flickering 0.9806 vs 0.9800, Motion Smoothness 0.9921 vs 0.9909), and the VBench evaluation uses only 355 images. The FID/FVD values are also single numbers without variance; the user study reports only aggregate preference percentages. Without variance estimates or at least paired significance tests, it is hard to assess whether the reported improvements are robust. Please report results over multiple seeds or bootstrap confidence intervals, and include the user-study per-participant agreement or a significance test.","section":"Table 1 and Fig. 8"},{"comment":"The training data is only 567 short video clips, with pseudo-ground-truth trajectories generated by chaining RAFT optical flow and filtering via cycle consistency. This is a small and potentially noisy supervision source. Since the method finetunes the temporal transformer blocks on this data, the observed reduction in appearance drift could partly reflect adaptation to the specific training distribution (including reduced camera motion, as the authors acknowledge in the Conclusion) rather than a generalizable correspondence prior. Please report statistics on the quality and coverage of the generated tracklets (e.g., number of surviving tracks, distribution of motion magnitudes), and consider evaluating generation on a broader set of prompts to demonstrate generalization beyond the training distribution.","section":"Sec. 4.1 and Supplementary A.1"}],"minor_comments":[{"comment":"The weighting factor lambda in the joint loss Ldiff + lambda * Lcorr is not defined until Sec. 4.1; it would be helpful to introduce it at the point of the loss definition in Sec. 3.3.","section":"Sec. 3.3, Eq. (4)"},{"comment":"The formatting of Table 2 is dense and the column alignment is difficult to read; please reformat it for clarity, separating generation and tracking metrics more clearly.","section":"Sec. 4.2 and Table 2"},{"comment":"The cosine similarity threshold of 0.6 is used both for pruning correspondences in feature analysis and for occlusion prediction in tracking evaluation; this dual use should be clarified, as the two tasks may benefit from different thresholds.","section":"Sec. 3.2 and Sec. 4.3.1"},{"comment":"The description of the refiner network states that the last layer has no BatchNorm, while the first seven layers do; please confirm that this design choice is intentional and report whether BatchNorm in the last layer was tried.","section":"Supplementary A.2"}],"recommendation":"major_revision","confidential_remarks":"The core idea is attractive and the paper is generally well written. The missing no-Lcorr control is the key issue: without it, the causal claim that tracking supervision improves generation is not established. The paper would also be strengthened by error bars and a brief sensitivity analysis for the RAFT-derived pseudo-labels. If the authors can provide the missing control and statistical reporting, the manuscript would likely be suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: the core idea is genuinely new—supervise a video diffusion model's intermediate features with explicit point-trajectory targets to reduce appearance drift—and the implementation is clean and minimal. There's one gap that keeps the headline from being proven: the experimental design never isolates the tracking loss from the trainable residual path that carries it.\n\nThe novel part is concrete. The authors add a small refiner R_phi on the third-decoder block features, compute a cosine-similarity soft-argmax tracking loss against RAFT-chained pseudo-trajectories, and feed the refined features back into the backbone through a zero-convolution that starts as a no-op. They detach the refiner from the diffusion loss, so it is optimized solely by the correspondence loss. That is a neat way to add spatial supervision without throwing away the model's prior. The motivation analysis showing the third decoder block produces the strongest correspondences is a solid empirical grounding. The evaluation is broad: VBench metrics, FID/FVD, a user study, and tracking benchmarks, with honest discussion of limitations and failure cases.\n\nThe soft spot is the missing control. Track4Gen differs from the finetuned-SVD baseline in two coupled ways: the tracking loss Lcorr, and the zeta_psi residual adapter that can reshape the decoder features with Ldiff alone. The 'w/o refiner' ablation removes both the refiner and the injection path, so it does not answer whether the adapter alone—with no tracking loss—would produce the gains. That matters because Table 1's deltas are modest: subject consistency goes from 0.9665 (finetuned SVD) to 0.9746, FID from 27.0 to 26.6, with no error bars or significance tests. Without a full-architecture no-Lcorr run, the most plausible alternative explanation is that the extra adapter capacity, not the point-tracking supervision, drives the improvement. The title and abstract state the causal mechanism, so this is not a minor omission.\n\nThe pseudo-label concern is secondary but real: 567 clips with RAFT-chained tracks is a small base, and the tracking evaluation shows Track4Gen features with DINO-Tracker underperform plain DINO-Tracker on DAVIS (72.5 vs 80.4 delta_x_avg), which complicates the claim of unifying generation and tracking.\n\nNone of this is fatal. The idea is worth pursuing, and the missing ablation is a specific, fixable request. I would send this to peer review, and I would want the revision to include the full-architecture no-Lcorr control plus error bars or significance tests on the main VBench/FID/FVD numbers. If the control shows the gains vanish, the paper still stands as a practical recipe—just not as evidence for the tracking-supervision mechanism.\n\nWho it is for: people working on video diffusion consistency or on joint generative/discriminative feature learning. It is worth engaging with.","headline":"Track4Gen has a genuinely new idea with a clean implementation, but the missing no-Lcorr control means the reported generation gains are not causally pinned to tracking supervision.","tokens_in":17936,"tokens_out":6816,"would_cite":true,"duration_ms":59047,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Point-tracking supervision on diffusion features reduces video appearance drift.","keywords":["appearance drift","video generation","point tracking","diffusion model","spatial correspondence","Stable Video Diffusion","tracking-any-point","temporal coherence"],"falsifier":"Train Track4Gen with the same architecture but replace the correspondence labels by random point pairs within each video; if appearance-drift scores on VBench stay high and tracking accuracy stays low, the claimed causal role of the tracking supervision would be refuted.","tokens_in":16960,"feed_emoji":"🎬","tokens_out":8253,"duration_ms":69791,"temperature":0.7,"pith_summary":"Appearance drift—objects in generated video gradually changing identity, morphing, or degrading—persists in diffusion video generators even as they improve, and the paper attributes this to missing spatial-tracking supervision in the feature space. Track4Gen adds a point-tracking objective to the diffusion loss, forcing the intermediate features of a pretrained video generator to encode where points move across frames. A trainable refiner module and a zero convolution inject these correspondence-aware features back into the generator, so the same network both generates video and tracks points. The paper reports that this reduces appearance drift on Stable Video Diffusion, improving subject consistency and temporal coherence without lowering standard video-quality scores, and that the resulting features also track points in real videos.","feed_headline":"Point-tracking supervision stops video models from mutating objects","feed_subtitle":"Adding a tracking loss to Stable Video Diffusion yields stable subjects and turns the generator into a point tracker.","key_machinery":"The machinery is the pair of a trainable refiner module $R_\\phi$ and a zero-convolution gate $\\zeta_\\psi$ attached to the upsampler layer of the third decoder block of a Stable Video Diffusion U-Net. $R_\\phi$, eight stacked 2D convolution layers initialized as identity, transforms raw diffusion features $h^{1:N}$ into refined features $\\tilde{h}^{1:N}$; the correspondence loss computes cosine-similarity cost volumes between a query point's feature and a target frame's feature map, applies a differentiable soft-argmax over a radius-limited window to predict the target position, and penalizes prediction error with a Huber loss. The refined features are fed into the next U-Net block only through $\\zeta_\\psi$, with gradients detached before the refiner, so $L_\\text{corr}$ trains the refiner and the temporal transformer blocks while the diffusion loss continues to train the whole generator. This design is what lets tracking supervision reshape the feature space without destroying the pretrained generation prior.","core_discovery":"Track4Gen's central claim is that explicit spatial-correspondence supervision at the feature level is the missing ingredient for temporally consistent video generation. The paper demonstrates this by training a single network to minimize both the video diffusion denoising loss and a correspondence loss: raw U-Net features from the third decoder block's upsampler layer are projected by an identity-initialized refiner into a correspondence-rich space, cosine-similarity cost volumes with soft-argmax predict point tracks, and a Huber loss supervises those predictions against pseudo-ground-truth trajectories. The refined features are also routed back into the generation backbone through a zero convolution, so the generation path inherits spatial awareness while preserving the pretrained model's prior. On VBench, DAVIS, and BADJA, the paper finds meaningfully improved subject consistency, reduced flickering, and tracking accuracy that approaches dedicated optical-flow chaining, and concludes that video generation and point tracking can be unified in one architecture.","pith_inferences":["If the causal mechanism is the tracking supervision itself, then scaling the 567-clip training set with automatically annotated real videos should further reduce drift; the small data scale makes this an untested consequence of the paper's hypothesis.","The same refiner-plus-tracking-loss design should apply to transformer-based (DiT) video generators, but the paper only tests U-Net architectures, so this extension is speculative.","The paper's acknowledged trade-off of reduced camera motion suggests an extension that supervises global motion separately from point-level correspondences could recover dynamism without sacrificing identity stability.","Because feature tracking still fails on fast motion, occlusions, and semantically similar objects, adding an explicit occlusion-prediction term to the correspondence loss is a natural next step and is flagged by the authors as future work."],"forward_implications":["Video generators trained this way should keep a subject's identity stable across many frames, eliminating the object-mutation and object-replacement failures typical of baseline Stable Video Diffusion.","The same checkpoint can double as a point tracker: zero-shot feature matching with Track4Gen features approaches RAFT optical-flow chaining accuracy on DAVIS and BADJA.","Plugging Track4Gen features into a test-time-optimization tracker yields long-term tracking accuracy comparable to dedicated supervised trackers.","Because only a small refiner and the temporal transformer blocks are finetuned, the recipe should transfer to other U-Net video diffusion models with minimal changes.","Standard generation-quality metrics (FID, FVD, motion smoothness, image quality) do not degrade, so the drift reduction is not bought at the cost of overall video fidelity."],"supporting_citations":[{"why":"Supplies the pretrained Stable Video Diffusion backbone, the image-to-video conditioning, and the primary generation baseline that Track4Gen finetunes.","marker":"[5]"},{"why":"Provides the RAFT optical flow used to generate pseudo-ground-truth point trajectories by chaining flow fields and cycle-consistency filtering.","marker":"[64]"},{"why":"Supplies the VBench benchmark and its subject-consistency, flickering, motion-smoothness, and quality metrics used to measure generation improvements.","marker":"[33]"},{"why":"Provides the test-time-optimization tracking method whose features are replaced by Track4Gen features for long-term tracking evaluation.","marker":"[66]"},{"why":"Defines the TAP-Vid benchmark and metrics for tracking any point, and informs the foreground/background balancing of trajectory points.","marker":"[14]"},{"why":"Serves as the reference dataset for FID/FVD evaluation and as a source of training clips with segmentation maps for balanced point sampling.","marker":"[52]"},{"why":"Supplies the BADJA animal-keypoint tracking benchmark used to measure tracking accuracy.","marker":"[4]"},{"why":"Contributes the zero-convolution layer mechanism used to route refined features back into the generation backbone.","marker":"[82]"}],"fun_headline_variants":["Point tracking fixes appearance drift in video diffusion","Track points to stop video models from mutating objects","A tracking loss keeps video subjects from morphing","Video generation gets stable subjects via point tracking","Teach video diffusers to track, and they generate coherently"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the pseudo-ground-truth point trajectories—generated by chaining optical-flow estimates from a small set of 567 short video clips and keeping only cycle-consistent matches—are accurate and diverse enough to teach a generalizable spatial-correspondence prior that transfers to other videos.","fun_headline_variants_meta":{"raw":{"variants":["Point tracking fixes appearance drift in video diffusion","Track points to stop video models from mutating objects","A tracking loss keeps video subjects from morphing","Video generation gets stable subjects via point tracking","Teach video diffusers to track, and they generate coherently"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000137,"raw_usage":{"total_tokens":1122,"prompt_tokens":891,"completion_tokens":231,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":507,"completion_tokens_details":{"reasoning_tokens":158}},"tokens_in":507,"tokens_out":231,"duration_ms":2828,"temperature":1.0,"reasoning_tokens":158,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:05:28.759328+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train Track4Gen with the same architecture but replace the correspondence labels by random point pairs within each video; if appearance-drift scores on VBench stay high and tracking accuracy stays low, the claimed causal role of the tracking supervision would be refuted.","supporting_citations":[{"cited_title":"Vbench: Comprehensive bench- mark suite for video generative models","cited_arxiv_id":null,"evidence_quote":"Supplies the VBench benchmark and its subject-consistency, flickering, motion-smoothness, and quality metrics used to measure generation improvements."}],"review_version":1}