{"id":"0f5656b0-a7b8-48b8-b562-14f5c11a754c","arxiv_id":"2506.08694","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"MoSiC clusters dense point tracks in videos and propagates the cluster assignments along the tracks, improving DINOv2's dense representations by 1 to 6 percent on segmentation and in-context benchmarks.","lead":"This paper introduces MoSiC, a self-supervised method that uses point-track trajectories and optimal-transport clustering to make dense visual features consistent across video frames. It reports 1 to 6 percent accuracy gains over strong image-pretrained baselines on segmentation and scene-understanding benchmarks, with code released.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (10) does not actually supervise later-frame features: the propagated teacher label is constant and the student score is computed only at t0, so the loss reduces to a visibility-weighted first-frame clustering objective, leaving the claimed motion mechanism unsupported.","rationale":"The reader identified track-object consistency as the weakest assumption. On closer reading, that concern is partly mitigated by the structure of Eq. (10): because the propagated teacher assignment is constant from frame t0 and the student score is always evaluated at t0, a track that drifts onto another object changes only the visibility weight of the original first-frame point; it does not directly train the destination object's features. The deeper problem is that the loss, as formally written, contains no student or teacher feature at any t>0, so the claimed temporal-coherence mechanism is absent from the objective. This is the most load-bearing issue because the paper's novelty and central claim rest on motion-based temporal supervision; if Eq. (10) is accurate, the method is essentially a visibility-weighted first-frame clustering loss on video clips, and the reported gains over DINOv2 could arise from continued training on YouTube-VOS frames rather than from motion. The suggested control (static tracks) would settle whether trajectories matter. The paper's empirical study is otherwise carefully structured, with clean MoSiC-vs-DINOv2 comparisons and released code, so the appropriate response is to require the authors to correct or justify Eq. (10) and provide the control, not to reject the empirical findings outright. Hence the reader's CONDITIONAL verdict stands.","tokens_in":22926,"tokens_out":14809,"duration_ms":185948,"concrete_test":"Run the released MoSiC code with each trajectory replaced by a static track (the first-frame grid position for every t) while preserving the per-point visibility weights v_{t,i} and all other settings. If the mIoU values in Tables 1-4 remain within 0.5 of the reported MoSiC numbers, the trajectory motion is not the operative signal and the concern is confirmed. If the code instead computes student scores at each tracked location for t>0, then Eq. (10) misstates the method and a corrected loss definition is required.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Eq. (10) is the only training loss claimed to implement temporal cluster propagation, but as written it cannot do so. The student score S^{stu,k,i}_{t0} in Eq. (9) depends only on Z^{stu,i}_{t0}, and by Eq. (8) the propagated teacher label Q^{teach,i}_t equals Q^{teach,i}_{t0} for every t. Consequently Eq. (10) factors as -(sum_t v_{t,i}) * sum_k delta(Q^{teach,i}_{t0}=k) * log S^{stu,k,i}_{t0}: a standard cross-entropy clustering loss on the first frame, with each point weighted by its total visible-track length. The gradient of the loss with respect to student features at frames t>0 is identically zero; the trajectory coordinates beyond t0 enter only through the scalar visibility weight. Thus the objective does not align later-frame features, does not use the teacher's later-frame features, and does not enforce spatiotemporal consistency along trajectories. The central claim that 'propagating cluster assignments along motion trajectories' improves dense representations is therefore not realized by the stated loss. This is more decisive than the reader's track-drift concern: because the propagated label is frozen at t0, a drifted track mainly changes the point's weight rather than directly polluting the destination object's features, whereas the complete absence of t>0 feature terms removes the temporal mechanism entirely.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MoSiC, a dense self-supervised video pretraining method. It extracts point trajectories with CoTracker-v3, computes Sinkhorn-Knopp cluster assignments on the first frame for both teacher and student networks, then claims to propagate the teacher's first-frame cluster assignments along the trajectories and trains the student with a cross-entropy loss. The method is initialized from DINOv2, trained on YouTube-VOS, and evaluated on unsupervised video object segmentation, in-context scene understanding, frozen clustering, and linear segmentation, reporting gains over DINOv2 and TimeT.","tokens_in":23136,"tokens_out":6109,"duration_ms":63814,"significance":"If the stated mechanism were actually implemented, the idea of using long-range point tracks to provide dense temporal supervision for a strong image-pretrained backbone is timely and potentially valuable. The paper is strongest in its controlled comparisons against DINOv2 with identical architecture and initialization, where it reports consistent albeit modest gains (about 1 to 4 points on linear segmentation and clustering, up to 6 points in low-data in-context retrieval). The authors provide code, extensive ablations, and a candid discussion of tracker failure cases. However, as written, the central loss does not use any features from frames after t0, so the claimed temporal propagation mechanism is not realized in the stated equations. This issue must be resolved before the paper's significance can be fairly assessed.","major_comments":[{"comment":"The loss in Eq. (10) does not implement temporal cluster propagation. The student score S^{stu,k,i}_{t0} in Eq. (9) depends only on Z^{stu,i}_{t0}, and Eq. (8) sets Q^{teach,i}_t = Q^{teach,i}_{t0} for every t. Consequently, Eq. (10) factors as -(sum_t v_{t,i}) * sum_k delta(Q^{teach,i}_{t0}=k) * log S^{stu,k,i}_{t0}, which is a first-frame cross-entropy clustering loss weighted by the total visible-track length. The gradient with respect to student features at frames t>0 is identically zero, and the teacher's later-frame features are never used. The central claim that the method 'propagates cluster assignments along motion trajectories' to enforce spatiotemporal consistency is therefore not supported by the stated objective. The authors must either correct the equations to use per-frame student scores and per-frame teacher assignments, or revise the claimed mechanism to match the actual loss.","section":"Section 3.4, Eqs. (8)-(10)"},{"comment":"The headline comparison against TimeT is confounded. MoSiC-S14 uses a ViT-S/14 initialized from DINOv2, while TimeT uses a ViT-S/16 initialized from DINO. The footnote in Table 2 acknowledges the architecture difference but does not control for it, so the reported '8.7% and 9.4% mIoU' gains over TimeT partly reflect the stronger backbone and initialization rather than the proposed method. The controlled comparisons against DINOv2-S/14 (e.g., +4.3 and +3.7 on average) are much more modest and should be the primary basis for the paper's claims, or an ablation with a common backbone and initialization should be added.","section":"Section 4.2, Table 2"},{"comment":"The ablations do not isolate the temporal component of the method. All reported ablations vary parameters such as mask ratio, grid size, number of prototypes, clip length, and frame count, but there is no baseline that trains the same first-frame clustering objective without the trajectory-derived visibility weighting. Given that Eq. (10) reduces to a first-frame loss with visibility weights, the observed gains over DINOv2 could come from the weighting or from the extra training data rather than from any spatiotemporal consistency enforced along tracks. A single-frame training baseline with identical loss (e.g., setting all v_{t,i}=1 or using one frame per clip) is necessary to support the claim that motion trajectories provide the supervisory signal.","section":"Section 4.7, Ablations"}],"minor_comments":[{"comment":"The caption appears to swap the equation references: it attributes Q^{stu}_{t0} to Eq. (7) and Q^{teach}_{t0} to Eq. (6), while in the text Eq. (6) defines Q^{stu}_{t0} and Eq. (7) defines Q^{teach}_{t0}.","section":"Figure 3 caption"},{"comment":"There is a typo: 'epsilon is the regularization coefficient than controls' should be 'that controls'.","section":"Section 3.2"},{"comment":"The notation in Eqs. (6) and (7) is dimensionally unclear: M^{stu*} is introduced as an assignment matrix in R^{n_s x K}, but it is multiplied by Z^{stu}_{t0} in R^{n_s x d}, and in Eq. (9) it is treated as if it were the prototype matrix. Please clarify the exact matrix products and the roles of M and P.","section":"Section 3.3, Eqs. (6)-(9)"},{"comment":"The phrase 'time step per frame fixed at 3.2/12 seconds' is ambiguous; it should state the actual temporal stride in seconds between consecutive frames.","section":"Section 4.7, Table 6h"},{"comment":"The failure-case discussion is not reconciled with Eq. (10). As written, a drifted track that remains visible only changes the scalar visibility weight; it does not propagate a wrong cluster label to a later-frame destination feature, because the teacher label is frozen at t0. The authors should clarify how the illustrated tracker failure affects the actual loss under their formulation.","section":"Section 7.2, Figure 8"}],"recommendation":"major_revision","confidential_remarks":"The most serious issue is Eq. (10), which as written does not use any temporal features. This is a load-bearing problem for the paper's central claim. I recommend asking the authors to provide the actual training loss from their implementation or a corrected formulation. If the implementation matches the text, the contribution should be reframed as a visibility-weighted first-frame clustering objective, and the paper's empirical gains, though modest, may still be publishable. If the implementation is different, the equations must be corrected and the experimental section should include an ablation that isolates the temporal mechanism. The TimeT comparison should also be made apples-to-apples."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the stress-test note is right, and it is decisive. Eq. (10) as written cannot implement the paper's central claim. The student score S^{stu,k,i}_{t0} depends only on features from the first frame, and eq. (8) makes the teacher label constant along the track. So the loss is just a visibility-weighted first-frame clustering objective. The gradient with respect to features at t>0 is zero. The trajectory positions matter only through the visibility flag, not through any alignment of later-frame features. The paper's stated mechanism—propagating cluster assignments along motion trajectories to enforce temporal coherence—is not realized by the loss it defines.\n\nWhat the paper does well: the problem is real, the evaluation is thorough, and the comparison against DINOv2 with the same architecture and initialization is the right kind of control. The ablations over mask ratio, grid size, prototypes, and clip length are useful. Code is released. The writing is clear about the method's intent.\n\nThe soft spots beyond the main flaw: the headline video-segmentation gains over TimeT change both backbone initialization and patch size, so they are not apples-to-apples. There is no ablation that removes the trajectory-propagation component, which means we cannot tell whether the observed gains come from motion, from continued training on video, or from the EMA teacher and masking. The paper's own failure cases (the bird-beak track that ends on a bowl) show track drift is real, but because the label is frozen at t0, drift would mainly change weights rather than directly pollute destinations; that concern is secondary.\n\nNet: the empirical results may be genuine, but the paper as written does not support its central claim. The fix is straightforward—compute the student score at the tracked location in frame t, or use the teacher's later-frame features—so the method might work as intended after revision. I would send it to review, with a request that the authors either correct the loss or re-frame the contribution. If they fix it, this could be a solid paper. As is, it fails on its own terms.","headline":"The loss in eq. (10) does not supervise later frames, so the paper's central claim is unsupported; the empirical study is solid but needs a fix.","tokens_in":23790,"tokens_out":2335,"would_cite":false,"duration_ms":28096,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes that unlabeled video motion, channeled through point-track cluster propagation, can sharpen the dense features of a frozen-initialized image backbone such as DINOv2.","keywords":["self-supervised learning","dense representation learning","point tracking","optimal transport clustering","video object segmentation","DINOv2","temporal consistency","semantic segmentation"],"falsifier":"Measure, on a held-out video set, whether a propagated first-frame cluster assignment agrees with a fresh cluster assignment computed on the later frame's own features for the same track; if agreement is at chance for a substantial fraction of visible tracks, the cross-entropy signal is not carrying semantic identity.","tokens_in":22652,"feed_emoji":"🎥","tokens_out":4963,"duration_ms":54920,"temperature":0.7,"pith_summary":"The paper introduces MoSiC, a self-supervised method that turns point trajectories into a temporal supervisory signal for dense visual representations. It clusters first-frame features with an entropy-regularized optimal transport step, propagates those cluster assignments along tracked points across frames, and trains the student's first-frame cluster scores against the propagated teacher targets. The claim is that this motion-guided clustering improves dense representations without any labels, yielding 1 to 6 percent gains over DINOv2 across four benchmarks and 8.7 to 9.4 percent gains over TimeT on video segmentation. If correct, unlabeled video alone can improve downstream semantic segmentation, in-context scene understanding, and video object segmentation.","feed_headline":"Video motion alone lifts DINOv2 segmentation by up to 6%","feed_subtitle":"Unlabeled point tracks carry cluster labels across frames, sharpening dense features on four benchmarks.","key_machinery":"The load-bearing mechanism is trajectory-based cluster propagation. An off-the-shelf point tracker samples a grid of points in the first frame and produces long-range trajectories (eq. 1); teacher and student features are clustered into K prototypes via entropy-regularized optimal transport using Sinkhorn-Knopp; the teacher's first-frame hard assignments are copied along each trajectory (eq. 8); and the student is trained to reproduce those propagated assignments at the first frame for points that remain visible (eq. 10). The visibility flag restricts the loss to points that stay visible, which the ablations show is important for avoiding false similarity enforcement.","core_discovery":"The central claim is that motion can act as an implicit dense label: points that move together should keep the same cluster identity over time. Concretely, MoSiC computes Sinkhorn-Knopp optimal-transport cluster assignments for the first frame of a clip, propagates each tracked point's teacher assignment along its trajectory, and applies a cross-entropy loss (eq. 10) between the student's first-frame cluster scores and the propagated teacher one-hot targets, masked to visible points. The paper asserts that this mechanism mitigates occlusions and long-range tracking drift, and that training on YouTube-VOS consistently improves the dense features of a DINOv2-initialized ViT across four image and video benchmarks.","pith_inferences":["A testable extension is to verify track-object consistency by cycle consistency or mask agreement before propagating labels; the paper's failure cases suggest filtering bad tracks would add further gains.","If point trackers keep improving, tracker quality rather than clustering becomes the bottleneck; the paper's tracker ablation already points in this direction.","The same propagation objective could be applied to multi-object labels over longer clips, potentially replacing mask propagation in label-efficient video segmentation pipelines."],"forward_implications":["Temporal motion alone, without any dense annotations, can improve the dense features of a strong image-pretrained backbone.","Propagating cluster identities along tracks gives cleaner unsupervised video object segmentation than mask propagation, with 8.7 to 9.4 mIoU gains over TimeT.","In-context scene understanding improves most in low-data regimes, suggesting motion pretraining helps feature transfer.","The recipe transfers to other backbones, including vision-language encoders, and scales with model size.","End-to-end fine-tuning on videos alone can improve object detection and semantic segmentation over the image-only backbone."],"supporting_citations":[{"why":"Supplies the long-range point trajectories that define the supervisory signal for propagation.","marker":"[29]"},{"why":"Provides the strong image-pretrained backbone and the main baseline that MoSiC improves.","marker":"[42]"},{"why":"The closest prior video time-tuning method and the primary comparison for video segmentation and in-context learning.","marker":"[50]"},{"why":"Provides the Sinkhorn-Knopp algorithm used for entropy-regularized optimal transport clustering.","marker":"[15]"},{"why":"Defines the in-context scene understanding benchmark used for dense nearest-neighbor retrieval evaluation.","marker":"[5]"},{"why":"Supports the claim that simple masking and cropping augmentations suffice, guiding the choice of augmentations.","marker":"[41]"}],"fun_headline_variants":["Motion tracks as pseudo-labels for dense video SSL","Optimal transport on point tracks sharpens features","Unlabeled video motion improves dense features up to 6%","MoSiC: motion-guided clustering for self-supervised learning","Tracking points to propagate cluster assignments in video"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes the point tracker keeps each point attached to the same physical object across the clip, so a propagated cluster label is semantically correct; when a track drifts, the loss trains the student to merge or split features incorrectly.","fun_headline_variants_meta":{"raw":{"variants":["Motion tracks as pseudo-labels for dense video SSL","Optimal transport on point tracks sharpens features","Unlabeled video motion improves dense features up to 6%","MoSiC: motion-guided clustering for self-supervised learning","Tracking points to propagate cluster assignments in video"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000233,"raw_usage":{"total_tokens":1471,"prompt_tokens":898,"completion_tokens":573,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":496}},"tokens_in":514,"tokens_out":573,"duration_ms":6414,"temperature":1.0,"reasoning_tokens":496,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:04:45.769159+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure, on a held-out video set, whether a propagated first-frame cluster assignment agrees with a fresh cluster assignment computed on the later frame's own features for the same track; if agreement is at chance for a substantial fraction of visible tracks, the cross-entropy signal is not carrying semantic identity.","supporting_citations":[{"cited_title":"Dinov2: Learning robust visual features without supervision","cited_arxiv_id":null,"evidence_quote":"Provides the strong image-pretrained backbone and the main baseline that MoSiC improves."},{"cited_title":"Time does tell: Self-supervised time- tuning of dense image representations","cited_arxiv_id":null,"evidence_quote":"The closest prior video time-tuning method and the primary comparison for video segmentation and in-context learning."},{"cited_title":"Sinkhorn distances: Lightspeed computation of optimal transport","cited_arxiv_id":null,"evidence_quote":"Provides the Sinkhorn-Knopp algorithm used for entropy-regularized optimal transport clustering."},{"cited_title":"Towards in-context scene understanding","cited_arxiv_id":null,"evidence_quote":"Defines the in-context scene understanding benchmark used for dense nearest-neighbor retrieval evaluation."},{"cited_title":"You don’t need domain- specific data augmentations when scaling self-supervised learning","cited_arxiv_id":null,"evidence_quote":"Supports the claim that simple masking and cropping augmentations suffice, guiding the choice of augmentations."}],"review_version":1}