{"id":"04ab2550-6d4e-4749-ac8a-7dfb1a876b15","arxiv_id":"2505.21635","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Motion-based pseudo-labels from optical flow clustering, used with contrastive pretraining, produce visual features that transfer well to depth, 3D detection, and occupancy tasks.","lead":"The paper trains visual models on pairs of pixels grouped by optical flow, so that object-like motion in videos becomes the supervision signal for learning image features. The resulting representations beat several supervised and self-supervised baselines on depth, 3D detection, and occupancy prediction, suggesting motion can substitute for manual labels.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Motion-boundary pseudo-labels are never validated against any ground-truth instance boundary, so the claimed emergence of object concepts could be an artifact of the clustering thresholds and flow noise rather than genuine object grouping.","rationale":"The reader correctly identified the weakest assumption: the link between motion boundaries and object boundaries, and the lack of validation of the clustering thresholds. My analysis agrees that this is the single most load-bearing concern. The paper's central claim depends entirely on the validity of the pseudo-labels. If the pseudo-labels are noisy or dominated by non-object boundaries, the supervised signal is arbitrary and the claimed object-instance emergence is not demonstrated. However, the existence of some qualitative evidence and the empirical gains on downstream tasks suggest the method may still be effective, but the lack of quantitative validation of the pseudo-labels leaves the central mechanism unverified. Thus, a CONDITIONAL verdict is appropriate, as the paper needs to provide this validation or soften its claims. I do not see evidence of internal inconsistency, but the unproven assumption is critical. The concrete test I propose would directly settle whether the pseudo-labels capture object boundaries, which is the crux of the central claim.","tokens_in":13996,"tokens_out":755,"duration_ms":10390,"concrete_test":"Run the BFS clustering pipeline on a held-out set of video frames with ground-truth instance annotations, such as the KITTI instance segmentation subset or the MOTS data, and compute standard metrics (e.g., boundary IoU or instance-level segmentation accuracy) between the pseudo-labels and ground-truth object masks, across a sweep of θf and θs values. If the best pseudo-label quality is poor (e.g., boundary IoU near chance or only a small fraction of objects recovered), the claimed emergence of object concepts from motion is not supported by the data.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that motion boundaries provide an unsupervised supervisory signal yielding object-instance features. The training signal is generated by BFS-clustering optical flow (Sec. 3.2, thresholds θf=1.5, θs=100) with the assumption that flow discontinuities correspond to object boundaries (Sec. 3.1, Eq. 3). Eq. (3) shows flow gradients follow depth gradients, but it does not establish that depth discontinuities correspond to object boundaries; the link is an unproven domain assumption. The authors themselves note (Sec. 5.2) that performance is greatly limited by optical flow quality, and the pseudo-labels exhibit under-segmentation due to weak motion cues or flow errors (Sec. 3.2). No quantitative evaluation of the pseudo-label quality is provided: the θf and θs thresholds are fixed without validation against any ground-truth boundary or instance mask. The loss (Eq. 4) is designed so that background pairs are ignored and only pixels with pseudo-labels are pulled together, but if the clusters are dominated by parallax regions, texture edges, or flow noise, the contrastive objective would reinforce these artifacts instead of object instances. The paper's own qualitative results show that the model labels many objects not annotated in the pseudo-labels (Sec. 4.2), which is presented as generalization, but this could equally indicate that the pseudo-labels themselves are noisy and the model learns something other than the intended instance-level grouping. Because the entire pipeline's efficacy hinges on the validity of the pseudo-labels, the lack of any direct validation of the pseudo-label generation step makes the central claim that object concepts emerge from motion unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a self-supervised pretraining method for visual encoders that uses motion information from raw video. It generates pseudo instance masks by applying BFS clustering to optical flow with thresholds theta_f = 1.5 and theta_s = 100, on about 2,700 hours of driving video from OpenDV-YouTube and nuPlan. The resulting masks are used to train Swin and ResNet backbones with a contrastive loss: pixels in the same motion cluster are pulled together, pixels in different clusters are pushed apart, and background-background pairs are ignored. The method is evaluated on monocular depth estimation on KITTI, 3D object detection on nuScenes, and 3D occupancy prediction on nuScenes, with comparisons against ImageNet-22K supervised pretraining, SimMIM, DINOv2, and other baselines. The central claim is that motion boundaries provide an unsupervised supervisory signal that yields object-instance features, and that models trained this way outperform prior supervised and self-supervised baselines across model sizes.","tokens_in":14300,"tokens_out":5974,"duration_ms":66075,"significance":"If the central claim were fully substantiated, this would be a valuable contribution: a scalable, calibration-free, label-free alternative to existing visual foundation models that captures instance-level structure, with evidence across low-level (depth) and high-level (detection, occupancy) tasks. The paper's pipeline is simple and reproducible, uses a large amount of unlabeled video, and the experiments demonstrate measurable gains in several configurations, including consistent improvements over ImageNet-22K pretraining for most backbones. The complementarity with DINO features shown in Table 5 is also a useful empirical finding. However, the current significance is limited by two gaps: the motion-to-object-boundary link is asserted rather than validated quantitatively, and the paper's universal superiority claim is contradicted by some of its own results. Because the core value depends on whether the pseudo-labels actually correspond to object instances, the missing validation is load-bearing.","major_comments":[{"comment":"Equation (3) is a chain-rule identity showing that flow gradients follow depth gradients under rigid motion, but it does not establish that depth discontinuities are object boundaries. Depth changes also occur at surface orientation changes, occlusion edges, and non-object scene structures such as walls or ground-plane boundaries. The paper's central mechanism therefore rests on an unproven domain assumption. This would be acceptable if the pseudo-labels were quantitatively validated against ground-truth instance masks, but no precision, recall, boundary IoU, or similar measure is reported. The qualitative examples in Fig. 3 and the acknowledgment of under-segmentation in Sec. 3.2 do not fill this gap. I request a quantitative evaluation of pseudo-label quality on a labeled subset (e.g., KITTI or nuScenes annotations), and an ablation comparing motion-based clustering against a depth-discontinuity baseline to isolate whether the gains come from object instances or from generic depth edges.","section":"§3.1 and §3.2"},{"comment":"The claim that models 'outperform previous supervised and self-supervised baselines' is not supported by the full set of results. In Table 1, SimMIM (Swin-L, window 12) achieves SILog 6.542 while Ours (Swin-L, window 7) achieves 6.558, so the baseline wins on the primary depth metric. In Table 6, DINOv2 (ViT-L) achieves RayIoU 39.0 while Ours (Swin-L) achieves 38.7. In Table 4, DINOv2 (ViT-L) achieves mAP 42.05 versus Ours (Swin-L) 41.79. These are not isolated footnotes; they directly bear on the abstract's claim that the method offers 'a compelling alternative to existing vision foundation models.' The text should be revised to say that the method is competitive and often better in specific settings, and the cases where baselines are stronger should be analyzed rather than omitted.","section":"Abstract, Tables 1–6"},{"comment":"The baseline comparison is not controlled for model capacity and context. Section 4.1 states: 'All Swin models use a window size of 7, while the B and L variants of SimMIM and Semantic-SAM used for comparison adopt a larger window size of 12.' A larger window gives the baseline more spatial context, which is usually beneficial. Therefore, the fact that Ours is worse than SimMIM on Table 1 may partly reflect this design choice rather than the pretraining method alone. Conversely, some claimed wins over SimMIM could be due to other differences in training setup. To support the 'outperforms' claim, the authors should either report SimMIM and Semantic-SAM with window size 7, or clearly separate the effect of the pretraining objective from the effect of window size, for example by training Ours with window 12 as well.","section":"§4.1, Tables 1 and 6"},{"comment":"The training signal could reinforce flow clustering artifacts rather than object instances. The BFS clustering uses fixed thresholds θf=1.5 and θs=100, which are not validated against any ground truth. The loss in Eq. (4) pulls together all pixels in the same cluster, including pixels that may belong to a parallax region, a texture edge, or a flow estimation error. Furthermore, the rule that 'the label with the largest pixel count is background' is an assumption: if a large object fills the frame, its pixels would be ignored as background. The qualitative evidence in Sec. 4.2 that the model distinguishes objects 'not annotated in the pseudo-labels' is presented as generalization, but it could equally indicate that the pseudo-labels themselves are noisy and the model learns some other grouping. This alternative explanation should be addressed explicitly, for example by measuring whether the learned features separate instances better than the pseudo-labels do, or by ablating the clustering thresholds and reporting pseudo-label quality versus downstream performance.","section":"§3.2, §4.2, Eq. (4)"}],"minor_comments":[{"comment":"The notation in Eq. (3) is imprecise: ∇F is a 2x2 Jacobian (or a vector of two component gradients), while dφ/dD is not defined as a matrix. The chain rule should be written with an explicit Jacobian so that the dimensional relationship is clear.","section":"§3.1, Eq. (3)"},{"comment":"The statement 'we are the first to leverage motion as an unsupervised supervisory signal' is too strong, since the paper itself cites earlier work on motion- and flow-guided object discovery, such as [49] and [59]. The novelty claim should be qualified to reflect the specific combination of simple BFS clustering, contrastive learning, and large-scale driving video.","section":"§1 and §2.1"},{"comment":"The sentence 'We retained all samples with at least two pseudo-label(i.e. at least one foreground cluster)' contains a typo and a potentially confusing parenthetical; the writing should be cleaned up.","section":"§3.2"},{"comment":"The paper calls the framework 'fully label-free,' but it relies on VideoFlow, an optical flow model that is trained with supervision on labeled synthetic data. The label-free claim should be scoped to the downstream training data or the pseudo-label generation process, not extended to the entire pipeline.","section":"§3.2 and §4.1"},{"comment":"In Table 1, the claim that 'our model consistently outperforms both supervised ImageNet-22K pretraining and models pretrained on the Semantic-SAM' is technically true for the rows shown, but the table also includes SimMIM, which outperforms Ours in the Swin-L row. The sentence should be rephrased to acknowledge the SimMIM comparison.","section":"§4.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's central value is a simple and reproducible pretraining recipe, not a rigorously established conceptual claim. I would advise asking the authors to either provide quantitative validation of the pseudo-labels or to substantially weaken the 'object concepts emerge' framing. The window-size asymmetry in the baseline comparison should also be addressed before the paper can be accepted, as it directly affects the fairness of the headline comparisons."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful thing here is the recipe: optical flow plus BFS clustering gives instance-ish pseudo-masks, and contrastive pretraining on those masks beats ImageNet-22K on the same backbones across depth, 3D detection, and occupancy. That is a real, reproducible empirical contribution, and the simplicity is a plus. The paper is also honest about its limits — it admits flow quality caps performance and that pseudo-labels under-segment.\n\nThe soft spots are real but not fatal. The biggest is that the pseudo-labels are never validated against any ground-truth boundary or instance mask. The thresholds θf=1.5 and θs=100 are fixed without ablations, and the geometric \"derivation\" in Eq. 3 is just a chain rule showing flow gradients follow depth gradients — it doesn't establish that depth discontinuities are object boundaries. So the central claim that \"object concepts emerge\" is more asserted than shown. That said, the downstream gains suggest the motion-grouping signal is useful even if it isn't clean objecthood.\n\nThe abstract overreaches. The paper claims to outperform supervised and self-supervised baselines, but Table 1 has SimMIM (Swin-L, window 12) beating Ours (Swin-L, window 7) on SILog, and Table 6 has DINOv2 (ViT-L) beating Ours on RayIoU. The window-size confound makes those comparisons unfair in both directions, but the blanket claim is not supported. Also, \"first to leverage motion as an unsupervised supervisory signal\" ignores Pathak et al. and other motion-supervised representation learning; the novelty is the simple pipeline, not the concept.\n\nThe lack of code and error bars is minor but makes the gains harder to trust. The qualitative out-of-domain generalization is nice but only suggestive.\n\nOverall: the empirical core is worth taking seriously and the paper deserves a proper peer review, but it needs controlled baselines, a pseudo-label quality analysis, and softer claims before I'd accept it. I'd cite it as a motion-pretraining baseline; I'd also bring it to a reading group for the discussion on what counts as \"object\" supervision.","headline":"A genuinely simple and reproducible motion-pretraining recipe with consistent but modest gains, whose headline claims outrun the evidence — worth reviewing seriously, but needs controlled baselines and validation of the pseudo-labels.","tokens_in":14886,"tokens_out":1972,"would_cite":true,"duration_ms":23807,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Motion boundaries can serve as a label-free supervisory signal for learning instance-level visual representations.","keywords":["unsupervised representation learning","motion boundary","object-centric representations","optical flow clustering","contrastive learning","visual instance","monocular depth estimation","3D object detection"],"falsifier":"Take a held-out subset of the pretraining videos, run the BFS clustering, and compare every pseudo-mask against human instance-segmentation annotations: if boundary recall is low, or if clusters systematically split along texture or parallax gradients rather than object boundaries, the central motion-boundary premise is not doing the claimed work.","tokens_in":1586,"feed_emoji":"🎥","tokens_out":1609,"duration_ms":61335,"temperature":0.7,"pith_summary":"This paper tries to show that object concepts can be learned without any labels by watching motion. The authors generate pseudo instance masks from optical flow clustering on roughly 2,700 hours of unlabeled driving video, then use these masks in contrastive learning to train single-image encoders. They report that this motion-based pretraining outperforms supervised ImageNet-22K and self-supervised baselines on monocular depth estimation, 3D object detection, and occupancy prediction. If true, motion would provide a scalable, calibration-free route to instance-level visual representations, complementing the category-level features of existing foundation models.","feed_headline":"Motion boundaries can teach vision models object concepts","feed_subtitle":"Flow-based pseudo-labels beat ImageNet-22K pretraining on depth, 3D detection, and occupancy.","key_machinery":"The central machinery is a BFS pixel-clustering algorithm that converts optical flow into pseudo instance masks: for each pixel passing a forward-backward consistency check, neighboring pixels whose flow vectors differ by less than $\\theta_f = 1.5$ are grouped, and clusters smaller than $\\theta_s = 100$ pixels are discarded. These masks feed a contrastive loss that pulls features from the same mask together, pushes features from different masks apart with margin $m = 1.0$, and ignores background-background pairs so incomplete pseudo-labels do not corrupt training. The loss turns raw motion discontinuities into a training signal for a single-image encoder built on a backbone plus a feature pyramid network.","core_discovery":"The paper claims that motion boundaries provide an unsupervised supervisory signal for learning features that capture the visual instance: a distinct, coherent object entity rather than merely a semantic category. Under rigid scene motion, the spatial gradient of the optical flow field is proportional to the gradient of depth, so flow discontinuities serve as proxies for depth discontinuities and hence object boundaries. The authors implement this by clustering optical flow with a simple breadth-first search, obtaining pseudo instance masks without human labels or camera calibration, and then training a contrastive objective on those masks. Across monocular depth, 3D object detection, and 3D occupancy prediction, the resulting features outperform supervised and self-supervised pretraining baselines, generalize to unseen indoor scenes, and are complementary to DINOv2 features when fused.","pith_inferences":["A direct test of the core premise would be to evaluate the pseudo-masks against ground-truth instance boundaries on a held-out video subset; if boundary recall is low or clusters systematically follow parallax or texture gradients, the motion-boundary assumption is not doing the claimed work.","If optical flow quality improves through synthetic-data pretraining, the same motion-mask pipeline should scale to egocentric and unconstrained web video, where the paper reports current flow limits the approach.","The reported complementarity with DINOv2 suggests a joint pretraining objective combining a motion-mask contrastive term with a self-distillation term could produce a single backbone with both instance-level and category-level abstractions.","A controlled pretraining experiment on static-camera video with only independently moving objects would isolate whether moving-object boundaries, rather than depth parallax from ego-motion, drive the learned object concepts."],"forward_implications":["A single-image encoder pretrained only on motion-derived masks can replace supervised ImageNet-22K pretraining on monocular depth, 3D detection, and occupancy prediction, across both ResNet and Swin backbones.","Features learned from motion are complementary to category-level features from DINOv2; concatenating them improves depth estimation beyond either representation alone.","Because the pipeline needs no camera calibration, it can be applied to any large, unstructured video collection without annotation.","Although trained on outdoor driving video, the features segment objects in unseen indoor, egocentric, and robot-manipulation scenes, suggesting transfer of object structure rather than memorized appearances.","Instance-level features benefit both low-level tasks like depth and high-level tasks like 3D detection, indicating that object composition is a useful shared abstraction."],"supporting_citations":[{"why":"Supplies the developmental-neuroscience motivation that infants acquire object unity through observed motion.","marker":"[27]"},{"why":"Provides the optical flow estimates that are the raw material for pseudo-mask clustering.","marker":"[47]"},{"why":"Supplies the OpenDV-YouTube unlabeled driving video corpus used for pretraining.","marker":"[65]"},{"why":"Supplies the nuPlan multi-camera driving dataset merged with OpenDV-YouTube for pretraining.","marker":"[20]"},{"why":"Serves as the downstream monocular depth estimation framework in which pretrained features are evaluated.","marker":"[55]"},{"why":"Serves as the downstream 3D object detection framework in which pretrained features are evaluated.","marker":"[64]"},{"why":"Serves as the downstream 3D occupancy prediction framework in which pretrained features are evaluated.","marker":"[33]"},{"why":"DINOv2 is a baseline and fusion partner that provides category-level features shown to be complementary to the proposed motion-learned instance features.","marker":"[40]"},{"why":"SimMIM is a self-supervised masked-image-modeling baseline that the proposed method is compared against on depth and detection.","marker":"[62]"}],"fun_headline_variants":["Unsupervised motion boundaries beat supervised pretraining","Flow boundaries alone teach vision models object concepts","Motion edges unlock object-level features without labels","From motion to object concepts, no labels needed","Object concepts arise from motion, outperforming ImageNet"],"cache_read_input_tokens":16896,"weakest_assumption_plain":"The whole method depends on the assumption that where optical flow changes sharply, an object boundary is present; if that link fails, the pseudo-masks teach the model about parallax, texture edges, or flow noise instead of objects.","fun_headline_variants_meta":{"raw":{"variants":["Unsupervised motion boundaries beat supervised pretraining","Flow boundaries alone teach vision models object concepts","Motion edges unlock object-level features without labels","From motion to object concepts, no labels needed","Object concepts arise from motion, outperforming ImageNet"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000188,"raw_usage":{"total_tokens":1308,"prompt_tokens":899,"completion_tokens":409,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":515,"completion_tokens_details":{"reasoning_tokens":339}},"tokens_in":515,"tokens_out":409,"duration_ms":4990,"temperature":1.0,"reasoning_tokens":339,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:26:47.572829+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out subset of the pretraining videos, run the BFS clustering, and compare every pseudo-mask against human instance-segmentation annotations: if boundary recall is low, or if clusters systematically split along texture or parallax gradients rather than object boundaries, the central motion-boundary premise is not doing the claimed work.","supporting_citations":[{"cited_title":"VideoFlow: Exploiting temporal cues for multi-frame optical flow estimation","cited_arxiv_id":null,"evidence_quote":"Provides the optical flow estimates that are the raw material for pseudo-mask clustering."},{"cited_title":"Generalized predictive model for autonomous driving","cited_arxiv_id":null,"evidence_quote":"Supplies the OpenDV-YouTube unlabeled driving video corpus used for pretraining."},{"cited_title":"Perception of object unity in young infants.Infant Behavior and Development, 11(2):161–180, 1983","cited_arxiv_id":null,"evidence_quote":"Supplies the developmental-neuroscience motivation that infants acquire object unity through observed motion."},{"cited_title":"DCDepth: Progressive monocular depth estimation in discrete cosine domain","cited_arxiv_id":null,"evidence_quote":"Serves as the downstream monocular depth estimation framework in which pretrained features are evaluated."},{"cited_title":"BEVFormer v2: Adapting modern image backbones to bird’s-eye- view recognition via perspective supervision","cited_arxiv_id":null,"evidence_quote":"Serves as the downstream 3D object detection framework in which pretrained features are evaluated."},{"cited_title":"DINOv2: Learning robust visual features without supervision.TMLR, 2023","cited_arxiv_id":null,"evidence_quote":"DINOv2 is a baseline and fusion partner that provides category-level features shown to be complementary to the proposed motion-learned instance features."},{"cited_title":"SimMIM: A simple framework for masked image modeling","cited_arxiv_id":null,"evidence_quote":"SimMIM is a self-supervised masked-image-modeling baseline that the proposed method is compared against on depth and detection."}],"review_version":1}