{"id":"350a528f-8c0a-4a9a-a2d2-29f1dbd37cd4","arxiv_id":"2411.19141","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"M3Former fuses appearance and motion streams in a Mask2Former-style transformer and, when trained on a diverse mix that includes KITTI and DAVIS train splits, reaches state-of-the-art motion segmentation scores on those benchmarks.","lead":"This paper introduces M3Former, a two-stream transformer architecture that fuses image appearance with optical flow, scene flow, or motion costs to segment independently moving objects from monocular video. It reports state-of-the-art results on KITTI and DAVIS when trained on a diverse mix of datasets, including those datasets' own training splits.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SOTA claim rests on non-comparable training mixes; even within Table 7, M3Former is below Raptor on DAVIS (37.07 vs 40.9 AP), so the central claim is not supported as stated.","rationale":"The reader identified the correct primary weakness: the SOTA comparison compares Mix-3-trained M3Former against zero-shot baselines trained on Mix 1, and under the zero-shot Mix 1 protocol M3Former is clearly below Raptor. I agree this is the most load-bearing concern. My reading adds a second, protocol-independent problem: in the paper's own Table 7, Raptor reaches DAVIS AP 40.9 while the best M3Former entry is 37.07, so even before questioning training mixes, the stated claim of SOTA on both KITTI and DAVIS is not supported by the reported numbers. This does not invalidate the paper's useful contributions: the systematic study of 2D/3D motion representations, the fusion-strategy ablations, and the negative-example augmentation are informative and appear technically sound. However, the central SOTA claim needs either retrained baselines under the same data protocol or a restricted claim limited to KITTI under Mix 3. Because the reader's conditional verdict already requires exactly this kind of corrective evidence, I do not recommend changing the verdict.","tokens_in":21425,"tokens_out":2997,"duration_ms":27248,"concrete_test":"Run a protocol-controlled benchmark: retrain Raptor and Generic MoSeg on Mix 3 with the same backbone, epochs, batch size, and augmentation schedule used for M3Former, and evaluate them on the KITTI and DAVIS test splits; also evaluate M3Former trained on Mix 1 only. The SOTA claim survives only if M3Former exceeds the retrained baselines under Mix 3 and remains competitive under the zero-shot Mix 1 protocol.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline claim is that M3Former 'achieves SotA performance on Kitti and Davis.' The load-bearing condition for that claim is an apples-to-apples comparison with prior methods under the same training and evaluation protocol. That condition is not met. M3Former's best results use Mix 3, which includes the train splits of KITTI and DAVIS, while Raptor and Generic MoSeg were evaluated zero-shot after training on Mix 1. The paper itself acknowledges this in §4.4: 'Our results are not necessarily surprising, as we partially trained on the target domain.' Under the zero-shot Mix 1 protocol in Table 7, M3Former RGB+OF scores KITTI AP 25.67 versus Raptor's 40.07, so the SOTA claim collapses unless baselines are retrained under Mix 3. Independently of protocol, the same Table 7 shows Raptor at DAVIS AP 40.9 while the best M3Former entry (RGB+SF, Mix 3) is 37.07; no M3Former row exceeds Raptor on DAVIS. Thus, even taking the reported numbers at face value, the abstract's 'SotA on Kitti and Davis' is not supported for DAVIS. The data-diversity analysis and negative-example augmentation remain useful, but they do not rescue the central comparative claim without controlled baselines.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes M3Former, a two-stream transformer architecture for monocular motion segmentation that fuses appearance features (RGB) with motion features obtained from frozen expert models (optical flow, scene flow, and higher-dimensional motion costs). The authors systematically analyze different motion representations, fusion mechanisms (encoder, decoder, and bottleneck-based), and a negative-example augmentation that prevents over-reliance on appearance. They also study the effect of training-data diversity by designing several dataset mixes and report instance-segmentation metrics on KITTI and DAVIS. The paper claims state-of-the-art (SotA) performance on both benchmarks, attributing this to the flexible attention mechanism and diverse training data.","tokens_in":21684,"tokens_out":5507,"duration_ms":43497,"significance":"If the SotA claim were fully supported, the paper would be a valuable contribution to monocular motion segmentation, particularly because it provides a clean comparison of 2D vs. 3D motion representations and a simple, effective negative-example augmentation. The systematic ablation of fusion locations and the analysis of training-data composition are useful and reproducible extensions of Mask2Former. However, the central empirical claim is undermined by a non-comparable evaluation protocol: the best M3Former results are obtained with a training mix that includes the target-domain training splits, whereas the comparison methods were evaluated zero-shot after training only on SceneFlow. The paper's own admission that it 'partially trained on the target domain' (§4.4) makes this limitation explicit. Because the headline claim is load-bearing and is not supported by an apples-to-apples comparison, the contribution reduces to an architecture study plus a data-diversity analysis unless the comparison is redone.","major_comments":[{"comment":"The state-of-the-art claim rests on a non-comparable training protocol. Table 7 reports Mix 3 results for M3Former (e.g., KITTI AP 52.27 with RGB+SF), but the comparison methods Raptor [44] and Generic MoSeg [18] were trained only on Mix 1 (SceneFlow) and evaluated zero-shot. The text at §4.4 explicitly admits: 'Our results are not necessarily surprising, as we partially trained on the target domain.' Under the zero-shot Mix 1 rows of Table 7, M3Former scores KITTI AP 25.67 (RGB+OF), 26.70 (RGB+SF), and 32.40 (RGB+Cost), all below Raptor's 40.07. To substantiate the SotA claim, the authors must either retrain the baselines under the Mix 3 protocol or present Mix 1 as the primary comparison; otherwise the improvement cannot be attributed to the architecture.","section":"Table 7 / §4.4"},{"comment":"Even taking the reported Mix 3 numbers at face value, the abstract's claim of 'SotA performance on Kitti and Davis' is not supported on DAVIS. In Table 7, Raptor attains DAVIS AP 40.9, while the best M3Former entry (RGB+SF, Mix 3) reaches only 37.07; no M3Former row exceeds Raptor's DAVIS AP. The claim should be restricted to KITTI, or the DAVIS comparison should be reworked with baselines retrained under the same protocol.","section":"Table 7"},{"comment":"The statement that 'we could not replicate the performance of [44] with the training setting of Mix 1' is a self-reported negative result that is not adequately investigated. Under identical Mix 1 data, Raptor's RGB + Cost input reaches KITTI AP 40.07, whereas the M3Former variant with the same input (RGB + Cost [44]) reaches only 32.40. The paper does not analyze whether this gap stems from the motion-cost computation, the fusion architecture, or training details. Since this is the only controlled comparison where both methods use the same modality and same training set, this gap is decisive for the claim that M3Former itself is competitive.","section":"§4.4, Mix 1 rows of Table 7"},{"comment":"The conclusion that 'diverse datasets are required to achieve SotA performance' is confounded with target-domain supervision. Mix 2 includes DAVIS train frames and Mix 3 includes both KITTI and DAVIS train frames (Table 2), so the large gains reported on those benchmarks from Mix 1 to Mix 2/3 may reflect label-space familiarity rather than diversity per se. To support the diversity claim, the authors should isolate the effect of adding datasets that are neither KITTI nor DAVIS (e.g., YTVOS, which they explicitly exclude), or ablate Mix 3 with the target-domain splits removed.","section":"Table 2 and §4.4"}],"minor_comments":[{"comment":"There is an incomplete sentence in the paragraph on Multi-modal Alignment: '...the model needs to figure out to rely on the' — the sentence ends mid-thought. It should be completed accordingly.","section":"§3.2"},{"comment":"Typographical errors: 'adress' should be 'address' (§2) and 'indepent' should be 'independent' (§3.1).","section":"§2 and §3.1"},{"comment":"Table 5 reports raw FP/FN counts and notes they are not normalized, while Tables 6 and 7 report per-frame normalized values. Using different conventions without a clear cross-reference is confusing; state the convention explicitly in each caption or use a single convention throughout.","section":"Table 5 vs. Tables 6–7"},{"comment":"The caption refers to 'Colored datasets are used for evaluation,' but in monochrome the table does not indicate which datasets are colored. Name the evaluation datasets explicitly (KITTI and DAVIS) in the caption.","section":"Table 2"},{"comment":"The abstract states 'SotA performance on Kitti and Davis,' while the conclusion words the claim as 'our approach achieves SotA performance by leveraging the flexible attention mechanism and diverse training data.' The discrepancy in specificity should be resolved, especially in light of the protocol issues discussed above.","section":"Abstract / Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The reader's stress-test concern about the non-comparable training mixes is confirmed by the manuscript itself: §4.4 admits partial training on the target domain, and Table 7 shows that the Mix 1 zero-shot results are below Raptor. The abstract's 'SotA on Kitti and Davis' is therefore not supportable as written. The paper has useful components—the motion-representation ablation, the negative-example augmentation, and the data-mix analysis—but the central comparative claim needs to be either re-proven with controlled baselines or restated. I recommend major revision rather than rejection, because the issues are fixable by retraining baselines or substantially revising the claims. I saw no evidence of fabrication or misconduct, but the evaluation protocol as reported is not sufficient for the stated headline."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: useful empirical paper, but the headline SOTA claim is not supported by the numbers as reported. The comparison protocol is the problem, and the paper itself admits it.\n\nThe genuinely new pieces are the two-stream Mask2Former-style fusion, the systematic comparison of optical flow, scene flow and geometric cost embeddings, and the negative-example augmentation. The augmentation is simple and Table 5 shows it has a large effect. The data-diversity analysis (Mix 1 vs 2 vs 3) is the most usable contribution; it makes the point that many failure modes are training-data problems rather than architecture problems, and that point comes through clearly.\n\nThe soft spot is load-bearing. The best results use Mix 3, which includes the train splits of KITTI and DAVIS, while prior methods (Raptor, Generic MoSeg) were trained only on SceneFlow and evaluated zero-shot. Under the zero-shot Mix 1 condition, M3Former RGB+OF gets KITTI AP 25.67 versus Raptor 40.07. That is not a small gap. The paper's own sentence in Section 4.4 — \"Our results are not necessarily surprising, as we partially trained on the target domain\" — is the honest version, and it should be in the abstract. The abstract and conclusion still claim SOTA on KITTI and DAVIS, and that claim is not supported on DAVIS even taking Table 7 at face value: the best M3Former row (RGB+SF, Mix 3) is 37.07 AP, below Raptor's 40.9. No M3Former row exceeds Raptor on DAVIS.\n\nSo the fix is not a tweak; the SOTA claim needs to be downgraded or supported by retraining the baselines under the same data mix. That is a well-defined, doable experiment. The paper would also be stronger with released code and run-to-run variance, especially because the fusion-strategy and pneg ablations are small-sample and could shift with training time.\n\nI do not think this is sloppy work. The body text is honest about the target-domain training, and the data-mix analysis is done with care. The problem is that the packaging overstates what the evidence shows.\n\nI would send this to peer review, not desk reject. The architecture and the data-diversity analysis deserve referee time, and the SOTA comparison can be repaired. I would cite it for the negative-example augmentation and the data-mix discussion, not for the SOTA claim.","headline":"Useful transformer fusion and data-mix analysis for motion segmentation, but the SOTA claim rests on a training-mix mismatch and fails on DAVIS.","tokens_in":22226,"tokens_out":3297,"would_cite":true,"duration_ms":27876,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"M3Former, a two-stream transformer that fuses appearance and motion features through attention, claims state-of-the-art moving-object segmentation on KITTI and DAVIS when trained on a diverse mix of datasets.","keywords":["motion segmentation","moving object detection","monocular video","transformer","multi-modal fusion","optical flow","scene flow","training-data diversity"],"falsifier":"Retrain M3Former under the identical zero-shot protocol of the prior work—only the SceneFlow-based Mix 1 training set—and compare KITTI average precision against Raptor's reported 40.07; the paper already reports 25.67 for this setting, so a gap in either direction would settle whether the architecture itself is state of the art or whether the diverse training mix is doing the work.","tokens_in":21189,"feed_emoji":"🎥","tokens_out":8783,"duration_ms":81998,"temperature":0.7,"pith_summary":"Segmenting every independently moving object in a video from a single moving camera is treated here as a multi-modal fusion problem: the model must combine what objects look like with how they move. The paper proposes M3Former, a two-stream transformer in which an appearance branch and a motion branch exchange information through attention, and reports state-of-the-art results on KITTI and DAVIS when the model is trained on a diverse mix of datasets. The paper's central empirical claim is that training-data diversity, not the architecture, is what fixes the common failure modes—degenerate colinear motion, non-rigid objects, and objects that could move but do not. A second claim is that 3D motion representations such as scene flow are most informative when depth is accurate, yet with enough diverse data even 2D optical flow reaches comparable performance.","feed_headline":"Transformer that fuses motion and appearance tops KITTI and DAVIS","feed_subtitle":"Attention lets a single model weigh image context against noisy motion maps, if training data is diverse.","key_machinery":"The carrying mechanism is the two-stream attention fusion inside M3Former, a Multi-Modal Mask2Former built on the Mask2Former architecture. An appearance branch and a motion branch each use a ResNet-50 backbone, a multi-scale deformable-attention encoder, and a masked-attention decoder with 100 object queries; fusion happens by concatenating feature maps, query embeddings, and learned attention masks so that attention can flow between the streams, and a $1\\times1$ convolution merges the mask and class outputs of all queries. A second load-bearing mechanism is the negative-example augmentation: with probability $p_{\\mathrm{neg}}$, the motion input is replaced by a constant flow field, which forces the network not to over-rely on appearance data. The input motion representations are computed by frozen expert models—RAFT optical flow, RAFT-3D scene flow, DPT depth—so that the architecture consumes pseudo-modalities rather than raw video.","core_discovery":"On the paper's own terms, the central claim is that motion segmentation should be formulated as a fusion problem and that a two-stream transformer with dedicated parameters for appearance and motion—M3Former—solves it better than prior CNN-based fusion. The two branches produce multi-scale features and separate sets of object queries; at fusion points the feature maps, queries, and learned attention masks are concatenated, so attention can decide freely which modality to trust, and a $1\\times1$ convolution merges the final predictions. Systematic ablations show that no single fusion strategy wins on all datasets or modalities, that scene flow—a per-pixel rigid-body motion field—carries the most information when depth quality is sufficient, and that the largest improvement comes from training on several datasets with diverse motion patterns and semantic classes. The paper's stated result is state-of-the-art average precision on KITTI and DAVIS for the fusion model trained on its most diverse data mix.","pith_inferences":["The headline state-of-the-art claim is protocol-dependent: under the zero-shot Mix 1 protocol the paper reports KITTI AP 25.67 against Raptor's 40.07, so the published comparison mixes training protocols; a matched-protocol benchmark would settle how much of the gain is architecture versus training data.","The dataset-mix study suggests a practical recipe many practitioners could adopt: combine synthetic scene-flow data, real driving data, and casual video, sample datasets roughly equally, and set the negative-example probability near 0.3 for small mixes and near 0.05 for large ones.","If segmentation is indeed driven mostly by the appearance stream, a much lighter motion branch would likely suffice; the paper hints at this but keeps both streams equal in size.","A quantitative evaluation on held-out camouflage datasets (the paper shows only qualitative MoCA examples) would directly test the claim that the model discovers truly generic moving objects, not just familiar classes."],"forward_implications":["A practical motion-segmentation system can be assembled from an off-the-shelf appearance transformer, a motion branch, and a diverse training mix, without handcrafted geometric criteria.","Training-data construction and balancing become part of the method itself: adding real driving data and casual video with non-rigid motion removes failure modes that architecture changes cannot.","The negative-example augmentation is a cheap, transferable regularizer for any fusion setting where one modality is noisy or sometimes absent.","With enough diverse data, 2D optical flow suffices for competitive results on driving scenes; 3D scene flow only pays off when depth maps are reliable and scale-accurate.","Because fusion happens through attention, the same two-stream design extends to longer temporal windows or additional modalities by adding streams."],"supporting_citations":[{"why":"Provides the Mask2Former base architecture that both streams are built on, including masked attention, queries, and decoder.","marker":"[14]"},{"why":"Defines the main SOTA baseline (Raptor) and the three-frame motion-cost embedding; also supplies the Mix 1 zero-shot training protocol the paper must beat.","marker":"[44]"},{"why":"Defines the four-criteria motion cost, the Mix 1 dataset protocol, and the foreground/background precision metric used in comparisons.","marker":"[76]"},{"why":"Establishes the generic motion-segmentation task and evaluation metrics (precision, recall, F-score) and the Mix 4 training setting with KITTI and DAVIS.","marker":"[18]"},{"why":"RAFT optical flow is the frozen expert model that produces the 2D motion representation for the motion branch.","marker":"[54]"},{"why":"RAFT-3D is the frozen expert model that produces per-pixel SE(3) scene flow for the 3D motion ablation.","marker":"[56]"},{"why":"DPT monocular depth estimation supplies the depth needed to compute scene flow in the wild and for the SLAM-based alignment experiments.","marker":"[47]"},{"why":"Provides the attention-bottleneck fusion variant that the paper ablates as an alternative to naive decoder and encoder-decoder fusion.","marker":"[43]"}],"fun_headline_variants":["M3Former fuses motion and appearance for SotA monocular segmentation","Transformer fusion wins on KITTI and DAVIS with diverse training","Attention weighs motion maps: M3Former tops segmentation benchmarks","Diverse data drives M3Former to state-of-the-art motion segmentation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The state-of-the-art comparison is fair even though the new model's training data includes the training splits of the very KITTI and DAVIS sets it is evaluated on, while the prior methods it is compared with (Raptor, Generic MoSeg) were evaluated without seeing those datasets.","fun_headline_variants_meta":{"raw":{"variants":["M3Former fuses motion and appearance for SotA monocular segmentation","Transformer fusion wins on KITTI and DAVIS with diverse training","Attention weighs motion maps: M3Former tops segmentation benchmarks","Diverse data drives M3Former to state-of-the-art motion segmentation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000714,"raw_usage":{"total_tokens":3167,"prompt_tokens":856,"completion_tokens":2311,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":472,"completion_tokens_details":{"reasoning_tokens":2234}},"tokens_in":472,"tokens_out":2311,"duration_ms":16036,"temperature":1.0,"reasoning_tokens":2234,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:29:07.221603+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain M3Former under the identical zero-shot protocol of the prior work—only the SceneFlow-based Mix 1 training set—and compare KITTI average precision against Raptor's reported 40.07; the paper already reports 25.67 for this setting, so a gap in either direction would settle whether the architecture itself is state of the art or whether the diverse training mix is doing the work.","supporting_citations":[{"cited_title":"Monocular arbitrary moving object discovery and segmentation","cited_arxiv_id":null,"evidence_quote":"Defines the main SOTA baseline (Raptor) and the three-frame motion-cost embedding; also supplies the Mix 1 zero-shot training protocol the paper must beat."},{"cited_title":"Learning to seg- ment rigid motions from two frames","cited_arxiv_id":null,"evidence_quote":"Defines the four-criteria motion cost, the Mix 1 dataset protocol, and the foreground/background precision metric used in comparisons."},{"cited_title":"Raft-3d: Scene flow using rigid- motion embeddings","cited_arxiv_id":null,"evidence_quote":"RAFT-3D is the frozen expert model that produces per-pixel SE(3) scene flow for the 3D motion ablation."},{"cited_title":"Vi- sion transformers for dense prediction","cited_arxiv_id":null,"evidence_quote":"DPT monocular depth estimation supplies the depth needed to compute scene flow in the wild and for the SLAM-based alignment experiments."},{"cited_title":"Attention bottlenecks for multimodal fusion","cited_arxiv_id":null,"evidence_quote":"Provides the attention-bottleneck fusion variant that the paper ablates as an alternative to naive decoder and encoder-decoder fusion."}],"review_version":1}