{"id":"62f6d4b6-7b28-4512-bf91-df3a72803925","arxiv_id":"2412.16876","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"MAGIC++ trains a semantic segmentation backbone with all available sensors and then uses the plain backbone at test time, reporting strong average results on arbitrary sensor combinations.","lead":"This paper introduces MAGIC++, a training framework that aims to make semantic segmentation work with any subset of sensors, such as RGB, depth, LiDAR, or events. It reports large average gains on modality-agnostic benchmarks, but the test-time procedure is unspecified and the method is not state-of-the-art when all sensors are present.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The inference protocol for arbitrary modality subsets is never specified: MIM/MASM are training-only (Sec. III-C), so at test time no fusion mechanism exists and the reported MaSS gains may be an ensemble effect.","rationale":"The reader's weakest_assumption and my stress-test converge on the same load-bearing gap: the test-time aggregation rule for arbitrary modality subsets is absent. This is not a minor omission because the described system removes every fusion component at inference, leaving no defined way to combine multiple modalities. Without this rule, the headline numbers cannot be reproduced, the comparison to fusion baselines is uncontrolled, and the mechanism claimed by the paper is not actually exercised during evaluation. The concern lands, and it is sufficient to support the reader's REJECT verdict; I see no reason to change that verdict.","tokens_in":20662,"tokens_out":3365,"duration_ms":55382,"concrete_test":"Ask the authors to release the exact inference procedure, or independently implement the following check: train per-modality SegFormer-B0 models with only LM on DELIVER, then fuse at test time by averaging softmax predictions over the available modalities. If this reproduces the reported mean MaSS of 47.74% (or the +7.25% margin over MAGIC), the improvement is an ensemble effect and the claimed fusion mechanism is unsubstantiated. Alternatively, verify whether the official code feeds each modality through the backbone and averages predictions; either way the protocol must be stated in the paper.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is state-of-the-art MaSS performance under arbitrary modality combinations. The method as described cannot support this claim because the only fusion/selection modules, MIM and MASM (Sec. III-C), are stated to be 'only utilized in training while the inference is achieved by the backbone model, i.e., SegFormer.' At test time, for a subset such as {R,D}, the paper does not specify whether (i) each available modality is passed independently through the shared backbone and the resulting softmax maps are averaged, (ii) features are averaged before the segmentation head, (iii) one modality is selected, or (iv) modalities are concatenated as input to the backbone. None of these operations is described or justified. If (i) is used, the large gains shown in Tables I-III are at least partly an ensemble effect and the comparison against fusion baselines such as CMNeXt is unfair. If (ii) or (iv) is used, the architecture departs from the training setup, where MIM/MASM are trained as auxiliary heads. Moreover, because Eq. (2)'s ranking and MIM's channel/spatial rectification are present only in training, the 'hierarchical modality selection' described in the abstract is not actually performed at inference. The reported numbers are therefore irreproducible without an explicit inference rule, and the central claim does not follow from the described system.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"MAGIC++ proposes a training-time framework for modality-agnostic semantic segmentation, built on a shared backbone such as SegFormer, with two auxiliary modules: a Multi-modal Interaction Module (MIM) and a Multi-scale Arbitrary-modal Selection Module (MASM). During training, MIM fuses features across modalities, and MASM ranks per-scale modality features by cosine similarity to their mean, then supervises the backbone with losses from the top/fragile features plus a consistency loss. The paper reports MaSS evaluations on MUSES and DELIVER and claims state-of-the-art results in the common multi-modal setting and large gains in the arbitrary-modality setting (+2.19% and +7.25% mean mIoU vs. MAGIC). The central contribution, as stated, is a single backbone that at inference is robust to any available subset of modalities, without using MIM/MASM.","tokens_in":20958,"tokens_out":9542,"duration_ms":84636,"significance":"If fully supported, the paper would address a real and timely problem: training a single segmentation model that behaves gracefully under arbitrary sensor availability. The idea of using training-time auxiliary losses to make a backbone robust to missing modalities is potentially useful, and the paper explores multiple backbones and two datasets. The paper also gives clear credit to previous work and extends a prior ECCV paper with explicit incremental contributions. However, the significance is severely undercut by three load-bearing problems: (1) the test-time inference rule for an arbitrary modality subset is never defined, making the reported numbers unverifiable; (2) the abstract's claim of state-of-the-art performance in the common multi-modal setting is contradicted by the paper's own full-modality columns; and (3) the main comparison is against the authors' own prior MAGIC, with only one external modality-agnostic baseline (Any2Seg, also by the same group), so the external validation is thin.","major_comments":[{"comment":"The test-time inference procedure is unspecified, and this omission is load-bearing for every result in Tables I-IV. The paper states: \"The MIM and MASM is only utilized in training while the inference is achieved by the backbone model, i.e., SegFormer.\" At test time, given a subset such as {R,D}, the paper does not say whether each available modality is passed independently through the shared backbone and the resulting softmax maps are averaged, whether features from multiple modalities are averaged before the segmentation head, whether one modality is selected, or whether the subset is concatenated and fed as a single multi-channel input. None of these rules is described or justified, and each yields different predictions. Since MIM and MASM are removed at inference, the \"hierarchical modality selection\" named in the title and abstract is not actually performed at test time. Without an explicit inference rule, the reported modality-agnostic results are not reproducible, and the gains over a simple ensemble of per-modality backbones cannot be assessed. This is not a presentation issue; it directly undermines the central claim.","section":"Section III-C (Training paragraph), Section III-A/B"},{"comment":"The claim that \"Under the common multi-modal setting, our method achieves state-of-the-art performance\" is contradicted by the full-modality columns in the paper's own tables. On MUSES (Table I), with all three modalities (FEL), MAGIC++ obtains 40.58 mIoU, whereas Any2Seg reaches 50.00 and MAGIC reaches 49.02. On DELIVER (Table II), with all four modalities (RDEL), MAGIC++ obtains 61.67 compared to MAGIC's 63.40. In Table III, the full-combination RDL scores are 61.80 (MAGIC++) vs. 62.64 (MAGIC), and RDE scores are 61.83 vs. 62.49. Section VI.A explicitly acknowledges that \"MAGIC++ under-performs in some controlled multi-modal evaluation settings.\" The abstract and introduction therefore overstate the results; the gains are captured in the mean over all arbitrary combinations, not in the standard full-modality setting. The authors should revise the central claim to be consistent with their own data.","section":"Abstract, Section I, Table I, Table II, Table III, Section VI.A"},{"comment":"The Swin-tiny RDL row for MAGIC++ lists RD=61.39, RL=49.59, DL=53.97, RDL=61.80, which are exactly identical to the corresponding values for the SegFormer-B0 backbone in Table III (RDL block). Since the backbone is different, identical mIoU values to two decimal places for these multi-modality combinations are implausible and strongly suggest a copy-paste error. If these numbers are incorrect, the cross-backbone generalization claims in Section IV.B.4 are unsupported; the authors should verify and correct this row or explain why the values coincide.","section":"Table IV, Swin-tiny RDL row"},{"comment":"The definition of the consistency loss is unclear. The scalars c1 and c2 are defined with features f^1_rm and f^2_rm, but the summation runs over K-1, and it is not specified whether the index is over semantic classes, spatial locations, or feature scales. In addition, Eq. (5) introduces β as the only trade-off weight, while the text mentions both λ and β; λ is never defined. This ambiguity prevents the reader from understanding how the loss is computed and how the hyper-parameters are set in the ablations of Table V.","section":"Section III-C3, Eq. (4)"}],"minor_comments":[{"comment":"The text still refers to the previous version's \"MAM\" and to a semantic feature \"fse derived from the MAM\"; the new module is MIM, and the notation should be updated for consistency.","section":"Section III-C1"},{"comment":"The sentence mentions \"λ and β are hyper-parameters,\" but λ does not appear in Eq. (5). Use only β or define λ.","section":"Eq. (5)"},{"comment":"The \"Training\" column is not self-explanatory; for example, CMNeXt shows \"FEL\" under Training while MAGIC++ shows \"-\". Please clarify what training configuration is used for each method.","section":"Tables I and II headers"},{"comment":"Typo: \"propsoed\" should be \"proposed\" in both figure captions.","section":"Figure 4 and Figure 5 captions"},{"comment":"\"DELVIER\" should be \"DELIVER\".","section":"Footnote 1"},{"comment":"In the sentence \"all the features are resized for better visualization,\" a period is missing after \"visualization\".","section":"Section V.B"}],"recommendation":"reject","confidential_remarks":"The paper shares authors with several of the baselines it compares against (MAGIC, Any2Seg), and the primary improvement numbers are computed with respect to the authors' own ECCV 2024 method. The more serious concern is that the method's test-time behavior is undefined, so I cannot verify whether the reported gains reflect a fusion mechanism or simply an ensemble effect. If the authors can clarify that inference is obtained by averaging per-modality backbone outputs, they must also compare against a simple ensemble baseline and against training-time-only robustness methods; if they cannot clarify, the reported MaSS results are not reproducible. Either way, the current manuscript cannot be accepted as is. I would encourage the editor to request code or a companion technical appendix before any resubmission is considered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe reader's stress-test is right, and it lands on a load-bearing spot: the paper never specifies how the backbone handles an arbitrary subset of modalities at test time. Section III-C states that MIM and MASM are used only in training and inference is achieved by the backbone alone. But the paper also says the multi-modal mini-batch is directly processed by the encoder. So for, say, {RGB, Depth}, is each modality passed through the shared backbone and the softmax maps averaged? Are features averaged? Is one modality chosen? None of this is stated. Without this rule, the reported mean mIoU over all combinations cannot be reproduced, and the comparison to fusion baselines like CMNeXt is not fair. The stress-test's concern about an ensemble effect is plausible, and the paper gives no way to rule it out.\n\nTo give credit where it's due: the paper is a real extension of the authors' ECCV 2024 MAGIC. The multi-scale selection (MASM) and upgraded interaction (MIM) are new modules, the consistency loss is a reasonable idea, and the experiments cover two benchmarks and three backbones. The gains on sparse modalities like Event and LiDAR are large and would be interesting if they hold up. The discussion honestly concedes that MAGIC++ underperforms MAGIC on full-modality RDEL on DELIVER (61.67 vs 63.40). That admission is consistent with the tables.\n\nThe soft spots beyond the missing inference rule are substantial. The abstract's \"state-of-the-art under the common multi-modal setting\" is contradicted by the paper's own tables: on MUSES, Any2Seg gets 50.00 on FEL versus 40.58 for MAGIC++; on DELIVER full RDEL, MAGIC gets 63.40 versus 61.67. The paper's dismissal of this as \"slightly below\" doesn't square with the abstract. There are no error bars, no code, and the ablations cover only the loss combination, not the selection module or MIM. Eq. (5) mentions λ but only β appears; a minor typo. The core direction—training-time robustness to missing sensors—is reasonable and worth pursuing.\n\nWho is this for? Researchers working on modality-agnostic or missing-modality segmentation. They might read it as a cautionary case and as a source of ideas for training-time ranking. But I would not cite the numbers until the inference protocol is specified and the experiments become reproducible.\n\nRecommendation: this deserves a serious referee, but not acceptance as it stands. Send it to review with a demand for a precise test-time fusion rule, a corrected abstract, and ablations isolating the modules. If the authors can supply those, the paper could be salvageable. For now, the reader's reject verdict is justified.","headline":"The missing test-time fusion rule makes the headline numbers irreproducible, but the training-time robustness idea is worth a revision.","tokens_in":21502,"tokens_out":3218,"would_cite":false,"duration_ms":26371,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training-time modality ranking lets one segmentation backbone handle any sensor subset, with large gains on MUSES and DELIVER.","keywords":["Semantic Segmentation","Multi-modal Learning","Modality-agnostic Segmentation","Modality Selection","Sensor Failure Robustness","Multi-scale Feature Fusion"],"falsifier":"Train the same backbone on DELIVER with identical batches and losses but replace the similarity-based selection in MASM with a random modality choice; if random selection reproduces the reported mean mIoU (47.74 for the four-modality training), the ranking is not the cause. The minimal observation is the exact inference rule: the paper should state how predictions from available sensors are combined, since the results cannot be attributed without it.","tokens_in":20426,"feed_emoji":"🧩","tokens_out":5390,"duration_ms":47706,"temperature":0.7,"pith_summary":"The paper argues that semantic segmentation can be made resilient to arbitrary combinations of available sensors by training a single backbone that learns, at each feature scale, which modalities are reliable and which are fragile. It proposes MAGIC++, adding a multi-modal interaction module and a multi-scale arbitrary-modal selection module during training, and reports state-of-the-art results in the standard fully-combined setting alongside large margins in the modality-agnostic setting: +2.19% mean mIoU on MUSES and +7.25% on DELIVER across all modality subsets. A sympathetic reader would care because real autonomous systems lose sensors or face degraded conditions, and a model that treats every sensor as potentially valuable rather than RGB-centric would be safer in those situations.","feed_headline":"One backbone, any sensor mix: segmentation gains +7.25% on DELIVER","feed_subtitle":"Modality-selection modules train the network to lean on whichever sensors are reliable, so missing ones barely matter.","key_machinery":"The load-bearing mechanism is the Multi-scale Arbitrary-modal Selection Module (MASM), which at each transformer-block scale computes the cosine similarity of each modality's feature to their mean feature, ranks them, and forwards the top-ranked (robust) and bottom-ranked (fragile) features to the Multi-modal Interaction Module (MIM). MIM then rectifies these selected features with channel-wise and spatial-wise attention and produces a fused feature that also serves as a surrogate for aligning the remaining modality features through a consistency loss. This forces the encoder to learn from both reliable and degraded inputs, which the paper claims is what transfers modality-agnostic robustness to the plain backbone at inference.","core_discovery":"On the paper's own terms, the central discovery is that training a hierarchical backbone with a cross-modal similarity ranking over each scale's features—merging the most robust and most fragile modalities—makes the backbone itself modality-agnostic, so that at test time it can consume whatever sensors are available without retraining or extra fusion modules. The reported evidence is that MAGIC++ outperforms previous arbitrary-modality methods in mean mIoU across all possible modality combinations, with the largest gains appearing precisely in the sparse-sensor cases (e.g., Event-only from 0.52 to 19.03 on DELIVER and Frame/Event gains on MUSES). The modules are used only during training; inference runs on the backbone alone.","pith_inferences":["The paper does not state how the backbone combines outputs from multiple sensors at inference; if that rule is prediction averaging, part of the margin over prior methods may be an ensemble effect that those methods could also exploit—an editor-level concern, not a claim of the paper.","The ranking-by-similarity idea is a generic training-time regularizer and could be tested on other dense tasks such as depth estimation or object detection that must cope with missing sensors.","A direct way to check the mechanism is to replace the similarity ranking with random selection during training; if performance is unchanged, the specific ranking signal is not the source of the gains.","The reported single-modality gains suggest the backbone learns to extract usable signal from weak modalities; probing feature quality during training could tell whether that is learned or merely inherited from the fusion losses."],"forward_implications":["A vehicle or robot equipped with several sensors could be trained once on all of them, then continue segmenting accurately when some sensors fail or are unavailable, without per-scenario retraining.","Because the auxiliary modules are discarded at test time, deployment can run the same lightweight backbone as a unimodal model while keeping the robustness gains.","The approach transfers across at least three hierarchical backbones (SegFormer, PVTv2, Swin), suggesting the training-time selection recipe is not tied to one encoder.","Adding the cross-modal consistency loss improves mean mIoU by a small but consistent margin on DELIVER (47.10 to 47.74), including on all-sensor input."],"supporting_citations":[{"why":"the predecessor method whose results the paper extends and whose training scheme it modifies.","marker":"[1]"},{"why":"provides the DELIVER dataset and the CMNeXt baseline/training setup for arbitrary-modal segmentation.","marker":"[12]"},{"why":"provides the MUSES real-world multi-sensor dataset used for evaluation.","marker":"[13]"},{"why":"the SegFormer backbone on which the primary experiments are built.","marker":"[20]"},{"why":"an arbitrary-modality baseline compared in the main tables.","marker":"[9]"},{"why":"the CMX cross-modal fusion baseline used as a comparison point.","marker":"[107]"}],"fun_headline_variants":["Modality-agnostic segmentation: train once, use any sensor mix","No RGB? No problem: segmentation adapts to any sensor set","Train once, segment with any sensor combo—no retraining needed","Event-only sensors? MAGIC++ still delivers top segmentation","Hierarchical selection makes any-modality segmentation state-of-the-art"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper says the interaction and selection modules are used only in training and that inference is achieved by the backbone, but it never specifies how the backbone processes an arbitrary subset of modalities at test time—if the rule is simple averaging of per-modality predictions, the gains may be an ensemble effect rather than the new selection mechanism.","fun_headline_variants_meta":{"raw":{"variants":["Modality-agnostic segmentation: train once, use any sensor mix","No RGB? No problem: segmentation adapts to any sensor set","Train once, segment with any sensor combo—no retraining needed","Event-only sensors? MAGIC++ still delivers top segmentation","Hierarchical selection makes any-modality segmentation state-of-the-art"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000832,"raw_usage":{"total_tokens":3654,"prompt_tokens":992,"completion_tokens":2662,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":608,"completion_tokens_details":{"reasoning_tokens":2573}},"tokens_in":608,"tokens_out":2662,"duration_ms":18054,"temperature":1.0,"reasoning_tokens":2573,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T06:01:52.652439+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same backbone on DELIVER with identical batches and losses but replace the similarity-based selection in MASM with a random modality choice; if random selection reproduces the reported mean mIoU (47.74 for the four-modality training), the ranking is not the cause. The minimal observation is the exact inference rule: the paper should state how predictions from available sensors are combined, since the results cannot be attributed without it.","supporting_citations":[{"cited_title":"Cmx: Cross-modal fusion for rgb-x semantic segmentation with transform- ers,","cited_arxiv_id":null,"evidence_quote":"the CMX cross-modal fusion baseline used as a comparison point."}],"review_version":1}