{"id":"fdbaf92e-da96-45d0-bd87-3102082596ef","arxiv_id":"2411.17141","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"AnySeg trains a segmentor to handle arbitrary combinations of visual modalities through unimodal and cross-modal distillation, improving mean mIoU by +6.37% on MUSES and +6.15% on DELIVER over prior state-of-the-art.","lead":"This paper presents AnySeg, a training method that helps image-segmentation models keep working when some input sensors, like depth, lidar, or event cameras, are missing. The authors train a teacher model on all sensors, then teach a student model to imitate the teacher using any subset, which matters for autonomous driving and robotics where sensors fail intermittently.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's central distillation may not be load-bearing: on MUSES the no-distillation Lsup+dropout student already reaches 39.29 vs 40.23 final, and no Lsup-only baseline is reported on DELIVER, so the +6.15 headline could be driven by random anymodal dropout rather than the proposed UMD/CMD/MAD…","rationale":"The reader's review is CONDITIONAL for several reasons. My stress-test narrows this to a single, checkable attribution problem. The most damaging fact in the paper is Table 3: the Lsup-only student, before any distillation, attains 39.29 on MUSES, while the complete framework attains 40.23. Because the SoTA baselines cited in Table 1 are 33.34 and 33.86, the simple supervised anymodal-dropout baseline already beats them by a wide margin; the three distillation losses add less than one point. This is not an internal inconsistency, but it makes the title and abstract's claimed mechanism load-bearing without support. The paper does not give the equivalent Lsup-only number on DELIVER, so the headline +6.15 may be independent of distillation. A fair reviewer would require this ablation before accepting the central claim. This does not change the reader's CONDITIONAL verdict but sharpens the condition: release the DELIVER Lsup-only ablation and multi-seed MUSES comparison, and measure the PML teacher's own anymodal mIoU. I agree with the reader that the teacher assumption is a weak point, but the more precise failure mode is the unproven causal role of distillation given the strong no-distillation baseline.","tokens_in":15316,"tokens_out":11117,"duration_ms":105418,"concrete_test":"Run a controlled ablation on DELIVER: train the anymodal student with only Lsup plus the same random anymodal dropout used in the full method (no teacher, no Lumd/Lcmd/Lmad), using the same hyperparameters and evaluation protocol as Table 2, and report the mean and per-subset mIoU over all 15 modality combinations. Also run this Lsup-only configuration on MUSES with at least 3 seeds to assess whether the observed +0.94 gap is significant. If the Lsup-only mean is within roughly 1 mIoU of AnySeg on both benchmarks, the distillation framework is not the load-bearing component and the paper must re-attribute its gains; if it is substantially lower on DELIVER, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"AnySeg's central claim is that unimodal, cross-modal, and modality-agnostic distillation from a PML teacher produce robustness to arbitrary modality subsets. Table 3 undercuts this attribution. The Lsup-only student, trained with random anymodal dropout but with no teacher and none of the three distillation losses, reaches 39.29 mean mIoU on MUSES; the full method reaches 40.23, a gain of 0.94, and the intermediate Lsup+Lmad+Lumd configuration is actually 0.39 points below Lsup+Lmad. Meanwhile the reported SoTA baselines are 33.34 (MAGIC) and 33.86 (Any2Seg), so the no-distillation student already beats prior SoTA by roughly 5.4-5.9 points. The paper never reports the analogous Lsup-only baseline on DELIVER, where the headline +6.15 gain over MAGIC is claimed. If that baseline is within about 1 mIoU of the full method, the proposed distillation is not the operative ingredient; random anymodal dropout is. This is compounded by the fact that the PML teacher's own per-subset mIoU is never measured, so there is no direct evidence that the per-modality branch features used as distillation targets are valid supervisors for every missing-modality combination.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AnySeg, a two-stage framework for semantic segmentation that is robust to arbitrary combinations of visual modalities (RGB, depth, LiDAR, event). A multimodal teacher is first trained with a parallel modality learning (PML) strategy that averages per-block features across modalities, and is then frozen. The student is trained with random anymodal dropout and three distillation losses: unimodal distillation (Lumd) aligning per-modality features, cross-modal distillation (Lcmd) aligning pairwise cosine similarities, and modality-agnostic prediction-level distillation (Lmad). Experiments on MUSES and DELIVER report mean mIoU improvements of +6.37 and +6.15 over prior state-of-the-art. The paper includes extensive ablations of loss weights and teacher choices.","tokens_in":15541,"tokens_out":7975,"duration_ms":70221,"significance":"If the central claim holds, AnySeg would be a practical contribution to multimodal segmentation under sensor failure, with notable gains on weak modalities such as event and LiDAR. The paper provides a useful negative result about the ineffectiveness of distillation on fused features, and the PML teacher is a simple and parameter-free alternative to fusion-based teachers. However, the ablation study in Table 3 shows that the no-distillation baseline (Lsup with anymodal dropout) already achieves 39.29 mean mIoU on MUSES, only 0.94 below the full method, and that the proposed UMD and CMD losses add just 0.12 over Lsup+Lmad. This substantially weakens the attribution of the headline gains to the proposed distillation mechanism, which is the paper's central contribution. The significance of the work therefore depends on whether the authors can either demonstrate a larger marginal contribution of the distillation losses or reframe the contribution as a training strategy centered on anymodal dropout with modest distillation refinement.","major_comments":[{"comment":"The ablation in Table 3 undercuts the causal claim that unimodal and cross-modal distillation are the key ingredients. The Lsup-only student, trained with random anymodal dropout but no teacher and none of the proposed losses, already achieves 39.29 mean mIoU on MUSES, which is 5.43 points above the Any2Seg baseline of 33.86. Adding Lmad brings the mean to 40.11, and the full method including Lumd and Lcmd reaches only 40.23. Thus the marginal contribution of the paper's two feature-level distillation losses (UMD and CMD) is just 0.12 mIoU over Lsup+Lmad, while the gain from anymodal dropout plus supervised training is 5.43 points. The paper never reports the analogous Lsup-only baseline on DELIVER, where the headline +6.15 gain is claimed. Without that control, the reader cannot tell whether the reported improvement over MAGIC on DELIVER is attributable to the proposed distillation or simply to random anymodal dropout. Please report the Lsup-only result on DELIVER and, if the marginal gains remain small, reframe the contribution accordingly.","section":"Table 3, Sec. 5"},{"comment":"Equation (5) is described as a cross-modal distillation loss based on KL divergence, but the normalized cosine similarity \\tilde{S}(x,y) = (cos(x,y)+1)/2 is a scalar in [0,1] for each channel index j, and the sum over j is not normalized to yield a probability distribution. The statement that averaging the similarity scores across batch samples leads to non-negative values does not make these quantities a distribution over channels or over any finite set. Consequently, the logarithm of the ratio of such scalars does not define a valid KL divergence. The authors should either reformulate Eq. (5) as a proper divergence (for example, treating each channel's normalized similarity as a Bernoulli variable and summing a corresponding KL term) or explicitly present it as a heuristic distance objective with a justification for why it is effective.","section":"Eq. 5, Sec. 3.2"},{"comment":"The paper claims that the PML teacher, which averages multimodal features per block, is a strong supervisor for every unimodal and cross-modal combination, but it never reports the teacher's per-subset mIoU. Table 7 only compares student performance after distillation using different teachers, which is an indirect and confounded measure of teacher quality. If the teacher's per-modality branch features are weak because supervision is only applied to the final averaged output, then aligning the student to those features may not be the actual mechanism of improvement. Please report the teacher's mIoU for each input combination (F, E, L, FE, FL, EL, FEL) on MUSES, and similarly on DELIVER, to provide direct evidence that the averaged features are valid supervision targets for all subsets.","section":"Sec. 3.1, Tab. 7"},{"comment":"The hyperparameters λ, α, and β are tuned on the evaluation benchmark itself (e.g., λ=50, α=5, β=10 for MUSES), and all reported results come from a single run without error bars or multiple seeds. Given that the full method's gain over Lsup+Lmad is only 0.12 mIoU on MUSES, the differences between configurations could easily be within run-to-run variance. Without variance estimates, the reader cannot assess whether the reported improvements are statistically meaningful, and the selection of hyperparameters on the test set raises a mild overfitting concern. Please provide at least three seeds with standard deviations for the main tables, or otherwise discuss the stability of the results.","section":"Tabs. 4-6, Sec. 5"}],"minor_comments":[{"comment":"The claim of being \"the first framework for learning robust segmentor that can handle any combinations of visual modalities\" is contradicted by the paper's own references to MAGIC and Any2Seg, which already target modality-agnostic segmentation over arbitrary input subsets. Consider rephrasing to \"a distillation-based framework\" or limiting the novelty claim to the specific distillation mechanism.","section":"Abstract and Sec. 1"},{"comment":"The loss Lumd sums over channels only, but the features are spatial maps; the spatial dimensions are not explicitly summed in the equation, leaving the loss definition ambiguous. Additionally, the text says \"softmax to the teacher features g and log-softmax to the student features f\" but writes log(\\tilde{g}/\\tilde{f}) with \\tilde{f} in the denominator; if \\tilde{f} is a log-softmax output, the ratio is not the standard KL form. Please clarify the normalization and the spatial summation.","section":"Eq. 4, Sec. 3.2"},{"comment":"Equation (2) uses the symbol Fma for the teacher encoder, while the teacher is denoted Fms elsewhere (e.g., Sec. 3.1 and Fig. 1). Please unify the notation.","section":"Eq. 2, Sec. 3.2"},{"comment":"The row \"w.r.t SoTA\" compares against Any2Seg's 33.86, but the text states the method \"surpassing all state-of-the-art baselines.\" Since MAGIC and Any2Seg have different means, please specify which baseline is used for the margin in the table and in the abstract.","section":"Table 1, Sec. 4.1"},{"comment":"The text says \"distilling from fused features, as in MAGIC and CMNeXt, degrades performance,\" but Table 7 compares teacher models, not whether the student distills from fused features; the direct evidence for the fused-feature claim is Table 8. Please align the wording with the distinction between teacher design and distillation target.","section":"Sec. 5, Tab. 7 and text"}],"recommendation":"major_revision","confidential_remarks":"The paper's own ablations confirm a key concern: the no-distillation baseline with random anymodal dropout already nearly matches the full method on MUSES, and the proposed distillation losses add only a small margin. This needs to be addressed head-on before the paper can be accepted, either by demonstrating a larger marginal contribution on both benchmarks or by recasting the contribution as a dropout-based training strategy with distillation as a secondary refinement. The technical issue with Eq. (5) being called a KL divergence should also be fixed, as it affects the paper's theoretical framing. The paper may also need to reconsider the 'first' claim given the closely related prior work by the same authors."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful within-subfield paper on anymodal segmentation, but the attribution of the headline gains to the proposed distillation is not supported by the ablation table; the random anymodal dropout does the heavy lifting. Still deserves a serious referee because the problem is real and the experiments are extensive.\n\nWhat's new: the combination of a parallel multimodal learning teacher (PML), unimodal distillation, cross-modal correspondence distillation, and modality-agnostic prediction distillation. Individually these are known tools, but the specific pairing of unimodal + cross-modal distillation from an averaged-feature teacher is not in prior work. The paper is also honest about the failure of fused-feature KD (Tab. 8) and shows consistent mean gains on MUSES, including large jumps on weak modalities like event (+16.4) and LiDAR (+9.8).\n\nSoft spots, in order of severity. First, the stress-test note is right: on MUSES the Lsup-only student with anymodal dropout already hits 39.29, versus 40.23 for the full method. That 0.94 gain is the entire contribution of all three distillation losses, and some intermediate combinations are worse than Lsup+Lmad. Meanwhile prior SoTA is 33.3-33.9, so the dropout itself accounts for about 5.5 points. No Lsup-only baseline is reported on DELIVER, so we cannot tell whether the +6.15 headline is due to distillation or just to dropout. The claim that UMD/CMD/MAD are the operative ingredients is therefore unproven. Second, Eq.5 is labeled KL divergence but uses normalized cosine similarities; these are not probabilities over channels, and the paper's argument that averaging across batch ensures non-negativity is both empirically fragile and theoretically sloppy. That needs fixing or re-deriving. Third, no error bars or multiple seeds, and hyperparameters are tuned on the same benchmark, so the reported deltas in Tabs. 4-6 could be noise. The paper should report variance at least for the main table. Fourth, the 'first framework' claim is overstated given MAGIC and Any2Seg already target modality-agnostic segmentation; the real novelty is the distillation scheme from a parallel teacher, not the task itself.\n\nAlso worth noting: the teacher's per-subset mIoU is never measured, so there is no direct evidence that the per-modality branch features are valid supervisors for every missing-modality combination. That weakens the conceptual story.\n\nWho it's for: anyone working on missing-modality robustness in driving perception. It is not a big conceptual leap, but the experiments are solid enough to warrant a serious peer review with attention to the attribution problem. Recommendation: send it out, but require the Lsup-only DELIVER baseline, a corrected formulation for the cross-modal loss, and error bars before acceptance.","headline":"Useful anymodal segmentation paper, but the distillation losses add less than 1 mIoU over plain modality dropout on MUSES; the headline gains belong to the dropout, not the distillation.","tokens_in":16114,"tokens_out":2476,"would_cite":false,"duration_ms":22249,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AnySeg trains a single segmentor to serve every subset of RGB, depth, lidar, and event inputs, improving mean mIoU by up to 6.37 points over prior modality-agnostic methods.","keywords":["semantic segmentation","multimodal learning","missing modality robustness","knowledge distillation","unimodal bias","anymodal segmentation","cross-modal correspondence","multi-sensor fusion"],"falsifier":"One decisive check is to measure the PML teacher's own unimodal mIoU on event-only and lidar-only inputs on MUSES and compare it with the distilled student's. If the student matches or exceeds the teacher on those hard subsets, then the averaged teacher is not actually supplying the hard-modality knowledge and the claim that averaging is the right supervision for every subset is undermined; alternatively, replacing the averaged teacher with a learned-confidence-weighted fusion teacher and showing the student does no worse would falsify the claim that simple averaging is the key.","tokens_in":15083,"feed_emoji":"🚗","tokens_out":6409,"duration_ms":53527,"temperature":0.7,"pith_summary":"The paper proposes AnySeg, a framework that trains a single semantic segmentation model to work well for every subset of available visual modalities—RGB, depth, lidar, and event data—instead of assuming all sensors are present. It argues that the core obstacle is unimodal bias, the tendency of multimodal models to rely on easy modalities like RGB and collapse when those are missing. The solution is a two-stage distillation: first train a strong teacher that treats all modalities in parallel with simple feature averaging, then distill unimodal and cross-modal feature knowledge plus prediction-level semantic knowledge into an anymodal student trained under random modality dropout. On the real-world MUSES and synthetic DELIVER benchmarks, the resulting student improves mean mIoU by +6.37% and +6.15% over prior state-of-the-art, with the largest gains on the hardest modalities, event and lidar. If correct, this gives a practical recipe for sensor-failure-robust perception without changing the inference architecture.","feed_headline":"Anymodal distillation lifts missing-modality mIoU by 6.37 points","feed_subtitle":"With RGB, depth, lidar, or event data missing, averaged-teacher distillation keeps segmentation accurate.","key_machinery":"The load-bearing mechanism is a teacher-student distillation scheme built on four components: (1) Parallel Multimodal Learning (PML), which trains a teacher by averaging multimodal features block-by-block in a shared SegFormer backbone before the segmentation head; (2) anymodal dropout, which randomly masks modalities so the student sees every subset; (3) unimodal distillation ($\\mathcal{L}_{umd}$) and cross-modal correspondence distillation ($\\mathcal{L}_{cmd}$), KL-style losses in multi-scale feature space that respectively transfer per-modality distributions and the cosine-similarity structure between modality pairs; and (4) modality-agnostic semantic distillation ($\\mathcal{L}_{mad}$), a prediction-level KL loss from teacher to student segmentation maps. The total loss combines these with a supervised cross-entropy term. The paper's argument is that learning from an averaged, not fused, teacher is what prevents the student from inheriting unimodal bias.","core_discovery":"On the paper's own terms, the central discovery is that unimodal bias in multimodal semantic segmentation can be removed by distilling from a teacher that never fuses modalities in a learned way. The teacher, trained by Parallel Multimodal Learning, feeds each modality through a shared backbone and simply averages the per-block features before the segmentation head; this preserves both modality-specific and shared information. The student is then trained with random anymodal dropout, and three losses align its multi-scale features with the teacher's: unimodal distillation transfers each single-modality distribution, cross-modal correspondence distillation matches the cosine-similarity structure between modality pairs, and modality-agnostic prediction distillation transfers the teacher's segmentation maps. The paper reports that distilling from directly fused features, as in prior fusion-based teachers, degrades performance, whereas the averaged-feature teacher yields balanced gains across all modality combinations, including large improvements on event and lidar inputs that prior methods nearly ignore.","pith_inferences":["A natural extension is to apply the same averaged-teacher plus unimodal and cross-modal distillation recipe to other fusion tasks, such as RGB-thermal or audio-visual segmentation, where one modality similarly dominates learning.","The PML teacher's simple averaging could be interpreted as a strong regularizer that prevents the teacher from learning modality-specific shortcuts, which may be why distillation from it transfers balanced knowledge; this suggests learned fusion may be unnecessary for teacher construction when the goal is robustness rather than peak fused accuracy.","The method's success on event and lidar hints that anymodal segmentation models could serve as a diagnostic tool: the gap between teacher and student on a given modality subset may quantify how much information that sensor actually contributes in a scene.","A testable extension is to replace random dropout with curriculum dropout that drops easy modalities first, then check whether the student gains further on hard modalities without sacrificing RGB performance."],"forward_implications":["A single AnySeg student trained once can be deployed at inference time with any available subset of sensors, and it maintains higher mean mIoU across all combinations than prior modality-agnostic methods.","The largest relative gains appear exactly where unimodal bias is worst: event-only and lidar-only inputs improve by +16.40 and +9.80 mIoU on MUSES, and event and lidar gains are +16.81 and +18.62 on DELIVER.","Distilling from fused features is not merely suboptimal but actively harmful: adding fused-feature knowledge distillation drops mean mIoU from 40.23 to as low as 39.37, so the teacher representation must keep modality structure intact.","The framework is robust to scene degradations: on DELIVER RGB-D under cloud, fog, night, rain, and sensor-noise conditions, AnySeg stays above 53% mIoU in all settings."],"supporting_citations":[{"why":"Prior modality-agnostic segmentor that serves as the main baseline; the paper shows it still drops sharply when depth is missing.","marker":"[6]"},{"why":"Prior state-of-the-art modality-agnostic method that AnySeg outperforms; its equal-treatment insight motivates the PML averaging strategy.","marker":"[7]"},{"why":"Introduces the DELIVER benchmark and the CMNeXt arbitrary-modal fusion baseline; supplies the dataset and the fixed-fusion contrast.","marker":"[17]"},{"why":"Provides the real-world MUSES multi-sensor benchmark used for the main three-modality evaluation and ablations.","marker":"[18]"},{"why":"SegFormer is the shared backbone for teacher and student; all feature-level distillation operates on its multi-scale representations.","marker":"[37]"},{"why":"Supplies the theoretical account of modality competition that motivates the paper's definition of unimodal bias.","marker":"[29]"}],"fun_headline_variants":["Anymodal segmentation: averaged-teacher distillation beats fusion","No fusion, no bias: robust segmentation with any missing modality","Distilling from averaged features makes segmentor robust to missing inputs","Teacher that averages, not fuses: key to anymodal segmentation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole scheme rests on the premise that a teacher built by simply averaging multimodal features, block by block, in a shared backbone is strong enough to supervise every unimodal and cross-modal combination; if that averaged teacher is itself biased toward easy modalities or only strong when all inputs are present, the distillation would transfer the bias rather than remove it.","fun_headline_variants_meta":{"raw":{"variants":["Anymodal segmentation: averaged-teacher distillation beats fusion","No fusion, no bias: robust segmentation with any missing modality","Distilling from averaged features makes segmentor robust to missing inputs","Teacher that averages, not fuses: key to anymodal segmentation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000269,"raw_usage":{"total_tokens":1593,"prompt_tokens":887,"completion_tokens":706,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":635}},"tokens_in":503,"tokens_out":706,"duration_ms":8049,"temperature":1.0,"reasoning_tokens":635,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:27:49.656968+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"One decisive check is to measure the PML teacher's own unimodal mIoU on event-only and lidar-only inputs on MUSES and compare it with the distilled student's. If the student matches or exceeds the teacher on those hard subsets, then the averaged teacher is not actually supplying the hard-modality knowledge and the claim that averaging is the right supervision for every subset is undermined; alternatively, replacing the averaged teacher with a learned-confidence-weighted fusion teacher and showing the student does no worse would falsify the claim that simple averaging is the key.","supporting_citations":[{"cited_title":"Learning modality-agnostic representation for semantic segmentation from any modalities,","cited_arxiv_id":null,"evidence_quote":"Prior modality-agnostic segmentor that serves as the main baseline; the paper shows it still drops sharply when depth is missing."},{"cited_title":"Centering the value of every modality: Towards efficient and resilient modality-agnostic semantic segmentation,","cited_arxiv_id":null,"evidence_quote":"Prior state-of-the-art modality-agnostic method that AnySeg outperforms; its equal-treatment insight motivates the PML averaging strategy."},{"cited_title":"De- livering arbitrary-modal semantic segmentation,","cited_arxiv_id":null,"evidence_quote":"Introduces the DELIVER benchmark and the CMNeXt arbitrary-modal fusion baseline; supplies the dataset and the fixed-fusion contrast."},{"cited_title":"Muses: The multi-sensor semantic perception dataset for driving under uncertainty,","cited_arxiv_id":null,"evidence_quote":"Provides the real-world MUSES multi-sensor benchmark used for the main three-modality evaluation and ablations."},{"cited_title":"Segformer: Simple and efficient design for semantic segmentation with transformers,","cited_arxiv_id":null,"evidence_quote":"SegFormer is the shared backbone for teacher and student; all feature-level distillation operates on its multi-scale representations."},{"cited_title":"Modality competition: What makes joint training of multi-modal network fail in deep learning? (provably),","cited_arxiv_id":null,"evidence_quote":"Supplies the theoretical account of modality competition that motivates the paper's definition of unimodal bias."}],"review_version":1}