Pith. sign in

REVIEW 4 major objections 5 minor 7 cited by

Learning Robust Anymodal Segmentor with Unimodal and Cross-modal Distillation

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read AnySeg trains a single segmentor to serve every subset of RGB, depth, lidar, and event inputs, improving mean mIoU by up to 6.37 points over prior modality-agnostic methods.

desk verdict Useful anymodal segmentation paper, but the distillation losses add less than 1 mIoU over plain modality dropout on MUSES; the headline gains belong to the dropout, not the distillation. read the letter →

arxiv 2411.17141 v2 pith:FZIMGZPK submitted 2024-11-26 cs.CV

Xu Zheng , Haiwei Xue , Jialei Chen , Yibo Yan , Lutao Jiang , Yuanhuiyi Lyu , Kailun Yang , Linfeng Zhang
show 1 more author
Xuming Hu
This is my paper · ORCID
classification cs.CV
keywords semanticsegmentationmultimodallearningmissingmodalityrobustnessknowledgedistillationunimodalbiasanymodalcross-modalcorrespondencemulti-sensorfusion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes AnySeg, a framework that trains a single semantic segmentation model to work well for every subset of available visual modalities—RGB, depth, lidar, and event data—instead of assuming all sensors are present. It argues that the core obstacle is unimodal bias, the tendency of multimodal models to rely on easy modalities like RGB and collapse when those are missing. The solution is a two-stage distillation: first train a strong teacher that treats all modalities in parallel with simple feature averaging, then distill unimodal and cross-modal feature knowledge plus prediction-level semantic knowledge into an anymodal student trained under random modality dropout. On the real-world MUSES and synthetic DELIVER benchmarks, the resulting student improves mean mIoU by +6.37% and +6.15% over prior state-of-the-art, with the largest gains on the hardest modalities, event and lidar. If correct, this gives a practical recipe for sensor-failure-robust perception without changing the inference architecture.

What carries the argument

The load-bearing mechanism is a teacher-student distillation scheme built on four components: (1) Parallel Multimodal Learning (PML), which trains a teacher by averaging multimodal features block-by-block in a shared SegFormer backbone before the segmentation head; (2) anymodal dropout, which randomly masks modalities so the student sees every subset; (3) unimodal distillation ($\mathcal{L}_{umd}$) and cross-modal correspondence distillation ($\mathcal{L}_{cmd}$), KL-style losses in multi-scale feature space that respectively transfer per-modality distributions and the cosine-similarity structure between modality pairs; and (4) modality-agnostic semantic distillation ($\mathcal{L}_{mad}$), a prediction-level KL loss from teacher to student segmentation maps. The total loss combines these with a supervised cross-entropy term. The paper's argument is that learning from an averaged, not fused, teacher is what prevents the student from inheriting unimodal bias.

What would settle it

One decisive check is to measure the PML teacher's own unimodal mIoU on event-only and lidar-only inputs on MUSES and compare it with the distilled student's. If the student matches or exceeds the teacher on those hard subsets, then the averaged teacher is not actually supplying the hard-modality knowledge and the claim that averaging is the right supervision for every subset is undermined; alternatively, replacing the averaged teacher with a learned-confidence-weighted fusion teacher and showing the student does no worse would falsify the claim that simple averaging is the key.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that unimodal bias in multimodal semantic segmentation can be removed by distilling from a teacher that never fuses modalities in a learned way. The teacher, trained by Parallel Multimodal Learning, feeds each modality through a shared backbone and simply averages the per-block features before the segmentation head; this preserves both modality-specific and shared information. The student is then trained with random anymodal dropout, and three losses align its multi-scale features with the teacher's: unimodal distillation transfers each single-modality distribution, cross-modal correspondence distillation matches the cosine-similarity structure between modality pairs, and modality-agnostic prediction distillation transfers the teacher's segmentation maps. The paper reports that distilling from directly fused features, as in prior fusion-based teachers, degrades performance, whereas the averaged-feature teacher yields balanced gains across all modality combinations, including large improvements on event and lidar inputs that prior methods nearly ignore.

Load-bearing premise

The whole scheme rests on the premise that a teacher built by simply averaging multimodal features, block by block, in a shared backbone is strong enough to supervise every unimodal and cross-modal combination; if that averaged teacher is itself biased toward easy modalities or only strong when all inputs are present, the distillation would transfer the bias rather than remove it.

Editorial extensions

If this is right

  • A single AnySeg student trained once can be deployed at inference time with any available subset of sensors, and it maintains higher mean mIoU across all combinations than prior modality-agnostic methods.
  • The largest relative gains appear exactly where unimodal bias is worst: event-only and lidar-only inputs improve by +16.40 and +9.80 mIoU on MUSES, and event and lidar gains are +16.81 and +18.62 on DELIVER.
  • Distilling from fused features is not merely suboptimal but actively harmful: adding fused-feature knowledge distillation drops mean mIoU from 40.23 to as low as 39.37, so the teacher representation must keep modality structure intact.
  • The framework is robust to scene degradations: on DELIVER RGB-D under cloud, fog, night, rain, and sensor-noise conditions, AnySeg stays above 53% mIoU in all settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to apply the same averaged-teacher plus unimodal and cross-modal distillation recipe to other fusion tasks, such as RGB-thermal or audio-visual segmentation, where one modality similarly dominates learning.
  • The PML teacher's simple averaging could be interpreted as a strong regularizer that prevents the teacher from learning modality-specific shortcuts, which may be why distillation from it transfers balanced knowledge; this suggests learned fusion may be unnecessary for teacher construction when the goal is robustness rather than peak fused accuracy.
  • The method's success on event and lidar hints that anymodal segmentation models could serve as a diagnostic tool: the gap between teacher and student on a given modality subset may quantify how much information that sensor actually contributes in a scene.
  • A testable extension is to replace random dropout with curriculum dropout that drops easy modalities first, then check whether the student gains further on hard modalities without sacrificing RGB performance.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes AnySeg, a two-stage framework for semantic segmentation that is robust to arbitrary combinations of visual modalities (RGB, depth, LiDAR, event). A multimodal teacher is first trained with a parallel modality learning (PML) strategy that averages per-block features across modalities, and is then frozen. The student is trained with random anymodal dropout and three distillation losses: unimodal distillation (Lumd) aligning per-modality features, cross-modal distillation (Lcmd) aligning pairwise cosine similarities, and modality-agnostic prediction-level distillation (Lmad). Experiments on MUSES and DELIVER report mean mIoU improvements of +6.37 and +6.15 over prior state-of-the-art. The paper includes extensive ablations of loss weights and teacher choices.

Significance. If the central claim holds, AnySeg would be a practical contribution to multimodal segmentation under sensor failure, with notable gains on weak modalities such as event and LiDAR. The paper provides a useful negative result about the ineffectiveness of distillation on fused features, and the PML teacher is a simple and parameter-free alternative to fusion-based teachers. However, the ablation study in Table 3 shows that the no-distillation baseline (Lsup with anymodal dropout) already achieves 39.29 mean mIoU on MUSES, only 0.94 below the full method, and that the proposed UMD and CMD losses add just 0.12 over Lsup+Lmad. This substantially weakens the attribution of the headline gains to the proposed distillation mechanism, which is the paper's central contribution. The significance of the work therefore depends on whether the authors can either demonstrate a larger marginal contribution of the distillation losses or reframe the contribution as a training strategy centered on anymodal dropout with modest distillation refinement.

major comments (4)
  1. [Table 3, Sec. 5] The ablation in Table 3 undercuts the causal claim that unimodal and cross-modal distillation are the key ingredients. The Lsup-only student, trained with random anymodal dropout but no teacher and none of the proposed losses, already achieves 39.29 mean mIoU on MUSES, which is 5.43 points above the Any2Seg baseline of 33.86. Adding Lmad brings the mean to 40.11, and the full method including Lumd and Lcmd reaches only 40.23. Thus the marginal contribution of the paper's two feature-level distillation losses (UMD and CMD) is just 0.12 mIoU over Lsup+Lmad, while the gain from anymodal dropout plus supervised training is 5.43 points. The paper never reports the analogous Lsup-only baseline on DELIVER, where the headline +6.15 gain is claimed. Without that control, the reader cannot tell whether the reported improvement over MAGIC on DELIVER is attributable to the proposed distillation or simply to random anymodal dropout. Please report the Lsup-only result on DELIVER and, if the marginal gains remain small, reframe the contribution accordingly.
  2. [Eq. 5, Sec. 3.2] Equation (5) is described as a cross-modal distillation loss based on KL divergence, but the normalized cosine similarity \tilde{S}(x,y) = (cos(x,y)+1)/2 is a scalar in [0,1] for each channel index j, and the sum over j is not normalized to yield a probability distribution. The statement that averaging the similarity scores across batch samples leads to non-negative values does not make these quantities a distribution over channels or over any finite set. Consequently, the logarithm of the ratio of such scalars does not define a valid KL divergence. The authors should either reformulate Eq. (5) as a proper divergence (for example, treating each channel's normalized similarity as a Bernoulli variable and summing a corresponding KL term) or explicitly present it as a heuristic distance objective with a justification for why it is effective.
  3. [Sec. 3.1, Tab. 7] The paper claims that the PML teacher, which averages multimodal features per block, is a strong supervisor for every unimodal and cross-modal combination, but it never reports the teacher's per-subset mIoU. Table 7 only compares student performance after distillation using different teachers, which is an indirect and confounded measure of teacher quality. If the teacher's per-modality branch features are weak because supervision is only applied to the final averaged output, then aligning the student to those features may not be the actual mechanism of improvement. Please report the teacher's mIoU for each input combination (F, E, L, FE, FL, EL, FEL) on MUSES, and similarly on DELIVER, to provide direct evidence that the averaged features are valid supervision targets for all subsets.
  4. [Tabs. 4-6, Sec. 5] The hyperparameters λ, α, and β are tuned on the evaluation benchmark itself (e.g., λ=50, α=5, β=10 for MUSES), and all reported results come from a single run without error bars or multiple seeds. Given that the full method's gain over Lsup+Lmad is only 0.12 mIoU on MUSES, the differences between configurations could easily be within run-to-run variance. Without variance estimates, the reader cannot assess whether the reported improvements are statistically meaningful, and the selection of hyperparameters on the test set raises a mild overfitting concern. Please provide at least three seeds with standard deviations for the main tables, or otherwise discuss the stability of the results.
minor comments (5)
  1. [Abstract and Sec. 1] The claim of being "the first framework for learning robust segmentor that can handle any combinations of visual modalities" is contradicted by the paper's own references to MAGIC and Any2Seg, which already target modality-agnostic segmentation over arbitrary input subsets. Consider rephrasing to "a distillation-based framework" or limiting the novelty claim to the specific distillation mechanism.
  2. [Eq. 4, Sec. 3.2] The loss Lumd sums over channels only, but the features are spatial maps; the spatial dimensions are not explicitly summed in the equation, leaving the loss definition ambiguous. Additionally, the text says "softmax to the teacher features g and log-softmax to the student features f" but writes log(\tilde{g}/\tilde{f}) with \tilde{f} in the denominator; if \tilde{f} is a log-softmax output, the ratio is not the standard KL form. Please clarify the normalization and the spatial summation.
  3. [Eq. 2, Sec. 3.2] Equation (2) uses the symbol Fma for the teacher encoder, while the teacher is denoted Fms elsewhere (e.g., Sec. 3.1 and Fig. 1). Please unify the notation.
  4. [Table 1, Sec. 4.1] The row "w.r.t SoTA" compares against Any2Seg's 33.86, but the text states the method "surpassing all state-of-the-art baselines." Since MAGIC and Any2Seg have different means, please specify which baseline is used for the margin in the table and in the abstract.
  5. [Sec. 5, Tab. 7 and text] The text says "distilling from fused features, as in MAGIC and CMNeXt, degrades performance," but Table 7 compares teacher models, not whether the student distills from fused features; the direct evidence for the fused-feature claim is Table 8. Please align the wording with the distinction between teacher design and distillation target.

Circularity Check

0 steps flagged · score 1.0 of 10

No material circularity: the losses, teacher, and benchmarks are independently defined and compared; self-citations are contextual, not load-bearing.

full rationale

The paper's derivation chain is empirical rather than definitional, and no step reduces to its own inputs. The PML teacher is trained with a standard cross-entropy loss (Eq. 1) on raw inputs plus ground truth, and its averaged-feature construction is a design choice, not a restatement of the student's target. The student losses (Eqs. 4, 5, 6, 7, 8) are separate KL-divergence and cross-entropy objectives defined between teacher features, student features, predictions, and ground truth; none of them is defined in terms of the final benchmark metric, and the final mean mIoU is not a fitted parameter. The reported gains over MAGIC and Any2Seg are external comparisons to published methods, and although those baselines were authored by overlapping researchers, their results are not cited as proof of the current method's correctness; they are empirical comparison points. The ablation in Table 3 does raise a legitimate attribution concern: the Lsup-only student with random anymodal dropout already reaches 39.29 mIoU on MUSES versus 40.23 for the full method, and no Lsup-only baseline is reported on DELIVER. However, this is a question of whether the distillation losses are the operative ingredient, not a circularity where a prediction is identical to its input by construction. Hyperparameter selection on the validation set is ordinary tuning, not a fitted-input-called-prediction pattern, because the reported headline numbers are benchmark results rather than values recovered from the tuning process. The paper also openly states a limitation (extra training cost) and does not import any uniqueness theorem or ansatz solely through self-citation. Overall, the central claim is self-contained against external benchmarks, so the appropriate finding is no significant circularity; the low score reflects only the presence of same-author prior work used for context and comparison, which is not load-bearing.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on an empirical training recipe: three tuned loss weights plus assumptions about the teacher, the masking strategy, and the benchmarks. No new physical entities are introduced, and there is no mathematical derivation to audit beyond the loss formulations.

free parameters (3)
  • λmad = 50
    Weight for modality-agnostic distillation loss Lmad; selected by ablation on MUSES (Tab.4 and Tab.11). The final results use λ=50.
  • α = 5
    Weight for unimodal distillation loss Lumd; Tab.5 shows α=5 as the selected balance because higher values improve RGB but hurt Event and LiDAR.
  • β = 10
    Weight for cross-modal distillation loss Lcmd; Tab.6 shows β=10 gives the best mean mIoU of 40.23 on MUSES, and the method is sensitive to this choice.
assumptions (4)
  • domain assumption Feature-level averaging of all modalities in a shared backbone produces a teacher that is strong for both unimodal and multimodal distillation.
    Sec.3.1 and Fig.2. Tab.7 compares teacher choices but does not establish why averaging is the right supervision for every modality subset.
  • domain assumption Randomly masking modalities, keeping at least one modality, accurately simulates real-world sensor-failure patterns.
    Sec.3.2. The paper does not model any real failure distribution, only random dropout.
  • standard math Softmax-normalized features and cosine-similarity values can be treated as probability distributions for KL divergence.
    Eq.4 uses softmax, which is valid for distributions, but Eq.5 uses normalized cosine similarities that are not normalized across channels and do not sum to one, so the KL interpretation is not rigorous.
  • domain assumption MUSES and DELIVER evaluation protocols are representative of anymodal robustness.
    Sec.4. Only two benchmarks are used, and no real sensor-dropout dataset is included.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning Robust Anymodal Segmentor with Unimodal and Cross-modal Distillation." pith.science (2026). https://pith.science/paper/FZIMGZPK

@misc{pith2026241117141,
  author       = {Pith},
  title        = {Pith review of: Learning Robust Anymodal Segmentor with Unimodal and Cross-modal Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FZIMGZPK}},
  note         = {Machine review of arXiv:2411.17141}
}
read the original abstract

Simultaneously using multimodal inputs from multiple sensors to train segmentors is intuitively advantageous but practically challenging. A key challenge is unimodal bias, where multimodal segmentors over rely on certain modalities, causing performance drops when others are missing, common in real world applications. To this end, we develop the first framework for learning robust segmentor that can handle any combinations of visual modalities. Specifically, we first introduce a parallel multimodal learning strategy for learning a strong teacher. The cross-modal and unimodal distillation is then achieved in the multi scale representation space by transferring the feature level knowledge from multimodal to anymodal segmentors, aiming at addressing the unimodal bias and avoiding over-reliance on specific modalities. Moreover, a prediction level modality agnostic semantic distillation is proposed to achieve semantic knowledge transferring for segmentation. Extensive experiments on both synthetic and real-world multi-sensor benchmarks demonstrate that our method achieves superior performance.

Figures

Figures reproduced from arXiv: 2411.17141 by the authors.

Figure 1
Figure 1. (a) Overall of AnySeg with a two-stage training strategy: the multimodal teacher is first [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. PML for learning a strong multimodal segmentor [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Qualitative comparison on MUSES [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: TSNE visualization of multi-modal features (RGB-R, Depth-D, Event-E, and LiDAR-L) [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Additional qualitative comparisons that highlight our method’s robustness under diverse [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: t-SNE visualization of multi-modal features (RGB, Depth, Event, and LiDAR) extracted by [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. BiXFormer: A Robust Framework for Maximizing Modality Effectiveness in Multi-Modal Semantic Segmentation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A multi-modal semantic segmentation framework that processes RGB and non-RGB sensors separately, matches labels in two stages, and aligns cross-modal queries with a VAE refiner.

  2. Partial CLIP is Enough: Chimera-Seg for Zero-shot Semantic Segmentation

    cs.CV 2025-06 conditional novelty 5.0 of 10

    A partial, frozen CLIP block mounted on a segmentation backbone, plus selective distillation to CLIP's CLS token, improves zero-shot semantic segmentation by about 1 hIoU point on two datasets.

  3. Customize Segment Anything Model for Multi-Modal Semantic Segmentation with Mixture of LoRA Experts

    cs.CV 2024-12 reject novelty 5.0 of 10

    MLE-SAM adapts SAM2 with modality-specific LoRA experts and a routing mechanism, reporting state-of-the-art multi-modal segmentation results on DELIVER, MUSES, and MCubeS, although the comparisons are confounded by di...

  4. MLLMs are Deeply Affected by Modality Bias

    cs.AI 2025-05 conditional novelty 4.0 of 10

    A position paper with a case study showing that multimodal LLMs rely on language priors and underuse visual input, together with a research roadmap and calls for balanced training.

  5. RMMSS: Towards Advanced Robust Multi-Modal Semantic Segmentation with Hybrid Prototype Distillation and Feature Selection

    cs.CV 2025-05 conditional novelty 4.0 of 10

    RMMSS improves missing-modality segmentation mIoU by up to 3.89% on public benchmarks while keeping full-modality mIoU within 0.1% of a full-modality teacher.

  6. Reducing Unimodal Bias in Multi-Modal Semantic Segmentation with Multi-Scale Functional Entropy Regularization

    cs.CV 2025-05 reject novelty 4.0 of 10

    A plug-and-play functional-entropy regularizer applied at feature and prediction scales is claimed to reduce unimodal bias in multi-modal semantic segmentation, with large mIoU gains on MUSES, DELIVER, and MCubeS with...

  7. Segment Any RGB-Thermal Model with Language-aided Distillation

    cs.CV 2025-05 conditional novelty 3.0 of 10

    SARTM fine-tunes SAM2 with LoRA and distills CLIP text knowledge to improve RGB-thermal semantic segmentation, reporting top mIoU on PST900, MFNet, and FMB.

Reference graph

Works this paper leans on

37 extracted references · 28 canonical work pages · cited by 7 Pith papers

  1. [1]

    360sfuda++: Towards source-free uda for panoramic segmentation by learning reliable category prototypes,

    X. Zheng, P. Y . Zhou, A. V . Vasilakos, and L. Wang, “360sfuda++: Towards source-free uda for panoramic segmentation by learning reliable category prototypes,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  2. [2]

    Omnibind: Teach to build unequal-scale modality interaction for omni-bind of all,

    Y . Lyu, X. Zheng, D. Kim, and L. Wang, “Omnibind: Teach to build unequal-scale modality interaction for omni-bind of all,” arXiv preprint arXiv:2405.16108, 2024

    arXiv 2024

  3. [3]

    Transformer-cnn cohort: Semi-supervised semantic segmentation by the best of both students,

    X. Zheng, Y . Luo, C. Fu, K. Liu, and L. Wang, “Transformer-cnn cohort: Semi-supervised semantic segmentation by the best of both students,” in 2024 IEEE International Conference on Robotics and Automation (ICRA) , pp. 11147–11154, IEEE, 2024

    work page 2024

  4. [4]

    Unibind: Llm-augmented unified and balanced representation space to bind them all,

    Y . Lyu, X. Zheng, J. Zhou, and L. Wang, “Unibind: Llm-augmented unified and balanced representation space to bind them all,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26752–26762, 2024

    2024

  5. [5]

    Cmx: Cross-modal fusion for rgb- x semantic segmentation with transformers,

    J. Zhang, H. Liu, K. Yang, X. Hu, R. Liu, and R. Stiefelhagen, “Cmx: Cross-modal fusion for rgb- x semantic segmentation with transformers,” IEEE Transactions on intelligent transportation systems, 2023

    work page 2023

  6. [6]

    Learning modality-agnostic representation for semantic segmentation from any modalities,

    X. Zheng, Y . Lyu, and L. Wang, “Learning modality-agnostic representation for semantic segmentation from any modalities,” in Proceedings of the European Conference on Computer Vision, 2024

    work page 2024

  7. [7]

    Centering the value of every modality: Towards efficient and resilient modality-agnostic semantic segmentation,

    X. Zheng, Y . Lyu, J. Zhou, and L. Wang, “Centering the value of every modality: Towards efficient and resilient modality-agnostic semantic segmentation,” inProceedings of the European Conference on Computer Vision, 2024

    work page 2024

  8. [8]

    Fourier prompt tuning for modality-incomplete scene segmentation,

    R. Liu, J. Zhang, K. Peng, Y . Chen, K. Cao, J. Zheng, M. S. Sarfraz, K. Yang, and R. Stiefelha- gen, “Fourier prompt tuning for modality-incomplete scene segmentation,” in IEEE Intelligent V ehicles Symposium, pp. 961–968, 2024

    work page 2024

Show all 37 references
  1. [9]

    Eventbind: Learning a unified representation to bind them all for event-based open-world understanding,

    J. Zhou, X. Zheng, Y . Lyu, and L. Wang, “Eventbind: Learning a unified representation to bind them all for event-based open-world understanding,” in Proceedings of the European Conference on Computer Vision, 2024

    2024

  2. [10]

    Eventdance: Unsupervised source-free cross-modal adaptation for event-based object recognition,

    X. Zheng and L. Wang, “Eventdance: Unsupervised source-free cross-modal adaptation for event-based object recognition,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17448–17458, 2024

    2024

  3. [11]

    Mseg3d: Multi-modal 3d semantic segmentation for autonomous driving,

    J. Li, H. Dai, H. Han, and Y . Ding, “Mseg3d: Multi-modal 3d semantic segmentation for autonomous driving,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21694–21704, 2023

    2023

  4. [12]

    Memorysam: Memorize modalities and semantics with segment anything model 2 for multi-modal semantic segmentation,

    C. Liao, X. Zheng, Y . Lyu, H. Xue, Y . Cao, J. Wang, K. Yang, and X. Hu, “Memorysam: Memorize modalities and semantics with segment anything model 2 for multi-modal semantic segmentation,” arXiv preprint arXiv:2503.06700, 2025

    2025 arXiv

  5. [13]

    Benchmarking multi-modal semantic segmentation under sensor failures: Missing and noisy modality robustness,

    C. Liao, K. Lei, X. Zheng, J. Moon, Z. Wang, Y . Wang, D. P. Paudel, L. Van Gool, and X. Hu, “Benchmarking multi-modal semantic segmentation under sensor failures: Missing and noisy modality robustness,” arXiv preprint arXiv:2503.18445, 2025

    2025 arXiv

  6. [14]

    Both style and distortion matter: Dual- path unsupervised domain adaptation for panoramic semantic segmentation,

    X. Zheng, J. Zhu, Y . Liu, Z. Cao, C. Fu, and L. Wang, “Both style and distortion matter: Dual- path unsupervised domain adaptation for panoramic semantic segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 1285–1295, 2023

    2023

  7. [15]

    Semantics distortion and style matter: Towards source-free uda for panoramic segmentation,

    X. Zheng, P. Zhou, A. V . Vasilakos, and L. Wang, “Semantics distortion and style matter: Towards source-free uda for panoramic segmentation,” inProceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition , pp. 27885–27895, 2024

    2024

  8. [16]

    Reducing unimodal bias in multi-modal semantic segmentation with multi-scale functional entropy regularization,

    X. Zheng, Y . Lyu, L. Jiang, D. P. Paudel, L. Van Gool, and X. Hu, “Reducing unimodal bias in multi-modal semantic segmentation with multi-scale functional entropy regularization,” arXiv preprint arXiv:2505.06635, 2025. 10

    2025 arXiv

  9. [17]

    De- livering arbitrary-modal semantic segmentation,

    J. Zhang, R. Liu, H. Shi, K. Yang, S. Reiß, K. Peng, H. Fu, K. Wang, and R. Stiefelhagen, “De- livering arbitrary-modal semantic segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1136–1147, 2023

    2023

  10. [18]

    Muses: The multi-sensor semantic perception dataset for driving under uncertainty,

    T. Brödermann, D. Bruggemann, C. Sakaridis, K. Ta, O. Liagouris, J. Corkill, and L. Van Gool, “Muses: The multi-sensor semantic perception dataset for driving under uncertainty,” in Pro- ceedings of the European Conference on Computer Vision , 2024

    2024

  11. [19]

    Unveiling the potential of segment anything model 2 for rgb-thermal semantic segmentation with language guidance,

    J. Zhao, F. Teng, K. Luo, G. Zhao, Z. Li, X. Zheng, and K. Yang, “Unveiling the potential of segment anything model 2 for rgb-thermal semantic segmentation with language guidance,” arXiv preprint arXiv:2503.02581, 2025

    2025 arXiv

  12. [20]

    Hrfuser: A multi-resolution sensor fusion architecture for 2d object detection,

    T. Broedermann, C. Sakaridis, D. Dai, and L. Van Gool, “Hrfuser: A multi-resolution sensor fusion architecture for 2d object detection,” in IEEE International Conference on Intelligent Transportation Systems, pp. 4159–4166, 2023

    2023

  13. [21]

    Mmanet: Margin-aware distillation and modality-aware regular- ization for incomplete multimodal learning,

    S. Wei, C. Luo, and Y . Luo, “Mmanet: Margin-aware distillation and modality-aware regular- ization for incomplete multimodal learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20039–20049, 2023

    2023

  14. [22]

    Abmdrnet: Adaptive-weighted bi- directional modality difference reduction network for rgb-t semantic segmentation,

    Q. Zhang, S. Zhao, Y . Luo, D. Zhang, N. Huang, and J. Han, “Abmdrnet: Adaptive-weighted bi- directional modality difference reduction network for rgb-t semantic segmentation,” in Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 2633–2642, 2021

    2021

  15. [23]

    Bev-guided multi-modality fusion for driving perception,

    Y . Man, L.-Y . Gui, and Y .-X. Wang, “Bev-guided multi-modality fusion for driving perception,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 21960–21969, 2023

    2023

  16. [24]

    Multimodal token fusion for vision transformers,

    Y . Wang, X. Chen, L. Cao, W. Huang, F. Sun, and Y . Wang, “Multimodal token fusion for vision transformers,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12186–12195, 2022

    2022

  17. [25]

    Spatial information guided convolution for real-time rgbd semantic segmentation,

    L.-Z. Chen, Z. Lin, Z. Wang, Y .-L. Yang, and M.-M. Cheng, “Spatial information guided convolution for real-time rgbd semantic segmentation,”IEEE Transactions on Image Processing, vol. 30, pp. 2313–2324, 2021

    2021

  18. [26]

    Customize segment anything model for multi- modal semantic segmentation with mixture of lora experts,

    C. Zhu, B. Xiao, L. Shi, S. Xu, and X. Zheng, “Customize segment anything model for multi- modal semantic segmentation with mixture of lora experts,” arXiv preprint arXiv:2412.04220, 2024

    2024 arXiv

  19. [27]

    Critical learning periods for multisensory integration in deep networks,

    M. Kleinman, A. Achille, and S. Soatto, “Critical learning periods for multisensory integration in deep networks,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24296–24305, 2023

    2023

  20. [28]

    Balanced multimodal learning via on-the-fly gradient modulation,

    X. Peng, Y . Wei, A. Deng, D. Wang, and D. Hu, “Balanced multimodal learning via on-the-fly gradient modulation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8238–8247, 2022

    2022

  21. [29]

    Modality competition: What makes joint training of multi-modal network fail in deep learning? (provably),

    Y . Huang, J. Lin, C. Zhou, H. Yang, and L. Huang, “Modality competition: What makes joint training of multi-modal network fail in deep learning? (provably),” in International Conference on Machine Learning, pp. 9226–9259, 2022

    2022

  22. [30]

    Understanding unimodal bias in multimodal deep linear networks,

    Y . Zhang, P. E. Latham, and A. M. Saxe, “Understanding unimodal bias in multimodal deep linear networks,” in International Conference on Machine Learning , 2024

    2024

  23. [31]

    Multi-modal 3d object detection in autonomous driving: a survey,

    Y . Wang, Q. Mao, H. Zhu, J. Deng, Y . Zhang, J. Ji, H. Li, and Y . Zhang, “Multi-modal 3d object detection in autonomous driving: a survey,”International Journal of Computer Vision, pp. 1–31, 2023

    2023

  24. [32]

    Missing modality robustness in semi-supervised multi-modal semantic segmentation,

    H. Maheshwari, Y .-C. Liu, and Z. Kira, “Missing modality robustness in semi-supervised multi-modal semantic segmentation,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1020–1030, 2024. 11

    2024

  25. [33]

    Robust multimodal learning with missing modalities via parameter-efficient adaptation,

    M. K. Reza, A. Prater-Bennette, and M. S. Asif, “Robust multimodal learning with missing modalities via parameter-efficient adaptation,” arXiv preprint arXiv:2310.03986, 2023

    2023 arXiv

  26. [34]

    Redundancy-adaptive multimodal learning for imperfect data,

    M. Chen, J. Yao, L. Xing, Y . Wang, Y . Zhang, and Y . Wang, “Redundancy-adaptive multimodal learning for imperfect data,” arXiv preprint arXiv:2310.14496, 2023

    2023 arXiv

  27. [35]

    Multi-modality guidance network for missing modality inference,

    Z. Zhao, H. Palani, T. Liu, L. Evans, and R. Toner, “Multi-modality guidance network for missing modality inference,” arXiv preprint arXiv:2309.03452, 2023

    2023 arXiv

  28. [36]

    Learnable cross- modal knowledge distillation for multi-modal learning with missing modality,

    H. Wang, C. Ma, J. Zhang, Y . Zhang, J. Avery, L. Hull, and G. Carneiro, “Learnable cross- modal knowledge distillation for multi-modal learning with missing modality,” in International Conference on Medical Image Computing and Computer-Assisted Intervention , pp. 216–226, 2023

    2023

  29. [37]

    Segformer: Simple and efficient design for semantic segmentation with transformers,

    E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo, “Segformer: Simple and efficient design for semantic segmentation with transformers,” Advances in neural information processing systems, vol. 34, pp. 12077–12090, 2021. 12 A Appendix A.1 Implementation Details. ...

    2021

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.