Pith. sign in

REVIEW 4 major objections 4 minor 87 references

BiXFormer: A Robust Framework for Maximizing Modality Effectiveness in Multi-Modal Semantic Segmentation

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that reformulating multi-modal semantic segmentation as mask-level classification, with separate RGB and X backbones and a two-step label-matching scheme, yields state-of-the-art mIoU (+2.75% on DELIVER, +22.74% on MUSES)…

desk verdict A genuinely novel two-step modality-matching idea with careful ablations, but the robustness claims rest on an underspecified zero-filling protocol and the SOTA numbers are selective. read the letter →

arxiv 2506.03675 v1 pith:EAO5HDOM submitted 2025-06-04 cs.CV

Jialei Chen , Xu Zheng , Danda Pani Paudel , Luc Van Gool , Hiroshi Murase , Daisuke Deguchi This is my paper
classification cs.CV
keywords multi-modalsemanticsegmentationmask-levelclassificationquerymatchingmissingmodalityrobustnesscross-modalityalignmentRGB-XUnified
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that feature fusion and knowledge distillation are not the right way to combine RGB with other sensors for semantic segmentation, and that a better way is to let each modality produce its own mask-level predictions and then assign labels through a coordinated two-step matching process. It presents BiXFormer, which processes RGB and all non-RGB (X) modalities in separate backbones and combines their outputs only at the prediction stage. Labels are assigned by Unified Modality Matching: Modality Agnostic Matching first gives labels to the globally most suitable queries across both modalities, and Complementary Matching then assigns leftover labels to remaining queries within each modality so that no single sensor hogs all classes. Cross Modality Alignment refines the weaker second-step queries using the stronger first-step ones. The reported result is mIoU gains over prior methods of +2.75% on the synthetic DELIVER benchmark and +22.74% on the real-world MUSES benchmark, with the design explicitly aimed at staying accurate when some sensors are missing.

What carries the argument

The load-bearing mechanism is Unified Modality Matching (UMM), a two-step Hungarian-style assignment over modality-specific query predictions. In step one, Modality Agnostic Matching (MAM) solves a global matching over all $2L$ predictions, assigning each ground-truth label to whichever query, RGB or X, has the lowest combined classification and mask loss. In step two, Complementary Matching (CM) splits the leftover queries by modality and re-solves matching against the labels each modality has not yet received, so that labels not claimed globally are still assigned within the modality that has free capacity. The final label of every query is the component-wise maximum of the matches found in MAM and CM, and the segmentation loss (Eq. 5) is computed under this merged matching. Cross Modality Alignment (CMA) then refines the suboptimal queries from CM by aligning them with the optimal queries of the same class from MAM, using a VAE refiner and an alignment loss combining MSE and MMD; this is what the paper credits with preserving modality-specific separation while sharpening category-level consistency.

What would settle it

Run the published evaluation protocol, then swap the zero-filled tensor for a different out-of-distribution filler, such as random Gaussian noise or a constant-gray image of the same shape, when a sensor is missing. If the model's missing-modality mIoU collapses or changes sharply under the alternative filler, the claimed robustness depends on the specific zero input rather than on the matching mechanism; a decisive version of this test is to train a comparison model with random modality dropout during training and see whether it beats the zero-filling model when a sensor is absent.

Watch

Extended reading notes

Core claim

The central discovery, as stated by the authors, is that multi-modal semantic segmentation can be reformulated as a mask-level classification task with modality-specialized query matching, and that doing so outperforms both feature-fusion and knowledge-distillation pipelines. BiXFormer uses two backbones: one ImageNet-pretrained for RGB, and one shared backbone for all X modalities with a modified first convolution layer that accepts the concatenated non-RGB channels. The two streams refine their features through a shared transformer encoder and decoder with modality-specific queries, avoiding any fusion of feature maps. Unified Modality Matching then assigns ground-truth labels at the query level in two steps: a global modality-agnostic matching that is optimal over all $2L$ predictions, followed by complementary matching within each modality that recycles unmatched labels to unassigned queries. The final matching takes the per-query maximum over both steps, and the segmentation loss is computed under that merged assignment. Cross Modality Alignment transfers information from the strong MAM-matched queries to the weak CM-matched queries through a VAE-based refiner, with an MMD-plus-MSE alignment loss that increases modality distance while decreasing class distance. The paper's own ablations show that both matching steps and the refiner contribute to the reported gains.

Load-bearing premise

The load-bearing premise is that replacing a missing non-RGB sensor with a zero tensor $0^{3\times H\times W}$ at test time is an adequate substitute that does not push the shared X backbone far outside its training distribution, since the paper trains only on complete modality batches and reports no missing-modality augmentation during training.

Editorial extensions

If this is right

  • If the central claim is correct, feature fusion and knowledge distillation are not prerequisites for strong multi-modal segmentation; two separate backbones plus prediction-stage label matching can match or beat them, as the reported MUSES and DELIVER comparisons show.
  • The two-backbone design extends to any non-RGB modality, such as depth, event streams, or LiDAR, by concatenating the X channels and modifying only the first convolution layer, so adding a new sensor does not require a new backbone.
  • The learned label allocation is category-dependent rather than fixed: the paper's matching visualizations show some classes going to RGB and others to X, so the framework adapts the division of labor to each dataset's sensor characteristics.
  • Because the final prediction combines modality-specific masks rather than a fused feature map, the model can keep producing meaningful segmentations when one sensor fails, as long as the zero-filled placeholder stays within the network's learned operating range.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper only tests missing sensors by zero-filling at inference time; a natural extension is to train with randomly dropped modalities and check whether the robustness numbers improve further, which would separate the benefit of the matching design from the model's tolerance of the zero input.
  • Because UMM operates on labels and masks rather than on modality-specific feature statistics, the same two-step assignment could transfer to other query-based dense-prediction tasks with heterogeneous inputs, such as RGB-thermal or RGB-event panoptic segmentation.
  • The category-dependent division of labor that UMM discovers suggests a testable prediction: on datasets where a sensor's noise characteristics change, such as rainy versus clear weather, the learned label allocation should shift toward the modality that is more reliable per class.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This manuscript proposes BiXFormer, a mask-classification framework for multi-modal semantic segmentation that processes RGB and non-RGB ('X') modalities with two separate backbones, assigns ground-truth labels to modality-specific queries through a two-step Unified Modality Matching (Modality Agnostic Matching followed by Complementary Matching), and refines the weaker complementary-matched queries with a VAE-based Cross Modality Alignment. The authors report mean mIoU improvements of +22.74 on MUSES and +2.75 on DELIVER over prior methods, and claim robustness to missing modalities by replacing absent sensors with zero tensors at inference. Ablations on MUSES support the contributions of the two backbones, UMM, and CMA.

Significance. If the reported numbers are taken at face value, BiXFormer would be a strong contribution to multi-modal segmentation: it avoids feature fusion, lets each modality specialize, and the two-step matching is a principled way to balance modality contributions while providing graceful degradation when sensors fail. The paper is also unusually thorough in ablation coverage (module ablations, matching ablations, refiner variants, backbone configurations, initialization). However, the headline comparisons are currently not controlled for model capacity, the missing-modality evaluation relies on a zero-filling input pattern that is never seen in training, and on the full-modality DELIVER setting the method is below MAGIC. These issues must be resolved before the strength of the claim can be assessed.

major comments (4)
  1. [Table I; Table II] The central +22.74 mean-mIoU claim is based on a capacity-confounded comparison. In Table I, the Ours rows use ResNet-18/34/50 (28.91-58.94M parameters) or a double Seg-B0 (17.70M), while CMNeXt and CMX use a single Seg-B0 (10.30-10.32M) and MAGIC/Any2Seg use 24.74M. The Ours Seg-B0 row is closer in scale but still roughly 1.7 times the baseline parameters. Please provide matched-capacity comparisons (e.g., baselines re-run with the same backbone and roughly the same parameter count, or Ours with a single/shared backbone of comparable capacity) so that the gain can be attributed to the method. Without this, the abstract's '+22.74%' is not a supported statement.
  2. [Section III-D; Methods paragraph] The robustness claim rests on an out-of-distribution test input. Section III-D states that a missing sub-modality is replaced by 0^{3xHxW}, while the Methods section states that during training each batch contains samples from all modalities. No modality-dropout augmentation is described. Consequently, the zero-filled inputs used in the F/E/L/FE/FL/EL columns of Tables I and II were never encountered during training, and the shared transformer encoder/decoder and BatchNorm statistics are optimized on fully populated inputs. Please add training with random modality dropout, or at minimum an analysis comparing zero-filling to an imputation baseline and to a model explicitly trained with missing modalities. As written, the missing-modality numbers measure robustness to a specific synthetic pattern, not to missing sensors generally.
  3. [Table II, RDEL column] On the full-modality DELIVER setting (RDEL), MAGIC achieves 63.40 mIoU while Ours achieves 58.29. The +2.75 gain reported in the abstract is thus only on the mean over all modality subsets, and the method does not improve over the prior art when all four modalities are available. The claim of 'significant improvements over the prior arts' should be qualified accordingly, and the full-modality comparison should be part of the headline statement.
  4. [Table III] The paper's contribution statement (III) claims that BiXFormer 'surpasses existing multimodal segmentation methods,' but on NYUv2 (Table III) Ours (MiT-B4) reaches 54.5 mIoU, which is below CMNext (MiT-B4) at 56.9 mIoU. The method is competitive on this standard benchmark but not state-of-the-art; please either adjust the claim or analyze why the proposed modules do not transfer to the RGB-D setting.
minor comments (4)
  1. [Section III-A] The text 'To be spcific' should read 'To be specific'.
  2. [Eq. (4)] The expression in Eq. (4) involving 'L_r X_{i=1}' appears to be a malformed summation; it should be written as a sum over the number of remaining unassigned predictions.
  3. [Section IV-C] The ablation text refers to 'integrating CMA (T + U + R)', but Table V labels the same configuration as 'T + U + A'; the notation should be made consistent.
  4. [Table II caption] The caption says 'using ResNet-34 as backbone model' but does not state whether CMNeXt and MAGIC were re-trained with ResNet-34 or use their original backbones; please clarify, as this determines whether the DELIVER comparison is capacity-controlled.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the framework is trained with standard losses and evaluated on external benchmarks, and no claimed result reduces by construction to its inputs or to a self-citation chain.

full rationale

There is no significant circularity in this paper. The reported gains are measured mIoU differences on external benchmarks (MUSES, DELIVER, NYUv2) rather than quantities fitted from those same benchmarks. The matching procedures UMM and CMA are training-time assignment and alignment mechanisms defined by Eqs. (2)-(6), with losses L_seg and L_a in Eq. (7); these are standard supervised objectives and not predictions derived from the fitted parameters. The method does invoke Mask2Former-style mask classification and Hungarian matching from prior work [31,32], but this is external, established machinery and not a self-citation carrying the paper's central claim. Several baselines and related works are authored by the present authors' group (e.g., MAGIC, Any2Seg, CMNeXt), but these are used as comparison methods or related context, not as load-bearing justification for the proposed architecture's correctness. The paper's limitation regarding missing modalities - using zero-filled tensors 0^{3xHxW} for absent sub-modalities at test time while training on complete modality batches - is an empirical robustness concern about whether the evaluation protocol matches the claimed generalization, not a circular argument. Therefore the derivation and experimental claims are self-contained, and the appropriate circularity score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

No physical or conceptual entities are introduced. The free parameters are standard deep-learning hyperparameters, most of which are underreported. The domain assumptions concern channel concatenation, ImageNet initialization, and zero-filling of missing sensors; the last is the most fragile because it assumes out-of-distribution test-time inputs are harmless.

free parameters (5)
  • Learning rate and schedule = 6e-5, polynomial decay power 0.9, 200 epochs, 10-epoch warm-up
    Chosen by hand in Section IV-A; the training dynamics and final accuracy depend on these values, yet no sensitivity analysis is provided.
  • Batch size and crop size = 16, 1024x1024
    Fixed across benchmarks in Section IV-A; not varied or justified.
  • Number of queries per modality L = not reported
    L controls matching capacity in UMM and CMA; without it, the architecture cannot be reproduced exactly.
  • MMD and MSE combination weights in L_r = not reported
    The alignment loss in Eq. 6 is described only as a combination of MSE and MMD without weights, leaving the strength of CMA unspecified.
  • VAE encoder and decoder dimensions = not reported
    Section IV-A says both are four layers of Linear-Layernorm-ReLU, but hidden sizes and latent dimensionality are omitted.
assumptions (4)
  • standard math Hungarian matching produces a globally optimal assignment for a fixed matching cost.
    Invoked in Eqs. 2 and 4 for MAM and CM; standard combinatorial optimization, but the practical optimum depends on the matching cost hyperparameters.
  • domain assumption Concatenating all non-RGB modalities into one channel stack and processing them with a single shared backbone preserves enough information.
    Section III-A concatenates I_i for i>=1 along the channel dimension; the paper provides no evidence that this loses less than per-modality backbones.
  • ad hoc to paper A backbone pretrained on ImageNet remains effective when its first convolutional layer is randomly re-initialized for X-channel input.
    Section III-A adopts this design to reuse pretraining; Table XIII shows ImageNet init helps, but the random first layer is not independently studied.
  • domain assumption Zero-filling a missing sub-modality is a valid test-time input for the shared X backbone.
    Section III-D replaces missing sub-modalities with 0^(3xHxW); this input distribution is never seen during training.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BiXFormer: A Robust Framework for Maximizing Modality Effectiveness in Multi-Modal Semantic Segmentation." pith.science (2026). https://pith.science/paper/EAO5HDOM

@misc{pith2026250603675,
  author       = {Pith},
  title        = {Pith review of: BiXFormer: A Robust Framework for Maximizing Modality Effectiveness in Multi-Modal Semantic Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EAO5HDOM}},
  note         = {Machine review of arXiv:2506.03675}
}
read the original abstract

Utilizing multi-modal data enhances scene understanding by providing complementary semantic and geometric information. Existing methods fuse features or distill knowledge from multiple modalities into a unified representation, improving robustness but restricting each modality's ability to fully leverage its strengths in different situations. We reformulate multi-modal semantic segmentation as a mask-level classification task and propose BiXFormer, which integrates Unified Modality Matching (UMM) and Cross Modality Alignment (CMA) to maximize modality effectiveness and handle missing modalities. Specifically, BiXFormer first categorizes multi-modal inputs into RGB and X, where X represents any non-RGB modalities, e.g., depth, allowing separate processing for each. This design leverages the well-established pretraining for RGB, while addressing the relative lack of attention to X modalities. Then, we propose UMM, which includes Modality Agnostic Matching (MAM) and Complementary Matching (CM). MAM assigns labels to features from all modalities without considering modality differences, leveraging each modality's strengths. CM then reassigns unmatched labels to remaining unassigned features within their respective modalities, ensuring that each available modality contributes to the final prediction and mitigating the impact of missing modalities. Moreover, to further facilitate UMM, we introduce CMA, which enhances the weaker queries assigned in CM by aligning them with optimally matched queries from MAM. Experiments on both synthetic and real-world multi-modal benchmarks demonstrate the effectiveness of our method, achieving significant improvements in mIoU of +2.75% and +22.74% over the prior arts.

Figures

Figures reproduced from arXiv: 2506.03675 by the authors.

Figure 1
Figure 1. Existing methods (top) rely on feature fusion and knowl [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 4
Figure 4. How is hungarian matching applied in segmentation. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Overview of cross modality alignment where we only [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figures from the paper (3 more)
Figure 6
Figure 6. Figure 6: Label matching distribution under UMM. is category-dependent, with fence and traffic signs favoring the X modality, while road and pole are predominantly assigned to RGB. This suggests that UMM effectively leverages the strengths of RGB and non-RGB modality rather than…
Figure 7
Figure 7. Figure 7: Prediction of RGB and non-RGB modality under different conditions. [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Predictions visualization compared with MAGIC [ [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

87 extracted references · 57 canonical work pages

  1. [1]

    Fully convolu- tional networks for semantic segmentation,

    J. Long, E. Shelhamer, and T. Darrell, “Fully convolu- tional networks for semantic segmentation,” inCVPR, 2015

    2015

  2. [2]

    Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,

    L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,”TPAMI, 2017

    2017

  3. [3]

    Encoder-decoder with atrous separable con- volution for semantic image segmentation,

    L.-C. Chen, Y . Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-decoder with atrous separable con- volution for semantic image segmentation,” inECCV, 2018

    2018

  4. [4]

    Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers,

    S. Zheng, J. Lu, H. Zhao, X. Zhu, Z. Luo, Y . Wang, Y . Fu, J. Feng, T. Xiang, P. H. Torret al., “Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers,” inCVPR, 2021. JOURNAL OF LATEX CLASS FILES, VOL. 18, NO. 9, SEPTEMBER 2020 11

    2021

  5. [5]

    Contour knowledge-aware perception learning for seman- tic segmentation,

    C. You, L. Jiao, L. Li, X. Liu, F. Liu, W. Ma, and S. Yang, “Contour knowledge-aware perception learning for seman- tic segmentation,”IEEE Transactions on Circuits and Systems for Video Technology, 2024

    2024

  6. [6]

    Omnisam: Omnidirectional seg- ment anything model for uda in panoramic semantic segmentation,

    D. Zhong, X. Zheng, C. Liao, Y . Lyu, J. Chen, S. Wu, L. Zhang, and X. Hu, “Omnisam: Omnidirectional seg- ment anything model for uda in panoramic semantic segmentation,”arXiv preprint arXiv:2503.07098, 2025

    arXiv 2025

  7. [7]

    Muses: The multi-sensor semantic perception dataset for driving under uncertainty,

    T. Br ¨odermann, D. Bruggemann, C. Sakaridis, K. Ta, O. Liagouris, J. Corkill, and L. Van Gool, “Muses: The multi-sensor semantic perception dataset for driving under uncertainty,” inECCV, 2024

    2024

  8. [8]

    Delivering arbitrary- modal semantic segmentation,

    J. Zhang, R. Liu, H. Shi, K. Yang, S. Reiß, K. Peng, H. Fu, K. Wang, and R. Stiefelhagen, “Delivering arbitrary- modal semantic segmentation,” inCVPR, 2023

    2023

Show all 87 references
  1. [9]

    Indoor segmentation and support inference from rgbd images,

    N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, “Indoor segmentation and support inference from rgbd images,” in Computer Vision–ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part V 12. Springer, 2012, pp. 746–760

    2012

  2. [10]

    Centering the value of every modality: Towards efficient and resilient modality-agnostic semantic segmentation,

    X. Zheng, Y . Lyu, J. Zhou, and L. Wang, “Centering the value of every modality: Towards efficient and resilient modality-agnostic semantic segmentation,” inECCV, 2024

    2024

  3. [11]

    Learning robust anymodal segmentor with unimodal and cross-modal distillation,

    X. Zheng, H. Xue, J. Chen, Y . Yan, L. Jiang, Y . Lyu, K. Yang, L. Zhang, and X. Hu, “Learning robust anymodal segmentor with unimodal and cross-modal distillation,” arXiv preprint arXiv:2411.17141, 2024

    2024 arXiv

  4. [12]

    Cafuser: Condition-aware multimodal fusion for robust semantic perception of driving scenes,

    T. Br ¨odermann, C. Sakaridis, Y . Fu, and L. Van Gool, “Cafuser: Condition-aware multimodal fusion for robust semantic perception of driving scenes,”RAL, 2025

    2025

  5. [13]

    Learning modality- agnostic representation for semantic segmentation from any modalities,

    X. Zheng, Y . Lyu, and L. Wang, “Learning modality- agnostic representation for semantic segmentation from any modalities,” inECCV, 2024

    2024

  6. [14]

    Cmx: Cross-modal fusion for rgb-x semantic segmenta- tion with transformers,

    H. Liu, J. Zhang, K. Yang, X. Hu, and R. Stiefelhagen, “Cmx: Cross-modal fusion for rgb-x semantic segmenta- tion with transformers,”arXiv e-prints, pp. arXiv–2203, 2022

    2022

  7. [15]

    Unveiling the potential of segment anything model 2 for rgb-thermal semantic segmentation with language guidance,

    J. Zhao, F. Teng, K. Luo, G. Zhao, Z. Li, X. Zheng, and K. Yang, “Unveiling the potential of segment anything model 2 for rgb-thermal semantic segmentation with language guidance,”arXiv preprint arXiv:2503.02581, 2025

    2025 arXiv

  8. [16]

    Benchmarking multi-modal semantic segmentation under sensor failures: Missing and noisy modality robustness,

    C. Liao, K. Lei, X. Zheng, J. Moon, Z. Wang, Y . Wang, D. P. Paudel, L. Van Gool, and X. Hu, “Benchmarking multi-modal semantic segmentation under sensor failures: Missing and noisy modality robustness,”arXiv preprint arXiv:2503.18445, 2025

    2025 arXiv

  9. [17]

    X-prompt: Multi-modal visual prompt for video object segmentation,

    P. Guo, W. Li, H. Huang, L. Hong, X. Zhou, Z. Chen, J. Li, K. Jiang, W. Zhang, and W. Zhang, “X-prompt: Multi-modal visual prompt for video object segmentation,” inMM, 2024

    2024

  10. [18]

    Ro- bust multimodal learning with missing modalities via parameter-efficient adaptation,

    M. K. Reza, A. Prater-Bennette, and M. S. Asif, “Ro- bust multimodal learning with missing modalities via parameter-efficient adaptation,”TPAMI, 2024

    2024

  11. [19]

    Dformer: Rethinking rgbd representation learning for semantic segmentation,

    B. Yin, X. Zhang, Z. Li, L. Liu, M.-M. Cheng, and Q. Hou, “Dformer: Rethinking rgbd representation learning for semantic segmentation,”arXiv preprint arXiv:2309.09668, 2023

    2023 arXiv

  12. [20]

    Cpal: Cross- prompting adapter with loras for rgb+ x semantic seg- mentation,

    Y . Liu, P. Wu, M. Wang, and J. Liu, “Cpal: Cross- prompting adapter with loras for rgb+ x semantic seg- mentation,”TCSVT, 2025

    2025

  13. [21]

    Context-aware interaction network for rgb-t semantic segmentation,

    Y . Lv, Z. Liu, and G. Li, “Context-aware interaction network for rgb-t semantic segmentation,”TMM, 2024

    2024

  14. [22]

    Embracing events and frames with hierarchical feature refinement network for object detection,

    H. Cao, Z. Zhang, Y . Xia, X. Li, J. Xia, G. Chen, and A. Knoll, “Embracing events and frames with hierarchical feature refinement network for object detection,” inECCV, 2024

    2024

  15. [23]

    Mawkdn: A multimodal fusion wavelet knowledge distillation approach based on cross-view attention for action recognition,

    Z. Quan, Q. Chen, M. Zhang, W. Hu, Q. Zhao, J. Hou, Y . Li, and Z. Liu, “Mawkdn: A multimodal fusion wavelet knowledge distillation approach based on cross-view attention for action recognition,”TCSVT, 2023

    2023

  16. [24]

    Cpal: Cross- prompting adapter with loras for rgb+ x semantic seg- mentation,

    Y . Liu, P. Wu, M. Wang, and J. Liu, “Cpal: Cross- prompting adapter with loras for rgb+ x semantic seg- mentation,”IEEE Transactions on Circuits and Systems for Video Technology, 2025

    2025

  17. [25]

    S3f2net: Spatial- spectral-structural feature fusion network for hyperspectral image and lidar data classification,

    X. Wang, L. Song, Y . Feng, and J. Zhu, “S3f2net: Spatial- spectral-structural feature fusion network for hyperspectral image and lidar data classification,”IEEE Transactions on Circuits and Systems for Video Technology, 2025

    2025

  18. [26]

    T 2 ea: Target-aware taylor expansion approx- imation network for infrared and visible image fusion,

    Z. Huang, C. Lin, B. Xu, M. Xia, Q. Li, Y . Li, and N. Sang, “T 2 ea: Target-aware taylor expansion approx- imation network for infrared and visible image fusion,” IEEE Transactions on Circuits and Systems for Video Technology, 2025

    2025

  19. [27]

    Nuc-net: Non- uniform cylindrical partition network for efficient lidar semantic segmentation,

    X. Wang, W. Feng, L. Kong, and L. Wan, “Nuc-net: Non- uniform cylindrical partition network for efficient lidar semantic segmentation,”IEEE Transactions on Circuits and Systems for Video Technology, 2025

    2025

  20. [28]

    Reducing unimodal bias in multi-modal semantic segmentation with multi-scale functional entropy regularization,

    X. Zheng, Y . Lyu, L. Jiang, D. P. Paudel, L. Van Gool, and X. Hu, “Reducing unimodal bias in multi-modal semantic segmentation with multi-scale functional entropy regularization,”arXiv preprint arXiv:2505.06635, 2025

    2025 arXiv

  21. [29]

    Primkd: Primary modality guided multimodal fusion for rgb-d semantic segmentation,

    Z. Hao, Z. Xiao, Y . Luo, J. Guo, J. Wang, L. Shen, and H. Hu, “Primkd: Primary modality guided multimodal fusion for rgb-d semantic segmentation,” inMM, 2024

    2024

  22. [30]

    Imagenet large scale visual recognition challenge,

    O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein et al., “Imagenet large scale visual recognition challenge,” IJCV, 2015

    2015

  23. [31]

    Masked-attention mask transformer for universal image segmentation,

    B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar, “Masked-attention mask transformer for universal image segmentation,” inCVPR, 2022

    2022

  24. [32]

    Per-pixel classi- fication is not all you need for semantic segmentation,

    B. Cheng, A. Schwing, and A. Kirillov, “Per-pixel classi- fication is not all you need for semantic segmentation,” inNeurIPS, 2021

    2021

  25. [33]

    Group detr: Fast detr training with group-wise one-to-many assignment,

    Q. Chen, X. Chen, J. Wang, S. Zhang, K. Yao, H. Feng, J. Han, E. Ding, G. Zeng, and J. Wang, “Group detr: Fast detr training with group-wise one-to-many assignment,” inICCV, 2023

    2023

  26. [34]

    Detrs with collaborative hybrid assignments training,

    Z. Zong, G. Song, and Y . Liu, “Detrs with collaborative hybrid assignments training,” inICCV, 2023

    2023

  27. [35]

    Detrs with hybrid matching,

    D. Jia, Y . Yuan, H. He, X. Wu, H. Yu, W. Lin, L. Sun, C. Zhang, and H. Hu, “Detrs with hybrid matching,” in CVPR, 2023

    2023

  28. [36]

    Ms-detr: Efficient detr training with mixed supervision,

    C. Zhao, Y . Sun, W. Wang, Q. Chen, E. Ding, Y . Yang, JOURNAL OF LATEX CLASS FILES, VOL. 18, NO. 9, SEPTEMBER 2020 12 and J. Wang, “Ms-detr: Efficient detr training with mixed supervision,” inCVPR, 2024

    2020

  29. [37]

    Detection transformer with stable matching,

    S. Liu, T. Ren, J. Chen, Z. Zeng, H. Zhang, F. Li, H. Li, J. Huang, H. Su, J. Zhuet al., “Detection transformer with stable matching,” inICCV, 2023

    2023

  30. [38]

    Rank-detr for high quality object detection,

    Y . Pu, W. Liang, Y . Hao, Y . Yuan, Y . Yang, C. Zhang, H. Hu, and G. Huang, “Rank-detr for high quality object detection,”NeurIPS, 2024

    2024

  31. [39]

    Dino: Detr with improved denoising anchor boxes for end-to-end object detection,

    H. Zhang, F. Li, S. Liu, L. Zhang, H. Su, J. Zhu, L. M. Ni, and H.-Y . Shum, “Dino: Detr with improved denoising anchor boxes for end-to-end object detection,”arXiv preprint arXiv:2203.03605, 2022

    2022 arXiv

  32. [40]

    Emo2-detr: Efficient-matching oriented object detection with transformers,

    Z. Hu, K. Gao, X. Zhang, J. Wang, H. Wang, Z. Yang, C. Li, and W. Li, “Emo2-detr: Efficient-matching oriented object detection with transformers,”IEEE TGRS, 2023

    2023

  33. [41]

    Hybrid proposal refiner: Revisiting detr series from the faster r-cnn perspective,

    J. Zhao, F. Wei, and C. Xu, “Hybrid proposal refiner: Revisiting detr series from the faster r-cnn perspective,” inCVPR, 2024

    2024

  34. [42]

    Salience detr: Enhancing detection transformer with hierarchical salience filtering refinement,

    X. Hou, M. Liu, S. Zhang, P. Wei, and B. Chen, “Salience detr: Enhancing detection transformer with hierarchical salience filtering refinement,” inCVPR, 2024

    2024

  35. [43]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” inCVPR, 2016

    2016

  36. [44]

    Going deeper with convolutions,

    C. Szegedy, W. Liu, Y . Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V . Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” inCVPR, 2015

    2015

  37. [45]

    A convnet for the 2020s,

    Z. Liu, H. Mao, C.-Y . Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A convnet for the 2020s,” inCVPR, 2022

    2022

  38. [46]

    Cus- tomize segment anything model for multi-modal semantic segmentation with mixture of lora experts,

    C. Zhu, B. Xiao, L. Shi, S. Xu, and X. Zheng, “Cus- tomize segment anything model for multi-modal semantic segmentation with mixture of lora experts,”arXiv preprint arXiv:2412.04220, 2024

    2024 arXiv

  39. [47]

    Adversarial co-training for semantic segmen- tation over medical images,

    H. Xie, C. Fu, X. Zheng, Y . Zheng, C.-W. Sham, and X. Wang, “Adversarial co-training for semantic segmen- tation over medical images,”Computers in biology and medicine, vol. 157, p. 106736, 2023

    2023

  40. [48]

    Look at the neighbor: Distortion-aware unsupervised domain adaptation for panoramic semantic segmentation,

    X. Zheng, T. Pan, Y . Luo, and L. Wang, “Look at the neighbor: Distortion-aware unsupervised domain adaptation for panoramic semantic segmentation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 18 687–18 698

    2023

  41. [49]

    Deformable convolutional networks,

    J. Dai, H. Qi, Y . Xiong, Y . Li, G. Zhang, H. Hu, and Y . Wei, “Deformable convolutional networks,” inICCV, 2017

    2017

  42. [50]

    Segformer: Simple and efficient design for semantic segmentation with transformers,

    E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo, “Segformer: Simple and efficient design for semantic segmentation with transformers,”NeurIPS, 2021

    2021

  43. [51]

    Segnext: Rethinking convolutional attention design for semantic segmentation,

    M.-H. Guo, C.-Z. Lu, Q. Hou, Z. Liu, M.-M. Cheng, and S.-M. Hu, “Segnext: Rethinking convolutional attention design for semantic segmentation,” inNeurIPS, 2022

    2022

  44. [52]

    Frozen is better than learning: A new design of prototype- based classifier for semantic segmentation,

    J. Chen, D. Deguchi, C. Zhang, X. Zheng, and H. Murase, “Frozen is better than learning: A new design of prototype- based classifier for semantic segmentation,”PR, 2024

    2024

  45. [53]

    Uncertainty-aware deep co-training for semi- supervised medical image segmentation,

    X. Zheng, C. Fu, H. Xie, J. Chen, X. Wang, and C.- W. Sham, “Uncertainty-aware deep co-training for semi- supervised medical image segmentation,”Computers in Biology and Medicine, vol. 149, p. 106051, 2022

    2022

  46. [54]

    Both style and distortion matter: Dual-path unsupervised domain adaptation for panoramic semantic segmentation,

    X. Zheng, J. Zhu, Y . Liu, Z. Cao, C. Fu, and L. Wang, “Both style and distortion matter: Dual-path unsupervised domain adaptation for panoramic semantic segmentation,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 1285–1295

    2023

  47. [55]

    A good student is cooperative and reliable: Cnn-transformer collaborative learning for semantic segmentation,

    J. Zhu, Y . Luo, X. Zheng, H. Wang, and L. Wang, “A good student is cooperative and reliable: Cnn-transformer collaborative learning for semantic segmentation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 11 720–11 730

    2023

  48. [56]

    Semantics distortion and style matter: Towards source- free uda for panoramic segmentation,

    X. Zheng, P. Zhou, A. V . Vasilakos, and L. Wang, “Semantics distortion and style matter: Towards source- free uda for panoramic segmentation,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 27 885–27 895

    2024

  49. [57]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,”NeurIPS, 2017

    2017

  50. [58]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gellyet al., “An image is worth 16x16 words: Transformers for image recognition at scale,”arXiv preprint arXiv:2010.11929, 2020

    2010 arXiv

  51. [59]

    E-clip: Towards label-efficient event-based open-world understanding by clip,

    J. Zhou, X. Zheng, Y . Lyu, and L. Wang, “E-clip: Towards label-efficient event-based open-world understanding by clip,”arXiv preprint arXiv:2308.03135, vol. 2, no. 5, p. 6, 2023

    2023 arXiv

  52. [60]

    Omnibind: Teach to build unequal-scale modality interaction for omni-bind of all,

    Y . Lyu, X. Zheng, D. Kim, and L. Wang, “Omnibind: Teach to build unequal-scale modality interaction for omni-bind of all,”arXiv preprint arXiv:2405.16108, 2024

    2024 arXiv

  53. [61]

    Exact: Language-guided conceptual reasoning and uncertainty estimation for event-based action recognition and more,

    J. Zhou, X. Zheng, Y . Lyu, and L. Wang, “Exact: Language-guided conceptual reasoning and uncertainty estimation for event-based action recognition and more,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 18 633–18 643

    2024

  54. [62]

    Missing modal- ity robustness in semi-supervised multi-modal semantic segmentation,

    H. Maheshwari, Y .-C. Liu, and Z. Kira, “Missing modal- ity robustness in semi-supervised multi-modal semantic segmentation,” inWACV, 2024

    2024

  55. [63]

    Towards good practices for missing modality robust action recognition,

    S. Woo, S. Lee, Y . Park, M. A. Nugroho, and C. Kim, “Towards good practices for missing modality robust action recognition,” inAAAI, 2023

    2023

  56. [64]

    Unified multi-modal image synthesis for missing modality imputation,

    Y . Zhang, C. Peng, Q. Wang, D. Song, K. Li, and S. K. Zhou, “Unified multi-modal image synthesis for missing modality imputation,”TMI, 2024

    2024

  57. [65]

    M3ae: multimodal representation learning for brain tumor segmentation with missing modalities,

    H. Liu, D. Wei, D. Lu, J. Sun, L. Wang, and Y . Zheng, “M3ae: multimodal representation learning for brain tumor segmentation with missing modalities,” inAAAI, 2023

    2023

  58. [66]

    Semi-mamba: Mamba-driven semi-supervised multimodal remote sensing feature classification,

    Y . Li, D. Li, W. Xie, J. Ma, S. He, and L. Fang, “Semi-mamba: Mamba-driven semi-supervised multimodal remote sensing feature classification,”IEEE Transactions on Circuits and Systems for Video Technology, 2025

    2025

  59. [67]

    Deep learning for event-based vision: A comprehensive survey and benchmarks,

    X. Zheng, Y . Liu, Y . Lu, T. Hua, T. Pan, W. Zhang, D. Tao, and L. Wang, “Deep learning for event-based vision: A comprehensive survey and benchmarks,”arXiv preprint arXiv:2302.08890, 2023

    2023 arXiv

  60. [68]

    Chasing day and night: Towards robust and efficient all-day object detection guided by an event camera,

    J. Cao, X. Zheng, Y . Lyu, J. Wang, R. Xu, and L. Wang, “Chasing day and night: Towards robust and efficient all-day object detection guided by an event camera,” in 2024 IEEE International Conference on Robotics and JOURNAL OF LATEX CLASS FILES, VOL. 18, NO. 9, SEPTEMBER 2020 ...

    2024

  61. [69]

    Unibind: Llm- augmented unified and balanced representation space to bind them all,

    Y . Lyu, X. Zheng, J. Zhou, and L. Wang, “Unibind: Llm- augmented unified and balanced representation space to bind them all,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 26 752–26 762

    2024

  62. [70]

    Eventdance: Unsupervised source-free cross-modal adaptation for event-based object recognition,

    X. Zheng and L. Wang, “Eventdance: Unsupervised source-free cross-modal adaptation for event-based object recognition,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 17 448–17 458

    2024

  63. [71]

    Eventbind: Learning a unified representation to bind them all for event-based open-world understanding,

    J. Zhou, X. Zheng, Y . Lyu, and L. Wang, “Eventbind: Learning a unified representation to bind them all for event-based open-world understanding,” inEuropean Conference on Computer Vision. Springer, 2024, pp. 477–494

    2024

  64. [72]

    Memorysam: Memorize modal- ities and semantics with segment anything model 2 for multi-modal semantic segmentation,

    C. Liao, X. Zheng, Y . Lyu, H. Xue, Y . Cao, J. Wang, K. Yang, and X. Hu, “Memorysam: Memorize modal- ities and semantics with segment anything model 2 for multi-modal semantic segmentation,”arXiv preprint arXiv:2503.06700, 2025

    2025 arXiv

  65. [73]

    Magic++: Efficient and resilient modality-agnostic se- mantic segmentation via hierarchical modality selection,

    X. Zheng, Y . Lyu, L. Jiang, J. Zhou, L. Wang, and X. Hu, “Magic++: Efficient and resilient modality-agnostic se- mantic segmentation via hierarchical modality selection,” arXiv preprint arXiv:2412.16876, 2024

    2024 arXiv

  66. [74]

    Fcos: Fully convolutional one-stage object detection,

    Z. Tian, C. Shen, H. Chen, and T. He, “Fcos: Fully convolutional one-stage object detection,” inICCV, 2019

    2019

  67. [75]

    Focal loss for dense object detection,

    T.-Y . Lin, P. Goyal, R. Girshick, K. He, and P. Doll ´ar, “Focal loss for dense object detection,” inICCV, 2017

    2017

  68. [76]

    End-to-end object detection with transformers,

    N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” inECCV, 2020

    2020

  69. [77]

    Mask dino: Towards a unified transformer- based framework for object detection and segmentation,

    F. Li, H. Zhang, H. Xu, S. Liu, L. Zhang, L. M. Ni, and H.-Y . Shum, “Mask dino: Towards a unified transformer- based framework for object detection and segmentation,” inCVPR, 2023

    2023

  70. [78]

    Primitive generation and semantic-related alignment for universal zero-shot segmentation,

    S. He, H. Ding, and W. Jiang, “Primitive generation and semantic-related alignment for universal zero-shot segmentation,” inCVPR, 2023

    2023

  71. [79]

    Gen- eralizable semantic vision query generation for zero- shot panoptic and semantic segmentation,

    J. Chen, D. Deguchi, C. Zhang, and H. Murase, “Gen- eralizable semantic vision query generation for zero- shot panoptic and semantic segmentation,”arXiv preprint arXiv:2402.13697, 2024

    2024 arXiv

  72. [80]

    Acnet: Attention based network to exploit complementary features for rgbd semantic segmentation,

    X. Hu, K. Yang, L. Fei, and K. Wang, “Acnet: Attention based network to exploit complementary features for rgbd semantic segmentation,” in2019 IEEE international conference on image processing (ICIP). IEEE, 2019, pp. 1440–1444

    2019

  73. [81]

    Shapeconv: Shape-aware convolutional layer for indoor rgb-d semantic segmentation,

    J. Cao, H. Leng, D. Lischinski, D. Cohen-Or, C. Tu, and Y . Li, “Shapeconv: Shape-aware convolutional layer for indoor rgb-d semantic segmentation,” inProceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 7088–7097

    2021

  74. [82]

    Aggregated residual transformations for deep neural networks,

    S. Xie, R. Girshick, P. Doll ´ar, Z. Tu, and K. He, “Aggregated residual transformations for deep neural networks,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 1492– 1500

    2017

  75. [83]

    Efficient rgb-d semantic segmentation for indoor scene analysis,

    D. Seichter, M. K ¨ohler, B. Lewandowski, T. Wengefeld, and H.-M. Gross, “Efficient rgb-d semantic segmentation for indoor scene analysis,” in2021 IEEE international conference on robotics and automation (ICRA). IEEE, 2021, pp. 13 525–13 531

    2021

  76. [84]

    Multimodal token fusion for vision transform- ers,

    Y . Wang, X. Chen, L. Cao, W. Huang, F. Sun, and Y . Wang, “Multimodal token fusion for vision transform- ers,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 12 186– 12 195

    2022

  77. [85]

    Omnivore: A single model for many visual modalities,

    R. Girdhar, M. Singh, N. Ravi, L. Van Der Maaten, A. Joulin, and I. Misra, “Omnivore: A single model for many visual modalities,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 16 102–16 112

    2022

  78. [86]

    Swin transformer: Hierarchical vision transformer using shifted windows,

    Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” inICCV, 2021

    2021

  79. [87]

    Prompting multi-modal image segmentation with semantic grouping,

    Q. He, “Prompting multi-modal image segmentation with semantic grouping,” inProceedings of the AAAI conference on artificial intelligence, vol. 38, no. 3, 2024, pp. 2094–2102

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.