Pith. sign in

REVIEW 4 major objections 4 minor 51 references

MM SAM-adapter tops DeLiVER, FMB, MUSES segmentation by injecting fused RGB-plus-sensor features into a SAM encoder through a side adapter.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 17:51 UTC pith:FL3OKW3P

load-bearing objection Solid SAM-adapter paper with a genuine architectural delta and careful ablations, but the SOTA claim rests on capacity-uncontrolled comparisons; worth refereeing with revisions. the 4 major comments →

arxiv 2509.10408 v1 pith:FL3OKW3P submitted 2025-09-12 cs.CV cs.AI

Multimodal SAM-adapter for Semantic Segmentation

Iacopo Curti , Pierluigi Zama Ramirez , Alioscia Petrelli , Luigi Di Stefano This is my paper
classification cs.CV cs.AI
keywords semantic segmentationmultimodal fusionSAM adapterfoundation modelsLiDARthermal imagingevent camerasRGB-hard benchmark
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that the pretrained SAM image encoder, designed for promptable instance segmentation on RGB, can be repurposed for multimodal semantic segmentation without losing its strong RGB priors. Its proposal, MM SAM-adapter, runs a lightweight adapter alongside SAM: a multimodal fusion encoder combines RGB and one auxiliary modality (LiDAR, depth, thermal, or event data), and injector–extractor blocks with cross-attention weave those fused features into SAM's layers. On the DeLiVER, FMB, and MUSES benchmarks the authors report state-of-the-art mIoU in every considered two-modality setup, and they introduce RGB-easy/RGB-hard splits to show the largest gains appear exactly when the RGB content is poor. If the results hold, the practical consequence is that one foundational RGB model can serve as the backbone for many sensor configurations.

Core claim

At its core, the paper argues that the optimal role of auxiliary sensors is to adjust a strong RGB foundation model rather than to share the burden equally. Traditional multimodal segmentation builds symmetric networks; MM SAM-adapter instead keeps SAM's ViT-Large encoder as the main branch and uses a second, smaller branch—two ConvNeXt-Small encoders plus a fusion module—to produce multi-scale features that are injected into SAM's layers via deformable cross-attention (injectors) and extracted back out (extractors). This lets the adapter learn when auxiliary information helps without overwriting SAM's general knowledge. The authors claim this design outperforms prior methods on all three be

What carries the argument

MM SAM-adapter's load-bearing piece is the side adapter built from repeated Injector–Extractor pairs. Each Injector takes SAM's current tokens as queries and the stacked multi-scale fused multimodal features as keys/values in a multi-scale deformable cross-attention, adding the attended result with a zero-initialized learned scale; each Extractor reverses the roles and refines multimodal features with an FFN. The adapter input comes from a Multimodal Fusion Encoder with two modality-specific ConvNeXt-Small encoders and a Road-Fusion module (self-attention, spatial attention, coordinate attention, convolutions) that preserves each modality's information so the model can discard noisy auxiliar

Load-bearing premise

The load-bearing premise is that the head-to-head benchmark comparisons are fair: most competitors were run with different backbones and several were retrained by the authors under configurations borrowed from other datasets, with no parameter or FLOP counts reported; if any baseline was undertrained or capacity-mismatched, the reported mIoU margins would not establish the adapter design as the cause.

What would settle it

Run a controlled experiment on DeLiVER RGB-LiDAR with exactly the same SAM ViT-L backbone, training budget, and segmentation head, comparing three variants: RGB-only adapter, LiDAR-only adapter, and MM SAM-adapter. The central claim requires the multimodal variant to beat the RGB-only variant on the RGB-hard split by a nontrivial margin (roughly the 7 mIoU reported in Table 9) while staying at parity on RGB-easy. If the margin collapses or the RGB-only adapter matches it, the paper's explanation for the gains is falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • On DeLiVER, FMB, and MUSES, MM SAM-adapter reports the best mean IoU in every RGB-plus-auxiliary setup tested, including night, fog, rain, and snow splits.
  • Because accuracy on RGB-easy samples stays almost constant across auxiliary modalities (e.g., roughly 57.62–57.75 mIoU on DeLiVER for RGB-Depth versus RGB-LiDAR), the model appears able to ignore auxiliary input when RGB is sufficient.
  • In RGB-hard samples, fused multimodal features produce documented gains, e.g., 45.46 versus 40.93 mIoU on DeLiVER RGB-LiDAR and 62.59 versus 61.79 on FMB against the best competing method.
  • Using only two modalities at a time, the method is reported to outperform four-modality competitors on both DeLiVER and MUSES test sets, suggesting that feature quality and fusion mechanism matter more than raw sensor count.
  • If these results hold, the recipe is portable: a single pretrained RGB segmentation foundation plus a lightweight fused-sensor adapter can serve many sensor configurations without retraining the foundation from scratch.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • An untested but plausible extension is to treat the adapter as a generic sensor-to-foundation bridge: replace the fusion encoder with a learned embedding for each new modality (radar, depth, thermal) and attach it to the same SAM encoder.
  • The reported margins over competitors are set against different backbone capacities (e.g., MiT-B2, Swin-Large, ConvNeXt-Large) without parameter or FLOP counts, so the specific contribution of the adapter's cross-attention design would be better isolated by a same-backbone, same-budget comparison; the paper's own LoRA comparison is one such control but uses only a single LoRA layer.
  • The RGB-easy/RGB-hard split methodology could become a reusable diagnostic for multimodal segmentation: future papers should report per-condition gains, not only aggregate mIoU, since aggregate numbers on these datasets hide that roughly 97% of DeLiVER is easy for RGB alone.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes MM SAM-adapter, a multimodal semantic segmentation method that adapts the SAM image encoder (ViT-L) to dense prediction by injecting fused multimodal features through an external side-tuning adapter. The Multimodal Fusion Encoder uses two ImageNet-pretrained ConvNeXt-S encoders plus a Road-Fusion module; the adapter exchanges information with SAM blocks via multi-scale deformable cross-attention injectors/extractors, and a SegFormer head produces predictions. The method is evaluated on DeLiVER (RGB-Depth, RGB-LiDAR, RGB-Event), FMB (RGB-Thermal), and MUSES (RGB-LiDAR, RGB-Event), reporting state-of-the-art mIoU on all three benchmarks and also on manually created RGB-easy/RGB-hard splits. Ablations examine asymmetric vs symmetric design, fusion-module choice, modality-specific vs modality-agnostic encoders, frozen vs fine-tuned SAM, and the value of SAM pretraining.

Significance. Should the reported numbers be accepted, the paper would show that a large pretrained RGB foundation model can be adapted to multimodal semantic segmentation with a comparatively light adapter, and that using only two modalities can beat methods that combine all available modalities (Tables 7 and 8). The paper's strengths include a careful ablation suite (Tables 9-14), release of code, and public online-benchmark submissions for MUSES. The RGB-easy/RGB-hard split is a useful diagnostic for modality complementarity. However, the central SOTA claim is not yet established: the headline comparisons in Tables 4-6 are not capacity-controlled, and several design decisions, including the fusion module, are selected using the DeLiVER test set. The limitation acknowledged in the Conclusion (only two input modalities are supported) is appropriate and does not by itself undermine the method's value, but it should be kept in the final version.

major comments (4)
  1. [Section 3.3.3; Tables 4-6] The headline state-of-the-art claim is based on capacity-uncontrolled comparisons. The proposed model uses a 24-layer SAM ViT-L encoder plus two ConvNeXt-S encoders and a Road-Fusion module, whereas the external baselines use MiT-B2 (CMNeXt, GeminiFusion on DeLiVER/MUSES), Swin-Large (GeminiFusion on FMB), and ConvNeXt-Large (RoadFormer+). No parameter counts or FLOPs are reported for any model. The reported margins (e.g., +2.58 mIoU over RoadFormer+ on DeLiVER RGB-LiDAR, +1.35 on FMB, +0.69 on MUSES) could therefore be due to the larger backbone and heavier fusion machinery rather than to the design principle of injecting fused features into a SAM adapter. The paper needs a matched-capacity comparison or, at minimum, a parameter/FLOP table and same-backbone external baselines.
  2. [Section 3.3.3; Tables 5-6] For FMB and MUSES, all baselines were retrained by the authors because official weights are unavailable, and for MUSES the configuration was "adopted" from DeLiVER. No training curves, validation performance, or per-dataset hyperparameter details are reported for these baselines. The margins on FMB (+1.35 over GeminiFusion) and MUSES (+0.69 over RoadFormer+) are small enough that undertrained or misconfigured baselines could flip the ranking. The authors should document the baseline training protocol in detail (learning rate, epochs, augmentations, convergence criteria) or compare with published numbers where they exist.
  3. [Section 4.3.3; Table 11] The fusion module (Road-Fusion vs addition vs concatenation) is chosen by comparing variants on the DeLiVER test set, and the other ablations in Section 4.3 (Tables 10, 12-14) also report test-set mIoU. This means the final DeLiVER result is not an unbiased evaluation on a held-out test set; it partially reflects model selection on the test set. The design choices should be made on the validation set, with the test set reserved for final evaluation. This is a load-bearing methodological concern for the reported DeLiVER SOTA.
  4. [Section 4.3.1; Table 9 and Table 13] The comparison between the proposed side-tuning adapter and LoRA is confounded with backbone fine-tuning. In Table 9, "SAM* + SH + LoRA" freezes SAM, while "SAM + SH + adapter" fine-tunes SAM. Table 13 shows that fine-tuning SAM improves the full model by 1.79 mIoU (55.35 vs 57.14). Therefore the conclusion that the ViT-adapter strategy is superior to LoRA is not supported; a LoRA variant with fine-tuned SAM is needed. This is a load-bearing point for the contribution claim in the introduction about outperforming LoRA.
minor comments (4)
  1. [Section 3.2.1] The RGB-easy/RGB-hard split is created by visual comparison, without a quantitative threshold or a public release of the split files. This makes the split hard to reproduce. Also, the text says "approximately 97%" of DeLiVER is RGB-easy, but the reported counts give 1797/1897 = 94.7%; please reconcile.
  2. [Table 6] There are formatting errors in several entries, e.g., "69.6469.29" and "80.6976.40"; spaces/separators are missing.
  3. [Figures 4 and 5] The qualitative figures are very small and hard to read, especially the overlay and input rows. Higher resolution or zoomed crops would help.
  4. [Conclusion] The limitation statement in the Conclusion (only two input modalities are supported) is honest and useful; please ensure it remains in the final version, and consider discussing its implications for the all-modality comparisons in Tables 7-8.

Circularity Check

0 steps flagged

No circularity found: the paper is an empirical composition of external components evaluated on external benchmarks.

full rationale

The paper's derivation chain is architectural, not formal: it assembles external, independently published components (SAM ViT-L encoder, ConvNeXt-S encoders, RoadFormer+'s Road-Fusion module, ViT-adapter-style cross-attention injectors/extractors, and a SegFormer head) and evaluates them on external benchmarks (DeLiVER, FMB, MUSES). The central claim—state-of-the-art performance—is an empirical result from Tables 4-6, not a quantity derived from the method's equations. The injector/extractor formulas (Eqs. 1-3) describe the forward pass and do not encode the performance outcome. There are no load-bearing self-citations: the reference list contains no papers by the present authors, and no uniqueness theorem or first-principles result is imported from prior work by the same group. The RGB-easy/RGB-hard splits are defined by where an RGB-only network fails; evaluating the proposed method on those splits is a comparative empirical analysis, not a definitional reduction, since the split is independent of the proposed architecture and the claim is about outperforming other multimodal methods on those subsets. The ablations in Section 4.3, including the choice of Road-Fusion in Table 11, involve test-set selection and capacity-uncontrolled comparisons, which are experimental-validity and fairness concerns rather than circularity: no fitted parameter is renamed as a prediction, and no equation is equivalent to its input by construction. Therefore the paper is not circular; any weaknesses belong to experimental protocol, not to a self-referential derivation.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The paper is empirical; its ledger is the set of design choices and background assumptions the results rest on, not fitted constants in a derivation. The main entries: pre-trained weight sources (SAM ViT-L, ImageNet-22k ConvNeXt-S), a hand-chosen training schedule, the test-set-selected fusion module, and the domain assumption that pixel-aligned auxiliary inputs are available. No new physical or conceptual entities are introduced; the gamma_i gate vector in Eq. (1) is a learnable parameter, not an entity.

free parameters (5)
  • Fusion module choice = Road-Fusion (RoadFormer+), 57.14 test mIoU
    Selected because it yields the best DeLiVER test mIoU in Table 11 (57.14 vs 56.87 concatenation, 56.09 addition); effectively a test-set model selection (Section 4.3.3).
  • Layer-wise learning-rate decay gamma = 0.9
    Handpicked in Section 3.3.2, inherited from ViT-adapter; no sensitivity analysis reported.
  • Base LR, warmup, epochs = 2e-4, warmup 10, 100 epochs (200 for FMB)
    Handpicked training schedule (Section 3.3.2); no sensitivity analysis reported.
  • Adapter block split N = N = 4 blocks (6 SAM layers each)
    24-layer SAM ViT-L divided into 4 blocks, each with one injector/extractor pair (Section 3.1.1); a design constant borrowed from ViT-adapter.
  • RGB-easy / RGB-hard threshold = manual visual verdict, 1797/100 and 183/97 splits
    The splits rest on the authors' visual comparison of an unnamed RGB-only network's predictions against ground truth (Section 3.2.1); the threshold is informal and network-specific.
axioms (4)
  • domain assumption Auxiliary signals are pixel-aligned with RGB at identical resolution
    Required by the ConvNeXt encoders and cross-attention in Section 3.1.2; holds for DeLiVER, FMB, and MUSES by dataset construction.
  • domain assumption SAM ViT-L features, once fine-tuned, transfer to semantic segmentation
    Motivates the whole design; validated by ablation Table 14 (fine-tuned 57.14 vs scratch 53.73) and the SAM+SH baseline (53.32), so it is an assumption with supporting evidence.
  • domain assumption The adapter learns to ignore the auxiliary modality when RGB is clean
    Central mechanism claim (Section 1 and Section 4.3.1); supported only indirectly by stable RGB-easy mIoU, never probed directly inside the network.
  • standard math Standard supervised training with OHEM cross-entropy and a SegFormer head suffices
    Task head and loss (Section 3.3.2) are standard practice, adopted without analysis.

pith-pipeline@v1.3.0-alltime-deepseek · 21929 in / 19378 out tokens · 186265 ms · 2026-08-04T17:51:57.184796+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Multimodal SAM-adapter for Semantic Segmentation." pith.science (2026). https://pith.science/paper/FL3OKW3P

@misc{pith2026250910408,
  author       = {Pith},
  title        = {Pith review of: Multimodal SAM-adapter for Semantic Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FL3OKW3P}},
  note         = {Machine review of arXiv:2509.10408}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Semantic segmentation, a key task in computer vision with broad applications in autonomous driving, medical imaging, and robotics, has advanced substantially with deep learning. Nevertheless, current approaches remain vulnerable to challenging conditions such as poor lighting, occlusions, and adverse weather. To address these limitations, multimodal methods that integrate auxiliary sensor data (e.g., LiDAR, infrared) have recently emerged, providing complementary information that enhances robustness. In this work, we present MM SAM-adapter, a novel framework that extends the capabilities of the Segment Anything Model (SAM) for multimodal semantic segmentation. The proposed method employs an adapter network that injects fused multimodal features into SAM's rich RGB features. This design enables the model to retain the strong generalization ability of RGB features while selectively incorporating auxiliary modalities only when they contribute additional cues. As a result, MM SAM-adapter achieves a balanced and efficient use of multimodal information. We evaluate our approach on three challenging benchmarks, DeLiVER, FMB, and MUSES, where MM SAM-adapter delivers state-of-the-art performance. To further analyze modality contributions, we partition DeLiVER and FMB into RGB-easy and RGB-hard subsets. Results consistently demonstrate that our framework outperforms competing methods in both favorable and adverse conditions, highlighting the effectiveness of multimodal adaptation for robust scene understanding. The code is available at the following link: https://github.com/iacopo97/Multimodal-SAM-Adapter.

Figures

Figures reproduced from arXiv: 2509.10408 by Alioscia Petrelli, Iacopo Curti, Luigi Di Stefano, Pierluigi Zama Ramirez.

Figure 1
Figure 1. Figure 1: We propose adapting the Segment Anything [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the MM SAM-adapter architecture. The top row shows the four main modules: SAM Encoder, MM Fusion Encoder, adapter and Segmentation Head (SH). The bottom row details the Modality-specific Encoders and the Fusion Module utilized by the MM Fusion Encoder as well as the Injector and Extractors modules utilized by the adapter. ViT features through a series of injectors and extractors that maintain c… view at source ↗
Figure 3
Figure 3. Figure 3: RGB-easy and RGB-hard examples for DeLiVER and FMB. The DeLiVER RGB-hard image has been stretched to enhance visibility (zoom in to better notice it). 3.2 Datasets We employ the following multimodal semantic segmen￾tation datasets in our experiments: DeLiVER [48] is a synthetic dataset comprising 3,983 training, 2,005 validation, and 1,897 test samples. It in￾cludes LiDAR, Depth, Event, and RGB data from u… view at source ↗
Figure 4
Figure 4. Figure 4: DeLiVER [48] test set predictions in RGB-Depth, RGB-LiDAR, RGB-Event framework. RGB-easy (top) and RGB-hard (bottom) samples. Notably, the input RGB image in RGB-hard case has been stretched with an exponential operator. contexts. These results are publicly available in the MUSES online benchmark 1 . Therefore, the experi￾ments in this section confirm that our approach—based 1 “MUSES online benchmark,” May… view at source ↗
Figure 5
Figure 5. Figure 5: FMB test set and MUSES validation set predictions of the methods in [PITH_FULL_IMAGE:figures/full_fig_p013_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: DeLiVER test set predictions of some of the methods in [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

51 extracted references · 9 linked inside Pith

  1. [1]

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization.arXiv preprint arXiv:1607.06450, 2016

    Pith/arXiv arXiv 2016

  2. [2]

    Muses: The multi- sensor semantic perception dataset for driving un- der uncertainty

    Tim Br ¨odermann, David Bruggemann, Christos Sakaridis, Kevin Ta, Odysseas Liagouris, Jason Corkill, and Luc Van Gool. Muses: The multi- sensor semantic perception dataset for driving un- der uncertainty. InEuropean Conference on Com- puter Vision (ECCV), 2024

    2024

  3. [3]

    Cafuser: Condition-aware mul- timodal fusion for robust semantic perception of driving scenes.IEEE Robotics and Automation Letters, 10(4):3134–3141, 2025

    Tim Br ¨odermann, Christos Sakaridis, Yuqian Fu, and Luc Van Gool. Cafuser: Condition-aware mul- timodal fusion for robust semantic perception of driving scenes.IEEE Robotics and Automation Letters, 10(4):3134–3141, 2025

    2025

  4. [4]

    Collaborative compensative trans- former network for salient object detection.Pattern Recognition, 154:110600, 2024

    Jun Chen, Heye Zhang, Mingming Gong, and Zhifan Gao. Collaborative compensative trans- former network for salient object detection.Pattern Recognition, 154:110600, 2024

    2024

  5. [5]

    Sam2-adapter: Evaluating & adapting segment anything 2 in downstream tasks: Camouflage, shadow, medi- cal image segmentation, and more.arXiv preprint arXiv:2408.04579, 2024

    Tianrun Chen, Ankang Lu, Lanyun Zhu, Chao- tao Ding, Chunan Yu, Deyi Ji, Zejian Li, Lingyun Sun, Papa Mao, and Ying Zang. Sam2-adapter: Evaluating & adapting segment anything 2 in downstream tasks: Camouflage, shadow, medi- cal image segmentation, and more.arXiv preprint arXiv:2408.04579, 2024

    Pith/arXiv arXiv 2024

  6. [6]

    Sam- adapter: Adapting segment anything in underper- formed scenes

    Tianrun Chen, Lanyun Zhu, Chaotao Deng, Run- long Cao, Yan Wang, Shangzhan Zhang, Zejian Li, Lingyun Sun, Ying Zang, and Papa Mao. Sam- adapter: Adapting segment anything in underper- formed scenes. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 3367–3375, 2023

    2023

  7. [7]

    Vision transformer adapter for dense predictions

    Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. InThe Eleventh International Conference on Learning Representations, 2023

    2023

  8. [8]

    Segment any event streams via weighted adaptation of piv- otal tokens

    Zhiwen Chen, Zhiyu Zhu, Yifan Zhang, Junhui Hou, Guangming Shi, and Jinjian Wu. Segment any event streams via weighted adaptation of piv- otal tokens. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recogni- tion (CVPR), pages 3890–3900, June 2024

    2024

  9. [9]

    Schwing, Alexander Kirillov, and Rohit Girdhar

    Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. InCVPR, 2022

    2022

  10. [10]

    Schwing, and Alexander Kirillov

    Bowen Cheng, Alexander G. Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. In NeurIPS, 2021

    2021

  11. [11]

    The cityscapes dataset for semantic urban scene understanding.2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3213–3223, 2016

    Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding.2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3213–3223, 2016

    2016

  12. [12]

    Indoor semantic seg- mentation using depth information.arXiv preprint arXiv:1301.3572, 2013

    Camille Couprie, Cl ´ement Farabet, Laurent Na- jman, and Yann LeCun. Indoor semantic seg- mentation using depth information.arXiv preprint arXiv:1301.3572, 2013

    Pith/arXiv arXiv 2013

  13. [13]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In2009 IEEE Confer- ence on Computer Vision and Pattern Recognition, pages 248–255, 2009

    2009

  14. [14]

    Efficient multimodal semantic segmentation via dual-prompt learning.arXiv preprint arXiv:2312.00360, 2023

    Shaohua Dong, Yunhe Feng, Qing Yang, Yan Huang, Dongfang Liu, and Heng Fan. Efficient multimodal semantic segmentation via dual-prompt learning.arXiv preprint arXiv:2312.00360, 2023

    Pith/arXiv arXiv 2023

  15. [15]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. InInternational Conference on Learning Representations, 2021

    2021

  16. [16]

    Mfnet: Towards real-time semantic segmentation for au- tonomous vehicles with multi-spectral scenes

    Qishen Ha, Kohei Watanabe, Takumi Karasawa, Yoshitaka Ushiku, and Tatsuya Harada. Mfnet: Towards real-time semantic segmentation for au- tonomous vehicles with multi-spectral scenes. In 2017 IEEE/RSJ International Conference on In- telligent Robots and Systems (IROS), pages 5108– 5115, 2017

    2017

  17. [17]

    A survey on instance segmentation: state of the art

    Abdul Mueed Hafiz and Ghulam Mohiuddin Bhat. A survey on instance segmentation: state of the art. International journal of multimedia information retrieval, 9(3):171–189, 2020

    2020

  18. [18]

    Fusenet: Incorporating depth into semantic segmentation via fusion-based cnn architecture

    Caner Hazirbas, Lingni Ma, Csaba Domokos, and Daniel Cremers. Fusenet: Incorporating depth into semantic segmentation via fusion-based cnn architecture. In Shang-Hong Lai, Vincent Lepetit, 16 Ko Nishino, and Yoichi Sato, editors,Computer Vi- sion – ACCV 2016, pages 213–228, Cham, 2017. Springer International Publishing

    2016

  19. [19]

    Masked au- toencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll ´ar, and Ross Girshick. Masked au- toencoders are scalable vision learners. InPro- ceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 16000–16009, June 2022

    2022

  20. [20]

    Parameter-efficient transfer learning for NLP

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzeb- ski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Syl- vain Gelly. Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors,Proceedings of the 36th International Conference on Machine Learning, volume 97 ofProceedings of Machine L...

    2019

  21. [21]

    Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022

    2022

  22. [22]

    Roadformer+: Delivering rgb-x scene parsing through scale-aware information decou- pling and advanced heterogeneous feature fusion

    Jianxin Huang, Jiahang Li, Ning Jia, Yuxi- ang Sun, Chengju Liu, Qijun Chen, and Rui Fan. Roadformer+: Delivering rgb-x scene parsing through scale-aware information decou- pling and advanced heterogeneous feature fusion. IEEE Transactions on Intelligent Vehicles, 2024. DOI:10.1109/TIV.2024.3448251

    arXiv 2024

  23. [23]

    OneFormer: One Transformer to Rule Universal Image Segmenta- tion

    Jitesh Jain, Jiachen Li, MangTik Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi. OneFormer: One Transformer to Rule Universal Image Segmenta- tion. InCVPR, 2023

    2023

  24. [24]

    Gemini- Fusion: Efficient pixel-wise multimodal fusion for vision transformer

    Ding Jia, Jianyuan Guo, Kai Han, Han Wu, Chao Zhang, Chang Xu, and Xinghao Chen. Gemini- Fusion: Efficient pixel-wise multimodal fusion for vision transformer. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors,Proceedings of the 41st International Con- ference on Machine L...

    2024

  25. [25]

    Attention enhanced machine instinctive vision with human-inspired saliency detection.Image and Vision Computing, 152:105308, 2024

    Habib Khan, Muhammad Talha Usman, Imad Rida, and JaKeoung Koo. Attention enhanced machine instinctive vision with human-inspired saliency detection.Image and Vision Computing, 152:105308, 2024

    2024

  26. [26]

    Berg, Wan-Yen Lo, Piotr Dollar, and Ross Gir- shick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and Ross Gir- shick. Segment anything. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023

    2023

  27. [27]

    Stitchfusion: Weaving any vi- sual modalities to enhance multimodal semantic segmentation.arXiv preprint arXiv:2408.01343, 2024

    Bingyu Li, Da Zhang, Zhiyuan Zhao, Junyu Gao, and Xuelong Li. Stitchfusion: Weaving any vi- sual modalities to enhance multimodal semantic segmentation.arXiv preprint arXiv:2408.01343, 2024

    Pith/arXiv arXiv 2024

  28. [28]

    Roadformer: Duplex transformer for rgb-normal semantic road scene parsing.IEEE Transactions on Intelligent Vehicles, 2024

    Jiahang Li, Yikang Zhang, Peng Yun, Guangliang Zhou, Qijun Chen, and Rui Fan. Roadformer: Duplex transformer for rgb-normal semantic road scene parsing.IEEE Transactions on Intelligent Vehicles, 2024. DOI:10.1109/TIV.2024.3388726

    arXiv 2024

  29. [29]

    Multi-interactive feature learning and a full- time multi-modality benchmark for image fusion and segmentation

    Jinyuan Liu, Zhu Liu, Guanyao Wu, Long Ma, Risheng Liu, Wei Zhong, Zhongxuan Luo, and Xin Fan. Multi-interactive feature learning and a full- time multi-modality benchmark for image fusion and segmentation. InInternational Conference on Computer Vision, 2023

    2023

  30. [30]

    Segmenting anything in the dark via depth perception.IEEE Transactions on Multime- dia, pages 1–12, 2025

    Peng Liu, Jinhong Deng, Lixin Duan, Wen Li, and Fengmao Lv. Segmenting anything in the dark via depth perception.IEEE Transactions on Multime- dia, pages 1–12, 2025

    2025

  31. [31]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021

    2021

  32. [32]

    A convnet for the 2020s.Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Sain- ing Xie. A convnet for the 2020s.Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022

    2022

  33. [33]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. InInternational Con- ference on Learning Representations, 2019

    2019

  34. [34]

    Image segmentation using deep learn- ing: A survey.IEEE transactions on pattern anal- ysis and machine intelligence, 44(7):3523–3542, 2021

    Shervin Minaee, Yuri Boykov, Fatih Porikli, Anto- nio Plaza, Nasser Kehtarnavaz, and Demetri Ter- zopoulos. Image segmentation using deep learn- ing: A survey.IEEE transactions on pattern anal- ysis and machine intelligence, 44(7):3523–3542, 2021

    2021

  35. [35]

    SAM 2: Segment any- thing in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R ¨adle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Dollar, and Christoph Feichtenhofer. SAM 2: Segment any- thing in images and videos. InThe Thirteenth 17 Int...

    2025

  36. [36]

    Su-sam: A simple unified framework for adapting segment anything model in underperformed scenes.arXiv preprint arXiv:2401.17803, 2024

    Yiran Song, Qianyu Zhou, Xuequan Lu, Zhiwen Shao, and Lizhuang Ma. Su-sam: A simple unified framework for adapting segment anything model in underperformed scenes.arXiv preprint arXiv:2401.17803, 2024

    Pith/arXiv arXiv 2024

  37. [37]

    Rtfnet: Rgb-thermal fusion network for semantic segmen- tation of urban scenes.IEEE Robotics and Au- tomation Letters, 4(3):2576–2583, 2019

    Yuxiang Sun, Weixun Zuo, and Ming Liu. Rtfnet: Rgb-thermal fusion network for semantic segmen- tation of urban scenes.IEEE Robotics and Au- tomation Letters, 4(3):2576–2583, 2019

    2019

  38. [38]

    Semantic segmentation using vision transformers: A survey

    Hans Thisanke, Chamli Deshan, Kavindu Chamith, Sachith Seneviratne, Rajith Vi- danaarachchi, and Damayanthi Herath. Semantic segmentation using vision transformers: A survey. Engineering Applications of Artificial Intelligence, 126:106669, 2023

    2023

  39. [39]

    Adapting seg- ment anything model to multi-modal salient object detection with semantic feature fusion guidance

    Kunpeng Wang, Danying Lin, Chenglong Li, Zhengzheng Tu, and Bin Luo. Adapting seg- ment anything model to multi-modal salient object detection with semantic feature fusion guidance. arXiv preprint arXiv:2408.15063, 2024

    Pith/arXiv arXiv 2024

  40. [40]

    Multi- modal token fusion for vision transformers

    Yikai Wang, Xinghao Chen, Lele Cao, Wenbing Huang, Fuchun Sun, and Yunhe Wang. Multi- modal token fusion for vision transformers. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022

    2022

  41. [41]

    Medical sam adapter: Adapting segment anything model for medical image segmentation.arXiv preprint arXiv:2304.12620, 2023

    Junde Wu, Wei Ji, Yuanpei Liu, Huazhu Fu, Min Xu, Yanwu Xu, and Yueming Jin. Medical sam adapter: Adapting segment anything model for medical image segmentation.arXiv preprint arXiv:2304.12620, 2023

    Pith/arXiv arXiv 2023

  42. [42]

    Segment any- thing with multiple modalities.arXiv preprint arXiv:2408.09085, 2024

    Aoran Xiao, Weihao Xuan, Heli Qi, Yun Xing, Naoto Yokoya, and Shijian Lu. Segment any- thing with multiple modalities.arXiv preprint arXiv:2408.09085, 2024

    Pith/arXiv arXiv 2024

  43. [43]

    Seg- former: Simple and efficient design for semantic segmentation with transformers

    Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Seg- former: Simple and efficient design for semantic segmentation with transformers. InNeural Infor- mation Processing Systems (NeurIPS), 2021

    2021

  44. [44]

    Parameter-efficient fine-tuning for pre- trained vision models: A survey.arXiv preprint arXiv:2402.02242, 2024

    Yi Xin, Siqi Luo, Haodi Zhou, Junlong Du, Xiaohong Liu, Yue Fan, Qing Li, and Yun- tao Du. Parameter-efficient fine-tuning for pre- trained vision models: A survey.arXiv preprint arXiv:2402.02242, 2024

    arXiv 2024

  45. [45]

    Sam-event-adapter: Adapting segment anything model for event-rgb se- mantic segmentation

    Bowen Yao, Yongjian Deng, Yuhan Liu, Hao Chen, Youfu Li, and Zhen Yang. Sam-event-adapter: Adapting segment anything model for event-rgb se- mantic segmentation. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 9093–9100, 2024

    2024

  46. [46]

    Zeiler, Dilip Krishnan, Geoffrey W

    Matthew D. Zeiler, Dilip Krishnan, Geoffrey W. Taylor, and Rob Fergus. Deconvolutional net- works. In2010 IEEE Conference on Computer Vi- sion and Pattern Recognition (CVPR), pages 2528–

  47. [47]

    Cmx: Cross-modal fusion for rgb-x semantic segmen- tation with transformers.IEEE Transactions on Intelligent Transportation Systems, 2023

    Jiaming Zhang, Huayao Liu, Kailun Yang, Xinxin Hu, Ruiping Liu, and Rainer Stiefelhagen. Cmx: Cross-modal fusion for rgb-x semantic segmen- tation with transformers.IEEE Transactions on Intelligent Transportation Systems, 2023

    2023

  48. [48]

    Delivering arbitrary-modal semantic segmentation

    Jiaming Zhang, Ruiping Liu, Hao Shi, Kailun Yang, Simon Reiß, Kunyu Peng, Haodong Fu, Kai- wei Wang, and Rainer Stiefelhagen. Delivering arbitrary-modal semantic segmentation. InCVPR, 2023

    2023

  49. [49]

    Deep multimodal fusion for semantic image segmentation: A survey.Image and Vision Computing, 105:104042, 2021

    Yifei Zhang, D ´esir´e Sidib ´e, Olivier Morel, and Fabrice M´eriaudeau. Deep multimodal fusion for semantic image segmentation: A survey.Image and Vision Computing, 105:104042, 2021

    2021

  50. [50]

    Deformable{detr}: Deformable transformers for end-to-end object de- tection

    Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiao- gang Wang, and Jifeng Dai. Deformable{detr}: Deformable transformers for end-to-end object de- tection. InInternational Conference on Learning Representations, 2021

    2021

  51. [51]

    Segment everything every- where all at once.Advances in Neural Information Processing Systems, 36, 2024

    Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and Yong Jae Lee. Segment everything every- where all at once.Advances in Neural Information Processing Systems, 36, 2024. 18 Biographies Iacopo Curtireceived his Master Degree in Au- tomation Engineering in 2023 from the University of Bologna. He is a PhD s...

    2024