Pith. sign in

REVIEW 3 major objections 6 minor 80 references

Segment Any RGB-Thermal Model with Language-aided Distillation

T0 review · 3 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims SARTM, a SAM2-based RGB-thermal segmentation framework, sets new state-of-the-art results on three benchmarks by combining LoRA fine-tuning with CLIP-guided knowledge distillation.

desk verdict A promising PST900 recipe, undermined by an FMB table that duplicates SegMiF's per-class numbers with an incompatible mIoU, and an overclaim of novelty relative to its own references. read the letter →

arxiv 2505.01950 v1 pith:WIPFGDTU submitted 2025-05-04 cs.CV cs.AI

Dong Xing , Xianxun Zhu , Wei Zhou , Qika Lin , Hang Yang , Yuqing Wang This is my paper · ORCID
classification cs.CVcs.AI
keywords RGB-thermalsemanticsegmentationSAM2SegmentAnythingModelLoRACLIPknowledgedistillationcross-modalfusionmulti-scalefeature
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SARTM is a framework for RGB-thermal semantic segmentation built on SAM2. It freezes SAM2's image encoder, adds modality-specific LoRA layers so the backbone can adapt to thermal input without losing its pretrained knowledge, and adds a dual-path decoder with an auxiliary segmentation head for multi-scale fusion. The paper's central claim is that language helps: CLIP text embeddings of class names are distilled into the visual features by aligning their self-similarity matrices, which the authors say resolves cross-modal ambiguity and improves category discrimination. On the three standard benchmarks the framework reports state-of-the-art mean intersection-over-union scores of 89.88 on PST900, 60.03 on MFNet, and 61.57 on FMB. If true, this would matter because RGB-thermal segmentation is a practical route to reliable scene understanding in low-light, foggy, and overexposed conditions, and adapting a general segmentation foundation model with small trainable components would be cheaper than training a specialised network from scratch.

What carries the argument

The load-bearing object is the Cross-Modal Knowledge Distillation (CMKD) module. It performs Mask Average Pooling on the fused features, using ground-truth labels as class masks, to obtain one vector per category; then it computes the cosine self-similarity matrix of those vectors and of CLIP text embeddings for the class names, and minimizes the KL divergence between the two matrices. This relational transfer is what the paper says injects semantic understanding into the fused RGB-thermal representation. The rest of the machinery supports it: modality-specific LoRA layers adapt the frozen SAM2 Hiera encoder, and a dual-path decoder combines the semantic feature map, fine-grained feature pyramid, and intermediate-resolution feature pyramid through SAM2's mask decoder plus an auxiliary FPN-style head.

What would settle it

Train SARTM on PST900 and FMB with the CLIP text embeddings replaced by random fixed vectors of the same dimension, keeping every other component and loss weight unchanged; if the mIoU drop is the same as the reported drop when the language loss is removed, about 3.6 and 2.7 points, then the language content itself is not doing the work. A second check is to inspect the implementation of Equation 13: as written it compares pooled visual features to ground-truth labels, so if that is literally what is computed, the language term is partly label supervision rather than distillation from text.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a frozen SAM2 can be turned into a top RGB-thermal semantic segmenter by three coordinated changes: low-rank (LoRA) adaptation of the image encoder per modality, a redesigned dual-path segmentation head that fuses semantic, fine-grained, and intermediate-resolution feature pyramids, and a Cross-Modal Knowledge Distillation module that uses CLIP's text encoder as a teacher. The teacher's class-name embeddings are compared with mask-pooled visual features through their cosine self-similarity matrices, and a KL divergence (Equation 14) pushes the student to reproduce the semantic relations among categories. The authors report that this combination achieves the best published mIoU on PST900 and FMB and competitive state-of-the-art on MFNet, with the language term contributing about 3.6 and 2.7 mIoU points on PST900 and FMB respectively in the ablation.

Load-bearing premise

The framework's central premise is that CLIP's text embeddings for class names, when aligned to mask-pooled visual features through the similarity-matching loss, actually deliver semantic knowledge that resolves RGB-thermal ambiguity; the evidence for this is an ablation that only turns the language loss off, so the premise is asserted rather than demonstrated.

Editorial extensions

If this is right

  • SAM2 can be reused for RGB-thermal segmentation with only LoRA matrices and segmentation heads trainable, keeping the frozen image encoder, so deployment cost stays well below training a specialised RGB-T network from scratch.
  • The language-guided relational distillation should improve category separation specifically in low-light and overexposed scenes, where RGB and thermal evidence conflict, matching the PST900 qualitative results.
  • Because the LoRA layers are modality-specific, the same architecture should transfer to other RGB-X inputs such as depth, event streams, or near-infrared with little modification.
  • The reported state-of-the-art results on PST900 and FMB, and the narrow gap on MFNet, imply that foundation-model adaptation is now a competitive recipe for RGB-T semantic segmentation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the language mechanism is not cleanly isolated by the paper, since the only evidence is an ablation that removes the loss; a reviewer-friendly experiment would replace CLIP embeddings with random fixed vectors of the same dimension and check whether the mIoU drop persists.
  • Inference: the self-similarity alignment in Equation 14 is essentially a class-relation regularizer, so if it works, a purely visual prototype-consistency loss without any text encoder should reproduce most of the gain, which would be cheaper and work for class names unseen by CLIP.
  • Inference: the approach as described uses only static class-name embeddings, not richer language; extending the teacher to attribute phrases or scene captions is a natural test of whether semantic language content, rather than label statistics, is what helps.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes SARTM, a SAM2-based framework for RGB-thermal semantic segmentation. The SAM2 image encoder is frozen and adapted with modality-specific LoRA layers; three feature hierarchies (SFM, FFP, IFP) are fused; a dual-path decoder with an auxiliary segmentation head produces predictions; and CLIP text embeddings of class names are used in a cross-modal knowledge distillation loss. Experiments on PST900, MFNet, and FMB report state-of-the-art mIoUs of 89.88, 60.03, and 61.57, respectively, and the paper claims significant improvement over prior RGB-T methods.

Significance. If the reported results are correct, the paper offers a practical and timely recipe for adapting SAM2 to multimodal semantic segmentation through parameter-efficient LoRA fine-tuning, and the idea of injecting CLIP-derived language guidance into RGB-T feature fusion is worth investigating. The manuscript includes useful ablations of LoRA rank, loss weights, and component removal, as well as t-SNE and similarity-matrix visualizations. However, the central cross-benchmark claim is not currently trustworthy: the FMB table contains an apparent numerical inconsistency and an unexplained duplication of another method's per-class scores, the MFNet text contradicts its own table, and the main language-aided distillation loss is not formally defined. These issues are load-bearing because the headline claim rests on all three benchmark results and on the language-guidance mechanism.

major comments (3)
  1. [Table III] Table III reports per-class IoUs for only eight classes, while the FMB dataset has 15 categories. The SARTM row lists per-class values (78.3, 65.4, 47.3, 43.1, 74.8, 82.0, 85.0, 49.8) that are identical to the SegMiF row, yet SARTM's mIoU is 61.57 while SegMiF's is 54.8. The mean of the eight listed values is 65.71, not 61.57. If the table intentionally shows only a subset of classes, that must be stated and the reported mIoU must be derived from a defined class set; if the table shows the full set, the mIoU is arithmetically wrong. In either case, identical per-class numbers for two independently developed methods cannot be accepted without explanation. The FMB portion of the abstract's 'significantly outperforms' claim is therefore unsupported as printed.
  2. [Section IV.B, Table II] The text states that on the MFNet dataset 'our method slightly underperforms compared to the best-performing method in terms of mIoU,' but Table II shows SARTM with the highest mIoU (60.03) against CMX (59.7) and all other listed methods. This direct contradiction between the text and the table should be resolved, since the paper's characterization of its own results is part of the evidence for the claimed cross-benchmark superiority.
  3. [Section III.E, Eqs. (13)-(14)] The language-aided distillation loss is not well-defined as written. Eq. (13) sets Lcr = KL(f, y), where f is a feature tensor and y is ground-truth labels; KL divergence is not defined between such objects, so this term cannot be implemented as stated. Eq. (14) applies KL to cosine-similarity matrices without converting them into normalized probability distributions, and the dimensions of Te and the feature set {f0,...,fK} are not specified. Since this loss is the entire mechanism for the claimed language guidance, the ablation in Table V cannot be interpreted until the loss is properly defined.
minor comments (6)
  1. [Section III.F] The heading 'Ovearll Training Objectives' contains a typo; it should be 'Overall Training Objectives.'
  2. [Table IV] The text says LoRA ranks 2, 4, 16, 32, and 64 were tested, but the table lists 4, 8, 16, 32, and 64, with rank 2 missing and rank 8 appearing instead. The text and table should be aligned.
  3. [Section III.D] The description of the second pathway contains garbled and duplicated phrases, e.g., 'ierarchical feature integration framework that aggregates multi-scale features into a unified high-resolution embedding' and 'Finlly,fpn merge.' The paragraph needs careful rewriting for clarity.
  4. [Figures 4-6] The qualitative figure captions are inconsistently matched to the text: for example, the text describing Figure 5 mentions CMNext and CAINet while the caption references CAINet and MMSFormer, and the text in Section IV.C has typos such as 'ocjects' and 'metods.'
  5. [Figure 8] The caption contains repeated typos: 'Epoach' should be 'Epoch.'
  6. [Table VII] The loss-weight ablation reports values for w1, w2, and w3, but the mapping of these symbols to the weights in Eq. (15) is not stated, and the table does not specify how w0 is set. The optimal mIoU of 60.03 is reported with w3 = 100, but the table's rows do not show a single joint configuration, so the reader cannot verify how the combination was obtained.

Circularity Check

1 steps flagged · score 6.0 of 10

FMB SOTA claim is a relabeled SegMiF row with an inflated mIoU; the rest of the pipeline shows no circular fit.

  1. renaming known result [Section IV-B, Table III (FMB test set)]
    "On the FMB dataset, our method outperforms the comparison methods ... SegMiF [70] 78.3 65.4 47.3 43.1 74.8 82.0 85.0 49.8 54.8 ... SARTM (Ours) 78.3 65.4 47.3 43.1 74.8 82.0 85.0 49.8 61.57"

    The per-class IoUs in the SARTM row of Table III are identical, class by class, to those in the SegMiF row (78.3, 65.4, 47.3, 43.1, 74.8, 82.0, 85.0, 49.8), so the FMB 'prediction' is not an independent result but the known SegMiF result relabeled as SARTM. The only difference is the reported mIoU (61.57 vs. 54.8); neither number equals the mean of the eight printed class values (65.71). Thus the claimed FMB superiority is equivalent to its own input table row by construction, and the central 'state-of-the-art across three benchmarks' claim rests in part on this renamed result.

full rationale

Apart from Table III, the method is a supervised fine-tuning of SAM2 with LoRA and a CLIP-text distillation regularizer, evaluated on held-out test splits of PST900, MFNet, and FMB. The distillation losses (Eqs. 13-15) are defined on training features, labels, and CLIP text embeddings, not on the reported mIoU, so there is no fitted-parameter-renamed-as-prediction or self-definitional loop. Self-citations (e.g., Wei Zhou's MFFENet, EGFNet, GMNet) appear only as baselines or background, not as load-bearing justification. The one serious issue is Table III: the SARTM row's per-class IoUs are identical to SegMiF's while the mIoU is inflated and inconsistent with the per-class mean; this is a renamed known result for the FMB component of the central SOTA claim, warranting a partial-circularity score of 6 even though the PST900 and MFNet claims are not shown to be circular. Separately, Eq. 13 (Lcr = KL(f, y)) is mathematically undefined as written and Eq. 14 applies KL to unnormalized cosine-similarity matrices; these are correctness and reproducibility defects, not circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper rests on external pretrained models (SAM2, CLIP), benchmark dataset labels, and a set of manually tuned hyperparameters. No new physical or conceptual entities are introduced. The tuned loss weights and LoRA rank are the main free parameters, and the stated assumptions about transferability of RGB-pretrained representations and CLIP text guidance are taken for granted.

free parameters (5)
  • LoRA rank r = 16
    Chosen from ablation on PST900 (Table IV); rank 16 gives best mIoU 89.88.
  • Loss weight w1 (aux head CE) = 0.008
    Tuned on MFNet (Table VII); 0.008 gives 58.67 mIoU before other weights are set.
  • Loss weight w2 (Lcr) = 10000
    Tuned on MFNet (Table VII); value 10000 gives 59.32 mIoU.
  • Loss weight w3 (Lse) = 100
    Tuned on MFNet (Table VII); value 100 gives final 60.03 mIoU.
  • OHEM pixel threshold factor = ntotal/16
    Defined in Eq. 11; controls the number of hard pixels used in CE loss.
assumptions (4)
  • domain assumption SAM2, pretrained on RGB images, can be adapted to thermal input by LoRA adapters without replacing the encoder.
    The whole method in Sec. III-B freezes the SAM2 encoder and adds LoRA; no analysis validates that this is sufficient for thermal imagery.
  • domain assumption CLIP text embeddings of class names are a suitable semantic teacher for RGB-T segmentation.
    Sec. III-E uses CLIP text features extracted from class names as a distillation target; the paper provides no evidence that these embeddings align with thermal features.
  • domain assumption The SAM2 mask decoder can be repurposed for semantic segmentation with an auxiliary head and feature pyramids.
    Sec. III-D modifies the decoder without theoretical or empirical justification beyond the reported tables.
  • standard math Cross-entropy, KL divergence, and self-similarity matrices behave as standard.
    Eqs. 11-14 rely on standard definitions; Eq. 13 is under-specified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Segment Any RGB-Thermal Model with Language-aided Distillation." pith.science (2026). https://pith.science/paper/WIPFGDTU

@misc{pith2026250501950,
  author       = {Pith},
  title        = {Pith review of: Segment Any RGB-Thermal Model with Language-aided Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WIPFGDTU}},
  note         = {Machine review of arXiv:2505.01950}
}
read the original abstract

The recent Segment Anything Model (SAM) demonstrates strong instance segmentation performance across various downstream tasks. However, SAM is trained solely on RGB data, limiting its direct applicability to RGB-thermal (RGB-T) semantic segmentation. Given that RGB-T provides a robust solution for scene understanding in adverse weather and lighting conditions, such as low light and overexposure, we propose a novel framework, SARTM, which customizes the powerful SAM for RGB-T semantic segmentation. Our key idea is to unleash the potential of SAM while introduce semantic understanding modules for RGB-T data pairs. Specifically, our framework first involves fine tuning the original SAM by adding extra LoRA layers, aiming at preserving SAM's strong generalization and segmentation capabilities for downstream tasks. Secondly, we introduce language information as guidance for training our SARTM. To address cross-modal inconsistencies, we introduce a Cross-Modal Knowledge Distillation(CMKD) module that effectively achieves modality adaptation while maintaining its generalization capabilities. This semantic module enables the minimization of modality gaps and alleviates semantic ambiguity, facilitating the combination of any modality under any visual conditions. Furthermore, we enhance the segmentation performance by adjusting the segmentation head of SAM and incorporating an auxiliary semantic segmentation head, which integrates multi-scale features for effective fusion. Extensive experiments are conducted across three multi-modal RGBT semantic segmentation benchmarks: MFNET, PST900, and FMB. Both quantitative and qualitative results consistently demonstrate that the proposed SARTM significantly outperforms state-of-the-art approaches across a variety of conditions.

Figures

Figures reproduced from arXiv: 2505.01950 by the authors.

Figure 1
Figure 1. Overall framework of our proposed SARTM, consists of original [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the proposed SARTM framework for multi-modal semantic segmentation. The architecture combines multi-scale features from a frozen image encoder fine-tuned with LoRA layers. where the spatial dimensions at stage i are given by: Hi = H/si , Wi = W/si , si = 2i+2 . (4) A window-based multi-head self-attention mechanism is employed at each stage: Attention(Q, K, V ) = softmax  QK⊤ √ dk  V, (5) where Q, … view at source ↗
Figure 3
Figure 3. (a) Hierarchical Feature Fusion for Cross-Scale Information Aggregation using Semantic Feature Map (SFM), Fine-Grained Feature Pyramid (FFP), [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Qualitatively compared with the SoTA RGB-T scene resolution network on the PST900 test set, where areas of significant improvement are shown [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Qualitatively compared with the SoTA RGB-T scene resolution network on the MFNet test set, where areas of significant improvement are shown in [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Qualitatively compared with the SoTA RGB-T scene resolution network on the FMB test set, where areas of significant improvement are shown in [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: t-SNE visualization based on the FMB dataset. (a)Comparison between SAM and the addition of an auxiliary segmentation head. (b)Comparison [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Visualization of the similarity matrix at different epochs, showing the transition from a dispersed pattern at Epoch 1 to a more structured and unified [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

80 extracted references · 62 canonical work pages

  1. [31]

    Customize segment anything model for multi-modal semantic segmentation with mixture of lora experts,

    C. Zhu, B. Xiao, L. Shi, S. Xu, and X. Zheng, “Customize segment anything model for multi-modal semantic segmentation with mixture of lora experts,” arXiv preprint arXiv:2412.04220 , 2024. 3

    arXiv 2024

  2. [32]

    Unveiling the potential of segment anything model 2 for rgb- thermal semantic segmentation with language guidance,

    J. Zhao, F. Teng, K. Luo, G. Zhao, Z. Li, X. Zheng, and K. Yang, “Unveiling the potential of segment anything model 2 for rgb- thermal semantic segmentation with language guidance,” arXiv preprint arXiv:2503.02581, 2025. 3

    arXiv 2025

  3. [1]

    Deep learning for event-based vision: A comprehensive survey and benchmarks,

    X. Zheng, Y . Liu, Y . Lu, T. Hua, T. Pan, W. Zhang, D. Tao, and L. Wang, “Deep learning for event-based vision: A comprehensive survey and benchmarks,” arXiv preprint arXiv:2302.08890 , 2023. 1

    arXiv 2023

  4. [2]

    Eventdance: Unsupervised source-free cross- modal adaptation for event-based object recognition,

    X. Zheng and L. Wang, “Eventdance: Unsupervised source-free cross- modal adaptation for event-based object recognition,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 17 448–17 458. 1

    work page 2024

  5. [3]

    Exact: Language-guided conceptual reasoning and uncertainty estimation for event-based action recognition and more,

    J. Zhou, X. Zheng, Y . Lyu, and L. Wang, “Exact: Language-guided conceptual reasoning and uncertainty estimation for event-based action recognition and more,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 18 633–18 643. 1

    work page 2024

  6. [4]

    Dehazed image quality evaluation: From partial discrepancy to blind perception,

    W. Zhou, R. Zhang, L. Li, G. Yue, J. Gong, H. Chen, and H. Liu, “Dehazed image quality evaluation: From partial discrepancy to blind perception,” IEEE Transactions on Intelligent Vehicles, vol. 9, no. 2, pp. 3843–3858, 2024. 1

    work page 2024

  7. [5]

    Context-aware interaction network for rgb-t semantic segmentation,

    Y . Lv, Z. Liu, and G. Li, “Context-aware interaction network for rgb-t semantic segmentation,” IEEE Transactions on Multimedia , 2024. 1, 6, 7

    work page 2024

  8. [6]

    Mfnet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes,

    Q. Ha, K. Watanabe, T. Karasawa, Y . Ushiku, and T. Harada, “Mfnet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes,” in 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2017, pp. 5108–5115. 1, 2, 6, 7

    work page 2017

Show all 80 references
  1. [7]

    Multi-interactive feature learning and a full-time multi-modality benchmark for image fusion and segmentation,

    J. Liu, Z. Liu, G. Wu, L. Ma, R. Liu, W. Zhong, Z. Luo, and X. Fan, “Multi-interactive feature learning and a full-time multi-modality benchmark for image fusion and segmentation,” in Proceedings of the IEEE/CVF international conference on computer vision, 2023, pp. 8115–

    2023

  2. [8]

    Pst900: Rgb-thermal calibration, dataset and segmentation network,

    S. S. Shivakumar, N. Rodrigues, A. Zhou, I. D. Miller, V . Kumar, and C. J. Taylor, “Pst900: Rgb-thermal calibration, dataset and segmentation network,” in 2020 IEEE International Conference on Robotics and Automation (ICRA), 2020, pp. 9441–9447. 1, 2, 6, 7

    2020

  3. [9]

    Mffenet: Multiscale feature fusion and enhancement network for rgb–thermal urban road scene parsing,

    W. Zhou, X. Lin, J. Lei, L. Yu, and J.-N. Hwang, “Mffenet: Multiscale feature fusion and enhancement network for rgb–thermal urban road scene parsing,” IEEE Transactions on Multimedia , vol. 24, pp. 2526– 2538, 2022. 1

    2022

  4. [10]

    Rtfnet: Rgb-thermal fusion network for semantic segmentation of urban scenes,

    Y . Sun, W. Zuo, and M. Liu, “Rtfnet: Rgb-thermal fusion network for semantic segmentation of urban scenes,” IEEE Robotics and Automation Letters, vol. 4, no. 3, pp. 2576–2583, 2019. 1, 6, 7

    2019

  5. [11]

    Fuseseg: Semantic segmentation of urban scenes based on rgb and thermal data fusion,

    Y . Sun, W. Zuo, P. Yun, H. Wang, and M. Liu, “Fuseseg: Semantic segmentation of urban scenes based on rgb and thermal data fusion,” IEEE Transactions on Automation Science and Engineering , vol. 18, no. 3, pp. 1000–1011, 2020. 1, 7

    2020

  6. [12]

    Context-aware interaction network for rgb-t semantic segmentation,

    Y . Lv, Z. Liu, and G. Li, “Context-aware interaction network for rgb-t semantic segmentation,” IEEE Transactions on Multimedia , vol. 26, pp. 6348–6360, 2024. 1, 2, 7

    2024

  7. [13]

    Segment anything,

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Lo, P. Doll´ar, and R. Girshick, “Segment anything,” in 2023 IEEE/CVF International Conference on Computer Vision (ICCV) . IEEE, 2023, pp. 4015–4026. 1, 2

    2023

  8. [14]

    Sam 2: Segment anything in images and videos,

    N. Ravi, V . Gabeur, Y .-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. R ¨adle, C. Rolland, L. Gustafson, E. Mintun, J. Pan, K. V . Alwala, N. Carion, C.-Y . Wu, R. Girshick, P. Doll ´ar, and C. Feichtenhofer, “Sam 2: Segment anything in images and videos,” arXiv preprint JOURNA...

    2021 arXiv

  9. [15]

    Msgfusion: Medical semantic guided two-branch network for multi- modal brain image fusion,

    J. Wen, F. Qin, J. Du, M. Fang, X. Wei, C. L. P. Chen, and P. Li, “Msgfusion: Medical semantic guided two-branch network for multi- modal brain image fusion,” IEEE Transactions on Multimedia , vol. 26, pp. 944–957, 2024. 2

    2024

  10. [16]

    Cddfuse: Correlation-driven dual-branch feature decomposition for multi-modality image fusion,

    Z. Zhao, H. Bai, J. Zhang, Y . Zhang, S. Xu, Z. Lin, R. Timofte, and L. Van Gool, “Cddfuse: Correlation-driven dual-branch feature decomposition for multi-modality image fusion,” in 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 5906–5916. 2

    2023

  11. [17]

    Multi-focus image fusion based on multi-scale gradients and image matting,

    J. Chen, X. Li, L. Luo, and J. Ma, “Multi-focus image fusion based on multi-scale gradients and image matting,” IEEE Transactions on Multimedia, vol. 24, pp. 655–667, 2022. 2

    2022

  12. [18]

    Ifsepr: A general framework for image fusion based on separate representation learning,

    X. Luo, Y . Gao, A. Wang, Z. Zhang, and X.-J. Wu, “Ifsepr: A general framework for image fusion based on separate representation learning,” IEEE Transactions on Multimedia , vol. 25, pp. 608–623, 2023. 2

    2023

  13. [19]

    Imagebind-llm: Multi-modality instruction tuning,

    J. Han, R. Zhang, W. Shao, P. Gao, P. Xu, H. Xiao, K. Zhang, C. Liu, S. Wen, Z. Guo et al. , “Imagebind-llm: Multi-modality instruction tuning,” arXiv preprint arXiv:2309.03905 , 2023. 2

    2023 arXiv

  14. [20]

    Llama-adapter v2: Parameter-efficient visual instruction model,

    P. Gao, J. Han, R. Zhang, Z. Lin, S. Geng, A. Zhou, W. Zhang, P. Lu, C. He, X. Yue et al. , “Llama-adapter v2: Parameter-efficient visual instruction model,” arXiv preprint arXiv:2304.15010 , 2023. 2

    2023 arXiv

  15. [21]

    Instructblip: Towards general-purpose vision-language models with instruction tuning. arxiv 2023,

    W. Dai, J. Li, D. Li, A. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and S. Hoi, “Instructblip: Towards general-purpose vision-language models with instruction tuning. arxiv 2023,” arXiv preprint arXiv:2305.06500 , vol. 2, 2023. 2

    2023 arXiv

  16. [22]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning . PMLR, 2021, pp. 8748–8763. 2, 3, 5

    2021

  17. [23]

    Unleash the power of vision-language models by visual attention prompt and multi-modal interaction,

    W. Zhang, L. Wu, Z. Zhang, T. Yu, C. Ma, X. Jin, X. Yang, and W. Zeng, “Unleash the power of vision-language models by visual attention prompt and multi-modal interaction,” IEEE Transactions on Multimedia, 2024. 2

    2024

  18. [24]

    Multi-task paired masking with alignment modeling for medical vision- language pre-training,

    K. Zhang, Y . Yang, J. Yu, H. Jiang, J. Fan, Q. Huang, and W. Han, “Multi-task paired masking with alignment modeling for medical vision- language pre-training,” IEEE Transactions on Multimedia , vol. 26, pp. 4706–4721, 2024. 2

    2024

  19. [25]

    Joint bilateral upsampling,

    J. Kopf, M. F. Cohen, D. Lischinski, and M. Uyttendaele, “Joint bilateral upsampling,” ACM Transactions on Graphics , vol. 26, no. 3, pp. 96:1– 96:6, 2007. 2, 3

    2007

  20. [26]

    Distilling efficient vision transformers from cnns for semantic segmentation,

    X. Zheng, Y . Luo, P. Zhou, and L. Wang, “Distilling efficient vision transformers from cnns for semantic segmentation,” Pattern Recognition, vol. 158, p. 111029, 2025. 2

    2025

  21. [27]

    Eventbind: Learning a unified representation to bind them all for event-based open-world understanding,

    J. Zhou, X. Zheng, Y . Lyu, and L. Wang, “Eventbind: Learning a unified representation to bind them all for event-based open-world understanding,” in European Conference on Computer Vision. Springer, 2024, pp. 477–494. 2

    2024

  22. [28]

    Magic++: Efficient and resilient modality-agnostic semantic segmentation via hierarchical modality selection,

    X. Zheng, Y . Lyu, L. Jiang, J. Zhou, L. Wang, and X. Hu, “Magic++: Efficient and resilient modality-agnostic semantic segmentation via hierarchical modality selection,” arXiv preprint arXiv:2412.16876, 2024. 2

    2024 arXiv

  23. [29]

    Learning robust anymodal segmentor with unimodal and cross-modal distillation,

    X. Zheng, H. Xue, J. Chen, Y . Yan, L. Jiang, Y . Lyu, K. Yang, L. Zhang, and X. Hu, “Learning robust anymodal segmentor with unimodal and cross-modal distillation,” arXiv preprint arXiv:2411.17141 , 2024. 2, 3

    2024 arXiv

  24. [30]

    Mrfs: Mutually rein- forcing image fusion and segmentation,

    H. Zhang, X. Zuo, J. Jiang, C. Guo, and J. Ma, “Mrfs: Mutually rein- forcing image fusion and segmentation,” in 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 26 964– 26 973. 2, 6, 9

    2024

  25. [33]

    Omnisam: Omnidirectional segment anything model for uda in panoramic semantic segmentation,

    D. Zhong, X. Zheng, C. Liao, Y . Lyu, J. Chen, S. Wu, L. Zhang, and X. Hu, “Omnisam: Omnidirectional segment anything model for uda in panoramic semantic segmentation,” arXiv preprint arXiv:2503.07098 ,

  26. [34]

    Eviprompt: A training-free evidential prompt generation method for adapting segment anything model in medical images,

    Y . Xu, J. Tang, A. Men, and Q. Chen, “Eviprompt: A training-free evidential prompt generation method for adapting segment anything model in medical images,” IEEE Transactions on Image Processing , vol. 33, pp. 6204–6215, 2024. 3

    2024

  27. [35]

    Unleashing the potential of sam2 for biomedical images and videos: A survey,

    Y . Zhang and Z. Shen, “Unleashing the potential of sam2 for biomedical images and videos: A survey,” arXiv preprint arXiv:2408.12889 , 2024. 3

    2024 arXiv

  28. [36]

    Towards segment anything model (sam) for med- ical image segmentation: a survey,

    Y . Zhang and R. Jiao, “Towards segment anything model (sam) for med- ical image segmentation: a survey,” arXiv preprint arXiv:2305.03678 ,

    arXiv

  29. [37]

    Segment anything model for medical image segmentation: Current applications and future directions,

    “Segment anything model for medical image segmentation: Current applications and future directions,” Computers in Biology and Medicine, vol. 171, p. 108238, 2024. 3

    2024

  30. [38]

    Samrs: Scaling-up remote sensing segmentation dataset with segment anything model,

    D. Wang, J. Zhang, B. Du, M. Xu, L. Liu, D. Tao, and L. Zhang, “Samrs: Scaling-up remote sensing segmentation dataset with segment anything model,” Advances in Neural Information Processing Systems , vol. 36,

  31. [39]

    Ringmo-sam: A foundation model for segment anything in multimodal remote-sensing images,

    Z. Yan, J. Li, X. Li, R. Zhou, W. Zhang, Y . Feng, W. Diao, K. Fu, and X. Sun, “Ringmo-sam: A foundation model for segment anything in multimodal remote-sensing images,” IEEE Transactions on Geoscience and Remote Sensing , vol. 61, pp. 1–16, 2023. 3

    2023

  32. [40]

    Rsprompter: Learning to prompt for remote sensing instance seg- mentation based on visual foundation model,

    K. Chen, C. Liu, H. Chen, H. Zhang, W. Li, Z. Zou, and Z. Shi, “Rsprompter: Learning to prompt for remote sensing instance seg- mentation based on visual foundation model,” IEEE Transactions on Geoscience and Remote Sensing , 2024. 3

    2024

  33. [41]

    Uvosam: A mask-free paradigm for unsupervised video object segmentation via segment anything model,

    Z. Zhang, Z. Wei, S. Zhang, Z. Dai, and S. Zhu, “Uvosam: A mask-free paradigm for unsupervised video object segmentation via segment anything model,” ArXiv, vol. abs/2305.12659, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:258833355 3

    2023 arXiv

  34. [42]

    Foodsam: Any food segmentation,

    X. Lan, J. Lyu, H. Jiang, K. Dong, Z. Niu, Y . Zhang, and J. Xue, “Foodsam: Any food segmentation,” IEEE Transactions on Multimedia, pp. 1–14, 2023. 3

    2023

  35. [43]

    Recalling unknowns without losing precision: An effective solution to large model-guided open world object detection,

    Y . He, W. Chen, S. Wang, T. Liu, and M. Wang, “Recalling unknowns without losing precision: An effective solution to large model-guided open world object detection,” IEEE Transactions on Image Processing , vol. 34, pp. 729–742, 2025. 3

    2025

  36. [44]

    Segmenting anything in the dark via depth perception,

    P. Liu, J. Deng, L. Duan, W. Li, and F. Lv, “Segmenting anything in the dark via depth perception,” IEEE Transactions on Multimedia , pp. 1–12, 2025. 3

    2025

  37. [45]

    Segment and track anything,

    Y . Cheng, L. Li, Y . Xu, X. Li, Z. Yang, W. Wang, and Y . Yang, “Segment and track anything,” arXiv preprint arXiv:2305.06558 , 2023. 3

    2023 arXiv

  38. [46]

    Rog- sam: A language-driven framework for instance-level robotic grasping detection,

    Y . Mei, J. Sun, Z. Peng, F. Deng, G. Wang, and J. Chen, “Rog- sam: A language-driven framework for instance-level robotic grasping detection,” IEEE Transactions on Multimedia , pp. 1–13, 2025. 3

    2025

  39. [47]

    Frequency-guided spatial adaptation for camouflaged object detection,

    S. Zhang, D. Kong, Y . Xing, Y . Lu, L. Ran, G. Liang, H. Wang, and Y . Zhang, “Frequency-guided spatial adaptation for camouflaged object detection,” IEEE Transactions on Multimedia, vol. 27, pp. 72–83, 2025. 3

    2025

  40. [48]

    Nto3d: Neural target object 3d reconstruction with segment anything,

    X. Wei, R. Zhang, J. Wu, J. Liu, M. Lu, Y . Guo, and S. Zhang, “Nto3d: Neural target object 3d reconstruction with segment anything,” in 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 20 352–20 362. 3

    2024

  41. [49]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in Proceedings of the International conference on machine learning (ICML) , 2021, pp. 8748–

    2021

  42. [50]

    Show, attend and tell: Neural image caption generation with visual attention,

    K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. Zemel, and Y . Bengio, “Show, attend and tell: Neural image caption generation with visual attention,” in Proceedings of the International conference on machine learning (ICML) , 2015, pp. 2048–2057. 3

    2015

  43. [51]

    Instructpix2pix: Learning to follow image editing instructions,

    T. Brooks, A. Holynski, and A. A. Efros, “Instructpix2pix: Learning to follow image editing instructions,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 18 392–18 402. 3

    2023

  44. [52]

    Unibind: Llm-augmented unified and balanced representation space to bind them all,

    Y . Lyu, X. Zheng, J. Zhou, and L. Wang, “Unibind: Llm-augmented unified and balanced representation space to bind them all,” in Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 26 752–26 762. 3

    2024

  45. [53]

    Omnibind: Teach to build unequal-scale modality interaction for omni-bind of all,

    Y . Lyu, X. Zheng, D. Kim, and L. Wang, “Omnibind: Teach to build unequal-scale modality interaction for omni-bind of all,” arXiv preprint arXiv:2405.16108, 2024. 3

    2024 arXiv

  46. [54]

    Vision-language consistency guided multi-modal prompt learning for blind ai generated image quality assessment,

    J. Fu, W. Zhou, Q. Jiang, H. Liu, and G. Zhai, “Vision-language consistency guided multi-modal prompt learning for blind ai generated image quality assessment,” IEEE Signal Processing Letters, vol. 31, pp. 1820–1824, 2024. 3

    2024

  47. [55]

    Dall-e: Creating images from text,

    M. D. M. Reddy, M. S. M. Basha, M. M. C. Hari, and M. N. Penchalaiah, “Dall-e: Creating images from text,” UGC Care Group I Journal, vol. 8, no. 14, pp. 71–75, 2021. 3

    2021

  48. [56]

    Gpt-4 technical report,

    OpenAI, “Gpt-4 technical report,” ArXiv, vol. abs/2303.08774, 2023. [Online]. Available: https://arxiv.org/abs/2303.08774 3 JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 13

    2023 arXiv

  49. [57]

    Multi-modal interaction graph convolutional network for temporal language localization in videos,

    Z. Zhang, X. Han, X. Song, Y . Yan, and L. Nie, “Multi-modal interaction graph convolutional network for temporal language localization in videos,” IEEE Transactions on Image Processing , vol. 30, pp. 8265– 8277, 2021. 3

    2021

  50. [58]

    Prompt-driven referring image segmentation with instance contrasting,

    C. Shang, Z. Song, H. Qiu, L. Wang, F. Meng, and H. Li, “Prompt-driven referring image segmentation with instance contrasting,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, 2024, pp. 4124–4134. 3

    2024

  51. [59]

    Cris: Clip- driven referring image segmentation,

    Z. Wang, Y . Lu, Q. Li, X. Tao, Y . Guo, M. Gong, and T. Liu, “Cris: Clip- driven referring image segmentation,” in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022, pp. 11 676– 11 685. 3

    2022

  52. [60]

    Egfnet: Edge-aware guidance fusion network for rgb–thermal urban scene parsing,

    S. Dong, W. Zhou, C. Xu, and W. Yan, “Egfnet: Edge-aware guidance fusion network for rgb–thermal urban scene parsing,” IEEE Transactions on Intelligent Transportation Systems, vol. 25, no. 1, pp. 657–669, 2024. 6, 7

    2024

  53. [61]

    Abmdrnet: Adaptive-weighted bi-directional modality difference reduction network for rgb-t semantic segmentation,

    Q. Zhang, S. Zhao, Y . Luo, D. Zhang, N. Huang, and J. Han, “Abmdrnet: Adaptive-weighted bi-directional modality difference reduction network for rgb-t semantic segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 20...

    2021

  54. [62]

    Feanet: Feature-enhanced attention network for rgb-thermal real-time semantic segmentation,

    F. Deng, H. Feng, M. Liang, H. Wang, Y . Yang, Y . Gao, J. Chen, J. Hu, X. Guo, and T. L. Lam, “Feanet: Feature-enhanced attention network for rgb-thermal real-time semantic segmentation,” in 2021 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE...

    2021

  55. [63]

    Dbcnet: Dynamic bilateral cross- fusion network for rgb-t urban scene understanding in intelligent vehi- cles,

    W. Zhou, T. Gong, J. Lei, and L. Yu, “Dbcnet: Dynamic bilateral cross- fusion network for rgb-t urban scene understanding in intelligent vehi- cles,” IEEE Transactions on Systems, Man, and Cybernetics: Systems , vol. 53, no. 12, pp. 7631–7641, 2023. 6

    2023

  56. [64]

    Ex- plicit attention-enhanced fusion for rgb-thermal perception tasks,

    M. Liang, J. Hu, C. Bao, H. Feng, F. Deng, and T. L. Lam, “Ex- plicit attention-enhanced fusion for rgb-thermal perception tasks,” IEEE Robotics and Automation Letters , vol. 8, no. 7, pp. 4060–4067, 2023. 6, 7

    2023

  57. [65]

    Gmnet: Graded- feature multilabel-learning network for rgb-thermal urban scene seman- tic segmentation,

    W. Zhou, J. Liu, J. Lei, L. Yu, and J.-N. Hwang, “Gmnet: Graded- feature multilabel-learning network for rgb-thermal urban scene seman- tic segmentation,” IEEE Transactions on Image Processing, vol. 30, pp. 7790–7802, 2021. 6

    2021

  58. [66]

    Mmsformer: Multi- modal transformer for material and semantic segmentation,

    M. K. Reza, A. Prater-Bennette, and M. S. Asif, “Mmsformer: Multi- modal transformer for material and semantic segmentation,” IEEE Open Journal of Signal Processing , pp. 1–12, 2024. 6, 7

    2024

  59. [67]

    Complementary random masking for rgb-thermal semantic segmentation,

    U. Shin and J. Lee, Kyunghyun and, “Complementary random masking for rgb-thermal semantic segmentation,” in 2024 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2024, pp. 11 110–11 117. 6

    2024

  60. [68]

    Complementary random masking for rgb-thermal semantic segmentation,

    U. Shin, K. Lee, I. S. Kweon, and J. Oh, “Complementary random masking for rgb-thermal semantic segmentation,” 2024, pp. 11 110– 11 117. [Online]. Available: https://doi.org/10.1109/ICRA57147.2024. 10611200 6, 7

    2024

  61. [69]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in ICLR, 2019. 7

    2019

  62. [70]

    Multi-interactive feature learning and a full-time multi-modality bench- mark for image fusion and segmentation,

    J. Liu, Z. Liu, G. Wu, L. Ma, R. Liu, W. Zhong, Z. Luo, and X. Fan, “Multi-interactive feature learning and a full-time multi-modality bench- mark for image fusion and segmentation,” in International Conference on Computer Vision , 2023. 7, 9

    2023

  63. [71]

    Cmx: Cross-modal fusion for rgb-x semantic segmentation with transformers,

    J. Zhang, H. Liu, K. Yang, X. Hu, R. Liu, and R. Stiefelhagen, “Cmx: Cross-modal fusion for rgb-x semantic segmentation with transformers,” IEEE Transactions on Intelligent Transportation Systems, vol. 24, no. 12, pp. 14 679–14 694, 2023. 7

    2023

  64. [72]

    Delivering arbitrary-modal semantic segmentation,

    J. Zhang, R. Liu, H. Shi, K. Yang, S. Reiß, K. Peng, H. Fu, K. Wang, and R. Stiefelhagen, “Delivering arbitrary-modal semantic segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2023, pp. 1136–1147. 7

    2023

  65. [73]

    Gmnet: graded-feature multilabel-learning network for rgb-thermal urban scene semantic seg- mentation,

    W. Zhou, J. Liu, J. Lei, L. Yu, and J.-N. Hwang, “Gmnet: graded-feature multilabel-learning network for rgb-thermal urban scene semantic seg- mentation,” IEEE TIP, vol. 30, pp. 7790–7802, 2021. 9

    2021

  66. [74]

    Rgb-t semantic segmentation with location, activation, and sharpening,

    G. Li, Y . Wang, Z. Liu, X. Zhang, and D. Zeng, “Rgb-t semantic segmentation with location, activation, and sharpening,” IEEE TCSVT ,

  67. [75]

    Edge-aware guidance fusion network for rgb thermal scene parsing,

    W. Zhou, S. Dong, C. Xu, and Y . Qian, “Edge-aware guidance fusion network for rgb thermal scene parsing,” AAAI, 2022. 9

    2022

  68. [76]

    Didfuse: Deep image decomposition for infrared and visible image fusion,

    Z. Zhao, S. Xu, C. Zhang, J. Liu, P. Li, and J. Zhang, “Didfuse: Deep image decomposition for infrared and visible image fusion,” IJCAI,

  69. [77]

    Reconet: Recurrent correction network for fast and efficient multi-modality image fusion,

    Z. Huang, J. Liu, X. Fan, R. Liu, W. Zhong, and Z. Luo, “Reconet: Recurrent correction network for fast and efficient multi-modality image fusion,” in ECCV. Springer, 2022, pp. 539–555. 9

    2022

  70. [78]

    U2fusion: A unified unsupervised image fusion network,

    H. Xu, J. Ma, J. Jiang, X. Guo, and H. Ling, “U2fusion: A unified unsupervised image fusion network,” IEEE TPAMI, 2020. 9

    2020

  71. [79]

    Target-aware dual adversarial learning and a multi-scenario multi- modality benchmark to fuse infrared and visible for object detection,

    J. Liu, X. Fan, Z. Huang, G. Wu, R. Liu, W. Zhong, and Z. Luo, “Target-aware dual adversarial learning and a multi-scenario multi- modality benchmark to fuse infrared and visible for object detection,” in IEEE/CVF CVPR, 2022, pp. 5802–5811. 9

    2022

  72. [80]

    U3m: Unbiased multiscale modal fusion model for multimodal semantic segmentation,

    B. Li, D. Zhang, Z. Zhao, J. Gao, and X. Li, “U3m: Unbiased multiscale modal fusion model for multimodal semantic segmentation,” arXiv preprint arXiv:2405.15365, 2024. 9

    2024 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.