Pith. sign in

REVIEW 3 major objections 6 minor 83 references

M-SpecGene: Generalized Foundation Model for RGBT Multispectral Vision

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read M-SpecGene claims that one self-supervised ViT, pretrained on 548,238 RGB-thermal pairs, generalizes across eleven datasets and four multispectral tasks without handcrafted fusion modules.

desk verdict First serious RGBT foundation model with strong ablations, but the undisclosed pretraining-set composition and an internal contradiction in the masking schedule block the generalization claim. read the letter →

arxiv 2507.16318 v2 pith:OHT6O6Z3 submitted 2025-07-22 cs.CV

Kailai Zhou , Fuqiang Yang , Shixian Wang , Bihan Wen , Chongde Zi , Linsen Chen , Qiu Shen , Xun Cao This is my paper
classification cs.CV
keywords RGB-thermalfusionmultispectralvisionfoundationmodelself-supervisedlearningmaskedautoencodermodality-invariantrepresentationprogressivemasking
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

M-SpecGene sets out to show that RGB-thermal vision can be unified under a single self-supervised foundation model instead of a zoo of task-specific fusion networks. It constructs RGBT550K, a curated 548,238-pair pretraining corpus distilled from 41 existing RGBT datasets, and trains a Siamese masked autoencoder whose masking schedule is driven by a new cross-modality information-density metric. After full fine-tuning, one ViT encoder with plain concatenation of RGB and thermal tokens matches or outperforms previous methods on eleven datasets spanning detection, segmentation, feature matching, and saliency. The payoff, if the claim holds, is that handcrafted fusion modules become unnecessary and RGB-only architectures can be lifted to RGBT tasks simply by swapping in the pretrained encoder.

What carries the argument

The load-bearing mechanism is the GMM-CMSS progressive masking strategy. CMSS scores each RGB-thermal patch pair as $m = (1+\langle a/|a|, b/|b|\rangle)/(2\sigma_a^2\sigma_b^2)$, where the numerator is cosine similarity of the normalized patch embeddings and the denominator is the product of their structural variances; low scores mark information-rich object regions and high scores mark flat background. A Gaussian mixture model fitted to the corpus-wide CMSS distribution controls a sampling function that starts by keeping only high-information patches visible and, over training, shifts the visible set toward low-information patches, giving an easy-to-hard, object-centric curriculum. The same shared-weight ViT encoder then processes complementary masked RGB and thermal inputs, and two modality-specific decoders reconstruct each modality, which forces the encoder to store representations that are consistent across modalities.

What would settle it

Take the dataset list for RGBT550K and check it, frame by frame, against KAIST, FLIR, LLVIP, SemanticRT, MVSeg, FMB, VT821, VT1000, VT5000, and VI-RGBT1500; if any evaluation images or their temporal neighbors appear, rerun pretraining on an overlap-free subset and compare mAP and AUC. As a second independent check, replace GMM-CMSS progressive masking with uniform random masking at the same 90% ratio and same compute; if the detection and matching margins shrink to noise, the masking curriculum is not carrying the result.

Watch

Extended reading notes

Core claim

The central claim is that modality-invariant RGBT representations can be learned once, from unlabeled paired data, and then reused across tasks that previous work treated separately. The key evidence is the GMM-CMSS progressive masking schedule: patch pairs where the two modalities agree strongly and both vary little are treated as low-information background, while patch pairs with low cosine similarity and high structural variance are treated as informative objects, and the pretraining curriculum gradually shifts the visible patches from object-dense to background regions. The Siamese MAE with a shared-weight ViT encoder and independent decoders learns to align RGB and thermal features in one latent space; after fine-tuning on each task, concatenating the two modalities' encoder outputs is enough to beat or match task-specific fusion architectures on most benchmarks.

Load-bearing premise

The load-bearing premise is that the 548,238-pair RGBT550K pretraining corpus does not include the eleven evaluation datasets or near-duplicate frames of them, because the paper never lists the component datasets and the reported gains would otherwise reflect in-distribution memory rather than generalization.

Editorial extensions

If this is right

  • A single pretrained ViT backbone, fine-tuned with simple concatenation, can replace the handcrafted attention and fusion modules in RGBT detection, segmentation, matching, and saliency models.
  • RGB-only downstream architectures can be reused for RGBT tasks without architectural modification, lowering the barrier to multispectral deployment.
  • Large unlabeled RGBT corpora become a pretraining resource, reducing dependence on costly task-specific annotations.
  • The reported ViT-B results improve over the previous RGBT foundation-model baseline on KAIST, FLIR, LLVIP, SemanticRT, MVSeg, and VI-RGBT1500, while ViT-S already matches several specialized methods.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A stronger test of generalization would be linear probing or zero-shot evaluation on the four tasks, since full fine-tuning can mask how much of the gain comes from pretraining versus task adaptation.
  • The CMSS/GMM masking schedule does not depend on RGB-thermal specifically; the same easy-to-hard curriculum could transfer to RGB-depth, RGB-event, or SAR-optical pairs where one modality is information-sparse.
  • If RGBT550K is later shown to contain evaluation-set frames or their temporal neighbors, the reported margins would reflect in-distribution pretraining rather than transfer; holding out all evaluation sources is the decisive check.
  • ViT-S lags on several saliency benchmarks while ViT-B does not, suggesting the foundation-model benefit scales with capacity and that smaller backbones may need task-specific fusion after all.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces M-SpecGene, a masked-autoencoder-style RGB-thermal foundation model. It collects 548,238 RGBT pairs (RGBT550K) from 41 existing datasets, defines a Cross-Modality Structural Sparsity (CMSS) score per patch pair, fits a Gaussian mixture over CMSS values, and uses a progressive masking schedule during Siamese self-supervised pretraining. The pretrained ViT encoder is then fine-tuned on four RGBT tasks; the authors report state-of-the-art or competitive results on KAIST, FLIR, LLVIP, SemanticRT, MVSeg, FMB, LLVIP matching, and four saliency datasets. The central claims are that M-SpecGene learns modality-invariant representations without handcrafted fusion and generalizes across eleven datasets.

Significance. If the findings hold, the paper provides a useful step toward unified RGBT pretraining: it demonstrates that a single ViT backbone with concatenation-based fusion can match or exceed task-specific fusion modules on several benchmarks, and the masking strategy is a novel way to exploit cross-modal information imbalance. The paper's strengths are its broad experimental coverage across four tasks, the inclusion of ablations over architecture, masking, decoder depth, and masking ratio, and an explicit promise of code release. However, the validity of the headline generalization claim depends on two unresolved issues: the composition and overlap of RGBT550K with the evaluation datasets, and the consistency of the masking-direction description with the selection rule. Both are addressable in revision.

major comments (3)
  1. [§3 and §5.2–§5.5] The paper states that RGBT550K was refined from RGBT3M, which consists of three million RGBT samples drawn from 41 datasets and 10 multispectral tasks (§3), and that further details can be found in the appendix, but the v2 text does not enumerate the 41 datasets or provide the appendix. Because the downstream evaluation uses KAIST, FLIR, LLVIP, SemanticRT, MVSeg, FMB, VT821, VT1000, VT5000, and VI-RGBT1500, the authors must report which of these contributed pretraining frames and quantify any overlap (e.g., via temporal or sequence exclusion). Without this audit, the central generalization claim in §1 and §6 cannot be distinguished from in-distribution pretraining.
  2. [§4.3 (with §4.1)] The masking direction as written is inverted. §4.1 states that high CMSS corresponds to low-information regions such as sky and low CMSS to high-information regions such as pedestrians. In §4.3, at the start of pretraining S(x) has K=1, μ̂1=0, and Σ̂1=0.01, so the generated sampling points lie near x=0; because "we sample ... masked patches from miter that are nearest to the generated sampling points s," the model masks low-CMSS (high-information) patches, leaving low-information patches unmasked. This contradicts the claim that "unmasked patches are concentrated in high-information-density regions" and the earlier statement that unmasked patches move from foreground to background. Please clarify whether the sampling function selects masked or unmasked patches and correct the directional descriptions; as written the progressive easy-to-hard mechanism is internally inconsistent.
  3. [Eq. (1), §4.1] The CMSS formula is under-specified: σ_a and σ_b are introduced without a definition (structural variance of what—per-patch embedding dimensions or per-pixel intensities?), and the normalization that maps m to [0,1] is not shown. Since the GMM and the entire masking strategy operate on this normalized value, the metric must be defined precisely and reproducibly.
minor comments (6)
  1. [Eq. (3)] The posterior α_{ik} in Eq. (3) sums over i in the denominator ("Σᵢ₌₁ᴷ"); it should sum over k. Please correct the summation index.
  2. [§5.3, Table 3] On FMB, M-SpecGene (ViT-B) has 60.1 mIoU and ViT-S has 56.5 mIoU, both below MSRS's 61.2; the text "superior to other competitive methods but falls short of MSRS on certain metrics" should say it falls short of MSRS on mIoU and should acknowledge that the ViT-S variant is below several baselines.
  3. [§3] The SSIM threshold of 0.80 is ambiguous: SSIM is a between-image similarity measure, and the text does not state which two images are compared (RGB versus thermal, or original versus another reference) or how this threshold selects for "sufficient object information." Please specify.
  4. [§5.1] The paper says M-SpecGene is first pretrained on ImageNet and "single-modality thermal datasets," but it does not list those thermal datasets or their sizes; please include this information for reproducibility.
  5. [§5.2] The claim of "seven of the nine evaluation metrics" on KAIST is consistent with Table 1(a) only if ties are counted as wins; please state this explicitly to avoid ambiguity.
  6. [§3] The appendix referenced in §3 is not present in the v2 text; please either add the appendix or remove the reference.

Circularity Check

0 steps flagged · score 0.0 of 10

No material circularity: the GMM-CMSS masking is a self-referential pretraining heuristic, but the paper's downstream claims are externally validated on independent benchmarks.

full rationale

I find no circular step in the claimed derivation chain. The paper's central claim is that a Siamese masked-autoencoder pretrained with GMM-CMSS progressive masking generalizes to four RGBT tasks; this is evaluated against external benchmarks (KAIST, FLIR, LLVIP, SemanticRT, MVSeg, FMB, LLVIP matching, VT821, VT1000, VT5000, VI-RGBT1500) and published baselines, so the headline numbers are not produced by the pretraining loss or the CMSS/GMM quantities by construction. The CMSS metric is computed from the encoder's own patch embeddings via Eq. (1) and used to select masks, which is a self-referential training heuristic rather than a prediction whose output is equivalent to its input; the downstream results do not reduce to it. The only self-citation is [80], used as background for modality imbalance and as a baseline (MBNet); it is not load-bearing. The unresolved issue of whether RGBT550K overlaps the downstream evaluation sets, and the missing appendix details, are potential data-contamination and external-validity risks rather than circularity; they belong in a correctness review, not a circularity score.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The method's novelty rests on the CMSS metric and the pretraining dataset. CMSS is an invented heuristic defined in terms of the encoder's own embeddings, with no external validation. The pretraining dataset is assembled from existing RGBT datasets without a published list, so the evaluation may not be held out. These are the main unproven inputs the central claim depends on.

free parameters (4)
  • GMM component count K = 3
    Set to 3 by default in Sec. 4.2; no model selection or ablation is reported for this choice.
  • Masking ratio r = 90%
    Tuned in Sec. 5.6 (Table 7e); 90% gives the best mAP50 among 85%, 90%, and 95%.
  • SSIM quality threshold = 0.80
    Used to filter RGBT3M into RGBT550K in Sec. 3; chosen by the authors without a stated criterion.
  • Initial sampling variance Sigma_hat = 0.01
    Hand-initialized in the sampling function S(x) in Sec. 4.3.
assumptions (4)
  • ad hoc to paper The CMSS metric, defined via cosine similarity and structural variances, is a valid proxy for information density.
    Introduced in Sec. 4.1 without external validation; the entire masking strategy depends on this mapping.
  • domain assumption A Gaussian mixture model with 3 components adequately fits the CMSS distribution.
    Sec. 4.2 assumes K=3 is sufficient; no goodness-of-fit evidence is presented.
  • domain assumption Self-supervised reconstruction of masked RGBT patches on RGBT550K yields transferable representations.
    Inherited from MAE-style pretraining (Sec. 4); not independently proven for the RGBT domain.
  • domain assumption The downstream test datasets are not substantially included in RGBT550K pretraining.
    The paper does not list the 41 source datasets, so it is unclear whether KAIST, FLIR, LLVIP, SemanticRT, MVSeg, FMB, and VT/VI saliency sets were seen during pretraining; the generalization claim rests on this.
invented entities (1)
  • Cross-Modality Structural Sparsity (CMSS) metric
    purpose: Quantify information density of RGBT patch pairs to guide the masking strategy
    No external validation is provided; the metric is defined and used only within the paper's own pipeline.

how reviews work

0 comments
Cite this review

Pith. "Pith review of M-SpecGene: Generalized Foundation Model for RGBT Multispectral Vision." pith.science (2026). https://pith.science/paper/OHT6O6Z3

@misc{pith2026250716318,
  author       = {Pith},
  title        = {Pith review of: M-SpecGene: Generalized Foundation Model for RGBT Multispectral Vision},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OHT6O6Z3}},
  note         = {Machine review of arXiv:2507.16318}
}
read the original abstract

RGB-Thermal (RGBT) multispectral vision is essential for robust perception in complex environments. Most RGBT tasks follow a case-by-case research paradigm, relying on manually customized models to learn task-oriented representations. Nevertheless, this paradigm is inherently constrained by artificial inductive bias, modality bias, and data bottleneck. To address these limitations, we make the initial attempt to build a Generalized RGBT MultiSpectral foundation model (M-SpecGene), which aims to learn modality-invariant representations from large-scale broad data in a self-supervised manner. M-SpecGene provides new insights into multispectral fusion and integrates prior case-by-case studies into a unified paradigm. Considering the unique characteristic of information imbalance in RGBT data, we introduce the Cross-Modality Structural Sparsity (CMSS) metric to quantify the information density across two modalities. Then we develop the GMM-CMSS progressive masking strategy to facilitate a flexible, easy-to-hard, and object-centric pre-training process. Comprehensive experiments validate M-SpecGene's generalizability across eleven datasets for four RGBT downstream tasks. The code will be available at https://github.com/CalayZhou/M-SpecGene.

Figures

Figures reproduced from arXiv: 2507.16318 by the authors.

Figure 1
Figure 1. (a) Manually customized models: task-oriented repre [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. RGBT550K consists of diverse resources, it exhibits an [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. (a) The self-supervised pre-training of M-SpecGene. (b) [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: As the sampling function S(x) shifts from µˆ = 0 to µˆ = 1 (green box), unmasked patches transition from high- to low-information-density areas (blue box). Following these steps, we iteratively update the Gaus￾sian mixture model parameters {µk, Σk, πk} at each pre￾trai…
Figure 5
Figure 5. Figure 5: (a) Samples for feature visualization. (b-c) The t-SNE visualization of concatenated RGBT features for object and background [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

83 extracted references · 70 canonical work pages

  1. [1]

    Effectiveness guided cross-modal information sharing for aligned rgb-t object de- tection

    Zijia An, Chunlei Liu, and Yuqi Han. Effectiveness guided cross-modal information sharing for aligned rgb-t object de- tection. IEEE Signal Processing Letters , 29:2562–2566,

  2. [2]

    Foundational models defining a new era in vision: A survey and outlook

    Muhammad Awais, Muzammal Naseer, Salman Khan, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, and Fahad Shahbaz Khan. Foundational models defining a new era in vision: A survey and outlook. arXiv preprint arXiv:2307.13721, 2023. 2

    arXiv 2023

  3. [3]

    Multimodal object detection by channel switching and spatial attention

    Yue Cao, Junchi Bin, Jozsef Hamari, Erik Blasch, and Zheng Liu. Multimodal object detection by channel switching and spatial attention. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition , pages 403–411, 2023. 6

    2023

  4. [4]

    Encoder-decoder with atrous separable convolution for semantic image segmentation

    Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 801–818, 2018. 6

    2018

  5. [5]

    Multimodal object detection via probabilistic ensembling

    Yi-Ting Chen, Jinghao Shi, Zelin Ye, Christoph Mertz, Deva Ramanan, and Shu Kong. Multimodal object detection via probabilistic ensembling. In European Conference on Com- puter Vision, pages 139–158. Springer, 2022. 6

    work page 2022

  6. [6]

    Feanet: Feature-enhanced atten- tion network for rgb-thermal real-time semantic segmenta- tion

    Fuqin Deng, Hua Feng, Mingjian Liang, Hongmin Wang, Yong Yang, Yuan Gao, Junfeng Chen, Junjie Hu, Xiyue Guo, and Tin Lun Lam. Feanet: Feature-enhanced atten- tion network for rgb-thermal real-time semantic segmenta- tion. In 2021 IEEE/RSJ international conference on intel- ligent robots and systems (IROS) , pages 4467–4473. IEEE,

    work page 2021

  7. [7]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 2, 3, 4, 5

    work page 2009

  8. [8]

    Redfeat: Recoupling detection and description for multimodal feature learning.IEEE Trans- actions on Image Processing, 32:591–602, 2022

    Yuxin Deng and Jiayi Ma. Redfeat: Recoupling detection and description for multimodal feature learning.IEEE Trans- actions on Image Processing, 32:591–602, 2022. 2, 7

    work page 2022

Show all 83 references
  1. [9]

    Efficient multimodal seman- tic segmentation via dual-prompt learning

    Shaohua Dong, Yunhe Feng, Qing Yang, Yan Huang, Dong- fang Liu, and Heng Fan. Efficient multimodal seman- tic segmentation via dual-prompt learning. arXiv preprint arXiv:2312.00360, 2023. 6

    2023 arXiv

  2. [10]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 3

    2010 arXiv

  3. [11]

    Siamese network for rgb-d salient object detection and beyond

    Keren Fu, Deng-Ping Fan, Ge-Peng Ji, Qijun Zhao, Jianbing Shen, and Ce Zhu. Siamese network for rgb-d salient object detection and beyond. IEEE transactions on pattern analysis and machine intelligence, 44(9):5541–5559, 2021. 7

    2021

  4. [12]

    Fusion of multispectral data through illumination-aware deep neural networks for pedestrian de- tection

    Dayan Guan, Yanpeng Cao, Jiangxin Yang, Yanlong Cao, and Michael Ying Yang. Fusion of multispectral data through illumination-aware deep neural networks for pedestrian de- tection. Information Fusion, 50:148–157, 2019. 6

    2019

  5. [13]

    Mfnet: Towards real-time se- mantic segmentation for autonomous vehicles with multi- spectral scenes

    Qishen Ha, Kohei Watanabe, Takumi Karasawa, Yoshitaka Ushiku, and Tatsuya Harada. Mfnet: Towards real-time se- mantic segmentation for autonomous vehicles with multi- spectral scenes. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 510...

    2017

  6. [14]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16000– 16009, 2022. 3, 4, 6, 8

    2022

  7. [15]

    Multispec- tral object detection via cross-modal conflict-aware learning

    Xiao He, Chang Tang, Xin Zou, and Wei Zhang. Multispec- tral object detection via cross-modal conflict-aware learning. In Proceedings of the 31st ACM International Conference on Multimedia, page 1465–1474, New York, NY , USA, 2023. Association for Computing Machinery. 6

    2023

  8. [16]

    Spectralgpt: Spectral remote sensing foun- dation model

    Danfeng Hong, Bing Zhang, Xuyang Li, Yuxuan Li, Chenyu Li, Jing Yao, Naoto Yokoya, Hao Li, Pedram Ghamisi, Xi- uping Jia, et al. Spectralgpt: Spectral remote sensing foun- dation model. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. 3, 6

    2024

  9. [17]

    Milan: Masked image pretraining on language assisted representation

    Zejiang Hou, Fei Sun, Yen-Kuang Chen, Yuan Xie, and Sun- Yuan Kung. Milan: Masked image pretraining on language assisted representation. arXiv preprint arXiv:2208.06049 ,

    arXiv

  10. [18]

    Pos-gift: A geomet- ric and intensity-invariant feature transformation for multi- modal images

    Zhuolu Hou, Yuxuan Liu, and Li Zhang. Pos-gift: A geomet- ric and intensity-invariant feature transformation for multi- modal images. Information Fusion, 102:102027, 2024. 7

    2024

  11. [19]

    Multi-graph fusion and learning for rgbt im- age saliency detection

    Liming Huang, Kechen Song, Jie Wang, Menghui Niu, and Yunhui Yan. Multi-graph fusion and learning for rgbt im- age saliency detection. IEEE Transactions on Circuits and Systems for Video Technology, 32(3):1366–1377, 2021. 7

    2021

  12. [20]

    Multispectral pedestrian detection: Benchmark dataset and baseline

    Soonmin Hwang, Jaesik Park, Namil Kim, Yukyung Choi, and In So Kweon. Multispectral pedestrian detection: Benchmark dataset and baseline. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1037–1045, 2015. 6

    2015

  13. [21]

    Multispectral pedestrian detection: Benchmark dataset and baseline

    Soonmin Hwang, Jaesik Park, Namil Kim, Yukyung Choi, and In So Kweon. Multispectral pedestrian detection: Benchmark dataset and baseline. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 1037–1045, 2015. 6

    2015

  14. [22]

    Semanticrt: A large-scale dataset and method for robust semantic segmentation in multispectral images

    Wei Ji, Jingjing Li, Cheng Bian, Zhicheng Zhang, and Li Cheng. Semanticrt: A large-scale dataset and method for robust semantic segmentation in multispectral images. In Proceedings of the 31st ACM International Conference on Multimedia, pages 3307–3316, 2023. 2, 6

    2023

  15. [23]

    Multispectral video se- mantic segmentation: A benchmark dataset and baseline

    Wei Ji, Jingjing Li, Cheng Bian, Zongwei Zhou, Jiaying Zhao, Alan L Yuille, and Li Cheng. Multispectral video se- mantic segmentation: A benchmark dataset and baseline. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition , pages 1094–1104, 2023. 2, 6

    2023

  16. [24]

    Llvip: A visible-infrared paired dataset for low-light vision

    Xinyu Jia, Chuang Zhu, Minzhen Li, Wenqi Tang, and Wenli Zhou. Llvip: A visible-infrared paired dataset for low-light vision. In Proceedings of the IEEE/CVF international con- ference on computer vision, pages 3496–3504, 2021. 6, 7

    2021

  17. [25]

    What to hide from your students: Attention-guided masked image modeling

    Ioannis Kakogeorgiou, Spyros Gidaris, Bill Psomas, Yan- nis Avrithis, Andrei Bursuc, Konstantinos Karantzalos, and Nikos Komodakis. What to hide from your students: Attention-guided masked image modeling. In European Conference on Computer Vision , pages 300–318. Springer,

  18. [26]

    Crossformer: Cross-guided attention for multi-modal object detection.Pat- tern Recognition Letters, 179:144–150, 2024

    Seungik Lee, Jaehyeong Park, and Jinsun Park. Crossformer: Cross-guided attention for multi-modal object detection.Pat- tern Recognition Letters, 179:144–150, 2024. 2, 6

    2024

  19. [27]

    Multispectral Pedestrian Detection via Simulta- neous Detection and Segmentation

    Chengyang Li, Dan Song, Ruofeng Tong, and Min Tang. Multispectral Pedestrian Detection via Simulta- neous Detection and Segmentation. arXiv e-prints , art. arXiv:1808.04818, 2018. 6

    2018 arXiv

  20. [28]

    Lasher: A large-scale high- diversity benchmark for rgbt tracking.IEEE Transactions on Image Processing, 31:392–404, 2021

    Chenglong Li, Wanlin Xue, Yaqing Jia, Zhichen Qu, Bin Luo, Jin Tang, and Dengdi Sun. Lasher: A large-scale high- diversity benchmark for rgbt tracking.IEEE Transactions on Image Processing, 31:392–404, 2021. 3

    2021

  21. [29]

    Semmae: Semantic-guided mask- ing for learning masked autoencoders

    Gang Li, Heliang Zheng, Daqing Liu, Chaoyue Wang, Bing Su, and Changwen Zheng. Semmae: Semantic-guided mask- ing for learning masked autoencoders. Advances in Neural Information Processing Systems, 35:14290–14302, 2022. 3

    2022

  22. [30]

    Rift: Multi-modal image matching based on radiation-variation insensitive fea- ture transform

    Jiayuan Li, Qingwu Hu, and Mingyao Ai. Rift: Multi-modal image matching based on radiation-variation insensitive fea- ture transform. IEEE Transactions on Image Processing, 29: 3296–3310, 2019. 2, 7

    2019

  23. [31]

    Confidence-aware fusion using dempster- shafer theory for multispectral pedestrian detection

    Qing Li, Changqing Zhang, Qinghua Hu, Huazhu Fu, and Pengfei Zhu. Confidence-aware fusion using dempster- shafer theory for multispectral pedestrian detection. IEEE Transactions on Multimedia, 25:3420–3431, 2022. 6

    2022

  24. [32]

    Saratr-x: Toward building a foundation model for sar target recognition

    Weijie Li, Wei Yang, Yuenan Hou, Li Liu, Yongxiang Liu, and Xiang Li. Saratr-x: Toward building a foundation model for sar target recognition. IEEE Transactions on Image Pro- cessing, 34:869–884, 2025. 3, 6

    2025

  25. [33]

    Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detection

    Xiang Li, Wenhai Wang, Lijun Wu, Shuo Chen, Xiaolin Hu, Jun Li, Jinhui Tang, and Jian Yang. Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detection. Advances in Neural Information Processing Systems, 33:21002–21012, 2020. 6

    2020

  26. [34]

    Exploring plain vision transformer backbones for object de- tection

    Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object de- tection. In European conference on computer vision , pages 280–296. Springer, 2022. 5, 6

    2022

  27. [35]

    Lightglue: Local feature matching at light speed

    Philipp Lindenberger, Paul-Edouard Sarlin, and Marc Polle- feys. Lightglue: Local feature matching at light speed. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17627–17638, 2023. 7

    2023

  28. [36]

    Infmae: A foundation model in the infrared modality

    Fangcen Liu, Chenqiang Gao, Yaming Zhang, Junjie Guo, Jinghao Wang, and Deyu Meng. Infmae: A foundation model in the infrared modality. In European Conference on Computer Vision, pages 420–437. Springer, 2025. 3, 6

    2025

  29. [37]

    Jingjing Liu, Shaoting Zhang, Shu Wang, and Dimitris N. Metaxas. Multispectral Deep Neural Networks for Pedes- trian Detection. arXiv e-prints, art. arXiv:1611.02644, 2016. 6

    2016 arXiv

  30. [38]

    Multi- interactive feature learning and a full-time multi-modality benchmark for image fusion and segmentation

    Jinyuan Liu, Zhu Liu, Guanyao Wu, Long Ma, Risheng Liu, Wei Zhong, Zhongxuan Luo, and Xin Fan. Multi- interactive feature learning and a full-time multi-modality benchmark for image fusion and segmentation. In Proceed- ings of the IEEE/CVF international conference on computer ...

    2023

  31. [39]

    Learning selective self-mutual attention for rgb-d saliency detection

    Nian Liu, Ni Zhang, and Junwei Han. Learning selective self-mutual attention for rgb-d saliency detection. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13756–13765, 2020. 7

    2020

  32. [40]

    Local memory attention for fast video semantic segmentation

    Matthieu Paul, Martin Danelljan, Luc Van Gool, and Radu Timofte. Local memory attention for fast video semantic segmentation. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1102–1109. IEEE, 2021. 6

    2021

  33. [41]

    Cross- modality fusion transformer for multispectral object detec- tion

    Fang Qingyun, Han Dapeng, and Wang Zhaokui. Cross- modality fusion transformer for multispectral object detec- tion. arXiv preprint arXiv:2111.00273, 2021. 1, 2

    2021 arXiv

  34. [42]

    Icafusion: Iterative cross-attention guided feature fusion for multispectral object detection

    Jifeng Shen, Yifei Chen, Yue Liu, Xin Zuo, Heng Fan, and Wankou Yang. Icafusion: Iterative cross-attention guided feature fusion for multispectral object detection. Pattern Recognition, 145:109913, 2024. 2, 6

    2024

  35. [43]

    Pst900: Rgb- thermal calibration, dataset and segmentation network

    Shreyas S Shivakumar, Neil Rodrigues, Alex Zhou, Ian D Miller, Vijay Kumar, and Camillo J Taylor. Pst900: Rgb- thermal calibration, dataset and segmentation network. In 2020 IEEE international conference on robotics and au- tomation (ICRA), pages 9441–9447. IEEE, 2020. 6

    2020

  36. [44]

    Multiple graph affinity interactive network and a variable il- lumination dataset for rgbt image salient object detection

    Kechen Song, Liming Huang, Aojun Gong, and Yunhui Yan. Multiple graph affinity interactive network and a variable il- lumination dataset for rgbt image salient object detection. IEEE Transactions on Circuits and Systems for Video Tech- nology, 33(7):3104–3118, 2022. 1, 2, 7

    2022

  37. [45]

    Loftr: Detector-free local feature matching with transformers

    Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. Loftr: Detector-free local feature matching with transformers. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 8922–8931, 2021. 5, 7

    2021

  38. [46]

    Rtfnet: Rgb- thermal fusion network for semantic segmentation of urban scenes

    Yuxiang Sun, Weixun Zuo, and Ming Liu. Rtfnet: Rgb- thermal fusion network for semantic segmentation of urban scenes. IEEE Robotics and Automation Letters , 4(3):2576– 2583, 2019. 6

    2019

  39. [47]

    Rgb-t image saliency detection via collaborative graph learning

    Zhengzheng Tu, Tian Xia, Chenglong Li, Xiaoxiao Wang, Yan Ma, and Jin Tang. Rgb-t image saliency detection via collaborative graph learning. IEEE Transactions on Multi- media, 22(1):160–173, 2019. 7

    2019

  40. [48]

    Multi-interactive encoder-decoder network for rgbt salient object detection

    Zhengzheng Tu, Zhun Li, Chenglong Li, Yang Lang, and Jin Tang. Multi-interactive encoder-decoder network for rgbt salient object detection. arXiv e-prints, pages arXiv–2005,

    2005

  41. [49]

    Multi-interactive dual-decoder for rgb-thermal salient object detection

    Zhengzheng Tu, Zhun Li, Chenglong Li, Yang Lang, and Jin Tang. Multi-interactive dual-decoder for rgb-thermal salient object detection. IEEE Transactions on Image Processing , 30:5678–5691, 2021. 7

    2021

  42. [50]

    Rgbt salient object detection: A large- scale dataset and benchmark

    Zhengzheng Tu, Yan Ma, Zhun Li, Chenglong Li, Jieming Xu, and Yongtao Liu. Rgbt salient object detection: A large- scale dataset and benchmark. IEEE Transactions on Multi- media, 25:4163–4176, 2022. 7

    2022

  43. [51]

    Hypersigma: Hyperspectral intelligence comprehen- sion foundation model

    Di Wang, Meiqi Hu, Yao Jin, Yuchun Miao, Jiaqi Yang, Yichu Xu, Xiaolei Qin, Jiaqi Ma, Lingyu Sun, Chenxing Li, et al. Hypersigma: Hyperspectral intelligence comprehen- sion foundation model. arXiv preprint arXiv:2406.11519 ,

    arXiv

  44. [52]

    Rgb-t saliency detection benchmark: Dataset, baselines, analysis and a novel approach

    Guizhao Wang, Chenglong Li, Yunpeng Ma, Aihua Zheng, Jin Tang, and Bin Luo. Rgb-t saliency detection benchmark: Dataset, baselines, analysis and a novel approach. In Image and Graphics Technologies and Applications: 13th Confer- ence on Image and Graphics Technologies and Appl...

    2018

  45. [53]

    Hard patches mining for masked image modeling

    Haochen Wang, Kaiyou Song, Junsong Fan, Yuxi Wang, Jin Xie, and Zhaoxiang Zhang. Hard patches mining for masked image modeling. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition , pages 10375–10385, 2023. 3

    2023

  46. [54]

    Cgfnet: Cross-guided fusion network for rgb-t salient object detection

    Jie Wang, Kechen Song, Yanqi Bao, Liming Huang, and Yunhui Yan. Cgfnet: Cross-guided fusion network for rgb-t salient object detection. IEEE Transactions on Circuits and Systems for Video Technology, 32(5):2949–2961, 2022. 7

    2022

  47. [55]

    Sgfnet: semantic- guided fusion network for rgb-thermal semantic segmenta- tion

    Yike Wang, Gongyang Li, and Zhi Liu. Sgfnet: semantic- guided fusion network for rgb-thermal semantic segmenta- tion. IEEE Transactions on Circuits and Systems for Video Technology, 33(12):7737–7748, 2023. 7

    2023

  48. [56]

    Image quality assessment: from error visibility to structural similarity

    Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Si- moncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600–612, 2004. 3

    2004

  49. [57]

    Tirdet: Mono-modality thermal infrared object detection based on prior thermal-to-visible translation

    Zeyu Wang, Fabien Colonnier, Jinghong Zheng, Jyotib- dha Acharya, Wenyu Jiang, and Kejie Huang. Tirdet: Mono-modality thermal infrared object detection based on prior thermal-to-visible translation. In Proceedings of the 31st ACM International Conference on Multimedia , page 2...

    2023

  50. [58]

    Object-wise masked autoencoders for fast pre-training

    Jiantao Wu and Shentong Mo. Object-wise masked autoencoders for fast pre-training. arXiv preprint arXiv:2205.14338, 2022. 3

    2022 arXiv

  51. [59]

    Unified perceptual parsing for scene understand- ing

    Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understand- ing. In Proceedings of the European conference on computer vision (ECCV), pages 418–434, 2018. 5, 6, 7

    2018

  52. [60]

    Early convolutions help trans- formers see better

    Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Doll´ar, and Ross Girshick. Early convolutions help trans- formers see better. Advances in neural information process- ing systems, 34:30392–30400, 2021. 7

    2021

  53. [61]

    Segformer: Simple and efficient design for semantic segmentation with transform- ers

    Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transform- ers. Advances in neural information processing systems, 34: 12077–12090, 2021. 7

    2021

  54. [62]

    Semantics lead all: Towards unified image registration and fusion from a semantic perspective

    Housheng Xie, Yukuan Zhang, Junhui Qiu, Xiangshuai Zhai, Xuedong Liu, Yang Yang, Shan Zhao, Yongfang Luo, and Jianbo Zhong. Semantics lead all: Towards unified image registration and fusion from a semantic perspective. Infor- mation Fusion, 98:101835, 2023. 7

    2023

  55. [63]

    Eva- x: A foundation model for general chest x-ray analysis with self-supervised learning

    Jingfeng Yao, Xinggang Wang, Yuehao Song, Huangxuan Zhao, Jun Ma, Yajie Chen, Wenyu Liu, and Bo Wang. Eva- x: A foundation model for general chest x-ray analysis with self-supervised learning. arXiv preprint arXiv:2405.05237,

    arXiv

  56. [64]

    C 2 former: Calibrated and complementary transformer for rgb-infrared object de- tection

    Maoxun Yuan and Xingxing Wei. C 2 former: Calibrated and complementary transformer for rgb-infrared object de- tection. IEEE Transactions on Geoscience and Remote Sens- ing, 2024. 6

    2024

  57. [65]

    Transla- tion, scale and rotation: cross-modal alignment meets rgb- infrared vehicle detection

    Maoxun Yuan, Yinyan Wang, and Xingxing Wei. Transla- tion, scale and rotation: cross-modal alignment meets rgb- infrared vehicle detection. In European Conference on Com- puter Vision, pages 509–525. Springer, 2022. 6

    2022

  58. [66]

    Unirgb-ir: A unified framework for visible-infrared downstream tasks via adapter tuning

    Maoxun Yuan, Bo Cui, Tianyi Zhao, and Xingxing Wei. Unirgb-ir: A unified framework for visible-infrared downstream tasks via adapter tuning. arXiv preprint arXiv:2404.17360, 2024. 3, 6, 7

    2024

  59. [67]

    Improving rgb-infrared object detection with cascade alignment-guided transformer

    Maoxun Yuan, Xiaorong Shi, Nan Wang, Yinyan Wang, and Xingxing Wei. Improving rgb-infrared object detection with cascade alignment-guided transformer. Information Fusion, 105:102246, 2024. 6

    2024

  60. [68]

    Object- contextual representations for semantic segmentation

    Yuhui Yuan, Xilin Chen, and Jingdong Wang. Object- contextual representations for semantic segmentation. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VI 16, pages 173–190. Springer, 2020. 6

    2020

  61. [69]

    Infusion-net: inter-and intra-weighted cross-fusion network for multispectral object detection

    Jun-Seok Yun, Seon-Hoo Park, and Seok Bong Yoo. Infusion-net: inter-and intra-weighted cross-fusion network for multispectral object detection. Mathematics, 10(21): 3966, 2022. 1, 2

    2022

  62. [70]

    Mmi- det: Exploring multi-modal integration for visible and in- frared object detection

    Yuqiao Zeng, Tengfei Liang, Yi Jin†, and Yidong Li. Mmi- det: Exploring multi-modal integration for visible and in- frared object detection. IEEE Transactions on Circuits and Systems for Video Technology, pages 1–1, 2024. 6

    2024

  63. [71]

    Multispectral fusion for object detection with cyclic fuse-and-refine blocks

    Heng Zhang, Elisa Fromont, S ´ebastien Lefevre, and Bruno Avignon. Multispectral fusion for object detection with cyclic fuse-and-refine blocks. In 2020 IEEE International conference on image processing (ICIP) , pages 276–280. IEEE, 2020. 3, 6

    2020

  64. [72]

    Guided attentive feature fusion for multispectral pedestrian detection

    Heng Zhang, Elisa Fromont, S ´ebastien Lef `evre, and Bruno Avignon. Guided attentive feature fusion for multispectral pedestrian detection. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 72–80,

  65. [73]

    Mrfs: Mutually reinforcing image fusion and segmenta- tion

    Hao Zhang, Xuhui Zuo, Jie Jiang, Chunchao Guo, and Jiayi Ma. Mrfs: Mutually reinforcing image fusion and segmenta- tion. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 26974–26983,

  66. [74]

    Cross-modality interactive attention network for multispectral pedestrian de- tection

    Lu Zhang, Zhiyong Liu, Shifeng Zhang, Xu Yang, Hong Qiao, Kaizhu Huang, and Amir Hussain. Cross-modality interactive attention network for multispectral pedestrian de- tection. Information Fusion, 50:20–29, 2019. 6

    2019

  67. [75]

    Weakly Aligned Cross-Modal Learning for Multispectral Pedestrian Detection

    Lu Zhang, Xiangyu Zhu, Xiangyu Chen, Xu Yang, Zhen Lei, and Zhiyong Liu. Weakly Aligned Cross-Modal Learning for Multispectral Pedestrian Detection. arXiv e-prints, art. arXiv:1901.02645, 2019. 6

    1901 arXiv

  68. [76]

    Rgb-t salient object detec- tion via fusing multi-level cnn features

    Qiang Zhang, Nianchang Huang, Lin Yao, Dingwen Zhang, Caifeng Shan, and Jungong Han. Rgb-t salient object detec- tion via fusing multi-level cnn features. IEEE Transactions on Image Processing, 29:3321–3335, 2019. 7

    2019

  69. [77]

    Abmdrnet: Adaptive-weighted bi-directional modality difference reduc- tion network for rgb-t semantic segmentation

    Qiang Zhang, Shenlu Zhao, Yongjiang Luo, Dingwen Zhang, Nianchang Huang, and Jungong Han. Abmdrnet: Adaptive-weighted bi-directional modality difference reduc- tion network for rgb-t semantic segmentation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Patt...

    2021

  70. [78]

    Mitigating modality discrepancies for rgb-t semantic segmentation

    Shenlu Zhao, Yichen Liu, Qiang Jiao, Qiang Zhang, and Jungong Han. Mitigating modality discrepancies for rgb-t semantic segmentation. IEEE Transactions on Neural Net- works and Learning Systems, 2023. 7

    2023

  71. [79]

    Removal and selection: Improving rgb- infrared object detection via coarse-to-fine fusion

    Tianyi Zhao, Maoxun Yuan, Feng Jiang, Nan Wang, and Xingxing Wei. Removal and selection: Improving rgb- infrared object detection via coarse-to-fine fusion. arXiv preprint arXiv:2401.10731, 2024. 6

    2024

  72. [80]

    Improving mul- tispectral pedestrian detection by addressing modality im- balance problems

    Kailai Zhou, Linsen Chen, and Xun Cao. Improving mul- tispectral pedestrian detection by addressing modality im- balance problems. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVIII 16, pages 787–803. Springer, 2020...

    2020

  73. [81]

    Mffenet: Multiscale feature fusion and en- hancement network for rgb–thermal urban road scene pars- ing

    Wujie Zhou, Xinyang Lin, Jingsheng Lei, Lu Yu, and Jenq- Neng Hwang. Mffenet: Multiscale feature fusion and en- hancement network for rgb–thermal urban road scene pars- ing. IEEE Transactions on Multimedia , 24:2526–2538,

  74. [82]

    Edge-aware guidance fusion network for rgb–thermal scene parsing

    Wujie Zhou, Shaohua Dong, Caie Xu, and Yaguan Qian. Edge-aware guidance fusion network for rgb–thermal scene parsing. In Proceedings of the AAAI conference on artificial intelligence, pages 3571–3579, 2022. 2, 6

    2022

  75. [83]

    Lsnet: Lightweight spatial boosting network for de- tecting salient objects in rgb-thermal images

    Wujie Zhou, Yun Zhu, Jingsheng Lei, Rongwang Yang, and Lu Yu. Lsnet: Lightweight spatial boosting network for de- tecting salient objects in rgb-thermal images. IEEE Transac- tions on Image Processing, 32:1329–1340, 2023. 7

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.