Pith. sign in

REVIEW 4 major objections 6 minor 41 references

SMFusion: Semantic-Preserving Fusion of Multimodal Medical Images for Enhanced Clinical Diagnosis

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SMFusion claims that fusing medical images under diagnostic-text guidance preserves clinically critical information better than seven state-of-the-art methods.

desk verdict A plausible text-guided medical image fusion method let down by a circular clinical evaluation and impossible MS-SSIM values. read the letter →

arxiv 2505.12251 v1 pith:QEXJJV4O submitted 2025-05-18 cs.CV

Haozhe Xiang , Han Zhang , Yu Cheng , Xiongwen Quan , Wanwan Huang This is my paper · ORCID
classification cs.CV
keywords multimodalmedicalimagefusionsemanticguidancetext-guidedpriorknowledgediagnosticreportgenerationlosscross-attentionalignmentMRI-CT-PET-SPECT
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Medical image fusion usually optimizes pixel-level statistics, so the fused image can look good while losing what a clinician needs. This paper argues that fusion should be guided by clinical meaning, and it proposes SMFusion, a network that pairs each source image with a diagnostic text description and injects that semantic signal into the fusion process. The paper claims this yields fused images that score best among eight methods on most standard quality metrics and, more importantly, keep lesion-defining information visible. It also introduces a new way to check clinical value: generate diagnostic reports from fused images and measure how informative those reports are.

What carries the argument

The load-bearing component is the Semantic Feature Modulation block: text embeddings act as queries in cross-attention over the image feature map, producing per-position and per-channel scale parameters R and shift parameters S that realign visual features toward the clinical description. A text-injection module normalizes the aligned features and rescales them with learned parameters λ and μ through an MLP, and the medical semantic loss enforces cosine similarity above a threshold of 0.85 between the fused image encoding and the text embeddings. These pieces convert free-form diagnostic text into controllable fusion guidance.

What would settle it

Take an SMFusion output, erase the lesion region, and recompute the medical semantic loss; if the score remains above the 0.85 threshold, the loss is not actually tracking clinically critical information, and the semantic-preservation claim collapses.

Watch

Extended reading notes

Core claim

The central claim is that a fusion network steered by medical prior knowledge encoded as text can beat purely visual fusion methods on MRI-CT, MRI-PET, and MRI-SPECT pairs. The text descriptions are produced by a medical vision-language model, encoded by a frozen text encoder, and used as queries in a cross-attention alignment that computes scale and shift parameters over the image feature map. A text-injection module then applies learned affine modulation to the aligned features, and a medical semantic loss keeps the fused image close to the text embedding in cosine similarity. The authors report that SMFusion reaches the highest spatial frequency, average gradient, and MS-SSIM values, and that diagnostic reports generated from its fused images contain more specific findings than reports from comparison methods.

Load-bearing premise

The load-bearing premise is that cosine similarity between a frozen natural-image text encoder's embedding and the fused image's embedding correctly measures whether clinically critical medical information has been retained.

Editorial extensions

If this is right

  • Fused images retain lesion details such as posterior fossa defects and adenomas that comparison methods miss in generated reports.
  • Diagnostic report generation becomes a usable evaluation channel for fusion quality, not just pixel metrics.
  • Modality-specific text prompts let one fusion architecture handle CT, PET, and SPECT pairings without redesign.
  • A public multimodal medical image-text dataset makes semantic fusion reproducible for other teams.
  • Removing the medical semantic loss reduces fusion quality, showing that text guidance carries part of the performance gain.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the text descriptions are generated from the very images being fused, the semantic guidance may partly amplify what the caption model already sees; a stronger test would use independent radiologist-written annotations.
  • The 0.85 cosine threshold makes the semantic loss active mainly early in training; annealing the threshold or recalibrating the text encoder could change how strongly semantics shape the final image.
  • Information entropy and keyword counts are weak proxies for clinical utility; a direct follow-up would have radiologists rank fused images side by side for diagnostic confidence.
  • If semantic fusion generalizes, the same text-injection machinery could apply to other alignment tasks such as registration or super-resolution where clinical context matters.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes SMFusion, a semantic-guided multimodal medical image fusion method. It generates text descriptions of source images with BiomedGPT, encodes them with a frozen CLIP text encoder, and uses a semantic interaction alignment module plus a text injection module to guide an encoder-decoder fusion network based on Restormer blocks. A medical semantic loss (Eq. 9) aligns the fused image's CLIP embedding with the text embedding. The authors claim superior quantitative performance over seven baseline methods on five fusion metrics and report that diagnostic reports generated by BiomedGPT from the fused images demonstrate preservation of clinically critical information, supported by text statistics and a small mean-opinion-score survey.

Significance. If the claims were fully supported, the paper would introduce a useful new direction: injecting medical semantic knowledge into image fusion and evaluating fusion outputs through downstream text-based diagnostics. The framework is clearly described, the choice of domain-specific text generation (BiomedGPT) is motivated, and the idea of using diagnostic reports as an evaluation proxy is creative. However, two load-bearing issues undermine the current claims: the reported quantitative metrics contain values that are impossible for the stated measures, and the clinical-utility evaluation is circular because the same model family that supplies the guidance text also generates the diagnostic reports used as evidence. The paper's core idea is promising, but the evidence presented does not yet substantiate the central claims of superior performance and preserved clinical information.

major comments (4)
  1. [Table I, Section IV-B2] All reported MS-SSIM values in Table I exceed 1 (e.g., 1.298 for DDFM and Ours), which is impossible for the standard MS-SSIM index defined in Eq. (17), whose value is bounded above by 1. Additionally, the standard deviation reported for MetaFusion's Q_ab/f is 3.302, yet Q_ab/f is defined in Eq. (13) as a weighted average of quantities in [0,1] and cannot have a standard deviation greater than 1. These inconsistencies indicate an error in the metric implementation or reporting, and because the quantitative superiority claim in Section IV-B2 rests on these numbers, the comparison needs to be recomputed with correct implementations before the result can be accepted.
  2. [Section IV-C and Eq. (9)] The evaluation of clinical utility is circular. The text descriptions that guide the fusion are generated by BiomedGPT from the source images (Section III-B1), and the diagnostic reports that serve as the main evidence of clinical value are also generated by BiomedGPT from the fused images (Section IV-C). Eq. (9) additionally maximizes cosine similarity between the fused image's CLIP embedding and the CLIP embedding of those same generated descriptions. Consequently, the evaluation can only demonstrate that the fused image is semantically aligned with the prompt; it does not establish that clinically critical information is preserved. Independent evidence is required, such as radiologist readings of the fused images against source images, or a downstream diagnostic task with ground-truth labels.
  3. [Eq. (9), Section III-B1] The medical semantic loss relies on a CLIP text encoder that is frozen and described as parameter-freezing (Section III-B1). CLIP is trained on natural images and text, not on radiology or medical imaging, so it is not established that high cosine similarity in this embedding space corresponds to clinical relevance. The threshold theta = 0.85 in Eq. (9) is stated without any sensitivity analysis, and the paper does not validate the CLIP space for medical content. The authors should justify this choice or provide experiments showing that the CLIP-based semantic loss correlates with clinically meaningful preservation.
  4. [Section IV-C, Tables III and IV] The claim that SMFusion generates more detailed diagnostic reports is contradicted by Table III, where Ours has the lowest text length (33.8889) among all methods, while Table IV presents reports from Ours with length 44 versus SDNet's 48 as evidence of 'more detailed and specific descriptions.' The evaluation also relies on proxies such as information entropy and keyword counts, which do not measure diagnostic correctness; no comparison against a ground-truth diagnosis or an independent clinician review is provided. The mean-opinion-score survey in Table VII involves only 13 medical students and a three-level rating scale, which is not a validated clinical assessment.
minor comments (6)
  1. [Section IV-A1] The text states that hyperparameter alpha was set to 0.1, while beta and gamma were determined to be 1.5 respectively; the phrasing is ambiguous as to whether both beta and gamma equal 1.5, and the later ablation tunes gamma alone, which should be clarified.
  2. [Abstract and Section IV-A2] The paper claims to 'construct a publicly available multimodal medical image-text dataset,' but the contribution section says the dataset 'will be publicly released on Github soon.' The dataset is therefore not yet publicly available, and the wording should be corrected.
  3. [Section III-D, Figure 9, Table II] The notation L1 and L2 is used inconsistently: in Figure 9, L1 is the medical semantic loss and L2 is the gradient loss, while in Eq. (11) L1 denotes the reconstruction loss for the first modality. Please rename to avoid confusion.
  4. [References] Reference [26] is a duplicate of reference [25] (both are the same 'Text-IF' paper), and in Section III-B1 it is cited for a claim about Zhao et al. using ChatGPT; the citation appears incorrect.
  5. [Table V] The caption of Table V says 'six evaluation metrics,' but the table lists only five metrics (SF, AG, MS-SSIM, SD, Q_ab/f).
  6. [Section III-D1] The sentence 'Once the text similarity loss exceeds the threshold theta, the gradient will no longer be updated' is imprecise: Eq. (9) zeros the loss when the cosine similarity is at least theta, which is a threshold on similarity, not on the loss value.

Circularity Check

2 steps flagged · score 8.0 of 10

The clinical-information claim is validated only by a BiomedGPT-to-CLIP-to-BiomedGPT loop; the semantic loss and the report evaluation are generated by the same model, making the central clinical claim circular.

  1. fitted input called prediction [Section III-B1 (text generation) and Section III-D, Eq. (9)]
    "For input image pairs I1,I2, BiomedGPT generates precisely aligned pathological descriptions {T1,T2}. These textual outputs are then fed into a parameter-freezing CLIP text encoder to obtain semantic feature vectors ... Lsemantic = 0, if cos(F_i_v(I_f), φT) ≥ θ; 1−cos(F_i_v(I_f), φT), otherwise."

    The semantic supervision φT is not independent medical knowledge: it is the CLIP encoding of BiomedGPT's description of the very source images being fused. Eq. (9) then drives the fused image embedding toward that exact text embedding, so any subsequent claim that the fused image 'preserves' the clinical content of those descriptions is enforced by the training objective rather than discovered or independently verified. The prediction of clinical information preservation is thus fitted to the same generated texts that define it.

  2. self definitional [Section IV-C, 'Performance in Generating Diagnostic Reports']
    "we randomly selected 27 brain images from the test set and leveraged the medical reasoning capabilities of BiomedGPT, which has demonstrated human-level performance in generating complex radiology reports, to generate diagnostic reports based on different fusion methods. ... We evaluated eight fusion approaches quantitatively across three dimensions: average text length, information entropy, and keyword count."

    The same BiomedGPT that authored the descriptions used as fusion targets in Eq. (9) is also the generator of the diagnostic reports used as evidence of clinical informativeness. The fused image was explicitly optimized to align with BiomedGPT/CLIP embeddings, so the reports are not an independent reading of the fused image; they sample the same semantic space imposed by the loss. Text length, entropy, keyword counts, and a 13-student MOS do not compare against ground-truth diagnoses or independent radiologist readings, so the claimed 'critical medical information' is defined and measured by the same model family that supplied the supervision.

full rationale

The standard fusion-quality comparisons (SF, AG, SD, Qab/f, and the qualitative panels) are independent evidence and are not circular: they compare SMFusion against seven external baselines on the same images. If the claim were limited to those metrics, the circularity score would be low. However, the abstract's central claim—'preserving more critical medical information'—is supported by two mechanisms that share a single generator. First, the medical semantic loss (Eq. 9) maximizes cosine similarity between the fused image embedding and the CLIP text embedding of BiomedGPT's descriptions of the source images, so the fusion is explicitly optimized to reproduce the semantic content of those generated descriptions. Second, the diagnostic-report evaluation in Section IV-C invokes the same BiomedGPT model to produce the reports from fused images. The reports are scored by text length, entropy, keyword count, and a small MOS panel, with no ground-truth diagnosis or independent radiologist reading. The loop is therefore: BiomedGPT writes the target text from the source images; Eq. 9 trains the fused image to match that text; BiomedGPT then reads the fused image and produces reports whose content is taken as evidence of clinical preservation. That is a self-referential validation of the central clinical claim. Separately, Table I reports MS-SSIM values above the theoretical maximum of 1 (e.g., 1.298), which is impossible for the standard bounded MS-SSIM index and undermines the quantitative table, though this is a metric-implementation concern rather than a circularity issue. No load-bearing self-citation chain is involved; the circularity is internal to the use of BiomedGPT/CLIP as both supervisor and evaluator.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim depends on several unvalidated domain assumptions about using CLIP and BiomedGPT as semantic bridges for medical images, and on hyperparameters selected based on test-set behavior. No new physical entities are introduced.

free parameters (5)
  • alpha = 0.1
    Weight for the medical semantic loss in Eq. 12, chosen by experiments rather than derived.
  • beta = 1.5
    Weight for the gradient loss in Eq. 12, chosen by experiments.
  • gamma = 1.5
    Weight for the reconstruction loss in Eq. 12; swept over 0.1, 0.5, 1, 1.5, 2 in Sec. IV-D-2 and set to 1.5.
  • theta = 0.85
    Threshold in Eq. 9 above which the medical semantic loss is set to zero; set via extensive experiments.
  • omega
    Adjustment factor for SSIM in Eq. 11; not specified in the implementation details, so the value is unknown.
assumptions (4)
  • domain assumption BiomedGPT-generated text descriptions of source images are accurate, clinically meaningful medical prior knowledge.
    The entire guidance mechanism rests on these captions being reliable; no verification against expert annotations or ground truth is provided.
  • domain assumption CLIP text encoder embeddings are semantically aligned with medical image features, despite CLIP being trained on natural images.
    Eq. 9 uses cosine similarity in CLIP space as a semantic loss; the paper provides no medical-domain validation of this alignment.
  • domain assumption The cross-attention based affine transformation from FFCLIP transfers to medical image fusion.
    The semantic interaction alignment module (Eqs. 5-7) is adopted from a natural-image editing model without medical adaptation or validation.
  • domain assumption Diagnostic reports generated by BiomedGPT from fused images are a valid measure of clinical information preservation.
    Section IV-C uses text length, entropy, and keyword counts as evaluation; no external clinical ground truth or radiologist validation is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SMFusion: Semantic-Preserving Fusion of Multimodal Medical Images for Enhanced Clinical Diagnosis." pith.science (2026). https://pith.science/paper/QEXJJV4O

@misc{pith2026250512251,
  author       = {Pith},
  title        = {Pith review of: SMFusion: Semantic-Preserving Fusion of Multimodal Medical Images for Enhanced Clinical Diagnosis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QEXJJV4O}},
  note         = {Machine review of arXiv:2505.12251}
}
read the original abstract

Multimodal medical image fusion plays a crucial role in medical diagnosis by integrating complementary information from different modalities to enhance image readability and clinical applicability. However, existing methods mainly follow computer vision standards for feature extraction and fusion strategy formulation, overlooking the rich semantic information inherent in medical images. To address this limitation, we propose a novel semantic-guided medical image fusion approach that, for the first time, incorporates medical prior knowledge into the fusion process. Specifically, we construct a publicly available multimodal medical image-text dataset, upon which text descriptions generated by BiomedGPT are encoded and semantically aligned with image features in a high-dimensional space via a semantic interaction alignment module. During this process, a cross attention based linear transformation automatically maps the relationship between textual and visual features to facilitate comprehensive learning. The aligned features are then embedded into a text-injection module for further feature-level fusion. Unlike traditional methods, we further generate diagnostic reports from the fused images to assess the preservation of medical information. Additionally, we design a medical semantic loss function to enhance the retention of textual cues from the source images. Experimental results on test datasets demonstrate that the proposed method achieves superior performance in both qualitative and quantitative evaluations while preserving more critical medical information.

Figures

Figures reproduced from arXiv: 2505.12251 by the authors.

Figure 1
Figure 1. Comparison of Existing Medical Image Fusion Methods and Our Proposed Approach. (a) Existing fusion methods: These methods are primarily based on deep learning networks and inte￾grate information from multiple source images through unsupervised learning strategies. (b) Proposed Semantic-Guided Image Fusion: Our approach integrates medical prior knowledge to explicitly guide the fusion process, establishing an interac… view at source ↗
Figure 2
Figure 2. Schematic illustration of multimodal medical image-text dataset [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The workflow of SMFusion. It consists of three key components: the feature extraction module (a), the semantic interaction alignment module (b), [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison of our SMFusion with seven state-of-the-art methods on three MRI and CT image pairs. Enlarged detail patches are highlighted [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Qualitative comparison of SMFusion with seven state-of-the-art methods on three MRI and PET image pairs. Notably, our method demonstrates [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Qualitative comparison of ours with seven state-of-the-art methods on three MRI and SPECT image pairs. Enlarged detail patches are highlighted [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Ablation qualitative experiments were performed on three different [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Ablation experiments between different hyperparameter settings. The [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 23 canonical work pages

  1. [1]

    Multimodal medical image fusion via laplacian pyramid and convolutional neural network reconstruction with local gradient energy strategy,

    J. Fu, W. Li, J. Du, and B. Xiao, “Multimodal medical image fusion via laplacian pyramid and convolutional neural network reconstruction with local gradient energy strategy,” Computers in Biology and Medicine , vol. 126, p. 104048, 2020

    work page 2020

  2. [2]

    Medical image fusion with parameter-adaptive pulse coupled neural network in nonsubsampled shearlet transform domain,

    M. Yin, X. Liu, Y . Liu, and X. Chen, “Medical image fusion with parameter-adaptive pulse coupled neural network in nonsubsampled shearlet transform domain,” IEEE Transactions on Instrumentation and Measurement, vol. 68, no. 1, pp. 49–64, 2018

    work page 2018

  3. [3]

    Multiresolution-based image fusion with additive wavelet decompo- sition,

    J. Nunez, X. Otazu, O. Fors, A. Prades, V . Pala, and R. Arbiol, “Multiresolution-based image fusion with additive wavelet decompo- sition,” IEEE Transactions on Geoscience and Remote sensing , vol. 37, no. 3, pp. 1204–1211, 1999

    work page 1999

  4. [4]

    Principal components analysis (pca),

    A. Ma ´ckiewicz and W. Ratajczak, “Principal components analysis (pca),” Computers & Geosciences , vol. 19, no. 3, pp. 303–342, 1993

    1993

  5. [5]

    Image fusion using quaternion wavelet transform and multiple features,

    P. Chai, X. Luo, and Z. Zhang, “Image fusion using quaternion wavelet transform and multiple features,” IEEE access , vol. 5, pp. 6724–6734, 2017

    work page 2017

  6. [6]

    The curvelet transform,

    J. Ma and G. Plonka, “The curvelet transform,” IEEE signal processing magazine, vol. 27, no. 2, pp. 118–133, 2010

    work page 2010

  7. [7]

    Image denoising using nonsubsampled shearlet transform and twin support vector ma- chines,

    H.-Y . Yang, X.-Y . Wang, P.-P. Niu, and Y .-C. Liu, “Image denoising using nonsubsampled shearlet transform and twin support vector ma- chines,” Neural networks , vol. 57, pp. 152–165, 2014

    work page 2014

  8. [8]

    Sparse representation-based multi- focus image fusion method via local energy in shearlet domain,

    L. Li, M. Lv, Z. Jia, and H. Ma, “Sparse representation-based multi- focus image fusion method via local energy in shearlet domain,”Sensors, vol. 23, no. 6, p. 2888, 2023

    work page 2023

Show all 41 references
  1. [9]

    Multi-modal medical image fusion by laplacian pyramid and adaptive sparse representation,

    Z. Wang, Z. Cui, and Y . Zhu, “Multi-modal medical image fusion by laplacian pyramid and adaptive sparse representation,” Computers in Biology and Medicine , vol. 123, p. 103823, 2020

    2020

  2. [10]

    Ifcnn: A general image fusion framework based on convolutional neural network,

    Y . Zhang, Y . Liu, P. Sun, H. Yan, X. Zhao, and L. Zhang, “Ifcnn: A general image fusion framework based on convolutional neural network,” Information Fusion , vol. 54, pp. 99–118, 2020

    2020

  3. [11]

    Swinfusion: Cross-domain long-range learning for general image fusion via swin transformer,

    J. Ma, L. Tang, F. Fan, J. Huang, X. Mei, and Y . Ma, “Swinfusion: Cross-domain long-range learning for general image fusion via swin transformer,” IEEE/CAA Journal of Automatica Sinica , vol. 9, no. 7, pp. 1200–1217, 2022

    2022

  4. [12]

    Matr: Multimodal medical image fusion via multiscale adaptive transformer,

    W. Tang, F. He, Y . Liu, and Y . Duan, “Matr: Multimodal medical image fusion via multiscale adaptive transformer,”IEEE Transactions on Image Processing, vol. 31, pp. 5134–5149, 2022

    2022

  5. [13]

    Ddcgan: A dual-discriminator conditional generative adversarial network for multi- resolution image fusion,

    J. Ma, H. Xu, J. Jiang, X. Mei, and X.-P. Zhang, “Ddcgan: A dual-discriminator conditional generative adversarial network for multi- resolution image fusion,” IEEE Transactions on Image Processing , vol. 29, pp. 4980–4995, 2020

    2020

  6. [14]

    Addns: An asymmetric dual deep network with sharing mechanism for medical image fusion of ct and mr-t2,

    W. Huang, H. Zhang, H. Guo, W. Li, X. Quan, and Y . Zhang, “Addns: An asymmetric dual deep network with sharing mechanism for medical image fusion of ct and mr-t2,” Computers in Biology and Medicine , vol. 166, p. 107531, 2023

    2023

  7. [15]

    A semantic-based medical image fusion approach,

    F. Fan, Y . Huang, L. Wang, X. Xiong, Z. Jiang, Z. Zhang, and J. Zhan, “A semantic-based medical image fusion approach,” arXiv preprint arXiv:1906.00225, 2019

    1906 arXiv

  8. [16]

    Image fusion via vision-language model,

    Z. Zhao, L. Deng, H. Bai, Y . Cui, Z. Zhang, Y . Zhang, H. Qin, D. Chen, J. Zhang, P. Wang et al. , “Image fusion via vision-language model,” arXiv preprint arXiv:2402.02235 , 2024

    2024 arXiv

  9. [17]

    Biomedgpt: A generalist vision-language foundation model for diverse biomedical tasks,

    K. Zhang, R. Zhou, E. Adhikarla, Z. Yan, Y . Liu, J. Yu, Z. Liu, X. Chen, B. D. Davison, and H. Ren, “Biomedgpt: A generalist vision-language foundation model for diverse biomedical tasks,” Nature Medicine, 2023

    2023

  10. [18]

    Unsuper- vised vision-and-language pre-training via retrieval-based multi-granular 13 alignment,

    M. Zhou, L. Yu, A. Singh, M. Wang, Z. Yu, and N. Zhang, “Unsuper- vised vision-and-language pre-training via retrieval-based multi-granular 13 alignment,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 16 485–16 494

    2022

  11. [19]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al. , “Learning transferable visual models from natural language supervision,” in International conference on machine learning . PmLR, 2021, pp. 8748–8763

    2021

  12. [20]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,

    J. Li, D. Li, C. Xiong, and S. Hoi, “Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” in International conference on machine learning . PMLR, 2022, pp. 12 888–12 900

    2022

  13. [21]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    2023 arXiv

  14. [22]

    Learning to prompt for vision- language models,

    K. Zhou, J. Yang, C. C. Loy, and Z. Liu, “Learning to prompt for vision- language models,” International Journal of Computer Vision , vol. 130, no. 9, pp. 2337–2348, 2022

    2022

  15. [23]

    Promptfusion: Harmonized semantic prompt learning for infrared and visible image fusion,

    J. Liu, X. Li, Z. Wang, Z. Jiang, W. Zhong, W. Fan, and B. Xu, “Promptfusion: Harmonized semantic prompt learning for infrared and visible image fusion,” IEEE/CAA Journal of Automatica Sinica , 2024

    2024

  16. [24]

    Textfusion: Unveiling the power of textual semantics for controllable image fusion,

    C. Cheng, T. Xu, X.-J. Wu, H. Li, X. Li, Z. Tang, and J. Kittler, “Textfusion: Unveiling the power of textual semantics for controllable image fusion,” Information Fusion , vol. 117, p. 102790, 2025

    2025

  17. [25]

    Text-if: Leveraging semantic text guidance for degradation-aware and interactive image fusion,

    X. Yi, H. Xu, H. Zhang, L. Tang, and J. Ma, “Text-if: Leveraging semantic text guidance for degradation-aware and interactive image fusion,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 27 026–27 035

    2024

  18. [26]

    Text-if: Leveraging semantic text guidance for degradation-aware and interactive image fusion,

    ——, “Text-if: Leveraging semantic text guidance for degradation-aware and interactive image fusion,” IEEE, 2024

    2024

  19. [27]

    Restormer: Efficient transformer for high-resolution image restoration,

    S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, and M.-H. Yang, “Restormer: Efficient transformer for high-resolution image restoration,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 5728–5739

    2022

  20. [28]

    One model to edit them all: Free-form text-driven image ma- nipulation with semantic modulations,

    Y . Zhu, H. Liu, Y . Song, Z. Yuan, X. Han, C. Yuan, Q. Chen, and J. Wang, “One model to edit them all: Free-form text-driven image ma- nipulation with semantic modulations,” Advances in Neural Information Processing Systems, vol. 35, pp. 25 146–25 159, 2022

    2022

  21. [29]

    Hairclip: Design your hair by text and reference image,

    T. Wei, D. Chen, W. Zhou, J. Liao, Z. Tan, L. Yuan, W. Zhang, and N. Yu, “Hairclip: Design your hair by text and reference image,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 18 072–18 081

    2022

  22. [30]

    A semantic-driven coupled network for infrared and visible image fusion,

    X. Liu, H. Huo, J. Li, S. Pang, and B. Zheng, “A semantic-driven coupled network for infrared and visible image fusion,” Information Fusion, vol. 108, p. 102352, 2024

    2024

  23. [31]

    Loss functions for image restoration with neural networks,

    H. Zhao, O. Gallo, I. Frosio, and J. Kautz, “Loss functions for image restoration with neural networks,” IEEE Transactions on computational imaging, vol. 3, no. 1, pp. 47–57, 2016

    2016

  24. [32]

    Diff-if: Multi-modality image fusion via diffusion model with fusion knowledge prior,

    X. Yi, L. Tang, H. Zhang, H. Xu, and J. Ma, “Diff-if: Multi-modality image fusion via diffusion model with fusion knowledge prior,” Infor- mation Fusion , vol. 110, p. 102450, 2024

    2024

  25. [33]

    Metafusion: Infrared and visible image fusion via meta-feature embedding from object detection,

    W. Zhao, S. Xie, F. Zhao, Y . He, and H. Lu, “Metafusion: Infrared and visible image fusion via meta-feature embedding from object detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 13 955–13 965

    2023

  26. [34]

    Equivariant multi-modality image fusion,

    Z. Zhao, H. Bai, J. Zhang, Y . Zhang, K. Zhang, S. Xu, D. Chen, R. Timofte, and L. Van Gool, “Equivariant multi-modality image fusion,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2024, pp. 25 912–25 921

    2024

  27. [35]

    Ddfm: denoising diffusion model for multi-modality image fusion,

    Z. Zhao, H. Bai, Y . Zhu, J. Zhang, S. Xu, Y . Zhang, K. Zhang, D. Meng, R. Timofte, and L. Van Gool, “Ddfm: denoising diffusion model for multi-modality image fusion,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 8082–8093

    2023

  28. [36]

    U2fusion: A unified unsupervised image fusion network,

    H. Xu, J. Ma, J. Jiang, X. Guo, and H. Ling, “U2fusion: A unified unsupervised image fusion network,” IEEE transactions on pattern analysis and machine intelligence , vol. 44, no. 1, pp. 502–518, 2020

    2020

  29. [37]

    Sdnet: A versatile squeeze-and-decomposition network for real-time image fusion,

    H. Zhang and J. Ma, “Sdnet: A versatile squeeze-and-decomposition network for real-time image fusion,” International Journal of Computer Vision, vol. 129, no. 10, pp. 2761–2785, 2021

    2021

  30. [38]

    Infrared and visible image fusion methods and applications: A survey,

    J. Ma, Y . Ma, and C. Li, “Infrared and visible image fusion methods and applications: A survey,” Information fusion , vol. 45, pp. 153–178, 2019

    2019

  31. [39]

    Detail preserved fusion of visible and infrared images using regional saliency extraction and multi-scale image decomposition,

    G. Cui, H. Feng, Z. Xu, Q. Li, and Y . Chen, “Detail preserved fusion of visible and infrared images using regional saliency extraction and multi-scale image decomposition,” Optics Communications , vol. 341, pp. 199–209, 2015

    2015

  32. [40]

    Objective image fusion performance measure,

    C. S. Xydeas and V . Petrovic, “Objective image fusion performance measure,” Electronics letters, vol. 36, no. 4, pp. 308–309, 2000

    2000

  33. [41]

    Multiscale structural similarity for image quality assessment,

    Z. Wang, E. P. Simoncelli, and A. C. Bovik, “Multiscale structural similarity for image quality assessment,” in The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003 , vol. 2. Ieee, 2003, pp. 1398–1402

    2003

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.