Pith. sign in

REVIEW 3 major objections 6 minor 22 references

Structural Similarity based Anatomical and Functional Brain Imaging Fusion

T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that an end-to-end unsupervised CNN trained with a structural-similarity-based loss can fuse MRI-PET brain images faster and with better edge preservation than established multi-scale fusion pipelines.

desk verdict Worth a look for the speed and visualization, but the quality claim rests on the same metric the network optimizes, so the paper's 'better fusion' conclusion doesn't hold as written. read the letter →

arxiv 1908.03958 v4 pith:ZIEQ3LNE submitted 2019-08-11 eess.IV cs.CVcs.LG

Nishant Kumar , Nico Hoffmann , Martin Oelschlägel , Edmund Koch , Matthias Kirsch , Stefan Gumhold This is my paper · ORCID
classification eess.IVcs.CVcs.LG
keywords medicalimagefusionMRI-PETconvolutionalneuralnetworkunsupervisedlearningstructuralsimilarityindexSSIMlossbrainimagingmultimodal
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that MRI-PET brain image fusion can be done by an end-to-end unsupervised convolutional neural network rather than by multi-scale transform pipelines. The network is trained directly on registered medical image pairs, with a loss that combines structural similarity (SSIM) and pixel-level ℓ2 error, so no ground-truth fused image is needed. The authors report that the resulting fused images preserve structural edges from both modalities better than six prior methods on the SSIM and visual-information-fidelity metrics, and that fusion plus color visualization takes 0.37 seconds per pair, orders of magnitude faster than the compared methods. If the claim holds, real-time diagnostic or neurosurgical settings could use such fusion without dedicated hybrid scanning hardware.

What carries the argument

The load-bearing mechanism is a steerable total loss $L_{\text{total}} = \lambda L_{\text{SSIM}} + (1-\lambda) L_{\ell^2}$, with $L_{\text{SSIM}} = (1-\text{SSIM}(I_1,F)) + (1-\text{SSIM}(I_2,F))$ and $L_{\ell^2} = \|F-I_1\|_2 + \|F-I_2\|_2$, which lets the network trade structural perception against luminance fidelity during unsupervised training. Around this loss sits the fusion rule of Eq. (1), which combines high-frequency feature maps by max-average pooling and low-frequency maps by weighted averaging. The SSIM term is what pushes the fused output toward human-perceptible structure; the $\ell^2$ term prevents brightness drift; the hand-set $\lambda$ balances them. The same loss and fusion rules are what make the network trainable without ground truth and what, according to the paper, account for its edge preservation.

What would settle it

Train the same network and loss with the high-frequency fusion rule replaced by a learned per-channel weighted average. If that variant matches or beats the reported QSSIM (0.8610) and QVIF (0.6005) on the same 100 test pairs, then the paper's attribution of its edge-preserving advantage to the max-average rule would be unsupported.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a small fully convolutional network, trained without supervision on 500 ADNI MRI-PET pairs with the loss $L_{\text{total}} = \lambda L_{\text{SSIM}} + (1-\lambda) L_{\ell^2}$, can fuse new MRI-PET pairs in a single forward pass while keeping more structural information than established fusion pipelines. The fused image is produced by decomposing each input into high- and low-frequency feature maps with modality-specific kernels, combining high-frequency features by the paper's max-average pooling rule and low-frequency features by weighted averaging (Eq. 1), and reconstructing with three hidden layers. On 100 held-out pairs the method scores highest among all compared methods on the SSIM metric (0.8610) and on visual information fidelity (0.6005), while running in 0.37 seconds versus 13.43 seconds for guided filtering and 481.73 seconds for the CNN baseline. A gradient-based color visualization then shows, per pixel, how much each input modality contributed to the fused image.

Load-bearing premise

The paper's central result depends on the hand-coded fusion rules in Eq. (1) — max-average pooling for high-frequency features and weighted averaging for low-frequency features — being the right way to combine the decomposed inputs; the network learns the feature extractors and reconstruction around those fixed rules, so if another combination rule worked better the reported advantage could disappear.

Editorial extensions

If this is right

  • Fusion of aligned MRI-PET pairs becomes fast enough (0.37 seconds per image, including color visualization) for interactive or streamed use during surgery, a regime the multi-scale transform baselines cannot enter.
  • Because the network is trained unsupervised on registered medical pairs, it does not depend on natural-image pretraining or on a ground-truth fused image, which do not exist for this task.
  • The same architecture can be pointed at other anatomical-functional modality pairs by changing the feature-extraction kernel sizes, as the paper states as future work.
  • The gradient-based color visualization gives clinicians a per-pixel readout of whether anatomy (MRI) or function (PET) dominates each fused region, which could support interpretation during diagnosis.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's reported numbers, a same-hardware, same-software benchmark (GPU forward pass versus GPU implementations of the baselines) would be the natural next step to separate architectural speed from implementation environment.
  • If max-average pooling is the mechanism behind the edge preservation, the same fusion rule could be dropped into other unsupervised fusion networks or into multi-scale pipelines and evaluated independently.
  • The gradient-based contribution map, which the paper uses for color visualization, could also be read as a per-pixel attribution or confidence map for clinical decision support.
  • A learned variant of the fusion rule would directly test whether the fixed combination is necessary; the paper does not report such an ablation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The manuscript proposes an unsupervised convolutional neural network for fusing MRI and PET grayscale image pairs. The network decomposes each input into high- and low-frequency features, combines them with fixed rules (max-average pooling for high-frequency, weighted averaging for low-frequency), and is trained end-to-end with a loss that combines SSIM and ℓ2 terms. The authors also propose a gradient-based color visualization of the fused image. The model is trained on 500 ADNI pairs and tested on 100 pairs from ADNI and the Harvard Whole Brain Atlas, with quantitative comparison against six existing fusion methods. The paper reports the best QSSIM and QVIF scores and a runtime of 0.37 s per image, about three orders of magnitude faster than the compared methods.

Significance. If the quality claims were established, this would be a practically useful contribution, particularly because the method trains on medical image pairs rather than natural images, runs in real-time-compatible times, and the code is publicly available. The speed advantage in Table 1 is clear and not disputed. However, the evidence for improved fusion quality is substantially weakened by two issues: the headline metric QSSIM is the same measure used as the training loss, and the hyperparameters λ and Ω are selected by maximizing the reported metrics on the same test set. The independent edge metric QG in Table 1 ranks the proposed method 6th of 7, which directly contradicts the paper's edge-preservation claim. These problems affect the central claim and require revision.

major comments (3)
  1. [§3.4, Table 1, Eq. (4)] The QSSIM column in Table 1 is not independent evidence of fusion quality. In Eq. (4), the training loss is L_SSIM = (1−SSIM(I1,F)) + (1−SSIM(I2,F)); the reported QSSIM metric, as defined by the same reference [15], is an average or aggregate of these same per-input SSIM values. Minimizing the training loss is therefore essentially the same as maximizing the reported QSSIM. Reporting this metric as the main quantitative evidence that the method 'performs better' is circular. The authors should either report metrics that are not part of the training objective as primary evidence, or explicitly state that the QSSIM improvement is an expected consequence of optimizing that objective.
  2. [§3.3] The hyperparameters λ and Ω are selected by evaluating 10 values each on the 100 test images and choosing the values that maximize the evaluation metrics ('The evaluation resulted in highest scores with λ = 0.8 and Ω = 0.6'). This constitutes tuning on the test set, so the numbers in Table 1 are fitted values rather than independent predictions. In addition, Table 1 reports only mean values over the 100 test images, with no standard deviations or confidence intervals, so the reader cannot judge whether the differences are statistically meaningful. The authors should use a separate validation split or nested cross-validation for hyperparameter selection, and should report uncertainty measures for all metrics.
  3. [§3.4, Table 1] The qualitative claim that the proposed method 'is robust in preserving the edges' is contradicted by the paper's own independent edge-oriented metric, QG (Xydeas metric). In Table 1, the proposed method scores 0.5707, ranking 6th of 7, below GF (0.7555), NSCT-RPCNN (0.6702), NSST-PAPCNN (0.6685), and LP-SR (0.6501). Since the abstract and Section 3.4 emphasize better structural and edge preservation, this discrepancy must be addressed. Either the claim should be limited to the metrics that actually improve (QSSIM, QVIF), or the authors need to provide additional independent evidence of edge preservation that is consistent with Table 1.
minor comments (6)
  1. [Abstract] Please check the spelling of 'malignant' in the abstract; the full text version reads 'malign tissues'.
  2. [Eq. (1)] The expression for H_o(φ) appears to be missing an operator between the max term and the (H1(φ)+H2(φ)) term; please clarify whether the intended operation is element-wise multiplication or something else.
  3. [Eq. (5)] The notation in Eq. (5) is unclear: summing over all i,j of ∂F[i,j]/∂I1[n,m] appears to loose the spatial correspondence of the derivative. Please define the partial derivatives more precisely, for example through the chain rule of the network or as entries of a Jacobian.
  4. [Figure 2] In the visual results, the red box referenced in Section 3.4 is not clearly visible in the printed figure; please annotate the figure so the claimed PET edge preservation can be located visually.
  5. [Table 1] Consider reporting standard deviations or confidence intervals for all metrics; without them, it is impossible to tell whether the reported differences are meaningful, especially where the proposed method ties or nearly ties with another method.
  6. [References [15] and loss function] Because reference [15] is used both for the SSIM loss in Eq. (4) and for the QSSIM metric in Table 1, the relation should be stated explicitly in the text and the circularity concern acknowledged and discussed.

Circularity Check

2 steps flagged · score 7.0 of 10

The QSSIM-based structural-quality claim is the training objective itself, and the reported test-table configuration was tuned on the same test images, so the central 'better preservation' claim is partly circular.

  1. self definitional [Section 3.4, Objective assessment; Eq. (4)]
    "The results convey that our method performs better with the quality metric QSSIM and QVIF F . This is assertive from the fact that the neural network optimizes the loss function and subsequently improves the structural information in the fused image."

    Eq. (4) defines the training loss as LSSIM = (1 − SSIM(I1, F)) + (1 − SSIM(I2, F)), so the fused image is explicitly optimized to increase SSIM with each input. Table 1 then reports QSSIM [15] as the headline quality metric, and Section 3.4 explicitly attributes the QSSIM win to the network optimizing the loss function. Maximizing the reported QSSIM is therefore an affine transform of minimizing the optimized LSSIM: the advantage is a by-construction consequence of training, not an independent verification of structural/edge preservation. The paper's own independent edge-specific metric QG ranks the proposed method 6th of 7 (0.5707 vs 0.7555 for GF), so the structural-superiority claim is not separately established.

  2. fitted input called prediction [Section 3.3, Evaluation settings; Table 1]
    "The visualisation results of the test images were evaluated with 10 values of λ, Ω ∈ [0, 1] on four objective assessment metrics namely nonlinear correlation information entropy ( QIE ) [ 18], xydeas metric ( QG) [ 19], feature mutual information ( QF M I) [ 20], structural similarity metric ( QSSIM ) [ 15] and human perceptive visual information fidelity ( QV IF F ) [21] with higher values means better performances. The evaluation resulted in highest scores with λ = 0.8 and Ω = 0.6 for three of the mentioned metrics."

    Section 3.3 states that the test images were evaluated with 10 values of λ and Ω and that λ = 0.8, Ω = 0.6 were chosen because they gave highest scores on the metrics. These are the same 100 test pairs used to build Table 1. Therefore the reported QVIF and QSSIM values are selected rather than predicted: the configuration was fitted on the test set and then re-reported as the method's comparative performance. No separate validation split or confidence intervals are provided, so the Table 1 comparisons are test-set-tuned rather than an unbiased prediction.

full rationale

The central quality claim, that the fused images 'preserve structural information and edges better than previous methods,' is supported mainly by the QSSIM column, but QSSIM is the same index used in the training loss in Eq. (4); the paper even says the QSSIM improvement is 'assertive from the fact that the neural network optimizes the loss function.' This makes the headline structural-quality advantage partly a self-definitional result. Additionally, λ and Ω were selected on the same 100 test images using the same metrics (Section 3.3), so the numbers in Table 1 are fitted rather than independent predictions. The independent edge metric QG tells the opposite story for the edge-preservation claim: the proposed method scores 0.5707, below GF (0.7555), LP-SR (0.6501), NSCT-RPCNN (0.6702), and NSST-PAPCNN (0.6685), ranking 6th of 7. The runtime advantage (0.37 s versus 13.43 s and 481.73 s) is genuine and independent, but it does not support the 'better structural and edge preservation' clause. There is no load-bearing self-citation or uniqueness-theorem issue, so the circularity is partial rather than total; the score reflects that the paper's central quality claim reduces to its own training objective and test-set tuning.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no invented physical or conceptual entities. Its central result depends on hand-picked hyperparameters and heuristic fusion rules.

free parameters (3)
  • lambda (loss weight) = 0.8
    Selected by trying 10 values and choosing the one with highest test metrics (Section 3.3). This is a free parameter tuned to the test set.
  • Omega (saturation factor) = 0.6
    Selected by trying 10 values and choosing the one with highest test metrics (Section 3.3). Controls the visualization's occlusion of anatomical detail.
  • Kernel sizes (9x9, 7x7, 3x3, 5x5)
    Hand-picked for feature extraction layers based on assumed frequency properties of MRI and PET; no formal selection.
assumptions (4)
  • domain assumption Affine registration of MRI and PET image pairs is sufficiently accurate.
    The paper aligns pairs using 3D Slicer's Affine tool (Section 3.1); misregistration would propagate into the fused image.
  • domain assumption SSIM plus L2 loss is a good proxy for clinical perceptual quality.
    The training objective is assumed to reflect human perception; no clinical validation is provided.
  • ad hoc to paper The fixed fusion rules (max-average pooling for HF, weighted averaging for LF) are near-optimal.
    These rules are introduced without learned justification in Eq. (1); the network only optimizes feature extraction and reconstruction.
  • domain assumption The 500 training pairs from ADNI are representative of the test distribution.
    The model is trained on ADNI data and tested on ADNI plus Harvard; if the Harvard domain differs, the reported performance may not transfer.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Structural Similarity based Anatomical and Functional Brain Imaging Fusion." pith.science (2026). https://pith.science/paper/ZIEQ3LNE

@misc{pith2026190803958,
  author       = {Pith},
  title        = {Pith review of: Structural Similarity based Anatomical and Functional Brain Imaging Fusion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZIEQ3LNE}},
  note         = {Machine review of arXiv:1908.03958}
}
read the original abstract

Multimodal medical image fusion helps in combining contrasting features from two or more input imaging modalities to represent fused information in a single image. One of the pivotal clinical applications of medical image fusion is the merging of anatomical and functional modalities for fast diagnosis of malignant tissues. In this paper, we present a novel end-to-end unsupervised learning-based Convolutional Neural Network (CNN) for fusing the high and low frequency components of MRI-PET grayscale image pairs, publicly available at ADNI, by exploiting Structural Similarity Index (SSIM) as the loss function during training. We then apply color coding for the visualization of the fused image by quantifying the contribution of each input image in terms of the partial derivatives of the fused image. We find that our fusion and visualization approach results in better visual perception of the fused image, while also comparing favorably to previous methods when applying various quantitative assessment metrics.

Figures

Figures reproduced from arXiv: 1908.03958 by the authors.

Figure 1
Figure 1. The proposed fusion and visualisation framework. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The three sets of images shows visual results of compared m [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Training loss curves with 200 epochs and several [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 21 canonical work pages

  1. [15]

    IEEE Trans actions on Image Process- ing

    Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: I mage quality assessment: From error visibility to structural similarity. IEEE Trans actions on Image Process- ing. 13(4), 600–612 (2004)

    work page 2004

  2. [1]

    Information Fusion

    James, A.P., Dasarathy, B.V.: Medical Image Fusion: A Sur vey of the State of the Art. Information Fusion. 19, 4–19 (2014)

    work page 2014

  3. [2]

    Diagnost ic and interventional radiology

    Nensa, F., Beiderwellen, K., Heusch, P., Wetter, A.: Clin ical applications of PET/MRI: Current status and future perspectives. Diagnost ic and interventional radiology. 20(5), 438–447 (2014)

    work page 2014

  4. [3]

    IEEE Transactions on Multimedi a

    Bhatnagar, G., Wu, Q.M.J., Liu, Z.: Directive Contrast Ba sed Multimodal Medical Image Fusion in NSCT Domain. IEEE Transactions on Multimedi a. 15(5), 1014– 1024 (2013)

    work page 2013

  5. [4]

    Neurocomputing

    Du, J., Li, W., Xiao, B., Nawaz, Q.: Union Laplacian pyrami d with multiple features for medical image fusion. Neurocomputing. 194, 326–339 (2016)

    work page 2016

  6. [5]

    IEEE Transac tions on Image Pro- cessing

    Du, J., Li, W., Xiao, B.: Anatomical-Functional Image Fus ion by Information of Interest in Local Laplacian Filtering Domain. IEEE Transac tions on Image Pro- cessing. 26(12), 5855–5866 (2017)

    work page 2017

  7. [6]

    IEEE Transactions on Instrumentation and Measurement

    Yin, M., Liu, X., Liu, Y., Chen, X.: Medical Image Fusion Wi th Parameter-Adaptive Pulse Coupled Neural Network in Nonsubsampled Shearlet Tra nsform Domain. IEEE Transactions on Instrumentation and Measurement. 68(1), 49–64 (2019)

    work page 2019

  8. [7]

    IEEE Transactions on Image Processing

    Li, S., Kang, X., Hu, J.: Image Fusion With Guided Filterin g. IEEE Transactions on Image Processing. 22(7), 2864–2875 (2013)

    work page 2013

Show all 22 references
  1. [8]

    Pattern Recognition

    Li, H., He, X., Tao, D., Tang, Y., Wang, R.: Joint medical im age fusion, denoising and enhancement via discriminative low-rank sparse dictio naries learning. Pattern Recognition. 79, 130–146 (2018)

    2018

  2. [9]

    Information Fusion

    Yang, B., Li, S.: Pixel-level image fusion with simultane ous orthogonal matching pursuit. Information Fusion. 13(1), 10–19 (2012)

    2012

  3. [10]

    Information Fu sion

    Liu, Y., Liu, S., Wang, Z.: A general framework for image f usion based on multi- scale transform and sparse representation. Information Fu sion. 24, 147–164 (2015)

    2015

  4. [11]

    Information Fusion

    Daneshvar, S., Ghassemian, H.: MRI and PET image fusion b y combining IHS and retina-inspired models. Information Fusion. 11(2), 114–123 (2010) Structural Similarity based Anatomical and Functional Bra in Imaging Fusion 9

    2010

  5. [12]

    Bio medical Engineer- ing/Biomedizinische Technik

    Hoffmann, N., Weidner, F., Urban, P., et al.: Framework fo r 2D-3D image fusion of infrared thermography with preoperative MRI. Bio medical Engineer- ing/Biomedizinische Technik. 62(6), 599–607 (2017)

    2017

  6. [13]

    In: 20th International C onference on Information Fusion (Fusion), pp 1–7

    Liu, Y., Chen, X., Cheng, J., Peng, H.: A medical image fus ion method based on convolutional neural networks. In: 20th International C onference on Information Fusion (Fusion), pp 1–7. Xi’an (2017). https://doi.org/10.23919/ICIF.2017.8009769

    2017

  7. [14]

    Information Fusion

    Liu, Y., Chen, X., Wang, Z., Wang, Z.J., Ward, R.K., Wang, X.: Deep learning for pixel-level image fusion: Recent advances and future prosp ects. Information Fusion. 42, 158–173 (2018)

    2018

  8. [16]

    Journal of magnetic resonance imaging 27(4), 685–691 (2008)

    Clifford, R.J., et al.: The Alzheimer’s Disease neuroima ging initiative (ADNI). Journal of magnetic resonance imaging 27(4), 685–691 (2008)

    2008

  9. [17]

    Johnson, K., Becker, J.: http://www.med.harvard.edu/AANLIB/home.html

  10. [18]

    21st IEEE Instrumenta tion and Measurement Technology Conference (IEEE Cat

    Wang, Q., Shen, Y.: Performances evaluation of image fus ion techniques based on nonlinear correlation measurement. 21st IEEE Instrumenta tion and Measurement Technology Conference (IEEE Cat. No.04CH37510) 1, 472–475 (2004)

    2004

  11. [19]

    Electronics Letters 36, 308–309 (2000)

    Xydeas, C., Petrovic, Vladimir.: Objective image fusio n performance measure. Electronics Letters 36, 308–309 (2000)

    2000

  12. [20]

    Haghighat, M.B.A., Aghagolzadeh, A., Seyedarabi, H.: A Non- reference Image Fusion Metric Based on Mutual Information of Image Features . Comput. Electr. Eng. 37(5), 744–756 (2011)

    2011

  13. [21]

    Information Fusion

    Han, Y., Cai, Y., Cao, Y., Xu, X.: A new image fusion perfor mance metric based on visual information fidelity. Information Fusion. 14(2), 127–135 (2013)

    2013

  14. [22]

    Medi cal & Biological Engi- neering & Computing

    Das, S., Kundu, M.K.: NSCT-based multimodal medical ima ge fusion using pulse- coupled neural network and modified spatial frequency. Medi cal & Biological Engi- neering & Computing. 50(10), 1105–1114 (2012)

    2012

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.