REVIEW 3 major objections 5 minor 44 references
Complex Wavelet Mutual Information Loss: A Multi-Scale Loss Function for Semantic Segmentation
T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read This paper claims that a loss function computing mutual information between complex steerable pyramid subbands of prediction and ground truth improves both pixel-wise and topological segmentation metrics across four datasets with minimal…
desk verdict The CWMI loss is a plausible and well-ablated segmentation loss, but Table 1 contains a duplicated MASS ROAD/Attention U-Net block that undermines the abstract's across-dataset claims until it is fixed. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The engine is the complex steerable pyramid, a redundant wavelet transform with orientation-tuned band-pass filters whose analytic complex subbands encode local phase as well as magnitude, so edges and corners appear as phase structure. For each pyramid level, the K directional subband coefficients at each pixel are treated as a random vector, and mutual information between prediction and label is estimated by the closed-form Gaussian formula $I \approx -\frac{1}{2} \log \det(M_n)$, extended to complex coefficients through Hermitian covariance and cross-covariance matrices (Eq. 9). Summing $-I$ over the $N$ pyramid levels and combining with cross entropy yields the CWMI loss; because each decomposition level downsamples by a factor of four, the total cost stays near $O(HW \log(HW))$.
What would settle it
Train the same U-Net on the same SNEMI3D folds with CWMI and with Dice and RMI losses, freezing all other hyperparameters; if CWMI does not beat both on mIoU and VI across repeated runs, the reported advantage does not generalize. To test the mechanism separately, compare Eq. 9 against a nonparametric mutual-information estimate on identical subbands and check whether training outcomes change.
Extended reading notes
Core claim
The central claim is that segmentation quality improves when mutual information is maximized between complex steerable pyramid subbands of the prediction and ground truth, rather than between their raw pixels. The CWMI loss sums, over pyramid levels, a closed-form Gaussian estimate of mutual information between each complex subband pair, adds a cross-entropy term, and is reported to outperform eleven baseline loss functions on the majority of metrics across SNEMI3D, GlaS, DRIVE, and Massachusetts Roads with both U-Net and Attention U-Net, and on SNEMI3D with VMUNet. Its reported gains include overlap measures such as mIoU and mDice but also structural measures such as variation of information, adjusted Rand index, and Hausdorff distance. The paper also claims the complex phase representation carries part of the benefit, since the full CWMI beats real-only, magnitude-only, and phase-only variants in the ablation study.
Load-bearing premise
The load-bearing premise is that the complex wavelet subband coefficients of prediction and label behave like a jointly Gaussian random vector, so that mutual information can be computed from covariance matrices alone; the paper borrows this approximation from regional mutual information without checking it on binary segmentation masks, whose wavelet coefficients are highly non-Gaussian.
Editorial extensions
If this is right
- Networks trained with CWMI should segment small instances and thin boundaries better in class- and instance-imbalanced datasets, without any change to the network architecture.
- The reported gains on variation of information, adjusted Rand index, and Hausdorff distance mean CWMI can serve as a cheap substitute for expensive topology-preserving losses on thin structures.
- The full complex form matters: replacing it with real-only, magnitude-only, or phase-only subbands reduces at least some metrics, so phase information carries part of the signal.
- With measured overhead of roughly 0.23 seconds per epoch over cross entropy and complexity near linear in pixel count, CWMI remains practical for high-resolution images.
- The loss also improved a Mamba-based U-Net on SNEMI3D, indicating compatibility beyond convolutional attention architectures.
Reading between the lines
- A testable extension beyond the paper: replace the Gaussian mutual-information estimate in Eq. 9 with a nonparametric estimator on the same subbands; if the results change materially, the Gaussian surrogate is load-bearing.
- The layer ablation shows the third level is best for regional metrics while the fourth is best for Hausdorff distance, so a per-level weighting scheme could improve on the paper's uniform sum.
- Because the loss penalizes structural mismatch in a phase-sensitive, translation-tolerant representation, it should transfer to other dense prediction tasks such as image-to-image translation and super-resolution, which the paper mentions only as future work.
- The paper's evidence is restricted to binary 2D segmentation; extending to multi-class and volumetric data is plausible but untested, so the practical scope of the claim is currently narrower than the title suggests.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Complex Wavelet Mutual Information (CWMI) loss, a multi-scale loss for semantic segmentation. The method decomposes prediction and ground-truth images with a complex steerable pyramid, estimates a closed-form regional mutual information term in each subband (borrowing the Gaussian estimator from RMI), and adds it to cross-entropy. Experiments compare CWMI with 11 loss functions on SNEMI3D, GlaS, DRIVE, and MASS ROAD using U-Net and Attention U-Net, plus VMUNet on SNEMI3D, reporting pixel-wise (mIoU, mDice) and topological/clustering (VI, ARI, HD) metrics. The central claim is that CWMI achieves significant improvements in pixel-wise accuracy and topological metrics with minimal computational overhead.
Significance. If the empirical results are reproduced, CWMI would be a useful plug-in loss: it is architecture-agnostic, has a favorable O(HW log HW) complexity analysis, and the ablations (phase vs. magnitude, decomposition levels, orientations, and regularization weight) are informative. The public code and the breadth of comparisons are strengths. However, the current manuscript contains a serious data-integrity issue in Table 1 that invalidates one of the four dataset blocks for Attention U-Net, and the abstract's significance claim is not fully supported by the table's own asterisks. The core idea remains credible, but the evidence base needs repair before the claims can be accepted.
major comments (3)
- [Table 1 (MASS ROAD, Attention U-Net block)] The entire MASS ROAD Attention U-Net block for the baseline losses is numerically identical to the GlaS Attention U-Net block: for example, CE is .640±.209/.725±.195/.906±.107/.406±.327/7.914±6.637 in both, and the same holds for BCE, Dice, Focal, Jaccard, Tversky, WCE, ABW, Skea-topo, clDice, and Sensitive rows. Two different datasets cannot yield identical means and standard deviations for eleven independent baseline training runs. This invalidates the MASS ROAD Attention U-Net comparisons, including the starred CWMI/CWMI-Real rows, and removes the support for the Section 4.2 statement that CWMI outperforms other losses 'for all datasets using both U-Net and Attention U-Net architectures.' The authors must rerun these baselines (or remove the block) and verify that the correct results are reported.
- [Abstract and Section 4.2] The abstract claims 'significant improvements in both pixel-wise accuracy and topological metrics,' but the paper's own significance markers do not support the pixel-wise part for GlaS and MASS ROAD. In Table 1, CWMI has no asterisks on mIoU or mDice for GlaS (both U-Net and Attention U-Net) or for MASS ROAD (both architectures); only VI and ARI are starred for those datasets, and MASS ROAD U-Net mIoU/mDice improvements are within one standard deviation of RMI. The significance claim in Section 4.2 is more carefully worded ('majority of evaluation metrics'), but the abstract overstates the result. The authors should either provide significance tests for mIoU/mDice on all datasets or revise the abstract to match the evidence.
- [Section 3.2, Eqs. (7)-(9)] The mutual information estimator is a closed-form Gaussian surrogate borrowed from RMI, extended to complex subbands via Hermitian covariance. Ground-truth segmentation masks are binary, and their wavelet subband coefficients are highly non-Gaussian, so the interpretation of the objective as 'mutual information' is heuristic unless the surrogate is validated. The manuscript offers no such validation (e.g., comparison with a nonparametric MI estimator, or a synthetic experiment showing the surrogate tracks structural agreement). This does not necessarily invalidate the empirical results, but the theoretical motivation should be explicitly qualified or supported.
minor comments (5)
- [Section 4.1, Datasets] The text says 'three public segmentation datasets' but then enumerates four (SNEMI3D, GlaS, DRIVE, MASS ROAD); this should read 'four.'
- [Table 1 header] The column header 'SENMI3D' is a typo and should be 'SNEMI3D.'
- [Section 4.3, Ablation references] The paragraph on decomposition level N and orientation K refers to 'Table 3' for the N/K and layer ablations, but these results appear in Table 5; the cross-references should be corrected.
- [Section 4.3, Computational complexity] The text says the computational overhead comparison is 'as shown in Table 4,' but the timing results are in Table 6; the cross-reference is incorrect.
- [Section 1, Contributions] The phrase '11 state-of-art loss functions' should use the standard spelling 'state-of-the-art.'
Circularity Check
No circularity: CWMI is an independently evaluated training objective; no fitted input is renamed as a prediction.
full rationale
The paper's central claim is that adding CWMI loss improves segmentation metrics. The loss is defined in Eqs. 7-13 as a training objective: it sums an RMI-style Gaussian MI estimate over complex steerable pyramid subbands and combines it with cross-entropy. Nothing in the definition is fitted to the reported evaluation metrics, and no parameter is chosen by matching the benchmark results; the hyperparameters (N=4, K=4, lambda=0.1) are selected via ablation on SNEMI3D and then applied as fixed settings. The MI estimator is explicitly borrowed from Zhao et al. (2019) and extended by Hermitian transpose, which is an independent, citable derivation rather than a self-citation. CWMI-Real is used as an ablation control, not as a 'prediction' of CWMI. The paper's empirical superiority claim depends on the integrity of Table 1, and the duplicated MASS ROAD Attention U-Net baseline block (identical values to the GlaS Attention U-Net block for CE, BCE, Dice, etc.) is a correctness/data-reporting concern, not a circular reasoning concern. The Gaussian-surrogate assumption for MI on binary masks is a validity risk, but it is an assumption about the estimator, not a tautology. Therefore no circular step can be exhibited, and the score is 0.
Assumptions & free parameters
free parameters (3)
- lambda regularization weight =
0.1
- N decomposition levels =
4
- K orientation count =
4
assumptions (4)
- domain assumption Equation 7 from RMI loss provides a valid approximation to mutual information for the subband features.
- domain assumption Treating K directional complex coefficients at each pixel as a K-dimensional random variable and computing covariances over spatial positions is meaningful.
- domain assumption Maximizing mutual information between prediction and ground truth subbands improves structural and topological segmentation quality.
- domain assumption The complex steerable pyramid can decompose binary ground-truth masks with meaningful phase information.
Cite this review
Pith. "Pith review of Complex Wavelet Mutual Information Loss: A Multi-Scale Loss Function for Semantic Segmentation." pith.science (2026). https://pith.science/paper/7HEZOXKC
@misc{pith2026250200563,
author = {Pith},
title = {Pith review of: Complex Wavelet Mutual Information Loss: A Multi-Scale Loss Function for Semantic Segmentation},
year = {2026},
howpublished = {\url{https://pith.science/paper/7HEZOXKC}},
note = {Machine review of arXiv:2502.00563}
}
read the original abstract
Recent advancements in deep neural networks have significantly enhanced the performance of semantic segmentation. However, class imbalance and instance imbalance remain persistent challenges, where smaller instances and thin boundaries are often overshadowed by larger structures. To address the multiscale nature of segmented objects, various models have incorporated mechanisms such as spatial attention and feature pyramid networks. Despite these advancements, most loss functions are still primarily pixel-wise, while regional and boundary-focused loss functions often incur high computational costs or are restricted to small-scale regions. To address this limitation, we propose the complex wavelet mutual information (CWMI) loss, a novel loss function that leverages mutual information from subband images decomposed by a complex steerable pyramid. The complex steerable pyramid captures features across multiple orientations and preserves structural similarity across scales. Meanwhile, mutual information is well-suited to capturing high-dimensional directional features and offers greater noise robustness. Extensive experiments on diverse segmentation datasets demonstrate that CWMI loss achieves significant improvements in both pixel-wise accuracy and topological metrics compared to state-of-the-art methods, while introducing minimal computational overhead. Our code is available at https://github.com/lurenhaothu/CWMI
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
-
[2]
S., Vishwanathan, A., and Berger, D
Arganda-Carreras, I., Seung, H. S., Vishwanathan, A., and Berger, D. R. Snemi3d: 3d segmentation of neurites in em images. (No Title), 2013
work page 2013
-
[3]
Loss functions in the era of semantic segmentation: A survey and outlook
Azad, R., Heidary, M., Yilmaz, K., H \"u ttemann, M., Karimijafarbigloo, S., Wu, Y., Schmeink, A., and Merhof, D. Loss functions in the era of semantic segmentation: A survey and outlook. arXiv preprint arXiv:2312.05391, 2023
arXiv 2023
-
[4]
A computational approach to edge detection
Canny, J. A computational approach to edge detection. IEEE Transactions on pattern analysis and machine intelligence, 0 (6): 0 679--698, 1986
work page 1986
-
[5]
Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A. L., and Zhou, Y. Transunet: Transformers make strong encoders for medical image segmentation. arXiv preprint arXiv:2102.04306, 2021
arXiv 2021
-
[6]
I., Waldner, F., Caccetta, P., and Wu, C
Diakogiannis, F. I., Waldner, F., Caccetta, P., and Wu, C. Resunet-a: A deep learning framework for semantic segmentation of remotely sensed data. ISPRS Journal of Photogrammetry and Remote Sensing, 162: 0 94--114, 2020
work page 2020
-
[7]
Golub, G. H. and Van Loan, C. F. Matrix computations. JHU press, 2013
2013
-
[8]
Fixed-point deformable u-net for pancreas ct segmentation
Huang, M., Huang, C., Yuan, J., and Kong, D. Fixed-point deformable u-net for pancreas ct segmentation. In Proceedings of the Third International Symposium on Image Computing and Digital Medicine, pp.\ 283--287, 2019
work page 2019
Show all 44 references
-
[9]
Islam, M., Vibashan, V., Jose, V. J. M., Wijethilake, N., Utkarsh, U., and Ren, H. Brain tumor segmentation and survival prediction using 3d attention unet. In Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries: 5th International Workshop, BrainLes 20...
2019
-
[10]
Multi-instance imbalance semantic segmentation by instance-dependent attention and adaptive hard instance mining
Jiang, W., Li, Y., Yi, Z., Chen, M., and Wang, J. Multi-instance imbalance semantic segmentation by instance-dependent attention and adaptive hard instance mining. Knowledge-Based Systems, 304: 0 112554, 2024
2024
-
[11]
and Salcudean, S
Karimi, D. and Salcudean, S. E. Reducing the hausdorff distance in medical image segmentation with convolutional neural networks. IEEE Transactions on medical imaging, 39 0 (2): 0 499--513, 2019
2019
-
[12]
Kervadec, H., Bouchtiba, J., Desrosiers, C., Granger, E., Dolz, J., and Ayed, I. B. Boundary loss for highly unbalanced segmentation. In International conference on medical imaging with deep learning, pp.\ 285--296. PMLR, 2019
2019
-
[13]
Kim, M. W. and Cho, N. I. Whfl: Wavelet-domain high frequency loss for sketch-to-image translation. In Proceedings of the IEEE/CVF Winter Conference on applications of computer vision, pp.\ 744--754, 2023
2023
-
[14]
B., Bhatia, H., Loehr, T., Piraud, M., et al
Kofler, F., Shit, S., Ezhov, I., Fidon, L., Horvath, I., Al-Maskari, R., Li, H. B., Bhatia, H., Loehr, T., Piraud, M., et al. Blob loss: Instance imbalance aware loss functions for semantic segmentation. In International Conference on Information Processing in Medical Imaging,...
2023
-
[15]
and Tekalp, A
Korkmaz, C. and Tekalp, A. M. Training transformer models by wavelet losses improves quantitative and visual performance in single image super-resolution. arXiv preprint arXiv:2404.11273, 2024
2024 arXiv
-
[16]
Feature pyramid networks for object detection
Lin, T.-Y., Doll \'a r, P., Girshick, R., He, K., Hariharan, B., and Belongie, S. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 2117--2125, 2017
2017
-
[17]
Enhancing boundary segmentation for topological accuracy with skeleton-based methods
Liu, C., Ma, B., Ban, X., Xie, Y., Wang, H., Xue, W., Ma, J., and Xu, K. Enhancing boundary segmentation for topological accuracy with skeleton-based methods. arXiv preprint arXiv:2404.18539, 2024
2024 arXiv
-
[18]
Boundary learning by using weighted propagation in convolution network
Liu, W., Chen, J., Liu, C., Ban, X., Ma, B., Wang, H., Xue, W., and Guo, Y. Boundary learning by using weighted propagation in convolution network. Journal of Computational Science, 62: 0 101709, 2022
2022
-
[19]
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., and Darrell, T. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 3431--3440, 2015
2015
-
[20]
Mallat, S. G. A theory for multiresolution signal decomposition: the wavelet representation. IEEE transactions on pattern analysis and machine intelligence, 11 0 (7): 0 674--693, 1989
1989
-
[21]
V-net: Fully convolutional neural networks for volumetric medical image segmentation
Milletari, F., Navab, N., and Ahmadi, S.-A. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision (3DV), pp.\ 565--571. Ieee, 2016
2016
-
[22]
Machine Learning for Aerial Image Labeling
Mnih, V. Machine Learning for Aerial Image Labeling. PhD thesis, University of Toronto, 2013
2013
-
[23]
Nunez-Iglesias, J., Kennedy, R., Parag, T., Shi, J., and Chklovskii, D. B. Machine learning of hierarchical clustering to segment 2d and 3d images. PloS one, 8 0 (8): 0 e71715, 2013
2013
-
[24]
L., Lee, M., Heinrich, M., Misawa, K., Mori, K., McDonagh, S., Hammerla, N
Oktay, O., Schlemper, J., Folgoc, L. L., Lee, M., Heinrich, M., Misawa, K., Mori, K., McDonagh, S., Hammerla, N. Y., Kainz, B., et al. Attention u-net: Learning where to look for the pancreas. arXiv preprint arXiv:1804.03999, 2018
2018 arXiv
-
[25]
Persistent homology with improved locality information for more effective delineation
Oner, D., Garin, A., Kozi \'n ski, M., Hess, K., and Fua, P. Persistent homology with improved locality information for more effective delineation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45 0 (8): 0 10588--10595, 2023
2023
-
[26]
and Simoncelli, E
Portilla, J. and Simoncelli, E. P. A parametric texture model based on joint statistics of complex wavelet coefficients. International journal of computer vision, 40: 0 49--70, 2000
2000
-
[27]
Wavelet-based loss for high-frequency interface dynamics
Prantl, L., Bender, J., Kugelstadt, T., and Thuerey, N. Wavelet-based loss for high-frequency interface dynamics. arXiv preprint arXiv:2209.02316, 2022
2022 arXiv
-
[28]
Rahman, M. A. and Wang, Y. Optimizing intersection-over-union in deep neural networks for image segmentation. In International symposium on visual computing, pp.\ 234--244. Springer, 2016
2016
-
[29]
U-net: Convolutional networks for biomedical image segmentation
Ronneberger, O., Fischer, P., and Brox, T. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention--MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, ...
2015
-
[30]
and Doll \'a r, G
Ross, T.-Y. and Doll \'a r, G. Focal loss for dense object detection. In proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 2980--2988, 2017
2017
-
[31]
Salehi, S. S. M., Erdogmus, D., and Gholipour, A. Tversky loss function for image segmentation using 3d fully convolutional deep networks. In International workshop on machine learning in medical imaging, pp.\ 379--387. Springer, 2017
2017
-
[32]
P., Wang, Z., Gupta, S., Bovik, A
Sampat, M. P., Wang, Z., Gupta, S., Bovik, A. C., and Markey, M. K. Complex wavelet structural similarity: A new image similarity index. IEEE transactions on image processing, 18 0 (11): 0 2385--2401, 2009
2009
-
[33]
C., Sekuboyina, A., Ezhov, I., Unger, A., Zhylka, A., Pluim, J
Shit, S., Paetzold, J. C., Sekuboyina, A., Ezhov, I., Unger, A., Zhylka, A., Pluim, J. P., Bauer, U., and Menze, B. H. cldice-a novel topology-preserving loss function for tubular structure segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern ...
2021
-
[34]
P., Freeman, W
Simoncelli, E. P., Freeman, W. T., Adelson, E. H., and Heeger, D. J. Shiftable multiscale transforms. IEEE transactions on Information Theory, 38 0 (2): 0 587--607, 1992
1992
-
[35]
P., Chen, H., Qi, X., Heng, P.-A., Guo, Y
Sirinukunwattana, K., Pluim, J. P., Chen, H., Qi, X., Heng, P.-A., Guo, Y. B., Wang, L. Y., Matuszewski, B. J., Bruni, E., Sanchez, U., et al. Gland segmentation in colon histology images: The glas challenge contest. Medical image analysis, 35: 0 489--502, 2017
2017
-
[36]
D., Niemeijer, M., Viergever, M
Staal, J., Abr \`a moff, M. D., Niemeijer, M., Viergever, M. A., and Van Ginneken, B. Ridge-based vessel segmentation in color images of the retina. IEEE transactions on medical imaging, 23 0 (4): 0 501--509, 2004
2004
-
[37]
C., Shit, S., Menze, B., and Bauer, U
Stucki, N., Paetzold, J. C., Shit, S., Menze, B., and Bauer, U. Topologically faithful image segmentation via induced matching of persistence barcodes. In International Conference on Machine Learning, pp.\ 32698--32727. PMLR, 2023
2023
-
[38]
X., Epps, J., and Bailey, J
Vinh, N. X., Epps, J., and Bailey, J. Information theoretic measures for clusterings comparison: is a correction for chance necessary? In Proceedings of the 26th annual international conference on machine learning, pp.\ 1073--1080, 2009
2009
-
[39]
C., Sheikh, H
Wang, Z., Bovik, A. C., Sheikh, H. R., and Simoncelli, E. P. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13 0 (4): 0 600--612, 2004
2004
-
[40]
Woo, S., Park, J., Lee, J.-Y., and Kweon, I. S. Cbam: Convolutional block attention module. In Proceedings of the European conference on computer vision (ECCV), pp.\ 3--19, 2018
2018
-
[41]
H., and Tsai, Y.-C
Yang, H.-H., Yang, C.-H. H., and Tsai, Y.-C. J. Y-net: Multi-scale feature aggregation network with wavelet structure similarity loss function for single image dehazing. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\...
2020
-
[42]
Region mutual information loss for semantic segmentation
Zhao, S., Wang, Y., Yang, Z., and Cai, D. Region mutual information loss for semantic segmentation. Advances in Neural Information Processing Systems, 32, 2019
2019
-
[43]
M., Tajbakhsh, N., and Liang, J
Zhou, Z., Rahman Siddiquee, M. M., Tajbakhsh, N., and Liang, J. Unet++: A nested u-net architecture for medical image segmentation. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support: 4th International Workshop, DLMIA 2018, and 8th...
2018
-
[44]
Wavelet loss function for auto-encoder
Zhu, Q., Wang, H., and Zhang, R. Wavelet loss function for auto-encoder. IEEE Access, 9: 0 27101--27108, 2021
2021
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.