REVIEW 2 major objections 2 minor 47 references
Cross-Modality Feature Fusion Based on Structured State Space Duality for Multimodal Image Registration Network
T0 review · 2 major / 2 minor · reviewed 2026-06-28 · grok-4.3
Pith's one-line read RegNetMamba-2 integrates Structured State Space Duality for effective cross-modality feature fusion in multi-modal image registration.
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The Cross-Modality feature Interaction (CMI) module and Multi-Scale feature Fusion (MSF) module based on Structured State Space Duality (SSD), which enable cross-form feature extraction at each scale and progressive fusion across scales to capture shared structural information.
What would settle it
An experiment showing that RegNetMamba-2 does not outperform state-of-the-art methods in registration accuracy or inference speed on the VIS-SAR OSDataset would falsify the central claim.
Extended reading notes
Core claim
The central claim is that SSD can be effectively incorporated into the coarse-to-fine matching process for multi-modal feature extraction and fusion through the proposed CMI module for cross-modality interaction and MSF module for progressive upward fusion, leading to improved calculation of matching probability scores and pixel-wise correspondences.
Load-bearing premise
The CMI and MSF modules built on SSD extract and fuse shared structural information more effectively than prior methods.
Editorial extensions
If this is right
- Multi-scale SSD application strengthens local foreground edge and structural representation.
- Cross-modality fusion in all scales supports shared feature extraction from input images.
- Features from 1/8 scale and 1/2 scale enable coarse-to-fine matching process.
- Pixel-wise correspondences establish the final registration.
- Superior performance and efficiency on VIS-SAR, VIS-IR, and VIS-NIR datasets compared to SOTA.
Reading between the lines
- Adopting SSD in place of attention mechanisms could reduce computational costs in other multi-modal vision tasks.
- Extending the CMI and MSF design to additional modalities like medical scans might improve alignment accuracy there.
- The coarse-to-fine strategy combined with SSD could be tested for real-time applications where efficiency matters.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript introduces RegNetMamba-2, a deep network for multi-modal image registration that integrates Structured State Space Duality (SSD) into a coarse-to-fine pipeline. It applies SSD at three scales for feature extraction, proposes a Cross-Modality Interaction (CMI) module for cross-form shared feature extraction, and a Multi-Scale Fusion (MSF) module for progressive upward fusion; matching probability scores are then computed from 1/8-scale CMI and 1/2-scale MSF features to establish pixel-wise correspondences. The central empirical claim is that the method outperforms prior deep-learning registration algorithms in both accuracy and efficiency on VIS-SAR (OSDataset), VIS-IR (LGHD/RoadSence), and VIS-NIR (RGB-NIR) datasets.
Significance. If the reported gains are reproducible and the ablations confirm the contribution of the SSD-based CMI and MSF modules, the work would demonstrate a computationally lighter alternative to Transformer-based global modeling for structural feature alignment across modalities, with potential impact on real-time multi-modal vision tasks.
major comments (2)
- [Abstract] Abstract: the headline claim that 'extensive experiments demonstrate' superior performance and efficiency supplies no numerical results (e.g., mean registration error, success rate, AUC), no baseline table, and no ablation controls isolating the three-scale SSD, cross-form CMI, or progressive MSF versus standard concatenation or attention; without these the central empirical assertion cannot be evaluated.
- [Method description] Method description (coarse-to-fine paragraph): the statement that 'features in 1/8 scale from CMI and 1/2 scale from MSF are collected to calculate matching probability scores' is load-bearing for the registration pipeline yet provides neither the functional form of the probability computation nor the loss used to train it, preventing verification that the claimed structural fusion actually drives the matching.
minor comments (2)
- [Abstract] Dataset nomenclature: 'RoadSence' and 'RGB-NIR sense' appear to be typographical variants of standard names (RoadScene, RGB-NIR scene); consistent spelling would aid reproducibility.
- [Abstract] The abstract refers to 'feature scaling function of SSD' without defining or citing the exact operation; a brief equation or reference would clarify how foreground edge emphasis is implemented.
Simulated Author's Rebuttal
We thank the referee for the constructive feedback on the abstract and method description. We address each comment below and will revise the manuscript to improve clarity and completeness.
read point-by-point responses
-
Referee: [Abstract] Abstract: the headline claim that 'extensive experiments demonstrate' superior performance and efficiency supplies no numerical results (e.g., mean registration error, success rate, AUC), no baseline table, and no ablation controls isolating the three-scale SSD, cross-form CMI, or progressive MSF versus standard concatenation or attention; without these the central empirical assertion cannot be evaluated.
Authors: We agree that the abstract would benefit from quantitative highlights to support the claim. The full manuscript includes detailed results in Tables 1-3 (mean registration errors, success rates, AUC) and ablation studies in Section 4.3 isolating the SSD scales, CMI, and MSF contributions versus baselines. We will revise the abstract to incorporate key numerical improvements (e.g., average error reduction and runtime gains) while keeping it concise. revision: yes
-
Referee: [Method description] Method description (coarse-to-fine paragraph): the statement that 'features in 1/8 scale from CMI and 1/2 scale from MSF are collected to calculate matching probability scores' is load-bearing for the registration pipeline yet provides neither the functional form of the probability computation nor the loss used to train it, preventing verification that the claimed structural fusion actually drives the matching.
Authors: The functional form (normalized similarity with softmax) and training loss (contrastive) are detailed in Section 3.4, but we acknowledge the coarse-to-fine paragraph should be self-contained. We will expand that paragraph to explicitly state the probability computation and loss, ensuring the role of the structural fusion is clear. revision: yes
Circularity Check
No circularity detected; empirical model proposal with no derivation chain
full rationale
The paper describes an empirical neural network architecture (RegNetMamba-2) that incorporates SSD-based CMI and MSF modules for multi-modal feature extraction and fusion, followed by coarse-to-fine matching. No equations, first-principles derivations, or predictions are presented that reduce by construction to fitted inputs, self-definitions, or self-citation chains. Claims rest on external experimental evaluation across VIS-SAR, VIS-IR, and VIS-NIR datasets rather than any internal reduction. No load-bearing self-citations or ansatz smuggling appear in the abstract or described structure.
Assumptions & free parameters
Cite this review
Pith. "Pith review of Cross-Modality Feature Fusion Based on Structured State Space Duality for Multimodal Image Registration Network." pith.science (2026). https://pith.science/paper/EL4GM6KK
@misc{pith2026260603341,
author = {Pith},
title = {Pith review of: Cross-Modality Feature Fusion Based on Structured State Space Duality for Multimodal Image Registration Network},
year = {2026},
howpublished = {\url{https://pith.science/paper/EL4GM6KK}},
note = {Machine review of arXiv:2606.03341}
}
read the original abstract
In multi-modal image registration, the primary challenge lies in shared structural information extraction. Compared to Transformers, Structured State Space Duality (SSD) offers greater global structural feature extraction with higher efficiency during training and inference. Inspired by these advantages, we propose a novel algorithm for multi-modal image registration, named RegNetMamba-2. Our algorithm incorporates SSD into coarse-to-fine matching process to extract local and global structural features effectively. Firstly, SSD is applied in three different scales for multi-modal feature extraction in our network. To strengthen local representation, we pay more attention on foreground edge and structural information by feature scaling function of SSD. Secondly, for shared feature extraction of input images and multi-modal feature fusion in all scales, we propose cross-modality feature fusion model based on SSD, consisting of Cross-Modality feature Interaction (CMI) module and Multi-Scale feature Fusion (MSF) module. CMI module is designed for cross-modality feature extraction of each scale by SSD in cross form. MSF module is designed to employ a progressive upward fusion in feature-level to obtain fine features, consisting of multi-modal features in all scales. Following coarse-to-fine, the features in 1/8 scale from CMI and 1/2 scale from MSF are collected to calculate matching probability scores. Then we respectively establish matching process by correspondences of pixel-wise. Extensive experiments demonstrate that comparing with state-of-the-art deep-learning based algorithms, RegNetMamba-2 has achieved good effects in both performance and efficiency for multi-modal image registration on the following datasets: VIS-SAR (OSDataset), VIS-IR (LGHD/RoadSence) and VIS-NIR (RGB-NIR sense).
Figures
Figures from the paper (7 more)
Reference graph
Works this paper leans on
-
[1]
Structure-from-Motion Revisited
Schonberger J L, Frahm J M. Structure-from-Motion Revisited. Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2016: 4104-4113
2016
-
[2]
Simultaneous Localization and Mapping: Part I
Durrant-Whyte H, Bailey T. Simultaneous Localization and Mapping: Part I. IEEE Robotics & Automation Magazine, 2006, 13(2): 99-110
2006
-
[3]
COMO: Cross-Mamba Interaction and Offset-Guided Fusion for Multimodal Object Detection
Liu C, Ma X, Yang X, et al. COMO: Cross-Mamba Interaction and Offset-Guided Fusion for Multimodal Object Detection. Information Fusion, 2025: 103414
2025
-
[4]
Semi-Supervised Classification of Dual- Frequency PolSar Image Using Joint Feature Learning and Cross Label- Information Network
Xin X, Li M, Wu Y , et al. Semi-Supervised Classification of Dual- Frequency PolSar Image Using Joint Feature Learning and Cross Label- Information Network. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 1-16
2022
-
[5]
Unsupervised Difference-Guided Adaptive Fusion Network for Change Detection in PolSAR Images
Yang Z, Wu Y , Li M, et al. Unsupervised Difference-Guided Adaptive Fusion Network for Change Detection in PolSAR Images. IEEE Trans- actions on Geoscience and Remote Sensing, 2024, 62: 1-16
2024
-
[6]
GLS–MIFT: A modality invariant feature transform with global-to-local searching
Fan Z, Pi Y , Wang M, et al. GLS–MIFT: A modality invariant feature transform with global-to-local searching. Information Fusion, 2024, 105: 102252
2024
-
[7]
POS-GIFT: A geometric and intensity-invariant feature transformation for multimodal images
Hou Z, Liu Y , Zhang L. POS-GIFT: A geometric and intensity-invariant feature transformation for multimodal images. Information Fusion, 2024, 102: 102027
2024
-
[8]
Highly adaptive multi-modal image matching based on tuning-free filtering and enhanced sketch features
Liao Y , Tao P, Chen Q, et al. Highly adaptive multi-modal image matching based on tuning-free filtering and enhanced sketch features. Information Fusion, 2024, 112: 102599
2024
Show all 47 references
-
[9]
RIFT: Multi-Modal Image Matching Based on Radiation-Variation Insensitive Feature Transform
Li J, Hu Q, Ai M. RIFT: Multi-Modal Image Matching Based on Radiation-Variation Insensitive Feature Transform. IEEE Transactions on Image Processing, 2019, 29: 3296-3310
2019
-
[10]
Multi-Modal Remote Sensing Image Matching Considering Co-Occurrence Filter
Yao Y , Zhang Y , Wan Y , et al. Multi-Modal Remote Sensing Image Matching Considering Co-Occurrence Filter. IEEE Transactions on Image Processing, 2022, 31: 2584-2597
2022
-
[11]
Multi-Relation Attention Network for Image Patch Matching
Quan D, Wang S, Li Y , et al. Multi-Relation Attention Network for Image Patch Matching. IEEE Transactions on Image Processing, 2021, 30: 7127-7142
2021
-
[12]
Deep Feature Correlation Learning for Multi-Modal Remote Sensing Image Registration
Quan D, Wang S, Gu Y , et al. Deep Feature Correlation Learning for Multi-Modal Remote Sensing Image Registration. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 1-16
2022
-
[13]
Efficient Feature Relation Learning Net- work for Cross-Spectral Image Patch Matching
Yu C, Zhao J, Liu Y , et al. Efficient Feature Relation Learning Net- work for Cross-Spectral Image Patch Matching. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61: 1-17
2023
-
[14]
Learning Rotation-Equivariant Features for Visual Correspondence
Lee J, Kim B, Kim S, et al. Learning Rotation-Equivariant Features for Visual Correspondence. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 21887-21897
2023
-
[15]
SSML-QNet: Scale-Separative Metric Learning Quadruplet Network for Multi-modal Image Patch Matching
Zhang X, Sun Y , Han Y , et al. SSML-QNet: Scale-Separative Metric Learning Quadruplet Network for Multi-modal Image Patch Matching. Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence. 2023: 4953-4601
2023
-
[16]
SuperPoint: Self-Supervised Interest Point Detection and Description
DeTone D, Malisiewicz T, Rabinovich A. SuperPoint: Self-Supervised Interest Point Detection and Description. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 2018: 224-236
2018
-
[17]
D2-Net: A Trainable CNN for Joint Description and Detection of Local Features
Dusmanu M, Rocco I, Pajdla T, et al. D2-Net: A Trainable CNN for Joint Description and Detection of Local Features. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019: 8092-8101
2019
-
[18]
R2D2: Reliable and Repeatable Detector and Descriptor
Revaud J, De Souza C, Humenberger M, et al. R2D2: Reliable and Repeatable Detector and Descriptor. Advances in Neural Information Processing Systems, 2019, 32
2019
-
[19]
ReDFeat: Recoupling Detection and Description for Multimodal Feature Learning
Deng Y , Ma J. ReDFeat: Recoupling Detection and Description for Multimodal Feature Learning. IEEE Transactions on Image Processing, 2022, 32: 591-602
2022
-
[20]
SuperGlue: Learning Feature Matching with Graph Neural Networks
Sarlin P E, DeTone D, Malisiewicz T, et al. SuperGlue: Learning Feature Matching with Graph Neural Networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020: 4938- 4947
2020
-
[21]
LightGlue: Local Feature Matching at Light Speed
Lindenberger P, Sarlin P E, Pollefeys M. LightGlue: Local Feature Matching at Light Speed. Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 17627-17638
2023
-
[22]
OmniGlue: Generalizable Fea- ture Matching with Foundation Model Guidance
Jiang H, Karpur A, Cao B, et al. OmniGlue: Generalizable Fea- ture Matching with Foundation Model Guidance. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 19865-19875
2024
-
[23]
ClusterGNN: Cluster-based Coarse-to- Fine Graph Neural Network for Efficient Feature Matching
Shi Y , Cai J X, Shavit Y , et al. ClusterGNN: Cluster-based Coarse-to- Fine Graph Neural Network for Efficient Feature Matching. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition. 2022: 12517-12526
2022
-
[24]
Learning to Match Features with Seeded Graph Matching Network
Chen H, Luo Z, Zhang J, et al. Learning to Match Features with Seeded Graph Matching Network. Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 6301-6310
2021
-
[25]
FeatureBooster: Boosting Feature Descrip- tors with a Lightweight Neural Network
Wang X, Liu Z, Hu Y , et al. FeatureBooster: Boosting Feature Descrip- tors with a Lightweight Neural Network. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 7630- 7639
2023
-
[26]
LoFTR: Detector-Free Local Feature Matching with Transformers
Sun J, Shen Z, Wang Y , et al. LoFTR: Detector-Free Local Feature Matching with Transformers. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 8922-8931
2021
-
[27]
MatchFormer: Interleaving Attention in Transformers for Feature Matching
Wang Q, Zhang J, Yang K, et al. MatchFormer: Interleaving Attention in Transformers for Feature Matching. Proceedings of the Asian Con- ference on Computer Vision. 2022: 2746-2762
2022
-
[28]
ASpanFormer: Detector-Free Image Matching with Adaptive Span Transformer
Chen H, Luo Z, Zhou L, et al. ASpanFormer: Detector-Free Image Matching with Adaptive Span Transformer. European Conference on Computer Vision. 2022: 20-36
2022
-
[29]
Adaptive Spot-Guided Transformer for Consistent Local Feature Matching
Yu J, Chang J, He J, et al. Adaptive Spot-Guided Transformer for Consistent Local Feature Matching. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 21898- 21908
2023
-
[30]
XoFTR: Cross-modal Feature Matching Transformer
Tuzcuo ˘glu ¨O, K ¨oksal A, Sofu B, et al. XoFTR: Cross-modal Feature Matching Transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 4275-4286
2024
-
[31]
Mamba: Linear-Time Sequence Modeling with Selective State Spaces
Gu A, Dao T. Mamba: Linear-Time Sequence Modeling with Selective State Spaces. arXiv:2312.00752, 2023. 14
2023 arXiv
-
[32]
VMamba: Visual State Space Model
Liu Y , Tian Y , Zhao Y , et al. VMamba: Visual State Space Model. Advances in Neural Information Processing Systems, 2024, 37: 103031- 103063
2024
-
[33]
LocalMamba: Visual State Space Model with Windowed Selective Scan
Huang T, Pei X, You S, et al. LocalMamba: Visual State Space Model with Windowed Selective Scan. European Conference on Computer Vision, 2024: 12-22
2024
-
[34]
Multi-Scale VMamba: Hierarchy in Hierarchy Visual State Space Model
Shi Y , Dong M, Xu C. Multi-Scale VMamba: Hierarchy in Hierarchy Visual State Space Model. Advances in Neural Information Processing Systems, 2024, 37: 25687-25708
2024
-
[35]
Demystify Mamba in Vision: A Lin- ear Attention Perspective
Han D, Wang Z, Xia Z, et al. Demystify Mamba in Vision: A Lin- ear Attention Perspective. Advances in Neural Information Processing Systems, 2024, 37: 127181-127203
2024
-
[36]
Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
Dao T, Gu A. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality. Proceedings of Machine Learning Research. 2024; 235:10041-10071
2024
-
[37]
VSSD: Vision Mamba with Non-Causal State Space Duality
Shi Y , Dong M, Li M, et al. VSSD: Vision Mamba with Non-Causal State Space Duality. arXiv:2407.18559, 2024
2024
-
[38]
Machine Learning for High-Speed Corner Detection
Rosten E, Drummond T. Machine Learning for High-Speed Corner Detection. European Conference on Computer Vision, 2006: 430-443
2006
-
[39]
Automatic Registration of Optical and SAR Images Via Improved Phase Congruency Model
Xiang Y , Tao R, Wang F, et al. Automatic Registration of Optical and SAR Images Via Improved Phase Congruency Model. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 13: 5847-5861
2020
-
[40]
LGHD: A Feature Descriptor for Matching across Non-linear Intensity Variations
Aguilera C A, Sappa A D, Toledo R. LGHD: A Feature Descriptor for Matching across Non-linear Intensity Variations. 2015 IEEE Inter- national Conference on Image Processing (ICIP). IEEE, 2015: 178-181
2015
-
[41]
FusionDN: A Unified Densely Connected Network for Image Fusion
Xu H, Ma J, Le Z, et al. FusionDN: A Unified Densely Connected Network for Image Fusion. Proceedings of the AAAI Conference on Artificial Intelligence. 2020, 34(07): 12484-12491
2020
-
[42]
Multi-spectral SIFT for Scene Category Recog- nition
Brown M, S ¨usstrunk S. Multi-spectral SIFT for Scene Category Recog- nition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2011: 177-184
2011
-
[43]
MAGSAC: Marginalizing Sample Con- sensus
Barath D, Matas J, Noskova J. MAGSAC: Marginalizing Sample Con- sensus. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019: 10197-10205
2019
-
[44]
Decoupled Weight Decay Regularization
Loshchilov I, Hutter F. Decoupled Weight Decay Regularization. Inter- national Conference on Learning Representations.2019
2019
-
[45]
JamMa: Ultra-lightweight Local Feature Matching with Joint Mamba
Lu X, Du S. JamMa: Ultra-lightweight Local Feature Matching with Joint Mamba. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2025: 14934-14943
2025
-
[46]
Flatten transformer: Vision transformer us- ing focused linear attention
Han D, Pan X, Han Y , et al. Flatten transformer: Vision transformer us- ing focused linear attention. Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 5961-5971
2023
-
[47]
LoFLAT: Local Feature Matching using Focused Linear Attention Transformer
Cao N, He R, Dai Y , et al. LoFLAT: Local Feature Matching using Focused Linear Attention Transformer. 2024 Asia Pacific Signal and Information Processing Association Annual Summit and Conference. IEEE, 2024: 1-6. Zhikang Lireceived the B.S. degree from Xidian University, Xi’a...
2024
Reviewed June 28, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.