Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

4D Multimodal Co-attention Fusion Network with Latent Contrastive Alignment for Alzheimer's Diagnosis

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read M2M-AlignNet claims that a geometry-aware, many-to-many patch contrastive loss plus a latent-query co-attention module fuses 4D fMRI and 3D sMRI to outperform existing multimodal methods for Alzheimer's diagnosis.

desk verdict Solid engineering, overstated novelty: the M2M loss in Eq. 6 is a weighted InfoNCE, not the many-to-many geometry-aware alignment the paper advertises. read the letter →

arxiv 2504.16798 v1 pith:2AZVNCLS submitted 2025-04-23 cs.MM cs.CVcs.LG

Yuxiang Wei , Yanteng Zhang , Xi Xiao , Tianyang Wang , Xiao Wang , Vince D. Calhoun This is my paper · ORCID
classification cs.MMcs.CVcs.LG
keywords Alzheimer'sdiseasemultimodalfusionfMRIscontrastivelearningco-attentionlatentalignmentM2M-AlignNet
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that 4D fMRI and 3D sMRI can be fused for early Alzheimer's diagnosis by aligning many patches across modalities in a shared latent space, rather than forcing one-to-one correspondences or letting one modality dominate. The proposed M2M-AlignNet couples a patch-wise contrastive loss with adaptive self-weighting of negative pairs to a co-attention module that uses learnable latent queries instead of a fixed modality-as-query design. On three imaging cohorts, it reports higher PR-AUC and ROC-AUC than comparison multimodal methods, with the largest gains on presymptomatic risk classification. If correct, it would give clinicians a practical way to exploit complementary structural and functional MRI signals for earlier detection.

What carries the argument

The central mechanism is the M2M contrastive loss in Eq. 6: a reweighted InfoNCE-style loss computed patch-wise over the fMRI-sMRI similarity matrix $S_t$. Each positive pair contributes a numerator $\exp(S_{t,(i,j)}/\tau)$, while the denominator sums over negative partners $k \neq i$ with weights $w_{t,(i,k)} = T(D(L_f^{(t,i)}, L_s^k))$, where $D$ is a divergence or similarity measure and $T$ is a negative-correlation function. The claimed effect is that patches with similar semantics are penalized less as negatives, allowing one patch to align with several patches across modalities. The co-attention module uses a learnable latent query $J_{\mathrm{sp}}$ initialized from a normal distribution, attending over concatenated modality keys and values, followed by modality-specific refinement with a down-up bottleneck.

What would settle it

Compute the gradient of Eq. 6 with respect to an off-diagonal pair: if off-diagonal pairs never receive a positive pull, the many-to-many claim is false. Then retrain with several off-diagonal pairs designated as positives and check whether the reported PR-AUC and ROC-AUC improve.

Watch

Extended reading notes

Core claim

The paper's central claim is that a multi-patch-to-multi-patch (M2M) contrastive loss, with per-pair weights derived from the discrepancy between fMRI and sMRI latents, creates a geometry-aware alignment that improves multimodal Alzheimer's diagnosis. The authors argue this goes beyond CLIP-style pairwise alignment and beyond co-attention with a fixed modality hierarchy. The latent-as-query co-attention fuses sMRI, fMRI spatial, and fMRI temporal features through learnable query vectors, and a bottleneck refinement module suppresses redundancy. On the presymptomatic at-risk cohort, the full model reaches PR-AUC 64.49 and ROC-AUC 71.55 with sMRI and fMRI alone; adding tabular biomarker features raises PR-AUC to 67.66 and ROC-AUC to 73.46. Ablations show that removing the contrastive alignment or the spatial co-attention fusion causes the largest performance drops.

Load-bearing premise

The load-bearing premise is that Eq. 6 implements multi-patch-to-multi-patch alignment; if it is actually a weighted one-to-one contrastive loss with only diagonal positives, the named 'M2M' mechanism is not what drives the reported gains.

Editorial extensions

If this is right

  • If the central claim is correct, early Alzheimer's risk can be detected from resting fMRI and sMRI with PR-AUC 64.49 and ROC-AUC 71.55 on the presymptomatic at-risk cohort, beating all listed baselines on ROC-AUC.
  • Adding tabular biomarker features raises PR-AUC to 67.66 and ROC-AUC to 73.46, indicating that imaging and blood-based biomarkers can be combined within the same fusion architecture.
  • Ablating either the contrastive alignment or the spatial co-attention fusion causes the largest performance drops, so both modules are load-bearing for the reported gains.
  • The choice of discrepancy measure for the self-weighting is decisive: dot-product weighting gives the best results while Jensen-Shannon divergence performs worst.
  • The model is relatively lightweight (1.24M parameters) compared to the strongest baseline (35.82M parameters) while achieving higher ROC-AUC, which may matter for clinical deployment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • As written, Eq. 6 keeps one positive pair per row; the adaptive weight only down-weights negative-pair denominators. A true many-to-many positive alignment would require several sMRI patches to receive positive gradients from one fMRI patch, which the equations do not show.
  • The 'geometry' in 'geometry-aware' does not enter via spatial distances; $D$ is dot product, cosine similarity, KL divergence, JSD, or MMD on embeddings. Weighting by actual anatomical distance between patches would be a direct test of the geometric claim the name implies.
  • The paper notes the loss extends to more than two modalities by summing over modality pairs; a natural test is adding PET or DTI and checking whether the alignment gains persist or saturate.
  • A decisive experiment is to replace the self-weighting in Eq. 6 with a standard InfoNCE denominator and hold everything else fixed; if the reported performance does not drop, the M2M mechanism is not what drives the results.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes M2M-AlignNet, a multimodal framework for Alzheimer's disease diagnosis that fuses 4D fMRI, 3D sMRI, and tabular data. The method uses a SwiFT 4D Transformer backbone, a latent-as-query co-attention fusion module with bottleneck refinement, and a contrastive alignment loss that the authors claim performs multi-patch-to-multi-patch (M2M) alignment between fMRI and sMRI patches via geometry-weighted self-weighting. Experiments are reported on the EHBS, ADNI, and HCP datasets, with comparisons to several multimodal baselines, ablated modules, and qualitative visualizations of discriminative brain regions and latent embeddings.

Significance. If the central claims were supported, the paper would address an underexplored and clinically relevant fusion setting, namely combining 4D spatiotemporal fMRI with 3D structural MRI for early AD diagnosis. The architecture-level ideas, including latent queries for modality-agnostic fusion and bottleneck-based refinement, are reasonable, and the multi-dataset evaluation is a strength. The paper also engages with a clinically meaningful question about function-structure correspondence. However, the central mechanistic claim about the M2M loss is not supported by the equations, and the experimental evidence lacks significance testing and contains high-variance ablation results. No code or reproducibility artifacts are provided, which further limits verification.

major comments (4)
  1. [Sec. III-D, Eq. (6)] The loss in Eq. (6) does not implement multi-patch-to-multi-patch alignment. For a fixed fMRI row i, the numerator contains exactly one positive pair exp(S_{t,(i,j)}/tau), and the self-weight w_{t,(i,k)} multiplies only the negative-pair terms in the denominator (k != i). Thus each fMRI patch is attracted to exactly one sMRI patch, and the weighting merely down-weights some negatives. The text's claim that the method 'allows correspondence between a single patch from the fMRI embedding and multiple patches from the sMRI embedding, and vice versa' is not realized by the equation. Moreover, for any off-diagonal positive pair (i != j), the indicator 1_{k != i} does not exclude k = j, so the same pair is simultaneously attracted in the numerator and repelled in the denominator. This is a load-bearing error because the M2M loss is the paper's core contribution.
  2. [Sec. III-D, Eq. (7)] The 'geometry-weighted' and 'distance-adjusted' characterization is not supported. D(.) in Eq. (7) is a dot product, cosine similarity, KL divergence, JSD, or MMD computed on latent embeddings, not a spatial distance between brain regions. The weighting therefore reflects embedding similarity, not brain geometry. Additionally, the negative correlation function T is never defined, so the mapping from D to w is unspecified. This affects the framing of the method and the claimed novelty of geometry-aware alignment.
  3. [Table I] No statistical significance tests are reported, and several differences are within one standard deviation of the mean. For example, on EHBS PR-AUC the proposed method is 64.49 +/- 3.9 versus mmFormer at 64.06 +/- 5.8, and on HCP PR-AUC it is 97.59 +/- 1.2 versus mmFormer at 96.62 +/- 1.7. The phrase 'significantly outperforms' is not justified. Also, on EHBS accuracy the proposed method (78.01) is lower than SwiFT-EF (79.74), and this is not discussed. Paired tests across folds or bootstrap tests should be reported for all metrics.
  4. [Table II] The ablation table is difficult to interpret. The header lists modules, but the rows are not labeled with which module is removed, and several accuracy values have very large standard deviations (e.g., 54.36 +/- 31.8 and 83.72 +/- 11.2). The claim that removing spatial fusion or alignment causes the most substantial performance degradation needs a clearly labeled table and a statistical comparison; as presented, the evidence is inconclusive.
minor comments (6)
  1. [Abstract and Introduction] The word 'correspondance' is misspelled as 'correspondence' in the abstract and the introduction; the typo should be corrected.
  2. [Sec. III-A] The section heading contains a typo: 'Proprocessing' should be 'Preprocessing'.
  3. [Sec. III-D] The function T in Eq. (7) is left unspecified. The authors should state explicitly what T is and how the weights w are normalized (for example, whether a softmax is applied over k).
  4. [Sec. IV-A] Several hyperparameters are not reported, including the temperature tau, the relative weight of the M2M contrastive loss versus the classification loss, the number of latent co-attention queries, and the bottleneck dimensions. These are needed for reproducibility.
  5. [Sec. IV-E] The phrase 'to verify the reliability and interpretability of the proposed DRL' uses the undefined abbreviation 'DRL'; this should be replaced with the model name.
  6. [Sec. IV-F and Fig. 5] The claim that distributions appearing nearly 'orthogonal' to each other indicate 'effective alignment' is conceptually confusing, since alignment in a shared space should decrease distributional distance. The interpretation should be clarified.

Circularity Check

0 steps flagged · score 2.0 of 10

No load-bearing circularity: reported results are empirical; the M2M loss's many-to-many characterization is an overclaim, not a circular derivation.

full rationale

The paper's reported metrics (Table I) come from a self-contained five-fold cross-validation loop on external benchmarks (EHBS, ADNI, HCP); no final accuracy or AUC is obtained by fitting a parameter to that same metric, so no prediction reduces to a fitted input. The M2M loss (Eq. 6) is not equivalent to the paper's claimed 'multi-patch-to-multi-patch' alignment: for a fixed fMRI row the numerator still has exactly one positive pair, and the self-weight w appears only on negative-denominator terms; the paper itself says it 'loosed the constraint by introducing the adaptive self-weighting for the negative pairs.' That is a mismatch between claim and equation (Eq. 6 is a weighted InfoNCE over diagonal positives), not a circular derivation, because the many-to-many property is not assumed as input to derive Eq. 6. The self-citations (Wei et al. 2024; Khalilullah et al. 2023, sharing a senior author) support interpretability and design motivation but are not load-bearing for the reported classification gains, and no uniqueness theorem is imported to forbid alternatives. Consequently, the derivation chain is self-contained; the only mild issue is self-referential interpretive support, reflected in the score of 2.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central mechanism relies on standard contrastive learning and attention modules; no new physical entities are introduced. The hidden choices that matter are the temperature tau, the unstated relative weight of the M2M loss against the classification loss, and the adoption of a biomarker-ratio threshold as ground truth for EHBS. The premise that flattened fMRI and sMRI patches are comparable C-dimensional units is also assumed without geometric grounding.

free parameters (4)
  • Contrastive temperature tau = not reported
    In Eq. 5 and 6, tau controls the softmax sharpness; the paper never states its value or tuning procedure.
  • Relative weight of M2M loss vs classification loss = not reported
    The total training objective is not written; the paper does not state how much the contrastive alignment is weighted relative to the diagnostic classifier.
  • Channel dimensions per SwiFT stage = [24, 48, 96, 96]
    Section III-B fixes these channel counts by hand; no justification from data is provided.
  • Number of latent co-attention queries = not reported
    Jsp and Jte dimensions are not specified beyond initializing from a normal distribution.
assumptions (4)
  • standard math InfoNCE and CLIP-style softmax contrastive losses are a valid mechanism for cross-modal alignment.
    Eq. 5-6 are built directly on the InfoNCE and CLIP objectives cited from Oord et al. and Radford et al.
  • domain assumption The tTau/Aβ42 ratio threshold of 0.24 defines pre-symptomatic AD risk in EHBS.
    Section III-A assigns labels using Tandon et al.'s threshold without validating it on this specific cohort or adjusting for confounders.
  • domain assumption Flattened fMRI patches at each time point and flattened sMRI patches are comparable C-dimensional latent units.
    Eq. 4 computes dot products between these patches as if their dimensions correspond, despite the physical difference between spatiotemporal activity and static structure.
  • domain assumption A 4D Swin Transformer pretrained for fMRI transfers to sMRI by adding a temporal dimension.
    Section III-B reuses SwiFT for sMRI without describing pretraining, fine-tuning, or evidence that the temporal axis is harmless for static structure.

how reviews work

0 comments
Cite this review

Pith. "Pith review of 4D Multimodal Co-attention Fusion Network with Latent Contrastive Alignment for Alzheimer's Diagnosis." pith.science (2026). https://pith.science/paper/2AZVNCLS

@misc{pith2026250416798,
  author       = {Pith},
  title        = {Pith review of: 4D Multimodal Co-attention Fusion Network with Latent Contrastive Alignment for Alzheimer's Diagnosis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2AZVNCLS}},
  note         = {Machine review of arXiv:2504.16798}
}
read the original abstract

Multimodal neuroimaging provides complementary structural and functional insights into both human brain organization and disease-related dynamics. Recent studies demonstrate enhanced diagnostic sensitivity for Alzheimer's disease (AD) through synergistic integration of neuroimaging data (e.g., sMRI, fMRI) with behavioral cognitive scores tabular data biomarkers. However, the intrinsic heterogeneity across modalities (e.g., 4D spatiotemporal fMRI dynamics vs. 3D anatomical sMRI structure) presents critical challenges for discriminative feature fusion. To bridge this gap, we propose M2M-AlignNet: a geometry-aware multimodal co-attention network with latent alignment for early AD diagnosis using sMRI and fMRI. At the core of our approach is a multi-patch-to-multi-patch (M2M) contrastive loss function that quantifies and reduces representational discrepancies via geometry-weighted patch correspondence, explicitly aligning fMRI components across brain regions with their sMRI structural substrates without one-to-one constraints. Additionally, we propose a latent-as-query co-attention module to autonomously discover fusion patterns, circumventing modality prioritization biases while minimizing feature redundancy. We conduct extensive experiments to confirm the effectiveness of our method and highlight the correspondance between fMRI and sMRI as AD biomarkers.

Figures

Figures reproduced from arXiv: 2504.16798 by the authors.

Figure 1
Figure 1. M2M-AlignNet: Modalities are first encoded by the corresponding modality-specific encoders, then fused via co-attention. fMRI and sMRI [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. M2M contrastive loss to align pairs of fMRI patches with sMRI [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Visualizations of the key brain regions contribute to the framework. We compute the spatial co-attention scores for fMRI and sMRI, map them onto [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Visualizations of the top 3 brain states contributing to the diagnosis. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: t-SNE visualizations of fMRI and sMRI embeddings in the latent [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. BrainMAP: Multimodal Graph Learning For Efficient Brain Disease Localization

    cs.CV 2025-06 reject novelty 4.0 of 10

    BrainMAP combines atlas-guided subgraph selection with multimodal attention-gated fusion to classify and localize brain disease efficiently.

Reference graph

Works this paper leans on

4 extracted references · 2 canonical work pages · cited by 1 Pith paper

  1. [2]

    Neuroepidemiology 53, 3-4 (2019), 187–200

    Rationale and design of the emory healthy aging and emory healthy brain studies. Neuroepidemiology 53, 3-4 (2019), 187–200. [Jack Jr et al.(2008)] Clifford R Jack Jr, Matt A Bernstein, Nick C Fox, Paul Thompson, Gene Alexander, Danielle Harvey, Bret Borowski, Paula J Britson, Jennifer L. Whitwell, Chadwick Ward, et al. 2008. The Alzheimer’s disease neuroi...

    work page 2019

  2. [2018]

    arXiv preprint arXiv:1807.03748 (2018)

    Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018). [Qiu et al.(2024)] Zifeng Qiu, Peng Yang, Chunlun Xiao, Shuqiang Wang, Xiaohua Xiao, Jing Qin, Chuan-Ming Liu, Tianfu Wang, and Baiying Lei. 2024. 3D Multimodal Fusion Network with Disease-induced Joint Learning for Early Alzheimer’s Disease Diagnosis. IEEE...

    arXiv 2018

  3. [2019]

    Molec- ular neurodegeneration 14, 1 (2019), 32

    The neuropathological diagnosis of Alzheimer’s disease. Molec- ular neurodegeneration 14, 1 (2019), 32. [Ding et al.(2024)] Saisai Ding, Juncheng Li, Jun Wang, Shihui Ying, and Jun Shi. 2024. Multimodal Co-attention Fusion Network with Online Data Augmentation for Cancer Subtype Classification. IEEE Transac- tions on Medical Imaging (2024). [Du et al.(202...

    work page 2019

  4. [2024]

    arXiv preprint arXiv:2405.16108 (2024)

    OmniBind: Teach to Build Unequal-Scale Modality Interaction for Omni-Bind of All. arXiv preprint arXiv:2405.16108 (2024). [Nagrani et al.(2021)] Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun. 2021. Attention bottlenecks for multimodal fusion. Advances in neural information processing systems 34 (2021), 14200–14213. [Ni...

    arXiv 2024

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.