REVIEW 6 major objections 6 minor 37 references
Report-guided semantic priors turn visual-only anomaly scores into semantic-aware ones and set state-of-the-art results on multimodal OCT/OCTA.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-03 14:48 UTC pith:VWOFUK4K
load-bearing objection New method, real gains, but the causal role of report semantics is underdetermined—especially on the public set with a fixed GPT report. the 6 major comments →
ReMoE: Report-Guided Mixture-of-Experts for Multimodal OCT/OCTA Anomaly Detection
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper's central claim is that converting raw visual feature discrepancies into semantic-aware discrepancies improves multimodal anomaly detection. ReMoE computes report embeddings from normal clinical text with a frozen text encoder pretrained on image-text data, and trains an image-to-text prior student—an L2-normalized global-average-pooling plus linear projection applied to encoder and decoder features—to predict those embeddings, supervised by a report semantic distillation loss. At each feature level, the student's pseudo-text is combined with modality anchors (text descriptions of OCT/OCTA slabs) into modality-aware priors. Report-Guided Modality Modulation (RMM) then uses these pr
What carries the argument
The load-bearing object is Report-Guided Modality Modulation (RMM), a mixture-of-experts block that takes modality-aware priors—a student-predicted pseudo-text embedding summed with a modality anchor per input—and uses them to compute a softmax routing weight per modality over K=3 visual experts with distinct convolutional receptive fields. The experts' responses are aggregated with those routing weights on encoder and decoder features at each level, converting raw feature discrepancies into report-conditioned discrepancies. A second mechanism, the image-to-text prior student, is the carrier of the semantics: a GAP-plus-linear projection shared across feature levels that distills the frozen
Load-bearing premise
The load-bearing premise is that the frozen text encoder's embedding of normal clinical reports is a trustworthy semantic space—and that a linear projection of globally pooled visual features can predict it; on the public dataset this is close to a constant-text regression because all training samples share one fixed report.
What would settle it
A controlled experiment would settle the claim: take the same RMM architecture and train it with a random fixed text vector as the distillation target (not the report embedding), keeping the student and anchors. If that variant matches ReMoE on OCTA500-3MM, the improvement is a capacity or regularization effect rather than report semantics. A second check: compute the variance of the student pseudo-text embeddings across test samples in OCTA500-3MM; near-zero variance would show the model cannot be using per-sample semantics there, directly challenging the semantic-aware-discrepancy story for
If this is right
- If ReMoE is right, report-guided routing is a viable way to inject clinical semantics into unsupervised anomaly detectors without needing abnormal labels.
- The modality-combination results imply that richer structural and vascular inputs monotonically improve report-modulated detection, so pairing reports with all available slabs is better than pair-wise fusion.
- The fixed-report result on OCTA500-3MM suggests that even a generic normal semantic reference can help when patient-specific reports are unavailable, widening applicability to public datasets.
- The ablation ordering (random, shuffled, fixed, then student prior) implies that the gain is tied to meaningful normal-report semantics and increases as the text prior becomes more informative about the sample.
Where Pith is reading between the lines
- [Editorial inference] The public-setting design—where every training sample shares one fixed generated report—means the student cannot learn per-sample semantics there; the 4-point AUROC gain likely reflects a generic regularization effect rather than report grounding. A synthetic per-sample-report protocol would separate these.
- [Editorial inference] The gain attributed to the text encoder's semantic space may partly come from the added capacity of the expert bank itself; a control that routes experts with a learned constant vector (or with anchors only) would isolate the report-derived contribution, since anchor-only RMM already improves over the visual baseline.
- [Editorial inference] A testable extension is to swap the frozen text encoder for one fine-tuned on retinal reports; if performance drops, the paper's implicit claim about the transferability of generic image-text semantics needs qualification.
- [Editorial inference] The framework suggests report-guided priors could be dropped into other encoder-decoder anomaly detectors (e.g., reverse-distillation models) as a general semantic conditioning layer.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes ReMoE, a report-guided mixture-of-experts framework for unsupervised anomaly detection on multimodal retinal OCT/OCTA images. A frozen CLIP text encoder encodes normal clinical reports (Eq. 6); a lightweight image-to-text student predicts pseudo-text representations from globally averaged visual features (Eq. 7) under a distillation loss (Eq. 8); the pseudo-text is combined with modality anchors (Eqs. 9–14) and used by RMM to modulate encoder/decoder features through learned visual-expert routing (Eqs. 15–20). Anomaly scores are computed from top-ρ pixel cosine discrepancies (Eqs. 2–5). The authors report SOTA results on a private OCT/OCTA dataset (I-AUROC 0.8682, I-AP 0.9358) and on an OCTA500-3MM setting with a fixed GPT-generated normal report (I-AUROC 0.8995, I-AP 0.9130), with component ablations, routing-strategy ablations, and sensitivity analyses.
Significance. The idea of distilling normal report semantics into image-conditioned priors for anomaly detection is a novel and potentially useful extension of visual-centric anomaly detection, and the reported gains over AnyAD are substantial. The paper includes five-seed standard deviations, component ablations, and several control conditions, which are strengths. However, the central attribution of the gains to report semantics is not fully established: the public setting uses a constant report target, and the ablation controls do not isolate the student's image-conditioned mechanism from semantic content. If the authors can close these gaps, the contribution would be of interest to the medical anomaly-detection and multimodal learning communities.
major comments (6)
- [Datasets and Evaluation Protocol; Eqs. (6)–(8)] On OCTA500-3MM, all normal training samples share one fixed GPT-generated report, so t_b in Eq. (6) is constant for every sample and L_RSD in Eq. (8) regresses every image to the same vector. At test time the student can only emit a near-constant pseudo-text. The 4.06-point I-AUROC gain over AnyAD in Table 1 therefore cannot be attributed to patient-specific report semantics; it could come from the added RMM/expert parameters or from a constant conditioning input. Please add a control with a non-semantic constant student target or restrict the semantics claim to the private paired-report dataset.
- [Table 2] The ablation table does not isolate the student's semantic content from its image-conditioning mechanism. The 'Random text prior' and 'Shuffled report prior' rows have an empty Stu. column, meaning they use hand-supplied text embeddings rather than a trained student; 'Student only RMM' adds both a student and image-dependent modulation. Thus the 0.8007→0.8233 jump from 'Fixed report prior' to 'Student only RMM' could be due to any learnable image-to-vector mapping, not to normal-report semantics. Add ablations where the student is trained against random, shuffled, or fixed report targets while the student architecture is preserved.
- [Related Work; Table 1] The closest text-guided medical anomaly detection method, UF-MADS (Lin, Hwang, Lai 2026), is cited in the Related Work but is absent from Table 1. Without this baseline, the claim of SOTA over text-guided methods is incomplete. Please include it if it can be adapted to OCT/OCTA, or explicitly state the technical reason it cannot.
- [Implementation Details; Hyperparameter Sensitivity] Key hyperparameters (λ_RSD=0.15, ρ=0.01, K=3, α^q initialization=0, routing temperature=1.0) are fixed, but only ρ and α are ablated, and no validation split is described for the private dataset. If any of these values were selected using the private test set (1,436 normal / 2,981 abnormal), the reported 0.8682/0.9358 could be optimistically biased. Please report a validation protocol or nested hyperparameter selection.
- [Eqs. (6)–(9); Method] The frozen CLIP ViT-B/32 text encoder is a general-domain model not trained on OCT reports. The paper provides no evidence that its embedding space preserves ophthalmic report semantics; if it does not, the 'distilled semantics' may be a fixed or low-dimensional mapping unrelated to clinical meaning. Please compare against medical text encoders (e.g., PubMedCLIP/MedCLIP-type models) and include a semantic probe, such as report-retrieval accuracy or inter-report similarity, to verify the teacher space.
- [Table 1] On OCTA500-3MM, ReMoE-2I has I-AUROC 0.8612±0.0489 and ReMoE has 0.8995±0.0232, while AnyAD has 0.8589±0.0015. With five seeds, the 4.06-point mean improvement is within about two standard deviations of the ReMoE estimate. Please report seed-wise paired comparisons or a paired significance test to establish that the improvement is not noise.
minor comments (6)
- [Eq. (17)] The text states routing temperature is 1.0, but Eq. (17) uses a plain Softmax. If a temperature is used, write Softmax(·/τ) explicitly.
- [Eq. (9)] The modality anchor texts u_m are only mentioned as being in the Supplementary Material. Please include the anchor texts in the main text or an appendix to make the method reproducible.
- [Table 3] The 'Confidence routing' and 'Learned routing' variants are not defined by equations. Please specify how the fusion weights are computed.
- [Figure 2] The figure is dense and the legend does not make clear which paths are train-only vs test-only in the text. Please clarify the data flow for L_RSD and L_RMR.
- [Eq. (2)] The loss is called 'Report-Modulated Reconstruction Loss' although the model computes a cosine discrepancy between encoder and decoder features rather than a reconstruction. Consider renaming to avoid terminological confusion.
- [Introduction] Define 'DINOv2-R' at first use (presumably DINOv2 with registers) so readers do not confuse it with the DINOv2 base model.
Circularity Check
No significant circularity: report prior is an auxiliary distillation target, not a fitted version of the result; fixed-report public setting weakens attribution but is not circular.
full rationale
ReMoE's derivation chain is not circular. The normal-report embedding t_b is used only as a distillation target (Eq. 6) for the image-to-text student; at inference the pseudo-text s^q is a learned function of visual features (Eq. 7), and the anomaly score is the cosine discrepancy between encoder and decoder features after RMM modulation (Eqs. 1-5, 9-20). The report embedding is not inserted into the score at test time, nor is it fitted to the labels, so no equation reduces to its input by construction. The ablation table includes random/shuffled/fixed-report controls, giving an independent (though imperfect) check of the text prior's role. The clearest limitation is evidential rather than circular: on OCTA500-3MM, "all normal training samples share a fixed normal report generated by GPT (Singh et al. 2026), which serves as a common normal semantic reference for this public setting," so the student is regressed to a near-constant target and the reported gain over AnyAD could come from constant conditioning or added RMM parameters; this reduces attribution, not circularity. Table 2's random/shuffled controls also use hand-supplied text rather than a student trained on such text, so the report-specific causal role is not fully isolated; again, this is a validity concern, not a circular reduction. The authors' self-citations (Nie et al. 2026a,b; Xu et al. 2026) appear only in related-work context and are not load-bearing justifications, and no uniqueness theorem is imported. I find no fitted parameter renamed as a prediction and no definitional circularity, so the honest finding is score 0.
Axiom & Free-Parameter Ledger
free parameters (6)
- lambda_RSD =
0.15
- rho (top-pixel proportion) =
0.01
- alpha^q initialization =
0
- K (number of visual experts) =
3
- Routing temperature =
1.0
- Modality anchor texts u_m =
not disclosed (Supplementary)
axioms (5)
- domain assumption CLIP text embeddings capture meaningful semantics of short normal OCT/OCTA reports.
- domain assumption A linear projection of globally averaged DINOv2 features can predict the report embedding.
- ad hoc to paper A single fixed GPT-generated normal report is an adequate semantic reference for OCTA500-3MM.
- domain assumption Normal reports in the private dataset are accurate and paired to normal images.
- domain assumption Cosine discrepancy between modulated encoder and decoder features is a valid normality score.
read the original abstract
Multimodal medical anomaly detection identifies samples deviating from normal patterns, where scarce abnormal cases make normality modeling from normal data practical. In retinal Optical Coherence Tomography (OCT) and OCT Angiography (OCTA) anomaly detection, existing unsupervised methods rely on visual feature distributions, reconstruction residuals, or encoder-decoder discrepancies, making anomaly scores depend on appearance-level deviations, while multimodal normality also contains semantic organization described in normal medical reports. To this end, we propose Report-Guided Mixture-of-Experts (ReMoE), which distills normal report semantics into an image-to-text prior student, builds modality-aware priors, and uses Report-Guided Modality Modulation (RMM) to modulate features through mixture-of-experts routing. Experiments on a private OCT/OCTA dataset with paired normal reports and a public OCTA500-3MM setting using a fixed normal report demonstrate state-of-the-art performance.
Figures
Reference graph
Works this paper leans on
-
[1]
Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
MVTec AD--A comprehensive real-world dataset for unsupervised anomaly detection , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
-
[2]
Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
Cutpaste: Self-supervised learning for anomaly detection and localization , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
-
[3]
Multimedia tools and applications , volume=
A systematic literature review on multimodal medical image fusion , author=. Multimedia tools and applications , volume=. 2024 , publisher=
2024
-
[4]
Information Fusion , pages=
Towards reliable medical unsupervised anomaly detection: A benchmark and dataset for PET/CT with cross-modality knowledge distillation network , author=. Information Fusion , pages=. 2026 , publisher=
2026
-
[5]
science , volume=
Optical coherence tomography , author=. science , volume=. 1991 , publisher=
1991
-
[6]
Progress in retinal and eye research , volume=
Optical coherence tomography angiography , author=. Progress in retinal and eye research , volume=. 2018 , publisher=
2018
-
[7]
Medical image analysis , volume=
OCTA-500: a retinal dataset for optical coherence tomography angiography study , author=. Medical image analysis , volume=. 2024 , publisher=
2024
-
[8]
npj Digital Medicine , volume=
Fusing multispectral information for retinal layer segmentation , author=. npj Digital Medicine , volume=. 2025 , publisher=
2025
-
[9]
Medical image analysis , volume=
f-AnoGAN: Fast unsupervised anomaly detection with generative adversarial networks , author=. Medical image analysis , volume=. 2019 , publisher=
2019
-
[10]
Proceedings of the IEEE/CVF international conference on computer vision , pages=
Draem-a discriminatively trained reconstruction embedding for surface anomaly detection , author=. Proceedings of the IEEE/CVF international conference on computer vision , pages=
-
[11]
International conference on pattern recognition , pages=
Padim: a patch distribution modeling framework for anomaly detection and localization , author=. International conference on pattern recognition , pages=. 2021 , organization=
2021
-
[12]
Proceedings of the British Machine Vision Conference , year=
Student-teacher feature pyramid matching for anomaly detection , author=. Proceedings of the British Machine Vision Conference , year=
-
[13]
Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
Towards total recall in industrial anomaly detection , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
-
[14]
Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
Simplenet: A simple network for image anomaly detection and localization , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
-
[15]
Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
Anomaly detection via reverse distillation from one-class embedding , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
-
[16]
Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
Anoddpm: Anomaly detection with denoising diffusion probabilistic models using simplex noise , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
-
[17]
Machine learning for healthcare conference , pages=
Contrastive learning of medical visual representations from paired images and text , author=. Machine learning for healthcare conference , pages=. 2022 , organization=
2022
-
[18]
Proceedings of the IEEE/CVF international conference on computer vision , pages=
Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition , author=. Proceedings of the IEEE/CVF international conference on computer vision , pages=
-
[19]
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages=
Medclip: Contrastive learning from unpaired medical images and text , author=. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages=
2022
-
[20]
European conference on computer vision , pages=
Making the most of text semantics to improve biomedical vision--language processing , author=. European conference on computer vision , pages=. 2022 , organization=
2022
-
[21]
Nature biomedical engineering , volume=
Expert-level detection of pathologies from unannotated chest X-ray images via self-supervised learning , author=. Nature biomedical engineering , volume=. 2022 , publisher=
2022
-
[22]
npj digital medicine , volume=
A multimodal visual--language foundation model for computational ophthalmology , author=. npj digital medicine , volume=. 2025 , publisher=
2025
-
[23]
IEEE transactions on neural networks and learning systems , year=
Msflow: Multiscale flow-based framework for unsupervised anomaly detection , author=. IEEE transactions on neural networks and learning systems , year=
-
[24]
Proceedings of the AAAI Conference on Artificial Intelligence , volume=
Unlocking the potential of reverse distillation for anomaly detection , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=
-
[25]
Proceedings of the Computer Vision and Pattern Recognition Conference , pages=
Dinomaly: The less is more philosophy in multi-class unsupervised anomaly detection , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=
-
[26]
Pattern Recognition , pages=
Few-shot medical anomaly detection through centroid consultation back and test-time self-calibration , author=. Pattern Recognition , pages=. 2026 , publisher=
2026
-
[27]
Information Fusion , volume=
Adapting the segment anything model for multi-modal retinal anomaly detection and localization , author=. Information Fusion , volume=. 2025 , publisher=
2025
-
[28]
2025 , eprint=
AnyAD: Unified Any-Modality Anomaly Detection in Incomplete Multi-Sequence MRI , author=. 2025 , eprint=
2025
-
[29]
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
Unimmad: Unified multi-modal and multi-class anomaly detection via moe-driven feature decompression , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
-
[30]
Proceedings of the AAAI Conference on Artificial Intelligence , volume=
Unsupervised domain adaptation for medical image segmentation by selective entropy constraints and adaptive semantic alignment , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=
-
[31]
Leveraging Image-text Pairs for Generalized Category Discovery in Medical Image Classification , year=
Feng, Wei and Wang, Bingjie and Wang, Zhonghua and Zhou, Sijin and Ge, Zongyuan , journal=. Leveraging Image-text Pairs for Generalized Category Discovery in Medical Image Classification , year=
-
[32]
Transactions on Machine Learning Research , year=
Dinov2: Learning robust visual features without supervision , author=. Transactions on Machine Learning Research , year=
-
[33]
International conference on learning representations , volume=
Vision transformers need registers , author=. International conference on learning representations , volume=
-
[34]
2021 , eprint=
Learning Transferable Visual Models From Natural Language Supervision , author=. 2021 , eprint=
2021
-
[35]
2026 , eprint=
OpenAI GPT-5 System Card , author=. 2026 , eprint=
2026
-
[36]
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
UFMADS: A Unified Framework for Prompt-Guided Medical Anomaly Detection and Segmentation , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
-
[37]
2026 , eprint=
CFR-Net:Collaborative Feature Refinement Network for Medical Image Anomaly Detection , author=. 2026 , eprint=
2026
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.