{"id":"65bec737-a627-45f8-8288-f8e19b2f43ee","arxiv_id":"2507.18481","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"QFAE, combining a frozen DINOv2 encoder, a Q-Former bottleneck, and a masked-autoencoder perceptual loss, reaches 94.3, 91.8, and 83.8 AUROC on BraTS2021, RESC, and RSNA.","lead":"QFAE is an autoencoder that uses a frozen, natural-image-pretrained vision model as its encoder and a Q-Former as its bottleneck, and it reports top anomaly-detection scores on three medical benchmarks. It matters because it shows off-the-shelf foundation models can be repurposed for medical imaging without fine-tuning the encoder.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RSNA SOTA result uses a per-dataset anomaly-score aggregation that differs from Eq. (4) and was selected on the validation split, so the headline claim is not yet established for the method as specified.","rationale":"The reader's weakest_assumption concerns transfer of frozen DINOv2/MAE features; my concern is orthogonal but related: even if the features transfer, the headline result is sensitive to a scoring function that is not the one in Eq. (4). The supplementary is honest about the deviation, but the main text and abstract do not qualify the claim. The proposed check would settle whether the RSNA result is an artifact of per-dataset aggregation selection. I see no evidence of misconduct; the issue is that the method as specified and the method as evaluated diverge on one benchmark. This supports the reader's CONDITIONAL verdict rather than a rejection, because the core construction (frozen encoder, Q-Former bottleneck, MAE perceptual loss) is otherwise well ablated and the remaining numbers are plausible.","tokens_in":18892,"tokens_out":9333,"duration_ms":98058,"concrete_test":"Re-run the RSNA test evaluation with the aggregation exactly as in Eq. (4) (max within each MAE feature map, then mean across maps), keeping all other settings from supplementary Table 10 identical. If test AUROC falls toward the 78.6 validation value seen with max-then-mean, the RSNA SOTA claim is contingent on the alternative aggregation; if it remains near 83.8, Eq. (4) can be corrected and the claim stands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing point is that the RSNA state-of-the-art number is not produced by the anomaly score defined in the paper. Main-text Eq. (4) defines the image-level score as the maximum of each layer-wise anomaly map, averaged across maps (max-then-mean). Supplementary Section 8 reports that on the RSNA validation split this aggregation gave only 78.6 AUROC, whereas the alternative mean-then-max aggregation gave 84.3, and the test AUROC of 83.8 in Table 5 was obtained with the latter. The anomaly-score function is therefore dataset-specific and was selected on validation, not part of the framework as specified by Eq. (4). Since RSNA is one of the three claimed state-of-the-art benchmarks, the central claim that QFAE as defined achieves SOTA on RSNA is not yet established. The BraTS claim is additionally a statistical tie with P-VQ (94.3±0.18 vs 94.3±0.23), so the margin of the headline result is thin. The concern is not that validation tuning is illegitimate per se, but that the main text presents Eq. (4) as the score while the reported SOTA depends on a different, per-dataset aggregation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes QFAE, an unsupervised medical anomaly detection autoencoder that combines a frozen pretrained ViT encoder (DINO, DINOv2, OpenCLIP), a Q-Former bottleneck producing a fixed-length latent sequence, a lightweight transformer decoder, and a perceptual loss computed from a frozen Masked Autoencoder. Anomaly scores are obtained from cosine distances between MAE features of the input and its reconstruction. Experiments on the BMAD benchmarks report AUROC values of 94.3 on BraTS2021, 91.8 on RESC, and 83.8 on RSNA, with additional LiverCT results in the supplement.","tokens_in":19209,"tokens_out":6783,"duration_ms":68620,"significance":"If the reported numbers hold under a clearly specified protocol, the paper would demonstrate that a frozen foundation-model encoder plus a Q-Former bottleneck and MAE-based perceptual loss is a competitive reconstruction-based anomaly detector across MRI, OCT, and X-ray modalities. The paper has clear strengths: code and models are released, results are reported as mean and standard deviation over five seeds, and the ablation study examines the encoder, loss, aggregation, decoder patch size, and perceptual-model patch size. The authors also include a limitations paragraph in the conclusions, which is a positive indication of balanced reporting. However, the headline SOTA claims are weakened by two issues: the BraTS2021 result is a statistical tie with P-VQ, and the RSNA result is produced by an anomaly-score aggregation different from the one defined in the main text and selected per dataset on the validation split. As a result, the paper currently supports 'strong empirical results' more strongly than 'a single unified framework with three state-of-the-art benchmarks.'","major_comments":[{"comment":"The RSNA state-of-the-art result is not produced by the anomaly score defined in Eq. (4). Eq. (4) defines the image-level score as the mean, over feature layers, of each layer's maximum cosine distance (max-then-mean). Supplement Section 8 reports that this aggregation obtained 78.6 AUROC on the RSNA validation split, whereas the alternative mean-then-max aggregation obtained 84.3, and that the test AUROC of 83.8 in Table 5 was obtained with the latter. Thus the RSNA claim depends on a per-dataset, validation-selected anomaly-score function that the main text does not define. The paper should either incorporate this alternative aggregation into the method definition for RSNA, report results for the fixed score of Eq. (4) across all datasets, or clearly restate the RSNA result as coming from a variant evaluated under a separate validation-selected protocol. As written, the central claim that QFAE as specified achieves SOTA on RSNA is not established.","section":"Supplement Section 8 vs. Eq. (4), Table 5"},{"comment":"The 'state-of-the-art' claim on BraTS2021 is a statistical tie rather than an improvement: QFAE achieves 94.3±0.18 and P-VQ achieves 94.3±0.23. The abstract and the contributions bullet state SOTA on BraTS2021, and Table 5's caption says the method 'outperform[s] all methods,' while Section 4.4 says only that the result is 'on par' with P-VQ. This inconsistency should be resolved by either performing and reporting a significance test or softening the abstract and contributions to 'matches the state of the art on BraTS2021.'","section":"Section 4.4, Table 5, Abstract"},{"comment":"The evaluation configuration is not a single fixed protocol: perceptual patch sizes, feature layers, and the order of spatial/cross-feature aggregation change per dataset and were selected on validation splits (e.g., Eq. (4) is used for BraTS/RESC, mean-then-max for RSNA, and a different patch-size set for LiverCT). Validation-based tuning is not inherently illegitimate, but the main text presents Eq. (4) as the method's score while the supplement shows that different scores are used for different reported results. The paper should specify the model-selection protocol, report results for both the fixed configuration and the per-dataset configurations, and state in the main text which evaluation configuration produced each headline number. Without this, the claim that the framework generalizes across modalities is weaker than presented.","section":"Supplement Tables 6, 7, 9, 10; Section 6.3"}],"minor_comments":[{"comment":"The formula 'Apixel-max(j,k) = max_n (An(j,k))' contradicts the preceding sentence, which says the final pixel-level map is obtained by taking the mean across all resized anomaly maps; please correct the formula and the variable name.","section":"Supplement Section 6.3"},{"comment":"The first row reports '654.1' for the Main Config AUROC; this appears to be a typo for '65.4' and should be corrected.","section":"Supplement Table 8"},{"comment":"The main text says features are extracted from 'layers 20 and 22 of the ViT-L encoder and layers 8 and 10 of the ViT-B architecture,' while the supplement specifies '2nd and 4th to last blocks'; please align these descriptions for reproducibility.","section":"Section 4.2 vs. Supplement Table 6"},{"comment":"Table 1 row 3 reports 86.8 AUROC for the LPerceptual configuration, while Table 2a reports 88.5 AUROC for the same loss setting; please state the exact configuration for each table or explain the discrepancy.","section":"Table 1 vs. Table 2a"},{"comment":"The caption notes that MVFA-AD is few-shot rather than unsupervised; this qualification should also appear in the main text where MVFA-AD is compared with unsupervised methods.","section":"Figure 4 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the journal's scope and the core idea is promising, but the current manuscript overstates the state-of-the-art claim: the RSNA number is produced by a validation-selected score function different from the one in Eq. (4), and the BraTS result is a tie. If the authors add a fixed-protocol evaluation or clearly present the per-dataset validation protocol, and adjust the SOTA claims accordingly, I would support acceptance. The supplement is transparent about the RSNA configuration, which is to the authors' credit; the problem is that the main text does not reflect it."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a sensible engineering contribution: frozen DINOv2 encoder, Q-Former bottleneck, and MAE-based perceptual loss for unsupervised medical anomaly detection. The combination is new, and the authors run proper 5-run protocols, report standard deviations, ablate each component, and are unusually transparent in the supplement about what they tuned. That parts deserves credit.\n\nThe soft spot is real. The main-text Eq. (4) defines the image-level score as max-then-mean across MAE feature maps. The supplement for RSNA shows that on the validation split this aggregation gave 78.6 AUROC, while mean-then-max gave 84.3, and the test number in Table 5 was produced with the latter. So the RSNA state-of-the-art result is not produced by the score the paper specifies. Since RSNA is one of the three claimed SOTA benchmarks, that claim is not established. This is not fatal to the whole paper, but it is load-bearing and needs fixing: either change Eq. (4) to match what was actually used, or present the per-dataset aggregation as part of the method and justify the selection without overselling.\n\nTwo more things worth noting. On BraTS2021, the result is a statistical tie with P-VQ (94.3±0.18 vs 94.3±0.23), so the 'state-of-the-art' language there is overstated. And the evaluation configurations are tuned per dataset on validation (patch sizes, aggregation, even preprocessing for LiverCT), which weakens the claim of a single robust framework. The impact is moderate: the paper shows a reusable recipe, not a field-reorganizing result.\n\nWho gets value from this: people working on medical AD, especially those who want strong baselines built from frozen foundation models. The ablations are genuinely useful. The paper deserves peer review, but it needs a revision that reconciles the scoring rule with the RSNA experiments and tempers the SOTA claims. I would send it out, with that as the main request.","headline":"A useful empirical combination for medical AD, but the RSNA SOTA claim depends on a validation-selected aggregation that contradicts the main-text scoring rule.","tokens_in":19712,"tokens_out":1746,"would_cite":false,"duration_ms":22212,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"QFAE shows that an autoencoder built from a frozen vision foundation model, a Q-Former bottleneck, and a masked-autoencoder perceptual loss achieves top AUROC on three medical anomaly detection benchmarks.","keywords":["medical anomaly detection","unsupervised anomaly detection","Q-Former","vision foundation models","DINOv2","masked autoencoder","perceptual loss","autoencoder"],"falsifier":"Train QFAE replacing the frozen DINOv2 encoder with a randomly initialized ViT of the same architecture, keeping the Q-Former, decoder, and MAE perceptual loss unchanged; if BraTS2021 AUROC stays near 94.3, the frozen foundation-model features are not load-bearing, while if it collapses, the transfer claim is confirmed. A second check is to run the exact RSNA configuration on a different chest X-ray set and see whether the image-level score distribution separates known pathologies from healthy controls.","tokens_in":18739,"feed_emoji":"🩺","tokens_out":5135,"duration_ms":51067,"temperature":0.7,"pith_summary":"This paper tries to establish that a classical autoencoder, modernized with three components, is a competitive unsupervised medical anomaly detector: a frozen vision foundation model as encoder, a Q-Former bottleneck that compresses multi-scale features into a fixed-length latent, and a perceptual loss computed in the feature space of a pretrained Masked Autoencoder. The claim matters because anomaly detection in medical images must usually work without labeled anomalies and without domain-specific fine-tuning. The authors report that the resulting framework, QFAE, reaches 94.3 AUROC on BraTS2021 brain MRI, 91.8 on RESC retinal OCT, and 83.8 on RSNA chest X-rays, matching or beating prior methods on all three benchmarks. Ablations attribute large gains to the Q-Former bottleneck and to the perceptual loss, indicating that neither component alone explains the result.","feed_headline":"Q-Former autoencoder sets new AUROC highs on three medical benchmarks","feed_subtitle":"Ties or beats prior methods on BraTS2021, RESC, and RSNA using frozen encoders and no labeled anomalies.","key_machinery":"The Q-Former bottleneck is the central adaptive component: a single transformer block with learnable query tokens that self-attend among themselves, cross-attend to the frozen encoder's projected multi-layer features as keys and values, and are passed through an MLP to produce a fixed-length latent. The number of queries controls the decoder's patch count, decoupling encoder and decoder output lengths and letting the decoder reconstruct at 8x8 patches. The second load-bearing mechanism is the Masked-Autoencoder perceptual loss, computed as a multi-scale cosine distance in feature space; it replaces pixel-level losses and, together with taking the maximum reconstruction error per layer when scoring, is what separates anomalies from normal tissue.","core_discovery":"QFAE replaces the trainable encoder of a classical autoencoder with a frozen pretrained vision transformer, such as DINO, DINOv2, or OpenCLIP, feeds the resulting multi-scale features through a Q-Former bottleneck that compresses them into a fixed number of learnable queries, and reconstructs the image with a lightweight transformer decoder. Training is guided solely by a perceptual loss: the cosine distance between features of the input and the reconstruction, extracted from several layers of a frozen Masked Autoencoder at multiple patch sizes. At test time the same cosine distance, aggregated per-layer by taking the maximum spatial error and then averaging across layers, is the anomaly score. The paper reports that this setup reaches 94.3 AUROC on BraTS2021, 91.8 on RESC, and 83.8 on RSNA, matching or surpassing prior methods on each, with ablations showing the Q-Former bottleneck and the MAE perceptual loss each contribute large gains.","pith_inferences":["A testable extension is multi-class anomaly detection: the paper names it as future work, and the fixed-length latent produced by Q-Former could plausibly carry a class label, but the paper does not test this.","The frozen-encoder design implies the framework's ceiling is set by the feature spaces of DINOv2 and MAE; newer foundation models pretrained on medical or multimodal data could be swapped in without retraining the rest, which the paper does not evaluate.","The sensitivity of RSNA results to switching the anomaly-score aggregation suggests the method may need a per-dataset aggregation recipe; whether a single universal aggregation exists is left open.","Because all reported gains come from datasets with relatively homogeneous normal images, a harder test would be a modality with large normal variation, such as full-body CT; the paper's stated limitation anticipates this."],"forward_implications":["On the three BMAD benchmarks, QFAE matches or beats every compared method without any encoder fine-tuning or labeled anomalies.","Ablations attribute 12.9 AUROC points to the Q-Former bottleneck and further large gains to switching from pixel-level loss to MAE feature-space perceptual loss, so both components are needed.","Smaller decoder patch sizes (8x8) and larger perceptual patch sizes (32x32 and 56x56 combined) improve detection, indicating granularity choices matter on both the reconstruction and scoring sides.","Per-dataset evaluation tweaks, such as mean-then-max aggregation on chest X-rays with optical artifacts, change AUROC by several points, so the aggregation rule should be tuned to the modality."],"supporting_citations":[{"why":"Supplies the frozen self-supervised DINO encoder used as one feature extractor option.","marker":"[10]"},{"why":"Supplies the default frozen DINOv2 encoder whose features achieve 94.4 AUROC alone in ablations.","marker":"[43]"},{"why":"Supplies the Masked Autoencoder feature space used for the perceptual loss and as an alternative encoder.","marker":"[22]"},{"why":"Introduces the Q-Former architecture that the paper adapts as the bottleneck.","marker":"[35]"},{"why":"Provides precedent for using Q-Former to aggregate multi-scale features into fixed-length latents.","marker":"[28]"},{"why":"Provides the BMAD benchmark suite and dataset splits used for evaluation.","marker":"[4]"},{"why":"The previous best method on BraTS2021 that QFAE matches in AUROC.","marker":"[29]"},{"why":"The previous best method on RESC that QFAE surpasses.","marker":"[46]"},{"why":"The previous best method on RSNA that QFAE surpasses.","marker":"[47]"},{"why":"Supplies the OpenCLIP encoder evaluated as an alternative frozen feature extractor.","marker":"[52]"}],"fun_headline_variants":["Frozen vision transformers power new medical anomaly detection","Q-Former autoencoder hits SOTA on three medical anomaly benchmarks","No fine-tuning needed: Q-Former AE for medical anomaly detection","Q-Former AE uses frozen DINO to flag medical anomalies","Medical anomaly detection: Q-Former AE matches or beats prior"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that features learned on natural images by DINOv2 and the Masked Autoencoder transfer to brain MRI, retinal OCT, and chest X-ray without any fine-tuning, and that cosine distance in MAE feature space between an image and its reconstruction reliably separates anomalies from healthy tissue across all three modalities.","fun_headline_variants_meta":{"raw":{"variants":["Frozen vision transformers power new medical anomaly detection","Q-Former autoencoder hits SOTA on three medical anomaly benchmarks","No fine-tuning needed: Q-Former AE for medical anomaly detection","Q-Former AE uses frozen DINO to flag medical anomalies","Medical anomaly detection: Q-Former AE matches or beats prior"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000201,"raw_usage":{"total_tokens":1398,"prompt_tokens":985,"completion_tokens":413,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":328}},"tokens_in":601,"tokens_out":413,"duration_ms":4514,"temperature":1.0,"reasoning_tokens":328,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:32:01.042990+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train QFAE replacing the frozen DINOv2 encoder with a randomly initialized ViT of the same architecture, keeping the Q-Former, decoder, and MAE perceptual loss unchanged; if BraTS2021 AUROC stays near 94.3, the frozen foundation-model features are not load-bearing, while if it collapses, the transfer claim is confirmed. A second check is to run the exact RSNA configuration on a different chest X-ray set and see whether the image-level score distribution separates known pathologies from healthy controls.","supporting_citations":[],"review_version":1}