{"id":"6ec114cf-9c08-4bda-94c2-e9a41a663af2","arxiv_id":"2506.14919","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A frequency-band-masked reconstruction error, FCRE, improves membership inference on medical image diffusion models.","lead":"The authors show that filtering diffusion reconstruction errors to a mid-frequency band improves membership inference attacks on medical image diffusion models. Their FCRE method beats standard attacks on brain MRI and chest X-ray datasets, but its headline gains rely on frequency thresholds tuned on the same data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"FeTS 2022 evaluation confounds membership with institution: all non-members come from one held-out institution, so the reported AUC gain may reflect domain shift rather than membership detection.","rationale":"I agree with the reader that the threshold choice (15%/85% Laplacian percentiles) is tuned on the evaluation data and may not transfer, but the stronger issue is the FeTS split. The FeTS dataset is multi-institutional, and the paper's non-member set is exclusively from a held-out institution while members come from the other 16. This produces a distribution shift between member and non-member sets that any reconstruction-based score can exploit. The method's own motivation—medical images have polarized frequency distributions—makes it especially sensitive to acquisition differences, so the FeTS result cannot be interpreted as membership inference until this confound is removed. The ChestX-ray8 experiment uses a random split and is cleaner; FCRE's improvement there (AUC 0.971 vs 0.907 SecMI) is the strongest surviving evidence. The CIFAR-10 result does not support the medical claim and actually shows FCRE (L2+SSIM) performing no better than SecMI on AUC, which slightly weakens the 'consistently outperforms' narrative. Overall, the paper has a plausible mechanism and one clean positive dataset, so the conditional verdict is appropriate, but the FeTS numbers should not be cited as evidence of membership detection without a deconfounded rerun.","tokens_in":7274,"tokens_out":3837,"duration_ms":38578,"concrete_test":"Re-run the FeTS experiment with a random split that pools all institutions before assigning member/non-member cases (or at minimum draws non-member cases from the same institutions as members), keeping the same training procedure and FCRE thresholds. If the AUC and TPR@FPR1% in Table 1 drop substantially—toward ChestX-ray8 levels or toward SecMI/PIA—the reported FeTS advantage is a domain-shift artifact. A second useful control: train the diffusion model on a subset of institutions and test non-members from both seen and unseen institutions to quantify the institution effect.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is the FeTS 2022 experimental design (Sec. 3.1). Member images are 37,000 slices from 740 cases across 16 institutions, while non-member images are 25,600 slices from 511 cases of a single held-out institution. Because the diffusion model is trained only on the member institutions, reconstruction errors for the held-out institution can be inflated by scanner/protocol differences and anatomical distribution shift, not by non-membership. The Table 1 result (AUC 0.926, TPR@FPR1% 0.328) is therefore not a clean measurement of membership inference; it may be partly or largely a domain-shift detector. This is not a flaw in the FCRE computation, but it means the paper's headline medical claim is supported by only the ChestX-ray8 random-split experiment, where FCRE also improves over baselines. The reader's threshold-tuning concern is real but secondary; the FeTS confound threatens the validity of the strongest reported numbers independently of hyperparameter choice.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FCRE, a membership inference attack (MIA) for diffusion models trained on medical images. The method extends the SecMI inversion-symmetry idea by selecting only mid-frequency image patches (defined by Laplacian-score thresholds computed on the original image), computing masked L2 and SSIM reconstruction errors, and thresholding the combined score. Experiments on FeTS 2022, ChestX-ray8, and CIFAR-10 compare against Loss-based, SecMI, and PIA/PIAN baselines, reporting ASR, AUC, and TPR@FPR1%. The authors report large gains on the two medical datasets (e.g., FeTS AUC 0.926 and TPR@FPR1% 0.328 versus 0.708 and 0.052 for SecMI). I evaluate whether those gains are actually supported by the experimental design.","tokens_in":7636,"tokens_out":6786,"duration_ms":63141,"significance":"The problem is timely, and the frequency-calibration idea is a plausible and clearly motivated way to address the hard/easy sample confound in medical image reconstruction. The ChestX-ray8 random-split result is promising, and Table 2 shows that frequency masking improves over no masking. If the experimental confounds are corrected, FCRE could be a useful tool for privacy auditing of medical diffusion models. The paper does not release code or model weights, which limits reproducibility, and the method relies on several free thresholds rather than a parameter-free derivation.","major_comments":[{"comment":"The FeTS 2022 evaluation confounds membership with institutional origin. Member images are 37,000 slices from 740 cases across 16 institutions, while all 25,600 non-member images come from 511 cases of a single held-out institution, and the diffusion model is trained only on the member institutions. Consequently, the Table 1 FeTS results (AUC 0.926, TPR@FPR1% 0.328) can be explained by scanner, protocol, or anatomical domain shift between the held-out institution and the training institutions, rather than by membership. Please re-run with a patient-level split that spans institutions in both member and non-member sets, or add a control condition in which the held-out institution is included in training, so that membership and institution are not perfectly correlated.","section":"Sec. 3.1, FeTS 2022 split"},{"comment":"The ChestX-ray8 partition is described only as independently and randomly selecting over 3,600 images for each set. Because ChestX-ray8 contains multiple images per patient, a random image-level split can place images from the same patient in both the member and non-member sets, which can inflate reconstruction-similarity-based attacks through patient anatomy similarity. Please either perform a patient-level split (all images of a patient in one set) or explicitly justify why image-level splitting is appropriate for the membership definition used.","section":"Sec. 3.1, ChestX-ray8 split"},{"comment":"The main results use thresholds Lmin=15% and Lmax=85%, which are selected using the Table 2 ablation on the same FeTS 2022 and ChestX-ray8 evaluation sets. There is no held-out validation for threshold choice, so the reported TPR@FPR1% and AUC improvements are optimistically biased by selection on the test data. The no-masking row in Table 2 shows that frequency masking helps, but the magnitude of the headline gains in Table 1 should be re-estimated using a nested or validation-split procedure, and fixed default thresholds should be reported as a sensitivity check.","section":"Sec. 3.4, Table 2 vs. Table 1"},{"comment":"The thresholding mechanism is under-specified. The text says the Laplacian score L(p) is computed and that patches are selected within the mid-frequency range based on thresholds applied to the mean absolute Laplacian score, and Table 2 reports Lmin and Lmax as percentages, but it is not stated whether these are per-image percentiles, dataset-wide percentiles, or absolute values, nor how the binary mask is constructed from L(p). This ambiguity prevents replication of the method and makes the ablation difficult to interpret. Please provide a precise algorithmic description and release code or pseudocode.","section":"Sec. 2.2, Eqs. (3)-(5)"}],"minor_comments":[{"comment":"The column header 'TPR 1%' should be 'TPR@FPR1%' for consistency, and the table formatting runs numbers together (e.g., '0.3160.810' and '0.8490.291').","section":"Table 1"},{"comment":"The symbol L(p) is used both for the Laplacian score in Eq. (3) and for the binary mask in Eqs. (4)-(5); please rename one of the two to avoid confusion.","section":"Sec. 2.2, Eqs. (3)-(5)"},{"comment":"The claim that FCRE consistently outperforms all baselines across the two medical image datasets is not supported for FeTS until the institution confound is addressed; the wording should be softened or qualified pending re-analysis.","section":"Sec. 3.3"},{"comment":"'CIF AR-10' should be written as 'CIFAR-10' in the places where the space appears.","section":"Throughout"},{"comment":"The statement that the optimal thresholds vary across datasets is in tension with using the same 15%/85% thresholds for both medical datasets; please clarify how the same thresholds were justified if they are not optimal per dataset.","section":"Sec. 3.4"},{"comment":"No confidence intervals or standard deviations are reported for any of the AUC, ASR, or TPR values; given the evident sensitivity to thresholds, reporting variability across random splits or model seeds would strengthen the claims.","section":"Sec. 3"}],"recommendation":"major_revision","confidential_remarks":"The FeTS confound is the most serious issue and should be treated as a correctness problem rather than a presentation fix. The threshold-overfitting concern is also important and should be resolved with a proper validation split. If the authors re-run the experiments with institution-balanced and patient-level splits and the gains persist, the paper would likely be acceptable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here is my take on arXiv:2506.14919. The kernel is worth reading: FCRE takes the DDIM reconstruction error from SecMI, masks out patches with very high and very low Laplacian scores, and scores the remaining mid-frequency patches with SSIM plus L2. The motivation ties to the polarized frequency content of medical images, and the implementation is clean. On ChestX-ray8, where the split is random, FCRE (L2+SSIM) gives AUC 0.971 and TPR@FPR1% 0.409 against SecMI's 0.907 and 0.176. That is a solid, reproducible-sounding improvement, and it makes the frequency-selective idea credible.\n\nThe soft spots are real, though. The FeTS 2022 experiment puts all non-members in a single held-out institution while training on 16 others. The 0.926 AUC there is likely part membership detection and part domain shift. That is the biggest number in the paper, so it should not be trusted as clean evidence. The second issue is threshold selection: the 15%/85% cutoffs come from the Table 2 ablation on the same evaluation data, so the reported gains partly encode fitting. The authors honestly note that optimal thresholds vary across datasets, but they don't validate on a separate set. Minor issues: no code or error bars, and the \"first MIA for medical diffusion models\" claim is probably overstated given the amount of MIA work.\n\nIf you strip away the FeTS confound, the ChestX-ray8 result still supports the central claim. The method is incremental but useful for privacy auditors who want a stronger attack on medical models. For a serious referee, I would accept it with major revision: redo the FeTS split to mix institutions across member/non-member or use a true random split, and move the threshold selection to a validation fold before reporting final numbers. The core idea is sound, so I would not desk-reject it.","headline":"FCRE is a plausible, useful MIA tweak with a clean win on ChestX-ray8, but the FeTS headline is confounded by institution and thresholds are tuned on the test sets.","tokens_in":8013,"tokens_out":2967,"would_cite":false,"duration_ms":29309,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that membership inference attacks on medical-image diffusion models become far more accurate when the comparison is restricted to a mid-frequency band of the image, and it supports this claim with a new method, FCRE.","keywords":["membership inference attack","diffusion model","medical imaging","frequency calibration","DDIM inversion","structural similarity","privacy","reconstruction error"],"falsifier":"Run FCRE with fixed thresholds on a medical dataset with a different frequency profile, for example CT or ultrasound images where low-frequency background dominates; if the AUC drops to the SecMI baseline or below, the reported advantage is an artifact of threshold selection rather than a general property of mid-frequency reconstruction error.","tokens_in":7106,"feed_emoji":"🧠","tokens_out":4077,"duration_ms":49978,"temperature":0.7,"pith_summary":"This paper tries to show that membership inference attacks on medical-image diffusion models can be made far more accurate than existing reconstruction-error methods by scoring only the mid-frequency part of the image. It proposes FCRE, which computes the DDIM reconstruction error on image patches selected by Laplacian score, ignoring both high frequencies (which are hard to reconstruct for every image) and low frequencies (which carry little information), and combines an L2 distance with SSIM into a single score. On FeTS 2022 brain MRI and ChestX-ray8 the attack raises AUC from 0.708 to 0.926 and from 0.907 to 0.971 relative to the SecMI baseline. The practical point is that medical images shared as training data are more exposed to membership disclosure than commonly assumed.","feed_headline":"Mid-frequency mask lifts membership attack AUC to 0.926","feed_subtitle":"FCRE ignores hard high and flat low frequencies, beating SecMI on brain MRI and chest X-rays.","key_machinery":"The load-bearing object is the mid-frequency patch mask $L(p)$ built from Laplacian scores of the original image $\\mathbf{x}_0$, applied equally to $\\mathbf{x}_t$ and $\\tilde{\\mathbf{x}}_t$ so that both are compared only in mid-frequency regions. The attack score is the sum of an L2 reconstruction error and a structural dissimilarity, Eq. (7), $\\text{MIA score} = (1-\\text{SSIM}) + \\|\\tilde{\\mathbf{x}}_t^F - \\mathbf{x}_t^F\\|_2$, computed after reducing both images to their selected patches.","core_discovery":"The central claim is that frequency calibration resolves the confounding factor of inherent image difficulty in MIAs against diffusion models trained on medical images. FCRE first estimates a mid-frequency patch mask from the original image using Laplacian scores, applying percentile thresholds $L_{\\min}$ and $L_{\\max}$ (set to 15% and 85% in the main experiments). It then carries out a partial DDIM reverse step from $\\mathbf{x}_t$ to $\\mathbf{x}_{t+\\Delta t}$ and back to $\\tilde{\\mathbf{x}}_t$, masks both $\\mathbf{x}_t$ and $\\tilde{\\mathbf{x}}_t$ with the same patch mask, and scores membership as $(1-\\text{SSIM}) + \\|\\tilde{\\mathbf{x}}_t^F - \\mathbf{x}_t^F\\|_2$. The paper reports that this score separates member and non-member score distributions much better than SecMI or PIA on two medical datasets, and argues that the mid-frequency band is where the membership signal lives in medical images.","pith_inferences":["The fixed 15% and 85% thresholds were selected on the same datasets used for evaluation, so the reported gains may overstate performance on new medical domains; a truly adaptive threshold would test whether the mid-frequency band generalizes.","Because the Laplacian score is computed on the original candidate image, this attack requires the attacker to possess the exact image, not merely query access to the model, a constraint that bounds its real-world applicability.","The mid-frequency membership signal might also appear in other high-frequency-rich modalities such as pathology or retinal images, but the paper does not test those domains.","A testable defensive implication is that training diffusion models to reduce mid-frequency reconstruction asymmetry could lower MIA accuracy, though this is not investigated in the paper."],"forward_implications":["On FeTS 2022, FCRE (L2+SSIM) reports AUC 0.926 and TPR@FPR1% 0.328, compared with 0.708 and 0.052 for SecMI; on ChestX-ray8 AUC is 0.971 versus 0.907 for SecMI.","Excluding only high or only low frequencies gives intermediate results, so the full mid-frequency mask is necessary for the reported gains.","FCRE's advantage shrinks on low-resolution natural images (CIFAR-10), where the authors attribute the gap to limited fine-grained frequency information.","The method provides a concrete scoring tool for auditing whether a specific medical image was part of a diffusion model's training set."],"supporting_citations":[{"why":"Defines SecMI, the step-wise error baseline whose inversion-symmetry mechanism FCRE extends and against which FCRE is primarily compared.","marker":"[6]"},{"why":"Provides the PIA/PIAN baseline attack from which FCRE borrows the proximal-initialization comparison protocol.","marker":"[11]"},{"why":"Introduces DDIM, the deterministic sampler whose invertibility FCRE exploits for reconstruction error estimation.","marker":"[19]"},{"why":"Documents per-step inversion errors in DDIM, the phenomenon that makes reconstruction asymmetry a membership signal.","marker":"[16]"},{"why":"Supplies the general difficulty-calibration perspective that motivates the frequency-based calibration in FCRE.","marker":"[3]"},{"why":"Argues that difficulty calibration is necessary for practical membership inference, providing the direct conceptual basis of the proposed method.","marker":"[8]"},{"why":"Defines SSIM, the perceptual similarity metric used in the FCRE membership score.","marker":"[22]"},{"why":"Provides the FeTS 2022 brain MRI dataset used to train the target diffusion models and to evaluate FCRE.","marker":"[17]"},{"why":"Provides the ChestX-ray8 chest X-ray dataset used as the second medical benchmark.","marker":"[21]"},{"why":"Supplies the loss-based membership inference baseline that FCRE also outperforms.","marker":"[15]"}],"fun_headline_variants":["Mid-frequency mask improves membership attacks on medical diffusion models","Frequency-calibrated score beats SecMI on brain MRI and chest X-rays","Excluding high-frequency noise sharpens membership inference on medical images","FCRE: mid-band reconstruction errors unmask training images"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a patch mask fixed by 15% and 85% Laplacian-score thresholds on the candidate image picks out a consistent mid-frequency band in both the noisy and reconstructed images, and that these thresholds, chosen on the same datasets used for evaluation, will hold on new medical data.","fun_headline_variants_meta":{"raw":{"variants":["Mid-frequency mask improves membership attacks on medical diffusion models","Frequency-calibrated score beats SecMI on brain MRI and chest X-rays","Excluding high-frequency noise sharpens membership inference on medical images","FCRE: mid-band reconstruction errors unmask training images"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001199,"raw_usage":{"total_tokens":4955,"prompt_tokens":972,"completion_tokens":3983,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":3912}},"tokens_in":588,"tokens_out":3983,"duration_ms":29428,"temperature":1.0,"reasoning_tokens":3912,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:09:05.931686+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run FCRE with fixed thresholds on a medical dataset with a different frequency profile, for example CT or ultrasound images where low-frequency background dominates; if the AUC drops to the SecMI baseline or below, the reported advantage is an artifact of threshold selection rather than a general property of mid-frequency reconstruction error.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines SecMI, the step-wise error baseline whose inversion-symmetry mechanism FCRE extends and against which FCRE is primarily compared."},{"cited_title":"In: 2022 IEEE Symposium on Security and Privacy (SP)","cited_arxiv_id":null,"evidence_quote":"Supplies the general difficulty-calibration perspective that motivates the frequency-based calibration in FCRE."},{"cited_title":"In: Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Commu- nications Security","cited_arxiv_id":null,"evidence_quote":"Argues that difficulty calibration is necessary for practical membership inference, providing the direct conceptual basis of the proposed method."},{"cited_title":"In: 2023 IEEE Security and Privacy Workshops (SPW)","cited_arxiv_id":null,"evidence_quote":"Supplies the loss-based membership inference baseline that FCRE also outperforms."}],"review_version":1}