{"id":"03c90bd2-b4b9-48ba-bbe8-1852c266df90","arxiv_id":"2608.01094","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"On medical imaging federated unlearning, easy client removals make all utility-preserving methods indistinguishable, while hard class-level removals separate them, and residual membership rather than task accuracy is the main erasable signal.","lead":"A new benchmark, Lethe, tests twelve federated unlearning methods on eight medical imaging task families and finds that the difficulty of the forgetting request, not the method, decides whether methods can be told apart. It also finds that on tasks that generalize across hospitals, the task metric barely moves when a client is removed, so residual membership signal is what must be erased.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"F0 alone cannot measure forgetting difficulty: segmentation has high F0 yet is easy because gold F* is also high; the erasable gap F0−F*, not F0, drives separability, so the central claim needs retesting with the correct difficulty variable.","rationale":"The reader's weakest assumption was that F0 measures request difficulty independently of task transfer, so that observed method spread can be attributed to difficulty. My analysis agrees but sharpens it into an internal inconsistency: the paper itself uses F0 as the difficulty readout in Section 4.1, yet Section 4.2 reports high F0 values for segmentation that do not produce method separation because the gold retrain also scores highly on the forget set. Therefore the stated operationalization cannot be what the data show; the operative variable must be something like the gap between the pre-unlearn model and the gold retrain on the forget set. This is load-bearing because every downstream recommendation—evaluate on hard requests, route requests by hardness, target membership rather than task accuracy—presupposes that the benchmark has correctly identified what makes a request hard. The proposed concrete test uses data already in the paper (Tables 3 and 6) and a single controlled manipulation to decide whether F0 or the gap D is the causal difficulty measure. I do not reject the paper; the benchmark is substantial and the qualitative phenomenon is plausible, but the causal framing as written is not yet established. Since the reader's verdict was already CONDITIONAL, my stress-test does not move the verdict; it reinforces the need for the condition.","tokens_in":25529,"tokens_out":6095,"duration_ms":54494,"concrete_test":"Recompute the separability analysis with D = A(M0,Df) − A(M*,Df) as the difficulty variable instead of F0, using data already in Tables 3 and 6. A monotone relationship between D and method spread across all eight task families (including the high-F0, low-D segmentation cases) would support the paper's conclusion but require correcting the stated operationalization; no relationship would refute it. To separate correlation from causation, run a controlled experiment on one dataset (e.g., BloodMNIST): vary training length or regularization to produce F0 in {0.3, 0.5, 0.7} while holding D approximately fixed, then measure whether method spread changes. If spread tracks F0, the current claim is right; if it tracks D, the F0-based difficulty metric in Fig. 2b is the wrong variable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—'difficulty, not method, governs separability'—is operationalized in Section 4.1 as the pre-unlearn forget accuracy F0 (Fig. 2b), and F0 alone is used to explain why client-level removal separates methods on BloodMNIST but not PathMNIST. But F0 conflates two very different regimes. If a model has memorized the forget client, F0 is high and the gold retrain's forget accuracy F* is low, so there is a genuine erasable signal. If the task generalizes across sites, F0 is also high but F* is nearly as high, so there is almost nothing to erase. Section 4.2 is exactly the second regime: segmentation F0 values are 0.57–0.89, yet the gold retrain still scores within a few points of every method and the paper itself calls the forget metric 'nearly vacuous' (Sec. 4.2). Under the stated F0-as-difficulty rule, segmentation would be the hardest setting and should separate methods; observed behavior is the opposite. The quantity that actually tracks the observations is the erasable gap D = F0 − F*, not F0. This is not a minor wording issue: the benchmark's headline conclusion and the recommendation to route requests by hardness (Sec. 4, 'Where the Field Should Go Next') depend on knowing whether F0, D, or some other request property is the causal difficulty variable. The paper's cross-dataset correlation is consistent with D but is also confounded by dataset identity, label skew, and architecture.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Lethe, a benchmark for federated unlearning in medical imaging, spanning eight task families, sixteen datasets, twelve unlearning methods, three forgetting granularities (client, class, sample), and a seven-metric panel evaluated against a retrained gold standard. The central claimed finding is that the difficulty of the forgetting request, rather than the choice of unlearning method, determines whether methods can be distinguished: on easy client-level removals, utility-preserving methods are statistically indistinguishable, while hard class-level and sole-class removals separate methods. A second claim is that on clinical tasks that generalize across sites, client forgetting barely changes task performance, leaving residual membership as the primary erasable signal. The authors validate this with a Friedman-Nemenyi ranking, several robustness checks (heterogeneity, scale, architecture, sample-level forgetting), per-sample membership inference attacks, and case studies, and they release code and data generators.","tokens_in":25801,"tokens_out":3226,"duration_ms":30845,"significance":"If the central claim holds, the paper makes a useful contribution to federated unlearning evaluation: it would show that the field's client-level benchmarks are non-discriminative, that hard requests are necessary for method comparison, and that membership leakage rather than task-metric change is the relevant target on many medical tasks. The benchmark itself is large and well-scoped, with a shared protocol, a gold-standard retrain baseline, per-sample MIA, and cost/durability metrics, and the authors provide code and data-generation scripts, which supports reproducibility. The finding that 'gentle' methods are often statistically indistinguishable is supported by the reported Friedman-Nemenyi analysis, and the qualitatively consistent pattern across many tables is a genuine strength. However, the headline causal claim about difficulty is under-supported because the difficulty proxy used (F0) appears internally inconsistent with the paper's own segmentation results, and the evidence is correlational rather than manipulative.","major_comments":[{"comment":"The paper operationalizes 'forgetting difficulty' as the pre-unlearn forget accuracy F0 and uses it to conclude that difficulty governs separability (Sec. 4.1). This proxy conflates two regimes: high F0 with low gold retrain forget accuracy F* (a real erasable signal) versus high F0 with high F* (task transfer, little to erase). The paper's own segmentation results contradict the stated F0 rule: Table 6 reports pre-unlearn forget Dice F0 of 0.570–0.890, which under the F0-as-difficulty reading would make segmentation the hardest setting and should separate methods, yet the text reports that 'the gold retrain ... still segments its data within a few percentage points of every method' and that 'the forget metric is nearly vacuous' (Sec. 4.2). The quantity that actually tracks the observed separability is the erasable gap D = F0 − F*, not F0. The authors should retest the headline claim using D and should report the correlation of F0 and D with a quantitative measure of method spread (e.g., variance of retain accuracy or the Friedman test statistic). Without this, the recommendation to route requests by hardness (Sec. 4) is not supported.","section":"§4.1, Fig. 2(b), §4.2, Table 6"},{"comment":"Several reported results show exact numerical duplication between methods from different mechanism families: BFU exactly matches GA on OrganAMNIST, PathMNIST, and OCTMNIST (Tables 15–17, F/R columns identical to GA), and Table 7 shows many methods identical to three decimals in the open-ended VQA block. Exact duplication of a Bayesian method (BFU) by gradient ascent strongly suggests an implementation bug or that BFU degenerates to GA in those settings. This is load-bearing for the benchmark's credibility as a method zoo: the Friedman–Nemenyi ranking in App. B.2 treats these as distinct methods, and the conclusion that 'only the gradient-ascent methods fall significantly below' is confounded if some non-ascent methods are actually running ascent. The authors should investigate and clarify these exact matches, or remove the affected columns/rows with an explanation.","section":"§3, Table 3; App. B.2, Fig. 6"},{"comment":"The causal phrasing 'difficulty governs separability' is not justified by the evidence presented. The paper shows correlations across datasets that differ simultaneously in F0, label skew, architecture, and cross-site generalization, but it never manipulates difficulty within a fixed task and dataset while holding other factors constant. The sole-class protocol in App. B.3 (Table 18) is the natural experiment, but it only reports retain accuracy for a subset of gentle methods and does not report F0, F*, or a separability measure for the same dataset under client-level versus sole-class requests. A concrete test would be to vary the number of holders of a class (or the F0–F* gap) within one dataset and show that method spread tracks that variation. Absent such a test, 'difficulty' remains a post-hoc label for the observed pattern rather than a demonstrated causal variable.","section":"§4, 'Where the Field Should Go Next'; App. B.3, Table 18"}],"minor_comments":[{"comment":"The y-axis label reads 'gold forget acc', but F0 is the pre-unlearn model's forget accuracy, not the gold retrain's; please relabel to 'pre-unlearn forget accuracy F0'.","section":"Fig. 2(b)"},{"comment":"The caption says 'Green marks the two retain values nearest gold', but the table is rendered in monochrome in the text; please ensure the color coding is visible or replace with bold/asterisks.","section":"Table 3 caption"},{"comment":"The definition R_k = A(M_u^(k), D_f) is the recovered forget accuracy after k fine-tuning epochs, but the text says 'a durable erasure keeps R_k near M*'; since the gold retrain's forget accuracy is a scalar, please state the comparison target explicitly (e.g., 'near the gold retrain's forget accuracy').","section":"Eq. (4) and following text"},{"comment":"The speedup table reports negative or zero speedup for FedEraser (0.6–1.1×), which is mentioned in the text, but the caption does not note that values below 1 indicate slower-than-retrain; please add a clarifying note.","section":"Table 11"},{"comment":"The open-ended VQA block reports many identical values (e.g., 0.296±0.002 for several methods), which may be due to rounding or the reported precision; if so, please state the rounding convention in the caption.","section":"§4.3, Table 7"}],"recommendation":"major_revision","confidential_remarks":"The benchmark is a substantial resource for the federated unlearning community, and the empirical observation that gentle methods are statistically indistinguishable on easy client-level removals is timely. However, I would not accept the paper in its current form: the difficulty proxy (F0) is internally contradicted by the segmentation results, and the exact BFU-GA duplication suggests a possible implementation error that must be resolved. The authors should be asked to re-analyze the data using the erasable gap F0−F* and to clarify or fix the duplicated rows. If the re-analysis confirms that D rather than F0 tracks separability, the central claim can be stated more precisely; if not, the headline claim should be softened to a correlational observation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should read this one. It is the first federated-unlearning benchmark that takes medical imaging seriously: eight task families, sixteen datasets, twelve methods, three forgetting granularities, and a per-sample membership attack, all scored against a retrained gold standard. The headline empirical pattern is real and important. On easy client-level removals, utility-preserving methods are statistically indistinguishable; on class-level and sole-class removal they separate; and on tasks that generalize across sites, the residual signal is membership, not task accuracy. That is a useful corrective to a literature that mostly evaluates on easy natural-image requests.\n\nWhat is genuinely new is the breadth plus the meta-finding, and the Friedman-Nemenyi ranking in App. B.2 backs the \"gentle methods form one cluster\" claim. The per-sample LiRA and the retrain-relative metric panel are the right evaluation choices.\n\nThe soft spots are real but fixable. First, the difficulty variable is mis-specified. Section 4.1 reads difficulty off the pre-unlearn forget accuracy F0. But segmentation, which they themselves call \"nearly vacuous,\" has F0 values of 0.57-0.89 while the gold retrain still matches every method. Under F0-as-difficulty, segmentation should be the hardest setting; it is the easiest. What actually tracks the observations is the erasable gap D = F0 - F*, the pre-unlearn forget accuracy minus the gold retrain's forget accuracy. The paper never computes D. The correlation story in Fig. 2(b) is consistent with D, but F0 alone cannot support the causal claim \"difficulty governs separability,\" and the \"route requests by hardness\" recommendation depends on knowing which variable hardness is. This is the main revision.\n\nSecond, BFU exactly duplicates GA on OrganAMNIST, PathMNIST, and OCTMNIST in Tables 3 and 15-17, to three decimals. That is not a real method behavior; it is a degenerate run or an implementation artifact. The twelve-method comparison is effectively eleven on those datasets, and the paper should fix or disclose it. Third, several tables (9, 10, 14-17) lack error bars, and the ranking analysis is only on retain accuracy, not on the separability claim itself. The code repo is linked but I have not run it; the duplication above makes me want to before trusting the numbers.\n\nThe citation pattern is fine; self-citations are to relevant federated medical VLM and adapter work, not padding. The math is standard for this kind of empirical benchmark.\n\nBottom line: this deserves a serious referee and probably acceptance after a major revision that re-operationalizes difficulty as an erasable gap and cleans up BFU. It will be citable once those are fixed.","headline":"A serious benchmark with a real meta-finding, but the difficulty variable is measured wrong (F0 instead of the erasable gap) and BFU degenerates to GA on several datasets; revise before relying on the rankings.","tokens_in":26344,"tokens_out":4140,"would_cite":true,"duration_ms":36372,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Forgetting difficulty, not method choice, decides whether federated unlearning methods can be told apart in medical imaging.","keywords":["federated unlearning","machine unlearning","medical imaging","benchmark","membership inference","forgetting difficulty","cross-site generalization","right to be forgotten"],"falsifier":"On a single dataset, hold architecture, method set, and task transfer fixed and vary $F_0$ by changing which classes the forget client holds exclusively; if the spread in retain accuracy among the gentle methods does not increase with $F_0$, the claim that difficulty governs separability fails.","tokens_in":25292,"feed_emoji":"🧠","tokens_out":6797,"duration_ms":58905,"temperature":0.7,"pith_summary":"Lethe is a benchmark for federated unlearning in medical imaging that spans eight task families, sixteen datasets, twelve unlearning methods, and three forgetting granularities. Its central claim is that the difficulty of the forgetting request, not the choice of unlearning method, determines whether methods can be told apart. On easy client-level removals, all methods that preserve utility cluster indistinguishably close to a from-scratch retrained gold standard, and only class-level or sole-class removals widen the spread. On clinical tasks that generalize across sites, forgetting a client barely changes the task metric, so the signal an unlearner must erase is residual membership information rather than task contribution. The paper concludes that federated unlearning evaluation should be difficulty-aware and target membership.","feed_headline":"Hard forgetting requests alone separate unlearning methods","feed_subtitle":"Easy removals make unlearning methods indistinguishable; residual membership is the signal to erase.","key_machinery":"The load-bearing object is the Lethe protocol itself, a shared evaluation where every unlearned model is scored against a gold standard retrained from scratch on the retained data, and where three request granularities (client, class, sample) and task families with different cross-site generalization are varied systematically. Inside that protocol, the operative quantity is $F_0$, the pre-unlearn model's accuracy on the forget set, which the paper uses as a difficulty meter: $F_0$ falls from about 0.49 on BloodMNIST to 0.05 on PathMNIST, and method separability tracks this ordering. The protocol also substitutes a retrain-calibrated per-sample membership attack for population-level attacks, because membership is the signal that survives when task transfer makes forget accuracy vacuous.","core_discovery":"Lethe's central discovery is empirical: what separates federated unlearning methods is the hardness of the forgetting request, read from the pre-unlearn forget accuracy $F_0$, not the algorithm itself. Easy requests, like a departing client whose classes are also held elsewhere, leave utility-preserving methods statistically indistinguishable from each other and from the retrained gold standard, while hard requests—class-level removal, a sole-holder client leaving, hard 3D classification—produce measurable method spread. A second discovery is that many clinical tasks generalize across sites: the gold retrain, which never saw the forget client, still performs on its data within a few points of every method, making forget accuracy a near-vacuous metric. Under a retrain-calibrated per-sample membership attack, residual membership remains detectable for several gentle methods, which is why the paper identifies membership, not task accuracy, as the real erasure target.","pith_inferences":["One extension the paper does not run: use representation-level difficulty, such as the distance between forget-class features and the retained manifold, to see whether methods separate even on easy requests.","The task-transfer result likely generalizes beyond medicine: any federated deployment with redundant sites or strong server-side generalization can hide residual membership behind stable task metrics, so calibrated per-sample attacks should be standard there too.","If $F_0$ truly governs separability, a cheap pre-unlearn measurement could be used at request time to decide whether to skip unlearning, apply a gentle update, or pay for retraining."],"forward_implications":["Benchmarks should stress class-level and sole-class removal, because easy client-level requests cannot separate methods.","Privacy evaluation should report retrain-calibrated per-sample membership-inference scores, since forget accuracy can be matched while residual membership remains.","An unlearning system should estimate request difficulty from class support, sole-provider status, and site skew, and escalate to full retraining only on hard requests.","Client-level method rankings obtained on natural-image, single-heterogeneity benchmarks should not be assumed to transfer to medical imaging.","Forgetting should be treated as a design property: site-scoped or class-scoped adapters that can be dropped or retrained are a more promising direction than post-hoc weight surgery."],"supporting_citations":[{"why":"Supplies the FedAvg training procedure that produces both the pre-unlearn model and the retrained gold standard.","marker":"[2]"},{"why":"One of the twelve methods compared; its gentle update-replay behavior anchors the indistinguishable cluster.","marker":"[4]"},{"why":"A projected-gradient-ascent baseline whose poor retain accuracy marks the destructive end of method spread.","marker":"[5]"},{"why":"The per-sample membership attack that exposes residual membership when forget accuracy already matches retrain.","marker":"[24]"},{"why":"The calibrated multi-reference variant that confirms residual membership at low false-positive rate.","marker":"[25]"},{"why":"Provides the four MedMNIST classification datasets whose pre-unlearn forget accuracy spans 0.05 to 0.49, driving the difficulty comparison.","marker":"[26]"},{"why":"Supplies a harder real clinical classification set used to test whether the difficulty pattern transfers.","marker":"[27]"},{"why":"Provides a natural five-hospital split for clinical classification, grounding the cross-site generalization finding.","marker":"[28]"}],"fun_headline_variants":["Benchmark reveals unlearning methods differ only on hard requests","Hard forgetting requests expose unlearning method gaps","Medical unlearning benchmark: easy requests hide method differences","Unlearning in medical imaging: difficulty matters more than method","Membership erasure key when task performance stays similar"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes that the pre-unlearn forget accuracy $F_0$ measures request difficulty independently of the compared methods and of how much the task transfers across sites, so that method-spread can be attributed to difficulty rather than to architecture, label skew, or task generalization.","fun_headline_variants_meta":{"raw":{"variants":["Benchmark reveals unlearning methods differ only on hard requests","Hard forgetting requests expose unlearning method gaps","Medical unlearning benchmark: easy requests hide method differences","Unlearning in medical imaging: difficulty matters more than method","Membership erasure key when task performance stays similar"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000201,"raw_usage":{"total_tokens":1379,"prompt_tokens":949,"completion_tokens":430,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":354}},"tokens_in":565,"tokens_out":430,"duration_ms":3999,"temperature":1.0,"reasoning_tokens":354,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:12:28.521020+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a single dataset, hold architecture, method set, and task transfer fixed and vary $F_0$ by changing which classes the forget client holds exclusively; if the spread in retain accuracy among the gentle methods does not increase with $F_0$, the claim that difficulty governs separability fails.","supporting_citations":[{"cited_title":"FedEraser: Enabling efficient client-level data removal from federated learning models","cited_arxiv_id":null,"evidence_quote":"One of the twelve methods compared; its gentle update-replay behavior anchors the indistinguishable cluster."},{"cited_title":"Inexact unlearning needs more careful evaluations to avoid a false sense of privacy","cited_arxiv_id":null,"evidence_quote":"The calibrated multi-reference variant that confirms residual membership at low false-positive rate."},{"cited_title":"MedMNIST v2-a large-scale lightweight benchmark for 2D and 3D biomedical image classification.Scientific data, 10(1):41, 2023","cited_arxiv_id":null,"evidence_quote":"Provides the four MedMNIST classification datasets whose pre-unlearn forget accuracy spans 0.05 to 0.49, driving the difficulty comparison."},{"cited_title":"Kvasir: A multi-class image dataset for computer aided gastrointestinal disease detection","cited_arxiv_id":null,"evidence_quote":"Supplies a harder real clinical classification set used to test whether the difficulty pattern transfers."},{"cited_title":"From detection of individual metastases to classification of lymph node statusatthepatientlevel:thecamelyon17challenge.IEEEtransactionsonmedicalimaging, 38(2):550–560, 2018","cited_arxiv_id":null,"evidence_quote":"Provides a natural five-hospital split for clinical classification, grounding the cross-site generalization finding."}],"review_version":2}