{"id":"92be98b4-df45-4519-b092-8289758c3e1f","arxiv_id":"2505.08576","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"MUBox benchmarks 23 unlearning methods on six scenarios and 11 metrics, showing that method rankings depend heavily on the scenario and metric chosen.","lead":"This paper introduces MUBox, a benchmark that runs 23 machine unlearning methods across six data-removal scenarios and 11 metrics on two image datasets. It finds that even top published unlearning methods work inconsistently across scenarios, and that no single metric reliably measures unlearning success.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unvalidated method implementations and non-identical base models leave the core inconsistency finding unverified.","rationale":"The reader's weakest assumption is implementation fidelity; this is exactly the load-bearing point. The paper's own contribution includes a reproducibility study, so it should meet the same standard: no comparison to original results, no random seeds, and three methods on different base models. The code being public is real evidence and helps, but public code can contain bugs; the competition entries are not peer-reviewed. The 'inconsistent effectiveness' claim is the strongest version of the abstract, and it is supported by Tables 1-12; if the implementation of a method is not faithful, the rankings in those tables cannot be trusted. A reviewer can still say the benchmark is useful and the negative findings are plausible, but conditional on validation. Therefore, no change to the reader's conditional verdict is needed.","tokens_in":30891,"tokens_out":7592,"duration_ms":81853,"concrete_test":"Take six conclusion-driving methods (e.g., SCRUB, SalUn, Bad-T, SSD, PGU, and Second-order) and run the original authors' released code on the same CIFAR-10 splits for the forgetting-from-one-class and class-wise scenarios, using each paper's own training recipe instead of MUBox's 200-epoch cosine schedule. Compare TA, RA, FA, and MIA against Tables 1 and 4. If any method's results shift by more than 5 percentage points or move across the paper's effectiveness boundary, rerun the MUBox analysis with corrected implementations to see whether the inconsistency finding survives. Separately, rerun Fisher, NTK, and L-CODEC on the full-size ResNet-18 used for other methods (or, if infeasible, state the feasibility limit and exclude them from cross-method rankings) to test whether the 40%-size base model changes their utility and forgetting results.","verdict_should_be":"UNCHANGED","load_bearing_attack":"MUBox's central finding is that state-of-the-art unlearning methods are inconsistent across scenarios. That conclusion depends on each implemented method being the method named in the table and on all methods being compared on the same base model. Neither condition is demonstrated. Section 3.2 says the same unlearning hyperparameters as the original papers are used, but the appendix reports material deviations: Fisher, NTK, and L-CODEC run on a ResNet-18 reduced to 40% of the size used for every other method; Fisher/NTK are also pre-trained on CIFAR-100; Amnesiac is trained for only 50 epochs. A 40%-scale model can have very different loss geometry and forgetting behavior, so rows for these methods are not comparable to the rest. The paper also never validates any of the 23 implementations against the original papers' reported numbers; competition entries FCS, MSG, CT, and NIU are taken directly from Kaggle discussion threads. If one or two implementations are buggy, or the reduced-model rows drive a scenario's conclusion, the headline 'inconsistent effectiveness' could be an artifact of the evaluation harness rather than a property of the methods. This is load-bearing because the paper's abstract and Section 1 generalize from these exact tables.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents MUBox, a benchmark and evaluation platform for deep machine unlearning. It integrates 23 unlearning methods, six unlearning scenarios, and eleven evaluation metrics, and it reports experiments on CIFAR-10 (main text) and TinyImageNet (appendix). The main empirical claims are that state-of-the-art unlearning methods, including competition winners, show inconsistent effectiveness across scenarios; that no single metric adequately captures unlearning quality; and that depoisoning performance varies strongly with the attack type. The paper also contains a reproducibility survey of 49 unlearning papers.","tokens_in":31111,"tokens_out":5587,"duration_ms":58016,"significance":"If the empirical findings are robust, MUBox would be a useful community resource: it offers broad method coverage, a public codebase, a systematic scenario taxonomy, and a concrete reproducibility audit of prior work. The headline finding that methods with strong papers or competition results fail outside the simplest settings is important and would justify more diverse evaluation protocols. The reproducibility statistics and the explicit multi-metric comparisons are also valuable. However, the current evidence does not yet support the strength of the claims, because several load-bearing aspects of the evaluation protocol are not demonstrated: the FA metric is reported on inconsistent scales, the base models are not identical across methods, implementations are not validated against original results, and all numbers come from single runs.","major_comments":[{"comment":"The FA metric is not defined or reported consistently. The text defines FA as the accuracy change of the unlearned model on D_f compared with the Retrain model, but in every table the Original and Retrain rows report raw forgetting accuracy (e.g., Table 1 shows Retrain FA = 93.40 and Original FA = 100.00), while all other rows report signed differences (e.g., PGU FA = -1.60, SSD FA = -93.40). This makes the cross-row comparisons in the text, such as 'only a 1.6% decrease' and the ranking arguments in Section 4, formally invalid unless the reader manually converts to a common scale. Please report a single consistent FA definition, or report both raw forgetting accuracy and the deviation from Retrain in separate columns.","section":"§3.4 and Tables 1, 3-6"},{"comment":"Not all methods are evaluated on the same base model, despite the abstract and Section 3.2 claiming a unified comparison. The appendix states that Fisher and NTK use a ResNet-18 reduced to 40% of the original size and pre-trained on CIFAR-100, that L-CODEC also uses the 40%-scale ResNet-18, and that Amnesiac is trained for only 50 epochs. Different architectures, training durations, and pretraining distributions change loss geometry and forgetting behavior, so the rows for Fisher, NTK, L-CODEC, and Amnesiac are not directly comparable with the rows for the other methods. The paper should either run all methods on the same base model, or present these rows in a clearly separated section and remove them from the cross-method inconsistency claims.","section":"§3.2 and Appendix 'Additional Implementation Details'"},{"comment":"The implementations of the 23 methods are not validated against the original papers' reported results. Section 4 says 'we use the same unlearning hyperparameter settings as specified in each original paper,' but no table or figure compares any MUBox result with the accuracy, forgetting, or runtime numbers in the original publications. Four competition methods (FCS, MSG, CT, NIU) are taken directly from Kaggle discussion threads rather than from official codebases. Because the central conclusion is that methods are inconsistent, a single buggy or misconfigured implementation could change a row and alter the finding. Please include a validation table showing, for each method, the original reported metric and the corresponding MUBox reproduction, or explicitly state which implementations could not be validated.","section":"§3.2, §4, and Appendix"},{"comment":"All reported results are single-run numbers with no error bars, confidence intervals, or random-seed information. The main conclusions are comparative rankings across methods, and several distinctions in the text are small, such as FA values of +6.60 versus +4.10 in Table 1. Without multiple seeds, these differences could be noise, and the 'inconsistent effectiveness' finding would be stronger if shown to be stable under training and unlearning stochasticity. Please report at least three to five runs with means and standard deviations for the primary metrics, or clearly label the results as preliminary single-run observations.","section":"§4, Tables 1-6"},{"comment":"The MIA metric set appears to be selected post hoc. In Section 4.1, m_entropy and probability-based MIA are declared 'unreliable' and 'should not be used to evaluate unlearning performance in this set of experiments,' and in Section 4.3 m_entropy is dropped because of its 'failure.' Yet the columns remain in the tables and are later used in the discussion, including the statement that only SSD and SISA are effective 'across all five metrics.' This selective treatment weakens the abstract claim that no single metric can capture unlearning performance, because the conclusion depends on which metrics are retained. Please justify the inclusion or exclusion of metrics before presenting results, or report sensitivity analyses over all five MIA variants consistently.","section":"§4.1 and §4.3"}],"minor_comments":[{"comment":"The typo 'Secone-order' should read 'Second-order' in the model utility paragraph.","section":"§4.1"},{"comment":"There are formatting errors in the numeric entries: Table 3 shows '06.30' and '03.60' in the FCS row, and Table 9 shows '0.78.78' in the NIU row. These should be corrected to standard numeric formatting.","section":"Table 3 and Table 9"},{"comment":"The sentence containing 'while maintaining competitive while maintaining competitive TA and RA' contains a duplicated phrase and should be rewritten.","section":"§4.3"},{"comment":"The abstract states that TinyImageNet results 'will be provided as supplementary material upon acceptance of the paper,' but the appendix already contains Tables 8 through 12 with these results. Please update the wording to reflect the actual structure.","section":"Abstract and Appendix"},{"comment":"The table entry for reference [27] is labeled 'Amnesaic' instead of 'Amnesiac,' and the appendix contains a stray fragment ', i.e., phys_theta_u = A(D_r),' in the 'Key Ideas of Implemented Unlearning Methods' section. Both should be removed or corrected.","section":"Appendix, Table 7"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper is worth reading if you work on machine unlearning evaluation. What's new: it assembles 23 methods, six scenarios, and 11 metrics on CIFAR-10 and TinyImageNet, plus a reproducibility audit of 49 papers. That combination is genuinely broader than the existing benchmarks it cites. The audit numbers (30.6% no code, 50% partial/no README) are concrete and useful. The qualitative findings—methods behave differently across scenarios, no single metric captures efficacy, depoisoning success depends on attack type—are plausible and consistent with what the community suspects.\n\nWhere it gets soft: the load-bearing comparison is not fully controlled. Fisher, NTK, and L-CODEC run on a ResNet-18 shrunk to 40% of the size used for everyone else; Fisher and NTK are also pretrained on CIFAR-100; Amnesiac is trained for only 50 epochs. Those differences can change loss geometry and forgetting behavior, so those rows are not apples-to-apples. The paper puts this in the appendix and is honest about it, but it undermines the 'inconsistent effectiveness' claim drawn from the main tables. The implementations are never validated against the original papers' reported numbers; four competition entries come from Kaggle discussion threads. If any of those is buggy, a scenario's conclusion could shift. Also, all results are single-run, no error bars, so the rankings in Tables 1-12 carry unknown noise. The FA metric is reported as raw accuracy for Retrain but as a delta for every other method; that should be fixed or clearly defined. m_entropy and probability MIA are dropped post hoc in some scenarios because they 'fail,' which is fine, but it would be better decided before running.\n\nThat said, these are fixable weaknesses. The qualitative takeaways probably survive because even among methods on the same base model there is visible variation. What shouldn't be cited yet is the precise quantitative rank order.\n\nWho is this for? Anyone building an unlearning evaluation suite or deciding which baselines to run. It deserves serious peer review, but with the expectation that the authors validate implementations, run multiple seeds, and either make base models identical or explicitly exclude non-comparable rows from the headline claims. I'd bring it to a reading group over the next month.","headline":"Broad, useful unlearning benchmark whose central 'inconsistency' claim is real but quantitatively under-backed by unvalidated implementations and non-identical base models.","tokens_in":31575,"tokens_out":2505,"would_cite":true,"duration_ms":23117,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MUBox claims that even state-of-the-art machine unlearning methods, including competition winners, are inconsistent across different forgetting scenarios, and that no single metric can judge them.","keywords":["machine unlearning","benchmark","evaluation framework","membership inference attack","data poisoning","depoisoning","reproducibility","deep learning"],"falsifier":"Run, say, SSD, SalUn, and PGU under MUBox's exact protocol with several random seeds and the original authors' released code, then check whether the rankings persist and whether the numbers match the original papers; if rankings flip or the reproduced values differ sharply from the original reports, the inconsistency findings could be implementation or noise artifacts.","tokens_in":30728,"feed_emoji":"🧠","tokens_out":6310,"duration_ms":59073,"temperature":0.7,"pith_summary":"MUBox argues that the field's confidence in machine unlearning is largely an artifact of easy benchmarks. The platform runs 23 published unlearning methods through six scenarios — partial-class, all-class, class-wise, worst-case, best-case, and depoisoning — with 11 utility, efficacy, and cost metrics, all on the same models and data. Its central finding: even methods published at top venues and winners of unlearning competitions perform inconsistently once the scenario changes, and no single metric tells you whether forgetting actually happened. If this is right, published comparisons have been over-optimistic, and evaluations need harder scenarios and multiple metrics to be meaningful.","feed_headline":"Unlearning methods fail the consistency test across six scenarios","feed_subtitle":"A unified benchmark of 23 methods shows top-scoring unlearning can collapse when the forgetting scenario changes.","key_machinery":"The load-bearing object is MUBox itself: a modular benchmark that fixes one training setup (ResNet-18 on CIFAR-10 and MobileViT on TinyImageNet) and runs 23 published unlearning algorithms through six forgetting scenarios, scoring each with 11 metrics grouped into model utility (test and retaining accuracy), unlearning efficacy (forgetting accuracy, five membership-inference scores, and $\\ell^2$ parameter distance), and cost efficiency (runtime relative to retraining and storage overhead). The unifying device is the gold-standard Retrain model: every method is compared against a model retrained without the forgotten data. The scenario axis — especially worst-case (lowest-loss) and best-case (highest-loss) forgetting and depoisoning — is what exposes the inconsistency that the paper reports.","core_discovery":"The paper's claim is that current machine-unlearning evaluations, dominated by class-wise and random forgetting, give a misleading picture. In MUBox's controlled comparison, methods that look strong in the standard class-wise scenario often degrade sharply when only part of a class is forgotten, when the forgotten samples are the easiest or hardest for the model, or when the task is depoisoning. The same method can look successful under one membership-inference metric and fail under another; for example, Bad-T and SalUn score near 100% on confidence- and entropy-based membership inference but near 0% on correctness-based membership inference. Parameter-space distance ($\\ell^2$) does not track behavioral difference: SSD and PGU have identical $\\ell^2$ distances yet opposite utility outcomes. For depoisoning, effectiveness is attack-dependent — methods that restore accuracy after label-flipping can still leave a backdoor trigger effective. The paper also reports a reproducibility survey of 49 papers from 2019 to 2024: 30.6% provide no code, and half of the code that exists lacks full instructions.","pith_inferences":["If the inconsistency finding holds, benchmark design should add worst-case and best-case forgetting as default settings, because leaderboards built only on class-wise forgetting may reward methods that are tuned to that one scenario.","The paper's single-run design is a lower bound on reliability; adding multiple seeds and testing hyperparameter sensitivity would separate genuine scenario-dependence from tuning fragility, which the current data cannot fully distinguish.","The depoisoning results suggest unlearning can serve as a poisoning defense only when the attack family is known; an attack-agnostic unlearning defense would need to be evaluated against mixed or adaptive poisoning, which MUBox does not yet cover.","The reproducibility statistics imply that a code-plus-README norm is necessary but not sufficient; a shared harness with fixed scenarios and metrics may matter more than individual implementations for making comparisons meaningful."],"forward_implications":["Class-wise forgetting, the most common benchmark, is the easiest setting; methods that pass it can still fail when only part of a class is forgotten (Amnesiac, SSD, and L-CODEC lose at least 10 accuracy points) or when the forgotten samples are the model's best or worst examples.","A single efficacy metric can flip a method's verdict: Bad-T and SalUn look fully unlearned under confidence- and entropy-based membership inference yet almost un-unlearned under correctness-based membership inference, so multi-metric evaluation is necessary.","Depoisoning success is attack-dependent: many methods restore utility after label-flipping, but for backdoor attacks FCS, MSG, CT, and NIU leave a high attack success rate, so unlearning cannot be treated as a universal poison defense.","Cost claims are method- and budget-specific: most approximate methods are faster than retraining, but L-CODEC can be about eight times slower and Amnesiac needs roughly 428 GB of stored updates, so efficiency must be reported together with storage and unlearning budget.","Methods that work on CIFAR-10 often fail on TinyImageNet: SSD collapses, SISA's test accuracy falls to about 18%, and gradient-based methods degrade on well-trained models, showing poor transfer across datasets and architectures."],"supporting_citations":[{"why":"Supplies the SISA exact-unlearning approach and the sharded-retraining baseline against which approximate methods are compared.","marker":"[2]"},{"why":"Provides the Boundary-S and Boundary-E methods, which show strong class-wise performance but fail on partial-class and worst-case forgetting.","marker":"[6]"},{"why":"Provides the Bad-T teacher-student unlearning method whose membership-inference scores vary sharply across metrics.","marker":"[11]"},{"why":"Provides the SalUn weight-saliency method, whose efficacy conclusions depend heavily on which membership-inference metric is used.","marker":"[19]"},{"why":"Provides the SSD selective-synaptic-dampening method, which preserves utility on CIFAR-10 in some scenarios but collapses on TinyImageNet.","marker":"[21]"},{"why":"Provides the Fisher-information-based method whose utility degrades under the benchmark's hyperparameter settings.","marker":"[25]"},{"why":"Provides the NTK-based method and defines the pretraining and reduced-model setup the paper follows for scalability.","marker":"[26]"},{"why":"Provides the PGU gradient-projection method, which keeps utility close to retrain but shows mixed membership-inference results.","marker":"[36]"},{"why":"Provides the $\\ell^1$-sparsity unlearning method, which preserves utility but exhibits high forgetting-accuracy disparity in several scenarios.","marker":"[38]"},{"why":"Provides the SCRUB teacher-student method whose unlearning efficacy differs across class-wise and partial-class forgetting.","marker":"[41]"}],"fun_headline_variants":["Unlearning methods that top benchmarks fail on harder tasks","No single metric captures unlearning performance; combine metrics","Depoisoning unlearning effectiveness depends on attack type","MUBox: 23 unlearning methods show inconsistent effectiveness"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole comparison rests on the assumption that MUBox's implementations faithfully reproduce the 23 original methods and that a single run with each original paper's hyperparameters is representative enough to rank them.","fun_headline_variants_meta":{"raw":{"variants":["Unlearning methods that top benchmarks fail on harder tasks","No single metric captures unlearning performance; combine metrics","Depoisoning unlearning effectiveness depends on attack type","MUBox: 23 unlearning methods show inconsistent effectiveness"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000729,"raw_usage":{"total_tokens":3317,"prompt_tokens":1047,"completion_tokens":2270,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":663,"completion_tokens_details":{"reasoning_tokens":2205}},"tokens_in":663,"tokens_out":2270,"duration_ms":20084,"temperature":1.0,"reasoning_tokens":2205,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:50:40.037482+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run, say, SSD, SalUn, and PGU under MUBox's exact protocol with several random seeds and the original authors' released code, then check whether the rankings persist and whether the numbers match the original papers; if rankings flip or the reproduced values differ sharply from the original reports, the inconsistency findings could be implementation or noise artifacts.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the SalUn weight-saliency method, whose efficacy conclusions depend heavily on which membership-inference metric is used."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the SSD selective-synaptic-dampening method, which preserves utility on CIFAR-10 in some scenarios but collapses on TinyImageNet."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Fisher-information-based method whose utility degrades under the benchmark's hyperparameter settings."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the NTK-based method and defines the pretraining and reduced-model setup the paper follows for scalability."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the PGU gradient-projection method, which keeps utility close to retrain but shows mixed membership-inference results."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the $\\ell^1$-sparsity unlearning method, which preserves utility but exhibits high forgetting-accuracy disparity in several scenarios."}],"review_version":1}