{"id":"8259fbfe-2b09-4b9e-9821-ac07cfd173b5","arxiv_id":"2411.17126","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"ETID combines leave-one-out ensemble training with distillation-based unlearning to erase requested training samples while keeping model accuracy and consistency with retrained models.","lead":"This paper proposes ETID, a framework that lets companies delete specific users' data from trained predictive models without retraining from scratch. It trains an ensemble where each model skips a different slice of data, then uses the models that skipped the deleted slice as guides to remove that data from the others.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Consistency rests on equating data-overlap 'alikeness' with output-level closeness; Eq. 3 only aligns on Du, and Prop. 2 treats distilled references as freshly retrained models, so sustained consistency with naive retraining is unproven.","rationale":"The paper's contribution is plausible: ROEL's leave-one-part-out construction genuinely provides high data overlap, and the two-dataset experiments show ETID beats several baselines on single-round aggregate consistency, accuracy, efficiency, and verifiability metrics. The problem is that the proof language overstates what is established. The central term 'retrained-alike' is defined purely by data overlap, which makes Proposition 1 true by construction, but the framework then uses that term as if it carried an output-level equivalence guarantee. This is not a disagreement with an outside consensus; it is an internal gap between Definition 3/4 and the consistency desideratum in Definition 1. Proposition 2 compounds the issue by treating distilled unlearned models as if they were freshly trained on their remaining data, so an overlap-based condition cannot be an if-and-only-if criterion for the model that actually serves as the next reference. A two-round experiment would settle whether the gap is harmful in practice. Given the substantial empirical support for the single-round claim and the absence of a direct contradiction of the reported results, conditional acceptance remains appropriate; the authors should either weaken the formal claims or add explicit output-level validation and repeated-round experiments.","tokens_in":27916,"tokens_out":7608,"duration_ms":74604,"concrete_test":"Conduct a two-round unlearning experiment on CIFAR100 with K=5: after the first TID round, update references as in Eq. 5; then issue a second deletion request in a different data part and apply TID without retraining any reference from scratch. After each round, compute the per-sample L2 distance (and accuracy) of the ETID ensemble against a freshly retrained Retrain-ETID on a fixed held-out test set. If the round-2 mean L2 distance is substantially larger than the round-1 value (or exceeds the gap between two independently trained models), Proposition 2's assumption that updated references remain retrained-alike is falsified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Definition 3 and Definition 4 define a 'retrained-alike' model solely by the proportion of shared training samples; they impose no behavioral condition on the algorithm A. Proposition 1 therefore proves only a data-overlap fact: for Du⊂di, Mi shares (K-2)/K of its data with the true retrained model M_rt^j=A(D_{-j}\\Du). For deep networks (the experiments use MLPs and ResNet18), this overlap does not imply that Mi's predictions, on Du or anywhere else, are close to M_rt^j's with any quantitative bound. The TID objective (Eq. 3) then forces the target sub-model to match Mi's outputs only on Xu, not on arbitrary inputs, so nothing in the construction constrains the unlearned model to match M_rt^j on remaining or test data. The reported low Con(·) values are empirical aggregate L2 numbers from two datasets, not a consequence of the proof structure.\n\nThe gap is sharper in Proposition 2. After a first request, Eq. 5 replaces each reference model with the corresponding unlearned sub-model; that model has been modified by distillation (Eq. 3) and rectification (Eq. 4), so it is no longer a model of the form A(Dr_{-i}). Proposition 2's proof nonetheless counts 'the remaining data of M_rt^i and Mj' and states an if-and-only-if condition (Eq. 6) on data overlap, as if the updated reference were freshly retrained. The condition therefore does not characterize the object that will actually serve as the reference in the next round, and the 'if and only if' is unsupported. Since the experiments cover only one unlearning round, the repeated-request regime—essential to the business-value claim of handling frequent erasure requests—is exactly where the reference-update argument fails.\n\nThe verifiability proof (Prop. 3) similarly asserts Mj(Xu)≠Mi(Xu) without justification; on high-confidence training points the outputs may coincide, so even the distinguishability argument is conditional.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a holistic machine learning-to-unlearning framework, ETID, consisting of a reference-oriented ensemble learning method (ROEL) for model construction and an iterative information distillation method (TID) for responding to data-erasure requests. ROEL trains K overlapping sub-models, each on a leave-one-part-out subset, and TID distills unlearning data from affected sub-models under the supervision of sub-models that are claimed to be 'retrained-alike' references, then rectifies models on remaining data. The paper claims that ETID simultaneously achieves consistency, accuracy, efficiency, and verifiability, and presents experiments on two datasets (Purchase and CIFAR100) comparing against several unlearning baselines.","tokens_in":28278,"tokens_out":4647,"duration_ms":40881,"significance":"If the claims are established, ETID would be a practically valuable method for GDPR-style right-to-be-forgotten compliance, potentially enabling efficient data erasure while preserving predictive accuracy. The paper is among the first to frame machine unlearning as a holistic two-stage business problem, and the empirical study covers multiple desiderata and includes parameter sensitivity analysis. The framework is simple and reproducible in principle. However, the theoretical guarantees are substantially weaker than the abstract suggests: the 'retrained-alike' notion is defined purely by data overlap, not behavioral similarity, and the proofs of Propositions 2 and 3 rely on unjustified assumptions. The empirical results on two datasets are encouraging but do not compensate for the missing theoretical support for the central consistency claim.","major_comments":[{"comment":"The 'retrained-alike' notion is defined solely by the proportion of shared training samples, with no behavioral condition on the learning algorithm A. Proposition 1 therefore establishes only a data-overlap fact: Mi shares (K−2)/K of its data with the true retrained model M_rt^j. For deep networks (MLPs and ResNet18 in the experiments), data overlap does not imply that predictions are close on any input distribution, and no quantitative bound is given. Since the TID objective (Eq. 3) aligns outputs only on the unlearning data Xu, nothing in the construction constrains the unlearned model to match M_rt^j on remaining or test data. The abstract's claim that ETID delivers predictions 'consistent with those of naïve retrained models' is therefore not supported by the proof structure.","section":"§4.1, Definitions 3–4 and Proposition 1"},{"comment":"After a first unlearning request, Eq. 5 updates each reference model to the corresponding unlearned sub-model, which has been modified by distillation (Eq. 3) and rectification (Eq. 4). Such a model is no longer of the form A(Dr_-i), i.e., it is not a freshly retrained model on the remaining data. The proof of Proposition 2 nonetheless treats the updated reference as if it were freshly retrained, deriving an if-and-only-if condition (Eq. 6) purely on data overlap. The condition therefore does not characterize the object that actually serves as the reference in the next round, leaving the iterative consistency claim underived.","section":"§4.2.4, Proposition 2"},{"comment":"The proof of distinguishability assumes that Mj(Xu) ≠ Mi(Xu). This inequality is not guaranteed; in fact, if Mi is a good retrained-alike reference, its outputs on unlearning data may coincide with those of Mj. The argument that the difference between M^u_j(Xu) and Mj(Xu) can serve as a verification function therefore requires a behavioral separation that the paper does not establish. As a result, verifiability is not proven by the given reasoning.","section":"§4.3, Proposition 3"},{"comment":"The consistency results are aggregate L2 distances between the unlearned and true retrained models, computed on two datasets. These empirical numbers are favorable, but they cannot substitute for the missing theoretical guarantee, because the proofs in Section 4 only establish data-overlap relations, not output-level closeness. The paper should either provide a rigorous behavioral bound connecting data overlap to prediction proximity for the algorithm class used, or explicitly reframe the consistency claim as an empirical finding rather than a proven property.","section":"§5.2.2, Table 7"}],"minor_comments":[{"comment":"The section title contains a typo: 'Reference-Oriented Ensemble Leaning' should be 'Learning'. The abbreviation 'E.q.' appears in several equations (e.g., Eq. 3 and Eq. 4) and should be 'Eq.'.","section":"§4.1"},{"comment":"Several table cells contain repeated values in a single cell (e.g., '0.999 ± 0.0010.999 ± 0.0010.999 ± 0.001' in Table 6 and '8.63 ± 0.248.63 ± 0.248.63 ± 0.24' in Table 9). These appear to be copy-paste formatting errors and should be corrected.","section":"Tables 6, 9, 11–14"},{"comment":"The default K=5 is justified by a reference to Breiman (1996), but the sensitivity analysis does not connect the degradation at large unlearning ratios to the condition in Eq. 6. A discussion of when Proposition 2's condition fails (e.g., for UR=10% on either dataset) and how the recommended retraining-from-scratch remedy applies would make the practical guidance more concrete.","section":"§5.1 and §5.2.6"},{"comment":"No code or detailed hyperparameters are provided (e.g., learning rates, number of epochs for distillation and rectification, optimizer choices). Given that the paper makes quantitative claims about efficiency and consistency, releasing an implementation or at least full training details would materially improve reproducibility.","section":"§5 (entire experimental section)"},{"comment":"The statement that adopting ETID could help companies 'save up to EUR 20 million' is speculative and not derived from the experiments. The managerial implications would be stronger if such quantitative claims were either removed or accompanied by a clearly stated model and assumptions.","section":"§6"}],"recommendation":"major_revision","confidential_remarks":"The paper appears aimed at an IS/business audience rather than a core machine learning venue; the methodological novelty relative to the existing ML unlearning literature is modest, but the framing as a holistic business problem is useful. The theoretical gaps in Propositions 2 and 3 are substantial and need to be addressed explicitly, either by strengthening the assumptions (e.g., restricting to model classes where data-overlap implies behavioral closeness) or by softening the claims to empirical ones. The absence of code is a concern given the reproducibility expectations for empirical ML papers, but the two-dataset study is reasonably thorough if the theoretical overreach is curtailed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful core here is ROEL, a leave-one-out ensemble that trains K sub-models on all-but-one equal partitions, so each sub-model is a natural distillation reference for the others without extra training. TID then does the obvious thing: distill the unlearning data toward that reference and rectify on remaining data. The combination is simple, sensible, and the experiments on Purchase and CIFAR100 show it beats standard baselines on consistency, accuracy, and speed. That empirical case is the real contribution.\n\nThe soft spots are in the theory, and they mostly match the stress-test note. Proposition 1 is true by the paper's own definition, but the definition of 'retrained-alike' is purely a data-overlap condition. It never shows that overlapping training data makes the reference's outputs close to a true retrained model, which is what distillation actually needs. For deep networks, that gap is not cosmetic. The low Con(·) values are empirical, not consequences of the proof, and the paper would be stronger if it said so plainly.\n\nProposition 2 is the weakest spot. After the first round, the updated reference is an unlearned sub-model that has been distilled and rectified, so it is no longer of the form A(Dr). The if-and-only-if condition on data overlap does not characterize that object, and the proof just reuses the Proposition 1 argument. Since experiments run only one unlearning round, the repeated-request regime that matters for the business-value story is exactly where the paper's formal support breaks down. Proposition 3 also assumes output differences that are not guaranteed.\n\nNone of this kills the empirical claim. What it means is: the paper is a good systems/empirical study wearing a theory hat that does not fit. No code or full training details are provided, which makes the experiments harder to trust than they should be.\n\nWho is this for? Readers in business/IS who want a practical unlearning method with accuracy preservation will get real value. A CS/security venue would likely find it incremental and the proofs too weak. A serious referee could improve it by asking for code, multi-round experiments, and either proofs of behavioral closeness or a clear downgrade of the propositions to heuristics. I would send it to review, not desk-reject.","headline":"A practical ensemble-plus-distillation unlearning framework with solid empirical results on two datasets; the theoretical wrapper overclaims consistency and the repeated-request case is unproven.","tokens_in":28874,"tokens_out":1626,"would_cite":false,"duration_ms":18084,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ETID shows that machine unlearning can be fast, accurate, and consistent with naive retraining, all at once.","keywords":["machine unlearning","right to be forgotten","GDPR compliance","ensemble learning","knowledge distillation","data erasure","predictive analytics","model consistency"],"falsifier":"Take a ROEL ensemble with K=5, erase one entire data part, and compare the TID-unlearned model against a model naively retrained without that part; if their L2 prediction distance is no smaller than the distance between two independent retrains on identical data, the retrained-alike premise is not doing the work claimed.","tokens_in":76,"feed_emoji":"🧠","tokens_out":12161,"duration_ms":162558,"temperature":0.7,"pith_summary":"The paper tries to show that honoring a data-erasure request need not mean paying for a full retraining or accepting a degraded predictive model. It proposes ETID, a two-stage framework: build an ensemble whose sub-models are trained on overlapping slices of the data, then erase any requested samples by distilling the affected sub-models toward sibling sub-models that have never seen those samples. The claim is that this achieves all four desiderata of machine unlearning—consistency with naive retraining, accuracy, efficiency, and verifiability—simultaneously, which previous methods trade off against one another. If correct, companies subject to GDPR's Right to Be Forgotten could erase data in seconds rather than retraining from scratch, while keeping service quality and avoiding fines for both non-compliance and business loss.","feed_headline":"Overlapping sub-models make unlearning fast without hurting accuracy","feed_subtitle":"Each sub-model teaches its siblings to forget, cutting erasure to seconds while matching retrained predictions.","key_machinery":"The load-bearing object is the retrained-alike model: a model trained by the same algorithm on data that overlaps a target model's training set by at least a factor of one in shared versus unique samples, so that it can serve as a distillation reference for what the target would look like if retrained without a specific block of data. ROEL manufactures these references for free by training $K$ sub-models on leave-one-part-out subsets; TID then applies the standard distillation objective—minimizing the divergence between the reference's outputs and the target sub-model's outputs on the samples to be erased—iterates over all affected parts, rectifies the sub-models on the remaining data, and updates the references. The same overlap condition, stated as Proposition 2, governs whether the references remain valid after many rounds of unlearning.","core_discovery":"The paper's central claim is that a carefully constructed ensemble makes unlearning nearly free without sacrificing the model's business value. With Reference-Oriented Ensemble Learning (ROEL), the training data is split into $K$ equal parts and $K$ sub-models are trained, each on all parts except one; because every pair of sub-models shares $(K-2)/K$ of its training data, the sub-model that excludes a requested sample's part is, by the paper's definitions, a 'retrained-alike' model of any sub-model that saw that part. The iterative information distillation stage (TID) then uses that sibling sub-model as a reference teacher, fine-tuning each affected sub-model so its outputs on the erased samples match the reference, rectifying with the remaining data, and updating the reference for future requests. The paper argues these steps erase the target samples' influence while keeping predictions close to what a naïve retrained model would have produced, and reports experiments on consumer profiling and image classification where ETID meets all four machine-unlearning desiderata simultaneously.","pith_inferences":["Editorial inference: each unlearning round that preserves the Proposition 2 overlap condition refreshes the reference models, so the framework amortizes across many deletion requests without a full retraining.","Editorial inference: a natural stress test is to report per-sample consistency on rare classes and on the erased samples themselves, since the paper's consistency numbers are aggregate L2 distances over all predictions.","Editorial inference: the same overlap-and-distill recipe could be ported to distributed model training by letting participants share a small common anchor set, so one participant's model can serve as another's erasure reference."],"forward_implications":["Deletion requests can be answered at a tiny fraction of retraining cost: ETID erases 1% of the training data in 8.63 seconds on the Purchase task and 958 seconds on CIFAR-100 with parallel computing, versus 43.6 and 8,295 seconds for single-model naïve retraining.","The unlearned model stays accurate for remaining, test, and unlearning samples, with test accuracies of 0.952 on Purchase and 0.776 on CIFAR-100, exceeding the best comparison methods.","The distilled model's L2 prediction distance to a naïve retrained model is smaller than that of the leading non-retrained baselines on both datasets, so consistency and compliance are not bought at the cost of accuracy.","Membership-inference AUC drops significantly after unlearning, so the erased samples are no longer recognizable as training members, satisfying verifiability.","Because the ensemble's sub-models are built to be retrained-alike from the start, the same model stock serves future requests without extra reference training, as long as the overlap condition in Proposition 2 holds."],"supporting_citations":[{"why":"Defines the SISA ensemble-unlearning baseline and the partial-retraining paradigm that ROEL is designed to improve.","marker":"Bourtoule et al. 2021"},{"why":"Supplies the survey framing and the four desiderata used to define the unlearning problem.","marker":"Xu et al. 2023"},{"why":"Provides the Forsaken distillation baseline and the membership-inference verification protocol.","marker":"Ma et al. 2022"},{"why":"Provides the SCRUB distillation baseline, the strongest non-retrained consistency comparator in the experiments.","marker":"Kurmanji et al. 2023"},{"why":"Provides the Fisher theory-based baseline and the L2 distance convention for measuring consistency.","marker":"Golatkar et al. 2020a"},{"why":"Establishes the distillation objective that TID adapts for erasure.","marker":"Hinton et al. 2015"},{"why":"Documents memorization in deep networks, the reason data deletion requires altering the model rather than only deleting files.","marker":"Arpit et al. 2017"}],"fun_headline_variants":["Ensemble unlearning: erase data in seconds, keep accuracy","Forget fast, stay accurate: ETID unlearning framework","GDPR unlearning in seconds, model quality intact","New unlearning method: fast erasure, same predictions","Ensemble-based unlearning: speed without accuracy loss"],"cache_read_input_tokens":30848,"weakest_assumption_plain":"The guarantee rests on treating two models trained on almost the same data as interchangeable teachers: for deep networks, high data overlap does not by itself ensure that the two models make nearly the same predictions.","fun_headline_variants_meta":{"raw":{"variants":["Ensemble unlearning: erase data in seconds, keep accuracy","Forget fast, stay accurate: ETID unlearning framework","GDPR unlearning in seconds, model quality intact","New unlearning method: fast erasure, same predictions","Ensemble-based unlearning: speed without accuracy loss"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000842,"raw_usage":{"total_tokens":3689,"prompt_tokens":986,"completion_tokens":2703,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":602,"completion_tokens_details":{"reasoning_tokens":2622}},"tokens_in":602,"tokens_out":2703,"duration_ms":17525,"temperature":1.0,"reasoning_tokens":2622,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:29:42.683764+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a ROEL ensemble with K=5, erase one entire data part, and compare the TID-unlearned model against a model naively retrained without that part; if their L2 prediction distance is no smaller than the distance between two independent retrains on identical data, the retrained-alike premise is not doing the work claimed.","supporting_citations":[{"cited_title":"A., Jia, H., Travers, A., Zhang, B.,","cited_arxiv_id":null,"evidence_quote":"Defines the SISA ensemble-unlearning baseline and the partial-retraining paradigm that ROEL is designed to improve."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the SCRUB distillation baseline, the strongest non-retrained consistency comparator in the experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents memorization in deep networks, the reason data deletion requires altering the model rather than only deleting files."}],"review_version":1}