{"id":"beecb829-ee16-4383-8724-066bbc797693","arxiv_id":"2412.00761","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A diffusion-based hypernetwork can generate classifier weights with near-zero accuracy on a requested forget class and near-retrained accuracy on retained classes.","lead":"This paper uses a diffusion model as a hypernetwork that generates new weights for a classifier, weights that fail on chosen classes while staying accurate on others. A single such generator could serve many different forgetting requests without retraining, though the evidence comes from small MNIST models and selected best samples.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Zero-forget result is best-of-N selection, not a property of typical samples; without reporting the full sampling distribution the central claim is unsupported.","rationale":"The reader's verdict is CONDITIONAL with high confidence, and I agree with that verdict. My stress-test identifies the same empirical weakness the reader mentions in the rationale, but frames it as the primary load-bearing concern: the zero-forget result is obtained by selecting the best sample, so the central claim about 'sampled unlearned models' is not a claim about the sampler's typical behavior. This is more directly damaging than the behavioral-proxy assumption, because even if high loss on forget classes were a perfect proxy for unlearning, the reported numbers would still not support the stated claim without a distributional analysis. The paper's own figures show a wide spread of sampled forget accuracy, so the selection issue is not hypothetical. However, the underlying idea, generating unlearned models with a conditional hypernetwork, is novel and worth pursuing, and the authors explicitly acknowledge major limitations in Section 6. A CONDITIONAL verdict remains appropriate: the direction is promising, but the claims need revision to report typical rather than selected behavior, and the evaluation needs explicit baselines and error bars. I would not move to ACCEPT or REJECT on the basis of this concern alone.","tokens_in":24863,"tokens_out":4917,"duration_ms":49882,"concrete_test":"Using the same checkpoint data and DiHyFo training procedure, draw N=100 independent samples per forget set without applying the Appendix A.1 selection rule. Report the full histogram of forget-set accuracy and retain-set accuracy, plus the mean and 95% CI, and compare with the retrained baseline. Specifically, compute the fraction of samples with forget accuracy below 1%; if this fraction is not close to 1, the 'zero performance' claim is an artifact of best-of-N selection. As a secondary check, for the selected best model, record the predicted-label distribution on forget-set inputs and verify that it matches the retrained model's label distribution rather than collapsing onto a single degenerate class.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline result is not a property of the sampling distribution, it is a property of a hand-picked sample. Appendix A.1 states: 'We save the sampled model that obtains the lowest average accuracy on the forget set while obtaining the highest possible average accuracy on the retain set (best unlearned model).' Tables 1-4 then report only these selected models, so the reported zero accuracy on the forget set is the lower envelope of a broad distribution, not the typical outcome. Figures 3-4 show sampled models spanning 0-100% forget accuracy, confirming the distribution is wide, yet the paper does not report how many samples were drawn per forget set or the mean/median behavior of the sampler. Because the selection criterion is exactly the success metric (low forget accuracy, high retain accuracy), the evaluation is circular with respect to the claim that 'sampled unlearned models effectively achieve zero performance.' Additionally, selecting for low forget accuracy without constraining where forget inputs are mapped can select degenerate classifiers that always emit a fixed retain class. The output-overlap and JSD comparisons are computed only after this selection and cannot distinguish a genuinely unlearned model from a broken one. The Section 6 concession that the generative model retains forget-set knowledge further confirms that zero accuracy is a behavioral state of selected parameters, not evidence of removed data influence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HyperForget, a framework for machine unlearning in which a diffusion-based hypernetwork generates parameters for a classifier conditioned on per-class target losses. Two implementations, DiHyFo-1 and DiHyFo-2, are trained on checkpoints from MLPs trained on MNIST and are used to sample models intended to achieve high accuracy on retain classes and low accuracy on forget classes. The authors report that the sampled models achieve zero accuracy on the forget sets while maintaining retain accuracy close to retrained models, and claim this demonstrates a promising approach for adaptive unlearning.","tokens_in":25120,"tokens_out":5279,"duration_ms":46157,"significance":"If the results were supported, the idea of amortizing unlearning through a trained hypernetwork that can sample unlearned models for multiple forget sets without further gradient updates would be a novel contribution to the machine unlearning literature. The paper is honest about several limitations, including the generative model retaining forget-set knowledge, and it provides a detailed appendix on data collection and evaluation. However, the central empirical claim is compromised by the selection procedure and circular evaluation, as detailed below.","major_comments":[{"comment":"The claim that 'the sampled unlearned models effectively achieve zero performance on forget sets' is not supported by the sampling distribution. Appendix A.1 states that the authors save the model with the lowest forget accuracy and highest retain accuracy among all sampled models, and only those selected models are reported in Tables 1–4. Figures 3–4 show that non-selected samples span 0–100% forget accuracy, and Table 5 reports prompt alignment values as low as −10.7378. The paper does not report the number of samples drawn per forget set or the mean/median forget accuracy of the sampler, so the reported zero accuracy is a property of the selection rule, not of typical samples.","section":"Section 5 and Appendix A.1"},{"comment":"The evaluation is circular: the reported success metric (zero forget accuracy, high retain accuracy) is exactly the criterion used to select the 'best unlearned model.' Because the selection step is part of the evaluation pipeline, the Tables 1–4 results are not an independent test of the generator's unlearning capability. The paper should report the distribution of forget/retain accuracies over all sampled models, and specify how many samples were drawn, before claiming that the method 'samples unlearned models.'","section":"Appendix A.1 and Section 5"},{"comment":"The statement that 'the obtained MIA scores for all sampled unlearned models are very close to the retrained models' is contradicted by Table 3. For MNIST-4 with Df={2}, DiHyFo-1 samples have MIA 0.6260, versus 0.4171 for the retrained model; for Df={2,3}, the gap is 0.6484 versus 0.3398. These are large gaps on a scale where lower is better, indicating that the selected models may leak more information about the forget set than retraining would, which is a central privacy claim of the paper.","section":"Section 5 and Table 3"},{"comment":"The paper's own limitation statement that 'the generative model retains the knowledge of forget sets' is in direct tension with the abstract's claim of 'targeted data removal' and Section 5's assertion that the sampled models 'no longer rely on the associated data to make predictions.' At best, the method achieves a behavioral state at the level of the sampled classifier parameters, not removal of data influence from the generative model. The claims should be narrowed accordingly to avoid overstating the unlearning guarantee.","section":"Section 6 and Abstract"},{"comment":"The evaluation does not compare against any existing approximate unlearning method (e.g., fine-tuning on retain set, gradient ascent on forget set, or a competent-teacher baseline). The only baseline is retraining from scratch. Without such comparisons, the claimed benefits of HyperForget—such as speed, adaptability, and robustness relative to the state of the art—are not established. Adding at least one such baseline is necessary to support the positioning of the method.","section":"Sections 4–5"}],"minor_comments":[{"comment":"The captions use 'Dihyfo-1' and 'Dihyfo-2'; for consistency with the text, these should be 'DiHyFo-1' and 'DiHyFo-2'.","section":"Figure 2 captions"},{"comment":"The notation C_increase and C_minimize is used without a formal definition; please define these as subsets of classes in the text.","section":"Equation 3"},{"comment":"The sentence 'we implement two Diffusion HyperForget Networks and used them to sample...' mixes tenses; consider revising to 'use them' for a methods description.","section":"Abstract"},{"comment":"Table 5 reports prompt alignment and correlation for 24 models, but the paper does not state how many models were sampled in total or the selection pool size; including this context would help interpret the averages.","section":"Appendix A.2, Table 5"},{"comment":"The phrase 'with zeros across all entries' in the discussion of confusion matrices is ambiguous; clarify that these are zero rows/columns corresponding to the forgotten classes.","section":"Appendix A.1"}],"recommendation":"major_revision","confidential_remarks":"The paper appears to be a workshop submission (AdvML-Frontiers'24) with a proof-of-concept scope. The idea of using hypernetworks for unlearning is interesting, but the current evaluation does not support the central claims. If the authors can substantially revise to report sampling distributions, add baselines, and reconcile claims with limitations, the contribution may be suitable as a workshop or short paper. The journal should consider whether the scope aligns with its standards."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nWorth reading if you work on machine unlearning, but the headline result doesn't hold as stated. The paper proposes HyperForget, a diffusion-based hypernetwork that generates classifier parameters for class-level forgetting. The novel bit is real: it is the first application of hypernetworks to unlearning, it frames forgetting as a generative process, and the two DiHyFo variants are a sensible adaptation of G.pt to class-level conditioning and deoptimization. The authors are also honest about the big limitation: Section 6 admits the generative model itself retains forget-set knowledge, so this is not privacy-grade unlearning.\n\nThe main problem is the evaluation. Zero accuracy on the forget set is achieved by selecting the best of many sampled models. Appendix A.1 says they save \"the sampled model that obtains the lowest average accuracy on the forget set while obtaining the highest possible average accuracy on the retain set.\" Figures 3 and 4 show the sampled distribution spans 0 to 100% forget accuracy. So the reported zero is the lower envelope of a wide distribution, not the typical outcome. Table 5 confirms high variance: prompt alignment ranges from +0.96 to -10.7. The selection criterion is also the success metric, making the headline claim circular in the sense that they report the selected model's performance as if it were the sampler's performance.\n\nThere are other soft spots. No comparison to established unlearning baselines (e.g., fine-tuning, gradient ascent, amnesiac unlearning). No error bars or confidence intervals, and the selection heuristic could pick degenerate classifiers that always emit a fixed retain class. The experiments are tiny: two-layer MLP on MNIST and MNIST-4. And the MIA comparison, while a good instinct, is only reported on the selected models.\n\nThat said, none of this is fatal to the direction. The idea is plausible and the paper is a proof of concept. What is needed is a revision that reports the typical (median or mean) behavior of the sampling distribution, the number of samples drawn, and a comparison to at least one simple baseline. If the typical sample is poor, the claim should be revised to \"occasionally we can find an unlearned model by search.\" That is a much weaker claim, but still a potentially useful insight for generative unlearning.\n\nWho is this for? Researchers working on unlearning who want to see a fresh architectural idea, and anyone teaching about evaluation pitfalls in generative model papers. I would send it to serious peer review, with a request for major revision. The core idea deserves referee time, but the claims need to match the evidence.\n\nBest,\n[Your name]","headline":"Genuinely new idea but the zero-forget headline is best-of-N selection, so the claims need major revision before the result can be trusted.","tokens_in":25661,"tokens_out":2531,"would_cite":false,"duration_ms":23133,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Machine unlearning can be reframed as generative modelling: a diffusion hypernetwork samples parameters that score zero on forget classes and preserve retain accuracy.","keywords":["machine unlearning","hypernetworks","diffusion models","parameter generation","class-level unlearning","membership inference","MNIST proof-of-concept","adaptive unlearning"],"falsifier":"Take a trained DiHyFo, prompt it to forget class 2 of MNIST, and fit a linear probe on the penultimate-layer activations of the sampled model using forget-class labels. If the probe can decode class-2 structure well above chance while the model's output accuracy on class 2 is zero, the claimed equivalence to a retrained model fails: the network has zeroed the output without removing the information. A complementary check is to compare the sampled model's full output distribution on forget-class inputs with that of a model retrained without class 2; if the sampled model collapses to a single constant vector while the retrained model retains a spread of near-random but structured probabilities, the 'unlearned' model is not behaviourally mimicking retraining.","tokens_in":24669,"feed_emoji":"🧠","tokens_out":8148,"duration_ms":72387,"temperature":0.7,"pith_summary":"This paper aims to show that a model can be made to forget by generating its parameters, not by editing them. The authors introduce HyperForget, a framework in which a hypernetwork—a neural network that outputs parameters for another network—is trained on checkpoints and their per-class losses, then prompted to produce parameters that score high loss on forget-set classes and low loss on retain-set classes. In proof-of-concept experiments on MNIST and MNIST-4, the sampled models achieved zero accuracy on the forget sets while retaining accuracy close to a model retrained without those classes; they also resembled the retrained model in output overlap and in a Jensen-Shannon-based unlearning score. The property the paper highlights is that one trained hypernetwork can sample unlearned models for several different forget sets, so forgetting requests can be served without retraining. The paper itself notes that the generative model still retains forget-set knowledge, which makes the result a behavioural approximation rather than a strict privacy guarantee.","feed_headline":"One diffusion hypernetwork samples models that forget on demand","feed_subtitle":"Trained once, a single DiHyFo serves many forget sets, matching retrained baselines on MNIST.","key_machinery":"The load-bearing object is the diffusion hypernetwork: a diffusion transformer (DiT) that generates the parameters of a small MLP classifier. A hypernetwork is a neural network whose output is the parameter vector of another network; here it is trained on checkpoints from many MLP training runs, with each checkpoint labelled by per-class losses on the classification task. DiHyFo-1 tokenises layers and predicts parameter updates conditioned on current parameters, current losses, target losses, and a diffusion time step, learning both to optimise and to 'deoptimise' (increase loss on forget classes). DiHyFo-2 conditions directly on desired class losses and denoises Gaussian noise into parameters. A pivot-class mechanism fixes a subset of classes that must always stay high-performing, simplifying the learning problem, and the same trained network can be reprompted with different forget sets at inference.","core_discovery":"The paper's central claim, stated in Section 5, is that sampled unlearned models effectively achieve zero performance on forget sets while maintaining high accuracy on retain sets and closely mimic a retrained model. This is offered as the first use of hypernetworks for machine unlearning: instead of updating a trained model's weights, HyperForget learns a conditional distribution over weight configurations and samples from it at inference time. The two implementations, DiHyFo-1 and DiHyFo-2, differ in conditioning—the former is a learned optimizer-deoptimizer that takes current parameters, current losses, and target losses, while the latter is a generator conditioned directly on target losses—but both produce parameters that are evaluated against retrained baselines using accuracy, membership-inference score, output overlap, and the unlearning score. The paper also claims that a single DiHyFo can sample unlearned models for all tested forget sets, and it acknowledges in Section 6 that the generator retains knowledge of the forget sets, making the approach unsuitable for strict unlearning-for-privacy applications.","pith_inferences":["My reading: the zero-forget headline depends on the Appendix A.1 selection procedure, which keeps the sampled model with the lowest forget accuracy; a deployed system would likely need an additional selection or verification step rather than trusting any single sample.","The paper's admitted retention of forget knowledge in the generator implies that the framework removes behaviour, not information; a natural test is to probe sampled parameters with linear classifiers or attempt reconstruction from the generator to quantify what remains.","If the approach scales, the same conditioning machinery could be applied to continual learning and model editing, where a single generator would replace task-specific fine-tuning by prompting which capabilities to keep and which to drop.","The pivot-class simplification means the user must decide in advance which classes will never be forgotten, a practical constraint that may not hold in open-ended deployment."],"forward_implications":["A single trained DiHyFo can serve many forget requests at inference time, without retraining or per-request gradient updates.","Sampled models reach zero accuracy on forget classes while keeping retain-set accuracy close to retrained baselines, so class-level unlearning can be verified behaviourally.","MIA scores of sampled models are close to those of retrained models, suggesting the unlearned models do not leak forget-set membership more than a model that never saw the data.","Because unlearning is a forward pass through a conditional generative model, the marginal cost of a new forget set is small once checkpoint data and training are done.","The proof-of-concept results are limited to small MLPs on MNIST variants, with scalability and other unlearning tasks left open by the paper."],"supporting_citations":[{"why":"Supplies the diffusion-hypernetwork method (G.pt) that DiHyFo-1 extends to class-level losses, plus the prompt alignment metric.","marker":"[37]"},{"why":"Provides the DDPM denoising process used to sample parameters from Gaussian noise.","marker":"[22]"},{"why":"Gives the hypernetwork definition and conditioning framework that HyperForget builds on.","marker":"[20]"},{"why":"Provides the MIA and unlearning-score evaluation procedures used to compare sampled models with retrained models.","marker":"[7]"},{"why":"Defines the retrained-model gold standard and the desiderata used to assess the sampled unlearned models.","marker":"[35]"},{"why":"Supplies MIA implementation details used in the evaluation.","marker":"[13]"},{"why":"Shows a latent-space diffusion approach for generating neural-network parameters, a baseline the paper compares conceptually.","marker":"[54]"},{"why":"Demonstrates conditioned diffusion over model parameters, motivating DiHyFo-2's direct conditioning design.","marker":"[27]"}],"fun_headline_variants":["One hypernetwork samples models that forget on demand","HyperForget: a diffusion hypernetwork for unlearning","Sample a fresh model instead of updating weights","Forget targeted data with a single diffusion network","DiHyFo: one network, many forgetful models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a model whose loss is high on forget-set classes and low on retain-set classes is behaviourally equivalent to a model that was never trained on the forgotten data; if that proxy is wrong, the reported zero accuracy could describe a degenerate classifier rather than genuine unlearning.","fun_headline_variants_meta":{"raw":{"variants":["One hypernetwork samples models that forget on demand","HyperForget: a diffusion hypernetwork for unlearning","Sample a fresh model instead of updating weights","Forget targeted data with a single diffusion network","DiHyFo: one network, many forgetful models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000643,"raw_usage":{"total_tokens":2931,"prompt_tokens":893,"completion_tokens":2038,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":509,"completion_tokens_details":{"reasoning_tokens":1964}},"tokens_in":509,"tokens_out":2038,"duration_ms":14982,"temperature":1.0,"reasoning_tokens":1964,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:02:00.381422+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a trained DiHyFo, prompt it to forget class 2 of MNIST, and fit a linear probe on the penultimate-layer activations of the sampled model using forget-class labels. If the probe can decode class-2 structure well above chance while the model's output accuracy on class 2 is zero, the claimed equivalence to a retrained model fails: the network has zeroed the output without removing the information. A complementary check is to compare the sampled model's full output distribution on forget-class inputs with that of a model retrained without class 2; if the sampled model collapses to a single constant vector while the retrained model retains a spread of near-random but structured probabilities, the 'unlearned' model is not behaviourally mimicking retraining.","supporting_citations":[{"cited_title":"Learning to learn with generative models of neural network checkpoints","cited_arxiv_id":null,"evidence_quote":"Supplies the diffusion-hypernetwork method (G.pt) that DiHyFo-1 extends to class-level losses, plus the prompt alignment metric."},{"cited_title":"Can bad teaching induce forgetting? unlearning in deep networks using an incompetent teacher","cited_arxiv_id":null,"evidence_quote":"Provides the MIA and unlearning-score evaluation procedures used to compare sampled models with retrained models."},{"cited_title":"Fast machine unlearning without retraining through selective synaptic dampening","cited_arxiv_id":null,"evidence_quote":"Supplies MIA implementation details used in the evaluation."},{"cited_title":"Neural network diffusion","cited_arxiv_id":null,"evidence_quote":"Shows a latent-space diffusion approach for generating neural-network parameters, a baseline the paper compares conceptually."}],"review_version":1}