{"id":"610b4e89-f01b-4c3f-881d-b385dd766ecf","arxiv_id":"2412.01207","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Siamese unlearning with knowledge vaporization and concentration erases targeted data by dispersing its augmented-view logits while concentrating those of retained data.","lead":"The paper introduces a new machine unlearning method that uses Siamese networks to scatter the outputs of data to be forgotten while keeping outputs of remaining data stable. The method needs only a small sample of the remaining dataset and no extra teacher model, and it reports better forgetting, utility, and privacy-attack resistance than existing approaches on CIFAR benchmarks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Vaporization loss Eq. (4) may act on the predictor h rather than fω; Appendix 8.3 admits KVC alone fails to alter fω's logits, so the claimed mechanism may not drive forgetting.","rationale":"The reader's weakest-assumption concern is that the dispersion signature motivating vaporization is measured on only 10 data points from one VGG16-BN model. I agree that evidence is thin, but the more load-bearing issue is internal to the proposed loss: even a robust signature would not matter if Eq. (4) does not actually implement that signature on fω. The stop-gradient and the untrusted predictor h allow the vaporization term to be optimized by changing h rather than fω; the paper's Appendix 8.3 nearly admits this when it says KVC without cross-entropy fails to alter the output logits of the forgetting data and blames the predictor. If that is true in the full objective, the central claim that Eqs. (8)-(9) approximate the retrained model through knowledge vaporization is not supported, and the empirical forgetting may be attributable to the permuted-label cross-entropy term instead. A single matched ablation removing LKV would settle whether the vaporization loss is the active ingredient. The test-set hyperparameter tuning and missing error bars noted by the reader are real but secondary; they affect the comparative 'best trade-off' claim, not the mechanism. My concern does not change the verdict from CONDITIONAL, because the method may still work even if the mechanism is mis-specified, and the proposed check can confirm or refute it. I therefore recommend UNCHANGED and would expect the authors to add the ablation plus a logit-level comparison to the retrained model before final acceptance.","tokens_in":17837,"tokens_out":14598,"duration_ms":139307,"concrete_test":"Run the full objective Eqs. (8)-(9) and an identical run with LKV removed (keeping LKC, SCE, ALP, same λ, seeds, and epochs) on CIFAR-10 full-class and sub-class forgetting. Also run a variant with h removed so LKV acts directly between fω(x1) and fω(x2). Compare AccDf, TADf, MIA, and the KL divergence between augmented-view softmax outputs on Df against the retrained model. If removing LKV leaves these metrics within noise, Eq. (4) is not the active forgetting mechanism; if they degrade substantially, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central mechanistic claim is that Eq. (4) implements knowledge vaporization by dispersing fω's augmented-view logits, and that Eqs. (8)-(9) thereby approximate the retrained model. But LKV maximizes cosine distance between h(fω(x1)) and a stop-grad copy of fω(x2). The predictor h is a free MLP, and the stop-gradient prevents direct gradient flow into fω(x2); gradients reach fω only through h(fω(x1)). The network can therefore satisfy LKV by changing h's output directions while leaving fω's logits nearly unchanged. The manuscript's own Appendix 8.3 states that without cross-entropy terms, optimizing knowledge vaporization and concentration 'fails to alter the output logits of the forgetting data' and attributes this to the optimization 'focusing solely on maximizing the distance between the predictor h and p1 = h(fω(x1)), neglecting the impact on fω.' If this holds in the full objective, the observed forgetting on Df could be driven by the permuted-label cross-entropy SCE(x, p(y)) in Eq. (9) rather than by the claimed vaporization. The central claim in Section 3.3 is then not demonstrated as stated. This is an internal consistency check of the loss against the stated mechanism, not a disagreement with consensus.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"Motivated by the observation that, in a retrained-on-remaining-data model, augmented views of forgetting samples produce more dispersed output logits than augmented views of remaining samples, the paper introduces knowledge vaporization and knowledge concentration, operationalized through a Siamese network with a prediction head h and stop-gradient (Eqs. 4-5) plus a symmetric cross-entropy term with adaptively permuted labels for forgetting data (Eqs. 6-9). The combined objective requires no retraining, no teacher model, and only a 1000-sample subset of the remaining data. The method is evaluated on CIFAR-10 and CIFAR-100 with VGG16-BN, ResNet18, and ResNet50 under full-class, sub-class, and random-forgetting scenarios against eight baselines on training/test accuracy and membership-inference metrics. The authors report the best overall trade-off in most settings, MIA down to 0.00, and low runtime, and they include ablations of the vaporization/concentration loss, cross-entropy terms, and adaptive label permutation.","tokens_in":18129,"tokens_out":24248,"duration_ms":197098,"significance":"If the empirical results hold, this is a useful and practical contribution to approximate unlearning: no retraining, no stored teacher models, only about 1000 remaining samples (claimed less than 2% of the training set), and strong reported trade-offs between forgetting, utility, and MIA across full-class, sub-class, and random forgetting on CIFAR-10/100 with three backbones and three augmentation sets. The paper deserves credit for a concrete, falsifiable protocol: the per-class permutation probabilities are computed from the known deletion-request ratios r rather than fitted to test performance, so the design is not circular; the evaluation is unusually broad, including runtimes and an ablation; and source code is provided. The central reservation is that the mechanism described in Section 3.3 is not established by the evidence, parts of the manuscript contradict each other (Appendix 8.3 vs. Tables 7/10), the tables lack error bars, and Eq. (7) is not reproducible as printed.","major_comments":[{"comment":"The central mechanistic claim, that Eq. (4) disperses the backbone's augmented-view logits, is contradicted by the manuscript's own text and ablation tables. In LKV each term maximizes cosine distance between p1 = h(fω(x1)) and the stop-gradient copy sg(l2); gradients reach fω only through the auxiliary head h on the non-stopped branch, so the loss can be approximately minimized by changing h's output directions while leaving fω nearly unchanged. Appendix 8.3 concedes exactly this, stating that without cross-entropy terms, \"optimizing knowledge vaporization and concentration alone fails to alter the output logits of the forgetting data.\" However, the KVC-only rows of Table 7 and Table 10 report AccDf=0.00 and TADf=0.00, which requires that fω's logits on Df changed drastically. These two statements cannot both be true. Because the permuted-label cross-entropy term SCE(x, p(y)) in Eq. (9) is the only term with a direct gradient into fω's logits on forgetting data, the observed forgetting could be driven by that term (the RandLab/Amnesiac mechanism) rather than by the claimed vaporization, so the claim in Section 3.3 is not demonstrated as stated. I request a direct measurement of fω's logit dispersion on Df (e.g., the augmented-view KL divergence as in Table 9) before and after unlearning under each ablation configuration, and an ablation that removes LKV while keeping CE and ALP, to isolate the vaporization contribution.","section":"§3.3, Eq. (4); §8.3; Tables 7 and 10"},{"comment":"All results in Tables 1-6 are single runs without error bars, and Appendix 8.1 states that hyperparameters of all compared methods were tuned to achieve optimal metric values in each scenario. Several headline comparisons are close (Table 5: Ours and SCRUB both report AccDf=100.00 with gap 0.86; Table 2 TADr: BadT 74.80 vs. Ours 73.96; Table 4 TADr: SCRUB 73.26 vs. Ours 72.38), so the claims that the method \"consistently outperforms\" baselines and achieves the \"best trade-off\" are not statistically supported as reported. I ask for multiple seeds with means and standard deviations on the main tables, and for the λ and per-scenario epoch values used in the reported results, which are currently absent from Section 8.","section":"§4.1 and §8.1, Tables 1-6"},{"comment":"The adaptive label permutation probability in Eq. (7) is not reproducible as printed. If the numerator is r_k − 1, then for every r_k < 1 and K > 2 the probability of keeping the true label is negative (e.g., r_k = 0 with K = 10 gives −1/8), which contradicts the text's statement that r_k = 0 causes no permutation; if r_k^{-1} was intended, the notation must be corrected. The denominator also appears to be K − 1 rather than k − 1. Because the ablation (Table 7) credits ALP with improving the AccDf gap from −3.92 to −0.28 relative to the retrained model, the exact rule matters for reproducing the results; please restate Eq. (7) unambiguously and state precisely what was implemented.","section":"Eq. (7)"},{"comment":"Algorithm 1 is inconsistent with the restricted-access claim. The Input line names the remaining-data subset \"Sf\" (presumably Sr), and lines 5-8 draw the remaining-data mini-batches from Dr, whereas Section 3.3 and Appendix 8.1 state that the method uses only a 1000-sample subset Sr (claimed to be less than 2% of the dataset). As printed, the algorithm contradicts the paper's headline advantage of not needing full remaining-data access; please correct both the source of the mini-batches and the input variable name.","section":"Algorithm 1 vs. §3.3 and §8.1"},{"comment":"The motivating observation, that augmented views of forgetting data are more dispersed than those of remaining data in retrained models, is evidenced only by t-SNE plots and KL divergences computed on 10 data points per set from a single VGG16-BN model on CIFAR-10 (Appendix A). In the random-forgetting scenario with contrastive augmentation, the reported KL values are 1.6926 for Dr vs. 1.7256 for Df, which is effectively no dispersion gap, yet the method's random-unlearning results are among its strongest (Tables 5-6). This suggests the dispersion signature is neither robust nor necessary for the method's success, leaving the design rationale of Section 3.2 unverified across architectures, datasets, and scenarios. I would like a quantitative check of the dispersion gap on the ResNet models and on CIFAR-100, and a statement of how the method behaves when the gap is weak or absent.","section":"§3.2 and Appendix A, Table 9"}],"minor_comments":[{"comment":"The phrase \"without the need for additional neural networks\" is inaccurate because the method itself trains an auxiliary predictor MLP h (Section 7, two fully connected layers with batch normalization); the claim should be qualified as \"no teacher model\" or \"h is discarded after unlearning.\"","section":"§3.3, 'Memory efficiency' bullet"},{"comment":"The sentence \"which means it effectively erases the knowledge of Df and maintains the utility on Df\" should read \"maintains the utility on Dr\"; as printed it is self-contradictory.","section":"§4.2, first paragraph"},{"comment":"The column alignment of the checkmarks in Table 7 is ambiguous in the current rendering; in particular, the row described in the main text as CE-only (yielding 0 on AccDf) does not visibly match the row with AccDf=100.00(20.18). Please ensure each checkmark is placed under the correct column.","section":"Table 7"},{"comment":"In the random-forgetting scenario, the notation Df is used for the whole test set (\"we only consider the test accuracy on the entire test dataset Df\"), which conflicts with the definition of Df as forgotten training data in Section 3.1; rename the test set to avoid confusion.","section":"Tables 5-6"},{"comment":"The phrase \"state-of-the-art\" is too strong given that individual metrics favor some baselines (e.g., Table 2 TADr: BadT 74.80 vs. Ours 73.96; Table 4 TADr: SCRUB 73.26 vs. Ours 72.38); \"best overall trade-off\" is a more accurate description.","section":"§4.2 and Table 2"},{"comment":"Please state whether the reported runtimes include the per-method hyperparameter search described in Appendix 8.1 and whether all methods were evaluated on the same hardware.","section":"Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript has several signs of a hastily finalized draft: the contradiction between Appendix 8.3 and Tables 7/10, the Algorithm 1 input/sampling mismatch, and the garbled Eq. (7). These are fixable, but they are load-bearing rather than cosmetic. If the authors can reconcile the mechanism with the ablations, add error bars, and correct the equations and algorithms, the paper could become acceptable. Otherwise the novelty claim may rest largely on a permuted-label cross-entropy mechanism that is close to RandLab/Amnesiac, with the vaporization/concentration framing not empirically substantiated. The journal should also consider whether a purely empirical unlearning heuristic, without formal guarantees and without verification of its motivating observation across architectures, fits its scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth taking seriously, but the central mechanism is not actually demonstrated. What is new here is a concrete recipe: a SimSiam-style Siamese loss that pushes apart logits of augmented views of forgetting data and pulls together views of remaining data, combined with an adaptive label permutation for the cross-entropy term. That combination, plus the practical claims of no teacher model and only a small remaining-data subset, is a genuine contribution to approximate unlearning. The experiments are broad: three scenarios, three architectures, several baselines, and the authors ship code. If the method works as reported, it is a useful addition to the toolbox.\n\nThe soft spot is load-bearing. The paper says knowledge vaporization disperses the encoder's output logits, but the loss in Eq. (4) has a stop-gradient on one side and goes through the predictor h on the other. The network can satisfy that loss by changing h alone, leaving f_omega's logits nearly untouched. The paper's own Appendix 8.3 admits this: without cross-entropy terms, optimizing KVC fails to alter the output logits of the forgetting data. That directly undercuts the stated mechanism. Worse, the appendix's admission contradicts the ablation tables, where KVC-only rows are reported as achieving AccDf = 0.00. That is an internal inconsistency a referee would catch immediately. If the tables are right, the appendix is wrong; if the appendix is right, the core conceptual story is wrong and the actual forgetting is likely driven by the permuted-label cross-entropy, making the method a variant of random-label fine-tuning.\n\nThere are smaller problems too. All tables show single runs with no error bars. Hyperparameters for baselines were tuned with the test set in the loop. The motivating observation about retrained-model logit dispersion rests on ten data points per set from one VGG16-BN on CIFAR-10, with t-SNE plots and averaged KL divergences—suggestive but thin. The citation pattern is fine and the paper engages the relevant literature.\n\nMy take: the paper deserves a serious referee, but it needs major revision. The authors should resolve the contradiction, add multiple seeds, and directly measure whether f_omega's logits actually disperse after optimizing LKV (for example, by removing h from the measurement). If vaporization turns out to be a byproduct of the cross-entropy term, they should reframe the contribution honestly. As written, I would not cite it as a demonstrated mechanism, but I would send it to review with the expectation of heavy revision.","headline":"Interesting practical unlearning recipe, but its own ablation contradicts the central mechanism: the vaporization loss can be satisfied by the predictor head alone, and the paper admits this in an appendix.","tokens_in":18646,"tokens_out":3884,"would_cite":false,"duration_ms":33733,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Siamese unlearning vaporizes forgotten-data logits and concentrates the rest.","keywords":["machine unlearning","right to be forgotten","Siamese networks","knowledge vaporization","knowledge concentration","approximate unlearning","membership inference attack","data augmentation"],"falsifier":"Train retrained models on a dataset where forgotten and remaining samples are semantically well separated, such as synthetic classes with no shared features, and compute the per-sample KL divergence of augmented-view logits; if forgetting samples do not show clearly higher dispersion than remaining samples, the empirical premise collapses. A direct test is to run Siamese unlearning on such a setting and check whether the unlearned model's outputs diverge from the retrained model far more than the paper's CIFAR results, which would show the method rides on the dispersion signature rather than on the loss functions themselves.","tokens_in":17660,"feed_emoji":"🧹","tokens_out":5025,"duration_ms":43704,"temperature":0.7,"pith_summary":"Machine unlearning asks how to remove the influence of specific training points from an already-trained model without retraining from scratch. This paper argues that a model retrained on only the remaining data has a recognizable signature: augmented views of forgotten data produce dispersed, high-entropy logits, while augmented views of remaining data stay tightly clustered. On that basis it proposes two complementary objectives—knowledge vaporization, which disperses the logits of forgetting data, and knowledge concentration, which preserves representations of remaining data—implemented through a Siamese network with shared weights and a stop-gradient predictor. The paper claims this reproduces a retrained model's behavior using only a small subset of remaining data and no teacher models, and it reports strong forgetting, preserved utility, and low membership-inference success across full-class, sub-class, and random forgetting on CIFAR-10 and CIFAR-100.","feed_headline":"Siamese unlearning erases data by vaporizing and concentrating logits","feed_subtitle":"Mimics retrained models with under 2% of remaining data and no teacher networks, cutting memory and MIA risk.","key_machinery":"The carrying mechanism is the Siamese network with a prediction MLP head and stop-gradient, used to define a symmetric cosine-distance loss between two augmented views of the same sample. Knowledge vaporization minimizes this distance for forgetting data, pushing the two views' logits apart so the sample's representations become dispersed and unpredictable; knowledge concentration maximizes the same distance for remaining data, pulling each sample's augmented views together so its learned representation is preserved. The cross-entropy term with adaptively permuted labels, generated by a randomized-response scheme from the per-class unlearning ratio, prevents the process from destroying model utility on partially retained classes. Together these losses implement the paper's central identity: the unlearned model should exhibit the same augmentation-response signature as a model retrained without the forgetting data.","core_discovery":"The paper's central claim is that unlearning can be posed as matching the logit dispersion pattern of a retrained model, rather than approximating its weights. In a retrained model, augmented views of forgetting data scatter widely over the logit space while augmented views of remaining data concentrate, as the paper shows with t-SNE plots and KL-divergence measurements. The proposed Siamese unlearning method enforces this pattern directly: for each forgetting sample it minimizes the cosine similarity between two augmented views through a prediction head with stop-gradient, and for each remaining sample it maximizes that similarity, while a symmetric cross-entropy term keeps the outputs usable for classification. An adaptive label permutation, derived from the per-class fraction of unlearned data via randomized response, sets the target labels for forgetting data so that partially forgotten classes are not destroyed. The result, the paper argues, is an unlearned model whose outputs align with those of the retrained model, achieving the best accuracy-forgetting trade-off among the compared baselines and, in full-class scenarios, zero membership-inference success on forgetting data.","pith_inferences":["The augmentation-dispersion signature could itself be turned into a post-hoc unlearning verification test: measure KL divergence of augmented-view logits on a candidate forgotten point and compare it with a reference set of remaining points, without needing retrained checkpoints.","The adaptive label permutation is a natural place to connect unlearning with local differential privacy; treating per-class unlearning ratios as a privacy budget could yield formal guarantees on how much class-level information survives.","If the dispersion signature is architecture-dependent, the method's effectiveness on larger or differently trained models is a direct empirical question; testing on ImageNet-scale models or transformer backbones would settle whether the signature generalizes.","The vaporization objective might combine with parameter-space methods such as Fisher-information damping, targeting the same forgetting samples while limiting collateral damage on shared representations."],"forward_implications":["If the central claim holds, an unlearning request can be served with access to less than 2% of the remaining dataset, which matters when the full remaining set is unavailable or privacy-sensitive.","Because no teacher model or stored gradient history is needed, the memory footprint of unlearning stays the same as the original model, unlike distillation-based methods.","The method applies to instance-wise and random forgetting, not only whole-class removal, so it covers the common single-user deletion request.","In full-class scenarios the reported 0.00 MIA on forgetting data implies that membership-inference attacks cannot distinguish forgotten points from non-members, a direct privacy benefit.","The approach is robust across simple, contrastive, and Cutout augmentations, suggesting the result does not depend on one particular augmentation choice."],"supporting_citations":[{"why":"Supplies the Siamese architecture and stop-gradient technique that stabilize the vaporization and concentration losses and prevent representation collapse.","marker":"[8]"},{"why":"Provides the contrastive augmentation used in experiments and the principle that augmentation invariance indicates learned representation.","marker":"[7]"},{"why":"Gives the randomized-response mechanism that underlies the adaptive label permutation for forgetting data.","marker":"[34]"},{"why":"SCRUB is a teacher-student unlearning baseline whose positive and negative distillation the method is designed to avoid.","marker":"[20]"},{"why":"BadT is a teacher-student baseline used for comparison and motivates the paper's memory-efficient alternative.","marker":"[9]"},{"why":"SSD is a parameter-dampening baseline for comparison in the unlearning scenarios.","marker":"[12]"},{"why":"Supplies the NegGrad baseline and the Fisher-information perspective that positioned approximate unlearning as parameter influence removal.","marker":"[14]"},{"why":"Amnesiac is a random-label fine-tuning baseline, directly related to the cross-entropy component on forgetting data.","marker":"[15]"},{"why":"Provides the membership-inference-attack evaluation methodology used to measure privacy leakage of unlearned models.","marker":"[25]"}],"fun_headline_variants":["Vaporize and concentrate knowledge for targeted unlearning","Siamese unlearning matches retrained logit patterns","Zero-memory unlearning by vaporizing logit spread","Mimic retrained models with a Siamese twist for unlearning","Unlearn by scattering forgetting logits, clustering the rest"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method rests on the observation that in a retrained model the augmented views of forgotten data are much more dispersed than augmented views of remaining data; that observation is demonstrated with t-SNE plots and KL divergences of ten data points per set from a single VGG16-BN model on CIFAR-10, so if the signature does not hold for other architectures, datasets, or augmentation strengths, the vaporization and concentration losses will not reproduce retrained-model behavior.","fun_headline_variants_meta":{"raw":{"variants":["Vaporize and concentrate knowledge for targeted unlearning","Siamese unlearning matches retrained logit patterns","Zero-memory unlearning by vaporizing logit spread","Mimic retrained models with a Siamese twist for unlearning","Unlearn by scattering forgetting logits, clustering the rest"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000202,"raw_usage":{"total_tokens":1370,"prompt_tokens":924,"completion_tokens":446,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":364}},"tokens_in":540,"tokens_out":446,"duration_ms":4417,"temperature":1.0,"reasoning_tokens":364,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:34:32.879859+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train retrained models on a dataset where forgotten and remaining samples are semantically well separated, such as synthetic classes with no shared features, and compute the per-sample KL divergence of augmented-view logits; if forgetting samples do not show clearly higher dispersion than remaining samples, the empirical premise collapses. A direct test is to run Siamese unlearning on such a setting and check whether the unlearned model's outputs diverge from the retrained model far more than the paper's CIFAR results, which would show the method rides on the dispersion signature rather than on the loss functions themselves.","supporting_citations":[{"cited_title":"Exploring simple siamese rep- resentation learning","cited_arxiv_id":null,"evidence_quote":"Supplies the Siamese architecture and stop-gradient technique that stabilize the vaporization and concentration losses and prevent representation collapse."},{"cited_title":"A simple framework for contrastive learning of visual representations","cited_arxiv_id":null,"evidence_quote":"Provides the contrastive augmentation used in experiments and the principle that augmentation invariance indicates learned representation."},{"cited_title":"Randomized response: A survey tech- nique for eliminating evasive answer bias","cited_arxiv_id":null,"evidence_quote":"Gives the randomized-response mechanism that underlies the adaptive label permutation for forgetting data."},{"cited_title":"Can bad teaching induce forgetting? unlearning in deep networks using an incompetent teacher","cited_arxiv_id":null,"evidence_quote":"BadT is a teacher-student baseline used for comparison and motivates the paper's memory-efficient alternative."},{"cited_title":"Fast machine unlearning without retraining through selective synaptic dampening","cited_arxiv_id":null,"evidence_quote":"SSD is a parameter-dampening baseline for comparison in the unlearning scenarios."},{"cited_title":"Eternal sunshine of the spotless net: Selective forgetting in deep networks","cited_arxiv_id":null,"evidence_quote":"Supplies the NegGrad baseline and the Fisher-information perspective that positioned approximate unlearning as parameter influence removal."},{"cited_title":"Amne- siac machine learning","cited_arxiv_id":null,"evidence_quote":"Amnesiac is a random-label fine-tuning baseline, directly related to the cross-entropy component on forgetting data."}],"review_version":1}