{"id":"982b9dc2-140a-4ff5-b002-55ec42b5f395","arxiv_id":"2508.01324","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"DCUE evaluates LLM unlearning by comparing core-token confidence score distributions of the unlearned model and the original model, corrected by a validation set, using the Kolmogorov-Smirnov test.","lead":"The paper proposes DCUE, a new metric for verifying whether a large language model has really forgotten data it was asked to forget, without needing to retrain a reference model. It corrects for ordinary fine-tuning shifts using a validation set and measures remaining memory via core-token confidence scores and a Kolmogorov-Smirnov test.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The min-based correction in Eq. (6) can certify perfect unlearning whenever Sv_o,u exceeds Sf_o,u; the §4.5 validation only checks Mt and Mr, leaving exactness and ranking claims unestablished.","rationale":"The reader's verdict is CONDITIONAL with high correctness risk, and the reader's weakest_assumption is exactly the Eq. (6) approximation. I agree that this is the load-bearing point: every claim about DCUE's practicality and exactness depends on Scorr meaningfully approximating the true distance between Mu and Mr on the forget set. The min heuristic creates a sharp failure mode—if the validation shift is at least as large as the forget shift, Scorr is forced to zero and the p-value becomes 1, a perfect-unlearning verdict that is independent of any actual forgetting. The paper's §4.5 simulation is encouraging but only covers the two endpoints Mt and Mr; it does not establish the inequality direction for intermediate unlearned models or for different Dv construction, and the perfect exactness scores in Table 2 are measured on a single TOFU setup. The central conclusion about existing unlearning methods uses extremesmall p-values, so it may survive this particular failure, but the metric's use as a practical certification tool is not safe without either a derivation of Eq. (6) or a broader validation covering the failure region. The additional calibration concern about treating Scorr as a KS statistic reinforces, but does not replace, the Eq. (6) issue. Overall, the paper is a reasonable proposal with useful components—core-token extraction is validated with manual review, and the ablation study is informative—but the load-bearing approximation needs stronger theoretical or empirical support. The reader's CONDITIONAL verdict is appropriate; my read does not move it.","tokens_in":14645,"tokens_out":7329,"duration_ms":91315,"concrete_test":"Re-run the §4.5 validation on Phi-1.5B with Mu = Mt and 100 random Dv draws, but sample Dv from the retained split Dr (or a distribution closer to Dr) to inflate the validation shift. For each draw, compare Sf_o,t and Sv_o,t; if any draw gives Sf_o,t ≤ Sv_o,t, DCUE outputs Scorr = 0 and p ≈ 1 for a model that has not unlearned, directly falsifying the exactness− = 1 claim and the 'uncommon' assertion after Eq. (6).","verdict_should_be":"UNCHANGED","load_bearing_attack":"Eq. (6) approximates δS by min{Sv_o,u, Sf_o,u}, and Eq. (8) then gives Scorr = Sf_o,u - min{Sv_o,u, Sf_o,u}. Consequently, whenever Sv_o,u ≥ Sf_o,u, Scorr = 0 and the KS p-value in Eq. (9) is 1: DCUE certifies complete unlearning no matter how far Mu's forget-set distribution is from Mr's. The heuristic is not derived; the paper states it is 'uncommon' for δS to equal Sf_o,u, but the only evidence is §4.5, which simulates Mu = Mt and Mu = Mr on TOFU with 100 Dv draws. Those endpoints can behave differently from intermediate unlearned models (GA/GD/DPO/NPO/SimNPO in Table 3) or from settings where Dv is closer in distribution to Dr. If Mu = Mt and Dv is drawn from data similar to the retained set, fine-tuning on Dr can shift Mo's validation CTCS more than memorizing Df shifts its forget CTCS, giving Sv_o,t > Sf_o,t and Scorr = 0 for a model that has not unlearned at all. That would falsify the exactness− = 1 entries in Table 2. Because the paper's 'current methods remain far from complete' conclusion uses already tiny p-values, it may survive this particular failure, but the metric's claim to exactness and its use as a certification tool do not. A second, compounding issue is that Scorr is not a two-sample KS statistic; feeding it into the KS survival function (Eq. 9, Appendix A) assumes a null distribution that is not established for a min-corrected difference of KS statistics.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that existing LLM unlearning evaluation metrics (text similarity, multiple-choice accuracy, prediction probability, MIA) are impractical, inexact, or non-robust in real-world settings, and proposes a new metric called DCUE. DCUE extracts core-token confidence scores (CTCS) via LLM prompts, uses a validation set Dv to correct distributional drift caused by retained data, and then applies a Kolmogorov-Smirnov test to a corrected statistic Scorr defined in Eq. (8). Experiments on Phi-1.5B, LLaMA2-7B, and Qwen2.5-7B with TOFU and MUSE-News claim that DCUE achieves perfect practicality, exactness, and robustness scores, and that existing unlearning methods (GA, GD, IDK, DPO, NPO, SimNPO) remain far from complete unlearning.","tokens_in":14961,"tokens_out":5143,"duration_ms":63465,"significance":"If the statistical construction were sound, DCUE would be a practically valuable contribution: it removes the dependence on an inaccessible retrained model, addresses the known token-level noise of text-similarity metrics, and offers a unified protocol for third-party verification of unlearning. The paper's critique of existing metrics in Section 3.2 is clear and well-illustrated, and the manual validation of core-token extraction (97% and 96% precision across models) is a concrete strength. However, the central approximation in Eq. (6) is heuristic, the corrected statistic Scorr is not a two-sample KS statistic, and the validation experiments cover only the favorable endpoints Mt and Mr. These are load-bearing gaps: until the null distribution of Scorr is established and the approximation is tested on intermediate unlearned models, the claimed exactness and the quantitative p-values in Table 3 are not reliable. The paper is therefore promising but requires substantial revision.","major_comments":[{"comment":"The min correction makes Scorr identically zero whenever Sv_o,u ≥ Sf_o,u, so DCUE then reports p = 1 and certifies complete unlearning regardless of Mu's actual internal state. This is not a remote edge case: if Dv is drawn from data close to Dr, fine-tuning on Dr can shift the model's validation-set CTCS more than memorization of Df shifts its forget-set CTCS, giving Sv_o,t > Sf_o,t for a model that has not unlearned at all. The validation in §4.5 only tests Mu = Mt and Mu = Mr, the two endpoints for which the approximation is most favorable, and does not include intermediate unlearned models such as those in Table 3. The exactness+ and exactness− scores of 1.0000 in Table 2 therefore do not establish exactness for the general case the metric is designed for.","section":"§4.3, Eq. (8)"},{"comment":"Feeding Scorr into the two-sample KS survival function is not justified. Scorr = Sf_o,u − min{Sv_o,u, Sf_o,u} is a min-corrected difference of two KS statistics, not a KS statistic computed from the maximum difference between two ECDFs. The null distribution of the two-sample KS statistic, used in Eq. (9) and Appendix A, does not apply to this corrected statistic. Consequently, the values RDCUE reported in Table 3 and the approximate p-values in Figure 5 are not calibrated p-values. The paper needs either a derived null distribution for Scorr or a permutation/bootstrap calibration; without that, the quantitative comparisons among unlearning methods are not statistically interpretable.","section":"§4.4, Appendix A"},{"comment":"The approximation δS ≈ min{Sv_o,u, Sf_o,u} in Eq. (6) is introduced purely as a heuristic ('The intuition behind this approximation is as follows...') and is not derived from any distributional assumption. The numerical validation is limited to two models, one dataset (TOFU), 100 draws of Dv, and only two choices of Mu (Mt and Mr); no error bars or sensitivity analyses are reported. This is a load-bearing point because the entire distribution-correction mechanism and every subsequent exactness claim depend on Eq. (6). At minimum, the authors should validate the approximation on the intermediate unlearned models studied in Table 3 and on Dv distributions that are closer to Dr than to Df.","section":"§4.5"},{"comment":"The robustness experiments use Mr as Mu ('Considering Mr represents the gold standard... we use Mr as Mu in subsequent experiment of robustness'). Mr is already perfectly unlearned, so post-processing it with unlearning on other data or fine-tuning on new data is not representative of applying post-processing to a partially unlearned model from Table 3. This weakens the claim that DCUE is robust in realistic deployment, where the model being evaluated will typically be in an intermediate unlearning state. Additionally, the fact that DCUE's robustness scores are exactly 1.0000 in every configuration (Tables 2 and 4) suggests possible saturation rather than meaningful measurement of robustness; the authors should report experiments where the metric is expected to change and show that it tracks those changes.","section":"§5.1, Table 2"}],"minor_comments":[{"comment":"The statement 'It is uncommon for δS to equal Sf_o,u' is not quantified or supported; the paper should provide the empirical frequency of this event in the §4.5 simulation.","section":"§4.3"},{"comment":"The ablation results in Figures 6 and 7 are reported as single deterministic numbers without error bars or repeated runs; adding variability measures would strengthen the claim that the observed differences are meaningful.","section":"§5.2"},{"comment":"The p-value formatting is inconsistent: some rows use scientific notation without parentheses (e.g., Mt and Mr), while others use parentheses with mixed forms such as '6.88e2' and '1.55e7'; the table should use a uniform notation and define the base value for the multiples.","section":"Table 3"},{"comment":"There are typographical and notation issues: 'PostProul' appears to be a typo for 'PostPro_ul'; 'CT CSd m' should be typeset as a single variable; and 'Qween' in Table 4 should be 'Qwen'.","section":"General"},{"comment":"The core-token extraction precision is manually reviewed on only the first 200 tokens for each dataset; reporting inter-annotator agreement or confidence intervals would improve reproducibility.","section":"§4.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript does not include a code or data availability statement, which is concerning given that the method relies on a heuristic approximation and manually validated prompt-based extraction. I would encourage the editor to request code and data if a revision is invited. The stress-test concern about Eq. (8) is real and confirmed by reading the paper; the validation in §4.5 covers only the endpoints Mt and Mr, which is precisely where the min correction behaves as intended."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's main contribution is a new evaluation metric for LLM unlearning that avoids needing a retrained reference model. The combination of core token confidence scores, validation-set distribution correction, and the KS test is genuinely new relative to TOFU, MUSE, WMDP, and RWKU. The authors clearly lay out three desirable properties (practicality, exactness, robustness) and demonstrate real limitations in existing metrics. The experiments on multiple models and datasets are a useful start, and the reported precision of core-token extraction is honest and reproducible.\n\nThe soft spots are serious. The central approximation in Eq. (6) is a heuristic. Scorr = Sf_o,u - min{Sv_o,u, Sf_o,u} collapses to zero whenever Sv_o,u >= Sf_o,u, so DCUE would certify perfect unlearning regardless of how far Mu's forget-set distribution is from the retrained model's. The validation in Section 4.5 only checks the two endpoints Mt and Mr, which does not rule out this collapse for intermediate unlearned models. The stress-test's counterexample is plausible: fine-tuning on retained data can shift validation confidence more than memorizing the forget set shifts forget confidence, giving Sv_o,t > Sf_o,t and a false perfect score for a model that has not unlearned at all.\n\nA second, compounding issue is that Scorr is not a two-sample KS statistic. Feeding it into the KS survival function assumes a null distribution that has not been derived for a min-corrected difference of KS statistics. The p-values in Table 3 may therefore be miscalibrated, and the claim that current methods remain far from unlearning, while likely true, is not reliably supported by this metric.\n\nThe robustness experiments use Mr as the base unlearned model, which is not a realistic real-world setting, and all robustness scores are 1.0, suggesting saturation rather than sensitivity. Also, the requirement of a validation set Dv drawn from the fine-tunable data but excluded from fine-tuning is a stronger practical assumption than the paper acknowledges.\n\nThese issues are addressable. The metric could be reworked with a proper statistical model or at least evaluated on a wider range of unlearning checkpoints, and the approximation could be tested against intermediate models. If the authors can fix the collapse and justify the null distribution, this could become a solid tool for the LLM unlearning community. As it stands, I would not rely on DCUE for certification, but the paper deserves a serious referee because the problem is important and the approach is novel. I expect major revisions.","headline":"A useful critique of existing unlearning metrics with a novel evaluation pipeline, but the load-bearing distribution-correction heuristic is unproven and the KS-based p-value is statistically unjustified.","tokens_in":15511,"tokens_out":3032,"would_cite":false,"duration_ms":35168,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that LLM unlearning can be evaluated without the retrained model by correcting a Kolmogorov-Smirnov statistic over core-token confidence scores using a validation set.","keywords":["LLM unlearning","unlearning evaluation","Kolmogorov-Smirnov test","core token confidence scores","distribution correction","validation set","machine unlearning metrics","retrained model independence"],"falsifier":"Take a case where the retained data shifts the model on the validation set in a direction opposite to its effect on the forget set, compute DCUE's corrected p-values, and compare them with p-values obtained from a genuinely retrained model; if the two disagree sharply, the approximation $\\delta_S \\approx \\min\\{S^v_{o,u}, S^f_{o,u}\\}$ has failed.","tokens_in":14392,"feed_emoji":"📏","tokens_out":10487,"duration_ms":119103,"temperature":0.7,"pith_summary":"The paper's aim is to make LLM unlearning verifiable where it actually happens: in deployed models that no one will retrain just to check them. It argues that existing evaluation metrics fail on three counts: several need the retrained model as a gold standard, text-similarity and multiple-choice scores can be gamed by irrelevant wording or general reasoning, and scores shift after unrelated updates. To fix this, the paper proposes DCUE, which scores only confidence assigned to answer-bearing \"core tokens\", corrects the difference between the original and unlearned models using a validation set drawn from the fine-tuning domain, and reports a Kolmogorov-Smirnov p-value. Across multiple model architectures and datasets, DCUE achieves the paper's own ideals of practicality, exactness, and stability under post-processing. It also finds that six current unlearning methods, GA, GD, IDK, DPO, NPO, and SimNPO, remain far from the level of a truly retrained model.","feed_headline":"LLM unlearning can now be measured without a retrained model","feed_subtitle":"A corrected statistic over answer-bearing tokens flags real forgetting using only a validation set.","key_machinery":"The load-bearing object is the corrected Kolmogorov-Smirnov statistic on core-token confidence scores (CTCS), where CTCS are the next-token probabilities a model gives to the minimal subset of tokens that actually carry an answer. The paper builds empirical cumulative distribution functions of CTCS for the original and unlearned models on both the forget set and a validation set, computes the KS statistics $S^f_{o,u}$ and $S^v_{o,u}$, and estimates the retained-data shift as $\\delta_S \\approx \\min\\{S^v_{o,u}, S^f_{o,u}\\}$. That estimate is subtracted from $S^f_{o,u}$ to form $S_{\\mathrm{corr}}$, and the p-value of $S_{\\mathrm{corr}}$ is the evaluation result.","core_discovery":"DCUE's central claim is that the unattainable comparison against a retrained model $M_r$ can be replaced by a corrected comparison against the original open-source model $M_o$, as long as a validation set $D_v$ from the fine-tuning domain is available and unused. On the forget set $D_f$, DCUE compares the empirical cumulative distribution functions of core-token confidence scores for $M_o$ and the unlearned model $M_u$; on $D_v$, the same comparison estimates the distributional shift produced by fine-tuning on the retained data. Subtracting the smaller of the two observed shifts from the forget-set shift gives a corrected statistic, $$S_{\\mathrm{corr}} = S^f_{o,u} - \\min\\{S^v_{o,u}, S^f_{o,u}\\},$$ and the p-value from the two-sample Kolmogorov-Smirnov test on this statistic is the unlearning score. The paper reports that on its test cases this metric assigns the best possible score to a retrained model and the worst to an untouched model, stays stable under further unlearning and fine-tuning, and does not require access to $M_r$.","pith_inferences":["Beyond the paper: the same validation-set correction could be applied to other leakage signals, such as membership-inference scores or hidden-state probes, to make them independent of a retrained model.","Beyond the paper: a natural hardening of the method is to require p-value stability across multiple validation draws, since the current score is a single p-value derived from one validation split.","Beyond the paper: core-token extraction currently relies on an auxiliary language model, so replacing that step with a deterministic extractor would make DCUE fully auditable.","Beyond the paper: the method's assumption that validation data resembles retained data suggests that in settings where the fine-tuning split is unknown, cross-validation over the forget set itself could serve as a fallback."],"forward_implications":["Data owners and auditors can check whether a model forgot requested data using only the original model, a validation set, and the unlearned model, with no retraining required.","Evaluations can avoid being fooled by non-answer wording, because only probabilities on answer-bearing tokens enter the score.","Unlearning methods can be compared by a score that stays stable when the model is later fine-tuned on unrelated data or unlearns other samples.","Under DCUE, current unlearning methods look far from complete, so reported unlearning should be treated as partial until proven otherwise.","Future unlearning algorithms can use DCUE's p-value as a deployment-time signal or acceptance criterion rather than depending on benchmark-only comparisons."],"supporting_citations":[{"why":"Supplies the TOFU benchmark and the TR Eval prediction-probability baseline, the retrained-model-dependent approach that DCUE must outperform.","marker":"[26]"},{"why":"Defines the MUSE text-similarity and MIA baselines (VerbMem, KnowMem, PrivLeak) whose exactness and practicality limits DCUE is built against.","marker":"[32]"},{"why":"Provides the RWKU text-similarity evaluation variants (QA, FB, AA) that illustrate how non-critical tokens distort Rouge-style scores.","marker":"[18]"},{"why":"Introduces the multiple-choice accuracy and probability-recalibration baselines (QA Eval, Prob Eval) that DCUE shows are vulnerable to reasoning without memorization.","marker":"[20]"},{"why":"Establishes the Kolmogorov-Smirnov statistic that DCUE uses to compare empirical cumulative distribution functions.","marker":"[1]"},{"why":"Provides the two-sample KS distribution used to convert the corrected statistic into the p-value that is the evaluation score.","marker":"[33]"},{"why":"Supplies the language-model prompting approach used to extract the minimal set of core answer tokens.","marker":"[34]"}],"fun_headline_variants":["LLM unlearning scored without a retrained model","DCUE metric corrects bias to gauge real forgetting","Unlearning evaluation that skips retraining entirely","KS test on corrected confidences reveals true unlearning","Validation set fuels new unlearning metric DCUE"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing bet is that the shift caused by fine-tuning on the kept data is captured by the smaller of two directly observable shifts, and that a validation set from the fine-tuning domain is available but unused; if either fails, the corrected score is unreliable.","fun_headline_variants_meta":{"raw":{"variants":["LLM unlearning scored without a retrained model","DCUE metric corrects bias to gauge real forgetting","Unlearning evaluation that skips retraining entirely","KS test on corrected confidences reveals true unlearning","Validation set fuels new unlearning metric DCUE"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000169,"raw_usage":{"total_tokens":1223,"prompt_tokens":864,"completion_tokens":359,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":480,"completion_tokens_details":{"reasoning_tokens":285}},"tokens_in":480,"tokens_out":359,"duration_ms":4727,"temperature":1.0,"reasoning_tokens":285,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T05:40:37.310471+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a case where the retained data shifts the model on the validation set in a direction opposite to its effect on the forget set, compute DCUE's corrected p-values, and compare them with p-values obtained from a genuinely retrained model; if the two disagree sharply, the approximation $\\delta_S \\approx \\min\\{S^v_{o,u}, S^f_{o,u}\\}$ has failed.","supporting_citations":[{"cited_title":"Sulla determinazione empirica di una legge didistribuzione","cited_arxiv_id":null,"evidence_quote":"Establishes the Kolmogorov-Smirnov statistic that DCUE uses to compare empirical cumulative distribution functions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the two-sample KS distribution used to convert the corrected statistic into the p-value that is the evaluation score."},{"cited_title":"Chatgpt: Optimizing language models for dialogue","cited_arxiv_id":null,"evidence_quote":"Supplies the language-model prompting approach used to extract the minimal set of core answer tokens."}],"review_version":1}