{"id":"5ff049b4-97b3-4e16-afdf-163c78bbb170","arxiv_id":"2505.06027","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Unilogit unlearns text from LLMs by setting the target token probability to 1/|V| and distilling from the current model, reporting stronger forgetting at comparable retained utility than NPO and UnDIAL.","lead":"Unilogit is a new LLM unlearning method that removes a target piece of knowledge by resetting that token's probability to a uniform value and redistributing the rest of the probability mass using the model's own current predictions. It claims better forget-retain trade-offs than existing methods while removing one tuning hyperparameter.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Dynamic self-distillation targets assume monotone convergence to the retrained model; this premise is untested in failure regimes and can reinforce drift.","rationale":"The reader's weakest assumption identifies the same load-bearing point: Unilogit's dynamic targets are only as good as the current model's approximation of the retrained model. My stress-test confirms this is the most fragile link in the central claim. The derivation in Appendix B is internally consistent, and the method is a plausible contribution, but the claimed superiority rests on monotone convergence that is neither theoretically guaranteed nor empirically probed in adversarial regimes. The paper's Section 4.4 provides a single illustrative KL trace on 100 samples, not a robustness analysis. The duplicate rows in Table 1 and the lack of seeds/code further weaken the empirical basis, but these are addressable reporting issues; they do not by themselves refute the method. Since the reader already conditioned the verdict on exactly these concerns, I recommend no change: the appropriate outcome remains conditional acceptance pending a direct test of the convergence assumption and improved empirical transparency.","tokens_in":17644,"tokens_out":3591,"duration_ms":36870,"concrete_test":"On MUSE-News with Llama 2 7B, run Unilogit+KL at lr=1e-5 and at a deliberately high lr (e.g., 5e-5) for 10 epochs. After each epoch, compute on the held-out 100 forget samples: (i) KL( p(y|x; theta_t) || p(y|x; theta_r) ) and (ii) KL( tilde{p}(y|x; theta_t) || p(y|x; theta_r) ), splitting the KL over the target token and the non-target vocabulary. If either KL increases after some epoch while the frozen-target variant (Unilogit with targets from theta_0) continues to decrease, the monotone-convergence premise fails exactly where hyperparameters are aggressive, and the dynamic-target advantage is not intrinsic. Compare final KnowMem and UtilityPreserv between dynamic and frozen variants; if frozen matches or beats dynamic in that regime, the central claim must be qualified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Unilogit's central advantage over UnDIAL is that its distillation targets are rebuilt from the current model parameters theta_t, on the assumption (Section 3) that 'at each unlearning step, the output distribution of theta should increasingly resemble that of theta_r.' This assumption is load-bearing because the target logit for the target token is set to log( sum_{i != k} exp(h_i(x; theta_t)) / (|V|-1) ), so the entire shape of the soft target tilde{p}(y|x; theta_t) is the current model's conditional distribution over non-target tokens. Nothing in the loss prevents theta_t from drifting away from theta_r on the forget set after aggressive updates; if it drifts, the non-target logits become a corrupted prior and the RKL objective trains the model to match its own errors, which can amplify the drift. The paper's only direct evidence against this is Figure 5 (center), a single-run KL trace on 100 MUSE-News forget samples, which does not probe failure regimes (e.g., high learning rate, larger forget fraction) where the monotone-convergence premise is most at risk. In addition, the empirical support for the headline claim is weakened by Table 1 containing duplicate rows (NPO+KL lr=5e-6 beta=0.1 and Unilogit lr=1e-5 appear twice) and by the absence of released code and multiple seeds, so the claimed Pareto superiority is not yet robustly established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Unilogit, a self-distillation method for LLM machine unlearning. The method constructs soft targets by modifying the target token's logit so that, after softmax, the target token has probability exactly 1/|V|, leaving all other logits unchanged. The targets are recomputed from the current model parameters at each step rather than from the starting model. The forget loss is a reverse-KL divergence between the current model output and this target, and the retain loss is a KL divergence from the starting model. Unilogit is compared with NPO, UnDIAL, GA, RKLD, ME+GD, and SimNPO on MUSE-News, RWKU, and an in-house e-commerce benchmark. The paper claims Unilogit achieves better Pareto trade-offs between forgetting and retention, is robust to hyperparameter choices, and produces output distributions closer to a retrained model than competing methods. Appendix B derives the target logit formula exactly, and ablations support the use of reverse KL, current-model targets, and the uniform-target logit adjustment.","tokens_in":17906,"tokens_out":6021,"duration_ms":64105,"significance":"If the empirical claims hold, Unilogit is a simple and attractive contribution: it removes UnDIAL's extra hyperparameter gamma, has an exact and interpretable derivation (Appendix B), and the idea of rebuilding distillation targets from the current model is a plausible and clean way to approximate the retrained model. The paper also makes a good-faith effort to evaluate the method against a golden retrained model by measuring KL divergences on the forget set. The strengths are real: the mathematics is exact, the comparisons span three benchmarks including a practical e-commerce setting, and the ablation study isolates the contributions of the main design choices. However, the evidence for the headline 'superior/robust' claim is weakened by single runs without error bars, duplicate rows in Table 1, a private benchmark whose details are deferred to a camera-ready version, and the absence of released code. The central premise of monotone convergence to the retrained model is plausible but is supported only by a single-run trace on 100 samples.","major_comments":[{"comment":"The RWKU section states that no retain set is available during unlearning, yet the full Unilogit objective in Eq. (2) includes a retain KL term over Dr. The paper does not state how Unilogit is trained on RWKU: whether lambda is set to zero, whether the retain term is dropped, or whether some other data are used. This matters because Table 2 compares methods under a no-retain-set protocol, and the reader cannot tell whether all methods were trained under the same objective. Please specify the exact objective used for each benchmark.","section":"Section 4.2 vs. Eq. (2)"},{"comment":"The claimed advantage of dynamic targets over static UnDIAL targets rests on the assumption that the current model's output distribution moves monotonically closer to the retrained model. The only direct evidence is a single-run KL trace on 100 MUSE-News forget samples, with no repeated seeds and no failure regimes (e.g., higher learning rates, larger forget fractions, or different retain-set sizes). If the monotone-convergence premise fails, the self-generated targets can reinforce model drift, and the method's advantage over the 'original model target' variant shown in Figure 6 would not transfer. Please test the premise across a range of hyperparameters or provide conditions under which it provably holds.","section":"Section 3 and Figure 5 (center)"},{"comment":"Table 1 contains duplicate hyperparameter rows with different numbers: NPO+KL lr=5e-6, beta=0.1 appears twice, and Unilogit+KL lr=1e-5 appears twice. No runs are repeated with multiple seeds and no error bars are reported. Since the central claim is Pareto superiority across hyperparameter sweeps, the current evidence does not establish that the observed ordering is stable. Please clarify the duplicates and report multiple seeds, at least for the main Pareto curves, or otherwise quantify run-to-run variability.","section":"Table 1 and experimental evidence"},{"comment":"The in-house e-commerce benchmark is used to support the 'practical reliability' and 'diverse real-world scenarios' conclusions, but the footnote states that 'More details about the in-house benchmark will be available in the camera-ready version of the paper.' The e-commerce results are therefore not reproducible from the submitted manuscript, and the private nature of the data means the evaluation cannot be independently verified. At minimum, provide dataset statistics, preprocessing details, and exact evaluation settings in the appendix, or clearly mark the e-commerce results as a case study rather than a general benchmark.","section":"Section 4.3 and footnote 1"}],"minor_comments":[{"comment":"The phrase 'without additional hyperparameters' is too strong: Eq. (2) still contains the retain-loss weight lambda, and the experiments require choosing a learning rate. The precise and defensible claim, made later in Section 3, is that Unilogit eliminates UnDIAL's gamma hyperparameter.","section":"Abstract and Section 3"},{"comment":"The paper calls KL(p(y|x;theta) || tilde p(y|x;theta)) a 'reverse KL' divergence, but with p as the current model and tilde p as the target distribution, this is the reverse of the common teacher-to-student forward KL in knowledge distillation. Please define the direction explicitly at first use to avoid confusion.","section":"Eq. (2)"},{"comment":"Several typos and formatting inconsistencies appear in Table 1; for example, 'lr 8.5e-6' and 'lr 8.75e-6' lack the equals sign used elsewhere, and the duplicate rows described in the major comments should be resolved.","section":"Table 1"},{"comment":"In the left and center panels of Figure 5, the two Unilogit checkpoints are not identified with their learning rates in the caption; the reader has to infer them from Table 1. Please label the bars or add this information to the caption.","section":"Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is sound and the derivation in Appendix B is exact; I do not see a circularity problem because the target logit formula follows from the uniform-probability constraint. The main risk is empirical: single runs, duplicate table rows, and a deferred private-benchmark description leave the Pareto-superiority claim less solid than the text suggests. The monotone-convergence premise is also untested in failure regimes. I recommend major revision with the expectation that the authors can address these points with additional experiments and clarifications."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the target-logit derivation in Appendix B is correct, and the method is a legitimate, small but real step past UnDIAL. I'd send it out, but the empirical case for the reported Pareto curve is not yet as strong as the text claims.\n\nWhat's genuinely new: (1) the closed-form logit value that makes the target token exactly 1/|V| after softmax—no extra gamma hyperparameter, and the derivation is exact; (2) rebuilding the non-target distribution from the current model's parameters at each step, rather than from theta_o. The ablation in D.1 shows that this dynamic-target choice beats the static-target version, and Figure 5's KL traces give at least some direct evidence for the convergence-to-retrained-model assumption. That is the right kind of evidence to include.\n\nThe soft spots are real but mostly fixable. Every run appears to be a single seed; no error bars anywhere. Table 1 has duplicate rows: NPO+KL lr=5e-6 beta=0.1 appears twice with different numbers, and Unilogit+KL lr=1e-5 appears twice as well. The e-commerce benchmark is described as \"in-house\" with details promised for the camera-ready version, and no code is released. That is a lot of missing reproducibility for a paper whose headline claim is robustness.\n\nThe deeper concern is the monotone-convergence premise: the whole method assumes the current model drifts toward the retrained model, so its non-target logits become better targets. The paper states this and supports it with one 100-sample KL trace on MUSE-News. It does not test failure regimes—larger learning rate, bigger forget fraction, or cases where the forget set is not from the same distribution as the retain set. The ablation shows dynamic targets help in the tested regime, but that is not the same as showing the method is robust where the assumption breaks. I don't think this is fatal, but it needs probing.\n\nBottom line: worth a serious referee. I'd ask for multiple seeds, error bars, Table 1 cleanup, code or a real description of the private benchmark, and at least one stress test of the convergence assumption before publication. The method itself is sound.","headline":"Deriving the exact uniform-target logit is clean and the dynamic self-distillation step is a real advance over UnDIAL, but the Pareto-superiority claim rests on single runs and a private benchmark; fix that.","tokens_in":18459,"tokens_out":2155,"would_cite":false,"duration_ms":20841,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Unilogit claims that forcing the target token to uniform probability, with targets recomputed from the current model's logits each step, gives better unlearning than static-target distillation baselines.","keywords":["machine unlearning","large language models","self-distillation","reverse KL divergence","uniform target distribution","logit adjustment","forget-retain trade-off","right to be forgotten"],"falsifier":"Run Unilogit on a benchmark where the golden retrained model $\\theta_r$ is available and compute, on the forget set, the KL divergence between the unlearned model's output distribution and $\\theta_r$'s across the learning-rate sweep; the central claim fails if Unilogit's KL is not lower than UnDIAL's and NPO's at matched retention. A second check is the ablation comparing targets rebuilt from the current model against targets built once from the starting model: if the dynamic-target version does not beat the static-target version at the same utility level, the core mechanism is not doing the work.","tokens_in":17458,"feed_emoji":"🧹","tokens_out":8547,"duration_ms":77600,"temperature":0.7,"pith_summary":"The paper introduces Unilogit, a self-distillation method for machine unlearning in large language models. Its central claim is that a target token should be forcibly assigned a uniform probability of $1/|V|$ after softmax, with all other logits taken from the current model, and that these targets should be recomputed from the current parameters at every training step. Unilogit then optimizes reverse KL divergence between the model's output and these targets on the forget set, plus a KL regularizer to the starting model on the retain set. The paper reports that this dynamic target construction yields more accurate approximations of a fully retrained model's outputs and better forget/retain Pareto curves than UnDIAL and NPO, without adding a hyperparameter to tune. If correct, the method offers a practical way to comply with right to be forgotten requirements at a fraction of the cost of retraining.","feed_headline":"Uniform-target distillation unlearns LLM data without extra tuning","feed_subtitle":"Paper reports better forget-retain trade-offs than NPO and UnDIAL on three benchmarks.","key_machinery":"The central object is the uniform-target logit adjustment $\\tilde{h}(x;\\theta)=(1-t)\\,h(x;\\theta)+t\\,\\log\\!\\left(\\frac{\\sum_{i\\neq k}\\exp(h_i(x;\\theta))}{|V|-1}\\right)$, where $t$ is a one-hot vector selecting the target token $k$. This formula sets exactly one logit to the value that makes the corresponding softmax probability equal to $1/|V|$, the uniform distribution over the vocabulary, while leaving all non-target logits intact. The resulting soft-label distribution $\\tilde{p}=\\mathrm{softmax}(\\tilde{h})$ is the distillation target in a reverse-KL forget loss, and the retain objective is a forward KL divergence between the current model and the starting model on the retain set. The mechanism carries the argument by replacing the discrete choice of how strongly to suppress the target token with a self-consistent calculation from the model's own current distribution, which the paper argues should increasingly match the retrained model.","core_discovery":"Unilogit's core discovery is that the amount by which a target token's logit must be suppressed can be derived from the model's own current logits, rather than chosen by a hyperparameter. For a forget sample with target token $k$, the method sets $\\tilde{h}_k(x;\\theta)=\\log\\!\\left(\\frac{\\sum_{i\\neq k}\\exp(h_i(x;\\theta))}{|V|-1}\\right)$, which makes $\\mathrm{softmax}(\\tilde{h})_k=1/|V|$. All other logits remain unchanged, so the probability mass removed from the target token is redistributed according to the current model's belief about the other vocabulary items. The paper argues that this produces a better approximation of the golden retrained model's outputs than UnDIAL's heuristic subtraction of a fixed $\\gamma$, and that rebuilding the targets from the latest parameters each step yields progressively more accurate guidance. The reported experiments claim that Unilogit achieves the most optimal Pareto curves on MUSE-News, RWKU, and an e-commerce benchmark, and that its final output distributions are closer in KL divergence to the retrained model than those of NPO or UnDIAL.","pith_inferences":["If the monotonic-progress assumption holds, the dynamic-target recipe should transfer to other self-distillation objectives and to continual-learning settings where the target distribution needs to track an evolving model; this is an extension the paper does not claim.","A discriminating stress test would compare Unilogit against a pure uniform-target baseline that flattens all logits; if non-target logits are the main source of signal, Unilogit's edge over that baseline should grow as the forget set shrinks.","The paper's token-level uniformity is a simplification; weighting tokens by their importance to the forget objective (a limitation the paper names) is a natural next step that could improve precision on structured text without changing the core formula."],"forward_implications":["Unilogit removes the need for the extra $\\gamma$ hyperparameter that UnDIAL requires, so deployment needs only the learning rate to be tuned.","Because the targets are recomputed from the current model each step, Unilogit can be applied in settings where the retain set is not available during unlearning, as demonstrated on RWKU.","If the KL-divergence analysis holds, unlearned models produced by Unilogit are closer in output distribution to a model retrained without the forget data than models produced by NPO or UnDIAL.","The reverse-KL choice is load-bearing: the ablation shows it gives stronger forgetting than forward KL at the same level of utility preservation.","The method transfers to real-world structured-text products, as shown by the e-commerce benchmark where Unilogit maintains general utility while reducing forget-set completion."],"supporting_citations":[{"why":"Supplies the UnDIAL baseline and the logit-adjustment idea that Unilogit refines; the main comparison in all experiments.","marker":"Dong et al. 2024"},{"why":"Provides the NPO baseline, a state-of-the-art preference-optimization unlearning loss that Unilogit must outperform on the Pareto curves.","marker":"Zhang et al. 2024"},{"why":"Provides the RKLD baseline, a reverse-KL distillation unlearning method, and supports the comparison on forgetting effectiveness.","marker":"Wang et al. 2024a"},{"why":"Defines the MUSE-News benchmark, including the forget and retain sets and the evaluation tasks KnowMem, UtilityPreserv, and PrivLeak.","marker":"Shi et al. 2024"},{"why":"Defines the RWKU benchmark used for the stricter no-retain-set unlearning scenario.","marker":"Jin et al. 2024"},{"why":"Provides the ME+GD baseline that pushes predictions toward a uniform distribution and is compared in all three benchmarks.","marker":"Yuan et al. 2024"},{"why":"Supplies the observation that untrained models produce near-uniform outputs, which justifies the $1/|V|$ target probability.","marker":"Tang et al. 2024"}],"fun_headline_variants":["Unlearn LLM data using model's own logit targets","Uniform-target distillation: no hyperparameters needed","Robust LLM unlearning via self-derived suppression","No extra tuning: Unilogit sets its own unlearning targets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the current model's non-target logits are a reliable prior for what the retrained model would output on forget samples, and that the model's distribution moves monotonically closer to the retrained model as unlearning progresses; if either fails, the self-generated targets can reinforce the model's own errors.","fun_headline_variants_meta":{"raw":{"variants":["Unlearn LLM data using model's own logit targets","Uniform-target distillation: no hyperparameters needed","Robust LLM unlearning via self-derived suppression","No extra tuning: Unilogit sets its own unlearning targets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000252,"raw_usage":{"total_tokens":1562,"prompt_tokens":947,"completion_tokens":615,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":549}},"tokens_in":563,"tokens_out":615,"duration_ms":6016,"temperature":1.0,"reasoning_tokens":549,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:49:41.897294+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Unilogit on a benchmark where the golden retrained model $\\theta_r$ is available and compute, on the forget set, the KL divergence between the unlearned model's output distribution and $\\theta_r$'s across the learning-rate sweep; the central claim fails if Unilogit's KL is not lower than UnDIAL's and NPO's at matched retention. A second check is the ablation comparing targets rebuilt from the current model against targets built once from the starting model: if the dynamic-target version does not beat the static-target version at the same utility level, the core mechanism is not doing the work.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the observation that untrained models produce near-uniform outputs, which justifies the $1/|V|$ target probability."}],"review_version":1}