Pith. sign in

REVIEW 4 major objections 7 minor 1 cited by

RULE: Reinforcement UnLEarning Achieves Forget-Retain Pareto Optimality

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read RULE frames LLM unlearning as refusal-policy learning: a small forget sample plus synthesized near-miss queries, trained by on-policy RL, beat full-data baselines on forgetting, naturalness, and utility.

desk verdict The refusal-boundary RL idea is real and worth testing, but the paper oversells Pareto optimality and leans too hard on unverified synthetic data. read the letter →

arxiv 2506.07171 v1 pith:L622EYWZ submitted 2025-06-08 cs.CL cs.LG

Chenlong Zhang , Zhuoran Jin , Hongbang Yuan , Jiaheng Wei , Tong Zhou , Kang Liu , Jun Zhao , Yubo Chen This is my paper
classification cs.CLcs.LG
keywords LLMunlearningrefusalpolicyoptimizationreinforcementlearningforget-retaintrade-offParetooptimalityboundarydatasynthesisresponsenaturalness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RULE sets out to establish that unlearning a piece of knowledge from an LLM is best done by teaching the model where to refuse, not by erasing information. The recipe is two-stage: a short supervised 'rejection steering' pass on a small sample of the forget set so the model can produce natural refusals, then on-policy reinforcement learning on those forget queries plus a small set of synthesized 'boundary queries', reformulations in which the sensitive entity is swapped for a permissible one, with a verifiable reward that says refuse here, answer there. The paper's headline empirical claim is that with only about 12% of the forget set and 8% synthesized boundary data, RULE beats baselines that use the full forget set, by up to 17.5 points on forget quality and 16.3 points on response naturalness, while keeping general utility and extending refusal behavior to unseen related queries. If true, this matters because it removes two practical blockers of existing unlearning: the need for a cleanly partitioned retain set, which real training corpora almost never provide, and the tendency of gradient-based unlearning to produce collapsed, evasive, or hallucinated outputs that advertise the unlearning.

What carries the argument

The carrying object is the refusal boundary, and the machinery is two-stage refusal-policy optimization. Stage one, rejection steering, is a short supervised pass on a 12% forget sample with entity-specific refusal templates adapted from an existing 'I don't know' phrasing; it exists because pretrained models rarely emit refusals, so RL rollouts need a policy that can refuse at initialization. Stage two, refusal boundary optimization, runs on-policy RL over the forget sample plus the boundary set $\widetilde{\mathcal{D}}_r$: queries synthesized by prompting a general-purpose model to rewrite each forget query with the sensitive entity swapped for a permissible counterpart, forming hard negatives that are semantically close to forget queries but lie on the retain side. The reward function $r(x,y)$ is the mechanism that shapes the boundary: on $\mathcal{D}_f$ it credits refusal-template matches plus target-entity mention, while on $\widetilde{\mathcal{D}}_r$ it credits non-refusal plus ROUGE-L overlap with the original model's reference answer, and a KL term anchored on the rejection-steered model keeps updates near the refusal-competent policy. The theoretical claim in Theorem 1 is that this design yields a smaller worst-case mis-refusal risk than supervised fine-tuning for an equal token budget, because exploration adds $KmH$ additional observations and the boundary probes estimate retain-side risk instead of bounding it by the worst case.

What would settle it

Sample a batch of RULE's synthesized boundary queries for several forget targets and have annotators check each one for two properties: does the query still leak or require the forgotten content, and is the supplied reference answer factually correct? If a substantial share, say more than one in five, fails either check, then the boundary-reward signal is actively teaching the model to refuse or answer wrongly on genuine retain inputs, and the forget–retain Pareto claim would not survive a change of entity-replacement generator.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the unlearning objective can be written as a refusal-boundary constraint, where the ideal policy $\pi_\theta$ refuses forget-side queries ($\pi_\theta(\texttt{[refuse]}\mid x)\to 1$ for $x\in\mathcal{D}_f$) and answers retain-side ones, and that this boundary is learnable from drastically less data than current baselines use. RULE first fine-tunes the base model on a 12% sample of the forget set with supervised refusal outputs (rejection steering), producing a policy that can refuse at all; it then runs on-policy reinforcement learning with PPO, GRPO, or Reinforce++ over that same forget sample plus an 8%-scale boundary set $\widetilde{\mathcal{D}}_r$ of synthesized near-miss queries, where the sensitive entity has been replaced by a permissible counterpart while structure and length stay intact. The reward $r(x,y)$ is verifiable rather than learned: template-matched refusal plus mention of the key entity earns reward on $\mathcal{D}_f$, and non-refusal plus ROUGE-L agreement with the original model's answer earns reward on $\widetilde{\mathcal{D}}_r$. Empirically, the GRPO variant reports 27.7 overall forget quality on the RWKU benchmark (against 40.1 for the best baseline) with retain quality 73.7, a 89.1 score on a three-axis naturalness evaluation (versus 72.8 for the best baseline), and the highest Pareto-frontier AUC at every retain threshold from 0.4 to 0.7, which the paper reads as forget–retain Pareto optimality. The paper also claims a generalization advantage over supervised fine-tuning in its Theorem 1: on-policy rollouts enlarge the effective sample size and convert worst-case retain-side risk into a bound that decays with interaction.

Load-bearing premise

The load-bearing premise, which the paper's own conclusion flags as a scalability limit, is that the synthesized boundary queries truly sit on the answerable side of the refusal boundary and carry correct reference answers; if entity-swapped questions still touch the forgotten content, or the reference answers are wrong, the reward would train the model to refuse or misanswer exactly the inputs that ought to be kept.

Editorial extensions

If this is right

  • Unlearning no longer needs a cleanly partitioned retain set: RULE reports better forget quality with 12% of the forget queries and 8% synthesized boundary data than baselines that train on the full sets.
  • Refusal behavior generalizes: RULE transfers refusals to semantically related queries it never trained on, across all three RWKU forget categories (FB, QA, AA), which is the property that directly attacks the overfitting-to-$\mathcal{D}_f$ failure.
  • Response naturalness becomes a measurable axis of unlearning: RULE scores 89.1 on the combined readability/helpfulness/truthfulness evaluation versus 72.8 for the best baseline, so unlearned models need not leak the fact that they were unlearned.
  • The forget–retain trade-off is not forced to be zero-sum: RULE holds the largest Pareto-frontier AUC at every retain threshold from 0.4 to 0.7, supporting the claimed forget–retain Pareto optimality.
  • General utility can improve along with forgetting: RULE reports higher truthfulness and fluency than the original model on RWKU's utility dimensions, suggesting refusal learning reinforces honest abstention instead of damaging capability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My read is that the data economy, if it holds, relocates the hard part of unlearning from optimization to synthesis: the bottleneck becomes generating trustworthy near-miss queries with guaranteed-correct retain answers, and the RL stage might be portable to cheaper optimizers; a direct test is to vary the synthesizer, using smaller models or rule-based swaps, and see whether the forget–retain fro
  • The reward's ROUGE-L branch credits word-level overlap with the original model's answer on boundary queries, which may quietly reward reproducing the original phrasing rather than helpfulness; replacing ROUGE-L with an independent factuality check and re-measuring retain quality would separate faithful retention from paraphrase matching.
  • The paper's Theorem 1 suggests a transferable principle beyond unlearning: on-policy exploration enlarges the effective sample of any refusal-shaped objective, so safety refusal training and copyright takedown could inherit the same advantage; the paper's own relearning experiment is one hint, but checking relearning resistance across target types, longer fine-tuning budgets, and multilingual or m
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes RULE, a two-stage reinforcement-learning framework for LLM unlearning. Stage I (Rejection Steering) fine-tunes the base model on a small forget set to produce refusals; Stage II (Refusal Boundary Optimization) runs on-policy RL over the forget set and a synthetic boundary set generated by GPT-4o-mini via entity replacement. The reward function (Eq. 5) rewards template-matched refusals on forget queries and non-refusals with sufficient ROUGE-L overlap on boundary queries. The authors report strong forget quality and naturalness on RWKU and MUSE with only 12% forget data and 8% boundary data, and claim forget-retain Pareto optimality, supported by a generalization theorem in Appendix B.4.

Significance. If the empirical claims are reproducible, the paper makes a useful contribution: it reframes unlearning as refusal-policy learning with a verifiable reward, reports substantially more fluent refusals than GA/NPO/SimNPO, and demonstrates data efficiency and generalization to unseen queries. The naturalness evaluation is a valuable addition to unlearning benchmarks, and the two-stage design with rejection steering is a sensible practical recipe. However, the headline claims of retained utility and Pareto optimality are not supported by the reported operating points, and the method's utility preservation rests entirely on an unvalidated synthetic boundary set. The paper also contains an inconsistency about the source of gold answers and a theoretical appendix that is only a proof sketch. These issues are load-bearing for the central claims and require substantial revision.

major comments (4)
  1. [Tables 1 and 9; Sec. 5.3] The claim that RULE "maintains general utility" and achieves "forget-retain Pareto optimality" is contradicted by the reported retain-quality numbers. In Table 1, RULE GRPO has Retain Quality All = 73.7, below NPO (75.1), SimNPO (76.5), and the original model (87.6); in Table 9 the drop is larger (68.9 vs 75.1 and 76.5). A method that loses 13.9 or 18.7 points of neighbor-set retain quality relative to the original model cannot be described as maintaining utility unless the term is restricted to the four dimensions in Table 3, and even then the comparison is not made at matched forget levels. The Pareto analysis in Figure 4 uses thresholded AUC over training trajectories, but the final operating points themselves show RULE trading a large forget improvement for a meaningful retain loss; the paper should report matched-forget or matched-retain comparisons and confidence intervals before claiming Pareto optimality.
  2. [Appendix A.2; Eq. (5); Table 2; Sec. 6] The correctness of the synthetic boundary set Dtilde_r is the load-bearing component of the entire method, and it is not validated. The ablation in Table 2 shows that removing Dtilde_r collapses Retain to 23.6, so the method's utility preservation is purchased entirely by the entity-replacement queries. Yet Sec. 3.2 states that the gold answers y_gold are "generated by the original model," while the prompt in Appendix A.2 instructs GPT-4o-mini to return "ACCURATE_ANSWER_HERE"; the training-details paragraph in Sec. 4.1 says the boundary set is constructed by prompting GPT-4o. These are three different descriptions of the same data-generation step, and no code or data is released to resolve them. If GPT-4o-mini's replacements are not genuinely permissible or the answers are inaccurate, Eq. (5) rewards the model for refusing or mis-answering retain-side inputs. The authors should release the boundary data, report a human audit of permissibility and answer accuracy, and provide statistics on how often the replacement still touches the forget entity.
  3. [Eq. (5); Sec. 4.1; Figure 5] There is a circularity risk between the reward function and the forget-quality metric. The forget-branch reward in Eq. (5) directly optimizes template-matched refusal and key-entity mention, while RWKU's forget quality is measured by ROUGE-L against the original answer (lower is better), so a refusal will trivially produce a low score. The reported forget-quality gains may therefore reflect reward alignment rather than knowledge erasure. The relearning experiment in Figure 5 is also behavioral: it shows that the model continues to refuse after fine-tuning on forget passages, but it does not show that the target knowledge is absent from the model's representations. The authors should test for retained knowledge with indirect elicitation or representation-level probes, and should report whether the refusal behavior can be bypassed by rephrasing or jailbreaking.
  4. [Appendix B.4; Eqs. (1.1)-(1.2)] The theoretical analysis is not a proof and contains an unjustified step that is load-bearing for the paper's "theoretical evidence" claim. The bound in Eq. (1.2) is derived by treating RULE's exploration as if it samples from the true retain distribution P_r, but the actual algorithm samples from D_f union Dtilde_r, where Dtilde_r is a synthetic set whose relationship to P_r is exactly the unverified assumption from Appendix A.2. In addition, Step 4 asserts C_KL(pi) <= C(pi) exp(-1/2 E_x[KL(pi || pi_anchor)]) without justification; this capacity-reduction inequality is not generally true and appears dimensionally suspect. The authors should either provide a rigorous proof with explicit assumptions about Dtilde_r, or remove the theoretical claim and present the appendix as intuition.
minor comments (7)
  1. [Sec. 3.2; Appendix A.2] The notation D_r in Eq. (3) and Algorithm 1 is used for the combined retain/boundary set, but Sec. 3.1 defines D_r as the true retain set; this conflation makes the theory harder to follow.
  2. [Table 10] The MUSE-books table has formatting problems: several rows appear to list D_f/D_r token ratios as 0% even when the baseline is reported as using 100% of the forget set, and the columns are misaligned.
  3. [Appendix C.1] The claim that the refusal regex "aligns well with human annotation in over 95% of sampled cases" is not backed by any protocol, sample size, or inter-annotator agreement; please provide details or remove the number.
  4. [Appendix D.4] The case study in Table 8 is helpful, but the "Before" and "After" states in Figure 2b are not defined in the caption or text; please clarify which training steps they correspond to.
  5. [References] References [51] and [52] cite the same arXiv paper with different titles/venues; this should be corrected.
  6. [Sec. 5.3] The footnote "geq0.4" contains a LaTeX typo and should read ">=0.4".
  7. [General] The paper repeatedly says "codes will be available" but includes no repository link, no data, and no random seeds; given the proprietary GPT-4o-mini dependency, releasing the exact prompts, generated boundary sets, and evaluation code is essential for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: RULE's forget and retain results are measured on held-out benchmark queries and its generalization bound is an independent argument.

full rationale

I walked RULE's derivation chain from the two-stage training objective (Eqs. 4 and 5) to the reported forget quality, retain quality, naturalness, and Pareto claims. The training reward is a hand-designed proxy: on D_f it rewards refusal-template matches and target-entity mention, and on the synthesized boundary set it rewards non-refusal plus ROUGE-L overlap with reference outputs. The benchmark metrics are related (RWKU uses ROUGE-L, and refusal suppresses the forget metric), but the evaluation is not the training set: RULE trains on 12% of the forget set and 8% synthesized boundary data, while Table 1 reports metrics on the full benchmark forget and neighbor sets, so the improvements require genuine generalization and are not forced by construction. The ablation in Table 2 shows the synthesized boundary set is empirically load-bearing, and the Conclusion explicitly concedes dependence on synthetic boundary data; that is a data-quality and scalability limitation, not a circular derivation. The theoretical bound (Theorem 1) is a standard uniform-convergence argument comparing SFT sample size with RL rollouts; it does not import the conclusion via self-citation. Self-citations (RWKU and prior adversarial-unlearning work) are used as benchmarks and related work, not as the justification for RULE's central claim. I also note the unresolved discrepancy between §3.2 ("reference outputs ygold generated by the original model") and Appendix A.2's prompt asking GPT-4o-mini to return "ACCURATE_ANSWER_HERE"; this is a reproducibility concern, not circularity. I therefore find no step where a prediction reduces by definition or by self-citation to the paper's own inputs.

Assumptions & free parameters 6 free parameters · 5 assumptions · 1 invented entities

The method relies on several hand-set reward weights, an unreported ROUGE-L threshold, a hand-crafted refusal regex, and synthetic boundary data generated by a proprietary model. The theoretical analysis assumes standard generalization bounds and a particular operationalization of unlearning as refusal. No new physical entities are introduced.

free parameters (6)
  • alpha (forget reward weight) = 0.5
    Hand-set in Eq. 5 to balance refusal matching and key-entity mention; not fitted, but the reported results depend on this choice.
  • beta (boundary reward weight) = 0.5
    Hand-set in Eq. 5 to balance non-refusal and ROUGE-L content match.
  • tau (ROUGE-L threshold) = not specified in paper
    Threshold in Eq. 5 for rewarding boundary responses; its value is never reported, so the reward function is not fully specified.
  • refusal pattern set P_refuse = hand-crafted regex list in Appendix C.1
    The template matcher determines whether a response counts as a refusal, and its coverage directly shapes the reward and the resulting refusal behavior.
  • KL regularization coefficient = Table 7, appears garbled as 1e-2 or 1e-3
    The KL coefficient anchors RL around the rejection-steered policy; the exact value cannot be read reliably from the table.
  • boundary data ratio = about 8% of Dr on RWKU; 2.9% tokens on MUSE
    The amount of synthetic boundary data is a design choice, and performance may vary with this ratio.
assumptions (5)
  • standard math Standard Rademacher complexity generalization bounds apply to token-level sequences and to the mis-refusal risk R(pi).
    Invoked in Appendix B.4 Step 1 without proof; the proof sketch mixes token-level and sequence-level quantities.
  • ad hoc to paper The true retain distribution is approximated by the synthetic boundary set created by GPT-4o-mini entity replacement.
    Eq. 5 rewards non-refusal on the synthetic boundary set; if these queries are not representative of permissible inputs, the learned boundary is miscalibrated.
  • domain assumption Refusal behavior is the correct operationalization of unlearning.
    The objective in Eq. 2 defines unlearning as refusing forget-set queries and answering retain-set queries; alternative definitions are not considered.
  • domain assumption ROUGE-L against the original model's response measures acceptable response quality on boundary queries.
    Eq. 5 uses ROUGE-L(y, y_gold) above tau as a reward, assuming the original model's answers are correct and that overlap is a valid quality signal.
  • domain assumption LLM-as-judge naturalness scores approximate human judgments of readability, helpfulness, and truthfulness.
    Naturalness numbers come from automated evaluation with a rubric that leaks the intended behavior, and no human agreement study is reported.
invented entities (1)
  • refusal boundary
    purpose: Conceptual description of the learned policy that refuses forget-related queries and answers permissible queries.
    The refusal boundary is a framing device, not a new physical or ontological entity. It names the decision surface learned by the policy and has no independent falsifiable handle outside the paper's own reward and evaluation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RULE: Reinforcement UnLEarning Achieves Forget-Retain Pareto Optimality." pith.science (2026). https://pith.science/paper/L622EYWZ

@misc{pith2026250607171,
  author       = {Pith},
  title        = {Pith review of: RULE: Reinforcement UnLEarning Achieves Forget-Retain Pareto Optimality},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L622EYWZ}},
  note         = {Machine review of arXiv:2506.07171}
}
abstract

The widespread deployment of Large Language Models (LLMs) trained on massive, uncurated corpora has raised growing concerns about the inclusion of sensitive, copyrighted, or illegal content. This has led to increasing interest in LLM unlearning: the task of selectively removing specific information from a model without retraining from scratch or degrading overall utility. However, existing methods often rely on large-scale forget and retain datasets, and suffer from unnatural responses, poor generalization, or catastrophic utility loss. In this work, we propose Reinforcement UnLearning (RULE), an efficient framework that formulates unlearning as a refusal boundary optimization problem. RULE is trained with a small portion of the forget set and synthesized boundary queries, using a verifiable reward function that encourages safe refusal on forget--related queries while preserving helpful responses on permissible inputs. We provide both theoretical and empirical evidence demonstrating the effectiveness of RULE in achieving targeted unlearning without compromising model utility. Experimental results show that, with only $12%$ forget set and $8%$ synthesized boundary data, RULE outperforms existing baselines by up to $17.5%$ forget quality and $16.3%$ naturalness response while maintaining general utility, achieving forget--retain Pareto optimality. Remarkably, we further observe that RULE improves the naturalness of model outputs, enhances training efficiency, and exhibits strong generalization ability, generalizing refusal behavior to semantically related but unseen queries.

Figures

Figures reproduced from arXiv: 2506.07171 by the authors.

Figure 1
Figure 1. (a) Illustration of model behaviors under unlearning settings when queried about forgotten [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. (a) Comparison of model responses across three naturalness dimensions on forget queries [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Left: Train/Test reward curves of ReBOPPO and ReBOGRPO; Middle: Forget Quality (lower is better). Right: Retain Quality (higher is better). Each curve represents the mean ± standard deviation over different unlearning targets. prompt (w/o RS∗ ) yields only partial improvements, showing that instruction alone cannot substitute for behavior-driven learning [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: We compare unlearning methods under Retain Quality thresholds from 0.4 to 0.7. Each [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Evaluation of RULE’s robustness under the “ [PITH_FULL_IMAGE:figures/full_fig_p024_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. De-attribute to Forget for LLM Unlearning

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    DareU unlearns LLMs by PPO-optimizing attribution rewards so outputs are no longer attributable to forget owners, outperforming loss-based methods on forget-utility trade-offs.

Reference graph

Works this paper leans on

55 extracted references · 48 canonical work pages · cited by 1 Pith paper

  1. [1]

    Bubeck, V

    S. Bubeck, V . Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y . T. Lee, Y . Li, S. Lundberg, H. Nori, H. Palangi, M. T. Ribeiro, and Y . Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4, 2023

    work page 2023

  2. [2]

    Carlini, F

    N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-V oss, K. Lee, A. Roberts, T. B. Brown, D. Song, C. Raffel, et al. Extracting training data from large language models. InUSENIX Security Symposium, 2021

    work page 2021

  3. [3]

    H. Chen, F. Jiao, X. Li, C. Qin, M. Ravaut, R. Zhao, C. Xiong, and S. Joty. Chatgpt’s one-year anniversary: Are open-source large language models catching up?, 2024

    work page 2024

  4. [4]

    T. Chu, Y . Zhai, J. Yang, S. Tong, S. Xie, D. Schuurmans, Q. V . Le, S. Levine, and Y . Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training, 2025

    2025

  5. [5]

    Cobbe, V

    K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems.CoRR, abs/2110.14168, 2021

    arXiv 2021

  6. [6]

    Z. Di, Z. Zhu, J. Jia, J. Liu, Z. Takhirov, B. Jiang, Y . Yao, S. Liu, and Y . Liu. Label smoothing improves machine unlearning.arXiv preprint arXiv:2406.07698, 2024

    arXiv 2024

  7. [7]

    D’Oosterlinck, W

    K. D’Oosterlinck, W. Xu, C. Develder, T. Demeester, A. Singh, C. Potts, D. Kiela, and S. Mehri. An- chored preference optimization and contrastive revisions: Addressing underspecification in alignment. Transactions of the Association for Computational Linguistics, 13:442–460, 2025

    work page 2025

  8. [8]

    Dubey, A

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru, B. Roziere, B. Biron, B. Tang, B. Chern, C. Caucheteux, C. Nayak, C. Bi, C. Marra, C. McConnell, C. Keller,...

    work page 2024

Show all 55 references
  1. [9]

    C. Fan, J. Jia, Y . Zhang, A. Ramakrishna, M. Hong, and S. Liu. Towards llm unlearning resilient to relearning attacks: A sharpness-aware minimization perspective and beyond, 2025

    2025

  2. [10]

    C. Fan, J. Liu, L. Lin, J. Jia, R. Zhang, S. Mei, and S. Liu. Simplicity prevails: Rethinking negative preference optimization for llm unlearning, 2025

    2025

  3. [11]

    C. Fan, J. Liu, Y . Zhang, E. Wong, D. Wei, and S. Liu. Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation. InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11...

    2024

  4. [12]

    J. Hu, J. K. Liu, and W. Shen. Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models, 2025

    2025

  5. [13]

    J. Jang, D. Yoon, S. Yang, S. Cha, M. Lee, L. Logeswaran, and M. Seo. Knowledge unlearning for mitigating privacy risks in language models. In A. Rogers, J. Boyd-Graber, and N. Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Lingui...

    2023

  6. [14]

    J. Ji, Y . Liu, Y . Zhang, G. Liu, R. R. Kompella, S. Liu, and S. Chang. Reversing the forget-retain objectives: An efficient llm unlearning framework from logit difference.Advances in Neural Information Processing Systems, 37:12581–12611, 2024

    2024

  7. [15]

    J. Jia, J. Liu, P. Ram, Y . Yao, G. Liu, Y . Liu, P. Sharma, and S. Liu. Model sparsity can simplify machine unlearning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neural Information Processing Systems 36: Annual Conference on N...

    2023

  8. [16]

    Z. Jin, P. Cao, C. Wang, Z. He, H. Yuan, J. Li, Y . Chen, K. Liu, and J. Zhao. Rwku: Benchmarking real-world knowledge unlearning for large language models, 2024

    2024

  9. [17]

    Kassianik, B

    P. Kassianik, B. Saglam, A. Chen, B. Nelson, A. Vellore, M. Aufiero, F. Burch, D. Kedia, A. Zohary, S. Weerawardhena, A. Priyanshu, A. Swanda, A. Chang, H. Anderson, K. Oshiba, O. Santos, Y . Singer, and A. Karbasi. Llama-3.1-foundationai-securityllm-base-8b technical report, 2025

    2025

  10. [18]

    N. Li, A. Pan, A. Gopal, S. Yue, D. Berrios, A. Gatti, J. D. Li, A. Dombrowski, S. Goel, G. Mukobi, N. Helm-Burger, R. Lababidi, L. Justen, A. B. Liu, M. Chen, I. Barrass, O. Zhang, X. Zhu, R. Tamirisa, B. Bharathi, A. Herbert-V oss, C. B. Breuer, A. Zou, M. Mazeika, Z. Wang, ...

    2024

  11. [19]

    Z.-Z. Li, D. Zhang, M.-L. Zhang, J. Zhang, Z. Liu, Y . Yao, H. Xu, J. Zheng, P.-J. Wang, X. Chen, Y . Zhang, F. Yin, J. Dong, Z. Guo, L. Song, and C.-L. Liu. From system 1 to system 2: A survey of reasoning large language models, 2025

    2025

  12. [20]

    Liang, R

    J. Liang, R. Pang, C. Li, and T. Wang. Model extraction attacks revisited. InProceedings of the 19th ACM Asia Conference on Computer and Communications Security, ASIA CCS ’24, page 1231–1245, New York, NY , USA, 2024. Association for Computing Machinery

    2024

  13. [21]

    Lightman, V

    H. Lightman, V . Kosaraju, Y . Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe. Let’s verify step by step. InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024

    2024

  14. [22]

    C.-Y . Lin. ROUGE: A package for automatic evaluation of summaries. InText Summarization Branches Out, pages 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics

    2004

  15. [23]

    S. Liu, Y . Yao, J. Jia, S. Casper, N. Baracaldo, P. Hase, Y . Yao, C. Y . Liu, X. Xu, H. Li, et al. Rethinking machine unlearning for large language models.Nature Machine Intelligence, pages 1–14, 2025

    2025

  16. [24]

    S. Liu, Y . Yao, J. Jia, S. Casper, N. Baracaldo, P. Hase, Y . Yao, C. Y . Liu, X. Xu, H. Li, K. R. Varshney, M. Bansal, S. Koyejo, and Y . Liu. Rethinking machine unlearning for large language models, 2024

    2024

  17. [25]

    T. Liu, Y . Zhao, R. Joshi, M. Khalman, M. Saleh, P. J. Liu, and J. Liu. Statistical rejection sampling improves preference optimization. InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024

    2024

  18. [26]

    Y . Liu, G. Deng, Z. Xu, Y . Li, Y . Zheng, Y . Zhang, L. Zhao, T. Zhang, K. Wang, and Y . Liu. Jailbreaking chatgpt via prompt engineering: An empirical study, 2024

    2024

  19. [27]

    Z. Liu, G. Dou, Z. Tan, Y . Tian, and M. Jiang. Machine unlearning in generative ai: A survey, 2024

    2024

  20. [28]

    Maini, Z

    P. Maini, Z. Feng, A. Schwarzschild, Z. C. Lipton, and J. Z. Kolter. Tofu: A task of fictitious unlearning for llms, 2024

    2024

  21. [29]

    Rafailov, A

    R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. InAdvances in Neural Information Processing Systems, volume 36, 2024

    2024

  22. [30]

    Schulman, S

    J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In F. Bach and D. Blei, editors,Proceedings of the 32nd International Conference on Machine Learning, volume 37 ofProceedings of Machine Learning Research, pages 1889–1897, Lille, Fr...

    2015

  23. [31]

    Schulman, P

    J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation, 2018

    2018

  24. [32]

    Schulman, F

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms, 2017

    2017

  25. [33]

    Schwinn, D

    L. Schwinn, D. Dobre, S. Xhonneux, G. Gidel, and S. Gunnemann. Soft prompt threats: Attacking safety alignment and unlearning in open-source llms through the embedding space, 2024

    2024

  26. [34]

    Schwinn, D

    L. Schwinn, D. Dobre, S. Xhonneux, G. Gidel, and S. Günnemann. Soft prompt threats: Attacking safety alignment and unlearning in open-source llms through the embedding space. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors,Advances...

    2024

  27. [35]

    Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y . K. Li, Y . Wu, and D. Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024

    2024

  28. [36]

    Sheshadri, A

    A. Sheshadri, A. Ewart, P. Guo, A. Lynch, C. Wu, V . Hebbar, H. Sleight, A. C. Stickland, E. Perez, D. Hadfield-Menell, and S. Casper. Latent adversarial training improves robustness to persistent harmful behaviors in llms, 2024. 12

    2024

  29. [37]

    W. Shi, J. Lee, Y . Huang, S. Malladi, J. Zhao, A. Holtzman, D. Liu, L. Zettlemoyer, N. A. Smith, and C. Zhang. Muse: Machine unlearning six-way evaluation for language models, 2024

    2024

  30. [38]

    R. S. Sutton, D. McAllester, S. Singh, and Y . Mansour. Policy gradient methods for reinforcement learning with function approximation. In S. Solla, T. Leen, and K. Müller, editors,Advances in Neural Information Processing Systems, volume 12. MIT Press, 1999

    1999

  31. [39]

    P. Wang, L. Li, Z. Shao, R. Xu, D. Dai, Y . Li, D. Chen, Y . Wu, and Z. Sui. Math-shepherd: Verify and reinforce LLMs step-by-step without human annotations. In L.-W. Ku, A. Martins, and V . Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Compu...

    2024

  32. [40]

    Y . Wang, J. Wei, C. Y . Liu, J. Pang, Q. Liu, A. P. Shah, Y . Bao, Y . Liu, and W. Wei. Llm unlearning via loss adjustment with only forget data, 2024

    2024

  33. [41]

    B. Wei, W. Shi, Y . Huang, N. A. Smith, C. Zhang, L. Zettlemoyer, K. Li, and P. Henderson. Evaluating copyright takedown methods for language models, 2024

    2024

  34. [42]

    X. Wu, J. Li, M. Xu, W. Dong, S. Wu, C. Bian, and D. Xiong. DEPN: Detecting and editing privacy neurons in pretrained language models. In H. Bouamor, J. Pino, and K. Bali, editors,Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 287...

    2023

  35. [43]

    H. Xu, A. Sharaf, Y . Chen, W. Tan, L. Shen, B. Van Durme, K. Murray, and Y . J. Kim. Contrastive preference optimization: pushing the boundaries of llm performance in machine translation. InProceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024

    2024

  36. [44]

    H. Xu, N. Zhao, L. Yang, S. Zhao, S. Deng, M. Wang, B. Hooi, N. Oo, H. Chen, and N. Zhang. Relearn: Unlearning via learning for large language models, 2025

    2025

  37. [45]

    J. Yan, Y . Li, Z. Hu, Z. Wang, G. Cui, X. Qu, Y . Cheng, and Y . Zhang. Learning to reason under off-policy guidance, 2025

    2025

  38. [46]

    Y . Yao, J. Duan, K. Xu, Y . Cai, Z. Sun, and Y . Zhang. A survey on large language model (llm) security and privacy: The good, the bad, and the ugly.High-Confidence Computing, page 100211, 2024

    2024

  39. [47]

    Y . Yao, X. Xu, and Y . Liu. Large language model unlearning, 2024

    2024

  40. [48]

    F. Yu, A. Gao, and B. Wang. OVM, outcome-supervised value models for planning in mathematical reasoning. In K. Duh, H. Gomez, and S. Bethard, editors,Findings of the Association for Computational Linguistics: NAACL 2024, pages 858–875, Mexico City, Mexico, June 2024. Associati...

    2024

  41. [49]

    H. Yuan, Z. Jin, P. Cao, Y . Chen, K. Liu, and J. Zhao. Towards robust knowledge unlearning: An adversarial framework for assessing and improving unlearning robustness in large language models. In T. Walsh, J. Shah, and Z. Kolter, editors,AAAI-25, Sponsored by the Association ...

    2025

  42. [50]

    H. Yuan, Z. Jin, P. Cao, Y . Chen, K. Liu, and J. Zhao. Towards robust knowledge unlearning: An adversarial framework for assessing and improving unlearning robustness in large language models. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 2...

    2025

  43. [51]

    Zhang, L

    R. Zhang, L. Lin, Y . Bai, and S. Mei. Negative preference optimization: From catastrophic collapse to effective unlearning, 2024

    2024

  44. [52]

    Zhang, L

    R. Zhang, L. Lin, Y . Bai, and S. Mei. Negative preference optimization: From catastrophic collapse to effective unlearning.arXiv preprint arXiv:2404.05868, 2024

    2024 arXiv

  45. [53]

    K. Zhao, M. Kurmanji, G.-O. B˘arbulescu, E. Triantafillou, and P. Triantafillou. What makes unlearning hard and what to do about it, 2024

    2024

  46. [54]

    alignment

    G. Zhou, P. Qiu, C. Chen, J. Wang, Z. Yang, J. Xu, and M. Qiu. Reinforced mllm: A survey on rl-based reasoning in multimodal large language models, 2025. 13 A Data Construction A.1 Refusal Data Construction In the context of unlearning, we consider two essential types of queri...

    2025

  47. [55]

    Stephen King

    paper to ensure fair comparison. Models are fine-tuned using the llama3-8b-instruct checkpoint 19 under bf16 precision. We use cosine learning rate schedules and full-parameter tuning. Table 6 summarizes the key hyperparameters across different training stages. Table 6: Key tr...

    1974

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.