Pith. sign in

REVIEW 4 major objections 5 minor 39 references

Beyond Binary Rewards: A Comparative Study of Reward Design for Reinforcement Unlearning

T0 review · 4 major / 5 minor · reviewed 2026-07-31 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Richer verifiable rewards make reinforcement unlearning up to 3× faster without hurting model utility.

desk verdict Solid empirical extension of PURGE: denser verifiable rewards speed RUL, but the abstract oversells Exponential and the 3× claim is PageRank-only. read the letter →

arxiv 2607.27968 v1 pith:H4RJVQ4U submitted 2026-07-30 cs.LG

Efstratios Zaradoukas , Davide Gabrielli , Bardh Prenkaj , Gjergji Kasneci This is my paper
classification cs.LG
keywords machineunlearningreinforcementlearningRLVRGRPOrewarddesignlargelanguagemodelsverifiablerewardsRWKU
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Machine unlearning tries to erase specific knowledge from a trained language model without retraining from scratch. Recent methods cast this as reinforcement learning with rewards that can be checked directly from the model’s outputs, but they usually give only a yes/no signal: the answer either avoided forbidden concepts or it did not. That sparse signal is noisy and slow. This paper argues that verifiability does not require sparsity. It separates the two ideas and builds denser rewards that still need only the model’s text and a public forget list: one that penalizes by how many forbidden items appear, and one that weights those items by semantic importance via a PageRank-style graph. On a real-world knowledge unlearning benchmark, both beat the binary baseline, reach similar forgetting much sooner—up to three times fewer steps—and leave general capabilities intact. The practical claim is that reward design itself is a major lever for making selective forgetting scalable.

What carries the argument

A reward decomposition that splits any verifiable reward into an information component I (what is read from forbidden-match counts and optional side information) and a transformation Γ that maps that information into [0,1]. This hierarchy runs from a single bit (binary) through total violation counts (exponential) to importance-weighted matches (PageRank on a semantic graph of forget items).

What would settle it

Train the same setup on RWKU targets but score completions that paraphrase or allude to the forgotten facts without using any listed surface forms; if denser rewards no longer improve forget metrics or speed relative to binary, the central efficiency claim fails.

Watch

Extended reading notes

Core claim

Within reinforcement unlearning optimized by group-relative policy updates, denser but still verifiable rewards—count-based exponential decay and semantically weighted PageRank penalties—consistently forget target knowledge faster than binary presence/absence rewards, reaching comparable forget-set performance in up to one-third the training steps while preserving neighbor knowledge and general utility metrics.

Load-bearing premise

Counting surface mentions of a fixed list of extracted entities is treated as a faithful stand-in for whether the model has actually forgotten the underlying knowledge.

Editorial extensions

If this is right

  • Practitioners can keep the same verifiable RL unlearning pipeline and swap only the reward to cut training steps substantially.
  • Semantic structure among forget entities becomes usable training signal without needing the original training data.
  • Binary yes/no rewards are an implementation choice, not a requirement of verifiability, so denser designs remain legitimate under the same privacy constraints.
  • Utility and neighbor metrics staying flat implies efficiency gains need not trade off general capability in this regime.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same decomposition could be tried on other verifiable RL tasks (math, code, safety) where sparse binary checkers currently dominate.
  • If surface-form matching is the bottleneck, combining these rewards with paraphrase-aware or embedding-based match operators would be a direct next stress test.
  • Adaptive or curriculum schedules that start dense and anneal toward binary may further stabilize late-stage training once most violations are already rare.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies reward design for Reinforcement Unlearning (RUL) within the PURGE/GRPO setup. It argues that sparse binary verifiable rewards give a weak learning signal, introduces a decomposition that separates verifiability from sparsity via an information hierarchy (I_min ≺ I_count ≺ I_structured), and proposes two denser verifiable rewards: an exponential penalty on total forbidden-match counts and a PageRank-weighted penalty on a kNN semantic graph over the forget set. On RWKU with Phi-3-Mini, the authors report that these rewards improve unlearning efficiency versus a reproduced binary/PURGE baseline—claiming up to 3× fewer steps to comparable forget-set gains—while utility metrics stay roughly stable. Ablations cover the exponential decay constant τ and several PageRank weight-redistribution variants (raw, linear, softmax).

Significance. If the efficiency claims hold under honest reporting, the work is a useful, practical contribution to RL-based LLM unlearning: it shows that verifiability need not imply binary sparsity, and that semantic weighting of forget entities can materially speed GRPO-style unlearning without collapsing general capability on RWKU utility splits. Strengths include a clear conceptual framework, a reproduced external baseline (PURGE/Binary) on a public benchmark, systematic τ and PageRank-variant ablations, and public code. The contribution is incremental rather than foundational—it builds directly on PURGE’s pipeline and surface-form matching—but reward design as a lever for RUL efficiency is a legitimate and underexplored axis. The PageRank-softmax speedup curve (Fig. 2) is the most interesting empirical finding and would matter to practitioners if restated with appropriate scope.

major comments (4)
  1. [Abstract; §6.2; Table 1; Fig. 2] Abstract and §6.2 claim that “both rewards consistently outperform the binary setting, while reaching similar forgetting performance up to 3× faster.” Table 1 (primary 1500-step endpoint, 20 targets) does not support the Exponential half of that claim on the Forget metrics the method optimizes: Exponential FB/QA/AA are 0.379/0.382/0.417 vs Binary 0.372/0.365/0.408 (equal or worse within reported stds). Only PageRank-Softmax improves at convergence (0.346/0.350/0.390). Fig. 2’s 3× result is explicitly PageRank reaching Binary’s step-1500 aggregate Δ by step 500; no analogous curve or claim is established for Exponential. The abstract and contribution list must be revised to match the tables (e.g., “PageRank-softmax … up to 3×; exponential is competitive / faster early but not stronger at convergence”), or Exponential must be shown to dominate on a clearly defined efficiency metric with un
  2. [Table 1; §6.2] Final PageRank–Binary gaps on Forget are small relative to cross-target standard deviations (~0.15–0.21, n=20). Table 1 reports mean±std but no paired tests, confidence intervals on Δ, or multiple-comparison correction across metrics/variants. The “consistently outperform … at convergence” language is therefore weakly locked even for PageRank-Softmax. Please add paired per-target tests (or bootstrap CIs on the forget-set Δ) and temper claims accordingly; if significance is marginal, center the paper on efficiency-to-target rather than final superiority.
  3. [App. B; Theorem 1; §1 contributions (1)] Theorem 1 (App. B) asserts rate_bin < rate_exp < rate_pr with scaling formulas T_bin ∝ 1/(p(1−p)), T_exp ∝ τ², T_pr ∝ d(ε), but only a proof sketch under unspecified “reasonable assumptions” on violation distributions is given. Lemma 1’s bound Var[Â] ≤ Var[φ]/σ² is also informal (Â already divides by σ). The main text advertises a “theoretical framework” and “theory-principled” rewards (§1, contributions). Either upgrade App. B to a precise statement with explicit assumptions and a real argument linking GRPO advantage variance to iteration complexity, or demote the theory to a heuristic motivation and remove rate-ordering theorem language from the contributions.
  4. [§5 Level 3; App. B Eqs. (21); §6.3] The main-text Level-3 PageRank reward (§5) defines importance weights w_j via personalized PageRank but never states the exact map from (M(y;F), w) to φ ∈ [0,1], unlike Eq. (10)–(11) for binary/exponential. App. B then writes φ_pr = clip(1−P,0,1) with P = Σ_j w_j · 1[m_j(y)>0] (presence, not counts), which may disagree with a count-weighted reading of “weights penalties by semantic importance” and with the softmax/linear redistributions in §6.3. Please give a single canonical equation for φ_pr in the main text (including how softmax/linear act on π_PR), and ensure experiments, figures, and theory all use that definition.
minor comments (5)
  1. [passim] Several glued words appear throughout (e.g., “Machineunlearning”, “verifiableRewards”, “Weintroducetwoprincipled”, “AsshowninTable2”). A full copy-edit pass is needed.
  2. [Fig. 1] Fig. 1 caption and body refer to a Stephen King graph; ensure the figure is readable in grayscale and that node labels match the forget-set construction description.
  3. [Table 1; §6.1] MIA columns in Table 1 show large negative LOSS values with FM slightly more negative than RM for unlearned models; a one-sentence interpretation of the desired FM↑/RM↓ direction relative to Base would help non-RWKU readers.
  4. [§5; App. C.2] Hyperparameters k (kNN), softmax temperature, and exact entity-encoder model for the semantic graph are only partly specified (§5 / App. C). List them fully for reproducibility alongside θ=0.5 and α=0.85.
  5. [§2.3; References] Related work cites PURGE as arXiv:2601.20568 and RULE; ensure citation keys and year formatting are consistent with the venue style and that contemporaneous RL unlearning baselines are not omitted without comment.

Circularity Check

1 steps flagged · score 1.0 of 10

No equation-level circularity: main efficiency claims are empirical comparisons on external RWKU against a reproduced binary baseline; only mild non-load-bearing self-citation of the authors' PURGE pipeline.

  1. self citation load bearing [§2.3, §3.3, §6.1; citation [37] PURGE]
    "Recent work has introduced Reinforcement Learning with Verifiable Rewards (RLVR) as a promising foundation for machine unlearning, with PURGE [37] and RULE [38] providing some of the first instantiations... In this work, we build on PURGE to mitigate sparsity issues... our approach is compared against... PURGE [37] (labeled as Binary...)"

    The experimental scaffold (synthetic forget-set construction, binary reward baseline, GRPO unlearning loop) is taken from the authors' own prior PURGE paper rather than an independent external method. This is ordinary self-citation of prior systems work and is not load-bearing for the comparative claim: Binary is re-run, RWKU is external, and the ranking of rewards is measured rather than assumed. Flagged only as mild scaffold dependence, not as a definitional reduction of the 3× result.

full rationale

The paper's load-bearing claim is experimental: within GRPO/RUL, exponential and PageRank-style verifiable rewards improve forget-set progress versus a sparse binary reward, with PageRank-Softmax reaching binary's late-training aggregate Δ earlier (Fig. 2; Table 1; §6.2). That comparison is run on the external RWKU benchmark with a reproduced Binary/PURGE baseline and reported utility/neighbor/MIA controls; nothing in the reward definitions (Eqs. 10–13) or the GRPO objective (Eq. 3) forces the measured speedups by construction. The reward decomposition (Defs. 3–5) is framing, not a prediction that equals its inputs. Appendix B's convergence ordering is a variance sketch motivating denser signals; it does not fit parameters from the target curves and then relabel them as theory. Hyperparameter choices (τ from the §6.3 sweep; PageRank-Softmax from Table 2) are standard ablations on the same task family, not presented as out-of-sample first-principles predictions. The only mild circularity-adjacent element is dependence on the authors' own PURGE pipeline (forget-set construction, binary baseline, GRPO setup) via self-citation [37], which supplies the experimental scaffold but does not make the comparative ranking tautological. Per the rubric this is at most a minor non-load-bearing self-citation (score 1), not a self-definitional or fitted-as-prediction loop.

Assumptions & free parameters 5 free parameters · 5 assumptions · 2 invented entities

The central speedup claim rests on the PURGE-style verifiable unlearning setup (synthetic forget entities + GRPO), on ROUGE-L/MIA/utility metrics as success criteria, and on several hand-chosen reward hyperparameters. Theory adds informal variance arguments rather than a closed derivation. No new physical entities; the main invented machinery is the I/Γ reward decomposition and the PageRank forget-graph reward.

free parameters (5)
  • exponential decay constant τ = 0.5 (selected from {0.1,0.25,0.5,1.0,2.0,5.0})
    Shapes ϕ_exp=exp(-m/τ); swept and set to 0.5 because it jointly minimized forget FB/QA/AA on the evaluation metrics (§6.3, Fig. 3).
  • PageRank damping α = 0.85
    Personalized PageRank fixed at conventional 0.85 without sensitivity study in main text (App. C.2).
  • semantic graph threshold θ and kNN k = θ=0.5; k unspecified in main text
    Edges kept only for cosine similarity ≥θ among k nearest forget items; θ=0.5 stated, k not clearly fixed in main hyperparameter list (§5, App. C.2).
  • PageRank weight redistribution (softmax temperature / linear / exprank τ) = Softmax default (temperature not numerically stated)
    Raw PageRank mass collapses; softmax/linear/exprank transforms are design choices selected by the same RWKU forget metrics (Table 2, App. D).
  • GRPO group size G, clip ε, KL weight β, train steps = G=8, ε=0.2, β=0.001, 1500 steps
    Copied from PURGE defaults; control optimization noise and deviation from reference (App. C.2).
assumptions (5)
  • domain assumption A completion has unlearned a concept iff it avoids surface forms in a constructed forget entity set F (verifiable binary success criterion).
    Inherited from PURGE and restated in §3.3 Eq. (4) and Defs. 1–3; load-bearing for all rewards in the hierarchy.
  • domain assumption GRPO group-normalized advantages plus small KL to the original model are a valid optimizer for unlearning policies.
    Preliminaries §3.2–3.3; no alternative RL algorithms compared.
  • ad hoc to paper Advantage variance / reward support arguments imply rate_bin < rate_exp < rate_pr under 'reasonable assumptions' on violation distributions.
    Theorem 1 and Lemmas in App. B are proof sketches tying denser rewards to faster convergence; not a standard theorem from prior RL literature as stated.
  • domain assumption RWKU ROUGE-L forget/neighbor probes, LOSS MIA, and utility suite adequately measure unlearning quality and side effects.
    Entire §6 evaluation stack; standard for the benchmark but still an external validity assumption.
  • domain assumption Forget-set entities extracted via external LLM NER + Top-K (+ manual validation) faithfully represent knowledge to remove.
    §3.3 Entity Extraction / Forget Set Construction stages carried over from PURGE.
invented entities (2)
  • Reward decomposition ϕ=Γ(I(y;F,ψ);H) with information hierarchy I_min ≺ I_count ≺ I_structured
    purpose: Separate verifiability from sparsity and organize binary vs exponential vs PageRank designs.
    Introduced in §4; taxonomy rather than empirical object; useful framing but not independently measured outside this paper.
  • PageRank-inspired verifiable unlearning reward on a kNN semantic forget graph
    purpose: Weight penalties by semantic centrality to the primary forget target.
    §5 Level 3 and App. D; graph is built from public embeddings of F, so computable, but the reward as an unlearning objective is native to this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Binary Rewards: A Comparative Study of Reward Design for Reinforcement Unlearning." pith.science (2026). https://pith.science/paper/H4RJVQ4U

@misc{pith2026260727968,
  author       = {Pith},
  title        = {Pith review of: Beyond Binary Rewards: A Comparative Study of Reward Design for Reinforcement Unlearning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H4RJVQ4U}},
  note         = {Machine review of arXiv:2607.27968}
}
abstract

Machine unlearning seeks to selectively remove specific knowledge from trained language models without full retraining, a growing necessity under privacy regulations such as GDPR and the EU AI Act. Recent work has reformulated unlearning as a Reinforcement Learning with Verifiable Rewards (RLVR) problem, where models are optimized against verifiable rewards computed directly from their outputs. However, existing methods rely on sparse binary rewards that provide minimal learning signal, indicating only whether forbidden content was avoided, and limiting convergence speed. In this paper, we study how reward design affects unlearning efficiency within the Reinforcement Unlearning (RUL) framework. We introduce a principled reward decomposition framework that decouples verifiability from sparsity, and propose two new reward functions: an exponential reward that provides graded penalties based on the count of forbidden-concept occurrences, and a PageRank inspired reward that weights penalties by semantic importance. We conduct experiments on the Real World Knowledge Unlearning (RWKU) benchmark, demonstrating that both rewards consistently outperform the binary setting, while reaching similar forgetting performance up to $3\times$ faster and preserving general model utility. Our results show that reward design is a key driver of unlearning efficiency offering a practical path toward scalable and efficient machine unlearning.

Figures

Figures reproduced from arXiv: 2607.27968 by the authors.

Figure 1
Figure 1. PageRank semantic graph for Stephen King. Node color and size reflect importance weights. The author node dominates the graph, with works such as The Shining and The Stand forming the high-weight core. Level 3: PageRank Reward. The third in￾stantiation exploits the structural relation￾ships among forget items. We embed every forget item with a dense sentence encoder and build a sparse, weighted semantic graph G = (V… view at source ↗
Figure 2
Figure 2. Average ∆ over BASE on the For￾get Set across training for all reward de￾signs (Binary = PURGE [37]). Higher val￾ues indicate greater unlearning. PageR￾ank Softmax achieves the strongest, most efficient gains throughout training [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Left: Exponential reward ϕexp(m; τ ) = e −m/τ vs. the forbidden-word count m for varying τ . Small τ approaches the binary reference (sharp drop at m = 1); large τ produces a lenient, slowly decaying curve. Right: Forget-quality metrics (FB, QA, AA; lower = better) at convergence across τ . The dotted line marks τ = 0.5, the value that jointly minimises all three forget metrics and maximises the utility metric. degr… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: PageRank weight distribution variants for the Stephen King forget set (61 terms). PageRank exhibit a power-law distribution where the top-5 nodes capture 34% of total weight mass. PageRank-Softmax compresses this gap while preserving semantic ordering; PageRank-Linear …
Figure 5
Figure 5. Figure 5: Evaluation of PageRank variants on the RWKU benchmark at 1,500 train￾ing steps, averaged over 20 forget targets. Columns report per-metric scores (Forget, Neighbor, MIA, Utility) alongside per-group ranks, win counts, overall rank, and a ro￾bustness score (inverse mean…
Figure 6
Figure 6. Figure 6: PageRank graph visualisations for the Stephen King forget set (61 terms) across four weight-transform variants. Node colour and size reflect per-node weight. PageRank distributes penalty mass more evenly than PageRank, while PageRank-Linear yields a near-uniform gradie…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

39 extracted references · 2 linked inside Pith

  1. [1]

    In: 2021 IEEE symposium on security and privacy (SP)

    Bourtoule, L., Chandrasekaran, V., Choquette-Choo, C.A., Jia, H., Travers, A., Zhang, B., Lie, D., Papernot, N.: Machine unlearning. In: 2021 IEEE symposium on security and privacy (SP). pp. 141–159. IEEE (2021)

    2021

  2. [2]

    In: 2015 IEEE symposium on security and privacy

    Cao, Y., Yang, J.: Towards making systems forget with machine unlearning. In: 2015 IEEE symposium on security and privacy. pp. 463–480. IEEE (2015)

    2015

  3. [3]

    In: The Eleventh International Conference on Learning Representations (2022)

    Chien, E., Pan, C., Milenkovic, O.: Efficient model updates for approximate un- learning of graph-structured data. In: The Eleventh International Conference on Learning Representations (2022)

    2022

  4. [4]

    In: The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2025)

    Dorna, V., Mekala, A.R., Zhao, W., McCallum, A., Kolter, J.Z., Lipton, Z.C., Maini, P.: Openunlearning: Accelerating llm unlearning via unified benchmark- ing of methods and metrics. In: The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2025)

    2025

  5. [5]

    arXiv preprint arXiv:2310.02238 (2023) Reward Design for Reinforcement Unlearning 13

    Eldan, R., Russinovich, M.: Who’s harry potter? approximate unlearning in llms. arXiv preprint arXiv:2310.02238 (2023) Reward Design for Reinforcement Unlearning 13

    arXiv 2023

  6. [6]

    Official Journal of the European Union (2016)

    European Union: Regulation (eu) 2016/679 of the european parliament and of the council. Official Journal of the European Union (2016)

    2016

  7. [7]

    Official Journal of the European Union (2023)

    European Union: Laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts. Official Journal of the European Union (2023)

    2023

  8. [8]

    CoRR (2025)

    Geng, J., Li, Q., Woisetschlaeger, H., Chen, Z., Wang, Y., Nakov, P., Jacobsen, H.A., Karray, F.: A comprehensive survey of machine unlearning techniques for large language models. CoRR (2025)

    2025

Show all 39 references
  1. [9]

    Advances in neural information processing systems32(2019)

    Ginart, A., Guan, M., Valiant, G., Zou, J.Y.: Making ai forget you: Data deletion in machine learning. Advances in neural information processing systems32(2019)

    2019

  2. [10]

    In: International Conference on Machine Learning

    Guo, C., Goldstein, T., Hannun, A., Van Der Maaten, L.: Certified data removal from machine learning models. In: International Conference on Machine Learning. pp. 3832–3842. PMLR (2020)

    2020

  3. [11]

    Information & Commu- nications Technology Law28(1), 65–98 (2019)

    Hoofnagle, C.J., Van Der Sloot, B., Borgesius, F.Z.: The european union general data protection regulation: what it is and what it means. Information & Commu- nications Technology Law28(1), 65–98 (2019)

    2019

  4. [12]

    In: The 61st Annual Meeting Of The Association For Computational Linguistics (2023)

    Jang, J., Yoon, D., Yang, S., Cha, S., Lee, M., Logeswaran, L., Seo, M.: Knowledge unlearning for mitigating privacy risks in language models. In: The 61st Annual Meeting Of The Association For Computational Linguistics (2023)

    2023

  5. [13]

    In: The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2024)

    Jin, Z., Cao, P., Wang, C., He, Z., Yuan, H., Li, J., Chen, Y., Liu, K., Zhao, J.: Rwku: Benchmarking real-world knowledge unlearning for large language mod- els. In: The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2024)

    2024

  6. [14]

    In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

    Kassem, A., Mahmoud, O., Saad, S.: Preserving privacy through dememorization: An unlearning technique for mitigating memorization risks in language models. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. pp. 4360–4379 (2023)

    2023

  7. [15]

    Advances in Neural Information Processing Systems36, 20750–20762 (2023)

    Kim, S., Yun, S., Lee, H., Gubri, M., Yoon, S., Oh, S.J.: Propile: Probing pri- vacy leakage in large language models. Advances in Neural Information Processing Systems36, 20750–20762 (2023)

    2023

  8. [16]

    In: Second Conference on Language Modeling (2024)

    Lambert, N., Morrison, J., Pyatkin, V., Huang, S., Ivison, H., Brahman, F., Mi- randa, L.J.V., Liu, A., Dziri, N., Lyu, X., et al.: Tulu 3: Pushing frontiers in open language model post-training. In: Second Conference on Language Modeling (2024)

    2024

  9. [17]

    In: Proceedings of the 41st International Conference on Machine Learning

    Li, N., Pan, A., Gopal, A., Yue, S., Berrios, D., Gatti, A., Li, J.D., Dombrowski, A.K., Goel, S., Mukobi, G., et al.: The wmdp benchmark: measuring and reducing malicious use with unlearning. In: Proceedings of the 41st International Conference on Machine Learning. pp. 28525–...

    2024

  10. [18]

    Nature Machine Intelligence pp

    Liu, S., Yao, Y., Jia, J., Casper, S., Baracaldo, N., Hase, P., Yao, Y., Liu, C.Y., Xu, X., Li, H., et al.: Rethinking machine unlearning for large language models. Nature Machine Intelligence pp. 1–14 (2025)

    2025

  11. [19]

    CoRR (2024)

    Liu, Z., Dou, G., Tan, Z., Tian, Y., Jiang, M.: Machine unlearning in generative ai: A survey. CoRR (2024)

    2024

  12. [20]

    Advances in neural information processing systems35, 27591–27609 (2022)

    Lu, X., Welleck, S., Hessel, J., Jiang, L., Qin, L., West, P., Ammanabrolu, P., Choi, Y.: Quark: Controllable text generation with reinforced unlearning. Advances in neural information processing systems35, 27591–27609 (2022)

    2022

  13. [21]

    In: First Conference on Language Modeling (2024)

    Maini, P., Feng, Z., Schwarzschild, A., Lipton, Z.C., Kolter, J.Z.: Tofu: A task of fictitious unlearning for llms. In: First Conference on Language Modeling (2024)

    2024

  14. [22]

    In: Proceedings of the 31st International Conference on Computational Linguistics

    Mekala, A., Dorna, V., Dubey, S., Lalwani, A., Koleczek, D., Rungta, M., Hasan, S.A., Lobo, E.: Alternate preference optimization for unlearning factual knowledge in large language models. In: Proceedings of the 31st International Conference on Computational Linguistics. pp. 3...

    2025

  15. [23]

    In: Algorithmic Learning Theory

    Neel, S., Roth, A., Sharifi-Malvajerdi, S.: Descent-to-delete: Gradient-based meth- ods for machine unlearning. In: Algorithmic Learning Theory. pp. 931–962. PMLR (2021)

    2021

  16. [24]

    ACM Transactions on Intelligent Systems and Technology16(5), 1–46 (2025)

    Nguyen,T.T.,Huynh,T.T.,Ren,Z.,Nguyen,P.L.,Liew,A.W.C.,Yin,H.,Nguyen, Q.V.H.: A survey of machine unlearning. ACM Transactions on Intelligent Systems and Technology16(5), 1–46 (2025)

    2025

  17. [25]

    Pardau, S.L.: The california consumer privacy act: Towards a european-style pri- vacy regime in the united states. J. Tech. L. & Pol’y23, 68 (2018)

    2018

  18. [26]

    Advances in Neural Information Processing Systems36, 53728–53741 (2023)

    Rafailov, R., Sharma, A., Mitchell, E., Manning, C.D., Ermon, S., Finn, C.: Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems36, 53728–53741 (2023)

    2023

  19. [27]

    In: The Thirteenth International Conference on Learning Representations (2025)

    Scholten, Y., Günnemann, S., Schwinn, L.: A probabilistic perspective on unlearn- ing and alignment for large language models. In: The Thirteenth International Conference on Learning Representations (2025)

    2025

  20. [28]

    Schulman,J.:Approximatingkldivergence.http://joschu.net/blog/kl-approx.html (Mar 2020), accessed: 2025-07-03

    2020

  21. [29]

    Advances in Neural Information Processing Systems34, 18075–18086 (2021)

    Sekhari, A., Acharya, J., Kamath, G., Suresh, A.T.: Remember what you want to forget: Algorithms for machine unlearning. Advances in Neural Information Processing Systems34, 18075–18086 (2021)

    2021

  22. [30]

    arXiv preprint arXiv:2402.03300 (2024)

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., et al.: Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 (2024)

    2024 arXiv

  23. [31]

    In: The Thirteenth International Conference on Learning Representations (2025)

    Shi, W., Lee, J., Huang, Y., Malladi, S., Zhao, J., Holtzman, A., Liu, D., Zettle- moyer, L., Smith, N.A., Zhang, C.: Muse: Machine unlearning six-way evaluation for language models. In: The Thirteenth International Conference on Learning Representations (2025)

    2025

  24. [32]

    In: Conference on Learning Theory

    Ullah, E., Mai, T., Rao, A., Rossi, R.A., Arora, R.: Machine unlearning via algo- rithmic stability. In: Conference on Learning Theory. pp. 4126–4142. PMLR (2021)

    2021

  25. [33]

    In: The Thirteenth International Conference on Learning Representations (2025)

    Wang, Y., Wei, J., Liu, C.Y., Pang, J., Liu, Q., Shah, A., Bao, Y., Liu, Y., Wei, W.: Llm unlearning via loss adjustment with only forget data. In: The Thirteenth International Conference on Learning Representations (2025)

    2025

  26. [34]

    In: The Thirty-First International Joint Conference on Artificial Intelligence (2022)

    Yan, H., Li, X., Guo, Z., Li, H., Li, F., Lin, X.: Arcane: An efficient architecture for exact machine unlearning. In: The Thirty-First International Joint Conference on Artificial Intelligence (2022)

    2022

  27. [35]

    In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers)

    Yao, J., Chien, E., Du, M., Niu, X., Wang, T., Cheng, Z., Yue, X.: Machine un- learning of pre-trained large language models. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers). pp. 8403–8419 (2024)

    2024

  28. [36]

    In: The Thirteenth International Conference on Learning Representations (2025)

    Yuan, X., Pang, T., Du, C., Chen, K., Zhang, W., Lin, M.: A closer look at machine unlearning for large language models. In: The Thirteenth International Conference on Learning Representations (2025)

    2025

  29. [37]

    arXiv preprint arXiv:2601.20568 (2026)

    Zaradoukas, E., Prenkaj, B., Kasneci, G.: Reinforcement unlearning via group rel- ative policy optimization. arXiv preprint arXiv:2601.20568 (2026)

    2026

  30. [38]

    In: The Thirty- ninth Annual Conference on Neural Information Processing Systems (2025)

    Zhang, C., Jin, Z., Yuan, H., Wei, J., Zhou, T., Liu, K., Zhao, J., Chen, Y.: Rule: Reinforcement unlearning achieves forget-retain pareto optimality. In: The Thirty- ninth Annual Conference on Neural Information Processing Systems (2025)

    2025

  31. [39]

    does the completion contain any forbidden item?

    Zhang, R., Lin, L., Bai, Y., Mei, S.: Negative preference optimization: From catas- trophic collapse to effective unlearning. In: First Conference on Language Modeling (2024) Reward Design for Reinforcement Unlearning 15 Acknowledgments. This work was supported by the IT Found...

    2024

Pith tools

Reviewed July 31, 2026 · model on record in the stance chip above.