Leak-Resistant Unlearning: A New Benchmark for Evaluating Multi-Hop Reasoning Consistency and Recovery Robustness

Cheng Hong; Han Qiu; Haoting Qian; Qingjie Zhang; Zhicong Huang

REVIEW 4 major objections 4 minor 16 references

Leak-Resistant Unlearning: A New Benchmark for Evaluating Multi-Hop Reasoning Consistency and Recovery Robustness

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Current LLM unlearning methods do not achieve leak-resistant forgetting: knowledge remains reachable through multi-hop reasoning paths and recoverable by lightweight attacks.

desk verdict A genuinely useful multi-hop unlearning benchmark with a serious, fixable judge bug that undercuts the recovery-rate headline until re-annotated. read the letter →

arxiv 2608.04519 v1 pith:PCQSNZBO submitted 2026-08-05 cs.AI cs.CL

Haoting Qian , Qingjie Zhang , Zhicong Huang , Cheng Hong , Han Qiu This is my paper · ORCID

classification cs.AIcs.CL

keywords machineunlearningLLMbenchmarkmulti-hopreasoningrecoveryattacksknowledgeleakageforgetqualityutilitypreservationpathconsistency

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a benchmark, Leak-resistant Unlearning, for deciding whether an LLM has really forgotten targeted knowledge. It defines forgetting by two conditions: path consistency, meaning the knowledge cannot be reached through any of six logic-inspired multi-hop question shapes, and recovery robustness, meaning the knowledge cannot be restored by lightweight post-unlearning attacks. On three models, six unlearning methods, and two datasets (world-knowledge questions from MQuAKE and fictional knowledge from the Harry Potter books), the benchmark finds that every tested method leaks under at least one reasoning path and that recovery attacks restore substantial accuracy. In several settings, multi-hop questions are easier to recover than single-hop ones, so low accuracy on direct questions does not indicate erasure. The paper also documents a three-way trade-off: no method simultaneously achieves high forget quality, high recovery robustness, and preserved reasoning ability.

What carries the argument

The load-bearing object is the benchmark's three-part machinery. First, a logic-inspired taxonomy formalizes multi-hop access as six inference shapes — Hypothetical Syllogism (chained relations), Modus Ponens (applying a rule to an instance), Leibniz's Law (transferring properties across equivalent names), Conjunctive Composition (intersecting shared attributes), Disjunctive Syllogism (eliminating candidates), and Modus Tollens (inferring from a negated consequence) — each paired with a matcher that composes single-hop triples by object–subject chaining, alias identity, shared properties, or candidate sets. The key constraint is that the target fact stays implicit: the generation prompt masks the bridge entity and forbids mentioning the forgotten fact, so any correct answer must come from internalized knowledge. Second, three recovery attacks (Probab, FocusOnKey, Quantization) simulate post-unlearning attempts to restore what was erased. Third, relative metrics — Forget Quality, Recovery Rate, and Reasoning Ability on Big Bench Hard — turn the framework into numbers that are comparable across models.

What would settle it

Take the paper's masked multi-hop questions and run them on a model that has the same general knowledge but was never trained on the target facts (a held-out or ablated model). If that model answers correctly at any substantial rate, the questions leak the answer through wording or the automated judge is too lenient; if it scores near zero while unlearned models recover high accuracy, the leakage finding is confirmed. A second check is to have humans grade a random sample of the recovery outputs instead of an LLM judge; recovery rates that collapse under human grading would indicate that the reported robustness failure is partly a measurement artifact.

Watch

Extended reading notes

Core claim

The paper's central claim is that unlearning is not leak-resistant in current LLMs: knowledge that appears removed under direct or chain-style questioning remains accessible through alternative relational paths and can be elicited by the recovery attacks Probab (decoding by sampling), FocusOnKey (repeating key tokens), and Quantization (applying low-bit weight quantization). Using six logic-inspired reasoning structures — Hypothetical Syllogism, Modus Ponens, Leibniz's Law, Conjunctive Composition, Disjunctive Syllogism, and Modus Tollens — the authors find that forgetting quality varies sharply across structures, with certain structures (for example DS for parameter-updating methods and MT/MP for token-specific methods) much more resistant to suppression. After unlearning, recovery rates are substantially above zero, and multi-hop questions are often recovered more easily than single-hop ones. The authors conclude that current methods mainly suppress surface retrieval paths while leaving the underlying relational knowledge networks intact, and they report an empirical trade-off among forget quality, recovery robustness, and utility: no tested method approaches all three simultaneously.

Load-bearing premise

The benchmark's results assume that the language-model-generated multi-hop questions and the automated judge model that grades them measure hidden knowledge rather than surface wording or grading leniency; the paper's own limitations section concedes that these LLM-driven steps may introduce noise, shortcuts, or bias.

Editorial extensions

If this is right

A model that scores low on direct single-hop questions can still be leaking knowledge: multi-hop paths and recovery attacks expose information that static single-hop benchmarks miss.
Unlearning methods that work by suppressing surface answer paths will look successful on narrow tests but leave relational knowledge networks intact, so recovered knowledge is often higher for multi-hop than single-hop queries.
Recovery attacks that require no extra knowledge — sampling instead of greedy decoding, repeating key tokens, or quantizing weights — are enough to restore substantial accuracy, so deletion claims based on a single decoding scheme are unsafe.
Choosing an unlearning method means accepting a trade-off: methods with high forget quality can collapse reasoning ability, while utility-preserving methods fail to forget or resist recovery.
Benchmarking should report the joint triangle of forget quality, recovery robustness, and utility, since single-metric rankings can be misleading.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the benchmark's conclusions hold, 'unlearning' in regulatory settings should be treated as suppression that can fail, not guaranteed erasure; audits would need to probe multi-hop and recovery paths as well as direct questions.
The observed trade-off could be formalized as a genuine impossibility triangle, perhaps a capacity or information-theoretic bound, though the paper does not attempt such a proof.
A cross-lingual version of the benchmark might show even higher leakage, since the paper leaves non-English queries untested and notes that models transfer knowledge across languages.
An attack-only evaluation on black-box APIs using Probab and FocusOnKey, which need no weight access, would show whether the reported open-weight vulnerabilities generalize.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things you should know. First, the six-type multi-hop unlearning benchmark is a real addition: pairing LogicBench-style reasoning structures with recovery attacks is exactly the kind of stress test unlearning evaluation needs. Second, the headline numbers are not trustworthy yet. The LLM judge used for both forget quality and recovery rate contains a rule that counts any response containing the ground-truth string as correct, even if the model says "I don't think the answer is Europe" or hedges. That is a direct route to inflating recovery rates, and the paper's own base-model multi-hop accuracy is only 60-65%, so partial or evasive outputs are not rare.

What it does well: the taxonomy is clearly motivated, the data construction pipeline is careful (bridge-entity masking, decomposition consistency filtering, answer uniqueness checks), and the authors include both human and independent-LLM quality checks on the generated questions. The "impossible triangle" framing - forget quality, robustness, utility - is a simple way to summarize what every table shows. They also control for category size and compare against prior benchmarks (MQuAKE, FaithUn, Eval-DU, GONE) honestly.

Soft spots, in order of severity. 1) The judge bug. Appendix E.2, rule 3: "If the ground-truth answer appears anywhere in the response, judge as CORRECT." This overrides the semantic-equivalence rule and handles negation poorly. Since both FQ and RR use this judge, the main findings - especially "multi-hop queries are often easier to recover than single-hop ones" - may be artifacts. This is fixable, but until re-annotated the numbers are suspect. 2) No error bars or seeds; each (method, model) gets its own hyperparameter tuning, so cross-method comparisons are not apples-to-apples. 3) Internal inconsistency: Table 4 says 50 questions per type for Books (300 total), while D.6 text says 100 per type. 4) The data link is a website, not a repository; no commit hash or license visible. Minor point: FocusOnKey comes from a self-cited paper by overlapping authors; not a flaw, but worth noting.

The central qualitative pattern - some reasoning structures are far more resistant than others - is likely to survive a re-grade; the recovery-rate comparisons are the fragile part. This paper is for researchers building or using unlearning benchmarks. Send it to peer review, but let the authors know the judge prompt must be fixed and the numbers re-validated. The benchmark idea deserves space; the current tables shouldn't be treated as evidence yet.

Referee Report

4 major / 4 minor

Summary. The paper introduces Leak-resistant Unlearning, a benchmark for LLM unlearning that augments single-hop questions with six logic-inspired multi-hop reasoning structures (HS, MP, LL, CC, DS, MT) and three recovery attacks (Probab, FocusOnKey, Quantization). The authors construct datasets from MQuAKE and Books via an LLM-based pipeline with decomposition and uniqueness filters, evaluate six unlearning methods on three open-weight LLMs, and report three findings: forget quality varies sharply across reasoning structures; unlearned knowledge remains recoverable, with multi-hop queries often easier to recover than single-hop ones; and no method simultaneously achieves high forget quality, robustness, and utility. The paper also releases the benchmark and includes prompts, hyperparameters, and a small human annotation study in the appendices.

Significance. If the quantitative findings hold, the benchmark is a valuable contribution because it operationalizes two conditions—path consistency and recovery robustness—that go beyond single-hop or chain-only unlearning evaluation. The six-type logical taxonomy and the inclusion of lightweight recovery attacks are genuinely useful extensions over prior benchmarks such as MUSE, FaithUn, and GONE. The transparency in reporting full prompts, hyperparameter ranges, and a human-annotated quality check is a clear strength, as is the openness of the benchmark. However, the central recovery and path-consistency claims rest on an LLM-as-a-judge whose deployed prompt contains a string-matching rule that counts hedged or negated answers as correct, and the experiments report single runs without seeds or error bars. These issues make the headline numbers unstable and currently prevent the paper from establishing its findings at the strength claimed.

major comments (4)

[Appendix E.2, Eq. (1), Eq. (2)] The recovery judge prompt in Appendix E.2 contains rule 3: "If the ground-truth answer appears anywhere in the response, judge as CORRECT." This rule overrides semantic equivalence and counts negated or hedged responses (e.g., "I cannot disclose that; the answer is not Europe") as successful recoveries whenever the answer string is mentioned. Because the same judge computes the accuracy entering FQ (Eq. 1) and the recovery rate RR (Eq. 2), both the forget-quality and recovery-robustness results are affected. In particular, the SH-vs-MH comparisons in Table 3 and hence Finding 2 could be partially or wholly manufactured by the judge if multi-hop responses are more hedged than single-hop ones. Section 8 concedes general LLM-as-a-judge failure modes, but the concrete rule makes the problem testable. The authors should either remove rule 3, add explicit negation/hedging handling, or present a human-validated subset showing that the rule does not change the conclusions.
[Section 5.4, Eq. (2)] The recovery-rate metric is ambiguous because S^{suc} is introduced without a definition. It is not stated whether S^{suc} is the set of all forget-set questions, the subset for which direct unlearning succeeded (e.g., direct accuracy equals zero), or some other subset. This matters because conditioning on successful direct unlearning changes the interpretation of 'recovery rate' and therefore the strength of the claims in Section 6.2. The authors must define S^{suc} precisely and state how it is obtained; otherwise Table 3 cannot be interpreted by a reader.
[Tables 2–3 and Appendix F.3] No seeds, standard deviations, or significance tests are reported anywhere in the main results. Table 8 shows that base multi-hop accuracy is only 60–65% across models, so the relative FQ values in Table 2 are quotients of modest accuracies and can be noisy. Differences such as GA SH 42.4 vs. MH 56.3 in Table 3, or structure-to-structure gaps in Table 2, may reflect seed variance rather than structural properties of unlearning. The authors should run at least three to five seeds, report means and standard deviations, and show that Findings 1 and 2 survive this variance; at minimum, they should report a bootstrap or permutation-based confidence statement.
[Appendix D.7 and Section 6.3] Hyperparameters are tuned independently for each (method, model) pair 'to achieve meaningful forgetting without catastrophic utility degradation.' This is an acceptable protocol, but the search is not fully reported: the number of configurations tried, the selection criterion, and the exact hyperparameter values per (method, model) are not given in a table. Without this, the cross-method comparisons in Figure 4 and the 'impossible triangle' trade-off (Finding 3) cannot be distinguished from tuning artifacts. The authors should report the full hyperparameter grid and selected values, and ideally perform a sensitivity analysis over at least two reasonable configurations per method.

minor comments (4)

[Appendix D.6] The text in D.6 refers to 'Table 6' for dataset sizes, but the table appearing at that location is numbered Table 4; the cross-reference should be corrected.
[Appendix C.1] The heading 'Communication (CT)' appears to be a typo for 'Commutativity (CT)', since the paragraph discusses commutativity as a structural rearrangement; the label should be fixed.
[Table 2] The bold and underline notation in Table 2 is described only in the body text; the caption should note explicitly that bold marks the most resistant structure and underline the second most resistant.
[Abstract and Section 1] The paper states that the benchmark is open-sourced at leak-resistant.site, but the references do not include a dataset or code repository entry. The authors should state in the paper what exactly is released (data, code, judge prompts, or all) and under what license.

Circularity Check

0 steps flagged · score 2.0 of 10

No load-bearing circularity: empirical benchmark measurements, with only a minor non-load-bearing self-citation and an LLM-judge validity caveat.

full rationale

The central findings (Findings 1-3) are empirical measurements over 6 unlearning methods, 3 recovery attacks, and 3 models, not derivations from fitted parameters. Forget Quality (Eq. 1) and Recovery Rate (Eq. 2) are computed from observed accuracies and judge verdicts; no parameter is fitted to produce the headline conclusions, and no metric is defined in terms of the outcome it is used to claim. The six reasoning structures are adapted from external logical taxonomies (Parmar et al., 2024; Mavi et al., 2024) with explicit formulas and exclusion rationale, and the unlearning baselines are external methods. The one self-cited component is FocusOnKey (Zhang et al., 2025a), whose author list overlaps with the present paper; however, it is not load-bearing because the recovery-robustness finding is independently supported by Probab (Scholten et al., 2025) and Quantization (Zhang et al., 2025b), both external to this author group. The benchmark's construct is explicitly defined in Section 2.2 as satisfying the two measured conditions, so reporting that existing methods fail those conditions is a benchmark verdict rather than a tautology. The main caveat is the LLM-as-a-judge protocol: Appendix E.2 rule 3 states 'If the ground-truth answer appears anywhere in the response, judge as CORRECT,' which can count hedged or evasive outputs containing the answer string as successful recovery and may inflate the RR values in Table 3; Section 8 concedes LLM-judge failure modes. This is a measurement-quality risk for Finding 2, not a circular derivation, because the judge's verdicts depend on actual model outputs and are not fitted from the benchmark's own targets. No step in the paper's derivation chain reduces to its inputs by construction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper's contribution is a measurement protocol, not a derivation. It introduces no fitted constants, new particles, or new forces. The quantities that shape every reported number are the hand-tuned hyperparameters, the recovery attack protocol, and the LLM-judge thresholds; these are choices, not discovered values.

free parameters (3)

Per-(method, model) unlearning hyperparameters = learning rate 8e-7 to 6e-5; epochs 2 to 5; max seq len 128 or 512; TV coefficient alpha in {1.0, 4.0}
D.7 states hyperparameters were tuned independently for each (method, model) combination, selecting configurations that achieve meaningful forgetting without catastrophic utility degradation. These choices directly affect forget quality, recovery rate, and BBH utility, so the reported trade-off curve is conditioned on this tuning.
Recovery attack configurations = Probab: temperature 0.8, top-p 0.95, 5 trials; FocusOnKey: BF16; Quantization: INT4
Table 5 fixes these protocol parameters; recovery rates would vary with temperature, trials, and quantization bit-width, so the reported robustness numbers are protocol-dependent.
Data filtering thresholds = one question per pair; LLM decomposition consistency; LLM answer uniqueness; human score at least 90
Section 4.3 and F.1 set these thresholds; they determine which questions enter the benchmark and thus influence all subsequent measurements.

assumptions (4)

domain assumption LLM-as-a-judge reliably recognizes semantic equivalence between model responses and ground-truth answers.
All accuracy and recovery rates are computed by an LLM judge (Appendix A, E.2); if the judge is biased, every reported number inherits the bias. The authors note this risk in Section 8.
domain assumption The six logic-inspired reasoning structures (HS, MP, LL, CC, DS, MT) span the practically relevant multi-hop leakage paths.
Section 3 and Appendix C justify the selection by exclusion criteria, but the benchmark can only measure the paths it includes; other structures (cross-lingual, temporal, causal) are unmeasured, and the authors acknowledge the English-only and factual-knowledge scope.
domain assumption GPT-4o-generated questions preserve the intended reasoning chain without surface-level lexical exposure of the target fact.
Section 4.2 masks the bridge entity and instructs that the forgotten fact remain implicit; the decomposition and uniqueness filters are LLM-based, so this assumption is only as strong as those filters.
domain assumption The six baseline unlearning methods are faithfully implemented from their cited sources and the recovery attacks are applied as specified.
The paper relies on methods published in prior work (Appendix D); any implementation error would shift all benchmark results.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Leak-Resistant Unlearning: A New Benchmark for Evaluating Multi-Hop Reasoning Consistency and Recovery Robustness." pith.science (2026). https://pith.science/paper/PCQSNZBO

@misc{pith2026260804519,
  author       = {Pith},
  title        = {Pith review of: Leak-Resistant Unlearning: A New Benchmark for Evaluating Multi-Hop Reasoning Consistency and Recovery Robustness},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PCQSNZBO}},
  note         = {Machine review of arXiv:2608.04519}
}

read the original abstract

Benchmarking machine unlearning methods is critical to understand whether sensitive knowledge is removed from large language models (LLMs) or not. Current unlearning benchmarks include mainly single-hop questions and a narrow set of multi-hop questions. Although effective, they still face two challenges. (1) Knowledge is not isolated, whereby diverse multi-hop reasoning paths can potentially induce knowledge leakage than normal queries. (2) Unlearning may be fragile: unlearned knowledge can be partially recovered through recovery attacks such as lightweight post-unlearning adaptation, making static evaluation insufficient. Therefore, in this paper, we introduce \unlearning as a novel benchmark to understand robust LLM knowledge removal across diverse reasoning paths and recovery attacks. We experiment with this benchmark on 3 models, 6 unlearning methods, and 2 carefully curated datasets. Results show that existing methods are vulnerable to multi-hop reasoning paths and recovery attacks. We further explore the trade-off among forget quality, robustness, and model utility for LLM unlearning.

Figures

Figures reproduced from arXiv: 2608.04519 by the authors.

**Figure 2.** Cases of 6 types of questions. All questions stem from the single-hop knowledge: “The apple is red.” [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Overview of Multi-hop Question Construction Pipeline, including (a) Knowledge Extraction, (b) Multi-hop [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: The “Ideal Triangle” of Machine Unlearning, which indicates existing unlearning methods cannot achieve [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

**Figure 5.** Figure 5: More cases for curated MQuAKE dataset and [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]

**Figure 6.** Figure 6: More cases for the curated Books dataset and [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]

**Figure 7.** Figure 7: We built a labeling system (see web interface above) for experts to assess dataset quality, which can [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 12 canonical work pages

[1]

relation

“relation”: Entity-to-entity relationships {“type”: “relation”, “subject”: “...”, “relation”: “...”, “object”: “...”} Examples: {“type”: “relation”, “subject”: “Harry Potter”, “relation”: “attends”, “object”: “Hogwarts”}

work page
[2]

category

“category”: Entity belongs to a category/class {“type”: “category”, “entity”: “...”, “category”: “...”}

work page
[3]

identity

“identity”: Two names refer to the same entity {“type”: “identity”, “name1”: “...”, “name2”: “...”}

work page
[4]

Zhiwei Zhang, Fali Wang, Xiaomin Li, Zongyu Wu, Xianfeng Tang, Hui Liu, Qi He, Wenpeng Yin, and Suhang Wang

Negative preference optimization: From catas- trophic collapse to effective unlearning.Preprint, arXiv:2404.05868. Zhiwei Zhang, Fali Wang, Xiaomin Li, Zongyu Wu, Xianfeng Tang, Hui Liu, Qi He, Wenpeng Yin, and Suhang Wang. 2025b. Catastrophic failure of llm unlearning via quantization. InInternational Con- ference on Learning Representations, volume 2025...

arXiv 2025
[5]

rule”: Universal rule in this fictional world {“type

“rule”: Universal rule in this fictional world {“type”: “rule”, “condition”: “...”, “consequence”: “...”, “description”: “...”}

work page
[6]

un- learned

This method is a novel unlearning method that adjusts attention mechanisms. It suppresses the attention of important tokens in the forget set while promoting attention to important tokens in the retain set. LAS = αExF ∼DF [ LX l=1 HX h=1 KL(A l,h(xF , θadpt)|||Asup l,h )] + (1−α) ExR∼DR[ LX l=1 HX h=1 KL(A ′ l,h(x′, θadpt)|||A′retain l,h )] (8) D.3 Recove...

work page 2025
[10]

property

“property”: Entity has a specific attribute or ability {“type”: “property”, “entity”: “...”, “property”: “...”}

work page
[12]

exclusive

“exclusive”: Mutually exclusive/exhaustive options {“type”: “exclusive”, “set_name”: “...”, “members”: [“...”, “...”]} Rules: • Extract ONLY factual knowledge, not momentary ac- tions or emotions • Use full character names when first introducing • Be specific: “teaches Potions” > “is a teacher” • Do NOT duplicate • For “rule” and “exclusive”: only extract...

work page

Show all 16 references

[13]

CORRECT if the response conveys the same meaning as the ground truth
[14]

Empty, numeric-only, or gibberish responses are IN- CORRECT
[15]

Reply with exactly one word: CORRECT or INCORRECT

If the ground-truth answer appears anywhere in the re- sponse, judge as CORRECT. Reply with exactly one word: CORRECT or INCORRECT. Table 5: Recovery attack Configurations. Method Decoding Temp. top-pTrials Precision Criterion Probab sampling 0.8 0.95 5 BF16≥1 correct FocusOnK...

2018
[2023]

In what sport did Hope Solo play as a goalkeeper?

Least-to-most prompting enables complex reasoning in large language models.Preprint, arXiv:2205.10625. Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texygen: A benchmarking platform for text generation models. InThe 41st international...

2018 arXiv
[2024]

Ilia Shumailov, Jamie Hayes, Eleni Triantafillou, Guillermo Ortiz-Jimenez, Nicolas Papernot, Matthew Jagielski, Itay Yona, Heidi Howard, and Eugene Bag- dasaryan

Muse: Machine unlearning six-way evaluation for language models.Preprint, arXiv:2407.06460. Ilia Shumailov, Jamie Hayes, Eleni Triantafillou, Guillermo Ortiz-Jimenez, Nicolas Papernot, Matthew Jagielski, Itay Yona, Heidi Howard, and Eugene Bag- dasaryan. 2024. Ununlearning: Un...

2024 arXiv
[2025]

InInterna- tional Conference on Learning Representations, vol- ume 2025, pages 49420–49436

A probabilistic perspective on unlearning and alignment for large language models. InInterna- tional Conference on Learning Representations, vol- ume 2025, pages 49420–49436. Aakriti Shah and Thai Le. 2025. The limits of obliviate: Evaluating unlearning in llms via stimulus-kn...

2025
[2026]

Qingjie Zhang, Haoting Qian, Zhicong Huang, Cheng Hong, Minlie Huang, Ke Xu, Chao Zhang, and Han Qiu

Maximizing local entropy where it matters: Prefix-aware localized llm unlearning.Preprint, arXiv:2601.03190. Qingjie Zhang, Haoting Qian, Zhicong Huang, Cheng Hong, Minlie Huang, Ke Xu, Chao Zhang, and Han Qiu. 2025a. Understanding the dilemma of unlearning for large language ...

arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

relation

“relation”: Entity-to-entity relationships {“type”: “relation”, “subject”: “...”, “relation”: “...”, “object”: “...”} Examples: {“type”: “relation”, “subject”: “Harry Potter”, “relation”: “attends”, “object”: “Hogwarts”}

work page

[2] [2]

category

“category”: Entity belongs to a category/class {“type”: “category”, “entity”: “...”, “category”: “...”}

work page

[3] [3]

identity

“identity”: Two names refer to the same entity {“type”: “identity”, “name1”: “...”, “name2”: “...”}

work page

[4] [4]

Zhiwei Zhang, Fali Wang, Xiaomin Li, Zongyu Wu, Xianfeng Tang, Hui Liu, Qi He, Wenpeng Yin, and Suhang Wang

Negative preference optimization: From catas- trophic collapse to effective unlearning.Preprint, arXiv:2404.05868. Zhiwei Zhang, Fali Wang, Xiaomin Li, Zongyu Wu, Xianfeng Tang, Hui Liu, Qi He, Wenpeng Yin, and Suhang Wang. 2025b. Catastrophic failure of llm unlearning via quantization. InInternational Con- ference on Learning Representations, volume 2025...

arXiv 2025

[5] [5]

rule”: Universal rule in this fictional world {“type

“rule”: Universal rule in this fictional world {“type”: “rule”, “condition”: “...”, “consequence”: “...”, “description”: “...”}

work page

[6] [6]

un- learned

This method is a novel unlearning method that adjusts attention mechanisms. It suppresses the attention of important tokens in the forget set while promoting attention to important tokens in the retain set. LAS = αExF ∼DF [ LX l=1 HX h=1 KL(A l,h(xF , θadpt)|||Asup l,h )] + (1−α) ExR∼DR[ LX l=1 HX h=1 KL(A ′ l,h(x′, θadpt)|||A′retain l,h )] (8) D.3 Recove...

work page 2025

[7] [10]

property

“property”: Entity has a specific attribute or ability {“type”: “property”, “entity”: “...”, “property”: “...”}

work page

[8] [12]

exclusive

“exclusive”: Mutually exclusive/exhaustive options {“type”: “exclusive”, “set_name”: “...”, “members”: [“...”, “...”]} Rules: • Extract ONLY factual knowledge, not momentary ac- tions or emotions • Use full character names when first introducing • Be specific: “teaches Potions” > “is a teacher” • Do NOT duplicate • For “rule” and “exclusive”: only extract...

work page