Unlearning That Lasts: Utility-Preserving, Robust, and Almost Irreversible Forgetting in LLMs

Francesco Croce; Matthias Hein; Maximilian M\"uller; Naman Deep Singh

REVIEW 4 major objections 4 minor 3 cited by

Unlearning That Lasts: Utility-Preserving, Robust, and Almost Irreversible Forgetting in LLMs

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A Jensen-Shannon objective drives LLM unlearning to 0% worst-case recall of targeted facts while preserving utility, and the paper shows standard evaluations overstate how well previous methods forget.

desk verdict A useful empirical package—JSD loss plus a stricter evaluation protocol—but the 'almost irreversible' headline overstates what their own relearning table shows. read the letter →

arxiv 2509.02820 v1 pith:WYUMOYOE submitted 2025-09-02 cs.LG

Naman Deep Singh , Maximilian Müller , Francesco Croce , Matthias Hein This is my paper · ORCID

classification cs.LG

keywords machineunlearningJensen-Shannondivergencelargelanguagemodelsforget-utilitytrade-offbenignrelearningLLM-as-judgeevaluationworst-caseLKFdataset

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models absorb facts during pre-training, and 'unlearning' aims to delete specific facts from the parameters instead of retraining from scratch. This paper argues that existing unlearning methods look more successful than they are, because evaluating with ROUGE scores and a single question format misses paraphrases and in-context prompts that still elicit the forgotten answer. To fix both sides, the authors propose JensUn, which uses the Jensen-Shannon Divergence for both the forget and retain objectives — a bounded, symmetric loss that stays stable through long unlearning — and a harder evaluation protocol: an LLM judge plus worst-case accuracy over many paraphrases. On their new LKF dataset of lesser-known real-world facts, JensUn reaches 0% worst-case forget accuracy while keeping win rate and MMLU near the base model, and after 2000 unlearning steps followed by benign relearning it recovers only 8% of forgotten answers, versus 26% for NPO and 58% for NPO+SAM. If the claim holds, unlearning can be both effective and safe enough to deploy in privacy and safety settings.

What carries the argument

The central object is the Jensen-Shannon Divergence (JSD), a symmetric distance between probability distributions bounded between 0 and log 2 per token. JensUn uses it in both terms of the unlearning objective: the forget loss minimizes JSD from the model's output distribution to a one-hot refusal target such as 'No idea', and the retain loss minimizes JSD from the unlearning model to the frozen base model. The paper proves the JSD gradient norm is at most (log 2)/2 times the KL gradient norm, so updates stay well-behaved over long fine-tuning, while the retain loss starts at zero and only grows as the model drifts — a natural balance between forgetting and utility. The second piece of machi

What would settle it

Take a JensUn-unlearned model with 0% JW on LKF and probe it with prompt transformations outside the paper's set — e.g., translations of the forget questions into several unrelated languages, or fine-tuning on a curriculum built from facts topically adjacent to the forget set (the paper's own appendix D.5 shows retain-set relearning already lifts JW from 1% to 18%). If any translation or curriculum pushes worst-case recall well above zero, the permanence claim would be settled as suppression, not removal.

Watch

Extended reading notes

Core claim

Two claims carry the paper. First, standard unlearning evaluation overstates success: ROUGE does not track factual correctness, and single-format questions hide residual knowledge. The worst-case metric JW — LLM-judged accuracy over 15 paraphrases plus in-context retain samples, forgotten only if all fail — raises reported forget accuracy by up to 43%. Second, with both losses formulated as Jensen-Shannon Divergence, the bounded objective drives the forget set to a refusal string while staying near the base model: on the new LKF dataset JensUn reaches 0% JW with WR 0.47, and after 2000 unlearning steps plus benign relearning recovers only 8% of forgotten answers versus 26% for NPO and 58% fo

Load-bearing premise

That a fact is truly forgotten because the model fails a finite test suite — up to 15 paraphrases and in-context retain samples — with no argument that these variations cover the prompts or fine-tuning curricula a determined questioner could use; an untested phrasing or relearning recipe that surfaces the answer would break the 'almost irreversible' conclusion.

Editorial extensions

If this is right

Benchmark results for existing unlearning methods (TOFU, RWKU, MUSE, WHP) should be re-read under worst-case evaluation; the paper shows forget accuracy jumps by up to 43%, meaning several published comparisons overstated forgetting.
Unlearning can be run much longer than the standard 200 steps: JensUn stays stable and useful past 2000 steps, and longer unlearning is what makes benign relearning ineffective.
A provider can choose the behavior on forgotten facts — explicit refusal ('No idea'), token gibberish ('#'), or silence — by picking the JSD target string, without sacrificing forget quality or utility.
The LKF dataset and the LLM-judge-with-worst-case protocol give a realistic, low-cost testbed for unlearning evaluation, including a more reliable ranking of methods than ROUGE-based scores.
Combining the better loss with the harder evaluation shrinks the gap between reported unlearning success and actual forgetting, making unlearning a more credible tool for privacy and safety compliance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial extension: if bounded gradients are the true driver of JensUn's stability, other symmetric bounded divergences (e.g., the t-Jensen-Shannon family from the GAN literature) should behave similarly; the paper does not test this.
Editorial extension: the permanence claim is established against 'benign' relearning on data disjoint from the forget set; the paper's own retain-set relearning experiment (appendix D.5) pushes JensUn's JW from 1% to 18%, so adversarial curricula built from forget-adjacent or rephrased data remain an open threat model.
Editorial extension: the worst-case protocol could plausibly be extended to cross-lingual prompts and membership-inference probing; the paper tested fill-in-the-blank and hint variants (no additional signal) but not languages.
Editorial extension: LKF's recipe — real lesser-known facts, non-dichotomous questions, a semantically adjacent retain set — transfers naturally to privacy-sensitive domains such as medical or legal records, where 'obscure but real' is exactly the relevant regime.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

I read this one carefully. The core contributions are real: a JSD-based forget/retain loss that behaves better than the usual log-likelihood or NPO variants, a worst-case evaluation protocol (LLM judge over paraphrases and in-context retain) that exposes over-optimistic scores in existing benchmarks, and a new LKF dataset of non-dichotomous, lesser-known facts. The experiments are extensive, the ablations are honest (they sweep lambda_R and LR and report trade-off curves), and they include a human study showing the judge agrees with human ratings. The gradient analysis for JSD vs KL is a nice, checkable detail. I would absolutely use their evaluation ideas and the LKF dataset in my own work.

The soft spots are mostly in the framing. The title and abstract promise "almost irreversible forgetting," but Table 3 shows JensUn recovering 8% of forget accuracy after 2000 unlearning steps under benign relearning. That is better than the baselines but it is not irreversibility. The paper actually concedes this in the text, but the headline claim outruns the evidence. Second, the worst-case metric J_W is defined over a finite set of English paraphrases generated by three small models. That is a useful lower bound on retrieval, but it cannot support a claim that the fact is erased from the weights. A different phrasing, a non-English query, or a different task format could still elicit the answer. The authors do test fill-in-the-blank and hints in an appendix and find no improvement to J_W, which helps, but the set remains finite and closed.

I also note the reliance on a closed API judge (Gemini-2.5-Flash) for all headline numbers. The judge agrees with humans on a 360-sample study, so it is not arbitrary, but the reproducibility of exact numbers is limited. Low-severity concern: the forget loss explicitly trains refusal, so the 0% forget accuracy is partly by construction. That is legitimate unlearning behavior, not metric-gaming, but it should be stated more clearly.

Overall: the method and evaluation framework are solid, the fairness of the comparison is reasonable, and the empirical ranking of JensUn against baselines is probably robust. The permanence claim needs to be toned down or re-tested on out-of-distribution queries. This paper deserves a serious referee; I would send it out and ask for the permanence language to be fixed.

Take it for your reading group.

Referee Report

4 major / 4 minor

Summary. The paper proposes JensUn, an LLM unlearning method that replaces the common log-likelihood/DPO-style forget and retain losses with Jensen-Shannon Divergence terms (Eqs. 4-6), arguing that boundedness and well-behaved gradients improve the forget-utility trade-off and stability during longer unlearning. It also introduces LKF, a new 100-forget/400-retain QA dataset of lesser-known historical facts, and proposes an evaluation protocol based on an LLM semantic judge plus worst-case accuracy over 15 paraphrases and in-context retain samples (JW). Experiments on LKF (Llama-3.2-3B, Phi-3) and on RWKU compare JensUn with GradAscent, GradDiff, NPO, SimNPO, RMU, DPO, RT, ICU, and NPO+SAM, reporting that JensUn achieves 0% JW on LKF while preserving utility, and lower forget accuracy on RWKU than competitors. The paper further studies benign relearning and reports that JensUn retains higher WR and lower relearned JW than NPO/NPO+SAM at longer unlearning steps.

Significance. If the results hold, the paper makes three useful contributions: a simple, precisely specified unlearning loss that is competitive with or better than established baselines; a realistic QA-style unlearning dataset (LKF); and a stronger evaluation protocol that replaces ROUGE with a human-aligned LLM judge and uses worst-case paraphrases. The authors publicly release code and dataset, perform a human study for the judge, and provide a gradient analysis of JSD versus KL. These are concrete strengths. However, the central rhetorical claim of 'almost irreversible forgetting' is not supported by the paper's own relearning numbers, and the evaluation, while more robust than standard benchmarks, remains a finite set of English paraphrases. The method and the relative rankings may survive such criticism, but the permanence claim needs substantial reframing or additional out-of-distribution evidence.

major comments (4)

[§5.3, Table 3; Appendix D.5, Table 13; title/abstract] The 'almost irreversible' framing is contradicted by the paper's own data. In Table 3, JensUn's relearned JW is 27% at 200 unlearning steps, 24% at 400, 19% at 600, 14% at 1000, and 8% at 2000; in Appendix D.5, retain-set relearning of the 2000-step model gives JW=18%, a value the text describes as 'only slight'. These are non-negligible recovery rates under a single benign relearning curriculum. The claims in the abstract and §5.3 that the information has been 'truly removed' and that unlearning is 'almost irreversible' are therefore unsupported. The authors should either define an explicit irreversibility criterion with an attack/relearning budget and demonstrate it, or reframe the claim as 'resistant to the tested benign relearning protocols'.
[§4.2, Eq. (4), §3.1] The worst-case metric JW uses 15 English paraphrases generated by three small LLMs plus in-context retain samples. This is a finite query family, and Eq. (4) explicitly trains the model to emit a refusal string ('No idea') for the forget prompts and their training paraphrases. Consequently, 0% JW is consistent with a model that has learned a refusal behavior for those specific phrasings rather than having removed the underlying factual association. The paper's own Table 6 shows that adding fill-in-the-blank and hint variants did not change results, but this only extends the query family by one more finite set. To support 'knowledge has been truly removed', the authors should add genuine out-of-distribution probes — e.g., non-English questions, cloze completions, multiple-choice framing, task reframings, and paraphrases from held-out generators not used in evaluation — and report JW on th
[§4.1, Appendix A.4, all main tables] All headline forget/retain/utility numbers depend on calls to Gemini-2.5-Flash through a closed API. The human validation is limited to 360 LKF samples, and approximately 1.5% of RWKU samples are removed due to judge/API failures. As reported, there is no analysis of judge variance, prompt sensitivity, or agreement with an independent judge. Since Tables 1–3 and the RWKU results are the basis for every comparative claim, this is a reproducibility risk. The authors should release judge outputs, report repeated-call variability, and provide at least one additional judge or a manual audit for a subsample of the main tables.
[Appendix B.3, Table 2] For RWKU, DPO and NPO use the benchmark's predefined retain templates, while GradDiff, SimNPO, and JensUn use a custom train-time retain set constructed from disjoint targets, because the RWKU evaluation retain set has a format mismatch. Therefore, Table 2 compares methods trained with different retain distributions. This is disclosed, but it biases the retain and utility comparisons in the main table. The authors should either construct one common train-time retain set for all methods, or explicitly state in the main text that RWKU retain columns are not directly comparable across these method groups.

minor comments (4)

[Figure 17 caption] 'Problems with ROGUE-L based metrics' contains a typo: ROGUE should be ROUGE.
[§4.2 and Appendix E.1] The notation JP is described as 'average forget quality evaluated with paraphrases' in one place, but Eq. (7) defines a sample-wise worst-case (max) accuracy. Please make the definition of JP, JICR, and JW consistent throughout.
[Table 9] The table header says 'Forget-utility trade-off ... on the LKF dataset', but the 'selected' rows are not visually marked in the provided text. Please ensure the final selected λR values are clearly highlighted in the camera-ready version.
[Appendix B.4] The sentence 'we use 1k instruction based generated queries from AlpacaEval' is awkward; consider '1,000 instruction-following queries sampled from AlpacaEval'.

Circularity Check

0 steps flagged · score 1.0 of 10

No load-bearing circularity: JensUn's loss and evaluation are aligned by design, but the evaluation paraphrases are disjoint and external benchmarks anchor the utility claims.

full rationale

The paper's derivation chain is Eq. (6) -> unlearned model -> JW / utility metrics. The forget loss Eq. (4) drives the model to output 'No idea', and the LLM-Judge (Fig. 19) marks any output lacking GT information as incorrect, so low forget-accuracy is the intended behavioral effect of the objective. This is not a fitted parameter renamed as a prediction: LKF evaluation paraphrases are generated by Mistral-7B, Qwen2.5-3B, and Phi-3.5 and are disjoint from the Phi-4-mini training paraphrases (App. A.3, B.2), and RWKU provides an external paragraph-based forget set. Utility is benchmarked against MMLU and AlpacaEval rather than being derived from the loss. The only self-citations (Croce et al. 2024; Zhao et al. 2024) are motivational/template-related and non-load-bearing; the bounded-gradient advantage is proved in App. E.4. The title's 'almost irreversible' is an extrapolation: Sec. 5.3 says 'suggests' and Table 3 still shows 8% JW recovery for JensUn after 2000 relearning steps, and the finite English paraphrase set cannot prove weight-level erasure. That is a validity limitation, not a circular reduction.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

No new physical entities, particles, forces, or conserved quantities are introduced. The LKF dataset is a new benchmark, but it is not an invented entity in the sense of this ledger. All free parameters are standard training hyperparameters and design choices, tuned on the LKF benchmark.

free parameters (5)

λF (forget loss weight) = 0.5
Tuned via grid search on LKF (Table 4); fixed for all JensUn runs.
λR (retain loss weight) = 0.5
Tuned on LKF; sweeping λR changes the forget-utility trade-off (Table 9).
Learning rate = 8e-6
Selected after grid search on LKF (Table 10) and RWKU (Table 15).
Target refusal string tokens = 'No idea'
Design choice; other refusal strings work equivalently (Figure 14).
Unlearning duration = 10 epochs (LKF), 5 epochs (RWKU)
Chosen as best trade-off; longer training improves relearning robustness (Table 3).

assumptions (4)

standard math JSD is bounded and its gradient is bounded by a factor of log2/2 relative to KL divergence.
Proved in Appendix E.4; used to argue that JSD provides more stable unlearning dynamics.
domain assumption Gemini-2.5-Flash LLM judge binary judgments agree with human judgments.
Validated on 360 samples with 98% agreement (Fig. 9), but the API is closed and version-dependent.
domain assumption The LKF forget set contains facts the base model actually knows.
Verified empirically: Llama-3.2-3B retrieves 76% of the forget facts, so unlearning is meaningful.
domain assumption Benign relearning on 200 QA pairs for 600 steps is a strong enough attack to test irreversibility.
Used in Table 3 and Appendix B.5; the paper does not test other attack configurations or stronger relearning curricula.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Unlearning That Lasts: Utility-Preserving, Robust, and Almost Irreversible Forgetting in LLMs." pith.science (2026). https://pith.science/paper/WYUMOYOE

@misc{pith2026250902820,
  author       = {Pith},
  title        = {Pith review of: Unlearning That Lasts: Utility-Preserving, Robust, and Almost Irreversible Forgetting in LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WYUMOYOE}},
  note         = {Machine review of arXiv:2509.02820}
}

read the original abstract

Unlearning in large language models (LLMs) involves precisely removing specific information from a pre-trained model. This is crucial to ensure safety of LLMs by deleting private data or harmful knowledge acquired during pre-training. However, existing unlearning methods often fall short when subjected to thorough evaluation. To overcome this, we introduce JensUn, where we leverage the Jensen-Shannon Divergence as the training objective for both forget and retain sets for more stable and effective unlearning dynamics compared to commonly used loss functions. In extensive experiments, JensUn achieves better forget-utility trade-off than competing methods, and even demonstrates strong resilience to benign relearning. Additionally, for a precise unlearning evaluation, we introduce LKF, a curated dataset of lesser-known facts that provides a realistic unlearning scenario. Finally, to comprehensively test unlearning methods, we propose (i) employing an LLM as semantic judge instead of the standard ROUGE score, and (ii) using worst-case unlearning evaluation over various paraphrases and input formats. Our improved evaluation framework reveals that many existing methods are less effective than previously thought.

Figures

Figures reproduced from arXiv: 2509.02820 by the authors.

**Figure 1.** Our JensUn yields the best trade-off between unlearning quality (forget set accuracy) and utility of the LLM. (left) Our unlearning method JensUn achieves on our LKF dataset an optimal worstcase forget set accuracy of 0% while maintaining high response quality (AlpacaEval), the most similar to the original Llama-3.2-3B-Instruct pre-trained model. (right) Our novel worst-case evaluation using 15 paraphrases of the q… view at source ↗

**Figure 2.** Problems with ROUGE-L and LLM-Judge as a replacement. (left) We illustrate how ROUGE-L scores can inaccurately signal unlearning success (✔) or failure (✘) based on the LLM output and the ground truth answer. (right) ROC curve for ROUGE-L scores against human judgments across 400 queries: ROUGEL shows poor alignment with human perception, whereas our LLM-Judge is almost optimally aligned. 4 RETHINKING UNLEARNING EV… view at source ↗

**Figure 3.** JensUn forms the Pareto front in forget-utility trade-off for different utility measures. For the LKF dataset, we show the trade-off between the forget set accuracy and (left) repetitiveness (middle) win rate vs the original model, (right) general understanding (MMLU). The curves are generated by sweeping over λR from Equation (1) for each method individually, detailed discussion in Appendix D. 8 [PITH_FULL_IMAGE:f… view at source ↗

Figures from the paper (19 more)

**Figure 4.** Figure 4: Sample questions from the LKF forget set. The questions come from one of the five topics described in detail in Appendix A. 4. Selection of Retain Concepts. For each event in the forget set, we select a set of topically related but distinct events for the retain set. F…

**Figure 5.** Figure 5: Sample dichotomous questions from the TOFU forget set. Selected dichotomous questions from the TOFU forget set, where a binary Yes/No answer suffices, making it fairly easy for a LLM to guess without reflecting true unlearning quality. TEMPLATE FOR GENERATING FORGET SE…

**Figure 6.** Figure 6: Query templates used to generate LKF sets. The following queries were used to generate the forget and retain set queries for the LKF dataset. A.3 GENERATION OF PARAPHRASES As an important part of our proposed evaluation is creating diverse paraphrases of test queries, …

**Figure 7.** Figure 7: Paraphrased query generation template. We use this template to paraphrase questions from both LKF and RWKU datasets. The same template is used to query Mistral-7B, Phi-3.5-mini-instruct, and Qwen2.5-3B-Instruct models with different values of N for forget and retain se…

**Figure 9.** Figure 9: The LLM-Judge performs very well. According to 6 human evaluators, for all methods from [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]

**Figure 8.** Figure 8: Sample questions with our generated paraphrases for the RWKU FB forget set where the paraphrased question gets the LLM to output the correct answer. The original questions are paraphrased either with Mistral-7B, Qwen2.5-3B-Instruct, or Phi-3.5-mini-instruct. Colored bo…

**Figure 10.** Figure 10: LLM-Judge prompt template for Win Rate evaluation for the AlpacaEval instruction based generation task. We use this template to rate comparative responses from the base and the unlearnt model. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]

**Figure 11.** Figure 11: Across unlearnt models the forget set accuracy saturates after certain relearning steps. Benign relearning performed on 200 real-world QA samples manages to restore close to pre-trained model forget set accuracy for some methods with 600 update steps. Further relearni…

**Figure 12.** Figure 12: Worst-case over different evaluation methods enhances forget-quality assessment. In this plot, we unlearn with the respective method for 5 epochs without paraphrases on the LKF dataset. Then, we show (a) standard (single question) forget set accuracy (b) worst-case fo…

**Figure 13.** Figure 13: Diversity in paraphrase generation is crucial for true forget set accuracy. In this plot, we unlearn with the respective method for 5 epochs without paraphrases on the LKF dataset. Then, we show how forget set accuracy increases on going from the standard (single quer…

**Figure 14.** Figure 14: All variants of JensUn achieved via different y target t in Equation (4) yield good forget set accuracy v utility trade-off. On average all JensUn variants attain lower forget set accuracy while staying on the same level as the original model in comparison to the base…

**Figure 15.** Figure 15: Sample outputs on successful forgetting across unlearning methods. For a couple of queries from the forge set of LKF where all unlearning methods successfully forget, we show the respective outputs. The different variants of JensUn allow control over the desired outpu…

**Figure 16.** Figure 16: Training dynamics between accuracy and different losses of JensUn. In this plot for the LKF dataset, we show how forget/retain accuracies and losses look as a function of unlearning steps starting from the pre-trained LLM. Firstly, in terms of forget set, already afte…

**Figure 17.** Figure 17: Problems with ROGUE-L based metrics for short and factual answers. In first example we highlight that ROUGE is not a good measure when the reference texts are paraphrases. The second example highlights how non-crucial tokens in Ref-Output increase the ROUGE recall to …

**Figure 18.** Figure 18: ROUGE problems are evident in RWKU targets. Across targets and query types like FB and QA, ROUGE under/over-estimates the unlearning quality of model outputs (LLM-Output) given the ground truth in all examples. We show further two examples illustrating problems with R…

**Figure 19.** Figure 19: Prompt made to the LLM-Judge. We use the following query along with 5 test cases to solicit a "Yes/No" response form Gemini-2.5-Flash-Preview model. 39 [PITH_FULL_IMAGE:figures/full_fig_p039_19.png]

**Figure 20.** Figure 20: Sample outputs on the AlpacaEval instruction based question used for repetitiveness and Quality assessment. For all unlearning methods used to unlearn the forget set of LKF, we show how the outputs look. Colored boxes depict: correct answer w.r.t GT/Query , and answe…

**Figure 21.** Figure 21: Sample outputs on the AlpacaEval instruction based question used for repetitiveness and Quality assessment. For all unlearning methods used to unlearn the forget set of LKF, we show how the outputs look. Colored boxes depict: correct answer w.r.t GT/Query , and answe…

**Figure 22.** Figure 22: Sample outputs on the AlpacaEval instruction based question used for repetitiveness and Quality assessment. For all unlearning methods used to unlearn the forget set of RWKU, we show how the outputs look. Colored boxes depict: correct answer w.r.t GT/Query , and answ…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Crossing the Margin Cliff: Toward Relearn-Robust LLM Unlearning via Margin Calibration
cs.AI 2026-07 conditional novelty 7.0 of 10

Margin Calibration, a non-saturating margin-anchored LoRA polish, crosses the margin cliff and cuts post-attack relearn recovery on all 97 populated cells in the paper's stress matrix.
Forget Narrowly, Retain Broadly: Unlearning as an Asymmetric Generalization Problem
cs.LG 2026-07 accept novelty 7.0 of 10

SUITE defines the forget-retain boundary at semantic, syntactic and lexical levels; training on it plus JensUn++ yields near-complete forgetting with minimal retain and utility loss.
Metis: Memory Foundation Model
cs.CL 2026-07 conditional novelty 6.5 of 10

Metis puts a trainable fixed-size memory matrix inside a frozen LLM backbone and learns to remember, update, forget, and reflect across turns without replaying original context.

Reference graph

Works this paper leans on

15 extracted references · 15 canonical work pages · cited by 3 Pith papers

[1]

Details on LKF and our new evaluation protocol

Appendix A . . . Details on LKF and our new evaluation protocol

work page
[2]

Experimental details

Appendix B . . . Experimental details

work page
[3]

Additional evaluation experiments

Appendix C . . . Additional evaluation experiments

work page
[4]

Additional unlearning experiments

Appendix D . . . Additional unlearning experiments

work page
[5]

Extended discussions and proofs A D ATASET AND PARAPHRASING DETAILS In this section, we explain in detail the LKF generation process and the paraphrasing details

Appendix E . . . Extended discussions and proofs A D ATASET AND PARAPHRASING DETAILS In this section, we explain in detail the LKF generation process and the paraphrasing details. A.1 T HE NEED FOR LKF For controlled tests on paraphrases and worst-case evaluations, we create the Lesser Known Facts (LKF) dataset, an unlearning benchmark with QA-type querie...

work page 2023
[6]

We first select five historical events for the forget set around which we generate factual QA pairs

Pick forget concepts. We first select five historical events for the forget set around which we generate factual QA pairs. The selected events are: the Challenger Disaster, the Salem Witch Trials, the Cod Wars, the 1883 Krakatoa Eruption , and the Battle of Talas . These are chosen to span different time periods, geographic regions, and levels of general ...

work page
[7]

We use GPT-4 (OpenAI, 2023) and Gemini 2.5 (Google- Gemini-Team, 2025) to generate candidate QA pairs for each forget concept following the template in Figure 6

Generation of Candidate Forget QA Pairs. We use GPT-4 (OpenAI, 2023) and Gemini 2.5 (Google- Gemini-Team, 2025) to generate candidate QA pairs for each forget concept following the template in Figure 6. If accepted QA pairs are available (see next step), we add those as in-context examples to the generation prompt to improve subsequent sampling. Some exam...

work page 2023
[8]

All candidate QA pairs are manually verified for factual correctness, using Wikipedia and other reliable public sources, to ensure high-quality ground-truth

Verification of Forget QA Pairs. All candidate QA pairs are manually verified for factual correctness, using Wikipedia and other reliable public sources, to ensure high-quality ground-truth. 15 SAMPLE QUESTIONS , RESPECTIVE ANSWERS FROM THE FORGET SET OF LKF Question:After how many seconds of flight did the Space Shuttle Challenger break apart? Answer: 73...

work page

Show all 15 references

[9]

For each event in the forget set, we select a set of topically related but distinct events for the retain set

Selection of Retain Concepts. For each event in the forget set, we select a set of topically related but distinct events for the retain set. For example, for the Challenger Disaster we include other space missions such as Apollo 11, Moon landing, and the Sputnik Program; for t...

2004
[10]

Candidate QA pairs for the retain events are generated using a similar template approach as for the forget set (see Figure 6)

Generation of Candidate Retain QA Pairs. Candidate QA pairs for the retain events are generated using a similar template approach as for the forget set (see Figure 6)
[11]

role": "system

Verification of Retain QA Pairs. Retain QA pairs undergo an automated verification stage using GPT- 4 (OpenAI, 2023), Gemini 2.5 (Google-Gemini-Team, 2025), and DeepSeek V3 (DeepSeek-AI, 2025). The models are prompted to evaluate each QA pair for: (i) factual correctness, (ii)...

2023
[12]

This relearning set is disjoint of both the LKF forget and retain sets

Real-knowledge set. This relearning set is disjoint of both the LKF forget and retain sets. Specifically, we collect 200 QA pairs using the Mistral-7B model from topics like history, geography, biology, sports, etc
[13]

adversarial

LKF retain set. To simulate the attacker having access to some form of retain set, we take the non- paraphrased retain set of LKF which comprises of 400 distinct question-answer pairs. This is our adversarial relearning set. 0 50 100 200 600 1000 Relearn steps 0 20 40 60 80For...

2025
[14]

I Love Lucy,

**Computers:** The first commercial computers, Method: SimNPO 1950s Method: NPO 1950 was a time of great technological advancement and social change. Here are some of the current-day inventions and trends that I found from my research: Method: JensUn In 1950, the world was a v...

1950
[15]

Schrodinger’s cat

**Computing:** The first commercial computers, like UNIV Figure 20: Sample outputs on the AlpacaEval instruction based question used for repetitive- ness and Quality assessment. For all unlearning methods used to unlearn the forget set of LKF, we show how the outputs look. Col...

1935

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Details on LKF and our new evaluation protocol

Appendix A . . . Details on LKF and our new evaluation protocol

work page

[2] [2]

Experimental details

Appendix B . . . Experimental details

work page

[3] [3]

Additional evaluation experiments

Appendix C . . . Additional evaluation experiments

work page

[4] [4]

Additional unlearning experiments

Appendix D . . . Additional unlearning experiments

work page

[5] [5]

Extended discussions and proofs A D ATASET AND PARAPHRASING DETAILS In this section, we explain in detail the LKF generation process and the paraphrasing details

Appendix E . . . Extended discussions and proofs A D ATASET AND PARAPHRASING DETAILS In this section, we explain in detail the LKF generation process and the paraphrasing details. A.1 T HE NEED FOR LKF For controlled tests on paraphrases and worst-case evaluations, we create the Lesser Known Facts (LKF) dataset, an unlearning benchmark with QA-type querie...

work page 2023

[6] [6]

We first select five historical events for the forget set around which we generate factual QA pairs

Pick forget concepts. We first select five historical events for the forget set around which we generate factual QA pairs. The selected events are: the Challenger Disaster, the Salem Witch Trials, the Cod Wars, the 1883 Krakatoa Eruption , and the Battle of Talas . These are chosen to span different time periods, geographic regions, and levels of general ...

work page

[7] [7]

We use GPT-4 (OpenAI, 2023) and Gemini 2.5 (Google- Gemini-Team, 2025) to generate candidate QA pairs for each forget concept following the template in Figure 6

Generation of Candidate Forget QA Pairs. We use GPT-4 (OpenAI, 2023) and Gemini 2.5 (Google- Gemini-Team, 2025) to generate candidate QA pairs for each forget concept following the template in Figure 6. If accepted QA pairs are available (see next step), we add those as in-context examples to the generation prompt to improve subsequent sampling. Some exam...

work page 2023

[8] [8]

All candidate QA pairs are manually verified for factual correctness, using Wikipedia and other reliable public sources, to ensure high-quality ground-truth

Verification of Forget QA Pairs. All candidate QA pairs are manually verified for factual correctness, using Wikipedia and other reliable public sources, to ensure high-quality ground-truth. 15 SAMPLE QUESTIONS , RESPECTIVE ANSWERS FROM THE FORGET SET OF LKF Question:After how many seconds of flight did the Space Shuttle Challenger break apart? Answer: 73...

work page