Pith. sign in

REVIEW 31 cited by

In-Context Unlearning: Language Models as Few Shot Unlearners

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.07579 v4 pith:W3FPLZBB submitted 2023-10-11 cs.LG cs.AIcs.CR

Martin Pawelczyk , Seth Neel , Himabindu Lakkaraju This is my paper · ORCID
classification cs.LGcs.AIcs.CR
keywords modelunlearninginstancesllmsspecificaccessin-contextmodels
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Machine unlearning, the study of efficiently removing the impact of specific training instances on a model, has garnered increased attention in recent years due to regulatory guidelines such as the \emph{Right to be Forgotten}. Achieving precise unlearning typically involves fully retraining the model and is computationally infeasible in case of very large models such as Large Language Models (LLMs). To this end, recent work has proposed several algorithms which approximate the removal of training data without retraining the model. These algorithms crucially rely on access to the model parameters in order to update them, an assumption that may not hold in practice due to computational constraints or having only query access to the LLMs. In this work, we propose a new class of unlearning methods for LLMs called ``In-Context Unlearning.'' This method unlearns instances from the model by simply providing specific kinds of inputs in context, without the need to update model parameters. To unlearn specific training instances, we present these instances to the LLMs at inference time along with labels that differ from their ground truth. Our experimental results demonstrate that in-context unlearning performs on par with, or in some cases outperforms other state-of-the-art methods that require access to model parameters, effectively removing the influence of specific instances on the model while preserving test accuracy.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 31 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Space Complexity of Learning-Unlearning Algorithms

    cs.LG 2025-06 accept novelty 8.0 of 10

    The space complexity of machine unlearning for realizability testing is characterized by eluder dimension (central lower bound), star number (ticketed upper bound), and hollow star number (bounded deletions), separati...

  2. MOON: Multi-Objective OrthoNormalized Updates for Multitask Learning

    cs.LG 2026-08 conditional novelty 7.0 of 10

    MOON applies spectral-nuclear-norm geometry to multi-objective gradient manipulation and uses polar-factor updates, with O(T^-1/2) deterministic and O(T^-1/4) stochastic convergence to Pareto stationarity.

  3. LEMUR: Latent Entropy-aware Multimodal Unlearning via Visual-anchored Reasoning Redirection

    cs.LG 2026-08 conditional novelty 7.0 of 10

    An inference-time unlearning method uses token entropy to detect when RL-trained multimodal reasoning models are about to recite private facts in their chain of thought, then redirects decoding with a visual anchor.

  4. ExAnte: A Benchmark for Ex-Ante Inference in Large Language Models

    cs.LG 2025-05 conditional novelty 7.0 of 10

    Models leak future knowledge despite explicit temporal cutoffs, as quantified by the ExAnte benchmark across four tasks.

  5. Distribution Preference Optimization: A Fine-grained Perspective for LLM Unlearning

    cs.LG 2025-10 conditional novelty 6.0 of 10

    DiPO is a distribution-level unlearning method that constructs preference distributions from the model's own high-confidence logits and achieves state-of-the-art forget quality on TOFU while preserving utility.

  6. OFMU: Optimization-Driven Framework for Machine Unlearning

    cs.LG 2025-09 unverdicted novelty 6.0 of 10

    OFMU is a penalty-based bi-level optimizer for machine unlearning that alternates between a gradient-ascent forgetting step and a gradient-descent utility-restoration step, with a similarity penalty between forget and...

  7. AntiDote: Bi-level Adversarial Training for Tamper-Resistant LLMs

    cs.CL 2025-09 conditional novelty 6.0 of 10

    A bi-level adversarial training method where a hypernetwork generates malicious LoRA patches to attack the defender, and the defender learns to nullify them, improves tamper resistance across ten open-weight LLMs with...

  8. Towards Evaluation for Real-World LLM Unlearning

    cs.AI 2025-08 conditional novelty 6.0 of 10

    DCUE evaluates LLM unlearning by comparing core-token confidence score distributions of the unlearned model and the original model, corrected by a validation set, using the Kolmogorov-Smirnov test.

  9. Generating Project-Specific Test Cases with Requirement Validation Intention

    cs.SE 2025-07 unverdicted novelty 6.0 of 10

    IntentionTest retrieves a reusable test from the project and edits it with an LLM to match a supplied validation intention, yielding tests that kill 28.1-37.6% more mutants, share 16.9-23.9% more coverage, and produce...

  10. What Should LLMs Forget? Quantifying Personal Data in LLMs for Right-to-Be-Forgotten Requests

    cs.CL 2025-07 conditional novelty 6.0 of 10

    WikiMem, a Wikidata-derived canary dataset and a calibrated NLL-ranking metric, identifies which human-fact associations an LLM has memorized, with higher rates for famous people and larger models.

  11. Enhancing Safe and Controllable Protein Generation via Knowledge Preference Optimization

    cs.AI 2025-07 conditional novelty 6.0 of 10

    A knowledge-graph-guided preference optimization framework that fine-tunes protein language models to generate fewer sequences similar to known harmful proteins.

  12. Learning-Time Encoding Shapes Unlearning in LLMs

    cs.CL 2025-06 conditional novelty 6.0 of 10

    How knowledge is encoded during LLM fine-tuning strongly affects later unlearning: paraphrased training data helps unlearning, while entangled chunks hinder selective forgetting.

  13. Unlearning Isn't Invisible: Detecting Unlearning Traces in LLMs from Model Outputs

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Unlearning in LLMs leaves detectable 'fingerprints' that let a simple classifier distinguish an unlearned model from its original, even on unrelated prompts.

  14. R-TOFU: Unlearning in Large Reasoning Models

    cs.CL 2025-05 conditional novelty 6.0 of 10

    R-TOFU shows that answer-level unlearning is insufficient for large reasoning models because residual knowledge persists in chain-of-thought traces.

  15. GUARD: Generation-time LLM Unlearning via Adaptive Restriction and Detection

    cs.CL 2025-05 conditional novelty 6.0 of 10

    GUARD performs inference-time unlearning by classifying prompts, retrieving original answers, and penalizing token matches during beam search, preserving utility but with forget quality that collapses on larger TOFU f...

  16. Layered Unlearning for Adversarial Relearning

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Layered Unlearning sequentially forgets growing subsets of data to create fold-specific inhibition, blocking recovery of early folds by adversarial relearning.

  17. Unilogit: Robust Machine Unlearning for LLMs Using Uniform-Target Self-Distillation

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Unilogit unlearns text from LLMs by setting the target token probability to 1/|V| and distilling from the current model, reporting stronger forgetting at comparable retained utility than NPO and UnDIAL.

  18. Can Language Models Represent the Past without Anachronism?

    cs.CL 2025-04 conditional novelty 6.0 of 10

    Fine-tuned GPT-4o-mini still betrays its present-day training to human readers, while prompting alone fails to shift style, evidence that period pretraining may be required for historical simulation.

  19. Agents Are All You Need for LLM Unlearning

    cs.AI 2025-02 reject novelty 6.0 of 10

    A four-agent pipeline, Vanilla, AuditErase, Critic, and Composer, filters target references out of LLM responses, claiming robust and scalable inference-time unlearning without weight updates.

  20. Unified Parameter-Efficient Unlearning for LLMs

    cs.AI 2024-11 conditional novelty 6.0 of 10

    An influence-function-based parameter editing framework performs instance removal, query modification, and response correction on PEFT adapters without retraining.

  21. SEUF: Is Unlearning One Expert Enough for Mixture-of-Experts LLMs?

    cs.LG 2024-11 conditional novelty 6.0 of 10

    SEUF restricts MoE LLM unlearning to the top-1 most relevant expert and adds a router anchor loss, improving utility by up to 35% while keeping forget efficacy roughly comparable.

  22. GROM: Gradient-Free Rapid One-Shot Machine Unlearning

    cs.LG 2026-08 conditional novelty 5.0 of 10

    A single closed-form ridge update to selected MLP layers removes targeted knowledge from LLMs in seconds, with state-of-the-art forgetting-utility trade-offs and quantization robustness.

  23. SoK: Machine Unlearning for Large Language Models

    cs.LG 2025-06 conditional novelty 5.0 of 10

    A new taxonomy for LLM unlearning distinguishes removal-intended from suppression-intended methods, and argues that gradient ascent methods functionally behave like suppression.

  24. Exploring Criteria of Loss Reweighting to Enhance LLM Unlearning

    cs.LG 2025-05 conditional novelty 5.0 of 10

    The authors propose SatImp, a product of a saturation weight and an importance weight, and show it improves the unlearn-retain trade-off on TOFU, WMDP, and MUSE.

  25. Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyond

    cs.LG 2025-02 conditional novelty 5.0 of 10

    Adding sharpness-aware minimization to NPO unlearning slows relearning attacks on WMDP and MUSE benchmarks.

  26. Optimizing Code Runtime Performance through Context-Aware Retrieval-Augmented Generation

    cs.SE 2025-01 conditional novelty 5.0 of 10

    An LLM code optimizer using control-flow-graph differences and retrieved examples reports 7.3% average runtime reduction on 116 C++ programs versus zero-shot GPT-4o.

  27. Multi-Objective Large Language Model Unlearning

    cs.CL 2024-12 conditional novelty 5.0 of 10

    MOLLM formulates LLM unlearning as a three-objective optimization problem and uses a bounded unlearning loss plus a common descent direction to forget target data while preserving model utility.

  28. LLM Unlearning for Cyber Defense: A Survey on Methods, Challenges, and Emerging Threats

    cs.LG 2026-06 conditional novelty 4.0 of 10

    Most gradient-based LLM unlearning methods achieve behavioral suppression, not true forgetting, and current benchmarks cannot certify that knowledge has been removed.

  29. Step-by-Step Reasoning Attack: Revealing 'Erased' Knowledge in Large Language Models

    cs.CR 2025-06 reject novelty 4.0 of 10

    Step-by-step reasoning prompts can recover purportedly erased facts from unlearned LLMs, but the paper's quantitative evidence is internally inconsistent.

  30. Lifting Data-Tracing Machine Unlearning to Knowledge-Tracing for Foundation Models

    cs.CV 2025-06 conditional novelty 4.0 of 10

    A position paper urging a shift from data-tracing to knowledge-tracing machine unlearning for foundation models, supported by a CLIP case study that shows current methods struggle to generalize.

  31. UCD: Unlearning in LLMs via Contrastive Decoding

    cs.CL 2025-06 conditional novelty 4.0 of 10

    UCD steers an LLM away from forget-set content at inference time by mixing in the difference between forget-tuned and retain-tuned small models.

Pith tools