Pith. sign in

REVIEW 5 major objections 5 minor 5 cited by

Worst-case reweighting of the forget set synchronizes LLM unlearning, improving forget quality without sacrificing model utility.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 10:40 UTC pith:POAJ5PFD

load-bearing objection A solid empirical plug-in for LLM unlearning with a DRO flavor, but the balanced-forgetting mechanism needs direct validation before the story fully lands. the 5 major comments →

arxiv 2601.09172 v3 pith:POAJ5PFD submitted 2026-01-14 cs.LG

BalDRO: A Distributionally Robust Optimization based Framework for Large Language Model Unlearning

Pengyang Shao , Naixin Zhai , Lei Chen , Yonghui Yang , Fengbin Zhu , Xun Yang , Meng Wang This is my paper
classification cs.LG
keywords LLM unlearningdistributionally robust optimizationsample imbalanceasynchronous forgettingforget qualitymodel utilitynegative preference optimizationDonsker-Varadhan dual
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that the core obstacle to effective LLM unlearning is sample-wise imbalance: easy samples are over-forgotten while hard samples remain insufficiently erased, so no single stopping point leaves all forget targets cleanly removed. BalDRO reframes unlearning as a min–sup problem: instead of minimizing average loss on the forget set, the model minimizes the worst-case expected loss over a family of distributions that upweight hard-to-unlearn samples. Two tractable instantiations are given: a discrete GroupDRO variant that trains on the highest-loss subset of each batch, and a continuous Donsker–Varadhan dual that applies softmax-like exponential weighting. If correct, this gives a principled, plug-in upgrade for existing unlearning methods, and experiments on TOFU and MUSE show higher forget quality and privacy metrics with nearly unchanged utility.

Core claim

BalDRO's central discovery is that balancing the forget set through a distributionally robust objective — rather than through heuristic per-sample weights — yields synchronized forgetting across samples of varying difficulty. Formally, it solves min_θ sup_{Q:D_KL(Q||P)≤η} E_Q[ℓ_f(Z;θ)], whose Donsker–Varadhan dual reduces to minimizing β log E_{Z~P}[exp(ℓ_f(Z;θ)/β)]. This log-sum-exp loss automatically assigns exponentially larger gradient influence to samples with larger forget loss, preventing easy samples from being erased too early while hard samples lag behind. On TOFU, NPO+BalDRO-DV raises Forget Quality from 0.766 to 0.990 at 1% forget ratio with Model Utility essentially flat, and si

What carries the argument

The central object is the KL-divergence-bounded distributionally robust objective applied solely to the forget loss, combined with the Donsker–Varadhan dual representation. This dual turns the adversarial inner supremum into a smooth log-sum-exp function of per-sample losses, β log (1/n) Σ_i exp(ℓ_i/β), which serves as a differentiable soft-max reweighting. A second variant, BalDRO-G, uses GroupDRO to focus each batch on its top-50% highest-loss forget samples, approximating the same worst-case principle in discrete form. Together, these mechanisms self-regulate sample contributions during training: harder samples pull more gradient, already-forgotten samples shrink toward zero weighting.

Load-bearing premise

The load-bearing premise is that a sample's forget loss ℓ_f(z;θ) accurately measures how hard that sample is to unlearn, so that worst-case reweighting emphasizes exactly the samples that lag behind.

What would settle it

Measure per-sample forgetting speed independently — e.g., the number of unlearning steps needed for the model's probability of the target answer to fall below a fixed threshold — and correlate it with the per-sample loss used in BalDRO's reweighting. If samples with high loss are not the slow-to-forget ones, BalDRO's emphasis is misdirected and the claimed synchronization across difficulty levels should not hold.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Adding BalDRO's reweighting to existing unlearning objectives (NPO, SimNPO, SatImp) consistently improves forget quality across TOFU and MUSE without changing the base training loop.
  • On TOFU 1% forget ratio, NPO+BalDRO-DV improves Forget Quality from 0.7659 to 0.9900 while Model Utility stays near 0.58.
  • On MUSE, BalDRO variants lower verbatim memorization and improve membership-inference privacy scores in both News and Books domains.
  • Applying DRO to the retain set instead of the forget set yields no benefit, suggesting the imbalance phenomenon is specific to forget data.
  • As β→∞ the BalDRO-DV objective collapses to the standard mean-loss unlearning, and as β→0 it approaches maximum-loss optimization, unifying prior methods as two poles of the same continuum.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A testable implication not explored in the paper: if asynchronous forgetting is indeed the bottleneck, then even a simple explicit softmax reweighting over per-sample losses (without the DRO derivation) should reproduce most of BalDRO's gains; comparing BalDRO-DV against such a direct-weighting baseline would isolate whether the min–sup formulation adds value beyond the reweighting itself.
  • Because the KL radius η cancels out of the implemented DV objective, BalDRO-DV's behavior is fully determined by β; this suggests the 'uncertainty set' framing is a motivation rather than an enforced constraint, and tuning β is the only robustness dial.
  • The assumption that per-sample loss equals unlearning difficulty could be stress-tested on synthetic forget sets with known difficulty labels (e.g., facts with controlled pretraining frequencies); if high-loss samples are not the slow-to-forget ones, the balanced-forgetting claim would be weakened.
  • The paper's TOFU numbers show FQ near-perfect (0.99) with NPO+BalDRO-DV; an interesting extension would examine whether such extreme reweighting risks memorizing the weighting itself (e.g., producing uniform refusal patterns), which the reported EM/ES metrics only partially capture.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes BalDRO, a distributionally robust optimization (DRO) framework for LLM unlearning that reweights forget-set samples to emphasize hard-to-unlearn examples. It formulates unlearning as a min-sup problem over a KL ball around the empirical forget distribution, and gives two realizations: BalDRO-G, a GroupDRO-style top-50% selection, and BalDRO-DV, a Donsker-Varadhan log-sum-exp objective. Experiments on TOFU and MUSE with LLaMA-2-7B compare against GA, GradDiff, NPO, SimNPO, and SatImp, reporting improved forget quality with comparable model utility, plus analyses of forget ratio, hyperparameters, retain-set DRO, and extended privacy metrics. Appendix A derives the optimal adversarial distribution via a standard Lagrangian calculation.

Significance. The DRO reweighting idea is simple, practically attractive, and the Appendix A derivation is standard and correct. The paper ships code, covers two benchmarks and several base methods, and includes useful hyperparameter and extended-metric analyses. If the reported gains are reproducible, BalDRO would be a lightweight plug-in improvement for gradient-based unlearning. However, the central conceptual claim of 'balanced/synchronized' forgetting is not directly tested, the implemented DV objective drops the KL radius and is effectively a penalized exponential-tilt reweighting, and several numerical anomalies in the main tables need explanation before the empirical claims can be accepted. The idea is worth pursuing, but the current manuscript overstates the strength of the evidence.

major comments (5)
  1. [§4.2.2, Eq. (9)–(12)] The Lagrangian in Eq. (9) contains βη, and Eq. (10)–(11) retain it. The final implemented objective, Eq. (12), removes βη because it is constant in θ for fixed β, and η never appears in the algorithm or in the experiments (Sec. 5.1.4 tunes only β and λ). Consequently, the method is a KL-penalized exponential-tilt reweighting with temperature β, not an explicit minimization over the KL ball of radius η defined in Eq. (6). The paper should either reformulate Definition 4.1/Eq. (6) as a penalized DRO problem or reintroduce η as an actual hyperparameter and report its value; otherwise the 'uncertainty set' claim is not supported by the implemented objective.
  2. [§4.2.1, Eq. (8)] Eq. (8) defines BalDRO-G as the maximum over G fixed groups, but the text immediately switches to a percentile-based top-50% selection. A fixed group partition and a per-iteration top-50% loss cut are not the same: the latter redefines the 'worst group' at every step from current per-sample losses and does not correspond to optimizing a fixed set of groups. No grouping mechanism, group size, or G is specified. Please give the exact update rule (e.g., a binary mask over per-sample losses) and clarify the relationship to GroupDRO.
  3. [§4.1/Definition 4.1 and §5.2] The entire mechanism rests on the assumption that high per-sample forget loss ℓ_f(Z;θ) identifies samples that are hard to unlearn. The paper provides no direct validation of this proxy. Figure 1(a) shows PPL divergence, not a correlation between loss magnitude and time-to-forget, and no synchronization metric is reported. A random-weight control (matching the same weight distribution but decoupled from loss) would show whether the DRO weighting itself drives the FQ gains, as opposed to simply applying more optimization pressure. Please also report a direct synchronization metric, such as the variance of per-sample forgetting epochs, since aggregate FQ improvements do not by themselves demonstrate balanced forgetting.
  4. [Tables 1 and 2] Several numerical anomalies need to be addressed. Table 1 reports identical FQ = 0.5786 for both SimNPO+BalDRO-G and SimNPO+BalDRO-DV; Table 2 reports PL = 100.41 for SimNPO+BalDRO-DV on News, while the base SimNPO is −99.90 and the stated preference is PL → 0 (Sec. 5.1.2). These contradict the text's claim that BalDRO 'consistently enhances' the base methods across domains. In addition, SatImp in Table 1 has FQ = 0.0013, exactly equal to Original, and LOSS/ZLib values near 0.99 in Table 3, which suggests the SatImp baseline may not have unlearned at all. Please verify the numbers, correct the tables, and discuss any degenerate baseline configuration.
  5. [Tables 1–3 and Figs. 3–5] All main results are reported as single runs without error bars or seed information. Many comparisons are small in absolute terms (e.g., Table 1 MU for NPO vs NPO+BalDRO-DV: 0.5775 vs 0.5815), so it is hard to assess whether the differences are significant. Report mean ± std over at least three seeds and describe the hyperparameter selection protocol (validation split, number of trials, budget) to rule out selection on the test metrics.
minor comments (5)
  1. [Eq. (11)–(12)] The step from Eq. (11) to Eq. (12) should explicitly state that βη is omitted as a constant independent of θ; as written, the equality appears to drop a term without comment.
  2. [Figure 3] Each panel repeats the labels 'FQ MU' with no clear title indicating forget ratio or base method. Use distinct titles or annotations to make the figure self-contained.
  3. [§5.3.2] The top-50% threshold in BalDRO-G is a hyperparameter, but the sensitivity analysis covers only β and λ. Add a brief analysis of the percentile or state that it was fixed after preliminary experiments.
  4. [§5.2.2 and Table 2] For PrivLeak, explain the sign convention explicitly before interpreting changes such as −90.85 → −65.70 as improvements; the text says 'closer to zero is preferred,' but the signed interpretation is not spelled out for negative values.
  5. [§2.1] The reference to [11] on fact frequency and forgetting difficulty is relevant, but the paper should distinguish that corpus-frequency notion from the per-sample loss proxy used in BalDRO; they are not obviously the same quantity.

Circularity Check

0 steps flagged

No significant circularity: the min–sup objective is standard DRO, the dual derivation is self-contained, and the reported FQ/MU gains are measured on external benchmarks rather than read off the fitted objective.

full rationale

The paper's derivation chain starts with Definition 4.1, which explicitly formalizes balanced unlearning as a worst-case expected forget loss under a KL ball (Eq. 6). This is a modeling definition, not a result derived from the outcome it is later said to predict. The inner supremum is solved by the standard Donsker–Varadhan dual: Eq. (9)-Eq. (12) and Appendix A give a closed-form Q* proportional to exp(ℓ_f/β), and the final objective is a log-sum-exp reweighting. No equation in this chain assumes the empirical claims (FQ, MU, EM/ES) that are later reported; those metrics are computed from model outputs on TOFU/MUSE and are not algebraic consequences of Eq. (12). The only concern is semantic: the paper labels high-loss samples as 'harder forget samples' (after Eq. 6 and Eq. 10), so the statement that BalDRO 'emphasizes hard-to-unlearn samples' is an operational definition rather than a validated empirical correlation. That is an unverified proxy assumption (and the KL radius η cancels out of the implemented objective, so the 'uncertainty set' framing is loose), but it is not circular: the benchmark results would remain meaningful even if the proxy were wrong, and no fitted value is renamed as a prediction. Self-citations such as [51] (and [24], [25], [50]) appear in related-work enumerations only; none is load-bearing for the DRO derivation or the empirical evaluation. Accordingly the score is 2: minor non-load-bearing self-citation, no reduction of a prediction to its inputs by construction.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central claim rests on the DRO formulation (KL-penalized expected-loss minimization) and on the empirical proxy that high-loss forget samples are the under-unlearned ones. β and top-50% are tuned on the evaluation benchmarks; η appears only as a constant and is never set.

free parameters (4)
  • β (BalDRO-DV temperature) = searched over {1.0, 2.0, 5.0, 10.0}; chosen per benchmark
    Controls sharpness of exponential weighting in log-sum-exp objective (Eq. 12); selected by validation performance, not derived.
  • top-50% percentile (BalDRO-G) = 50% (fixed, not ablated)
    Determines which samples are 'hardest' and receive gradient; no sensitivity analysis given; manual choice.
  • λ (retain-loss weight) = searched over {0.25, 0.5, 1.0, 2.0}
    Balances forget vs retain loss in Eq. (1); tuned per experiment.
  • η (KL uncertainty radius) = never set; cancels out
    Appears in Eq. (6) but drops out of the final objective; effectively a free constant with no role.
axioms (4)
  • domain assumption The empirical distribution over the forget set is the reference measure; worst-case distributions are absolutely continuous w.r.t. it.
    Required for density-ratio reparameterization in Appendix A; ignores off-support forget samples.
  • standard math Strong duality holds for the Lagrangian relaxation of the KL-constrained DRO.
    The closed-form Q* (Eq. 10) requires the sup to be attained; Slater's condition for η>0 is not checked.
  • ad hoc to paper Per-sample forget loss ℓ_f is a reliable proxy for unlearning difficulty.
    Central premise of Definition 4.1; never validated directly; PPL trajectories in Fig. 1 are illustrative, not loss-difficulty correlation.
  • domain assumption The retain loss needs no balancing and can be added independently.
    They test DRO on retain in §5.3.3 and find it unhelpful, supporting the assumption, but it remains an empirical finding not a guarantee.

pith-pipeline@v1.3.0-alltime-deepseek · 18999 in / 15746 out tokens · 145843 ms · 2026-08-03T10:40:00.928760+00:00 · methodology

0 comments
read the original abstract

As Large Language Models (LLMs) increasingly shape online content, removing targeted information from well-trained LLMs (also known as LLM unlearning) has become critical for web governance. A key challenge lies in sample-wise imbalance within the forget set: different samples exhibit widely varying unlearning difficulty, leading to asynchronous forgetting where some knowledge remains insufficiently erased while others become over-forgotten. To address this, we propose BalDRO, a novel and efficient framework for balanced LLM unlearning. BalDRO formulates unlearning as a min-sup process: an inner step identifies a worst-case data distribution that emphasizes hard-to-unlearn samples, while an outer step updates model parameters under this distribution. We instantiate BalDRO via two efficient variants: BalDRO-G, a discrete GroupDRO-based approximation focusing on high-loss subsets, and BalDRO-DV, a continuous Donsker-Varadhan dual method enabling smooth adaptive weighting within standard training pipelines. Experiments on TOFU and MUSE show that BalDRO significantly improves both forgetting quality and model utility over existing methods, and we release code for reproducibility.

Figures

Figures reproduced from arXiv: 2601.09172 by Fengbin Zhu, Lei Chen, Meng Wang, Naixin Zhai, Pengyang Shao, Xun Yang, Yonghui Yang.

Figure 1
Figure 1. Figure 1: Illustration of sample-wise imbalance in LLM unlearning. a) Per-sample PPL (perplexity) at early and later epochs [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The overall min–sup process of BalDRO. The inner [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Performance with varying forget ratios (5% and 10%) on the TOFU benchmark. We focus on FQ and MU, the two most [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Performance of BalDRO-DV with varying 𝛽 and balancing parameter 𝜆 on the TOFU benchmark. forget set, we conduct additional experiments on TOFU using forget￾set ratios of 5% and 10% [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Performance comparisons between whether apply [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. One Anchor for All: Unified Multilingual and Multimodal Safety Alignment for LVLMs

    cs.AI 2026-07 conditional novelty 6.0

    A small set of model neurons shared across languages and modalities can transfer English-only safety training to multilingual and multimodal refusal behavior.

  2. Mitigating Error Amplification in Fast Adversarial Training

    cs.LG 2026-04 unverdicted novelty 6.0

    DDG dynamically adjusts perturbation magnitude and supervision strength in fast adversarial training according to sample confidence at the ground-truth class, mitigating catastrophic overfitting and the robustness-acc...

  3. Targeted Interpretable Safety Neuron Enhancement for Multilingual Vision-Language Large Models

    cs.CV 2026-04 unverdicted novelty 6.0

    Precise Shield identifies safety neurons in VLLMs via activation contrasts and aligns only them with gradient masking, boosting safety, preserving generalization, and enabling zero-shot cross-lingual and cross-modal transfer.

  4. VC-Soup: Value-Consistency Guided Multi-Value Alignment for Large Language Models

    cs.LG 2026-03 unverdicted novelty 6.0

    VC-Soup uses a cosine-similarity consistency metric to filter data, trains value-consistent policies, and applies linear merging with Pareto filtering to improve multi-value LLM alignment trade-offs.

  5. Targeted Interpretable Safety Neuron Enhancement for Multilingual Vision-Language Large Models

    cs.CV 2026-04 conditional novelty 5.0

    A neuron-targeted safety tuning method for VLLMs reduces attack success rates from ~20-30% to ~4-6% on average across ten languages while using less than 0.03% of parameters.

Reference graph

Works this paper leans on

65 extracted references · 12 linked inside Pith · cited by 4 Pith papers

  1. [1]

    Haoyue Bai, Haoyu Wang, Shengyu Chen, Zhengzhang Chen, Lu-An Tang, Wei Cheng, Haifeng Chen, and Yanjie Fu. 2025. Learning to Route: A Rule-Driven Agent Framework for Hybrid-Source Retrieval-Augmented Generation.arXiv preprint arXiv:2510.02388(2025)

    arXiv 2025

  2. [2]

    Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021. Extracting training data from large language models. In30th USENIX security symposium (USENIX Security 21). 2633–2650

    2021

  3. [3]

    Junkai Chen, Zhijie Deng, Kening Zheng, Yibo Yan, Shuliang Liu, PeiJun Wu, Peijie Jiang, Jia Liu, and Xuming Hu. 2025. Safeeraser: Enhancing safety in multimodal large language models through multimodal machine unlearning. arXiv preprint arXiv:2502.12520(2025)

    arXiv 2025

  4. [4]

    Feng-Qi Cui, Anyang Tong, Jinyang Huang, Jie Zhang, Dan Guo, Zhi Liu, and Meng Wang. 2025. Learning from heterogeneity: Generalizing dynamic facial expression recognition via distributionally robust optimization. InProceedings of the 33rd ACM International Conference on Multimedia. 5587–5596

    2025

  5. [5]

    Chongyu Fan, Jiancheng Liu, Licong Lin, Jinghan Jia, Ruiqi Zhang, Song Mei, and Sijia Liu. 2025. Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning. InNeurips Safe Generative AI Workshop 2024

    2025

  6. [6]

    Junfeng Fang, Zijun Yao, Ruipeng Wang, Haokai Ma, Xiang Wang, and Tat- Seng Chua. 2025. We Should Identify and Mitigate Third-Party Safety Risks in MCP-Powered Agent Systems.arXiv preprint arXiv:2506.13666(2025)

    Pith/arXiv arXiv 2025

  7. [7]

    Jiahui Geng, Qing Li, Herbert Woisetschlaeger, Zongxiong Chen, Fengyu Cai, Yuxia Wang, Preslav Nakov, Hans-Arno Jacobsen, and Fakhri Karray. 2025. A comprehensive survey of machine unlearning techniques for large language models.arXiv preprint arXiv:2503.01854(2025)

    Pith/arXiv arXiv 2025

  8. [8]

    Jinpeng Hu, Tengteng Dong, Gang Luo, Hui Ma, Peng Zou, Xiao Sun, Dan Guo, Xun Yang, and Meng Wang. 2024. Psycollm: Enhancing llm for psychological understanding and evaluation.IEEE Transactions on Computational Social Systems 12, 2 (2024), 539–551

    2024

  9. [9]

    Wonje Jeung, Sangyeon Yoon, Hyesoo Hong, Soeun Kim, Seungju Han, Youngjae Yu, and Albert No. 2025. Dusk: Do not unlearn shared knowledge.arXiv preprint arXiv:2505.15209(2025)

    Pith/arXiv arXiv 2025

  10. [10]

    Jinghan Jia, Yihua Zhang, Yimeng Zhang, Jiancheng Liu, Bharat Runwal, James Diffenderfer, Bhavya Kailkhura, and Sijia Liu. 2024. SOUL: Unlocking the Power of Second-Order Optimization for LLM Unlearning. InEMNLP

    2024

  11. [11]

    Aravind Krishnan, Siva Reddy, and Marius Mosbach. 2025. Not All Data Are Unlearned Equally. InSecond Conference on Language Modeling. https: //openreview.net/forum?id=Kd97lfFfTu

    2025

  12. [12]

    Solomon Kullback. 1951. Kullback-leibler divergence.Tech. Rep.(1951)

    1951

  13. [13]

    Claude Lemaréchal. 2001. Lagrangian relaxation. InComputational combinatorial optimization: optimal or provably near-optimal solutions. Springer, 112–156

    2001

  14. [14]

    Zexi Li, Xiangzhu Wang, William F Shen, Meghdad Kurmanji, Xinchi Qiu, Dongqi Cai, Chao Wu, and Nicholas D Lane. 2025. Editing as Unlearning: Are Knowledge Editing Methods Strong Baselines for Large Language Model Unlearning?arXiv preprint arXiv:2505.19855(2025)

    Pith/arXiv arXiv 2025

  15. [15]

    Fengming Lin, Xiaolei Fang, and Zheming Gao. 2022. Distributionally robust optimization: A review on theory and applications.Numerical Algebra, Control and Optimization12, 1 (2022), 159–212

    2022

  16. [16]

    Xinyu Lin, Wenjie Wang, Jujia Zhao, Yongqi Li, Fuli Feng, and Tat-Seng Chua

  17. [17]

    Chris Liu, Yaxuan Wang, Jeffrey Flanigan, and Yang Liu. 2024. Large language model unlearning via embedding-corrupted prompts.Advances in Neural Infor- mation Processing Systems37 (2024), 118198–118266

    2024

  18. [18]

    Jilong Liu, Pengyang Shao, Wei Qin, Fei Liu, Yonghui Yang, and Richang Hong

  19. [19]

    Jiashuo Liu, Jiayun Wu, Bo Li, and Peng Cui. 2022. Distributionally robust optimization with data geometry.Advances in neural information processing systems35 (2022), 33689–33701

    2022

  20. [20]

    Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Yuguang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li, et al. 2025. Rethinking machine unlearning for large language models.Nature Machine Intelligence(2025), 1–14

    2025

  21. [21]

    Haokai Ma, Javier Yong, Yunshan Ma, Chen Kuei, Anis Yusof, Zhenkai Liang, and Ee-Chien Chang. 2025. AttackSeqBench: Benchmarking Large Language Models in Analyzing Attack Sequences within Cyber Threat Intelligence. (2025)

    2025

  22. [22]

    Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary Chase Lipton, and J Zico Kolter. 2024. TOFU: A Task of Fictitious Unlearning for LLMs. InFirst Conference on Language Modeling

    2024

  23. [23]

    Anmol Reddy Mekala, Vineeth Dorna, Shreya Dubey, Abhishek Lalwani, David Koleczek, Mukund Rungta, Sadid A Hasan, and Elita AA Lobo. 2025. Alternate Preference Optimization for Unlearning Factual Knowledge in Large Language Models. InProceedings of the 31st International Conference on Computational Linguistics. 3732–3752

    2025

  24. [24]

    Haowen Pan, Yixin Cao, Xiaozhi Wang, Xun Yang, and Meng Wang. 2024. Finding and editing multi-modal neurons in pre-trained transformers. InFindings of the Association for Computational Linguistics: ACL 2024. 1012–1037

    2024

  25. [25]

    Haowen Pan, Xiaozhi Wang, Yixin Cao, Zenglin Shi, Xun Yang, Juanzi Li, and Meng Wang. 2025. Precise Localization of Memories: A Fine-grained Neuron- level Knowledge Editing Technique for LLMs.arXiv preprint arXiv:2503.01090 (2025)

    Pith/arXiv arXiv 2025

  26. [26]

    Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. 2025. Safety Alignment Should be Made More Than Just a Few Tokens Deep. InThe Thirteenth International Conference on Learning Representations

    2025

  27. [27]

    Wei Qin, Zetong Chen, Xun Yang, Lei Wang, Yunshi Lan, Weijieying Ren, and Richang Hong. 2025. Explainable and Interactive LLMs-Augmented Depression Detection in Social Media.IEEE Transactions on Computational Social Systems (2025)

    2025

  28. [28]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model.Advances in neural information processing systems36 (2023), 53728–53741

    2023

  29. [29]

    Fei Shen and Jinhui Tang. 2024. Imagpose: A unified conditional framework for pose-guided person generation.Advances in neural information processing systems37 (2024), 6246–6266

    2024

  30. [30]

    Fei Shen, Hu Ye, Jun Zhang, Cong Wang, Xiao Han, and Yang Wei. 2024. Ad- vancing Pose-Guided Image Synthesis with Progressive Conditional Diffusion Models. InThe Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=rHzapPnCgT

    2024

  31. [31]

    William F Shen, Xinchi Qiu, Meghdad Kurmanji, Alex Iacob, Lorenzo Sani, Yi- hong Chen, Nicola Cancedda, and Nicholas D Lane. 2025. LLM unlearning via neural activation redirection. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems

    2025

  32. [32]

    Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2023. Detecting pretraining data from large language models.arXiv preprint arXiv:2310.16789(2023)

    Pith/arXiv arXiv 2023

  33. [33]

    Smith, and Chiyuan Zhang

    Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtz- man, Daogao Liu, Luke Zettlemoyer, Noah A. Smith, and Chiyuan Zhang. 2025. MUSE: Machine Unlearning Six-Way Evaluation for Language Models. InThe Thirteenth International Conference on Learning Representations

    2025

  34. [34]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yas- mine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288(2023)

    Pith/arXiv arXiv 2023

  35. [35]

    Yaxuan Wang, Jiaheng Wei, Chris Yuhao Liu, Jinlong Pang, Quan Liu, Ankit Shah, Yujia Bao, Yang Liu, and Wei Wei. 2025. LLM Unlearning via Loss Adjustment with Only Forget Data. InThe Thirteenth International Conference on Learning Representations

    2025

  36. [36]

    Yaxuan Wang, Jiaheng Wei, Chris Yuhao Liu, Jinlong Pang, Quan Liu, Ankit Parag Shah, Yujia Bao, Yang Liu, and Wei Wei. 2024. Llm unlearning via loss adjustment with only forget data.arXiv preprint arXiv:2410.11143(2024)

    Pith/arXiv arXiv 2024

  37. [37]

    Zifan Wang, Yi Shen, Michael M Zavlanos, and Karl H Johansson. 2024. Outlier- robust distributionally robust optimization via unbalanced optimal transport. Advances in Neural Information Processing Systems37 (2024), 52189–52214

    2024

  38. [38]

    Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jiawei Chen, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. 2025. Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization. InThe Thirteenth International Conference on Learning Representations

    2025

  39. [39]

    Xiaobo Xia, Xiaofeng Liu, Jiale Liu, Kuai Fang, Lu Lu, Samet Oymak, William S Currie, and Tongliang Liu. 2025. Identifying Trustworthiness Challenges in Deep Learning Models for Continental-Scale Water Quality Prediction.arXiv preprint arXiv:2503.09947(2025)

    arXiv 2025

  40. [40]

    Hefei Xu, Le Wu, Chen Cheng, and Hao Liu. 2025. Multi-Value Alignment for LLMs via Value Decorrelation and Extrapolation.arXiv preprint arXiv:2511.17579 (2025)

    arXiv 2025

  41. [41]

    Yangyang Xu, Jinpeng Hu, Zhuoer Zhao, Zhangling Duan, Xiao Sun, and Xun Yang. 2025. MultiAgentESC: A LLM-based Multi-Agent Collaboration Framework for Emotional Support Conversation. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 4665–4681

    2025

  42. [42]

    Zaiyan Xu, Sushil Vemuri, Kishan Panaganti, Dileep Kalathil, Rahul Jain, and Deepak Ramachandran. 2025. Robust LLM Alignment via Distributionally Robust Direct Preference Optimization.arXiv preprint arXiv:2502.01930(2025)

    arXiv 2025

  43. [43]

    Yuwei Yan, Yu Shang, Qingbin Zeng, Yu Li, Keyu Zhao, Zhiheng Zheng, Xuefei Ning, Tianji Wu, Shengen Yan, Yu Wang, et al. 2025. Agentsociety challenge: Designing llm agents for user modeling and recommendation on web platforms. InCompanion Proceedings of the ACM on Web Conference 2025. 2963–2967

    2025

  44. [44]

    Kai Yang, Jianwei Huang, Yihong Wu, Xiaodong Wang, and Mung Chiang. 2014. Distributed robust optimization (DRO), part I: Framework and example.Opti- mization and Engineering15, 1 (2014), 35–67. WWW ’26, April 13–17, 2026, Dubai, United Arab Emirates Pengyang Shao et al

    2014

  45. [45]

    Puning Yang, Qizhou Wang, Zhuo Huang, Tongliang Liu, Chengqi Zhang, and Bo Han. 2025. Exploring Criteria of Loss Reweighting to Enhance LLM Unlearning. InForty-second International Conference on Machine Learning

    2025

  46. [46]

    Yuanshun Yao, Xiaojun Xu, and Yang Liu. 2024. Large language model unlearning. Advances in Neural Information Processing Systems37 (2024), 105425–105475

    2024

  47. [47]

    Shanshan Ye, Jie Lu, and Guangquan Zhang. 2025. Towards safe machine un- learning: A paradigm that mitigates performance degradation. InProceedings of the ACM on Web Conference 2025. 4635–4652

    2025

  48. [48]

    Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. 2018. Privacy risk in machine learning: Analyzing the connection to overfitting. In2018 IEEE 31st computer security foundations symposium (CSF). IEEE, 268–282

    2018

  49. [49]

    Miao Yu, Liang Lin, Guibin Zhang, Xinfeng Li, Junfeng Fang, Ningyu Zhang, Kun Wang, and Yang Wang. 2025. UniErase: Unlearning Token as a Universal Erasure Primitive for Language Models.arXiv preprint arXiv:2505.15674(2025)

    arXiv 2025

  50. [50]

    Zhen Zeng, Leijiang Gu, Xun Yang, Zhangling Duan, Zenglin Shi, and Meng Wang. 2025. Visual-oriented fine-grained knowledge editing for multimodal large language models. InProceedings of the IEEE/CVF International Conference on Computer Vision. 2491–2500

    2025

  51. [51]

    Naixin Zhai, Pengyang Shao, Binbin Zheng, Fei Shen, Long Bai, and Xun Yang

  52. [52]

    Jingyang Zhang, Jingwei Sun, Eric Yeats, Yang Ouyang, Martin Kuo, Jianyi Zhang, Hao Frank Yang, and Hai Li. 2025. Min-k%++: Improved baseline for pre-training data detection from large language models. InThe Thirteenth International Con- ference on Learning Representations

    2025

  53. [53]

    Lijun Zhang, Peng Zhao, Zhen-Hua Zhuang, Tianbao Yang, and Zhi-Hua Zhou

  54. [54]

    Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. 2024. Negative Preference Opti- mization: From Catastrophic Collapse to Effective Unlearning. InFirst Conference on Language Modeling

    2024

  55. [55]

    Kairan Zhao, Meghdad Kurmanji, George-Octavian Bărbulescu, Eleni Triantafil- lou, and Peter Triantafillou. 2024. What makes unlearning hard and what to do about it.Advances in Neural Information Processing Systems37 (2024), 12293– 12333

    2024

  56. [56]

    Mengnan Zhao, Lihe Zhang, Xingyi Yang, Tianhang Zheng, and Baocai Yin. 2024. Advanchor: Enhancing diffusion model unlearning with adversarial anchors. arXiv preprint arXiv:2501.00054(2024)

    Pith/arXiv arXiv 2024

  57. [57]

    Mengnan Zhao, Lihe Zhang, Tianhang Zheng, Yuqiu Kong, and Baocai Yin

  58. [58]

    Xuanning Zhou, Hao Zeng, Xiaobo Xia, Bingyi Jing, and Hongxin Wei. 2025. Semi-Supervised Conformal Prediction With Unlabeled Nonconformity Score. arXiv preprint arXiv:2505.21147(2025)

    arXiv 2025

  59. [59]

    Yuchen Zhou, Jiayu Tang, Shuo Yang, Xiaoyan Xiao, Yuqin Dai, Wenhao Yang, Chao Gou, Xiaobo Xia, and Tat-Seng Chua. 2025. Logic unseen: Revealing the logical blindspots of vision-language models.arXiv preprint arXiv:2508.11317 (2025)

    Pith/arXiv arXiv 2025

  60. [60]

    International Medal for Outstanding Discoveries in Earth Sciences

    Mingye Zhu, Yi Liu, Zheren Fu, Yongdong Zhang, and Zhendong Mao. 2025. Leveraging Robust Optimization for LLM Alignment under Distribution Shifts. arXiv preprint arXiv:2504.05831(2025). A Derivation of the Closed-form Solution of𝑄 ★ 𝑓 We now provide a detailed derivation of the optimal adversarial distribution𝑄★ 𝑓 used in the main text. Each transformatio...

    arXiv 2025

  61. [62]

    Separable multi-concept erasure from diffusion models.arXiv preprint arXiv:2402.05947(2024)

    Pith/arXiv arXiv 2024

  62. [2023]

    Stochastic approximation approaches to group distributionally robust optimization.Advances in Neural Information Processing Systems36 (2023), 52490– 52522

    2023

  63. [2024]

    InProceedings of the AAAI Conference on Artificial Intelligence, Vol

    Temporally and distributionally robust optimization for cold-start recom- mendation. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 8750–8758

  64. [2025]

    Debate over Mixed-knowledge: A Robust Multi-Agent Reasoning Frame- work for Incomplete Knowledge Graph Question Answering.arXiv preprint arXiv:2511.12208(2025)

    arXiv 2025

  65. [2026]

    Maximizing Local Entropy Where It Matters: Prefix-Aware Localized LLM Unlearning.arXiv preprint arXiv:2601.03190(2026)

    Pith/arXiv arXiv 2026