Pith. sign in

REVIEW 2 major objections 6 minor 67 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

Choosing training data is already alignment, not just efficiency

2026-07-09 21:27 UTC pith:DBSHU275

load-bearing objection Solid core finding with fixable gaps the 2 major comments →

arxiv 2607.07023 v1 pith:DBSHU275 submitted 2026-07-08 cs.LG

Online Data Selection Is Implicit Alignment

Aoxiong Zeng , Yuxin Yang , Xiangquan Yang This is my paper
classification cs.LG
keywords onlinedataalignmentbehavioralselectiondriftimplicitmodel
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that when you filter which examples to train a language model on during supervised fine-tuning, the selection rule acts as an implicit reward model that shifts the model's behavior along axes normally associated with alignment: refusal rate, verbosity, sycophancy, truthfulness, and jailbreak robustness. The authors formalize online data selection as a reweighting of the SFT objective, where the selector's scores define a tilted training distribution that over-represents certain response styles or safety postures. They show empirically that selectors producing statistically indistinguishable task accuracy can diverge by more than 8 points in harmful-refusal rate and over 3 points in benign over-refusal, with the direction of each shift predictable from which behavioral attributes the selected data enriches. A loss-based selector enriches premise-agreement examples and increases sycophancy; a quality-based selector enriches long answers and refusal markers, raising both helpfulness and over-refusal simultaneously. The paper introduces Alignment Drift Auditing, a protocol that measures this drift under equal token budgets, and Alignment-Aware Selection, a selector that constrains the behavioral attribute mixture of the chosen data to reduce drift while preserving data efficiency.

Core claim

Two SFT runs with identical base model, optimizer, token budget, and task accuracy can produce models with sharply different refusal rates, verbosity, sycophancy, and jailbreak robustness, solely because the online data selector over-represents certain behavioral attributes in the training subset. The direction and magnitude of this drift is predictable from the enrichment ratios of the selected data, formalized through a first-order bound linking the attribute mixture shift to behavioral change via the gradient-behavior Jacobian.

What carries the argument

The central object is the reweighted SFT objective, where an online selector assigns weights w_π(i) to candidate examples, inducing a tilted training distribution q_π. The alignment drift is formalized through Proposition 1, which gives a first-order bound: the behavioral change between two selectors equals the Jacobian mapping parameter updates to behavior, applied to the difference in their weighted gradient means, plus second-order remainder. Under a clustering assumption that gradients concentrate within behavioral attribute groups, this bound reduces to a function of the enrichment gap, the difference in attribute-group mass between the selected subset and the full pool, weighted by how

Load-bearing premise

The clustering assumption that training gradients concentrate within behavioral attribute groups, meaning examples sharing a style like verbosity or refusal format produce similar gradient directions. This bridges the abstract gradient-space theory to the empirical data-mixture diagnostics, but it is verified only on a subset of runs without full quantitative reporting across all attribute groups.

What would settle it

If two selectors with identical task accuracy and identical enrichment ratios on all measured behavioral attributes still produce significantly different behavioral drift, the claimed link from data-mixture diagnostics to behavioral outcomes would break, since the enrichment gap would no longer predict drift direction.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Any paper or system reporting online SFT data selection results should report behavioral drift metrics alongside task accuracy, since equal efficiency can mask divergent safety and style profiles.
  • Preference optimization stages in post-training pipelines are not starting from a neutral SFT base; the selected data has already moved the model toward or away from desired behavioral regions, so SFT selection and preference optimization should be co-designed.
  • The framing of 'data quality' as a neutral, behaviorally inert concept is insufficient; high-quality data carries a persona with specific verbosity, deference, and caution characteristics that become the model's default behavior.
  • Alignment-Aware Selection demonstrates that constraining the behavioral attribute mixture of selected data via a maximum mean discrepancy penalty can reduce drift by more than half relative to unconstrained selectors while preserving most task gains.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper argues that online data selection during SFT is not a neutral efficiency layer but an implicit alignment mechanism: different online selectors (loss-based, quality-based, diversity-based) induce directional behavioral drift along alignment-relevant axes (refusal rate, verbosity, sycophancy, jailbreak robustness) even when matched on task accuracy. The authors formalize selection as importance reweighting of the SFT objective (§3.1, Eqs. 4–7), derive a first-order drift bound (Proposition 1, Eq. 8), and connect it to data-mixture enrichment under a clustering assumption (Appendix A, Eq. 19). They introduce Alignment Drift Auditing (ADA), a controlled protocol for measuring behavioral drift under equal token budgets, and Alignment-Aware Selection (AAS), a diagnostic selector that constrains drift via an MMD penalty on attribute mixtures. Empirically, Table 1 shows that selectors within 0.8 points of task accuracy differ by >8 points in harmful-refusal rate; Table 2 shows enrichment ratios that track the observed drift directions; Tables 3–4 show budget trends and AAS ablations.

Significance. The conceptual reframing — that an online scorer occupies the role normally held by a reward model — is timely and practically important. The ADA protocol (matched budgets, paired seeds, eight-axis behavioral suite, enrichment diagnostics) is a genuine methodological contribution that could standardize how selection papers report side effects. The formal framework (selection as importance reweighting, first-order drift bound) is correct and provides useful vocabulary. The AAS selector is presented appropriately as a diagnostic rather than a production method. The paper ships falsifiable predictions (drift direction tracks enrichment) and a reproducible experimental grid (budget sweep, judge robustness in Appendix E, ablation in Table 4).

major comments (2)
  1. §5.6 and Appendix A: The clustering assumption A4 (within-group gradient concentration, bounded by τ) is the formal bridge between the abstract Proposition 1 and the empirical enrichment ratios (Eq. 19). The paper states this is 'verified empirically in the mechanistic diagnostics' (§3.1), but §5.6 only describes the diagnostics qualitatively for 'a subset of runs' and never reports numerical values of τ across attribute groups. Without quantifying τ, a reader cannot assess whether Eq. 19's bound is tight enough to be genuinely predictive or merely post-hoc consistent. The gradient alignment measure α(k)_π (Eq. 13) is defined but no actual values are reported in any table or figure. Reporting τ and α(k)_π for the attribute groups in Table 2 would substantially strengthen the mechanistic claim. This is load-bearing because the enrichment-to-drift link is the paper's stronger, more novel贡献
  2. §5.3 and Table 2: The predictability claim ('the direction of the shift is predictable from the attribute mixture of the selected data') is supported by enrichment-drift consistency across only four selectors (Loss, Quality, Diversity, AAS). With n=4 and no held-out validation — e.g., predicting drift direction for a new selector from its enrichment profile alone — 'predictable' overstates what is demonstrated. AAS's success (Tables 1, 4) provides indirect support (if enrichment didn't matter, constraining it shouldn't help), but AAS is evaluated within the same framework and metrics, functioning as a consistency check rather than independent validation. Adding one or two additional selectors with a priori enrichment predictions, or softening the claim to 'consistent with,' would address this without changing the paper's scope.
minor comments (6)
  1. Table 1: The 'Avg. tokens' column header is ambiguous — it is unclear whether this refers to selected-data tokens, output tokens at evaluation, or training tokens per example. Clarifying would help reproducibility.
  2. §4.1: The paper mentions a 'full-fine-tuning run to rule out an adapter artifact' but does not report its results in any table. A one-line summary (e.g., 'full FT confirms the same selector ordering on D_A') would suffice.
  3. Eq. (9): The coverage term c(S) and its submodular properties are referenced but c(S) is never explicitly defined. The greedy rule in Eq. (10) references λΔc(x|S), but the reader must infer the form of c from context.
  4. Figure 2 (right panel): The signed drift heatmap uses a color scale that is difficult to parse in grayscale; the values are readable but the visual encoding could be clearer.
  5. §3.2: The claim that quality and diversity scorers 'subsume the utility-style online selectors' (citing [15, 16]) is stated without justification. A brief argument for why utility-based selectors are instances of these families would help.
  6. Appendix B: The calibrated enrichment estimator (Eq. 20) is a nice detail, but the confusion rates ρ_fp, ρ_fn are reported only as a range (Cohen's κ 0.86–0.94). Reporting the actual confusion rates per attribute family would make the correction fully reproducible.

Circularity Check

0 steps flagged

No significant circularity found; derivation chain is self-contained with standard assumptions

full rationale

The paper's derivation chain is not circular. Proposition 1 (Eq. 8) is a standard first-order Taylor expansion of a behavior function b(θ) under gradient descent — the inputs (gradient means, Jacobian) are not defined in terms of the output (behavioral drift). The enrichment bound (Eq. 19) follows from the clustering assumption A4, which is an explicitly stated empirical assumption, not a definition that forces the conclusion. The 'predictability' claim — that drift direction tracks attribute enrichment — is tested by comparing enrichment ratios (Table 2, computed from selected data) against behavioral metrics (Table 1, measured on held-out benchmarks like TruthfulQA and HarmBench). These are independent measurements; the correlation between them is an empirical observation, not a tautology. AAS (Eq. 9-10) uses an MMD penalty to match attribute distributions, and its reduced drift (Table 1) is measured on the same independent behavioral benchmarks — this is a legitimate test of the theory, not circular reasoning. Self-citations ([26], [51], [52], [57], [58], [61]) appear in Related Work and Limitations for context, none are load-bearing for the central claim. The score of 2 reflects minor self-citation presence that is not load-bearing; the central derivation is parameter-free and independently testable.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 2 invented entities

The axiom ledger is lean. The paper introduces no new physical entities or ad-hoc mathematical objects. The free parameters (lambda, rho, w_j) are standard hyperparameters of the AAS method, not fitted constants of a theory. The clustering assumption A4 is the most consequential axiom and is only partially verified.

free parameters (3)
  • lambda (coverage weight in AAS) = not reported
    Weight on the coverage term c(S) in the AAS objective (Eq. 9). Value not specified in the paper.
  • rho (MMD penalty strength in AAS) = not reported
    Controls the trade-off between utility selection and attribute matching in AAS (Eq. 9). Value not specified; the paper says it interpolates between pure utility (rho=0) and attribute-matched (rho→∞).
  • w_j (safety-critical axis weights) = default 1
    Optional weights in the drift magnitude DA (Eq. 12) that upweight safety-critical axes. Default is w_j≡1 but the paper does not report what values were used.
axioms (4)
  • domain assumption Clustering assumption (A4): within-group gradient concentration, gi = ḡ(k) + εi with bounded ||Σεi/N|| ≤ τ
    Invoked in Appendix A to derive the enrichment bound (Eq. 19) that connects data-mixture diagnostics to behavioral drift. This is the load-bearing assumption for the paper's central bridge between selected-data attributes and model behavior.
  • domain assumption Behavior vector b(θ) is a smooth functional of parameters with bounded Jacobian (A2: β-smooth)
    Invoked in Appendix A for the first-order Taylor expansion in Prop. 1. Standard for neural networks but the smoothness constant β is not estimated.
  • domain assumption SFT loss is twice differentiable with bounded Hessian (A1: ||∇²ℓ|| ≤ L)
    Invoked in Appendix A. Standard assumption for gradient-based optimization analysis.
  • domain assumption LLM judges provide reliable behavioral evaluations
    The behavioral metrics (helpfulness, sycophancy, truthfulness) rely on LLM judges. The paper acknowledges this is imperfect (§Limitations) and provides a robustness check (Table 5) showing stable rankings, but absolute metric validity is assumed.
invented entities (2)
  • Alignment Drift Auditing (ADA) independent evidence
    purpose: Protocol for quantifying selection-induced behavioral movement across 8 axes under matched budgets
    ADA is a methodological framework, not a physical entity. Its validity is tested by the experiments in §5 showing measurable drift. It is falsifiable: if selectors produced no behavioral differences, ADA would show zero drift.
  • Alignment-Aware Selection (AAS) independent evidence
    purpose: Diagnostic online selector that constrains drift via MMD penalty on behavioral attributes
    AAS is a concrete algorithm (Eq. 9-10) with a greedy selection rule. Its effectiveness is tested in Table 1 and Table 4, showing reduced drift while preserving task performance. Falsifiable: if AAS did not reduce drift, the method would be invalid.

pith-pipeline@v1.1.0-glm · 21866 in / 2797 out tokens · 245696 ms · 2026-07-09T21:27:33.019742+00:00 · methodology

0 comments
read the original abstract

Supervised fine-tuning (SFT) is often treated as a capability-adaptation step, while alignment is attributed to later preference optimization or reinforcement learning. This separation is incomplete: when examples are scored and kept online during fine-tuning, the choice of which data to train on already changes the model's behavioral preferences. We study online data selection as an implicit alignment mechanism. Given the same base model, optimizer, and selected-token budget, we compare random, loss-based, quality-based, and diversity-based online selectors and measure the behavioral drift they induce without any preference optimization. The proposed evaluation tracks helpfulness, refusal rate, verbosity, truthfulness, sycophancy, calibration, and jailbreak robustness, together with diagnostics for which behavioral modes are over-represented in the selected data. We formalize online selection as a reweighted SFT objective whose weights define an implicit preference over response styles and safety postures, so that an online scorer plays the role usually assigned to a reward model. This view predicts that high-scoring data can systematically favor longer, more assertive, more compliant, or more refusal-prone behaviors depending on how the online score is defined. Empirically, selectors that are statistically indistinguishable in task accuracy diverge sharply in refusal rate, verbosity, and sycophancy, and we show that the direction of the shift is predictable from the attribute mixture of the selected data. We introduce Alignment Drift Auditing (ADA), a controlled protocol for quantifying selection-induced behavioral movement, and Alignment-Aware Selection (AAS), a diagnostic online selector that retains data efficiency while constraining drift along safety and style axes.

Figures

Figures reproduced from arXiv: 2607.07023 by Aoxiong Zeng, Xiangquan Yang, Yuxin Yang.

Figure 1
Figure 1. Figure 1: Data selection changes the empirical SFT ob [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Result visualizations. Left: task/helpfulness improvements and alignment drift are not monotonically [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Enrichment plot. High-utility and high-quality [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Budget sweep. Online selection pressure is [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Safety-boundary geometry. Off-diagonal movement indicates a conditional rather than global change in refusal behavior. AAS strengthens harmful refusal with little benign over-refusal. harmful refusal) while still raising benign refusal, a particularly undesirable combination. AAS lands near the harmful axis with little benign movement, i.e., it strengthens the boundary where it should without becoming glob… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

67 extracted references · 67 canonical work pages · 21 internal anchors

  1. [1]

    Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in Neural Informa- tion Processing Systems, 2022

    work page 2022

  2. [2]

    Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  3. [3]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Ste- fano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. InAdvances in Neural Information Processing Systems, 2023

    work page 2023

  4. [4]

    Fine-Tuning Language Models from Human Preferences

    Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning lan- guage models from human preferences.arXiv preprint arXiv:1909.08593, 2019

    work page internal anchor Pith review Pith/arXiv arXiv 1909

  5. [5]

    Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul F

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul F. Christiano. Learning to summarize with human feedback. InAdvances in Neural Information Processing Systems, 2020

    work page 2020

  6. [6]

    Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M

    Jason Wei, Maarten Bosma, Vincent Y . Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M. Dai, and Quoc V . Le. Finetuned language models are zero-shot learners. InInternational Con- ference on Learning Representations, 2022

    work page 2022

  7. [7]

    Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al

    Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. InInternational Conference on Learning Representations, 2022

    work page 2022

  8. [8]

    Self-Instruct: Aligning Language Models with Self-Generated Instructions

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning lan- guage models with self-generated instructions.arXiv preprint arXiv:2212.10560, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  9. [9]

    WizardLM: Empowering large pre-trained language models to follow complex instructions

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardLM: Empowering large language mod- els to follow complex instructions.arXiv preprint arXiv:2304.12244, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  10. [10]

    OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization

    Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pa- sunuru, Todor Mihaylov, Dima Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, et al. OPT-IML: Scaling language model instruc- tion meta learning through the lens of generalization. arXiv preprint arXiv:2212.12017, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  11. [11]

    A Survey on Data Selection for Language Models

    Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haeju Jeong, et al. A survey on data selection for language models.arXiv preprint arXiv:2402.16827, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  12. [12]

    AlpaGasus: Training a better alpaca with fewer data

    Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, and Hongxia Jin. AlpaGasus: Training a better alpaca with fewer data. InInternational Conference on Learning Rep- resentations, 2024

    work page 2024

  13. [13]

    LIMA: Less is more for align- ment

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, 10 Ping Yu, Lili Yu, et al. LIMA: Less is more for align- ment. InAdvances in Neural Information Processing Systems, 2023

    work page 2023

  14. [14]

    LESS: Select- ing influential data for targeted instruction tuning

    Mengzhou Xia, Sadhika Malladi, Suchin Gururan- gan, Sanjeev Arora, and Danqi Chen. LESS: Select- ing influential data for targeted instruction tuning. InInternational Conference on Machine Learning, 2024

    work page 2024

  15. [15]

    Wang, Tong Wu, Dawn Song, Prateek Mittal, and Ruoxi Jia

    Jiachen T. Wang, Tong Wu, Dawn Song, Prateek Mittal, and Ruoxi Jia. GREATS: Online selection of high-quality data for LLM training in every itera- tion. InAdvances in Neural Information Processing Systems, 2024

    work page 2024

  16. [16]

    Utility-diversity aware online batch se- lection for llm supervised fine-tuning.arXiv preprint arXiv:2510.16882, 2025

    Heming Zou, Yixiu Mao, Yun Qu, Qi Wang, and Xiangyang Ji. Utility-diversity aware online batch se- lection for llm supervised fine-tuning.arXiv preprint arXiv:2510.16882, 2025

    work page arXiv 2025

  17. [17]

    Textbooks Are All You Need

    Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need.arXiv preprint arXiv:2306.11644, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  18. [18]

    Data selection for language mod- els via importance resampling.Advances in Neural Information Processing Systems, 2023

    Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. Data selection for language mod- els via importance resampling.Advances in Neural Information Processing Systems, 2023

    work page 2023

  19. [19]

    A General Language Assistant as a Laboratory for Alignment

    Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for align- ment.arXiv preprint arXiv:2112.00861, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  20. [20]

    The Capacity for Moral Self-Correction in Large Language Models

    Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas I. Liao, Kamil ˙e Lukoši ¯ut˙e, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, et al. The capacity for moral self- correction in large language models.arXiv preprint arXiv:2302.07459, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  21. [21]

    Concrete Problems in AI Safety

    Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety.arXiv preprint arXiv:1606.06565, 2016

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  22. [22]

    Understanding black-box predictions via influence functions

    Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. InIn- ternational Conference on Machine Learning, 2017

    work page 2017

  23. [23]

    Estimating training data in- fluence by tracing gradient descent

    Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data in- fluence by tracing gradient descent. InAdvances in Neural Information Processing Systems, 2020

    work page 2020

  24. [24]

    Deep learning on a data diet: Finding important examples early in training

    Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. InAdvances in Neural Information Processing Systems, 2021

    work page 2021

  25. [25]

    Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S. Morcos. Beyond neural scaling laws: Beating power law scaling via data pruning. InAdvances in Neural Information Process- ing Systems, 2022

    work page 2022

  26. [26]

    The Long-Term Effects of Data Selection in LLM Fine-Tuning

    Yuxin Yang, Aoxiong Zeng, and Xiangquan Yang. The long-term effects of data selection in LLM fine- tuning.arXiv preprint arXiv:2605.30537, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  27. [27]

    Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei

    Paul F. Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep re- inforcement learning from human preferences. In Advances in Neural Information Processing Systems, 2017

    work page 2017

  28. [28]

    RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback

    Harrison Lee, Sang Michael Phatale, Hassan Man- soor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, and Ab- hinav Rastogi. Rlaif: Scaling reinforcement learn- ing from human feedback with ai feedback.arXiv preprint arXiv:2309.00267, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  29. [29]

    Kto: Model alignment as prospect theoretic optimization

    Kawin Ethayarajh, Winnie Xu, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. InInternational Conference on Machine Learning, 2024

    work page 2024

  30. [30]

    Scal- ing laws for reward model overoptimization.Pro- ceedings of the 40th International Conference on Machine Learning, 2023

    Leo Gao, John Schulman, and Jacob Hilton. Scal- ing laws for reward model overoptimization.Pro- ceedings of the 40th International Conference on Machine Learning, 2023

    work page 2023

  31. [31]

    Open problems and fun- damental limitations of reinforcement learning from human feedback.Transactions on Machine Learning Research, 2023

    Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fun- damental limitations of reinforcement learning from human feedback.Transactions on Machine Learning Research, 2023

    work page 2023

  32. [32]

    AI Alignment: A Comprehensive Survey

    Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al. Ai alignment: A comprehensive survey.arXiv preprint arXiv:2310.19852, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  33. [33]

    TruthfulQA: Measuring how models mimic human falsehoods

    Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. InProceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics, 2022

    work page 2022

  34. [34]

    Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned

    Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to re- duce harms: Methods, scaling behaviors, and lessons learned.arXiv preprint arXiv:2209.07858, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  35. [35]

    Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. InFindings of the Association for Com- putational Linguistics: EMNLP, 2020. 11

    work page 2020

  36. [36]

    Discovering language model behaviors with model-written evaluations

    Ethan Perez, Sam Ringer, Kamil ˙e Lukoši¯ut˙e, Ka- rina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kada- vath, et al. Discovering language model behaviors with model-written evaluations. InFindings of the As- sociation for Computational Linguistics: ACL, 2023

    work page 2023

  37. [37]

    Jailbroken: How does LLM safety training fail? InAdvances in Neural Information Processing Systems, 2023

    Alexander Wei, Nika Haghtalab, and Jacob Stein- hardt. Jailbroken: How does LLM safety training fail? InAdvances in Neural Information Processing Systems, 2023

    work page 2023

  38. [38]

    Harmbench: A standardized evaluation framework for automated red teaming and robust refusal

    Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, et al. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. InInternational Con- ference on Machine Learning, 2024

    work page 2024

  39. [39]

    On faithfulness and factuality in abstractive summarization

    Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. InProceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, 2020

    work page 2020

  40. [40]

    Weinberger

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural net- works. InInternational Conference on Machine Learning, 2017

    work page 2017

  41. [41]

    Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell

    Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language mod- els be too big? InProceedings of the 2021 ACM Conference on Fairness, Accountability, and Trans- parency, 2021

    work page 2021

  42. [42]

    The woman worked as a babysitter: On biases in language generation

    Emily Sheng, Kai-Wei Chang, Premkumar Natara- jan, and Nanyun Peng. The woman worked as a babysitter: On biases in language generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, 2019

    work page 2019

  43. [43]

    Energy-based Automated Model Evaluation

    Ru Peng, Heming Zou, Haobo Wang, Yawen Zeng, Zenan Huang, and Junbo Zhao. Energy- based automated model evaluation.arXiv preprint arXiv:2401.12689, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  44. [44]

    Parameter-efficient transfer learning for NLP

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzeb- ski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. InIn- ternational Conference on Machine Learning, 2019

    work page 2019

  45. [45]

    Hu, Yelong Shen, Phillip Wallis, Yizhi Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Yizhi Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. InInternational Conference on Learning Representations, 2022

    work page 2022

  46. [46]

    QLoRA: Efficient finetuning of quantized LLMs

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetuning of quantized LLMs. InAdvances in Neural Informa- tion Processing Systems, 2023

    work page 2023

  47. [47]

    LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugin

    Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Jun Zhao, Wei Shen, Yuhao Zhou, Zhiheng Xi, Xiao Wang, Xiaoran Fan, et al. LoRAMoE: Al- leviate world knowledge forgetting in large lan- guage models via MoE-style plugin.arXiv preprint arXiv:2312.09979, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  48. [48]

    MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts

    Dengchun Li, Yingzi Ma, Naizheng Wang, Zheng- mao Ye, Zhiyuan Cheng, Yinghao Tang, Yan Zhang, Lei Duan, Jie Zuo, Cal Yang, et al. MixLoRA: Enhancing large language models fine-tuning with LoRA-based mixture of experts.arXiv preprint arXiv:2404.15159, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  49. [49]

    Mixture of LoRA experts

    Xun Wu, Shaohan Huang, and Furu Wei. Mixture of LoRA experts. InInternational Conference on Learning Representations, 2024

    work page 2024

  50. [50]

    HydraLoRA: An asymmetric LoRA architecture for efficient fine-tuning.Ad- vances in Neural Information Processing Systems, 2024

    Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Cheng-Zhong Xu. HydraLoRA: An asymmetric LoRA architecture for efficient fine-tuning.Ad- vances in Neural Information Processing Systems, 2024

    work page 2024

  51. [51]

    Towards specialized generalists: A multi-task MoE- LoRA framework for domain-specific LLM adapta- tion.arXiv preprint arXiv:2601.07935, 2026

    Yuxin Yang, Aoxiong Zeng, and Xiangquan Yang. Towards specialized generalists: A multi-task MoE- LoRA framework for domain-specific LLM adapta- tion.arXiv preprint arXiv:2601.07935, 2026

    work page arXiv 2026

  52. [52]

    FlyLoRA: Boosting task decou- pling and parameter efficiency via implicit rank-wise mixture-of-experts.Advances in Neural Information Processing Systems, 38:10386–10419, 2026

    Heming Zou, Yunliang Zang, Wutong Xu, Yao Zhu, and Xiangyang Ji. FlyLoRA: Boosting task decou- pling and parameter efficiency via implicit rank-wise mixture-of-experts.Advances in Neural Information Processing Systems, 38:10386–10419, 2026

    work page 2026

  53. [53]

    Rusu, Kieran Milan, John Quan, Tiago Ramalho, Ag- nieszka Grabska-Barwinska, et al

    James Kirkpatrick, Razvan Pascanu, Neil Rabi- nowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Ag- nieszka Grabska-Barwinska, et al. Overcoming catas- trophic forgetting in neural networks.Proceedings of the National Academy of Sciences, 2017

    work page 2017

  54. [54]

    Learning without forgetting.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017

    Zhizhong Li and Derek Hoiem. Learning without forgetting.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017

    work page 2017

  55. [55]

    Fine-tuning can dis- tort pretrained features and underperform out-of- distribution

    Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can dis- tort pretrained features and underperform out-of- distribution. InInternational Conference on Learn- ing Representations, 2022

    work page 2022

  56. [56]

    Fernando Hernandez-Garcia, Qingfeng Lan, Parash Rahman, A

    Shibhansh Dohare, J. Fernando Hernandez-Garcia, Qingfeng Lan, Parash Rahman, A. Rupam Mahmood, and Richard S. Sutton. Loss of plasticity in deep continual learning.Nature, 632:768–774, 2024

    work page 2024

  57. [57]

    Heming Zou, Yunliang Zang, Wutong Xu, and Xi- angyang Ji. Fly-CL: A fly-inspired framework for enhancing efficient decorrelation and reduced train- ing time in pre-trained model-based continual repre- sentation learning.arXiv preprint arXiv:2510.16877, 2025. 12

    work page arXiv 2025

  58. [58]

    Representation Collapse in Sequential Post-Training of Large Language Models

    Yichen Liu, Mingyu Chen, Hao Wang, Xiaoran Xu, Chenxi Lin, Rui Zhang, Yutong Zhou, Yuxin Yang, Jiarui Wu, and Wei Sun. Representation collapse in sequential post-training of large language models. arXiv preprint arXiv:2605.30524, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  59. [59]

    Orthogonal subspace learning for lan- guage model continual learning

    Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xu- anjing Huang. Orthogonal subspace learning for lan- guage model continual learning. InFindings of the Association for Computational Linguistics: EMNLP, 2023

    work page 2023

  60. [60]

    Decouple and Orthogonalize: A Data-Free Framework for LoRA Merging

    Shenghe Zheng, Hongzhi Wang, Chenyu Huang, Xiaohui Wang, Tao Chen, Jiayuan Fan, Shuyue Hu, and Peng Ye. Decouple and orthogonalize: A data- free framework for LoRA merging.arXiv preprint arXiv:2505.15875, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  61. [61]

    Disentangling task con- flicts in multi-task LoRA via orthogonal gradient projection.arXiv preprint arXiv:2601.09684, 2026

    Ziyu Yang, Guibin Chen, Yuxin Yang, Aoxiong Zeng, and Xiangquan Yang. Disentangling task con- flicts in multi-task LoRA via orthogonal gradient projection.arXiv preprint arXiv:2601.09684, 2026

    work page arXiv 2026

  62. [62]

    Online Batch Selection for Faster Training of Neural Networks

    Ilya Loshchilov and Frank Hutter. Online batch selection for faster training of neural networks.arXiv preprint arXiv:1511.06343, 2015

    work page internal anchor Pith review Pith/arXiv arXiv 2015

  63. [63]

    Accelerating Deep Learning by Focusing on the Biggest Losers

    Angela H. Jiang, Daniel L. Wong, Giulio Zhou, David G. Andersen, Jeffrey Dean, Gregory R. Ganger, Gauri Joshi, Michael Kaminsky, Michael Kozuch, Zachary C. Lipton, et al. Accelerating deep learning by focusing on the biggest losers.arXiv preprint arXiv:1910.00762, 2019

    work page internal anchor Pith review Pith/arXiv arXiv 1910

  64. [64]

    Brauner, Muhammed Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N

    Soren Mindermann, Jan M. Brauner, Muhammed Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N. Gomez, Adrien Morisot, Sebastian Farquhar, and Yarin Gal. Pri- oritized training on points that are learnable, worth learning, and not yet learnt. InInternational Confer- ence on Machine Learning, 2022

    work page 2022

  65. [65]

    Active learning for convolutional neural networks: A core-set ap- proach

    Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set ap- proach. InInternational Conference on Learning Representations, 2018

    work page 2018

  66. [66]

    Ash, Chicheng Zhang, Akshay Krishna- murthy, John Langford, and Alekh Agarwal

    Jordan T. Ash, Chicheng Zhang, Akshay Krishna- murthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. InInternational Conference on Learn- ing Representations, 2020

    work page 2020

  67. [67]

    further from the center is better

    Kushal Tirumala, Dima Simig, Armen Aghajanyan, and Ari S. Morcos. D4: Improving LLM pretrain- ing via document deduplication and diversification. Advances in Neural Information Processing Systems, 2023. A Proof of Proposition 1 and an Enrichment Bound Setup and assumptions.We treat the behavior vector as a smooth functional b(θ)∈R 8 of the parameters and ...

    work page 2023