Pith. sign in

REVIEW 3 major objections 5 minor 91 references

This paper proves that selecting SFT data with a bilevel validation-guided objective is strictly better than mixing all data, and extends the idea to online self-refining with importance-weighted responses.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 20:05 UTC pith:K2VOK7PA

load-bearing objection Offline BDS/BMO theory is a real but conditional extension; the online importance-weight justification is mathematically wrong and the abstract overclaims a convergence result that isn't in the paper. the 3 major comments →

arxiv 2511.21056 v2 pith:K2VOK7PA submitted 2025-11-26 cs.LG cs.CLmath.OC

Bilevel Data Curation for LLM Fine-tuning: Offline Selection and Online Self-Refining Generation

Quan Xiao , Yutong Xuan , Gaowen Liu , Ramana Rao Kompella , Tianyi Chen This is my paper
classification cs.LG cs.CLmath.OC
keywords bilevel optimizationdata selectionLLM fine-tuningself-refining generationimportance samplingmulti-objective optimizationsupervised fine-tuningsafety-aware fine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to show that for fine-tuning large language models, choosing which supervised fine-tuning (SFT) samples to train on — guided by a small, high-quality validation set — is provably better than just pooling the low-quality SFT data with the validation data. It does this by unifying offline data selection and online self-refining generation under one bilevel optimization view, where each data sample is a separate objective. The central result is that the bilevel selection problem is equivalent to a bilevel multi-objective problem, and its optimal solution assigns zero weight to samples that are useless with respect to the validation set, keeping only useful ones. From this, the model trained on the selected data strictly improves validation loss over direct data mixing. The paper then extends this to an online setting where responses are regenerated and reused via importance sampling, showing the importance ratio serves as a response-level validation weight.

Core claim

The paper establishes that bilevel data selection (BDS) and bilevel multi-objective optimization (BMO) are equivalent under a separability assumption: any solution of one is a solution of the other (Theorem 1). Defining a sample as useful if its individual minimizer set intersects the validation minimizer set, the paper proves that BDS removes all useless samples and keeps only useful ones (Theorem 2), and as a result the optimal model strictly beats any direct mixing of the low-quality SFT dataset with validation data on validation loss (Theorem 3). For the online extension, the paper shows that the importance ratio for reusing old-policy responses is proportional to the softmax of negative

What carries the argument

The proof hinges on two insights: (1) the per-sample SFT loss is convex with respect to the backbone representation of the model, even though it is nonconvex in the parameters, which makes the sum-of-weights lower-level problem tractable; and (2) under individual separability, the weak Pareto front of the multi-objective loss equals the union of the lower-level solution sets over all simplex weights (Theorem 1). This lets the authors define useful versus useless samples via individual minimizers and then show that BDS zeroes out useless samples. The online extension relies on the softmax-of-negative-loss identity to interpret the importance ratio as a validation-aligned response weight.

Load-bearing premise

The proofs assume there exists a model that fits every low-quality SFT sample perfectly and another that fits every validation sample perfectly; if the two datasets are not each exactly separable, the strict-improvement results have no guarantee.

What would settle it

Train a model on a small dataset where one low-quality sample conflicts with the validation set (no joint zero-loss solution) but each set is individually separable. If direct mixing achieves lower validation loss than bilevel data selection, Theorem 3 is false. A simpler check: verify whether the trained LoRA models achieve exactly zero loss on all SFT and validation samples; if not, Assumption 1 is violated.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If Theorem 3 holds, practitioners can expect lower validation loss by selecting rather than pooling low-quality SFT data, even when the low-quality set is large.
  • The equivalence of BDS and BMO means the same validation-aligned data weights can be computed either explicitly (offline) or implicitly through multi-objective optimization, giving two algorithmic routes to the same solution.
  • The importance-ratio response weight enables principled reuse of old-policy generations in online self-refining, reducing generation cost without discarding the question.
  • The observed simple-to-hard learning progression suggests that the method implicitly builds a curriculum, which may improve generalization beyond what static selection achieves.
  • Safety-aware fine-tuning can benefit because harmful responses are down-weighted or refined rather than propagated, improving performance on unsafe data while keeping question diversity.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The equivalence argument may extend to any loss that is convex in a fixed feature map, suggesting that similar selection guarantees could apply to preference optimization or token-level curation, though the paper only proves the SFT case.
  • The likelihood-based importance weight is a general recipe for off-policy self-training: any iterative refinement loop that reuses old samples can weight them by their probability under the current policy, which is a cheap way to align with validation data.
  • The empirically observed simple-to-hard ordering could be developed into an automatic curriculum for LLMs, or used as a way to identify question difficulty without manual labels.
  • If the separability assumption fails, the strict improvement may become approximate; a practical test is to measure the gap between the selected model's validation loss and the direct-mixing minimum on datasets with varying degrees of conflict.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript studies bilevel data selection (BDS) and bilevel multi-objective optimization (BMO) for LLM fine-tuning. It derives an equivalence between BDS and BMO under an exact-separability assumption, defines useful/useless samples via individual minimizers, and proves that BDS can discard useless samples and achieve strictly lower validation loss than direct mixing (Theorems 1–4). For online self-refining generation, the paper proposes an importance-sampling response weight and claims it is proportional to the implicit BMO weight (Lemma 6). Experiments on quality enhancement and safety fine-tuning compare online/offline selection with direct mixing. The offline theory is internally consistent under Assumption 1, but the online-weighting claim is incorrect and the advertised global-convergence result is absent from the manuscript.

Significance. If the offline theory held as stated, the paper would provide a useful theoretical justification for validation-guided data selection. It offers formal statements and proof sketches for the equivalence and selection properties, and it includes extensive experiments on two LLM families and two tasks. However, the online IS/BMO-weight proportionality is wrong, and the abstract's promised global convergence analysis for a one-layer Transformer does not appear anywhere in the main text or appendices. These are load-bearing issues affecting central advertised contributions; the empirical results alone do not compensate for the invalid theoretical claim.

major comments (3)
  1. [§4.3, Lemma 6, Algorithm 1 (lines 10–12), Eq. (8)] The claim that r_g = π_θ(y_i,g^old)/π_old(y_i,g^old) is proportional to the BMO weight λ_i,g ∝ π_θ(y_i,g^old) is false. The factor π_old(y_i,g^old) is not a common scaling constant across g; it varies with each generated response. For example, with (π_θ, π_old) = (0.5,0.01) and (0.6,0.99) for two responses, the BMO weights rank the second response higher, while r ranks the first higher. Since Algorithm 1 uses r_g directly as the response-level weight, the algorithm does not in fact assign the validation-aligned BMO weight. Additionally, if the intention is to correct an expectation over the current policy, r_g∇L_SFT omits the score-function term from differentiating the sampling distribution, so it is not an unbiased gradient of E_{π_θ}L_SFT. This invalidates the stated theoretical foundation of the online method.
  2. [Abstract vs. full text (Appendices A–F)] The arXiv abstract states: 'Moreover, we provide a global convergence analysis for gradient-based BDS approach for one-layer Transformer, showing that the epsilon-global optimum of offline BDS is achievable in finite time.' This result does not appear in the main text or in any appendix. The only theoretical statements are Theorems 1–4 (equivalence, selection, loss comparison) and Lemma 6; Appendix E discusses the PBGD algorithm and a lemma on expected weight updates but contains no convergence theorem, let alone a one-layer Transformer global convergence result. This advertised contribution is unsupported and would need to be either provided or removed.
  3. [Assumption 1 (§3.1) and experimental setup (§5.1, §5.2, Appendix F.1)] All theoretical guarantees depend on exact separability: zero per-sample SFT loss on the low-quality set and zero validation loss. The experiments use LoRA rank 16 and report evaluation losses in the range 0.7–1.5, with no check that the zero-loss solution exists. The strict-improvement claims are therefore conditional on a strong assumption that the paper does not verify empirically. The manuscript should state this limitation explicitly, and ideally either test a setting where the assumption is plausible or provide a robustness analysis for the non-separable case.
minor comments (5)
  1. [Eq. (8), Algorithm 1] The condition in Eq. (8) is written 'if i_k /∈ Ik', which appears to be a typo. The surrounding text indicates the online branch applies when i_k ∈ I_M; the equation should be updated for consistency.
  2. [§4.3] Notation 'πθ(xi, yi,g old)' should be written with explicit conditioning, e.g., π_θ(y_i,g^old | x_i), to avoid confusion with joint probabilities.
  3. [References] References [21] and [22] are duplicate entries for the Llama 3 herd paper; one should be removed.
  4. [Appendix C.3, Theorem 3 proof] The proof is very terse. The step 'we can only add it to the validation dataset' needs expansion to explain why θ* for BDS is upper-bounded by the model trained on validation plus a single useful sample.
  5. [Tables 5, 6, 10] These tables report point estimates without standard errors or per-run variance, unlike Tables 1 and 3. Adding error bars or confidence intervals would strengthen the empirical comparison.

Circularity Check

2 steps flagged

The Section 3 'selection beats mixing' theorem is largely a restatement of the paper's Definition 3 of 'useful', and Section 4.3's equivalence between the importance ratio and the BMO response weight is asserted by treating a per-response denominator as a constant.

specific steps
  1. self definitional [Section 3.2, Definition 3; Theorem 3; Appendix C.3 proof]
    "We say that (x_i, y_i) is • useful if Θ_i ∩ Θ_val ≠ ∅; • useless otherwise, i.e., Θ_i ∩ Θ_val = ∅. ... First we have L0(θ∗) = 0. This is because there exists at least one useful data point (x_i, y_i) in the lower-level SFT dataset and we can only add it to the validation dataset."

    Definition 3 labels a sample 'useful' exactly when one of its SFT minimizers also minimizes the validation loss. The proof of Theorem 3 then obtains L0(θ*)=0 from the existence of such a sample, since BDS may place all weight on it; the failure of mixing follows from the separately assumed absence of a joint minimizer for the full SFT set. The claimed strict gain of BDS over direct mixing is therefore a direct restatement of the usefulness definition plus the non-joint-separability assumption, rather than an independently derived prediction.

  2. other [Section 4.3 (Importance ratio as response weight), Lemma 6 and Algorithm 1 lines 10–12]
    "Since L_SFT(θ; x_m, y_{i,g}^{old}) = −log π_θ(x_m, y_{i,g}^{old}) and the denominator of λ_{i,g} is same to each g-th response for i-th question, we have the importance ratio of each response r_g = π_θ(x_i, y_{i,g}^{old}) / π_old(x_i, y_{i,g}^{old}) ∝ π_θ(x_i, y_{i,g}^{old}) ∝ λ_{i,g} where we treat π_old(x_i, y_{i,g}^{old}) as the scaling factor."

    Lemma 6 defines λ_{i,g} as the softmax of per-response SFT losses under the current policy, hence proportional to π_θ(y_g|x_i). The IS ratio also contains the per-response denominator π_old(y_g|x_i), which varies across g. Calling it a 'scaling factor' is exactly the move that postulates r_g ∝ λ_g; without it r_g and λ_g are different functions of g. Algorithm 1 uses r_g as the response-level weight and the paper claims it recovers the BMO/validation-aligned weight. The online self-refining justification is thus an asserted identification, not a derived proportionality.

full rationale

Two reductions drive the paper's headline claims. First, Theorem 3's proof obtains L0(θ*)=0 directly from the existence of a 'useful' sample, and 'useful' is defined (Def. 3) as a sample whose individual SFT minimizer also minimizes L0; mixing is worse only because joint separability of the full SFT set with validation is explicitly not assumed. Thus the claimed strict superiority of BDS over direct mixing is a restatement of the definition plus an assumption, not a derivation from model properties; the genuinely nontrivial equivalence BMO=BDS does not rescue the comparative claim. Second, the online section defines the BMO response weight as softmax over current-policy SFT losses and then asserts the IS ratio is proportional to it by treating π_old as a constant scaling factor; since π_old differs per response, this is not a theorem and Algorithm 1's use of r_g as the validation-aligned weight is an asserted identification. These are central claims, so score 6. I do not weight the Assumption 1 fragility as circularity, and the self-citations ([58], [6], [74]) are not load-bearing: the core theorems are proved in the appendix from stated assumptions. The experiments are self-contained external benchmarks, but the theoretical 'benefits over direct mixing' and the online IS equivalence reduce by construction.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

The central theory rests on exact zero-loss separability of both the low-quality and validation datasets (Assumption 1), plus the existence of at least one 'useful' sample and distributional closeness for generalization. These are strong structural assumptions, not derived from external benchmarks. Experimental hyperparameters (R, G, γ_k schedule) are chosen by hand; no fitted physical constants are involved.

free parameters (3)
  • online ratio R = N_M/N = 10% (best in ablations; 20% hurts)
    Fraction of questions whose responses are regenerated online. Tuned on validation performance; Table 1 and Figure 4 show R=10% G=5 best for OpenOrca. This is a hand-chosen hyperparameter, not derived.
  • number of generated responses per question G = 5 for best quality (G=1 for runtime efficiency)
    Generating multiple candidates improves response quality but adds ~1x runtime; authors note 'G=1 is an ideal choice' in Section 5.1, yet best results use G=5. Hand-chosen.
  • penalty constant schedule gamma_k = rho_k/(1-rho_k) = rho_0=0.1, increment 0.1 per epoch
    Algorithm hyperparameter from [58], chosen by hand; affects the trade-off between upper-level and lower-level objectives.
axioms (3)
  • domain assumption Assumption 1 (Separable data): There exists θ with L_SFT(θ; x_i, y_i)=0 for all i in [N], and θ' with L_0(θ'; x̃_i, ỹ_i)=0 for all i in [N']
    Invoked in Section 3.1 (Assumption 1) and used to prove Theorems 1–3. Exact zero-loss separability of the full low-quality SFT set is a strong idealization; if it fails, the equivalence and usefulness theorems do not apply.
  • domain assumption Existence of at least one useful sample in the low-quality SFT dataset
    Assumed in Theorem 2 and Theorem 3 (Section 3.2) to guarantee the upper-level validation loss can be driven to zero by selecting a single sample. Without it, the strict improvement over direct mixing may not hold.
  • domain assumption Evaluation distribution close to validation distribution: KL(p_eval||p_val) ≤ ε and bounded per-sample losses 0 ≤ L_0(θ;x,y) ≤ B
    Invoked in Theorem 4 (Section 3.2) to transfer improvement from empirical validation to empirical evaluation loss. If the evaluation distribution diverges, the guarantee vanishes.

pith-pipeline@v1.3.0-alltime-deepseek · 30055 in / 15045 out tokens · 145359 ms · 2026-08-03T20:05:40.248481+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Bilevel Data Curation for LLM Fine-tuning: Offline Selection and Online Self-Refining Generation." pith.science (2026). https://pith.science/paper/K2VOK7PA

@misc{pith2026251121056,
  author       = {Pith},
  title        = {Pith review of: Bilevel Data Curation for LLM Fine-tuning: Offline Selection and Online Self-Refining Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K2VOK7PA}},
  note         = {Machine review of arXiv:2511.21056}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Supervised fine-tuning (SFT) datasets are critical to the downstream performance of large language models, yet they often contain low-quality or harmful question-response pairs. To improve SFT data quality, we develop a unified bilevel framework that combines offline data selection with the online self-refining generation. In the offline setting, bilevel data selection (BDS) selects question-response pairs from the offline SFT dataset to maximize the validation performance. We theoretically show that the optimal model given by BDS outperforms direct data mixing approach in useful data coverage. Moreover, we provide a global convergence analysis for gradient-based BDS approach for one-layer Transformer, showing that the epsilon-global optimum of offline BDS is achievable in finite time. Although efficient, offline BDS discards potentially harmful questions together with responses, thereby reducing question diversity. We address this limitation by refining the responses to selected questions using online self-refining generation framework. However, BDS is inefficient to update the response weights when responses are regenerated online. To address this issue, we introduce bilevel multi-objective optimization (BMO) for response-level weighting. We show that BMO recovers the same validation-aligned solution as BDS, but admits a closed-form importance-ratio weight that adapts to regenerated responses. Experiments on LLM quality enhancement and safety-aware fine-tuning demonstrate that the proposed framework consistently improves both data quality and downstream fine-tuning performance.

Figures

Figures reproduced from arXiv: 2511.21056 by Gaowen Liu, Quan Xiao, Ramana Rao Kompella, Tianyi Chen, Yutong Xuan.

Figure 1
Figure 1. Figure 1: An overview of bilevel data selection principle. ‘WP’ in the figure is short for weak Pareto optimal [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: An overview of our online self-refining algo [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: An overview of key steps for establishing the [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Ablation study of our algorithm (online) and comparisons with other baselines. Fine-tuning loss on [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Average response length of top 10% ques [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: An overview of how PBGD selects the data for [PITH_FULL_IMAGE:figures/full_fig_p021_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

91 extracted references · 15 linked inside Pith

  1. [1]

    A convergence theory for deep learning via over- parameterization

    Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterization. InProc. International Confer- ence on Machine Learning, pages 242–252, 2019

    2019

  2. [2]

    Amortized im- plicit differentiation for stochastic bilevel optimiza- tion

    Michael Arbel and Julien Mairal. Amortized im- plicit differentiation for stochastic bilevel optimiza- tion. InProc. International Conference on Learn- ing Representations, virtual, 2022

    2022

  3. [3]

    What data en- ables optimal decisions? an exact characteri- zation for linear optimization.arXiv preprint arXiv:2505.21692, 2025

    Omar Bennouna, Amine Bennouna, Saurabh Amin, and Asuman Ozdaglar. What data en- ables optimal decisions? an exact characteri- zation for linear optimization.arXiv preprint arXiv:2505.21692, 2025

    Pith/arXiv arXiv 2025

  4. [4]

    Pythia: A suite for analyzing large language models across training and scaling

    Stella Biderman, Hailey Schoelkopf, Quentin Gre- gory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivan- shu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. InProc. Inter- national Conference on Machine Learning, 2023

    2023

  5. [5]

    Cambridge uni- versity press, 2004

    Stephen Boyd, Stephen P Boyd, and Lieven Van- denberghe.Convex optimization. Cambridge uni- versity press, 2004

    2004

  6. [6]

    Ef- ficient first-order optimization on the pareto set for multi-objective learning under preference guid- ance

    Lisha Chen, Quan Xiao, Ellen Hidemi Fukuda, Xinyi Chen, Kun Yuan, and Tianyi Chen. Ef- ficient first-order optimization on the pareto set for multi-objective learning under preference guid- ance. InProc. International Conference on Ma- chine Learning, Vancouver, BC, Canada, 2025

    2025

  7. [7]

    Closing the gap: Tighter analysis of alternating stochastic gradient methods for bilevel problems

    Tianyi Chen, Yuejiao Sun, and Wotao Yin. Closing the gap: Tighter analysis of alternating stochastic gradient methods for bilevel problems. InProc. Offline Data Selection and Online Self-refining Generation Advances in Neural Information Processing Sys- tems, virtual, 2021

    2021

  8. [8]

    Safety-aware fine-tuning of large language models

    Hyeong Kyu Choi, Xuefeng Du, and Yixuan Li. Safety-aware fine-tuning of large language models. InNeurips Safe Generative AI Workshop, 2024

    2024

  9. [9]

    A framework for bilevel op- timization that enables stochastic and global vari- ance reduction algorithms

    Mathieu Dagr´ eou, Pierre Ablin, Samuel Vaiter, and Thomas Moreau. A framework for bilevel op- timization that enables stochastic and global vari- ance reduction algorithms. InProc. Advances in Neural Information Processing Systems, New Or- leans, LA, 2022

    2022

  10. [10]

    Solving bilevel multi-objective optimization problems using evolu- tionary algorithms

    Kalyanmoy Deb and Ankur Sinha. Solving bilevel multi-objective optimization problems using evolu- tionary algorithms. InInternational conference on evolutionary multi-criterion optimization, pages 110–124. Springer, 2009

    2009

  11. [11]

    Dempe and P

    S. Dempe and P. Mehlitz. Semivectorial bilevel programming versus scalar bilevel programming. Optimization, 69(4):657–679, 2020

    2020

  12. [12]

    Springer, 2005

    Matthias Ehrgott.Multicriteria optimization. Springer, 2005

    2005

  13. [13]

    Mitigating forgetting in llm su- pervised fine-tuning and preference learning.arXiv preprint arXiv:2410.15483, 2024

    Heshan Fernando, Han Shen, Parikshit Ram, Yi Zhou, Horst Samulowitz, Nathalie Baracaldo, and Tianyi Chen. Mitigating forgetting in llm su- pervised fine-tuning and preference learning.arXiv preprint arXiv:2410.15483, 2024

    arXiv 2024

  14. [14]

    Forward and reverse gradient-based hyperparameter optimization

    Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil. Forward and reverse gradient-based hyperparameter optimization. In Proc. International Conference on Machine Learn- ing, Sydney, Australia, 2017

    2017

  15. [15]

    Bilevel programming for hyperparameter optimization and meta-learning

    Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimilano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. InProc. International Con- ference on Machine Learning, Stockholm, Sweden, 2018

    2018

  16. [16]

    Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.arXiv preprint arXiv:2209.07858, 2022

    Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Ka- mal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.arXiv preprint arXiv:2209.07858, 2022

    Pith/arXiv arXiv 2022

  17. [17]

    On the properties of the softmax function with application in game theory and reinforcement learning.arXiv preprint arXiv:1704.00805, 2017

    Bolin Gao and Lacra Pavel. On the properties of the softmax function with application in game theory and reinforcement learning.arXiv preprint arXiv:1704.00805, 2017

    Pith/arXiv arXiv 2017

  18. [18]

    The language model evaluation harness, 07 2024

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lin- tang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The langu...

    arXiv 2024

  19. [19]

    Learning and data selection in big datasets

    Hossein Shokri Ghadikolaei, Hadi Ghauch, Carlo Fischione, and Mikael Skoglund. Learning and data selection in big datasets. InProc. Interna- tional Conference on Machine Learning, 2019

    2019

  20. [20]

    Approximation methods for bilevel programming.arXiv preprint arXiv:1802.02246, 2018

    Saeed Ghadimi and Mengdi Wang. Approximation methods for bilevel programming.arXiv preprint arXiv:1802.02246, 2018

    Pith/arXiv arXiv 2018

  21. [22]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ah- mad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  22. [23]

    On the iteration com- plexity of hypergradient computation

    Riccardo Grazzi, Luca Franceschi, Massimiliano Pontil, and Saverio Salzo. On the iteration com- plexity of hypergradient computation. InProc. In- ternational Conference on Machine Learning, vir- tual, 2020

    2020

  23. [24]

    Reinforced self-training (rest) for language modeling.arXiv preprint arXiv:2308.08998, 2023

    Caglar Gulcehre, Tom Le Paine, Srivatsan Srini- vasan, Ksenia Konyushkova, Lotte Weerts, Ab- hishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (rest) for language modeling.arXiv preprint arXiv:2308.08998, 2023

    Pith/arXiv arXiv 2023

  24. [25]

    A two-timescale stochastic algo- rithm framework for bilevel optimization: Com- plexity analysis and application to actor-critic

    Mingyi Hong, Hoi-To Wai, Zhaoran Wang, and Zhuoran Yang. A two-timescale stochastic algo- rithm framework for bilevel optimization: Com- plexity analysis and application to actor-critic. SIAM Journal on Optimization, 33(1):147–180, 2023

    2023

  25. [26]

    User prefer- ence meets pareto-optimality in multi-objective bayesian optimization

    Joshua Hang Sai Ip, Ankush Chakrabarty, Ali Mesbah, and Diego Romeres. User prefer- ence meets pareto-optimality in multi-objective bayesian optimization. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 20246–20254, 2025

    2025

  26. [27]

    Bilevel optimization: Convergence analysis and enhanced design

    Kaiyi Ji, Junjie Yang, and Yingbin Liang. Bilevel optimization: Convergence analysis and enhanced design. InProc. International Conference on Ma- chine Learning, virtual, 2021. Quan Xiao, Tianyi Chen

    2021

  27. [28]

    Directional conver- gence and alignment in deep learning

    Ziwei Ji and Matus Telgarsky. Directional conver- gence and alignment in deep learning. InProc. Ad- vances in Neural Information Processing Systems, virtual, 2020

    2020

  28. [29]

    Get more for less: Principled data selection for warming up fine- tuning in llms

    Feiyang Kang, Hoang Anh Just, Yifan Sun, Hi- manshu Jahagirdar, Yuanzhi Zhang, Rongxing Du, Anit Kumar Sahu, and Ruoxi Jia. Get more for less: Principled data selection for warming up fine- tuning in llms. InProc. International Confer- ence on Learning Representations, Vienna, Aus- tria, 2024

    2024

  29. [30]

    A near-optimal algorithm for stochastic bilevel op- timization via double-momentum

    Prashant Khanduri, Siliang Zeng, Mingyi Hong, Hoi-To Wai, Zhaoran Wang, and Zhuoran Yang. A near-optimal algorithm for stochastic bilevel op- timization via double-momentum. InProc. Ad- vances in Neural Information Processing Systems, virtual, 2021

    2021

  30. [31]

    Adam: A method for stochas- tic optimization

    Diederik P Kingma. Adam: A method for stochas- tic optimization. InProc. International Conference on Learning Representations, 2015

    2015

  31. [32]

    A fully first-order method for stochastic bilevel optimization

    Jeongyeol Kwon, Dohyun Kwon, Stephen Wright, and Robert D Nowak. A fully first-order method for stochastic bilevel optimization. InProc. Inter- national Conference on Machine Learning, Hon- olulu, HI, 2023

    2023

  32. [33]

    On penalty methods for non- convex bilevel optimization and first-order stochas- tic approximation

    Jeongyeol Kwon, Dohyun Kwon, Steve Wright, and Robert Nowak. On penalty methods for non- convex bilevel optimization and first-order stochas- tic approximation. InProc. International Confer- ence on Learning Representations, Vienna, Aus- tria, 2024

    2024

  33. [34]

    A fully sin- gle loop algorithm for bilevel optimization without hessian inverse

    Junyi Li, Bin Gu, and Heng Huang. A fully sin- gle loop algorithm for bilevel optimization without hessian inverse. InProc. Association for the Ad- vancement of Artificial Intelligence, virtual, 2022

    2022

  34. [35]

    Hashimoto

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval, 5 2023

    2023

  35. [36]

    Data mixing optimization for supervised fine-tuning of large language models

    Yuan Li, Zhengzhong Liu, and Eric Xing. Data mixing optimization for supervised fine-tuning of large language models. InProc. International Conference on Machine Learning, Vancouver, BC, Canada, 2025

    2025

  36. [37]

    Preserving diversity in supervised fine-tuning of large language models

    Ziniu Li, Congliang Chen, Tian Xu, Zeyu Qin, Jiancong Xiao, Zhi-Quan Luo, and Ruoyu Sun. Preserving diversity in supervised fine-tuning of large language models. InProc. International Con- ference on Learning Representations, Singapore, Singapore, 2025

    2025

  37. [38]

    Data- efficient fine-tuning for llm-based recommenda- tion

    Xinyu Lin, Wenjie Wang, Yongqi Li, Shuo Yang, Fuli Feng, Yinwei Wei, and Tat-Seng Chua. Data- efficient fine-tuning for llm-based recommenda- tion. InProceedings of the 47th international ACM SIGIR conference on research and development in information retrieval, pages 365–374, 2024

    2024

  38. [39]

    Goedel- prover-v2: Scaling formal theorem proving with scaffolded data synthesis and self-correction.arXiv preprint arXiv:2508.03613, 2025

    Yong Lin, Shange Tang, Bohan Lyu, Ziran Yang, Jui-Hui Chung, Haoyu Zhao, Lai Jiang, Yihan Geng, Jiawei Ge, Jingruo Sun, et al. Goedel- prover-v2: Scaling formal theorem proving with scaffolded data synthesis and self-correction.arXiv preprint arXiv:2508.03613, 2025

    Pith/arXiv arXiv 2025

  39. [40]

    Regmix: Data mixture as re- gression for language model pre-training

    Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, and Min Lin. Regmix: Data mixture as re- gression for language model pre-training. InProc. International Conference on Learning Representa- tions, Singapore, Singapore, 2025

    2025

  40. [41]

    A generic first-order algo- rithmic framework for bi-level programming be- yond lower-level singleton

    Risheng Liu, Pan Mu, Xiaoming Yuan, Shangzhi Zeng, and Jin Zhang. A generic first-order algo- rithmic framework for bi-level programming be- yond lower-level singleton. InProc. International Conference on Machine Learning, virtual, 2020

    2020

  41. [42]

    Averaged method of multipliers for bi-level optimization without lower-level strong convexity

    Risheng Liu, Yaohua Liu, Wei Yao, Shangzhi Zeng, and Jin Zhang. Averaged method of multipliers for bi-level optimization without lower-level strong convexity. InProc. International Conference on Machine Learning, Honolulu, HI, 2023

    2023

  42. [43]

    What makes good data for align- ment? a comprehensive study of automatic data selection in instruction tuning

    Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for align- ment? a comprehensive study of automatic data selection in instruction tuning. InProc. Inter- national Conference on Learning Representations, Vienna, Austria, 2024

    2024

  43. [44]

    The flan col- lection: Designing data and methods for effective instruction tuning

    Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan col- lection: Designing data and methods for effective instruction tuning. InProc. International Confer- ence on Machine Learning, Honolulu, HI, 2023

    2023

  44. [45]

    First-order penalty methods for bilevel optimization.SIAM Journal on Optimization, 34(2):1937–1969, 2024

    Zhaosong Lu and Sanyou Mei. First-order penalty methods for bilevel optimization.SIAM Journal on Optimization, 34(2):1937–1969, 2024

    1937

  45. [46]

    Learning what reinforcement learning can’t: Interleaved online fine-tuning for hardest questions.arXiv preprint arXiv:2506.07527, 2025

    Lu Ma, Hao Liang, Meiyi Qiang, Lexiang Tang, Xi- aochen Ma, Zhen Hao Wong, Junbo Niu, Chengyu Shen, Runming He, Bin Cui, et al. Learning what reinforcement learning can’t: Interleaved online fine-tuning for hardest questions.arXiv preprint arXiv:2506.07527, 2025

    arXiv 2025

  46. [47]

    Multi- task learning with user preferences: Gradient de- scent with controlled ascent in pareto optimiza- Offline Data Selection and Online Self-refining Generation tion

    Debabrata Mahapatra and Vaibhav Rajan. Multi- task learning with user preferences: Gradient de- scent with controlled ascent in pareto optimiza- Offline Data Selection and Online Self-refining Generation tion. InProc. International Conference on Ma- chine Learning, virtual, 2020

    2020

  47. [48]

    Emergence of separable manifolds in deep language representations

    Jonathan Mamou, Hang Le, Miguel Del Rio, Cory Stephenson, Hanlin Tang, Yoon Kim, and Sueyeon Chung. Emergence of separable manifolds in deep language representations. InProc. International Conference on Machine Learning, virtual, 2020

    2020

  48. [49]

    MIT press, 2018

    Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar.Foundations of machine learning. MIT press, 2018

    2018

  49. [50]

    Orca: Progressive learning from complex explanation traces of gpt-4.arXiv preprint arXiv:2306.02707, 2023

    Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4.arXiv preprint arXiv:2306.02707, 2023

    Pith/arXiv arXiv 2023

  50. [51]

    Token cleaning: Fine-grained data selection for llm su- pervised fine-tuning

    Jinlong Pang, Na Di, Zhaowei Zhu, Jiaheng Wei, Hao Cheng, Chen Qian, and Yang Liu. Token cleaning: Fine-grained data selection for llm su- pervised fine-tuning. InProc. International Con- ference on Machine Learning, Vancouver, BC, Canada, 2025

    2025

  51. [52]

    Hyperparameter optimization with approximate gradient

    Fabian Pedregosa. Hyperparameter optimization with approximate gradient. InProc. International Conference on Machine Learning, New York City, NY, 2016

    2016

  52. [53]

    Learning dy- namics of LLM Finetuning

    Yi Ren and Danica J Sutherland. Learning dy- namics of LLM Finetuning. InProc. International Conference on Learning Representations, Singa- pore, Singapore, 2025

    2025

  53. [54]

    Cos-dpo: Condi- tioned one-shot multi-objective fine-tuning frame- work

    Yinuo Ren, Tesi Xiao, Michael Shavlovsky, Lexing Ying, and Holakou Rahmanian. Cos-dpo: Condi- tioned one-shot multi-objective fine-tuning frame- work. InThe 41st Conference on Uncertainty in Artificial Intelligence

  54. [55]

    Springer Science & Business Media, 2009

    R Tyrrell Rockafellar and Roger J-B Wets.Vari- ational analysis, volume 317. Springer Science & Business Media, 2009

    2009

  55. [56]

    Large language models encode seman- tics in low-dimensional linear subspaces.arXiv preprint arXiv:2507.09709, 2025

    Baturay Saglam, Paul Kassianik, Blaine Nelson, Sajana Weerawardhena, Yaron Singer, and Amin Karbasi. Large language models encode seman- tics in low-dimensional linear subspaces.arXiv preprint arXiv:2507.09709, 2025

    arXiv 2025

  56. [57]

    Equilibrium propagation: Bridging the gap between energy- based models and backpropagation.Frontiers in computational neuroscience, 11:24, 2017

    Benjamin Scellier and Yoshua Bengio. Equilibrium propagation: Bridging the gap between energy- based models and backpropagation.Frontiers in computational neuroscience, 11:24, 2017

    2017

  57. [58]

    Seal: Safety-enhanced aligned llm fine- tuning via bilevel data selection

    Han Shen, Pin-Yu Chen, Payel Das, and Tianyi Chen. Seal: Safety-enhanced aligned llm fine- tuning via bilevel data selection. InProc. Inter- national Conference on Learning Representations, 2025

    2025

  58. [59]

    On penalty-based bilevel gradient descent method

    Han Shen, Quan Xiao, and Tianyi Chen. On penalty-based bilevel gradient descent method. Mathematical Programming, pages 1–51, 2025

    2025

  59. [60]

    The importance of online data: Un- derstanding preference fine-tuning via coverage

    Yuda Song, Gokul Swamy, Aarti Singh, J Bagnell, and Wen Sun. The importance of online data: Un- derstanding preference fine-tuning via coverage. In Proc. Advances in Neural Information Processing Systems, Vancouver, BC, Canada, 2024

    2024

  60. [61]

    The im- plicit bias of gradient descent on separable data

    Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The im- plicit bias of gradient descent on separable data. Journal of Machine Learning Research, 19(70):1– 57, 2018

    2018

  61. [62]

    Preference fine-tuning of llms should leverage sub- optimal, on-policy data

    Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Ste- fano Ermon, Chelsea Finn, and Aviral Kumar. Preference fine-tuning of llms should leverage sub- optimal, on-policy data. InProc. International Conference on Machine Learning, Vienna, Aus- tria, 2024

    2024

  62. [63]

    Fukuda, and Nobuo Ya- mashita

    Hiroki Tanabe, Ellen H. Fukuda, and Nobuo Ya- mashita. New merit functions for multiobjective optimization and their properties.arXiv preprint arXiv:2010.09333, 2022

    Pith/arXiv arXiv 2010

  63. [64]

    Stanford al- paca: An instruction-following llama model, 2023

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford al- paca: An instruction-following llama model, 2023

    2023

  64. [65]

    Transform- ers as support vector machines.arXiv preprint arXiv:2308.16898, 2023

    Davoud Ataee Tarzanagh, Yingcong Li, Christos Thrampoulidis, and Samet Oymak. Transform- ers as support vector machines.arXiv preprint arXiv:2308.16898, 2023

    Pith/arXiv arXiv 2023

  65. [66]

    Importance sampling: a review.Wiley Interdisciplinary Re- views: Computational Statistics, 2(1):54–60, 2010

    Surya T Tokdar and Robert E Kass. Importance sampling: a review.Wiley Interdisciplinary Re- views: Computational Statistics, 2(1):54–60, 2010

    2010

  66. [67]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. InProc. Advances in Neural Infor- mation Processing Systems, 2017

    2017

  67. [68]

    On implicit bias in overparameterized bilevel op- timization

    Paul Vicol, Jonathan P Lorraine, Fabian Pe- dregosa, David Duvenaud, and Roger B Grosse. On implicit bias in overparameterized bilevel op- timization. InProc. International Conference on Machine Learning, Baltimore, MD, 2022

    2022

  68. [69]

    Interpretable preferences via multi-objective reward modeling and mixture- of-experts

    Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture- of-experts. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 10582–10592, 2024. Quan Xiao, Tianyi Chen

    2024

  69. [70]

    Hbo: Hierarchical balancing op- timization for fine-tuning large language models

    Weixuan Wang, Minghao Wu, Barry Haddow, and Alexandra Birch. Hbo: Hierarchical balancing op- timization for fine-tuning large language models. arXiv preprint arXiv:2505.12300, 2025

    arXiv 2025

  70. [71]

    Self-instruct: Aligning language models with self-generated instructions

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint arXiv:2212.10560, 2022

    Pith/arXiv arXiv 2022

  71. [72]

    Finetuned language models are zero-shot learners

    Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. InProc. Inter- national Conference on Learning Representations, 2022

    2022

  72. [73]

    Ldc-mtl: Balancing multi-task learning through scalable loss discrepancy control.arXiv preprint arXiv:2502.08585, 2025

    Peiyao Xiao, Chaosheng Dong, Shaofeng Zou, and Kaiyi Ji. Ldc-mtl: Balancing multi-task learning through scalable loss discrepancy control.arXiv preprint arXiv:2502.08585, 2025

    arXiv 2025

  73. [74]

    Unlocking global optimality in bilevel optimization: A pilot study

    Quan Xiao and Tianyi Chen. Unlocking global optimality in bilevel optimization: A pilot study. arXiv preprint arXiv:2408.16087, 2024

    Pith/arXiv arXiv 2024

  74. [75]

    A gen- eralized alternating method for bilevel optimiza- tion under the polyak- lojasiewicz condition

    Quan Xiao, Songtao Lu, and Tianyi Chen. A gen- eralized alternating method for bilevel optimiza- tion under the polyak- lojasiewicz condition. In Proc. Advances in Neural Information Processing Systems, New Orleans, LA, 2023

    2023

  75. [76]

    Self-training with noisy student im- proves imagenet classification

    Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student im- proves imagenet classification. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10687–10698, 2020

    2020

  76. [77]

    Doremi: Optimizing data mixtures speeds up lan- guage model pretraining

    Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up lan- guage model pretraining. InProc. Advances in Neural Information Processing Systems, New Or- leans, LA, 2023

    2023

  77. [78]

    Data selection for language models via importance resampling

    Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang. Data selection for language models via importance resampling. InProc. Ad- vances in Neural Information Processing Systems, New Orleans, LA, 2023

    2023

  78. [79]

    Prov- ably faster algorithms for bilevel optimization

    Junjie Yang, Kaiyi Ji, and Yingbin Liang. Prov- ably faster algorithms for bilevel optimization. In Proc. Advances in Neural Information Processing Systems, virtual, 2021

    2021

  79. [80]

    Llm data selection and utilization via dynamic bi-level opti- mization.arXiv preprint arXiv:2507.16178, 2025

    Yang Yu, Kai Han, Hang Zhou, Yehui Tang, Kaiqi Huang, Yunhe Wang, and Dacheng Tao. Llm data selection and utilization via dynamic bi-level opti- mization.arXiv preprint arXiv:2507.16178, 2025

    Pith/arXiv arXiv 2025

  80. [81]

    Understanding why generalized reweighting does not improve over ERM

    Runtian Zhai, Chen Dan, J Zico Kolter, and Pradeep Kumar Ravikumar. Understanding why generalized reweighting does not improve over ERM. InProc. International Conference on Learn- ing Representations, Kigali, Rwanda, 2023

    2023

Showing first 80 references.