Pith. sign in

REVIEW 1 major objections 4 minor 72 references

Estimating the Effects of Sample Training Orders for Large Language Models without Retraining

T0 review · 1 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that the effect of any training-sample order on a large language model's final parameters can be estimated from one reference training run, using Taylor expansions of the optimizer update and random-projection storage, so…

desk verdict A genuinely useful question and a concrete pipeline, but the core Taylor derivation ignores Adam's path-dependent optimizer state and the appendix's Hessian substitute is not a Hessian, so the estimates are not grounded. read the letter →

arxiv 2505.22042 v1 pith:6KHFQ2IJ submitted 2025-05-28 cs.LG cs.AI

Hao Yang , Haoxuan Li , Mengyue Yang , Xu Chen , Mingming Gong This is my paper
classification cs.LGcs.AI
keywords sampleorderretraining-freeestimationAdamoptimizerTaylorexpansionrandomprojectioncurriculumlearningmemorizationgeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Training order influences how large language models learn, but measuring that influence by retraining for each order is infeasible. The paper proposes a retraining-free framework (FUT) that reconstructs the parameter trajectory under a permuted batch order from checkpoints of a single reference run. It does this by treating the Adam update term as a function of the current parameters and applying first- or second-order Taylor expansions, so the update for any batch at any step can be expressed using stored terms. Random projection keeps the stored terms small enough for LLM-scale parameter counts. The paper reports that estimated validation perplexities land within roughly 0.0085 to 0.0703 of retrained values on a 636M-parameter model and that the method becomes increasingly cheaper than retraining as more orders are evaluated.

What carries the argument

The load-bearing object is the Adam update term $\Gamma(\theta, B)$, viewed as a function of the parameters $\theta$; the framework expands $\Gamma(\gamma_t, B_{l_t})$ around the reference checkpoint $\theta_t$ using a Taylor series, converting an unknown update under a permuted order into known quantities $\Gamma(\theta_t, B_{l_t})$, $\nabla_\theta \Gamma(\theta_t, B_{l_t})$, and in the enhanced version $\nabla^2_\theta \Gamma(\theta_t, B_{l_t})$. To avoid storing a full matrix for every pair, the framework compresses these terms with a Gaussian random projection based on the Johnson-Lindenstrauss dimension-reduction guarantee and reconstructs them via the Moore-Penrose pseudoinverse. The Hessian of the loss is not computed exactly; it is replaced by a secant quotient $(\nabla_\theta L(B; \theta_t) - \nabla_\theta L(B; \theta_{t-1}))/(\theta_t - \theta_{t-1})$, and the third-order term in the enhanced version is approximated similarly. These pieces let the estimation stage run recursively, one forward or backward pass per stored pair, with $O(T^2 C)$ total cost compared with $O(T \cdot C \cdot T!)$ for exhaustive retraining.

What would settle it

Train a 636M-parameter model once for $T=16$ batches in a reference order and store the checkpoints; use the paper's method to predict the validation perplexity for a randomly permuted order, then retrain from the same initialization on that permuted order. If the absolute difference between predicted and retrained perplexity is larger than the spread among random-order retraining runs, or grows as $T$ increases from 8 to 256, the central claim fails.

Watch

Extended reading notes

Core claim

The central discovery is that the trajectory of an LLM trained with a permuted batch order can be written as a recursion over the reference trajectory: $\gamma_{t+1} = \gamma_t - \eta \Gamma(\theta_t, B_{l_t}) - \eta(\gamma_t - \theta_t)\nabla_\theta \Gamma(\theta_t, B_{l_t})$, where $\Gamma(\theta, B) = m/(\sqrt{v}+\epsilon)$ is the Adam update term and the gradient term is evaluated at the reference checkpoint $\theta_t$. All quantities on the right come from the reference run, so arbitrary orders can be simulated without retraining. A second-order version adds a curvature term $c(\gamma_t - \theta_t)^2 \nabla^2_\theta \Gamma(\theta_t, B_{l_t})$ and generally improves accuracy in the reported experiments. The paper validates the approximation by comparing predicted perplexity to retrained perplexity across $T=8$ to $256$ batches, and applies it to curriculum search and to measuring position-dependent memorization and generalization.

Load-bearing premise

The load-bearing premise is that shifting the model's parameters a little changes the optimizer update in a smooth, nearly linear way, and that the curvature of the loss can be read off the previous step's change in gradients; if those approximations fail, the predicted checkpoints for a reshuffled order will drift from the actually retrained ones.

Editorial extensions

If this is right

  • After one reference run, any number of candidate training curricula can be scored by estimated validation perplexity; the paper's genetic search uses those scores to choose an order and reports lower perplexity than random, sample-length, and perplexity-based curricula.
  • The estimated parameter trajectories reproduce the retrained memorization pattern that later batches are memorized better (lower perplexity) and earlier batches are more prone to forgetting, without running the model again for each position.
  • Batches whose content is similar to the test set generalize better when placed later in training, while dissimilar batches show little or random positional effect; the estimated curves match the retrained curves in the reported experiments.
  • Because only the update rule changes, the same Taylor-transfer construction extends to other batch-based optimizers such as SGD, as the paper states.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the Taylor transfer holds as reported, order effects are effectively determined by pairwise interactions between reference checkpoints and batches, suggesting a low-dimensional 'order kernel' might predict order effects on new datasets without additional full runs.
  • The same stored update terms could answer influence-style questions—removing, duplicating, or reweighting a batch—since those are limiting cases of reordering, although the paper does not test this.
  • The framework implies that estimate error should grow with learning rate and total batch count, because larger parameter displacements push the expansion further from the reference trajectory; the reported $T=16$ case has the largest gap, so a natural extension is an adaptive rule deciding when the second-order term is necessary.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 4 minor

Summary. The paper proposes FUT, a framework that claims to estimate, without retraining, the model parameters and perplexity of a large language model trained on arbitrary permutations of a given mini-batch sequence. A single reference training run provides checkpoints θ_t; for every pair (θ_t, B_l) the paper precomputes the Adam update term Γ(θ_t, B_l) and its first- and second-order derivatives, compresses them with random projections, and then simulates a target permutation by recursively applying a Taylor expansion of Γ around the reference checkpoints. The framework is applied to training-curriculum search and to memorization/generalization analysis. Experiments on a 636M-parameter LLaMA-style model report AbsDiff values below a Random baseline, curriculum perplexity improvements, and scaling experiments up to 1.4B parameters.

Significance. If the central derivation were valid, the framework would address a genuinely important computational bottleneck: it would turn an O(T!)-cost problem into an O(T^2)-cost precomputation plus cheap recursive estimation. The paper has real empirical ambition, including repeated retraining for validation, intermediate-step perplexity tracking, and two downstream applications, and it is honest in reporting that only perplexity is used as the outcome metric. However, the load-bearing mathematical steps are not sound: the Taylor recurrence is built on an invalid secant substitute for the Hessian, and the Adam momentum state is taken from the reference trajectory rather than from the permuted one. These issues are not cosmetic; they mean Eqs. (3)-(8) do not describe the trajectory of the permuted training run, and the empirical numbers cannot be read as validating the proposed mechanism. The self-reported limitations in Appendix F (Taylor accuracy and projection noise) do not mention either of these structural defects.

major comments (1)
  1. [Eqs. (5)-(6), Appendix A.1] The manuscript text in Section 5.1 reports AbsDiff values as evidence that the method reproduces retrained perplexities. However, the second-order weight c in Eq. (7) is never specified or selected in the paper, the parameter clipping threshold in Section B.1.3 is tuned from a range, and the random-projection dimension k is chosen empirically per layer and update type (Appendix A.2). With these tunable elements undisclosed, Table 1 measures the tuned pipeline rather than the Taylor expansion itself, and the main empirical claim is not reproducible from the information given.
minor comments (4)
  1. [Abstract and Figure 1] The abstract says the method uses random projection to store intermediate checkpoints, whereas Figure 1 and Section 3 correctly state that update terms and gradient terms are stored; the wording should be aligned.
  2. [Appendix A.1, Eq. (11)] The sentence describing m_{t-1} and v_{t-1} says they are constructed from the gradient at the last step of the original training order, which is a special choice and not the only possible one; this should be stated as a modeling assumption rather than as a consequence of the Adam update.
  3. [Table 1 and Section 5.1] The Random baseline draws uniformly from the min-max range of the ground-truth perplexities, which are obtained by retraining; the comparison therefore does not isolate the difficulty of estimation without retraining and should be discussed as such.
  4. [Section 3, complexity comparison] The claimed O(T^2 C) complexity counts only forward/backward passes over batches but omits the cost of computing ∇²_θ L in Eq. (5) and the pseudoinverse recovery in Appendix A.2, both of which are non-negligible for LLM-scale parameters.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the FUT estimates are an approximation built from the standard Adam update rule and are validated against external retraining benchmarks.

full rationale

The paper's derivation chain is self-contained rather than circular. The central equations (1)-(2) are the standard Adam optimizer update rule, and Eq. (3)-(4) are a Taylor expansion of the update term as a function of the parameters, with the reference trajectory used only as the expansion base. The estimated trajectories are then compared against actual retraining results in Table 1 and Figures 3-5, i.e., external ground truth, not against the framework's own stored terms. The stored terms Γ(θ_t, B_l), ∇_θΓ(θ_t, B_l), and ∇²_θΓ(θ_t, B_l) are computed from reference checkpoints and then used recursively; no fitted parameter is renamed as a prediction. The concerns raised by the reader's take—the secant approximation of the Hessian in Appendix A.1, reuse of reference Adam moments, the unspecified constant c in Eq. (7), and the tunable clipping range in Appendix B.1.3—are correctness and robustness limitations, not cases where a predicted quantity is equal to an input by construction. The paper itself flags the Taylor-expansion validity, random-projection noise, and the perplexity-only evaluation in Section F (Limitations). These are legitimate scientific caveats but do not make the central claim circular. Consequently, no circular step can be exhibited from the paper's own equations.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central derivation rests on several unproved modeling choices. Most serious is the secant replacement for the Hessian, which is not a true second derivative and enters every ∇θΓ term. The Adam state is carried over from the reference order, so permuted-order dynamics are not actually simulated. The second-order weight c and clipping range are tuned but not reported, and random projection recovery from JL-compressed rows has no reconstruction guarantee for the full Jacobian terms.

free parameters (4)
  • c (second-order Taylor weight) = not reported
    Equation (7) introduces c to weight the second-order term, but the paper never states its value or how it was chosen; the FUT++ results depend on this undisclosed constant.
  • parameter clipping threshold = selected from [-1.1,-0.3] U [0.3,1.1]
    Appendix B.1.3 clips estimated parameters to a tunable range chosen for numerical stability; no criterion or sensitivity analysis is given.
  • random projection dimension k = 300, 200, 160, 80, 20, or 8 by layer/update type
    Appendix A.2 sets k empirically to balance accuracy and memory; estimation error depends on these choices.
  • GA population size N and generations K = N in [16,12,8,4,2], K in [16,12,8,4,2,1] per T
    Table 3 in Appendix B.2.2 tunes these to the batch count; they affect the curriculum-search results and are not derived from any principle.
assumptions (5)
  • domain assumption The Adam update term Γ(θ,B) is sufficiently smooth that first- and second-order Taylor expansions around the reference checkpoint θ_t are accurate across the whole permuted trajectory.
    Invoked in Eqs (3) and (7) without an error bound; training dynamics of LLMs are highly nonlinear, so this is an unverified smoothness assumption.
  • ad hoc to paper The momentum state (m_t, v_t) from the reference training order can be reused when evaluating Γ(θ_t, B_l) for a permuted order.
    Appendix A.1 equations (10)-(11) construct m_t and v_t using the previous reference gradient rather than the permuted trajectory's gradient, effectively assuming the Adam state is order-independent.
  • ad hoc to paper The Hessian ∇²L(B_l; θ_t) can be replaced by (∇L(B_l; θ_t) - ∇L(B_l; θ_{t-1}))/(θ_t - θ_{t-1}) and the third derivative by a similar secant quotient.
    Appendix A.1 uses this elementwise vector division to approximate second and third order gradients; it is not a valid Hessian or third-derivative formula in parameter space.
  • domain assumption Recovering update matrices from random projections via the Moore-Penrose pseudoinverse preserves enough information for accurate parameter estimation.
    The JL theorem preserves pairwise distances, not individual high-dimensional matrices; Appendix A.2 provides no reconstruction error bound for the update terms after projection to dimensions as low as 8.
  • domain assumption Validation perplexity on WikiText-103 is a sufficient measure of LLM performance and of training-order effects.
    All experiments use perplexity only; Appendix F acknowledges downstream reasoning and understanding tasks are not validated, so the scope of the empirical claim is narrower than the abstract suggests.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Estimating the Effects of Sample Training Orders for Large Language Models without Retraining." pith.science (2026). https://pith.science/paper/6KHFQ2IJ

@misc{pith2026250522042,
  author       = {Pith},
  title        = {Pith review of: Estimating the Effects of Sample Training Orders for Large Language Models without Retraining},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6KHFQ2IJ}},
  note         = {Machine review of arXiv:2505.22042}
}
read the original abstract

The order of training samples plays a crucial role in large language models (LLMs), significantly impacting both their external performance and internal learning dynamics. Traditional methods for investigating this effect generally require retraining the model with various sample orders, which is computationally infeasible for LLMs. In this work, we improve traditional methods by designing a retraining-free framework. By approximating Adam optimizer updates with first- and second-order Taylor expansions and utilizing random projection methods to store intermediate checkpoints, our framework can efficiently estimate model parameters for arbitrary training sample orders. Next, we apply our framework to two downstream research problems: (1) Training curriculum design for LLMs -- we base our retraining-free framework to propose a novel curriculum learning strategy that augments curriculum proposals with estimated model performances, enabling more informed sample scheduling. (2) LLMs' memorization and generalization effect analysis -- we use our retraining-free framework to estimate how the positions of training samples influence LLMs' capacity for memorization and generalization. We conduct extensive experiments to validate the effectiveness of our retraining-free framework in reproducing the true model performances, and further demonstrate its potential in optimizing LLM training curricula and analyzing the memorization and generalization effects of LLMs.

Figures

Figures reproduced from arXiv: 2505.22042 by the authors.

Figure 1
Figure 1. Overview of the FUT framework. FUT operates in three stages: Stage 1: Compute the reference trajectory Θ = {θt} T t=0 using a fixed data order r. Stage 2: Store update and gradient terms for all (θt, Blt ) pairs, compressing them via random projection. Stage 3: Estimate trajectories {γ ki t } T t=0 under permuted data orders {ki} N i=1 using first-order Taylor expansion based on stored terms. A toy example along the… view at source ↗
Figure 2
Figure 2. Time cost comparison. In our experiments, assuming the dataset consists of T batches, we randomly select N training orders from the total of T! possible permutations. For each selected order, we use our method to estimate the model performance rˆ and also train the LLM using that order to obtain the ground-truth performance r. The performance gap is then calculated as: AbsDiff = 1 N X N k=1 |rˆk − rk|, where k index… view at source ↗
Figure 3
Figure 3. Memorization effects. Heatmaps in (a) and (b) are estimated by our FUT and FUT++ [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The generalization effect of batch Bi on dataset D, with sim(Bi , D) >= τ . 0 1 2 3 4 5 6 7 Position 1.36 1.38 1.40 1.42 1.44 1.46 1.48 1.50 Perplexity Generalize B0 to D Real Values FUT FUT++ 0 1 2 3 4 5 6 7 Position 1.36 1.38 1.40 1.42 1.44 1.46 1.48 1.50 Perplexity …
Figure 6
Figure 6. Figure 6: Scalable estimation performance across model sizes. We evaluate the estimation accuracy of FUT and FUT++ across model scales {0.8B, 1.0B, 1.2B, 1.4B} under training batch numbers T = 8 (left) and T = 16 (right). FUT and FUT++ consistently outperform the Random baseline…
Figure 7
Figure 7. Figure 7: Perplexity estimation at intermediate training steps. We visualize the validation perplexity estimated by FUT and FUT++ compared to the real validation perplexity after each batch, for training schedules with T ∈ {8, 16, 32} total batches. FUT and FUT++ both closely fo…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

72 extracted references · 42 canonical work pages

  1. [1]

    Critical learning periods in deep networks

    Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical learning periods in deep networks. InInternational Conference on Learning Representations, 2018

    work page 2018

  2. [2]

    If influence functions are the answer, then what is the question?Advances in Neural Information Processing Systems, 35:17953–17967, 2022

    Juhan Bae, Nathan Ng, Alston Lo, Marzyeh Ghassemi, and Roger B Grosse. If influence functions are the answer, then what is the question?Advances in Neural Information Processing Systems, 35:17953–17967, 2022

    work page 2022

  3. [3]

    Influence functions in deep learning are fragile

    S Basu, P Pope, and S Feizi. Influence functions in deep learning are fragile. InInternational Conference on Learning Representations (ICLR), 2021

    2021

  4. [4]

    Curriculum learning

    Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. InProceedings of the 26th annual international conference on machine learning, pages 41–48, 2009

    2009

  5. [5]

    Pythia: A suite for analyzing large language models across training and scaling

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397–2430. PMLR, 2023

    2023

  6. [6]

    Andrei Z. Broder. On the resemblance and containment of documents.Proceedings. Compres- sion and Complexity of SEQUENCES 1997 (Cat. No.97TB100171), pages 21–29, 1997

    1997

  7. [7]

    Generalization potential of large language models.Neural Computing and Applications, 37(4):1973–1997, 2025

    Mikhail Budnikov, Anna Bykova, and Ivan P Yamshchikov. Generalization potential of large language models.Neural Computing and Applications, 37(4):1973–1997, 2025

    work page 1973

  8. [8]

    Curriculum learning for language modeling.arXiv preprint arXiv:2108.02170, 2021

    Daniel Campos. Curriculum learning for language modeling.arXiv preprint arXiv:2108.02170, 2021

    arXiv 2021

Show all 72 references
  1. [9]

    Fast and accurate network embeddings via very sparse random projection

    Haochen Chen, Syed Fahad Sultan, Yingtao Tian, Muhao Chen, and Steven Skiena. Fast and accurate network embeddings via very sparse random projection. InProceedings of the 28th ACM international conference on information and knowledge management, pages 399–408, 2019

    2019

  2. [10]

    Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models

    Pin-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models. InProceedings of the 10th ACM workshop on artificial intelligence and security, pages 15–26, 2017

    2017

  3. [11]

    Zo- adamm: Zeroth-order adaptive momentum method for black-box optimization.Advances in neural information processing systems, 32, 2019

    Xiangyi Chen, Sijia Liu, Kaidi Xu, Xingguo Li, Xue Lin, Mingyi Hong, and David Cox. Zo- adamm: Zeroth-order adaptive momentum method for black-box optimization.Advances in neural information processing systems, 32, 2019

    2019

  4. [12]

    Optimal rates for zero-order convex optimization: The power of two function evaluations.IEEE Transactions on Information Theory, 61(5):2788–2806, 2015

    John C Duchi, Michael I Jordan, Martin J Wainwright, and Andre Wibisono. Optimal rates for zero-order convex optimization: The power of two function evaluations.IEEE Transactions on Information Theory, 61(5):2788–2806, 2015

    2015

  5. [13]

    What neural networks memorize and why: Discovering the long tail via influence estimation.Advances in Neural Information Processing Systems, 33:2881–2891, 2020

    Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation.Advances in Neural Information Processing Systems, 33:2881–2891, 2020

    2020

  6. [14]

    Online convex opti- mization in the bandit setting: gradient descent without a gradient.arXiv preprint cs/0408007, 2004

    Abraham D Flaxman, Adam Tauman Kalai, and H Brendan McMahan. Online convex opti- mization in the bandit setting: gradient descent without a gradient.arXiv preprint cs/0408007, 2004. 10

    2004 arXiv

  7. [15]

    The early phase of neural network training.arXiv preprint arXiv:2002.10365, 2020

    Jonathan Frankle, David J Schwab, and Ari S Morcos. The early phase of neural network training.arXiv preprint arXiv:2002.10365, 2020

    2002 arXiv

  8. [16]

    Stochastic first-and zeroth-order methods for nonconvex stochastic programming.SIAM journal on optimization, 23(4):2341–2368, 2013

    Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming.SIAM journal on optimization, 23(4):2341–2368, 2013

    2013

  9. [17]

    Deep curriculum learning optimization.SN Computer Science, 1(5):245, 2020

    Henok Ghebrechristos and Gita Alaghband. Deep curriculum learning optimization.SN Computer Science, 1(5):245, 2020

    2020

  10. [18]

    Gradientless descent: High-dimensional zeroth-order optimization.arXiv preprint arXiv:1911.06317, 2019

    Daniel Golovin, John Karro, Greg Kochanski, Chansoo Lee, Xingyou Song, and Qiuyi Zhang. Gradientless descent: High-dimensional zeroth-order optimization.arXiv preprint arXiv:1911.06317, 2019

    1911 arXiv

  11. [19]

    Training dynamics for text summarization models

    Tanya Goyal, Jiacheng Xu, Junyi Jessy Li, and Greg Durrett. Training dynamics for text summarization models. InFindings of the Association for Computational Linguistics: ACL 2022, pages 2061–2073, 2022

    2022

  12. [20]

    Au- tomated curriculum learning for neural networks

    Alex Graves, Marc G Bellemare, Jacob Menick, Remi Munos, and Koray Kavukcuoglu. Au- tomated curriculum learning for neural networks. Ininternational conference on machine learning, pages 1311–1320. Pmlr, 2017

    2017

  13. [21]

    Curriculum learning for facial expression recognition

    Liangke Gui, Tadas Baltrušaitis, and Louis-Philippe Morency. Curriculum learning for facial expression recognition. In2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), pages 505–511. IEEE, 2017

    2017

  14. [22]

    Fastif: Scalable influence functions for efficient model interpretation and debugging

    Han Guo, Nazneen Rajani, Peter Hase, Mohit Bansal, and Caiming Xiong. Fastif: Scalable influence functions for efficient model interpretation and debugging. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10333–10350, 2021

    2021

  15. [23]

    On the power of curriculum learning in training deep networks

    Guy Hacohen and Daphna Weinshall. On the power of curriculum learning in training deep networks. InInternational conference on machine learning, pages 2535–2544. PMLR, 2019

    2019

  16. [24]

    Can perplexity reflect large language model’s ability in long text understanding?ArXiv, abs/2405.06105, 2024

    Yutong Hu, Quzhe Huang, Mingxu Tao, Chen Zhang, and Yansong Feng. Can perplexity reflect large language model’s ability in long text understanding?ArXiv, abs/2405.06105, 2024

    2024 arXiv

  17. [25]

    A review on genetic algorithm: past, present, and future.Multimedia Tools and Applications, 80:8091 – 8126, 2020

    Sourabh Katoch, Sumit Singh Chauhan, and Vijay Kumar. A review on genetic algorithm: past, present, and future.Multimedia Tools and Applications, 80:8091 – 8126, 2020

    2020

  18. [26]

    Strategic data ordering: Enhancing large language model performance through curriculum learning.ArXiv, abs/2405.07490, 2024

    Jisu Kim and Juhwan Lee. Strategic data ordering: Enhancing large language model performance through curriculum learning.ArXiv, abs/2405.07490, 2024

    2024 arXiv

  19. [27]

    Understanding black-box predictions via influence functions

    Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. InInternational conference on machine learning, pages 1885–1894. PMLR, 2017

    2017

  20. [28]

    On the accuracy of influence functions for measuring group effects.Advances in neural information processing systems, 32, 2019

    Pang Wei W Koh, Kai-Siang Ang, Hubert Teo, and Percy S Liang. On the accuracy of influence functions for measuring group effects.Advances in neural information processing systems, 32, 2019

    2019

  21. [29]

    Crossover operators in genetic algorithms: A review

    Padmavathi Kora and Priyanka Yadlapalli. Crossover operators in genetic algorithms: A review. International Journal of Computer Applications, 162(10), 2017

    2017

  22. [30]

    Causal estimation of memorisation profiles

    Pietro Lesci, Clara Meister, Thomas Hofmann, Andreas Vlachos, and Tiago Pimentel. Causal estimation of memorisation profiles. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15616–15635, 2024

    2024

  23. [31]

    Environment curriculum generation via large language models

    William Liang, Sam Wang, Hung-Ju Wang, Osbert Bastani, Dinesh Jayaraman, and Yecheng Ja- son Ma. Environment curriculum generation via large language models. In8th Annual Conference on Robot Learning, 2024

    2024

  24. [32]

    Token-wise influential training data retrieval for large language models

    Huawei Lin, Jikai Long, Zhaozhuo Xu, and Weijie Zhao. Token-wise influential training data retrieval for large language models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 841–860, 2024. 11

    2024

  25. [33]

    A primer on zeroth-order optimization in signal processing and machine learning: Principals, recent advances, and applications.IEEE Signal Processing Magazine, 37(5):43–54, 2020

    Sijia Liu, Pin-Yu Chen, Bhavya Kailkhura, Gaoyuan Zhang, Alfred O Hero III, and Pramod K Varshney. A primer on zeroth-order optimization in signal processing and machine learning: Principals, recent advances, and applications.IEEE Signal Processing Magazine, 37(5):43–54, 2020

    2020

  26. [34]

    Probing across time: What does roberta know and when? InFindings of the Association for Computational Linguistics: EMNLP 2021, pages 820–842, 2021

    Zeyu Liu, Yizhong Wang, Jungo Kasai, Hannaneh Hajishirzi, and Noah A Smith. Probing across time: What does roberta know and when? InFindings of the Association for Computational Linguistics: EMNLP 2021, pages 820–842, 2021

    2021

  27. [35]

    Fine-tuning language models with just forward passes.Advances in Neural Information Processing Systems, 36:53038–53075, 2023

    Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes.Advances in Neural Information Processing Systems, 36:53038–53075, 2023

    2023

  28. [36]

    Teacher–student curriculum learning.IEEE transactions on neural networks and learning systems, 31(9):3732–3740, 2019

    Tambet Matiisen, Avital Oliver, Taco Cohen, and John Schulman. Teacher–student curriculum learning.IEEE transactions on neural networks and learning systems, 31(9):3732–3740, 2019

    2019

  29. [37]

    Scalable language modeling: Wikitext-103 on a single gpu in 12 hours.Proceedings of the SYSML, 18, 2018

    Stephen Merity, Nitish Shirish Keskar, James Bradbury, and Richard Socher. Scalable language modeling: Wikitext-103 on a single gpu in 12 hours.Proceedings of the SYSML, 18, 2018

    2018

  30. [38]

    An analysis of neural language modeling at multiple scales.arXiv preprint arXiv:1803.08240, 2018

    Stephen Merity, Nitish Shirish Keskar, and Richard Socher. An analysis of neural language modeling at multiple scales.arXiv preprint arXiv:1803.08240, 2018

    2018 arXiv

  31. [39]

    Curriculum learning for small code language models.arXiv preprint arXiv:2407.10194, 2024

    Marwa Naïr, Kamel Yamani, Lynda Said Lhadj, and Riyadh Baghdadi. Curriculum learning for small code language models.arXiv preprint arXiv:2407.10194, 2024

    2024 arXiv

  32. [40]

    Random gradient-free minimization of convex functions

    Yurii Nesterov and Vladimir Spokoiny. Random gradient-free minimization of convex functions. Foundations of Computational Mathematics, 17(2):527–566, 2017

    2017

  33. [41]

    Dataman: Data manager for pre-training large language models.arXiv preprint arXiv:2502.19363, 2025

    Ru Peng, Kexin Yang, Yawen Zeng, Junyang Lin, Dayiheng Liu, and Junbo Zhao. Dataman: Data manager for pre-training large language models.arXiv preprint arXiv:2502.19363, 2025

    2025 arXiv

  34. [42]

    Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability.Advances in neural information processing systems, 30, 2017

    Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability.Advances in neural information processing systems, 30, 2017

    2017

  35. [43]

    Sutherland

    Yi Ren and Danica J. Sutherland. Learning dynamics of llm finetuning, 2025

    2025

  36. [44]

    Training dynamics of neural language models

    Naomi Saphra. Training dynamics of neural language models. 2021

    2021

  37. [45]

    Understanding learning dynamics of language models with svcca.arXiv preprint arXiv:1811.00225, 2018

    Naomi Saphra and Adam Lopez. Understanding learning dynamics of language models with svcca.arXiv preprint arXiv:1811.00225, 2018

    2018 arXiv

  38. [46]

    On the dynamics of gender learning in speech translation

    Beatrice Savoldi, Marco Gaido, Luisa Bentivogli, Matteo Negri, and Marco Turchi. On the dynamics of gender learning in speech translation. InProceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pages 94–111. Association for Computational Ling...

    2022

  39. [47]

    Emergent structures and training dynamics in large language models

    Ryan Teehan, Miruna Clinciu, Oleg Serikov, Eliza Szczechla, Natasha Seelam, Shachar Mirkin, and Aaron Gokaslan. Emergent structures and training dynamics in large language models. In Proceedings of BigScience Episode# 5–Workshop on Challenges & Perspectives in Creating Large L...

    2022

  40. [48]

    Memorization without overfitting: Analyzing the training dynamics of large language models.Advances in Neural Information Processing Systems, 35:38274–38290, 2022

    Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics of large language models.Advances in Neural Information Processing Systems, 35:38274–38290, 2022

    2022

  41. [49]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  42. [50]

    How hard can it be? estimating the difficulty of visual search in an image

    Radu Tudor Ionescu, Bogdan Alexe, Marius Leordeanu, Marius Popescu, Dim P Papadopoulos, and Vittorio Ferrari. How hard can it be? estimating the difficulty of visual search in an image. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2157...

    2016

  43. [51]

    The johnson-lindenstrauss transform: an empirical study

    Suresh Venkatasubramanian and Qiushi Wang. The johnson-lindenstrauss transform: an empirical study. In2011 Proceedings of the Thirteenth Workshop on Algorithm Engineering and Experiments (ALENEX), pages 164–173. SIAM, 2011

    2011

  44. [52]

    Curriculum learning for multimedia in the era of large language models

    Xin Wang, Yuwei Zhou, Hong Chen, and Wenwu Zhu. Curriculum learning for multimedia in the era of large language models. InProceedings of the 32nd ACM International Conference on Multimedia, pages 11296–11297, 2024

    2024

  45. [53]

    Stochastic zeroth-order optimization in high dimensions

    Yining Wang, Simon Du, Sivaraman Balakrishnan, and Aarti Singh. Stochastic zeroth-order optimization in high dimensions. InInternational conference on artificial intelligence and statistics, pages 1356–1365. PMLR, 2018

    2018

  46. [54]

    Curriculum learning by transfer learning: Theory and experiments with deep networks

    Daphna Weinshall, Gad Cohen, and Dan Amir. Curriculum learning by transfer learning: Theory and experiments with deep networks. InInternational conference on machine learning, pages 5238–5246. PMLR, 2018

    2018

  47. [55]

    Curriculum learning for natural language understanding

    Benfeng Xu, Licheng Zhang, Zhendong Mao, Quan Wang, Hongtao Xie, and Yongdong Zhang. Curriculum learning for natural language understanding. InProceedings of the 58th annual meeting of the association for computational linguistics, pages 6095–6104, 2020

    2020

  48. [56]

    To repeat or not to repeat: Insights from scaling llm under token-crisis, 2023

    Fuzhao Xue, Yao Fu, Wangchunshu Zhou, Zangwei Zheng, and Yang You. To repeat or not to repeat: Insights from scaling llm under token-crisis, 2023

    2023

  49. [57]

    Dpzero: dimension- independent and differentially private zeroth-order optimization

    Liang Zhang, Kiran Koshy Thekumparampil, Sewoong Oh, and Niao He. Dpzero: dimension- independent and differentially private zeroth-order optimization. InInternational Workshop on Federated Learning in the Age of Foundation Models in Conjunction with NeurIPS 2023, 2023

    2023

  50. [58]

    An empirical exploration of curriculum learning for neural machine translation.arXiv preprint arXiv:1811.00739, 2018

    Xuan Zhang, Gaurav Kumar, Huda Khayrallah, Kenton Murray, Jeremy Gwinnup, Marianna J Martindale, Paul McNamee, Kevin Duh, and Marine Carpuat. An empirical exploration of curriculum learning for neural machine translation.arXiv preprint arXiv:1811.00739, 2018

    2018 arXiv

  51. [59]

    Frames: Boosting llms with a four-quadrant multi-stage pretraining strategy.arXiv preprint arXiv:2502.05551, 2025

    Xuemiao Zhang, Feiyu Duan, Liangyu Xu, Yongwei Zhou, Sirui Wang, Rongxiang Weng, Jingang Wang, and Xunliang Cai. Frames: Boosting llms with a four-quadrant multi-stage pretraining strategy.arXiv preprint arXiv:2502.05551, 2025

    2025 arXiv

  52. [60]

    Preference curriculum: Llms should always be pretrained on their preferred data.arXiv preprint arXiv:2501.13126, 2025

    Xuemiao Zhang, Liangyu Xu, Feiyu Duan, Yongwei Zhou, Sirui Wang, Rongxiang Weng, Jingang Wang, and Xunliang Cai. Preference curriculum: Llms should always be pretrained on their preferred data.arXiv preprint arXiv:2501.13126, 2025

    2025 arXiv

  53. [61]

    Revisiting zeroth-order optimization for memory-efficient llm fine-tuning: A benchmark.arXiv preprint arXiv:2402.11592, 2024

    Yihua Zhang, Pingzhi Li, Junyuan Hong, Jiaxiang Li, Yimeng Zhang, Wenqing Zheng, Pin-Yu Chen, Jason D Lee, Wotao Yin, Mingyi Hong, et al. Revisiting zeroth-order optimization for memory-efficient llm fine-tuning: A benchmark.arXiv preprint arXiv:2402.11592, 2024

    2024 arXiv

  54. [62]

    Billion-scale network embedding with iterative random projection

    Ziwei Zhang, Peng Cui, Haoyang Li, Xiao Wang, and Wenwu Zhu. Billion-scale network embedding with iterative random projection. In2018 IEEE international conference on data mining (ICDM), pages 787–796. IEEE, 2018

    2018

  55. [63]

    Curriculum learning for deep generative models with clustering.arXiv preprint arXiv:1906.11594, 2019

    Deli Zhao, Jiapeng Zhu, Zhenfang Guo, and Bo Zhang. Curriculum learning for deep generative models with clustering.arXiv preprint arXiv:1906.11594, 2019

    1906 arXiv

  56. [64]

    An empirical study of memorization in nlp

    Xiaosen Zheng and Jing Jiang. An empirical study of memorization in nlp. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6265–6278, 2022. 13 Contents A Technical Details 15 A.1 Precomputation in Update Te...

    2022

  57. [65]

    , T}}, with|ST |=T!

    Population Initialization:Randomly select N sample orders POP={πi}N i=1 from ST as the initial populations, whereS T ={π|πis a permutation of{1, . . . , T}}, with|ST |=T!

  58. [66]

    Retain the top 50% individuals with the highest fitness scores for reproduction, and discard the rest

    Fitness Selection:For each πi ∈POP , evaluate the model performance R(γπi T ,D val) as its fitness, where γπi T is estimated via the FUT method. Retain the top 50% individuals with the highest fitness scores for reproduction, and discard the rest

  59. [67]

    Specifically, randomly choose two crossover points l and r such that 1≤l < r≤T , then exchange the subsequences πa l:r and πb l:r between the parents

    Crossover:Generate new children by applying the partially matched crossover (PMX) [ 29] to randomly selected parent pairs πa and πb from the surviving population. Specifically, randomly choose two crossover points l and r such that 1≤l < r≤T , then exchange the subsequences πa...

  60. [68]

    This operation introduces diversity and prevents premature convergence

    Mutation:With a predefined mutation probability pm, randomly select two indices i and j in πc and swap their values: πc ←π c i↔j. This operation introduces diversity and prevents premature convergence

  61. [69]

    The updated population then forms the basis for the next generation

    Replacement:Insert the newly generated children into the population, replacing the dis- carded individuals. The updated population then forms the basis for the next generation. By iteratively performing 2-5 steps over a fixed number of generations K, or until a convergence cri...

  62. [70]

    The accuracy of our estimates relies on the validity of Taylor expansions, particularly when higher-order nonlinearities dominate the optimization dynamics—scenarios where our first- and second-order approximations may fall short

  63. [71]

    Although the use of random projection significantly reduces memory overhead, it may introduce approximation noise, especially for models with extremely large parameter spaces

  64. [72]

    We evaluate the effectiveness of our FUT framework solely based on perplexity performance. This is because downstream natural language understanding and reasoning tasks typically re- quire large-scale models, which are infeasible to retrain repeatedly under varying conditions....

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.