Pith. sign in

REVIEW 4 major objections 6 minor 45 references

Steering the Noise: Turning Random Perturbations into Effective Descent for Memory-Efficient LLM Fine-Tuning

T0 review · 4 major / 6 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By ranking a handful of random perturbations and combining the low-loss ones, zeroth-order fine-tuning of large language models can converge far faster than standard MeZO while keeping the memory cost of forward-only training.

desk verdict Plausible guiding-vector idea and broad experiments, but unequal forward-pass budgets and a Lemma 4 algebra slip undermine the strongest efficiency and convergence claims. read the letter →

arxiv 2601.04710 v2 pith:PSQOCR56 submitted 2026-01-08 cs.CL cs.LG

Feihu Jin , Shipeng Cen , Ying Tan This is my paper
classification cs.CLcs.LG
keywords zeroth-orderoptimizationMeZOguidingvectorgreedyperturbationLLMfine-tuningmemory-efficientgradientalignmentforwardpasses
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to establish that the random noise used in zeroth-order (forward-only) fine-tuning of large language models can itself be steered into a better descent direction. Instead of updating along a single random perturbation — the MeZO recipe — the authors propose drawing a small pool of perturbations, evaluating each with a forward pass, and either picking the single best one (greedy) or averaging the best and worst groups to form a 'guiding vector' (GV). They prove that both strategies inflate the component of the estimated gradient that points along the true gradient: the aligned component grows like O(log k) for greedy and O(s log k) for the guiding vector, versus O(1) for vanilla MeZO. Empirically, on OPT-13B the resulting methods beat all zeroth-order baselines on all 11 benchmark tasks and beat gradient-based fine-tuning on 9 of 11, while keeping memory at inference levels. The reason to care: if the effect is real, forward-only fine-tuning of very large models becomes substantially more competitive with backpropagation.

What carries the argument

The load-bearing object is the 'guiding vector' v, built as the mean of the elite (lowest-loss) perturbations minus the mean of the non-elite (highest-loss) perturbations among M Gaussian draws, or its greedy cousin z*, the single lowest-loss perturbation. The paper's measure of effectiveness is the ratio of the component of the estimated gradient parallel to the true gradient to the perpendicular component (ratio1) and to the true gradient norm (ratio2). The identity that carries the argument is Lemma 4's estimate ratio2 ≈ 8 s log k, which quantifies how much the contrast between low-loss and high-loss perturbations amplifies the useful signal; this is what converts a random pool into a nea

What would settle it

Run MeZO-GV and MeZO with the same total number of forward passes (e.g., 40,000 each) and compare final task accuracy and wall-clock time; if MeZO-GV does not beat MeZO under equal query budgets, the 'faster convergence' claim is a step-count artifact. A second, more local check: at ε=1e-3, compute g^T z_i for the pool and compare this ordering with the loss ordering used to select z* — if the Spearman correlation is low, Lemmas 3 and 4 do not apply.

Watch

Extended reading notes

Core claim

The authors' central claim is that a zeroth-order gradient estimator built on a single random Gaussian perturbation wastes most of its information, because the estimated direction is dominated by the perpendicular (noise) component. They show that by evaluating M candidate perturbations on the loss and using the loss ranking to select or combine them, the resulting perturbation has a much larger projection onto the true gradient. Concretely, the paper defines v as the difference between the mean of the lowest-loss perturbations and the mean of the highest-loss perturbations, and shows that the parallel-to-true-gradient ratio ||V_|| ||/||g|| is approximately 8 s log k for this guiding vector

Load-bearing premise

The reported speed-ups rest on the premise that halving the step budget (10,000 vs 20,000 steps) with each step costing M+2 forward passes still nets no more wall-clock time than the baseline; if that parity fails, the gains are partially an artifact of a 1.5–3× larger query budget — and the theory additionally assumes the loss at θ+ε z_i ranks perturbations the same way the true-gradient projection g^T z_i would.

Editorial extensions

If this is right

  • If the central claim holds, zeroth-order fine-tuning can reach a given accuracy in roughly half the training steps, making forward-only tuning of 10B+ parameter models practical on a single GPU.
  • Because the trick operates entirely on the perturbation distribution, it can be dropped into any ZO optimizer that uses random directions — the paper demonstrates this with MeZO and SubZero, in full, LoRA, and prefix-tuning modes — so existing ZO pipelines inherit the speedup without architectural changes.
  • The extra forward passes used to build the guiding vector are the only added cost, and they remain backprop-free, so the memory profile of MeZO (inference-level memory) is preserved.
  • The theory predicts the advantage grows with the pool size M and the elite fraction s (via 8 s log k), so users can trade a few extra forward passes for a larger per-step descent.
  • Against multi-query SPSA baselines that spend the same number of forward passes per step, the GV method achieves higher accuracy in less wall-clock time, indicating the ranking/combining step extracts more value from each query than simply averaging independent SPSA estimates.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A fair comparison that the paper does not fully make would hold the total forward-pass budget constant (e.g., 40k forward passes for both methods) rather than halving the step count; if the benefit persists, the mechanism—not the extra queries—is the cause.
  • The theoretical lemmas assume that loss values at θ+ε z_i rank the same as dot products g^T z_i (local linearity). A reader could test this directly by computing the Spearman correlation between the two orderings at the actual ε=1e-3 used in the experiments; the paper does not report this check.
  • If the alignment amplification is as strong as Lemma 4 suggests, the same loss-ranked selection could be applied block-wise (per layer or per LoRA rank) to reduce the variance of individual perturbation directions, potentially improving convergence further without increasing the forward budget.
  • The diminishing returns shown when M grows from 4 to 12 suggest the optimal pool size depends on the task's loss landscape; a task-adaptive stopping rule (increase M until the selected direction stops changing) would be a natural extension.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes two plug-and-play modifications to zeroth-order (ZO) optimizers such as MeZO and SubZero: MeZO-GV, which samples M random perturbations, evaluates their losses on the current minibatch, splits them into elite/non-elite groups, and uses their difference as a guiding vector v for a two-forward-pass finite-difference step; and MeZO-Greedy, which selects the single lowest-loss perturbation z* from M candidates and uses it for the finite-difference step. The theoretical section (Lemmas 2-4) claims that these strategies increase the gradient-aligned component of the ZO update, yielding larger per-step descent. Experiments on OPT-1.3B/13B/30B and Llama2-7B/13B across classification, multiple-choice, and QA tasks report accuracy gains over MeZO, SubZero, and several ZO baselines, with memory usage comparable to MeZO. The paper concludes that prior-informed perturbation selection improves convergence and final accuracy while preserving the memory efficiency of forward-only optimization.

Significance. If the claims were fully established, the paper would make a useful practical contribution: the proposed mechanisms are simple, architecture-agnostic, and memory-efficient, and the experimental scope is broad (multiple model families, scales, PEFT schemes, and tasks). The strengths include the explicit algorithmic description, the provision of a code link, the memory-usage measurements, and a comparison against a multi-query SPSA baseline in Section V.E. However, the two load-bearing pillars of the paper — the theoretical scaling of the guiding-vector alignment and the claim of equal computational cost in the main experiments — both need correction. The theoretical centerpiece contains an algebraic error, and the main accuracy tables compare against baselines using a substantially smaller forward-pass budget. These issues compromise the central 'same computational cost, faster convergence' claim, although they do not definitively show the method is ineffective.

major comments (4)
  1. [Section IV, Lemma 4] The proof contains an algebraic error that reverses the claimed scaling with s. The text defines zhat = (1/s)(m g + N), then writes V = zhat zhat^T g = (1/s)(m^2 g + m N). Since zhat = (m g + N)/s, the correct expression is V = (1/s^2)(m^2 g + m N). Consequently ||V_parallel||/||g|| = m^2 / s^2 ≈ 8 log k, not 8 s log k. The claimed linear amplification in s is therefore unsupported. This affects Table I (last row, O(s log k)) and the qualitative conclusion that increasing s or k 'significantly strengthens' the parallel component beyond the constant-factor gain. The corrected formula still gives a constant-factor improvement over standard MeZO, but it removes the theoretical argument that the guiding-vector strategy increasingly outperforms the greedy strategy as s grows.
  2. [Section V, Setup and Tables III-VIII] The 'identical computational cost' comparison is invalid. Each MeZO-GV or MeZO-Greedy iteration uses M forward passes to evaluate the candidate perturbations in Algorithm 2/4 plus two forward passes for the finite-difference estimate in Eq. 6, i.e., M+2 forward passes per step. With M=4 (Table III), 10,000 steps consume 60,000 forward passes, while MeZO/SubZero at 20,000 steps consume 40,000 forward passes. Thus the main tables compare GV methods with a 1.5x query-budget advantage (up to 3x for 5,000-step runs), so the reported accuracy and convergence gains may reflect additional compute rather than algorithmic superiority. The claim 'To maintain identical computational cost' should be restated in terms of forward passes or wall-clock time, and the main experiments need to be rerun or reanalyzed at equal query budgets.
  3. [Section IV, Lemmas 3-4 and Algorithm 2/4] The theory assumes that ranking candidates by L(theta + epsilon z_i) is equivalent to ranking them by z_i^T g, but this holds only up to second-order terms and minibatch noise. The extreme-value calculations (Y_1 = min z_i^T g, order statistics, etc.) describe the algorithm only if the loss ranking and gradient projection ranking coincide. The paper does not check this local-linearity assumption at the actual perturbation scales used in Table III (epsilon = 1e-3 or 1e-2). Without such a check, the link between Lemma 3-4 and the implemented selection rule is not established. A simple empirical test — measuring the rank correlation between L(theta + epsilon z_i) and g^T z_i on a few checkpoints — would clarify this.
  4. [Section IV and V (general claim)] The theoretical analysis establishes per-step directional alignment ratios, not convergence rates. The abstract and conclusion infer 'faster convergence' and 'improved convergence rates,' but no optimization bound maps the corrected ratio to iteration complexity under a forward-pass budget. Given that the method spends M extra forward passes per step, the theory does not by itself support the central efficiency claim. The empirical evidence must therefore carry the burden, which makes the query-budget confound in the main experiments even more consequential.
minor comments (6)
  1. [Section V, Table VII caption] Typo: 'Tabel VII' should be 'Table VII'.
  2. [References] SubZero is cited as [14], but reference [14] in the bibliography is Hansen and Ostermeier's CMA-ES paper; the SubZero paper appears as [30] (Yu et al.). The in-text citation needs to be corrected.
  3. [Section V.E, Table X] The text quotes training times that do not match the table (e.g., 'reducing training time from 1.44h to 0.84h' versus the table's 10.24h and 4.52h for BoolQ q=2). Please reconcile the narrative with the numerical results.
  4. [Algorithm 2] The subroutine stores only seeds and losses, then later reconstructs the z_i from the seeds. This is correct only if the random-number generator is deterministic and the parameter perturbations are applied and reverted exactly; please state this reproducibility requirement explicitly.
  5. [Figures and notation] The notation 'q' is overused: q denotes the number of SPSA evaluations in Eq. (1), but in Table X 'MeZO-GV (q=2)' seems to denote something else. Also, the term 'fireworks size M' (Algorithm 1) is nonstandard; 'candidate pool size' would be clearer.
  6. [Figure 6] The cosine similarity values reported (about 0.02-0.12) are very small; the text says GV 'closely follows the true gradient,' which is an overstatement. Please add a note that these are typical magnitudes for high-dimensional ZO estimates and that the comparison is relative, not absolute.

Circularity Check

0 steps flagged · score 2.0 of 10

No circularity in the derivation; the only self-citation is a non-load-bearing related-work reference.

full rationale

No circular step can be exhibited. The guiding vector v is selected from loss evaluations (Eq. 5) and then used in a two-point SPSA estimator (Eq. 6), which is a legitimate use of function values to shape an update, not a quantity fitted to the target benchmark. Lemmas 2-4 derive alignment ratios from Gaussian order statistics; the only questionable step is that selection by loss is assumed to equal selection by z_i^T g (local linearity), an unvalidated modeling assumption rather than a definitional circularity. The per-step decrease claim follows from the update rule plus Taylor expansion, so it is not circular. The experiments are checked against external baselines (MeZO, SubZero, ZO-AdaMU, HiZOO, FT), so the empirical claims are self-contained. The only self-citation (ref. [17], the authors' prior DFO-LoRA work, likely the intended source for the garbled [33] in Sec. II.C) is used for motivation and is not load-bearing. The unequal forward-pass budget (M+2 vs 2 per step) is a real cost-fairness flaw but is an experimental-design issue, not circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The method introduces no new physical or structural entities; the guiding vector is an algorithmic quantity derived from sampled perturbations. The central claim rests on hyperparameters M, alpha, epsilon, and on a local-linearity assumption about the loss landscape, plus the premise that step-budget halving matches computational cost.

free parameters (4)
  • M (candidate pool size) = 4 (default; 2 in some runs; swept 4-12 in Fig. 5)
    Chosen by hand. The convergence-rate claims scale as log M (or s log k in the paper's Lemma 4), so results depend on this budget.
  • alpha (elite split ratio) = 0.5
    Default split between elite/non-elite groups in Eq. (5); enters the guiding-vector formula.
  • epsilon (perturbation scale) = 1e-3 (FT), 1e-2 (LoRA), 1e-1 (Prefix)
    Inherited from MeZO, but the theory's Taylor linearization depends on it being small; reported values are chosen per method.
  • Learning rate / rank / subspace frequency = grid-selected per task/model (e.g., 1e-7-5e-7; rank 32/64; frequency 500/1000/2000)
    Best hyperparameters are selected on validation; neither the selection procedure nor error bars around the chosen values are reported.
assumptions (5)
  • domain assumption Local linearity of the loss: f(theta+epsilon z) ≈ f(theta) + epsilon grad f(theta)^T z over the perturbation radius
    Used throughout Lemmas 3 and 4 to map loss-ranking of perturbations to g^T z. For epsilon=1e-3 in high-dimensional, non-smooth transformer losses this is approximate and not empirically checked.
  • domain assumption Gradient normalized to unit norm in the alignment lemmas
    Lemmas 2-4 assume ||g||=1. Actual gradient norms vary by task and model, so quantitative cross-task comparisons of ratio2 are only meaningful after rescaling.
  • domain assumption Minibatch loss at theta±epsilon z with the same minibatch is a stable proxy for ranking candidate directions
    The selection and finite-difference evaluations use one minibatch B_t; minibatch noise is not modeled in the theory.
  • standard math Matrix Bernstein inequality
    Used in Lemma 1 to bound ||S_k - I_d||.
  • standard math Gaussian order statistics and tail approximation: Phi^{-1}(1/(k+1)) ≈ sqrt(2 log k) for the minimum of k standard normals
    Used to estimate Y_1 and the sums of extreme Y_i in Lemmas 3 and 4.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Steering the Noise: Turning Random Perturbations into Effective Descent for Memory-Efficient LLM Fine-Tuning." pith.science (2026). https://pith.science/paper/PSQOCR56

@misc{pith2026260104710,
  author       = {Pith},
  title        = {Pith review of: Steering the Noise: Turning Random Perturbations into Effective Descent for Memory-Efficient LLM Fine-Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PSQOCR56}},
  note         = {Machine review of arXiv:2601.04710}
}
read the original abstract

Fine-tuning large language models (LLMs) achieves strong performance but is often limited by the memory overhead of backpropagation. Zeroth-order (ZO) optimization avoids this overhead by estimating gradients through forward passes alone, yet it typically converges slowly because random Gaussian perturbations yield high-variance gradient estimates in high-dimensional parameter spaces. In this paper, we propose a plug-and-play framework that turns random perturbations into more effective descent directions. The key idea is to draw a small pool of candidate perturbations, evaluate their loss values, and then select or combine those that are best aligned with the optimization objective. We develop two instantiations of this idea: MeZO-GV, which forms a guiding vector from the contrast between low-loss and high-loss perturbation groups, and MeZO-Greedy, which keeps the single best perturbation within a fixed evaluation budget. We theoretically show that both strategies yield a larger per-step reduction in the objective than standard ZO estimation, leading to improved convergence rates. Experiments on LLMs of different scales and architectures confirm that the proposed methods integrate naturally with existing ZO optimizers and consistently improve convergence speed and task accuracy. On OPT-13B, our approach outperforms all ZO baselines across 11 benchmarks and exceeds gradient-based methods on 9 of them, while retaining the memory efficiency of forward-only optimization.

Figures

Figures reproduced from arXiv: 2601.04710 by the authors.

Figure 1
Figure 1. The training loss curves for the WSC, SST-2, and BoolQ tasks are evaluated using the OPT-1.3B model. Our proposed methods (MeZO-Greedy and [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Validation Accuracy on SST2 and BoolQ Tasks for Llama2-7B and Llama2-13B. All experiments are conducted with a batch size of 16. For LoRA [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Training loss on SST2, BoolQ, and CB Tasks for OPT-1.3B/13B Models. We employ a learning rate of 2e-7. All experiments are conducted with a [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Training loss on BoolQ and RTE Tasks with Llama2-7B Model. [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 6
Figure 6. Figure 6: Cosine similarity between the estimated gradient [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

45 extracted references · 8 linked inside Pith

  1. [1]

    Language models are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amode...

    2020

  2. [2]

    Gpt-4 technical report,

    O. J. Achiam, S. Adler, S. Agarwal, and et al., “Gpt-4 technical report,” 2023

    2023

  3. [3]

    Learning represen- tations by back-propagating errors,

    D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning represen- tations by back-propagating errors,” Nature, vol. 323, pp. 533–536, 1986

    1986

  4. [4]

    LoRA: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” In International Conference on Learning Representations, 2022

    2022

  5. [5]

    Parameter-efficient transfer learning for NLP,

    N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter-efficient transfer learning for NLP,” in Proceedings of the 36th International Conference on Machine Learning (K. Chaudhuri and R. Salakhutdinov, eds.), vol. 97 of Proceedings of Machine Learning Research, pp. 2790–2799, PMLR, 09–15 Jun 2019

    2019

  6. [6]

    Prefix-tuning: Optimizing continuous prompts for generation,

    X. L. Li and P. Liang, “Prefix-tuning: Optimizing continuous prompts for generation,” In ACL, 2021

    2021

  7. [7]

    Opt: Open pre-trained transformer language models,

    S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. T. Diab, X. Li, X. V . Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettle- moyer, “Opt: Open pre-trained transformer language models,” ArXiv, vol. abs/2205.01068, 2022. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 12

    arXiv 2022

  8. [8]

    Fine-tuning language models with just forward passes,

    S. Malladi, T. Gao, E. Nichani, A. Damian, J. D. Lee, D. Chen, and S. Arora, “Fine-tuning language models with just forward passes,” In Thirty- seventh Conference on Neural Information Processing Systems, 2023

    2023

Show all 45 references
  1. [9]

    Sparse mezo: Less parameters for better performance in eroth-order llm fine- tuning,

    Y . Liu, Z. Zhu, C. Gong, M. Cheng, C.-J. Hsieh, and Y . You, “Sparse mezo: Less parameters for better performance in eroth-order llm fine- tuning,” ArXiv, vol. abs/2402.15751, 2024

    2024

  2. [10]

    Zeroth-order fine-tuning of LLMs with extreme sparsity,

    W. Guo, J. Long, Y . Zeng, Z. Liu, X. Yang, Y . Ran, J. R. Gardner, O. Bastani, C. D. Sa, X. Yu, B. Chen, and Z. Xu, “Zeroth-order fine-tuning of LLMs with extreme sparsity,” in 2nd Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resou...

    2024

  3. [11]

    Secondorder fine-tuning without pain for llms: A hessian informed zeroth-order optimizer,

    Y . Zhao, S. Dang, H. Ye, G. Dai, Y . Qian, and I. W.-H. Tsang, “Secondorder fine-tuning without pain for llms: A hessian informed zeroth-order optimizer,” ArXiv, vol. abs/2402.15173, 2024

    2024 arXiv

  4. [12]

    AdaZeta: Adaptive zeroth-order tensor-train adaption for memory- efficient large language models fine-tuning,

    Y . Yang, K. Zhen, E. Banijamali, A. Mouchtaris, and Z. Zhang, “AdaZeta: Adaptive zeroth-order tensor-train adaption for memory- efficient large language models fine-tuning,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (Y . Al Onai...

    2024

  5. [13]

    Multivariate stochastic approximation using a simultaneous perturbation gradient approximation,

    J. Spall, “Multivariate stochastic approximation using a simultaneous perturbation gradient approximation,” IEEE Transactions on Automatic Control, vol. 37, no. 3, pp. 332–341, 1992

    1992

  6. [14]

    Completely derandomized self- adaptation in evolution strategies,

    N. Hansen and A. Ostermeier, “Completely derandomized self- adaptation in evolution strategies,” Evolutionary Computation, vol. 9, no. 2, pp. 159–195, 2001

    2001

  7. [15]

    Black-box tuning for language-model-as-a-service,

    T. Sun, Y . Shao, H. Qian, X. Huang, and X. Qiu, “Black-box tuning for language-model-as-a-service,” in Proceedings of ICML, 2022

    2022

  8. [16]

    BBTv2: Towards a gradient-free future with large language models,

    T. Sun, Z. He, H. Qian, Y . Zhou, X. Huang, and X. Qiu, “BBTv2: Towards a gradient-free future with large language models,” in Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing (Y . Goldberg, Z. Kozareva, and Y . Zhang, eds.), (Abu Dhabi,...

    2022

  9. [17]

    Derivative-free optimization for low-rank adaptation in large language models,

    F. Jin, Y . Liu, and Y . Tan, “Derivative-free optimization for low-rank adaptation in large language models,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 32, pp. 4607–4616, 2024

    2024

  10. [18]

    Zero-offload: Democratizing billion-scale model training,

    J. Ren, S. Rajbhandari, R. Y . Aminabadi, O. Ruwase, S. Yang, M. Zhang, D. Li, and Y . He, “Zero-offload: Democratizing billion-scale model training,” ArXiv, vol. abs/2101.06840, 2021

    2021 arXiv

  11. [19]

    Bpipe: Memory-balanced pipeline parallelism for training large language models,

    T. Kim, H. Kim, G.-I. Yu, and B.-G. Chun, “Bpipe: Memory-balanced pipeline parallelism for training large language models,” in International Conference on Machine Learning, 2023

    2023

  12. [20]

    Deepzero: Scaling up zeroth- order optimization for deep model training,

    A. Chen, Y . Zhang, J. Jia, J. Diffenderfer, K. Parasyris, J. Liu, Y . Zhang, Z. Zhang, B. Kailkhura, and S. Liu, “Deepzero: Scaling up zeroth- order optimization for deep model training,” in The Twelfth International Conference on Learning Representations, 2024

    2024

  13. [21]

    Distributed zero-order algorithms for nonconvex multiagent optimization,

    Y . Tang and N. Li, “Distributed zero-order algorithms for nonconvex multiagent optimization,” 2019 57th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pp. 781–786, 2019

    2019

  14. [22]

    A zeroth-order block coordinate descent algorithm for huge-scale black-box optimization,

    H. Cai, Y . Lou, D. Mckenzie, and W. Yin, “A zeroth-order block coordinate descent algorithm for huge-scale black-box optimization,” ArXiv, vol. abs/2102.10707, 2021

    2021 arXiv

  15. [23]

    signsgd via zeroth-order oracle,

    S. Liu, P.-Y . Chen, X. Chen, and M. Hong, “signsgd via zeroth-order oracle,” in International Conference on Learning Representations, 2019

    2019

  16. [24]

    Gradientless descent: High-dimensional zeroth-order optimization,

    D. Golovin, J. Karro, G. Kochanski, C. Lee, X. Song, and Q. Zhang, “Gradientless descent: High-dimensional zeroth-order optimization,” in International Conference on Learning Representations, 2020

    2020

  17. [25]

    Simple random search of static linear policies is competitive for reinforcement learning,

    H. Mania, A. Guy, and B. Recht, “Simple random search of static linear policies is competitive for reinforcement learning,” in Neural Information Processing Systems, 2018

    2018

  18. [26]

    The forward-forward algorithm: Some preliminary in- vestigations,

    G. E. Hinton, “The forward-forward algorithm: Some preliminary in- vestigations,” ArXiv, vol. abs/2212.13345, 2022

    2022 arXiv

  19. [27]

    Variance-reduced zeroth-order methods for fine-tuning language models,

    T. Gautam, Y . Park, H. Zhou, P. Raman, and W. Ha, “Variance-reduced zeroth-order methods for fine-tuning language models,” in Forty-first International Conference on Machine Learning, 2024

    2024

  20. [28]

    Revisiting zerothorder optimization for memory efficient llm fine-tuning: A benchmark,

    Y . Zhang, P. Li, J. Hong, J. Li, Y . Zhang, W. Zheng, P.-Y . Chen, J. D. Lee, W. Yin, M. Hong, Z. Wang, S. Liu, and T. Chen, “Revisiting zerothorder optimization for memory efficient llm fine-tuning: A benchmark,” in Forty-first International Conference on Machine Learning, 2024

    2024

  21. [29]

    Zo-adamu optimizer: Adapting perturbation by the momentum and uncertainty in zeroth order optimization,

    S. Jiang, Q. Chen, Y . Pan, Y . Xiang, Y . Lin, X. Wu, C. Liu, and X. Song, “Zo-adamu optimizer: Adapting perturbation by the momentum and uncertainty in zeroth order optimization,” in Thirty-Eighth AAAI Confer- ence on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conferen...

    2024

  22. [30]

    Subzero: Random sub- space zeroth-order optimization for memory-efficient LLM fine-tuning,

    Z. Yu, P. Zhou, S. Wang, J. Li, and H. Huang, “Subzero: Random sub- space zeroth-order optimization for memory-efficient LLM fine-tuning,” CoRR, vol. abs/2410.08989, 2024

    2024 arXiv

  23. [31]

    Superglue: A stickier benchmark for generalpurpose language understanding systems,

    A. Wang, Y . Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, “Superglue: A stickier benchmark for generalpurpose language understanding systems,” in NeurIPS, 2019

    2019

  24. [32]

    The commitment- bank: Investigating projection in naturally occurring discourse,

    M.-C. De Marneffe, M. Simons, and J. Tonhauser, “The commitment- bank: Investigating projection in naturally occurring discourse,” Proceed- ings of Sinn und Bedeutung, vol. 23, pp. 107–124, Jul. 2019

    2019

  25. [33]

    Choice of plausible alternatives: An evaluation of commonsense causal reasoning,

    M. Roemmele, C. A. Bejan, and A. S. Gordon, “Choice of plausible alternatives: An evaluation of commonsense causal reasoning,” in Logical Formalizations of Commonsense Reasoning, Papers from the 2011 AAAI Spring Symposium, Technical Report SS-11-06, Stanford, California, USA, ...

    2011

  26. [34]

    Looking beyond the surface: A challenge set for reading comprehen- sion over multiple sentences,

    D. Khashabi, S. Chaturvedi, M. Roth, S. Upadhyay, and D. Roth, “Looking beyond the surface: A challenge set for reading comprehen- sion over multiple sentences,” in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistic...

    2018

  27. [35]

    Benchmarking applied seman- tic inference: The PASCAL recognising textual entailment challenges,

    R. Bar-Haim, I. Dagan, and I. Szpektor, “Benchmarking applied seman- tic inference: The PASCAL recognising textual entailment challenges,” in Language, Culture, Computation. Computing - Theory and Technology - Essays Dedicated to Yaacov Choueka on the Occasion of His 75th Birt...

    2014

  28. [36]

    WiC: the word-in-context dataset for evaluating context-sensitive meaning representations,

    M. T. Pilehvar and J. Camacho-Collados, “WiC: the word-in-context dataset for evaluating context-sensitive meaning representations,” in Pro- ceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo-...

    2019

  29. [37]

    The winograd schema challenge,

    H. J. Levesque, “The winograd schema challenge,” in Logical For- malizations of Commonsense Reasoning, Papers from the 2011 AAAI Spring Symposium, Technical Report SS 11-06, Stanford, California, USA, March 21-23, 2011, AAAI, 2011

    2011

  30. [38]

    BoolQ: Exploring the surprising difficulty of natural yes/no questions,

    C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova, “BoolQ: Exploring the surprising difficulty of natural yes/no questions,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human La...

    2019

  31. [39]

    Record: Bridging the gap between human and machine commonsense reading comprehension,

    S. Zhang, X. Liu, J. Liu, J. Gao, K. Duh, and B. V . Durme, “Record: Bridging the gap between human and machine commonsense reading comprehension,” CoRR, vol. abs/1810.12885, 2018

    2018 arXiv

  32. [40]

    Recursive deep models for semantic compositionality over a sentiment treebank,

    R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng, and C. Potts, “Recursive deep models for semantic compositionality over a sentiment treebank,” in Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (D. Yarowsky, T. Baldwin, A...

    2013

  33. [41]

    SQuAD: 100,000+ questions for machine comprehension of text,

    P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, “SQuAD: 100,000+ questions for machine comprehension of text,” in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (J. Su, K. Duh, and X. Carreras, eds.), (Austin, Texas), pp. 2383–2392, As...

    2016

  34. [42]

    DROP: A reading comprehension benchmark requiring discrete reason- ing over paragraphs,

    D. Dua, Y . Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner, “DROP: A reading comprehension benchmark requiring discrete reason- ing over paragraphs,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: ...

    2019

  35. [43]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. R. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, and et.al, “Llama 2: Open foundation and fine-tuned chat models,” ArXiv, vol. abs/2307.09288, 2023

    2023 arXiv

  36. [44]

    Making pre-trained language models better few-shot learners,

    T. Gao, A. Fisch, and D. Chen, “Making pre-trained language models better few-shot learners,” in ACL, 2021

    2021

  37. [45]

    Exploiting cloze-questions for few-shot text classification and natural language inference,

    T. Schick and H. Sch¨utze, “Exploiting cloze-questions for few-shot text classification and natural language inference,” in EACL, 2021

    2021

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.