Pith. sign in

REVIEW 4 major objections 5 minor 26 references

Decoupled Relative Learning Rate Schedules

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Per-component relative learning-rate schedules, tuned on a small model and reused on larger ones, speed up Transformer pretraining by up to 23% for MoE models and 17% for dense models.

desk verdict A practical, mostly convincing empirical method for per-component LR schedules in Transformers, with real transfer value for MoE training but a few reproducibility and comparison gaps that keep it from being a slam dunk. read the letter →

arxiv 2507.03526 v1 pith:ZHE5LMQI submitted 2025-07-04 cs.LG

Jan Ludziejewski , Jan Ma{l}aśnicki , Maciej Pióro , Micha{l} Krutul , Kamil Ciebiera , Maciej Stefaniak , Jakub Krajewski , Piotr Sankowski
show 3 more authors
Marek Cygan Kamil Adamczewski Sebastian Jaszczur
This is my paper
classification cs.LG
keywords relativelearningratesdecoupledrateschedulesTransformerpretrainingMixtureofExpertssampleefficiencyhyperparametertransfercosinescheduletrainingstability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RLRS is the claim that Transformer components should not share a single learning-rate schedule. The paper defines, for each module (Embedding, Unembedding, Attention, Feed-Forward, and for Mixture-of-Experts models also Router and Experts), two multipliers that reshape the global cosine decay. Tuning these multipliers on a 34M-active-parameter model speeds up convergence by up to 23% for MoE and up to 17% for dense Transformers. The same multipliers, reused without retuning, give up to 19% faster training on models up to 27x larger in active parameters. If the transfer finding holds, this is a low-cost way to reduce pretraining compute and evidence that some hyperparameters are architecture-dependent rather than scale-dependent.

What carries the argument

The load-bearing object is the decoupled cosine schedule. With warmup, each module $m$ receives a learning rate $\eta_t^m = \eta_{base}\left[\alpha_{end}\lambda^m_{end} + \frac{1}{2}(\lambda^m_{start}-\alpha_{end}\lambda^m_{end})(1+\cos(\pi t/T))\right]$, so the schedule is fully specified by $\eta_{base}$, $\alpha_{end}$, and the per-module multipliers. The second piece is the transfer procedure: tune the multipliers by local search on a small proxy model, then reuse them for larger models, retuning only the base learning rate. The empirical claim is that the multipliers encode architecture-specific dynamics rather than size-specific ones.

What would settle it

Retrain the largest MoE model with each transferred multiplier varied one local-search step above and below the small-model value, holding the others fixed. If any non-Embedding component's optimum lies outside the flat region around the transferred value, or if the Embedding start optimum shifts enough to erase the measured speedup, the scale-transfer claim is falsified.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the optimal learning-rate schedule is not one curve but a family of curves indexed by module, and that the relative shapes of these curves survive a 27x scale-up. Concretely, RLRS keeps a global peak learning rate $\eta_{base}$ and final-rate fraction $\alpha_{end}$, then multiplies them by per-component constants $\lambda^m_{start}$ and $\lambda^m_{end}$, giving each module its own cosine decay. On small models the decoupled schedules reach a given loss in up to 22.8% fewer steps for MoE and 17.2% for dense. Transferred without tuning to 113M and 906M active-parameter models, the same values give up to 19% speedups and also remove the loss spikes seen in the MoE baseline. The one caveat the paper identifies is that the Embedding start multiplier should increase with model size; every other relative value transfers unchanged.

Load-bearing premise

The load-bearing premise is that relative learning rates tuned on a small 34-million-parameter model remain near-optimal for models up to 906 million parameters; the paper itself notes this already fails to hold for the Embedding start rate.

Editorial extensions

If this is right

  • Any Transformer-based pretraining run can adopt RLRS without changing the architecture, optimizer, or data; the only extra cost is a local search on a small proxy model.
  • At the tested scale, the component multipliers transfer across a 27x parameter range, so large-scale runs can reuse a small-model configuration and still gain up to 19% in speed.
  • MoE training with RLRS is more stable: the loss spikes visible in the baseline vanish, and RLRS beats the baseline across a range of base learning rates.
  • An overtrained MoE with nearly 130 tokens per active parameter also benefits from transferred rates (14.6% speedup), so the gain is not limited to compute-optimal runs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because only the Embedding start rate shows scale dependence in the ablations, a natural extension is a scaling law for that multiplier, letting future work predict its value at any model size instead of retuning.
  • The speedup metric used in the paper compares steps to reach a fixed loss while the cosine schedule is still active, so true wall-clock savings from RLRS may be larger than the reported percentages.
  • RLRS separates schedule shape from base rate; combining it with a parameterization that also transfers the base rate could eliminate per-size learning-rate tuning entirely, a direction the paper notes and does not test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces RLRS, a method that assigns each Transformer component (Embedding, Unembedding, Attention, Feed-Forward, and for MoE Router and Experts) its own cosine learning-rate schedule through relative multipliers lambda_start and lambda_end applied to a shared base schedule. It proposes tuning these multipliers on a small model and then transferring them to larger models (Algorithm 1). Experiments on dense and MoE models from 34M to 906M active parameters report speedups up to 23% for small models and up to 19% in transferred settings, along with stability improvements and an ablation study. The paper positions the method as a practical, zero-overhead training improvement that is orthogonal to base-LR tuning and complementary to Tensor Programs.

Significance. If the empirical claims hold, the paper offers a simple and practical training-speedup method and, more importantly, evidence that per-component relative learning-rate schedules can transfer across model scale, which would be genuinely valuable for large-scale pretraining. The paper has notable strengths: it tests both dense and MoE architectures, spans a wide range of model sizes, performs out-of-sample transfer to models 27x larger, defines an explicit speedup metric, and includes an ablation that checks sensitivity around the transferred values. The main caveats are the absence of released code, the lack of error bars on the headline speedups, the in-sample nature of the small-model tuning results, and a partially validated transfer premise that is contradicted by the paper's own Embedding ablation.

major comments (4)
  1. [Section 3.3 / Algorithm 1 / Section 4.3] The central cross-scale transfer claim is only partially supported. Algorithm 1 assumes that lambda_m values tuned on a 34M-active-parameter model can be reused on a 906M model without scale correction. However, Section 4.3 and Figure 6 report that the Embedding start rate has a clear preference to increase with model size; the transferred value 5.0 is at the edge of the tested grid at 113M, and the optimum lies beyond it. Since the remaining components were swept only at 113M, there is no direct evidence that Expert, Unembedding, or Attention rates do not also drift between 113M and 906M. This is a load-bearing limitation for the claim that relative rates can be effectively reused on models up to 27x larger. Please either restrict the transfer claim to components for which scale invariance is demonstrated, provide a scale sweep for all components up to 906M, or add a scale-dependent correction for the Embedding start rate.
  2. [Section 3.1 / Tables 2 and 3] The headline speedups are reported without error bars or statistical significance. The speedup is defined as (T_base / T_relative - 1) x 100% and is computed over the means of 3 runs, but Tables 2 and 3 report only a single number per configuration. For example, the Dense906M speedup of 8.7% could easily be within run-to-run noise. Please report per-seed speedups, confidence intervals, or a significance test for at least the main configurations, and clarify how the step count is determined when the two loss curves cross multiple times.
  3. [Section 3.2 / Algorithm 2] The small-model speedups in Table 2 are in-sample: relative learning rates are tuned on the same small model using the local search in Algorithm 2, and the speedup is then measured on that same model. This risks optimistic bias from overfitting to the specific seeds and initialization used for tuning. Please validate the small-model gains on held-out seeds or a separate small-model configuration, or otherwise quantify the selection bias.
  4. [Section 5.1] There is a contradiction between the transfer claim and the text in Section 5.1. The section states that 'our extrapolations demonstrate that our optimization scheme depends on the architecture rather than the model size' and also that 'our method does not aim to facilitate learning rate transfer between different model sizes,' which conflicts with Algorithm 1, the abstract, and Section 2.1. Moreover, the first statement is not consistent with the Section 4.3 finding that the Embedding start rate should increase with model size. Please reconcile these statements and clarify the intended scope of the transfer claim.
minor comments (5)
  1. [Section 3.1] The text refers to 'Base LR Final Fraction (lambda_base)' but the notation in Equation (2) is alpha_end; please unify the notation.
  2. [Figures 3 and 7] Figure 3's caption says 'Medium (33M)' while Table 1 reports Dense34M and MoE8x34M, and Figure 7's caption says 'MoE8x116M' while the text and Table 1 use MoE8x113M. These inconsistencies should be corrected.
  3. [Section 3.1] The paper says that code and configuration files 'will be provided in a public repository soon.' For reproducibility, please make the code available at submission or provide a more concrete availability statement.
  4. [Section 4.1] The interpretations of the relative rates are marked as not experimentally backed, which is good, but some statements, such as 'the Router often stabilizes early in training, leading to deterministic routing,' cite prior work; please ensure the citations are complete and distinguish observed results from hypotheses.
  5. [Table 4] The table entries for the MoE relative learning rates contain spacing artifacts (e.g., '5 0 .6 0 .6 0 .3 1'). These should be formatted consistently with clear decimal points and column alignment.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the RLRS speedups and the cross-scale transfer results are measured empirically, with the transfer premise explicitly tested rather than assumed.

full rationale

The paper makes empirical claims rather than deriving results from definitions, so there is no derivation-to-input circularity. Equations (1) and (2) merely define component learning rates as the base learning rate times relative factors; the reported speedups are computed from measured training-loss curves (Section 3.1, 'Speed-up metric'), not implied algebraically by the definition. Small-model speedups in Table 2 use relative values obtained by local search on those same models, but the paper presents these as tuning outcomes, not as predictions, and the baseline is tuned with the same procedure, so the comparison is not statistically forced. The central transfer claim is genuinely out-of-sample: Algorithm 1 assumes relative values tuned on MoE8x34M transfer to larger models, and Section 3.3 tests that assumption at 113M and 906M active parameters against independently tuned baselines, reporting speedups of up to 19%. Section 4.3 explicitly identifies one component, the Embedding start rate, whose transferred value is not optimal at the larger size and shows a preference to increase with model size; this is stated as an exception and is a scope limitation, not a circular step, because the speedups at the tested configurations are still measured outcomes. The only self-citation, Ludziejewski et al. (2024), is used to set compute-optimal training durations (Section 3.1) and is not load-bearing for the RLRS speedup or transfer conclusions. No uniqueness theorem or ansatz is imported from prior work by the same authors, and the paper explicitly disclaims that its interpretation of relative rates is 'not backed by experimental evidence' (Section 4.1), which further separates explanation from evidence. Therefore no circular step meets the evidentiary bar of the review rules, and the honest finding is no circularity.

Assumptions & free parameters 9 free parameters · 5 assumptions · 0 invented entities

RLRS introduces no new physical entities. Its load-bearing content is a set of tuned hyperparameters (relative multiplier schedules) plus the empirical assumptions that small-model values transfer across scale and that the chosen speedup metric reflects real training efficiency.

free parameters (9)
  • MoE relative LR schedule for Embedding (start/end) = 5.0 / 0.6
    Tuned on MoE8x34M via local search; high early rate stabilizes embeddings quickly, lower late rate avoids disrupting other components.
  • MoE relative LR schedule for Unembedding (start/end) = 0.6 / 0.4
    Tuned on MoE8x34M; lower late rates align with reported unembedding divergence instabilities.
  • MoE relative LR schedule for Router (start/end) = 0.6 / 1.0
    Tuned on MoE8x34M; early low rate delays router stabilization, late high rate supports adaptation.
  • MoE relative LR schedule for Experts (start/end) = 0.3 / 1.125
    Tuned on MoE8x34M; lowest early rate prevents early expert specialization and later high rate fine-tunes experts.
  • MoE relative LR schedule for Attention (start/end) = 1.0 / 1.0
    Local search kept attention at baseline value.
  • Dense relative LR schedule (Embedding, Unembedding, Feed-Forward, Attention start/end) = 5/0.6, 1/0.4, 1/0.6, 1/0.2
    Tuned on Dense34M and used for Dense113M and Dense906M.
  • Base LR final fraction alpha_end = 0.04 (MoE), 0.06 (dense)
    Optimized jointly with relative rates; sets the final cosine value and affects all schedules.
  • Baseline learning rate per model = 1e-4 to 3e-3 depending on model
    Tuned separately for baseline and RLRS on a grid to ensure fair comparison.
  • Weight decay and initialization scale = 0.1 and 0.15
    Optimized alongside RLRS; shared by baseline and RLRS.
assumptions (5)
  • domain assumption Relative learning-rate multipliers tuned on a 34M-active-parameter proxy remain near-optimal for models up to 906M active parameters and 27x total parameters.
    Algorithm 1 and Section 3.3 rely on this transfer; Section 4.3 notes the Embedding start rate is an exception, so the assumption is only partially validated.
  • domain assumption Speedup, defined as steps until RLRS loss exceeds baseline loss, is a valid measure of training efficiency.
    Section 3.1 defines this metric and states it likely underestimates the gain; the claim depends on this proxy being meaningful.
  • domain assumption Three random seeds per configuration are sufficient to support the reported speedups and stability claims.
    Section 3.1 reports means of three runs without variance on speedup, so statistical strength is limited.
  • domain assumption AdamW with cosine schedule is representative enough that RLRS gains generalize to other optimizers.
    Section 2 claims RLRS is optimizer-independent but only AdamW is tested.
  • domain assumption Transformer components can be grouped into Embedding, Attention, Unembedding, Expert, and Router with shared schedules per component.
    Section 2 defines these modules; per-layer variation within a component is ignored, though Figure 10 shows different layers have different update magnitudes.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Decoupled Relative Learning Rate Schedules." pith.science (2026). https://pith.science/paper/ZHE5LMQI

@misc{pith2026250703526,
  author       = {Pith},
  title        = {Pith review of: Decoupled Relative Learning Rate Schedules},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZHE5LMQI}},
  note         = {Machine review of arXiv:2507.03526}
}
abstract

In this work, we introduce a novel approach for optimizing LLM training by adjusting learning rates across weights of different components in Transformer models. Traditional methods often apply a uniform learning rate across all network layers, potentially overlooking the unique dynamics of each part. Remarkably, our introduced relative learning rates, RLRS, method accelerates the training process by up to $23\%$, particularly in complex models such as Mixture of Experts (MoE). Hyperparameters of RLRS can be efficiently tuned on smaller models and then effectively reused on models up to $27\times$ larger. This simple and effective method results in a substantial reduction in training time and computational resources, offering a practical and scalable solution for optimizing large-scale neural networks.

Figures

Figures reproduced from arXiv: 2507.03526 by the authors.

Figure 1
Figure 1. Magnitudes of weight updates for different components of the Transformer with MoE [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Visualization of distinct cosine learning rate schedules for each model component, compared [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 5
Figure 5. Loss of RLRS and baseline for different ηbase on MoE8×113M models. RLRS results in better loss across a range of learning rates. Type LR Type Base LR Train Tokens Total Params Speedup Dense34M baseline 2 × 10−3 1.3B 34M - relative 2 × 10−3 1.3B 34M 17.2% MoE8×34M baseline 3 × 10−3 1.3B 210M - relative 3 × 10−3 1.3B 210M 22.8% [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figures from the paper (4 more)
Figure 6
Figure 6. Figure 6: The performance of relative learning rates, [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Varying λ Attention start and λ Attention end for MoE8×116M. For this component, our optimization algorithm kept the relative value unchanged (λ = 1.0). 0.0B 0.2B 0.4B 0.6B 0.8B 1.0B 1.2B 1.4B Tokens 3.4 3.5 3.6 3.7 3.8 3.9 4.0 Loss Medium (33M) Baseline - mean Baselin…
Figure 8
Figure 8. Figure 8: Training loss for Baseline versus RLRS for MoE8×33M (210M total parame￾ters), averaged among 3 runs. 0B 2B 5B 8B 10B 12B 15B 18B 20B Tokens 2.65 2.70 2.75 2.80 2.85 2.90 Loss Large (906M) Baseline - mean Baseline - std Relative - mean Relative - std [PITH_FULL_IMAGE:f…
Figure 10
Figure 10. Figure 10: Magnitudes of weight updates for different components of all 8 layers of [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

26 extracted references · 9 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradb...

    2022

  3. [3]

    Alemi, Roman Novak, Peter J

    Katie Everett, Lechao Xiao, Mitchell Wortsman, Alexander A. Alemi, Roman Novak, Peter J. Liu, Izzeddin Gur, Jascha Sohl-Dickstein, Leslie Pack Kaelbling, Jaehoon Lee, and Jeffrey Pennington. Scaling exponents across parameterizations and optimizers. arXiv preprint arXiv:2407.05872, 2024

    arXiv 2024

  4. [4]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research, 23 0 (1): 0 5232--5270, 2022

    work page 2022

  5. [5]

    Lora+: Efficient low rank adaptation of large models

    Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models. arXiv preprint arXiv:2402.12354, 2024

    arXiv 2024

  6. [6]

    Rae, and Laurent Sifre

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laurent Sifre...

    work page 2024

  7. [7]

    Universal language model fine-tuning for text classification

    Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Long Papers), 2018

    work page 2018

  8. [8]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    arXiv 2021

Show all 26 references
  1. [9]

    Averaging weights leads to wider optima and better generalization

    Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407, 2018

    2018 arXiv

  2. [10]

    Sgdr: Stochastic gradient descent with warm restarts

    Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. ICLR 2017 (5th International Conference on Learning Representations), 2016

    2017

  3. [11]

    Decoupled weight decay regularization, 2019

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019

    2019

  4. [12]

    Scaling laws for fine-grained mixture of experts

    Jan Ludziejewski, Jakub Krajewski, Kamil Adamczewski, Maciej Pióro, Michał Krutul, Szymon Antoniak, Kamil Ciebiera, Krystian Król, Tomasz Odrzygóźdź, Piotr Sankowski, Marek Cygan, and Sebastian Jaszczur. Scaling laws for fine-grained mixture of experts. In Forty-first Internat...

    2024

  5. [13]

    Improving language understanding by generative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018

    2018

  6. [14]

    Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J

    Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21: 0 140:1--140:67, 2019

    2019

  7. [15]

    D eep S peed- M o E : Advancing mixture-of-experts inference and training to power next-generation AI scale

    Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. D eep S peed- M o E : Advancing mixture-of-experts inference and training to power next-generation AI scale. In Kamalika Chaudhuri, Stefanie Jegelk...

    2022

  8. [16]

    Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. How to fine-tune bert for text classification? In Chinese computational linguistics: 18th China national conference, CCL 2019, Kunming, China, October 18--20, 2019, proceedings 18, pp.\ 194--206. Springer, 2019

    2019

  9. [17]

    Llama: Open and efficient foundation language models, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language...

    2023

  10. [18]

    Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John D

    Mitchell Wortsman, Peter J. Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John D. Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, Jeffrey Pennington, Jascha Sohl-dickstein, Kelvin Xu, Jaehoon Lee, Justin Gilmer, and Simon Kornblith. Small-scale proxies for large...

    2023 arXiv

  11. [19]

    Openmoe: An early effort on open mixture-of-experts language models

    Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou, and Yang You. Openmoe: An early effort on open mixture-of-experts language models. arXiv preprint arXiv:2402.01739, 2024

    2024 arXiv

  12. [20]

    Tensor programs ii: Neural tangent kernel for any architecture

    Greg Yang. Tensor programs ii: Neural tangent kernel for any architecture. arXiv preprint arXiv:2006.14548, 2020

    2006 arXiv

  13. [21]

    Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer

    Greg Yang, Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiv:2203.03466, 2022

    2022 arXiv

  14. [22]

    Adalora: Adaptive budget allocation for parameter-efficient fine-tuning

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter-efficient fine-tuning. In International Conference on Learning Representations (ICLR), 2023

    2023

  15. [23]

    St-moe: Designing stable and transferable sparse expert models

    Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models. arXiv preprint arXiv:2202.08906, 2022

    2022 arXiv

  16. [24]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  17. [25]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  18. [26]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.