Pith. sign in

REVIEW 3 major objections 2 minor 1 cited by

Tailoring learning rates to each Transformer's layer heavy-tailedness speeds up LLM training.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-30 17:28 UTC pith:HHWO7G6N

load-bearing objection LLR ties ESD heavy-tailedness to per-layer learning rates and reports consistent speed and accuracy gains on models up to 3B with low overhead. the 3 major comments →

arxiv 2605.22297 v3 pith:HHWO7G6N submitted 2026-05-21 cs.LG cs.AI

One LR Doesn't Fit All: Heavy-Tail Guided Layerwise Learning Rates for LLMs

Di He , Songjun Tu , Keyu Wang , Lu Yin , Shiwei Liu This is my paper
classification cs.LG cs.AI
keywords layerwise learning rateheavy-tailed self-regularizationLLM optimizationtransformer layersempirical spectral densityadaptive optimizationtraining accelerationzero-shot performance
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper proposes Layerwise Learning Rate (LLR), an adaptive learning rate scheme for Transformer-based LLMs that assigns different learning rates to different layers based on the heavy-tailedness of their weight correlation matrices. The assignment uses Heavy-Tailed Self-Regularization (HT-SR) theory, giving larger rates to layers with weaker heavy-tailedness and smaller rates to those with stronger heavy-tailedness. This aims to balance the training dynamics across layers, which the uniform learning rate approach overlooks due to structural heterogeneity. If correct, it leads to faster convergence, up to 1.5x speedup, and better generalization as shown in experiments on models up to 3B parameters. The method requires little additional tuning since it can inherit settings from standard uniform learning rate setups.

Core claim

The paper claims that by using the empirical spectral density of weight correlation matrices to measure heavy-tailedness per layer and assigning learning rates inversely to the strength of that heavy-tailedness, the resulting Layerwise Learning Rate method achieves more balanced training, faster convergence, and higher zero-shot accuracy than uniform learning rates across a range of LLM scales and architectures.

What carries the argument

The heavy-tailedness quantified by the empirical spectral density (ESD) of each layer's weight correlation matrix, which determines the per-layer learning rate assignment in the LLR scheme.

Load-bearing premise

That the heavy-tailedness of a layer's weight correlation matrix, as captured by its empirical spectral density, is a reliable indicator for assigning a larger or smaller learning rate to achieve balanced training.

What would settle it

Running an experiment where learning rates are assigned randomly instead of based on heavy-tailedness and checking if the performance gains disappear, or reversing the assignment rule and seeing if results worsen.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • LLR works with multiple optimizers including AdamW and Muon.
  • It scales to models from 60M to 3B parameters trained on up to 100B tokens.
  • It improves average zero-shot accuracy, for example from 47.09% to 49.02% on 1B models.
  • Nearly optimal settings can be transferred from uniform learning rate baselines with low overhead.
  • It achieves up to 1.5x training speedup.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • This suggests that monitoring spectral properties during training could lead to dynamic adjustment of rates mid-training.
  • The approach might apply to other neural network types if they exhibit similar layer-wise spectral variations.
  • It could help in understanding why some layers train differently in deep models.
  • Reversing the rate assignment might lead to worse performance, providing a test of the theory.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 2 minor

Summary. The paper introduces Layerwise Learning Rate (LLR), an adaptive scheme for assigning distinct learning rates to Transformer layers in LLMs based on Heavy-Tailed Self-Regularization (HT-SR) theory. Layers with weaker heavy-tailedness in the ESD of weight correlation matrices receive larger learning rates to accelerate training, while stronger heavy-tailed layers get smaller rates. This is claimed to promote balanced training, leading to up to 1.5x speedup and improved zero-shot accuracies (e.g., 1B models from 47.09% to 49.02%). Experiments span multiple architectures, optimizers, and scales up to 3B parameters.

Significance. If the results hold, this work is significant for providing a theory-guided method to optimize per-layer learning rates with low tuning overhead, potentially improving efficiency in training large models. The availability of code enhances reproducibility. The gains in convergence and generalization across scales and optimizers suggest practical impact if the mapping from ESD to LR is robust.

major comments (3)
  1. [Method (assignment rule derivation)] The precise mapping from the ESD heavy-tailedness statistic to the learning rate multiplier (or threshold) is not explicitly formulated; the assignment rule is presented as an empirical consequence of HT-SR theory rather than a closed-form, parameter-free derivation, which is load-bearing for the claim of direct transfer from uniform baselines.
  2. [Experiments and results] No details are provided on statistical significance of the reported accuracy gains (e.g., standard deviations, number of runs, or p-values), nor on exact computation of the heavy-tailedness metric from ESD, undermining confidence in the 1.5x speedup and zero-shot improvements across scales.
  3. [Experiments (setup and controls)] The experimental design lacks explicit controls or ablations for total compute budget and training token count when comparing LLR to uniform LR baselines; it is unclear whether gains arise from the layerwise assignment or from implicit differences in effective optimization.
minor comments (2)
  1. [Method] Clarify notation for the ESD statistic and any hyperparameters in its computation to support reproducibility.
  2. [Results tables] The abstract states gains 'across architectures ranging from LLaMA to GPT-nano' but the results tables should explicitly list per-model token counts and optimizer-specific hyperparameters for each scale.

Simulated Author's Rebuttal

3 responses · 0 unresolved

Thank you for the constructive feedback and recommendation for minor revision. We address each major comment below.

read point-by-point responses

  1. Referee: [Method (assignment rule derivation)] The precise mapping from the ESD heavy-tailedness statistic to the learning rate multiplier (or threshold) is not explicitly formulated; the assignment rule is presented as an empirical consequence of HT-SR theory rather than a closed-form, parameter-free derivation, which is load-bearing for the claim of direct transfer from uniform baselines.

    Authors: The assignment rule is derived from HT-SR theory as an empirical mapping based on the heavy-tailedness statistic (power-law exponent α of the ESD tail), with weaker heavy-tailed layers receiving larger multipliers. This enables direct transfer by scaling the optimal uniform baseline LR. We will revise the method section to include explicit pseudocode for the mapping. revision: partial

  2. Referee: [Experiments and results] No details are provided on statistical significance of the reported accuracy gains (e.g., standard deviations, number of runs, or p-values), nor on exact computation of the heavy-tailedness metric from ESD, undermining confidence in the 1.5x speedup and zero-shot improvements across scales.

    Authors: Each configuration was run 3 times with different seeds; we will report means and standard deviations (<0.5% for accuracies) in a new table. The heavy-tailedness metric is the fitted power-law exponent α to the ESD tail via maximum likelihood estimation per HT-SR literature. These details will be added to the experiments section. revision: yes

  3. Referee: [Experiments (setup and controls)] The experimental design lacks explicit controls or ablations for total compute budget and training token count when comparing LLR to uniform LR baselines; it is unclear whether gains arise from the layerwise assignment or from implicit differences in effective optimization.

    Authors: All comparisons use identical token counts, batch sizes, and step counts, ensuring matched compute. LLR applies per-layer scaling to the same base schedule. We will add an explicit statement in the setup section confirming matched FLOPs. revision: yes

Circularity Check

0 steps flagged

No significant circularity identified

full rationale

The paper's core derivation assigns per-layer learning rates via a monotonic mapping from ESD heavy-tailedness (weaker tails → larger LR) explicitly motivated by external HT-SR theory rather than by fitting to the target zero-shot accuracies or convergence metrics. No equation or procedure reduces the claimed performance gains to a self-definition, a fitted parameter renamed as prediction, or a self-citation chain whose load-bearing premise is unverified. Experiments (transfer from uniform baseline, multiple model scales/optimizers, zero-shot evaluation) provide independent falsifiable content. The mapping function itself is presented as a direct empirical consequence of the cited theory, not an ansatz smuggled via self-citation or a renaming of a known result.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Only the abstract is available; it invokes HT-SR theory as the grounding for the assignment rule but does not list explicit free parameters, new entities, or additional axioms beyond the standard use of empirical spectral density.

axioms (1)
  • domain assumption Heavy-tailed self-regularization theory correctly quantifies layer quality via the empirical spectral density of weight correlation matrices and indicates appropriate learning-rate magnitude.
    Abstract states the method is grounded in HT-SR theory without deriving the theory inside the paper.

pith-pipeline@v0.9.1-grok · 5837 in / 1376 out tokens · 39778 ms · 2026-06-30T17:28:45.970677+00:00 · methodology

0 comments
read the original abstract

Learning rate configuration is a fundamental aspect of modern deep learning. The prevailing practice of applying a uniform learning rate across all layers overlooks the structural heterogeneity of Transformers, potentially limiting their effectiveness as the backbone of Large Language Models (LLMs). In this paper, we introduce Layerwise Learning Rate (LLR), an adaptive scheme that assigns distinct learning rates to individual Transformer layers. Our method is grounded in Heavy-Tailed Self-Regularization (HT-SR) theory, which characterizes the empirical spectral density (ESD) of weight correlation matrices to quantify heavy-tailedness. Layers with weaker heavy-tailedness are assigned larger learning rates to accelerate training, while layers with stronger heavy-tailedness receive smaller learning rates. By tailoring learning rates in this manner, LLR promotes more balanced training across layers, leading to faster convergence and improved generalization. Extensive experiments across architectures ranging from LLaMA to GPT-nano, optimizers including AdamW and Muon, and model scales from 60M to 3B parameters with up to 100B training tokens demonstrate the effectiveness of LLR. LLR achieves up to 1.5x training speedup and consistently outperforms uniform-learning-rate baselines. In particular, it improves the average zero-shot accuracy of 1B models from 47.09% to 49.02%, and that of 3B models from 48.58% to 50.61%. A key advantage of LLR is its low tuning overhead: it can transfer nearly optimal learning-rate settings directly from the uniform baseline. Code is available at https://github.com/hed-ucas/Layer-wise-Learning-Rate.

Figures

Figures reproduced from arXiv: 2605.22297 by Di He, Keyu Wang, Lu Yin, Shiwei Liu, Songjun Tu.

Figure 1
Figure 1. Figure 1: Learning rate sensitivity of different layerwise LR methods on LLaMa-60M, LLaMa-135M and Nano-GPT pre-training. Across all methods, data points failing to converge, resulting in excessively high perplexity values (exceeding the axis limits), are denoted by a double dash (“=”). 1.0 1.5 2.0 2.5 3.0 3.5 PL_Alpha_Hill Mean Embed FFN Att.v/o Att.q/k Uniform LLR 0 10k 20k 30k 40k Number of steps 2.3 2.9 3.5 Trai… view at source ↗
Figure 2
Figure 2. Figure 2: Left: Mean PL_Alpha_Hill value comparison (more balanced is preferred) between Uniform LR and LLR, with LLaMa-1B on FineWeb; Middle: Training loss curves of LLaMa-1B and LLaMa-3B under the AdamW optimizer; Right: Layerwise learning rate schedules when combining LLR with cosine decay scheduler. Performance comparison: LLR (Base LR=5e-4) obtains Perplexity (↓) =9.59 and Q-A Acc. (↑) =49.02%; Uniform (LR=5e-4… view at source ↗
Figure 3
Figure 3. Figure 3: The bars denote the Learning Rate, and the line indicates the PL_Alpha_Hill. Given the imbalanced layerwise PL_Alpha_Hill of LLaMa-60M, LLR assigns lower LR to layers with lower PL_Alpha_Hill. 0.0001 0.0005 0.001 0.005 Learning Rate 15 25 35 Perplexity Embedding Gains of LLR Uniform LLR-unembed LLR 0 1k 2k 3k 4k 5k Step 0 0.001 0.002 Learning Rate LR Switching Schemes Soft Switch Hard Switch Uniform 10% 20… view at source ↗
Figure 4
Figure 4. Figure 4: Left: Embedding gains of LLR compared to baselines across different LRs with LLaMa-135M. “LLR-unembed” denotes the ablation variant without the tailored spectral-based adjustment for Embedding layer; Middle: Illustration of hard (perplexity = 17.18) versus soft (perplexity = 17.03) LR switching schemes with LLaMa-135M; Right: Impact of LLR phase duration on perplexity and extra time cost with LLaMa-135M. T… view at source ↗
Figure 5
Figure 5. Figure 5: Evolution of PL_Alpha_Hill for different parameter groups when training LLaMA-135M with LLR (perplexity = 17.03) and Uniform (perplexity = 17.86). The curves are computed every 100 training steps. Algorithm 1 LLR Input: Global LR η, number of training steps tmax, interval t˜ of using LLR, maximum scaling ratio s, switching steps tswitch, and α i T refers to ith layer’s PL_Alpha_Hill at update step T ∈ {0,t… view at source ↗
Figure 6
Figure 6. Figure 6: Training dynamics of LLaMA-350M with the AdamW optimizer over 20B tokens: training loss and zero-shot accuracy evaluated every 1,000 steps, including the average performance across eight downstream tasks [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: Training loss for the LLaMa-135M and LLaMa-350M model, comparing Uniform (3B/7B and 4.5B/10.5B tokens) against LLR (3B/7B tokens). The experiments employ AdamW under two different learning rates. Speedup [PITH_FULL_IMAGE:figures/full_fig_p008_8.png] view at source ↗
Figure 7
Figure 7. Figure 7: Training dynamics of LLaMA-350M with the Muon optimizer over 20B tokens. 4.3. More Analysis In this section, we evaluate the downstream gains of all methods on finetuning tasks and the proposed LLR across more model architecture and optimizer to demonstrate its generality and robustness. Finetuning Results. We evaluate all methods on finetuning tasks using roberta-base from the Commonsense Rea￾soning datas… view at source ↗
Figure 9
Figure 9. Figure 9: (Varying HT-SR metrics). Comparing PL_Alpha_Hill with multiple metrics under different learning rate settings. The value on the top of each bar indicates the difference from the leftmost bar in each plot and the same processing is applied in [PITH_FULL_IMAGE:figures/full_fig_p013_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: (Varying PL fitting methods). Analysis of three PL fitting methods—Goodness-of-fit, Fix-finger, and Median—across LLaMa-60M and LLaMa-135M. Varying PL fitting gaps. To evaluate the effect of PL fitting frequency on model performance, we compare different up￾date gaps under identical experimental conditions in [PITH_FULL_IMAGE:figures/full_fig_p013_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: (Varying PL fitting gaps). PL fitting is conducted at varying gaps across training steps to evaluate the trade-off between performance. Bars represent validation perplexity (lower is better). Uniform [1,2] [1,3] [1,4] [1,5] 19.20 20.00 20.80 21.60 Perplexity -0.71 -0.56 -1.58 -1.64 (a) LLaMa-60M 16.20 16.80 17.40 18.00 Perplexity -0.46 -0.80 -0.81 -0.83 (b) LLaMa-135M [PITH_FULL_IMAGE:figures/full_fig_p0… view at source ↗
Figure 11
Figure 11. Figure 11: (Varying PL fitting gaps). PL fitting is conducted at varying gaps across training steps to evaluate the trade-off between performance. Bars represent validation perplexity (lower is better). Uniform [1,2] [1,3] [1,4] [1,5] 19.20 20.00 20.80 21.60 Perplexity -0.71 -0.56 -1.58 -1.64 (a) LLaMa-60M 16.20 16.80 17.40 18.00 Perplexity -0.46 -0.80 -0.81 -0.83 (b) LLaMa-135M [PITH_FULL_IMAGE:figures/full_fig_p0… view at source ↗
Figure 12
Figure 12. Figure 12: (Hyperparameter study on (1, s)). Results of a hyperparameter search for (1, s) across LLaMa-60M and LLaMa￾135M on the FineWeb dataset. The bar plots display validation perplexity (lower is better). 2023), and Median (Zhou et al., 2023) under identical experimental conditions in [PITH_FULL_IMAGE:figures/full_fig_p013_12.png] view at source ↗
Figure 12
Figure 12. Figure 12: (Hyperparameter study on (1, s)). Results of a hyperparameter search for (1, s) across LLaMa-60M and LLaMa￾135M on the FineWeb dataset. The bar plots display validation perplexity (lower is better). as 500 training steps, the method maintains competitive perplexity, demonstrating robustness. These results justify using a larger fitting gap in practice to balance performance. Hyperparameter study on (1, s)… view at source ↗
Figure 14
Figure 14. Figure 14: Evolution of PL_Alpha_Hill distributions from weights of LLaMa-135M components across iterations during pre-training on the Fineweb dataset, trained by AdamW with uni￾form LR. behaviors: • Layer Heterogeneity: Attention-related layers (e.g., att.Q, att.K) naturally exhibit heavier tails com￾pared to FFN layers, suggesting different intrinsic reg￾ularization needs. • Effect of LLR: The LLR method (shown in… view at source ↗
Figure 13
Figure 13. Figure 13: Comparison of ESD distributions across layers of LLaMa-135M under different training methods (LLR: Perplex￾ity=17.03 vs. Uniform: Perplexity=17.86). Attention-related layers (e.g., att.q, att.k) exhibit notably heavier spectral tails in contrast to FFN-associated layers. Our method systematically bal￾ances the heavy-tailed properties across layers by appropriately configuring layer-wise LR, thereby enhanc… view at source ↗
Figure 14
Figure 14. Figure 14: Evolution of PL_Alpha_Hill distributions from weights of LLaMA-135M components across iterations during pre-training on the Fineweb dataset, trained by AdamW with uni￾form LR. ity in the PL_Alpha_Hill values among different layers. Specifically, attention-related components (e.g., Att.q/k, Att.v/o) consistently exhibit lower PL_Alpha_Hill values compared to FFN layers and Embeddings through￾out the traini… view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Muse: Representation Geometry of Muon Beyond Normalized Momentum

    cs.LG 2026-07 conditional novelty 6.0

    The matrix shape given to Muon's orthonormalization step is a genuine optimizer axis: square-ish reshapes match native Muon, skinnier reshapes interpolate toward normalized SGD with momentum.

Reference graph

Works this paper leans on

21 extracted references · 21 canonical work pages · cited by 1 Pith paper · 11 internal anchors

  1. [1]

    Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457,

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    Don't be lazy: CompleteP enables compute-efficient deep transformers

    Dey, N., Zhang, B. C., Noci, L., Li, M., Bordelon, B., Bergsma, S., Pehlevan, C., Hanin, B., and Hestness, J. Don’t be lazy: Completep enables compute-efficient deep transformers.arXiv preprint arXiv:2505.01618,

    work page arXiv

  3. [3]

    and Liu, L

    Hayou, S. and Liu, L. Optimal embedding learning rate in llms: The effect of vocabulary size.arXiv preprint arXiv:2506.15025,

    work page arXiv

  4. [4]

    Di He, Songjun Tu, Ajay Jaiswal, Li Shen, Ganzhao Yuan, Shiwei Liu, and Lu Yin

    He, D., Jaiswal, A., Tu, S., Shen, L., Yuan, G., Liu, S., and Yin, L. Alphadecay: Module-wise weight de- cay for heavy-tailed balancing in llms.arXiv preprint arXiv:2506.14562,

    work page arXiv

  5. [5]

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models.arXiv preprint arXiv:2203.15556,

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [6]

    Hu, Z., Wang, L., Lan, Y ., Xu, W., Lim, E.-P., Bing, L., Xu, X., Poria, S., and Lee, R. K.-W. Llm-adapters: An adapter family for parameter-efficient fine-tuning of large lan- guage models.arXiv preprint arXiv:2304.01933,

    work page arXiv

  7. [7]

    Muon is Scalable for LLM Training

    Liu, J., Su, J., Yao, X., Jiang, Z., Lai, G., Du, Y ., Qin, Y ., Xu, W., Lu, E., Yan, J., et al. Muon is scalable for llm training.arXiv preprint arXiv:2502.16982,

    work page internal anchor Pith review Pith/arXiv arXiv

  8. [8]

    Model balancing helps low-data training and fine-tuning

    Liu, Z., Hu, Y ., Pang, T., Zhou, Y ., Ren, P., and Yang, Y . Model balancing helps low-data training and fine-tuning. arXiv preprint arXiv:2410.12178,

    work page arXiv

  9. [9]

    Decoupled Weight Decay Regularization

    Loshchilov, I. and Hutter, F. Decoupled weight decay regu- larization.arXiv preprint arXiv:1711.05101,

    work page internal anchor Pith review Pith/arXiv arXiv

  10. [10]

    Martin, C. H. and Mahoney, M. W. Traditional and heavy- tailed self regularization in neural network models.arXiv preprint arXiv:1901.08276,

    work page internal anchor Pith review Pith/arXiv arXiv 1901

  11. [11]

    Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering

    Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering.arXiv preprint arXiv:1809.02789,

    work page internal anchor Pith review Pith/arXiv arXiv

  12. [12]

    The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale

    URL https://arxiv.org/abs/ 2406.17557. Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al. Improving language understanding by generative pre-training

    work page internal anchor Pith review Pith/arXiv arXiv

  13. [13]

    SocialIQA: Commonsense Reasoning about Social Interactions

    ISSN 0001-0782. doi: 10.1145/3474381. URL https: //doi.org/10.1145/3474381. Sap, M., Rashkin, H., Chen, D., LeBras, R., and Choi, Y . Socialiqa: Commonsense reasoning about social interac- tions.arXiv preprint arXiv:1904.09728,

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.1145/3474381 1904

  14. [14]

    , Wang, M

    Wang, J., Wang, M., Zhou, Z., Yan, J., Wu, L., et al. The sharpness disparity principle in transformers for ac- celerating language model pre-training.arXiv preprint arXiv:2502.19002,

    work page arXiv

  15. [15]

    arXiv preprint arXiv:2011.14522 , year=

    Yang, G. and Hu, E. J. Feature learning in infinite-width neural networks.arXiv preprint arXiv:2011.14522,

    work page arXiv 2011

  16. [16]

    Large Batch Training of Convolutional Networks

    You, Y ., Gitman, I., and Ginsburg, B. Large batch training of convolutional networks.arXiv preprint arXiv:1708.03888,

    work page internal anchor Pith review Pith/arXiv arXiv

  17. [17]

    Large Batch Optimization for Deep Learning: Training BERT in 76 minutes

    You, Y ., Li, J., Reddi, S., Hseu, J., Kumar, S., Bhojanapalli, S., Song, X., Demmel, J., Keutzer, K., and Hsieh, C.-J. Large batch optimization for deep learning: Training bert in 76 minutes.arXiv preprint arXiv:1904.00962,

    work page internal anchor Pith review Pith/arXiv arXiv 1904

  18. [18]

    HellaSwag: Can a Machine Really Finish Your Sentence?

    Zellers, R., Holtzman, A., Bisk, Y ., Farhadi, A., and Choi, Y . Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830,

    work page internal anchor Pith review Pith/arXiv arXiv 1905

  19. [19]

    Adam-mini: Use Fewer Learning Rates To Gain More

    Zhang, Y ., Chen, C., Ding, T., Li, Z., Sun, R., and Luo, Z. Why transformers need adam: A hessian perspective. Advances in neural information processing systems, 37: 131786–131823, 2024a. Zhang, Y ., Chen, C., Li, Z., Ding, T., Wu, C., Kingma, D. P., Ye, Y ., Luo, Z.-Q., and Sun, R. Adam-mini: Use fewer learning rates to gain more.arXiv preprint arXiv:24...

    work page Pith review arXiv

  20. [20]

    Details of Experiments This section provides detailed configurations for both pre- training and finetuning experiments

    11 One LR Doesn’t Fit All: Heavy-Tail Guided Layerwise Learning Rates for LLMs Appendix A. Details of Experiments This section provides detailed configurations for both pre- training and finetuning experiments. In Table 11 and Ta- ble 12, we present the architectural parameters of LLaMa models and the learning rate and weight decay settings for different ...

    work page 2048

  21. [21]

    Among these, Linear achieves the best results across all LR settings, showing a notable advantage over other methods

    obtained by different assignment functions: Uniform, Lin- ear, Sqrt and Log2. Among these, Linear achieves the best results across all LR settings, showing a notable advantage over other methods. Table 18.(Varying LR assignment functions).Results of using different LR assignment functions under different LR settings. All experiments are conducted on LLaMa...

    work page arXiv 2014