REVIEW 3 major objections 7 minor 42 references
GPAS: Accelerating Convergence of LLM Pretraining via Gradient-Preserving Activation Scaling
T0 review · 3 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Adding a gradient-preserving activation-scaling step to Pre-LayerNorm transformers lowers pretraining perplexity from 71M to 1B parameters, and the gains survive fine-tuning.
desk verdict GPAS is a simple, plausible stop-gradient activation scaling trick with consistent but single-run evidence; it deserves peer review, not desk rejection. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the GPAS block itself: a per-layer learnable scalar gate $\alpha_l$, a SiLU activation applied to it, and a stop-gradient operator $\mathrm{sg}(\cdot)$ that multiplies the intermediate activation. The update $x_{l+1}=x'_{l+1}-\mathrm{SiLU}(\alpha_l)\cdot \mathrm{sg}(x'_{l+1})$ performs two jobs at once: in the forward pass it scales $x'_{l+1}$ by $(1-\mathrm{SiLU}(\alpha_l))$, and in the backward pass it makes $\partial x_{l+1}/\partial x'_{l+1}=I$, so the scale term contributes nothing to the gradient. The learned $\alpha_l$ therefore acts as a per-layer mixing ratio between the shortcut and the sub-layer output, and the paper's Theorem 1 derives gradient-norm bounds for the GPAS-augmented network in which the gate-dependent factor $\exp(-M(\alpha)/2)$ exponentially suppresses the upper bound, replacing Pre-LN's $O(L)$ growth; the proof relies on a variance recursion $\sigma^2_{x_{\ell+1}} = \sigma^2_{x_\ell}(1+1/\sigma_{x_\ell})(1-\mathrm{SiLU}(\alpha_\ell))$.
What would settle it
Either measure the Jacobian $\partial x_{l+1}/\partial x'_{l+1}$ of the GPAS step during BFloat16 training and check whether it stays identity within numerical precision, or run the paper's own control at 1B (Equation 12 without stop-gradient): the 350M ablation shows a 0.99 perplexity gap between the stop-gradient and no-stop-gradient versions, so seeing that gap vanish at 1B would falsify the gradient-preservation mechanism.
Extended reading notes
Core claim
The paper's central claim is that the residual path, not the normalization, is the place to fix Pre-LN's depth pathology. GPAS changes the residual update to $x_{l+1}=x'_{l+1}-\mathrm{SiLU}(\alpha_l)\cdot \mathrm{sg}(x'_{l+1})$, where $x'_{l+1}=x_l+f(\mathrm{LN}(x_l))$ is the usual Pre-LN output and $\mathrm{sg}$ means stop-gradient. In the forward pass the sub-layer output is damped by the learned factor $1-\mathrm{SiLU}(\alpha_l)$, which compresses the layer-by-layer growth of activation variance; in the backward pass the Jacobian of the GPAS step is exactly the identity, so the dampening does not shrink gradients. The paper reports that this decoupling improves every architecture it is added to (Pre-LN, Sandwich-LN, Mix-LN, LayerNorm Scaling, and DeepNorm) across five model sizes, and that the pretraining gains carry over to supervised fine-tuning on seven reasoning benchmarks. The partial 7B run also shows faster convergence (13.82 vs 15.27 evaluation perplexity after 40K steps), which the authors read as evidence the mechanism scales.
Load-bearing premise
The load-bearing premise is that the stop-gradient operator makes the backward pass behave as if the forward scaling had not happened, and that the learned per-layer scalars stay stable while training; the paper's own gradient spike at step 10K and the need to clip gate gradients for the 1B Sandwich-LN run show this stability premise is already strained.
Editorial extensions
If this is right
- Pretraining to a fixed perplexity would take fewer steps for Pre-LN transformers at scales from 71M to 1B, and the partial 7B run (13.82 vs 15.27 evaluation perplexity after 40K steps) suggests the effect can persist at larger scale.
- The same residual-scale mechanism transfers to other normalization schemes: Sandwich-LN, Mix-LN, LayerNorm Scaling, and DeepNorm all improve with GPAS, with DeepNorm + GPAS avoiding vanilla DeepNorm's 1B divergence.
- Layer-removal importance becomes more uniform, meaning deeper layers carry real information instead of being dominated by the shortcut; this reduces the "unreasonable ineffectiveness of deeper layers" phenomenon and should make pruned-LLM pipelines retain more accuracy.
- The gain is not erased by downstream fine-tuning: 1B GPAS models score higher on averaged commonsense-reasoning benchmarks, with gates frozen during supervised fine-tuning.
Reading between the lines
- As an extension the paper does not pursue, a clamped or scheduled gate value might keep most of the variance compression while removing the step-10K gradient spike; this is directly testable with a fixed schedule rather than learned gates.
- If variance flattening is the active ingredient, GPAS should show diminishing returns on shallow or already variance-flat models, so a controlled comparison across depths would separate the variance hypothesis from the gradient-preservation hypothesis.
- The paper uses one shared gate per layer for both attention and FFN sub-layers; per-sub-layer gates are a plausible refinement, but untested, and could change both the gains and the stability behavior.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Gradient-Preserving Activation Scaling (GPAS), a per-layer modification of the residual stream, x_{l+1} = x'_{l+1} - SiLU(alpha_l)*sg(x'_{l+1}), which scales forward activations by 1 - SiLU(alpha_l) while the stop-gradient operator keeps the local Jacobian equal to the identity. The method is motivated by the known exponential activation-variance growth of Pre-LN transformers, and is instantiated on Pre-LN, Sandwich-LN, Mix-LN, LayerNorm Scaling, and DeepNorm. The authors pretrain LLaMA-style models from 71M to 1B parameters on C4 (plus a partial 40K-step 7B run in Appendix A), reporting perplexity improvements in 18 of 19 GPAS-versus-base comparisons (Table 2), with carry-over to supervised finetuning (Table 3). Section 5 analyzes learned gate values, activation variance, gradient norms, weight norms, and layer importance; Section 5.7 ablates the activation, insertion location, stop-gradient necessity, and learnable versus predefined gates; Section 5.6 and Appendix C provide a variance-recurrence analysis culminating in Theorem 1.
Significance. If the reported gains survive repeated-seed evaluation, GPAS is a genuinely useful plug-in: it is a one-line architectural change with one extra scalar per layer, it is orthogonal to the normalization scheme, the stop-gradient control experiment (Table 5) cleanly isolates the mechanism, and the paper ships code. Strengths include the breadth of the sweep (five scales, six architectures), the SFT carry-over, the honest Limitations paragraph, and analysis figures linking the method to variance and layer-importance effects. The main gap is statistical: every Table 2 cell is a single run, several deltas are small, and three baseline entries are diverged runs; I agree with the stress-test assessment that this is the primary obstacle. The theoretical section is the weakest part, as written it does not establish the 'no vanishing gradients / no explosion' claims it asserts. Overall, the empirical finding is plausible but not yet established at the level the abstract claims.
major comments (3)
- [Table 2, Section 4.2] The central claim of consistent performance gains rests entirely on single-run comparisons, and the evidence is weaker than the table's format suggests. Every cell in Table 2 is one training run with no seeds and no error bars, while several reported improvements are small enough to be within typical run-to-run noise at these token budgets (e.g., Sandwich+GPAS at 250M: -0.07; Mix+GPAS at 250M: -0.12 and at 1B: -0.24). Moreover, three baseline entries are diverged runs (Post-LN at 250M: 1412.13; Post-LN at 1B: 1404.98; DeepNorm at 1B: 1404.22), so the deltas reported against them, including the '-1386' for DeepNorm+GPAS at 1B, are not evidence about the method; they only show that GPAS avoided divergence. The 7B result (Appendix A) is a single partial run stopped at 40K of 150K steps. I would ask for at least three seeds with mean and standard deviation for the headline comparisons (Pre+GPAS at 250M and 1B, LNS+GPAS at 1B, Sandwich+GPAS at 250M) and for a table that separates converged from diverged baselines, before the 'consistent gains' claim can be taken as established.
- [Section 5.6, Theorem 1, Eq. (10), Appendix C] The theoretical support does not establish the conclusions drawn from it. Theorem 1 bounds UP(||d y_L / d x_1||^2), which is itself an upper bound on the gradient norm; from a lower bound on this quantity the text concludes that GPAS 'precludes the possibility of vanishing gradients,' but a lower bound on an upper estimate says nothing about the true gradient norm. The variance recurrence in Eq. (16), sigma^2_{x_{l+1}} = sigma^2_{x_l}(1 + 1/sigma_{x_l})(1 - SiLU(alpha_l)), is asserted without derivation, uses inconsistent layer indices, and is dimensionally unusual; the constants A and B appearing in Eq. (25) are never defined; and the lower bound in Eq. (10) does not follow from the appendix's Eq. (39), where the geometric-series sum converges to a constant as L grows unless the exponent is negative, with an unexplained '+1' in the denominator of Eq. (10). The manuscript's own Limitations paragraph concedes that gate-update stability and theoretical understanding remain open; I would suggest either substantially repairing the proof or explicitly presenting Section 5.6 as a heuristic analysis rather than a theorem.
- [Section 5.3, Section 5.7, Table 5] Two additional observations bear directly on the robustness of the central claim. First, Section 5.3 reports a gradient spike at step 10K for Pre+GPAS at 1B and states that Sandwich+GPAS at 1B required gate-gradient clipping (threshold 0.01) 'to avoid crashing,' while Section 5.7's predefined-gate ablation (Table 5) shows that freezing the final learned gate profile from the start makes training worse than the no-GPAS baseline (22.46 vs 21.35). Together these results indicate that the benefit is carried by the adaptive gate schedule rather than by the attained variance profile, and that the method's stability depends on conditions that are not yet characterized; with a single run per configuration it is unclear which of the 18 positive deltas reflect a robust effect. The manuscript would be strengthened by repeated-seed runs for at least the configurations that exhibit these instabilities, and by a statement of whether gate clipping was applied in each reported configuration.
minor comments (7)
- [Section 5.3, Figure 5 caption] The caption contains a duplicated word ('across across training steps'), and Section 5.1's 'the first layer tend to learn negative gate values' should read 'tends.'
- [Section 5.6, Lemma 1] The lower bound M in 'M <= UP(...) <= O(L)' is never defined, and the theorem statement in Eq. (10) is badly garbled in the compiled text; please restate both cleanly.
- [Appendix C, Eq. (16)] The variance recurrence is asserted as 'following the variance analysis in [13]' without derivation; given that Theorem 1 depends on it, at least a brief derivation or a precise pointer to the specific equation in [13] is needed.
- [Table 2] The DeepNorm+GPAS 1B entry reports an improvement of '-1386' over a diverged baseline; report the absolute perplexity and note that the baseline diverged instead of reporting a delta.
- [Section 3.2] The statement that applying GPAS after LayerNorm 'does not bring performance gain' for Post-LN is an empirical claim without shown results; either include the experiment in an appendix or mark it as preliminary.
- [Section 4.1, Section 5.3] The gate-gradient clipping exception for Sandwich+GPAS at 1B (threshold 0.01) is not listed in the configuration summary of Table 1; move it into the setup description.
- [Table 2] DeepNorm+GPAS at 71M (34.97) and 130M (26.65) is worse than the unaugmented Sandwich-LN and LNS baselines; the 'improvement over base' format hides this, so add cross-architecture comparisons in the text when claiming versatility.
Circularity Check
No significant circularity: the GPAS gains are empirical, and the theory section, while relying on an overlapping-author variance recurrence, is post-hoc analysis rather than the source of the predictions.
full rationale
The central claim—consistent perplexity improvements from GPAS—rests on Table 2, where the learnable gates α_l are optimized by Adam, not fitted to the evaluation perplexity or to target deltas; no reported number is a renamed fitted parameter. The predefined-gate ablation actually fixes gates extracted from a trained model and shows worse performance, which is the opposite of fitting-to-predict. The gradient-preservation identity ∂x_{l+1}/∂x'_{l+1}=I is a definitional consequence of the stop-gradient operator, but the paper uses it as a mechanism, not as a fitted prediction, and the empirical gains are verified independently in Figures 2 and 4. Section 5.6 and Appendix C do import the variance recurrence from [13], a paper with overlapping authors, and insert the GPAS factor (1−SiLU(α_l)) into that recurrence; this makes Theorem 1 an assumption-based analysis rather than an independent theoretical check, but it is explicitly presented as post-hoc ('Based on our experimental findings') and is not load-bearing for the headline result. The paper also self-reports instability (gradient spike near step 10K; gradient clipping required for Sandwich+GPAS at 1B) and a partial 7B run; these are robustness limitations, not circularity. Overall, the empirical derivation chain is self-contained, so circularity is minimal aside from the minor self-citation in the theoretical motivation.
Assumptions & free parameters
free parameters (2)
- Per-layer gate alpha_l =
learned during pretraining; initialized to 0
- Gate gradient clipping threshold =
0.01 for Sandwich+GPAS 1B and for 7B; 1.0 for other parameters
assumptions (4)
- domain assumption Variance recurrence sigma^2_x_{l+1} = sigma^2_x_l (1 + 1/sigma_x_l) (1 - SiLU(alpha_l))
- domain assumption Gaussian and mutually independent activations and weights
- ad hoc to paper alpha_l bounded and slowly varying across layers
- domain assumption LayerNorm before each sub-layer cancels any scaling applied to x'_{l+1}
Cite this review
Pith. "Pith review of GPAS: Accelerating Convergence of LLM Pretraining via Gradient-Preserving Activation Scaling." pith.science (2026). https://pith.science/paper/FFZ7H5JW
@misc{pith2026250622049,
author = {Pith},
title = {Pith review of: GPAS: Accelerating Convergence of LLM Pretraining via Gradient-Preserving Activation Scaling},
year = {2026},
howpublished = {\url{https://pith.science/paper/FFZ7H5JW}},
note = {Machine review of arXiv:2506.22049}
}
read the original abstract
Modern Large Language Models, such as the LLaMA, Qwen and DeepSeek series, predominantly adopt the Pre-LayerNorm (Pre-LN) Transformer architecture. While being stable during pretraining and scalable to large model sizes, Pre-LN suffers from an exponential growth in activation variance across layers, causing the shortcut to dominate over sub-layer outputs in the residual connection and limiting the learning capacity of deeper layers. To mitigate this issue, we propose Gradient-Preserving Activation Scaling (GPAS), a simple technique that can be used in combination with existing approaches. GPAS works by scaling down the intermediate activations while keeping their gradients unchanged. This leaves information in the activations intact, and avoids the gradient vanishing problem associated with gradient downscaling. Extensive experiments across various model sizes from 71M to 1B show that GPAS achieves consistent performance gains. Beyond enhancing Pre-LN Transformers, GPAS also shows promise in improving alternative architectures such as Sandwich-LN and DeepNorm, demonstrating its versatility and potential for improving training dynamics in a wide range of settings. Our code is available at https://github.com/dandingsky/GPAS.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
Gomez, Lukasz Kaiser, and Illia Polosukhin
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 2017
work page 2017
-
[2]
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024
arXiv 2024
-
[3]
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024
arXiv 2024
-
[4]
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025
arXiv 2025
-
[5]
Adaptive input representations for neural language modeling
Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. ICLR, 2019
work page 2019
-
[6]
Generating long sequences with sparse transformers
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019
arXiv 1904
-
[7]
Learning deep transformer models for machine translation
Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F Wong, and Lidia S Chao. Learning deep transformer models for machine translation. ACL, 2019
work page 2019
-
[8]
Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning llms to high sparsity
Lu Yin, You Wu, Zhenyu Zhang, Cheng-Yu Hsieh, Yaqing Wang, Yiling Jia, Mykola Pech- enizkiy, Yi Liang, Zhangyang Wang, and Shiwei Liu. Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning llms to high sparsity. ICML, 2024
work page 2024
Show all 42 references
-
[9]
The unreasonable ineffectiveness of the deeper layers
Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A Roberts. The unreasonable ineffectiveness of the deeper layers. arXiv preprint arXiv:2403.17887, 2024
2024 arXiv
-
[10]
Shortgpt: Layers in large language models are more redundant than you expect
Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprint arXiv:2403.03853, 2024
2024 arXiv
-
[11]
Layer normalization
Jimmy Lei Ba. Layer normalization. arXiv preprint arXiv:1607.06450, 2016
2016 arXiv
-
[12]
Mix-ln: Unleashing the power of deeper layers by combining pre-ln and post-ln
Pengxiang Li, Lu Yin, and Shiwei Liu. Mix-ln: Unleashing the power of deeper layers by combining pre-ln and post-ln. arXiv preprint arXiv:2412.13795, 2024
2024 arXiv
-
[13]
The curse of depth in large language models
Wenfang Sun, Xinyuan Song, Pengxiang Li, Lu Yin, Yefeng Zheng, and Shiwei Liu. The curse of depth in large language models. arXiv preprint arXiv:2502.05795, 2025
2025
-
[14]
Deepnet: Scaling transformers to 1,000 layers
Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. Deepnet: Scaling transformers to 1,000 layers. TPAMI, 2024
2024
-
[15]
Cogview: Mastering text-to-image generation via transformers
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. NeurIPS, 34:19822–19835, 2021
2021
-
[16]
Group normalization
Yuxin Wu and Kaiming He. Group normalization. In ECCV, pages 3–19, 2018
2018
-
[17]
Root mean square layer normalization
Biao Zhang and Rico Sennrich. Root mean square layer normalization. NeurIPS, 32, 2019
2019
-
[18]
Understanding and improving layer normalization
Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, and Junyang Lin. Understanding and improving layer normalization. arXiv preprint arXiv:1911.07013, 2019
1911 arXiv
-
[19]
Transformers without normalization, 2025
Jiachen Zhu, Xinlei Chen, Kaiming He, Yann LeCun, and Zhuang Liu. Transformers without normalization, 2025
2025
-
[20]
On layer normalization in the transformer architecture
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. In ICML, pages 10524–10533. PMLR, 2020. 12
2020
-
[21]
B2t connection: Serving stability and performance in deep transformers
Sho Takase, Shun Kiyono, Sosuke Kobayashi, and Jun Suzuki. B2t connection: Serving stability and performance in deep transformers. ACL, 2023
2023
-
[22]
Peri-ln: Revisiting normalization layer in the transformer architecture
Jeonghoon Kim, Byeongchan Lee, Cheonbok Park, Yeontaek Oh, Beomjun Kim, Taehwan Yoo, Seongjin Shin, Dongyoon Han, Jinwoo Shin, and Kang Min Yoo. Peri-ln: Revisiting normalization layer in the transformer architecture. arXiv preprint arXiv:2502.02732, 2025
2025 arXiv
-
[23]
Fu, Stefano Ermon, Atri Rudra, and Christopher Ré
Tri Dao, Daniel Y . Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. arXiv preprint arXiv:2205.14135, 2022
2022 arXiv
-
[24]
Gqa: Training generalized multi-query transformer models from multi-head checkpoints
Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245, 2023
2023 arXiv
-
[25]
Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y . Wu, Zhenda Xie, Y . K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. Deepseekmoe: Towards ultimate expert specialization in mixture-of-...
2024 arXiv
-
[26]
Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model
DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, 2024
2024 arXiv
-
[27]
Prajit Ramachandran, Barret Zoph, and Quoc V . Le. Searching for activation functions, 2017
2017
-
[28]
Relora: High- rank training through low-rank updates
Vladislav Lialin, Sherin Muckatira, Namrata Shivagunde, and Anna Rumshisky. Relora: High- rank training through low-rank updates. In ICLR, 2023
2023
-
[29]
Galore: Memory-efficient llm training by gradient low-rank projection
Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient llm training by gradient low-rank projection. ICML, 2024
2024
-
[30]
Transformer-xl: Attentive language models beyond a fixed-length context
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. ACL, 2019
2019
-
[31]
Sandwich batch normal- ization: A drop-in replacement for feature distribution heterogeneity
Xinyu Gong, Wuyang Chen, Tianlong Chen, and Zhangyang Wang. Sandwich batch normal- ization: A drop-in replacement for feature distribution heterogeneity. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2494–2504, 2022
2022
-
[32]
Llama: Open and efficient foundation language models
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023
2023 arXiv
-
[33]
Glu variants improve transformer
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020
2002 arXiv
-
[34]
Spike no more: Stabilizing the pre-training of large language models
Sho Takase, Shun Kiyono, Sosuke Kobayashi, and Jun Suzuki. Spike no more: Stabilizing the pre-training of large language models. arXiv preprint arXiv:2312.16903, 2023
2023 arXiv
-
[35]
Adam: A method for stochastic optimization
Diederik P Kingma. Adam: A method for stochastic optimization. ICLR, 2015
2015
-
[36]
Exploring the limits of transfer learning with a unified text-to-text transformer
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020
2020
-
[37]
Documenting large webtext corpora: A case study on the colossal clean crawled corpus, 2021
Jesse Dodge, Maarten Sap, Ana Marasovi´c, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus, 2021
2021
-
[38]
Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models
Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka-Wei Lee. Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. EMNLP, 2023
2023
-
[39]
Lisa: Layerwise importance sampling for memory-efficient large language model fine-tuning
Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. Lisa: Layerwise importance sampling for memory-efficient large language model fine-tuning. 2024. 13
2024
-
[40]
The language model evaluation harness, 07 2024
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...
2024
-
[41]
Llama 2: Open foundation and fine-tuned chat models, 2023
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models, 2023
2023
-
[42]
E. T. Whittaker and G. N. Watson. A Course of Modern Analysis. Cambridge Mathematical Library. Cambridge University Press, Cambridge, 4 edition, 1996. 14 A Pretrain Results on 7B-Parameter Models To further verify the effectiveness of GPAS on larger scale models, we perform pr...
1996
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.