Pith. sign in

REVIEW 3 major objections 6 minor 39 references

MERIT: Maximum-normalized Element-wise Ratio for Language Model Large-batch Training

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MERIT claims that replacing LAMB's layer-wise $l_2$ trust ratio with max-norm, row/column-local ratios lets GPT-2 Medium train at a 6k batch size with no quality loss versus the 480-batch baseline.

desk verdict Useful optimizer with an honest mechanism and consistent empirical gains, but the headline 'no degradation at 6k batch' is under-supported by the experimental design. read the letter →

arxiv 2508.20577 v1 pith:SGJNRTRP submitted 2025-08-28 cs.LG cs.AIcs.CL

Yang Luo , Zangwei Zheng , Ziheng Qin , Zirui Zhu , Yong Liu , Yang You This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords large-batchtraininglanguagemodelpretrainingoptimizertrustratiomaxattentionlogitLAMBGPT-2entropycollapse
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the performance collapse of large-batch language-model training is driven by runaway max attention logits in middle self-attention layers, and that an optimizer can fix it without changing the model or the data. It introduces MERIT, which rescales each parameter update by a trust ratio built from max-norms and row/column-local norms instead of LAMB's single layer-wide $l_2$-norm ratio. On GPT-2 models, the paper reports that MERIT beats AdamW, LAMB, Lion, and Sophia-G in validation loss, and that with 48B training tokens GPT-2 Medium can use a batch size of 6,000 with no measured generalization drop against AdamW's standard batch size of 480. A reader would care because, if true, an optimizer-only swap would cut wall-clock pretraining time by using many more GPUs in parallel without sacrificing model quality.

What carries the argument

The load-bearing object is the element-wise trust ratio $s^{(i,j)}_t=\max\{r^{(i)}_t, c^{(j)}_t, b_t\}$, where $r$ and $c$ are row-wise and column-wise max-norm ratios $\|w^{(i)}\|_m/\|u^{(i)}+\lambda w^{(i)}\|_m$ and $\|w^{(j)}\|_m/\|u^{(j)}+\lambda w^{(j)}\|_m$, and $b_t$ is the whole-matrix max-norm ratio. Rows are similar because attention is multi-headed, and columns are similar because of outlier dimensions, so a per-element ratio isolates local structure instead of letting one extreme value contaminate the whole layer's update scale. An element-wise clipping operation caps every update at magnitude 1. Together these pieces bound the max-norm of query and key weights, hence bound the max attention logit and prevent the one-hot attention collapse that the paper identifies as the large-batch failure mode.

What would settle it

Repeat the paper's 48B-token GPT-2 Medium comparison with, say, five seeds each for MERIT at batch 6K and AdamW at batch 480, and compute the five-task average with standard errors. If the MERIT average does not exceed the AdamW average, or their confidence intervals overlap broadly, the "no performance degradation" claim fails as stated. Separately, if the max attention logit in the 12th self-attention layer under MERIT grows to the level LAMB reaches at the same step count, the proposed mechanism is not what is doing the work.

Watch

Extended reading notes

Core claim

MERIT's central claim is that a finer-grained, max-norm-based version of LAMB's trust ratio removes the large-batch training gap for decoder-only language models. In LAMB, the trust ratio is $R=\|w_t\|/\|u_t+\lambda w_t\|$ with the $l_2$ norm; MERIT computes a weight-wise max-norm ratio, row- and column-wise max-norm ratios, and sets each element's ratio to the maximum of its row ratio, column ratio, and the weight-wise ratio, then clips the update magnitude to 1. Because the attention logit is bounded by $\sqrt{d}\,M_Q M_K C_X^2$, where $M_Q$ and $M_K$ are the max-norms of the query and key weight matrices, controlling max-norm keeps the max attention logit from spiking in mid-depth layers where LAMB still fails. The paper reports that this lowers validation loss on GPT-2 Small, Medium, and Large at batch sizes 1K, 4K, and 8K, and that on GPT-2 Medium with 48B tokens, MERIT at batch 6K matches AdamW at batch 480 on five downstream tasks, averaging 46.70 versus 46.59.

Load-bearing premise

The load-bearing premise is that the reported single-run numbers are representative: the headline "no degradation" for GPT-2 Medium rests on a 0.11-point average downstream difference (46.70 vs 46.59) with no seeds or variance estimate, so if that gap is run-to-run noise the central claim collapses.

Editorial extensions

If this is right

  • An optimizer-only change, with no architectural or data changes, can remove the large-batch generalization gap at the tested GPT-2 scales.
  • GPT-2 Medium can be trained with a batch size of 6,000 under a 48B-token budget and still match AdamW's 480-batch downstream average, implying about 12.5x more data parallelism at equal quality.
  • The gap between MERIT and LAMB grows with model size in the paper's runs, so the reported advantage is not confined to the smallest model.
  • MERIT's convergence point has a smaller top Hessian eigenvalue and trace than AdamW's, consistent with landing in a flatter region of the loss landscape.
  • Because only the optimizer changes, the speed-up composes with other large-batch techniques such as gradient accumulation and distributed data parallelism.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the max-attention-logit diagnosis is right, the same failure signature could be monitored online during pretraining: a rising max attention logit in mid-depth layers would signal that the trust-ratio granularity is too coarse, independent of validation-loss movement.
  • A testable extension is to apply MERIT at 1B-plus parameter scale with in-context-learning benchmarks; the paper evaluates up to 770M parameters and explicitly leaves larger-scale validation open.
  • The optimizer applies its element-wise ratio to all tensors, not just query and key weights, so ablating attention-only updates could reveal whether the gains come from constraining attention logits or from a more general update-scaling benefit.
  • The paper does not report a MERIT small-batch run that matches a LAMB small-batch run, so part of the advantage could be optimizer quality rather than batch-size invariance; comparing both optimizers at small and large batches would separate the two effects.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes MERIT, a new optimizer for large-batch training of language models. MERIT modifies LAMB by replacing the l2-norm weight-wise trust ratio with a max-norm weight-wise ratio, adding element-wise row/column trust ratios, and applying an element-wise clipping operation. The motivation is that large-batch AdamW training causes a sharp increase in max attention logits and that LAMB's l2-norm-based ratio does not control the maximum query/key weight values. Experiments compare MERIT against AdamW, LAMB, Lion, and Sophia-G on GPT-2 small/medium/large and on two Llama sizes, reporting validation loss, zero-shot downstream tasks, Hessian curvature, and computational overhead. The headline claim is that MERIT enables a 6k batch size for GPT-2 Medium without performance degradation compared with AdamW at batch size 480, using 48B training tokens. A convergence theorem is provided for a simplified weight-wise max-norm variant, MERIT-W.

Significance. If the central claim were established, the paper would make a practically valuable contribution: an optimizer-only change that supports roughly 12x larger batches while matching small-batch generalization would substantially reduce wall-clock time for LLM pretraining. The work has several genuine strengths: it evaluates four model sizes across two architectures, releases code, provides component ablations, measures computational overhead, and offers a Hessian-curvature analysis that connects MERIT to flatter minima. The max-attention-logit diagnosis is a useful, falsifiable observation that connects the optimizer design to a measurable training instability. However, the headline claim currently rests on a single-run 0.11-point average gap, and the convergence theorem does not analyze the actual algorithm as submitted; both issues are fixable but are load-bearing for the paper's main assertions.

major comments (3)
  1. [Section 5.3, Table 1] The headline claim that MERIT enables a 6k batch size without performance degradation compared with the standard 480 batch size is not supported by the reported evidence. For GPT-2 Medium, Table 1 shows MERIT at batch 6k averaging 46.70 across five tasks versus AdamW at batch 480 averaging 46.59; this 0.11-point difference comes from a single run of each configuration and is within typical run-to-run variability for a 355M-parameter model. In addition, per-task results are mixed: MERIT is worse on COPA (70 vs 71) and HellaSwag (32.32 vs 32.39). More importantly, the table contains no MERIT run at batch 480 and no AdamW run at batch 4k or 6k, so the experiment cannot distinguish a genuine batch-size-invariance property of MERIT from an optimizer-quality advantage that would appear at any batch size. The authors should provide multiple seeds with standard errors for the headline comparison and add the two missing baselines (MERIT at small batch, AdamW at large batch) to support the claimed 12x batch-size benefit.
  2. [Section 4.4, Appendix J, Eq. (5)] Theorem 1 analyzes a different algorithm from the one proposed as MERIT. The proof studies MERIT-W with the update in Appendix J Eq. (5), w_{t+1} = w_t - eta_t * (||w_t||_m / ||u_t||_m) * u_t, whereas Algorithm 1 lines 10-12 use the element-wise ratio s_t(i,j) = max{r_t(i), c_t(j), b_t} followed by clip(s_t * (u_t + lambda w_t), 1). The paper's statement that clipping makes it sufficient to study the weight-wise max-norm ratio as a lower bound is not justified: although b_t <= s_t(i,j), this means the pre-clip MERIT update can be larger than the analyzed MERIT-W update, and the clipping operation is not part of the proof. The proof also sets beta_1 = 0 and lambda = 0, so it additionally omits the weight-decay term present in Algorithm 1. As written, the convergence guarantee does not apply to the actual algorithm. The authors should either prove convergence for Algorithm 1 or explicitly restrict the theorem to MERIT-W and support the full algorithm only empirically.
  3. [Section 5.1, Figures 4-6, Table 3] All main comparisons and ablations are reported as single runs without seeds or variance estimates. The text states in Section 5.2 that MERIT 'consistently achieves lower validation loss' than the baselines, and the ablation study in Figure 9 and Table 3 draws conclusions from differences such as 3.470 (AdamW) versus 3.355 (LAMB) versus 3.280 (MERIT) in GPT-2 Small validation loss. Without at least a small number of seeds and confidence intervals, these differences cannot be distinguished from optimization noise, and the word 'consistently' is not statistically supported. The authors should report seed variation for the central comparisons and for each of the ablations, or explicitly characterize the reported numbers as single-trial observations and soften the claims accordingly.
minor comments (6)
  1. [Table 1] The column header 'HelllaSwag' is a typo and should read 'HellaSwag'.
  2. [Section 5.2, Figure 6] The caption of Figure 6 says the Llama experiments use '2.6B tokens ... for GPT-2 small and 8B tokens ... for GPT-2 medium', which mixes the Llama architecture with GPT-2 size names; please clarify that these are Llama models of approximately those sizes.
  3. [Algorithm 1, Section 4.3] The clip operation in line 12 is written as clip(·, 1), while the text says it 'limits the max update magnitude to 1'; please specify whether this is an element-wise clamp to [-1, 1] or a max-norm projection, since the two interpretations lead to different algorithms.
  4. [Appendix J] The proof bounds ||u_t||_m but uses u_t = m_t / (sqrt(v_t) + epsilon) in the algorithm while the analysis omits epsilon; please define the exact normalized update used in the proof and justify the bound on its max-norm.
  5. [Section 5.3] The text says experiments use '48 billion tokens' following the protocol of Liu et al. (2024), but the rest of Section 5.1 specifies 2B/8B/16B token budgets under the Chinchilla protocol; please state explicitly which runs produced Table 1 and how the token budgets relate to the rest of the experiments.
  6. [Appendix D] The quantity C_X is introduced as a token-dependent sum of absolute input values and then treated as a constant upper bound; if LayerNorm is used, please state explicitly under what conditions C_X is uniformly bounded across tokens.

Circularity Check

0 steps flagged · score 2.0 of 10

No material circularity: MERIT's empirical claims are judged by external validation loss and downstream tasks, not by the max-attention-logit diagnostic that motivated the design; the only notable overlap is a minor non-load-bearing self-citation in the convergence analysis.

full rationale

I walked the derivation chain and found no step in which a claimed prediction reduces by construction to an input or to a self-citation chain. The optimizer design is motivated by an observed correlation between large-batch training and growth of the max attention logit, and the paper argues that max-norm trust ratios bound that logit through the inequality in Appendix D. However, the success criterion is external: validation loss on OpenWebText, perplexity on LAMBADA and WikiText, and SuperGLUE accuracies (Section 5.2, Table 1, Figure 5). Those metrics are not defined in terms of the max attention logit or of the trust-ratio formula, so the empirical comparison is not circular. The trust-ratio framework is a modification of LAMB, and the paper states 'Following the analysis in (You et al., 2020), we focus on the setting where beta1 = 0 and lambda = 0' before proving Theorem 1. This is a self-citation by an overlapping author, but the theorem is re-derived in Appendix J rather than merely asserted, and the central empirical claim does not depend on the convergence proof. One non-circular weakness should be flagged: Section 4.4 says 'we only need to consider the convergence analysis of weight-wise maximum-normalized ratio that is the lower bound of the proposed MERIT, which is noted as MERIT-W,' while Algorithm 1 includes element-wise row/column ratios and clipping. The proof therefore does not establish convergence of the actual algorithm; this is a formal coverage gap, not a circularity. Similarly, the 'no degradation' claim in Table 1 rests on single training runs and lacks a MERIT-small-batch or AdamW-large-batch control, which is a statistical and comparison-design limitation rather than a circular reduction. Because the only notable issue is a minor self-citation that is not load-bearing for the headline result, the circularity score is 2.

Assumptions & free parameters 5 free parameters · 7 assumptions · 0 invented entities

The central empirical result is an optimizer comparison, so the mathematical axioms are mostly confined to the auxiliary convergence theorem. The main domain assumptions are the row/column similarity of attention weights and the causal role of max attention logits, both supported only by the paper's own observations. No invented physical or conceptual entities are introduced beyond the optimizer itself.

free parameters (5)
  • Peak learning rates (MERIT, LAMB, AdamW) = MERIT: 9e-3,9e-3,6e-3; LAMB: 1e-2,1e-2,8e-3; AdamW: 1e-3,4e-3,2e-3
    Tuned by grid search per optimizer and model (Appendix A Table 4); all comparisons depend on these choices.
  • Peak learning rates (Lion, Sophia-G) = Lion: 1e-4,8e-5,8e-5; Sophia-G: 1e-4,1e-4,2e-4
    Grid-searched baseline LRs for the additional optimizers compared in Figure 4.
  • Element-wise clipping bound = 1
    Algorithm 1 line 12 caps every update magnitude at 1; no sensitivity analysis of this value is provided.
  • Beta and epsilon settings = AdamW 0.9/0.95; Lion 0.95/0.98; Sophia-G 0.92/0.99; MERIT epsilon not reported
    Baseline betas follow Liu et al. (2024); MERIT's epsilon and betas are not fully specified in Section 5.1.
  • Weight decay, warmup, gradient clipping = weight decay 0.1; warmup ratio 0.02; grad clip 1.0
    Global hyperparameters chosen for fair comparison and standard training practice.
assumptions (7)
  • standard math The loss function is Li-smooth per block and stochastic gradients have bounded variance (Assumptions 1 and 2).
    Used in the convergence proof in Section 4.4 and Appendix J to obtain the descent inequality.
  • standard math Gradients are bounded element-wise by G (Assumption 3).
    Invoked in the proof of Theorem 1 to bound the adaptive update terms.
  • ad hoc to paper The max-norm of the normalized update u satisfies the bound sqrt(2 log(d)/(1-beta2)) and sqrt(v_t) <= G.
    Stated without derivation in Appendix J; this bound is not a standard consequence of Assumptions 1 to 3 and appears to be carried over from sign-SGD style analyses.
  • domain assumption Input embeddings X satisfy a bounded absolute-sum constant C_X after LayerNorm.
    Used in Appendix D to bound the max attention logit by sqrt(d) * M_Q * M_K * C_X^2.
  • domain assumption Rows and columns of query and key weight matrices are highly similar during large-batch training.
    Motivates the row/column max-norm ratios in Section 4.2; the supporting evidence is Figure 3, which shows only the first attention layer of GPT-2 small.
  • domain assumption The gradients of the loss with respect to W_Q and W_K are independent of the learning rate eta.
    Used in Appendix C to argue that max attention logit change is linearly proportional to eta; this is not justified for adaptive optimizers.
  • ad hoc to paper Max attention logit growth is a primary cause of large-batch AdamW performance degradation.
    Core motivating observation in Section 3.1 and Figure 2; the paper shows correlation but no controlled causal evidence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MERIT: Maximum-normalized Element-wise Ratio for Language Model Large-batch Training." pith.science (2026). https://pith.science/paper/SGJNRTRP

@misc{pith2026250820577,
  author       = {Pith},
  title        = {Pith review of: MERIT: Maximum-normalized Element-wise Ratio for Language Model Large-batch Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SGJNRTRP}},
  note         = {Machine review of arXiv:2508.20577}
}
abstract

Large-batch training has become a cornerstone in accelerating the training of deep neural networks, yet it poses challenges in optimization and generalization. Existing optimizers like AdamW present performance degradation during language models' large-batch training, due to the information bottleneck in attention layers caused by the sharp increase of max attention logit. While the LAMB optimizer partially addresses this issue, some attention layers still face this issue. The reason is that $l_2$-norm-based trust ratios in LAMB are less effective in directly influencing the max value of query/key weights. Furthermore, the weight-wise trust ratio in LAMB is error-prone as it overlooks relationships of weight values within rows or columns. Building on these observations, we propose a novel optimizer, MERIT, which leverages the max-norm to calculate the trust ratio to constrain the max attention logit more effectively. Moreover, we further construct element-wise trust ratios to provide more robust update scaling by focusing on local weight structures. Extensive experiments of large-batch training across various sizes of GPT-2 models demonstrate the superior performance of MERIT. Notably, during the training of GPT-2 Medium, MERIT enables a 6k batch size without any performance degradation compared to the standard batch size (480) with 48B training tokens. This work highlights the importance of considering the max attention logit and finer-granularity trust ratio in large-batch training. It successfully improves the training stability and paves the way for larger batch usage, enabling faster development and iteration of large language models. Code is available at https://github.com/NUS-HPC-AI-Lab/MERIT.

Figures

Figures reproduced from arXiv: 2508.20577 by the authors.

Figure 1
Figure 1. For both GPT-2 models, the connection between batch size and the number of steps required to reach a specific validation loss follows a similar pattern. At first, as the batch size increases, there is a phase of ideal scaling (shown by a dotted line) where doubling the batch size cuts the necessary steps in half. This is followed by a period where the benefits start to decrease. Eventually, a point is reached where … view at source ↗
Figure 2
Figure 2. Max attention logit of self-attention layers during the large-batch training of GPT-2 medium model using three optimizers. (a) Max Attention Logit of first self-attention layer. (b) Max Attention Logit of medium (12th) self-attention layer. A comprehensive visualization is availabe in Appendix I. 2.2. Max Attention Logit The relationship between max attention logit and training stability of transformers has been exp… view at source ↗
Figure 3
Figure 3. Analysis of GPT-2 small’s first attention layer reveals patterns in query-key weight magnitudes: weights show high similarities within both rows (arising from multi-headed attention architecture) and columns (due to the outlier dimension phenomenon). w (i) . For vectors u and v ∈ R d , we use u 2 to represent the element-wise square operation and u/v to represent the element-wise division operation. We use ∥ · ∥, ∥ … view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Final validation loss. (a) GPT-2 Small (125M, batch size=1K). AdamW: 3.470, LAMB: 3.355, MERIT: 3.280 (b) GPT-2 Medium (355M, batch size=4K). AdamW: 3.172, LAMB: 3.068, MERIT: 2.982. (c) GPT-2 Large (770M, batch size=8K). AdamW: 3.039, LAMB: 2.971, MERIT: 2.897. 5. Exp…
Figure 5
Figure 5. Figure 5: Zero-shot evaluation on LAMBADA and WikiText. With the same number of steps, language models large-batch pre-trained with MERIT outperform models pre-trained with AdamW and LAMB on both tasks with lower perplexity scores [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Final validation loss. (a) Llama 130M (batch size=1K). AdamW: 3.277, LAMB: 3.265, MERIT: 3.199 (b) Llama 350M (batch size=4K). AdamW: 3.014, LAMB: 3.001, MERIT: 2.957. flat regions, characterized by small eigenvalues and trace, are frequently associated with improved g…
Figure 7
Figure 7. Figure 7: A graphical representation comparing the eigenvalues of Hessian matrices at convergence points, contrasting models pre-trained using AdamW versus MERIT.. Comparison of wall-clock time and computational re￾sources. In [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: QK-Norm leads to performance degradation although improving the feasible learning rates of GPT-2 models pre-training without divergence. 8 [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Three ablations present obvious performance degrada￾tions, which validates the necessity of all three design choices. Ablation 4: Norm Choice. To investigate the impact of the norm choice in the trust ratio computation, we conducted an ablation study where we replaced …
Figure 10
Figure 10. Figure 10: Relative ratios between max and l2 norm of WK/WQ ∈ R 1024×1024 in GPT-2 medium. Ratio is calculated as (∥W∥ − ∥W∥m)/∥W∥, in which ∥ · ∥ and ∥ · ∥m denote l2-norm and max norm [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: shows the distribution of max attention logits in small-batch (512) training of the GPT-2 medium model using the same chinchilla scaling law setting. Notably, these max attention logits are significantly lower than those observed in large-batch training scenarios. Thi…
Figure 12
Figure 12. Figure 12: Zero-shot evaluation on SuperGLUE benchmark. Given an equivalent number of training steps, models that undergo large-batch pre-training using MERIT exhibit higher accuracy than those pre-trained with AdamW and LAMB on most tasks. H. Clipping Ratio in GPT-2 Small Large…
Figure 13
Figure 13. Figure 13: presents the element-wise clipping ratio in GPT-2 small training setting (2B tokens) for the 1st, 6th, and 12th layers. The analysis of clipping effects across different layers reveals distinct patterns in gradient update behavior during training. The input layer (Lay…
Figure 14
Figure 14. Figure 14: Max attention logit of self-attention layers during the large-batch training of GPT-2 medium using three optimizers of all layers 17 [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]
Figure 15
Figure 15. Figure 15: The trigger ratio of the weight-wise trust ratio lower bound during the large-batch training of GPT-2 Medium [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 9 canonical work pages

  1. [1]

    M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., Chu, E., Clark, J

    Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., Chu, E., Clark, J. H., Shafey, L. E., Huang, Y., Meier-Hellstern, K., Mishra, G., Moreira, E., Omernick, M., Robinson, K., Ruder, S., Tay, Y., Xiao, K., Xu, Y., Zhang, Y., Abrego, G. H., Ahn, J., Austin, J., Barham, P., Botha, J., Bradbur...

    arXiv 2023

  2. [2]

    L., Kiros, J

    Ba, J. L., Kiros, J. R., and Hinton, G. E. Layer normalization, 2016. URL https://arxiv.org/abs/1607.06450

    arXiv 2016

  3. [3]

    signsgd: Compressed optimisation for non-convex problems, 2018

    Bernstein, J., Wang, Y.-X., Azizzadenesheli, K., and Anandkumar, A. signsgd: Compressed optimisation for non-convex problems, 2018. URL https://arxiv.org/abs/1802.04434

    arXiv 2018

  4. [4]

    Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, ...

    arXiv 2020

  5. [5]

    Chen, X., Liang, C., Huang, D., Real, E., Wang, K., Liu, Y., Pham, H., Dong, X., Luong, T., Hsieh, C.-J., Lu, Y., and Le, Q. V. Symbolic discovery of optimization algorithms, 2023. URL https://arxiv.org/abs/2302.06675

    arXiv 2023

  6. [6]

    Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer, N., Prabhakaran, V., Reif, E., Du, N., Hutchinson, B., Pope, R., Bradbury, J., Austin, J., Isard, M., Gur-Ari, G., Yin, P., Duke, T., Levska...

    arXiv 2022

  7. [7]

    Dehghani, M., Djolonga, J., Mustafa, B., Padlewski, P., Heek, J., Gilmer, J., Steiner, A. P., Caron, M., Geirhos, R., Alabdulmohsin, I., Jenatton, R., Beyer, L., Tschannen, M., Arnab, A., Wang, X., Riquelme Ruiz, C., Minderer, M., Puigcerver, J., Evci, U., Kumar, M., Steenkiste, S. V., Elsayed, G. F., Mahendran, A., Yu, F., Oliver, A., Huot, F., Bastings,...

    work page 2023

  8. [8]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=YicbFdNTTy

    2021

Show all 39 references
  1. [9]

    The llama 3 herd of models, 2024

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., and e...

    2024 arXiv

  2. [10]

    Sharpness-aware minimization for efficiently improving generalization

    Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=6Tm1mposlrM

    2021

  3. [11]

    and Cohen, V

    Gokaslan, A. and Cohen, V. Openwebtext corpus. http://Skylion007.github.io/OpenWebTextCorpus, 2019

    2019

  4. [12]

    Accurate, large minibatch sgd: Training imagenet in 1 hour, 2018

    Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., and He, K. Accurate, large minibatch sgd: Training imagenet in 1 hour, 2018. URL https://arxiv.org/abs/1706.02677

    2018 arXiv

  5. [13]

    A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Vinyals, O., Rae, J

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Vinyals, O., Ra...

    2022

  6. [14]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models, 2020. URL https://arxiv.org/abs/2001.08361

    2020 arXiv

  7. [15]

    S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, P

    Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, P. T. P. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=H1oyRlYgg

    2017

  8. [16]

    Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization, 2017. URL https://arxiv.org/abs/1412.6980

    2017 arXiv

  9. [17]

    BERT busters: Outlier dimensions that disrupt transformers

    Kovaleva, O., Kulshreshtha, S., Rogers, A., and Rumshisky, A. BERT busters: Outlier dimensions that disrupt transformers. In Zong, C., Xia, F., Li, W., and Navigli, R. (eds.), Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp.\ 3392--3405, Online, ...

    2021 doi

  10. [18]

    Liu, H., Li, Z., Hall, D. L. W., Liang, P., and Ma, T. Sophia: A scalable stochastic second-order optimizer for language model pre-training. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=3xHDeA8Noi

    2024

  11. [19]

    Towards efficient and scalable sharpness-aware minimization, 2022

    Liu, Y., Mai, S., Chen, X., Hsieh, C.-J., and You, Y. Towards efficient and scalable sharpness-aware minimization, 2022. URL https://arxiv.org/abs/2203.02714

    2022 arXiv

  12. [20]

    and Hutter, F

    Loshchilov, I. and Hutter, F. Decoupled weight decay regularization, 2019. URL https://arxiv.org/abs/1711.05101

    2019 arXiv

  13. [21]

    Came: Confidence-guided adaptive memory efficient optimization

    Luo, Y., Ren, X., Zheng, Z., Jiang, Z., Jiang, X., and You, Y. Came: Confidence-guided adaptive memory efficient optimization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 4442--4453, 2023

    2023

  14. [22]

    Pointer sentinel mixture models

    Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=Byj72udxe

    2017

  15. [23]

    Muennighoff, N., Rush, A., Barak, B., Le Scao, T., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T., and Raffel, C. A. Scaling data-constrained language models. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Proce...

    2023

  16. [24]

    Gpt-4 technical report, 2024

    OpenAI. Gpt-4 technical report, 2024. URL https://arxiv.org/abs/2303.08774

    2024 arXiv

  17. [25]

    Q., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fern \'a ndez, R

    Paperno, D., Kruszewski, G., Lazaridou, A., Pham, N. Q., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fern \'a ndez, R. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Erk, K. and Smith, N. A. (eds.), Proceedings of the 54th Annual Meeti...

    2016 doi

  18. [26]

    Pytorch: An imperative style, high-performance deep learning library, 2019

    Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. Pytorch: An impera...

    2019 arXiv

  19. [27]

    Outlier dimensions that disrupt transformers are driven by frequency

    Puccetti, G., Rogers, A., Drozd, A., and Dell ' Orletta, F. Outlier dimensions that disrupt transformers are driven by frequency. In Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp.\ 1286--1304, Abu Dh...

    2022 doi

  20. [28]

    Language models are unsupervised multitask learners

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. 2019. URL https://api.semanticscholar.org/CorpusID:160025533

    2019

  21. [29]

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer, 2023. URL https://arxiv.org/abs/1910.10683

    2023 arXiv

  22. [30]

    J., Lee, J., Antognini, J., Sohl-Dickstein, J., Frostig, R., and Dahl, G

    Shallue, C. J., Lee, J., Antognini, J., Sohl-Dickstein, J., Frostig, R., and Dahl, G. E. Measuring the effects of data parallelism on neural network training, 2019. URL https://arxiv.org/abs/1811.03600

    2019 arXiv

  23. [31]

    Dropout: a simple way to prevent neural networks from overfitting

    Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res., 15 0 (1): 0 1929–1958, January 2014. ISSN 1532-4435

    1929

  24. [32]

    Llama: Open and efficient foundation language models, 2023

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient foundation language models, 2023. URL https://arxiv.org/abs/2302.13971

    2023 arXiv

  25. [33]

    N., Kaiser, L

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Informatio...

    2017

  26. [34]

    Superglue: A stickier benchmark for general-purpose language understanding systems

    Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. Superglue: A stickier benchmark for general-purpose language understanding systems. In Wallach, H., Larochelle, H., Beygelzimer, A., d Alch\' e -Buc, F., Fox, E., and Garnett, R...

    2019

  27. [35]

    J., Xiao, L., Everett, K

    Wortsman, M., Liu, P. J., Xiao, L., Everett, K. E., Alemi, A. A., Adlam, B., Co-Reyes, J. D., Gur, I., Kumar, A., Novak, R., Pennington, J., Sohl-Dickstein, J., Xu, K., Lee, J., Gilmer, J., and Kornblith, S. Small-scale proxies for large-scale transformer training instabilitie...

    2024

  28. [36]

    Large batch training of convolutional networks, 2017

    You, Y., Gitman, I., and Ginsburg, B. Large batch training of convolutional networks, 2017. URL https://arxiv.org/abs/1708.03888

    2017 arXiv

  29. [37]

    Large batch optimization for deep learning: Training bert in 76 minutes, 2020

    You, Y., Li, J., Reddi, S., Hseu, J., Kumar, S., Bhojanapalli, S., Song, X., Demmel, J., Keutzer, K., and Hsieh, C.-J. Large batch optimization for deep learning: Training bert in 76 minutes, 2020. URL https://arxiv.org/abs/1904.00962

    2020 arXiv

  30. [38]

    Zhai, S., Likhomanenko, T., Littwin, E., Busbridge, D., Ramapuram, J., Zhang, Y., Gu, J., and Susskind, J. M. Stabilizing transformer training by preventing attention entropy collapse. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), ...

    2023

  31. [39]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.