{"id":"3f82d9ac-a6d3-41ce-898f-e067caba2ba0","arxiv_id":"2508.20577","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"MERIT, a max-norm and element-wise trust-ratio optimizer, improves large-batch GPT-2 and Llama training and matches small-batch downstream scores at 6k batch size.","lead":"This paper introduces MERIT, an optimizer that scales weight updates with maximum-norm and element-wise ratios to keep attention scores stable during large-batch language model training. It reports that MERIT matches small-batch quality at batch sizes up to 6,000 for GPT-2 Medium, roughly 12 times larger than the standard 480.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline 'no degradation' rests on a single-run 0.11-point average gap and a MERIT small-batch baseline the paper never reports.","rationale":"The reader identified the missing seeds/variance and absent MERIT-small-batch baseline as the weakest assumption, and I agree those are the most immediate evidence-quality gaps. My independent reading confirms the strongest claim — no degradation at 6k versus 480 — is a load-bearing single-run comparison against only one reference point. I would weight the concern slightly more onto the missing AdamW-large-batch control, because without it the experiment does not even measure the 'degradation' the claim says was avoided; the phrasing 'no degradation' should mean large-batch MERIT matches small-batch AdamW, but the paper's own Figure 1 and Section 5.2 show AdamW at large batch degrades substantially, so the correct comparison for the batch-size-invariance claim is between MERIT 6k and AdamW 6k, not just against AdamW 480. This also meshes with the proof limitation: the convergence theorem explicitly analyzes MERIT-W (a lower-bound simplification, as the authors state) and the element-wise ratio and clipping that constitute the core novelty are not part of the analyzed update, so theoretical support is conditional on an unproved extension. I do not see internal inconsistency or a stronger attack; the empirical claims are plausible, and the 1% overhead and consistent loss curves across GPT-2 and Llama sizes provide independent support, so CONDITIONAL rather than REJECT or UNVERDICTED is the right adjustment. The condition should be: re-run the key comparison with multiple seeds and an AdamW-large-batch arm, and provide a proof (or explicit proof sketch) for the full Algorithm 1 rather than its weight-wise restriction. If those checks pass, the central claim is credible; if the seed spread swamps the 0.11-point gap, the headline claim needs to be weakened to 'comparable within noise.'","tokens_in":18250,"tokens_out":2051,"duration_ms":16843,"concrete_test":"Run GPT-2 Medium (355M) under the Section 5.3 protocol with three seeds for each of: AdamW batch 480, AdamW batch 6k, and MERIT batch 6k, all with 48B tokens; report mean and per-task standard deviation of the five-task average. If the AdamW-large-batch mean is more than one standard deviation below AdamW 480 and MERIT 6k does not exceed AdamW 480 by a margin larger than the seed spread, the 'no degradation' claim is not supported. Independently, re-derive Theorem 1's bound from Algorithm 1's actual update (row/column max element-wise ratio with clip), checking whether the sign-matching probability argument survives the element-wise ratio's dependence on ||w(i)||_m and ||w(j)||_m; if the proof requires MERIT-W assumptions at any step, state that convergence for Algorithm 1 is unproved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that MERIT enables a 6k batch size with no degradation versus AdamW at batch 480 (Section 5.3, Table 1) depends on a single run of each configuration. Table 1 shows GPT-2 Medium MERIT averaging 46.70 versus AdamW's 46.59 across five tasks, a gap smaller than typical seed-to-seed variance for 355M-parameter pretraining. The same table compares MERIT at 4k/6k against AdamW at 480, and no MERIT-small-batch or AdamW-large-batch balanced baseline is given; the design cannot separate the optimizer's general quality from its batch-size invariance, since Section 5.2 shows MERIT also beats AdamW at the same (already large) batch size. Crucially, the claimed 12x batch-size benefit is not demonstrated by the table: no AdamW-large-batch row with the same training budget is shown in that protocol, so 'no degradation' is inferred from a comparison to AdamW-small-batch rather than measured against the large-batch AdamW baseline that would quantify what was recovered. Additionally, the convergence proof (Theorem 1, Section 4.4, Appendix J) explicitly analyzes only MERIT-W, the weight-wise max-norm variant, and defends this by asserting the element-wise clipping bounds updates; the proof never covers the row/column element-wise ratio or the clipping's interaction with it, so the convergence guarantee does not apply to the actual Algorithm 1. The paper itself notes a need for caution when extrapolating beyond 770M (Appendix B), and Section 5.3's headline is presented at only 125M and 355M scales.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MERIT, a new optimizer for large-batch training of language models. MERIT modifies LAMB by replacing the l2-norm weight-wise trust ratio with a max-norm weight-wise ratio, adding element-wise row/column trust ratios, and applying an element-wise clipping operation. The motivation is that large-batch AdamW training causes a sharp increase in max attention logits and that LAMB's l2-norm-based ratio does not control the maximum query/key weight values. Experiments compare MERIT against AdamW, LAMB, Lion, and Sophia-G on GPT-2 small/medium/large and on two Llama sizes, reporting validation loss, zero-shot downstream tasks, Hessian curvature, and computational overhead. The headline claim is that MERIT enables a 6k batch size for GPT-2 Medium without performance degradation compared with AdamW at batch size 480, using 48B training tokens. A convergence theorem is provided for a simplified weight-wise max-norm variant, MERIT-W.","tokens_in":18666,"tokens_out":9385,"duration_ms":86616,"significance":"If the central claim were established, the paper would make a practically valuable contribution: an optimizer-only change that supports roughly 12x larger batches while matching small-batch generalization would substantially reduce wall-clock time for LLM pretraining. The work has several genuine strengths: it evaluates four model sizes across two architectures, releases code, provides component ablations, measures computational overhead, and offers a Hessian-curvature analysis that connects MERIT to flatter minima. The max-attention-logit diagnosis is a useful, falsifiable observation that connects the optimizer design to a measurable training instability. However, the headline claim currently rests on a single-run 0.11-point average gap, and the convergence theorem does not analyze the actual algorithm as submitted; both issues are fixable but are load-bearing for the paper's main assertions.","major_comments":[{"comment":"The headline claim that MERIT enables a 6k batch size without performance degradation compared with the standard 480 batch size is not supported by the reported evidence. For GPT-2 Medium, Table 1 shows MERIT at batch 6k averaging 46.70 across five tasks versus AdamW at batch 480 averaging 46.59; this 0.11-point difference comes from a single run of each configuration and is within typical run-to-run variability for a 355M-parameter model. In addition, per-task results are mixed: MERIT is worse on COPA (70 vs 71) and HellaSwag (32.32 vs 32.39). More importantly, the table contains no MERIT run at batch 480 and no AdamW run at batch 4k or 6k, so the experiment cannot distinguish a genuine batch-size-invariance property of MERIT from an optimizer-quality advantage that would appear at any batch size. The authors should provide multiple seeds with standard errors for the headline comparison and add the two missing baselines (MERIT at small batch, AdamW at large batch) to support the claimed 12x batch-size benefit.","section":"Section 5.3, Table 1"},{"comment":"Theorem 1 analyzes a different algorithm from the one proposed as MERIT. The proof studies MERIT-W with the update in Appendix J Eq. (5), w_{t+1} = w_t - eta_t * (||w_t||_m / ||u_t||_m) * u_t, whereas Algorithm 1 lines 10-12 use the element-wise ratio s_t(i,j) = max{r_t(i), c_t(j), b_t} followed by clip(s_t * (u_t + lambda w_t), 1). The paper's statement that clipping makes it sufficient to study the weight-wise max-norm ratio as a lower bound is not justified: although b_t <= s_t(i,j), this means the pre-clip MERIT update can be larger than the analyzed MERIT-W update, and the clipping operation is not part of the proof. The proof also sets beta_1 = 0 and lambda = 0, so it additionally omits the weight-decay term present in Algorithm 1. As written, the convergence guarantee does not apply to the actual algorithm. The authors should either prove convergence for Algorithm 1 or explicitly restrict the theorem to MERIT-W and support the full algorithm only empirically.","section":"Section 4.4, Appendix J, Eq. (5)"},{"comment":"All main comparisons and ablations are reported as single runs without seeds or variance estimates. The text states in Section 5.2 that MERIT 'consistently achieves lower validation loss' than the baselines, and the ablation study in Figure 9 and Table 3 draws conclusions from differences such as 3.470 (AdamW) versus 3.355 (LAMB) versus 3.280 (MERIT) in GPT-2 Small validation loss. Without at least a small number of seeds and confidence intervals, these differences cannot be distinguished from optimization noise, and the word 'consistently' is not statistically supported. The authors should report seed variation for the central comparisons and for each of the ablations, or explicitly characterize the reported numbers as single-trial observations and soften the claims accordingly.","section":"Section 5.1, Figures 4-6, Table 3"}],"minor_comments":[{"comment":"The column header 'HelllaSwag' is a typo and should read 'HellaSwag'.","section":"Table 1"},{"comment":"The caption of Figure 6 says the Llama experiments use '2.6B tokens ... for GPT-2 small and 8B tokens ... for GPT-2 medium', which mixes the Llama architecture with GPT-2 size names; please clarify that these are Llama models of approximately those sizes.","section":"Section 5.2, Figure 6"},{"comment":"The clip operation in line 12 is written as clip(·, 1), while the text says it 'limits the max update magnitude to 1'; please specify whether this is an element-wise clamp to [-1, 1] or a max-norm projection, since the two interpretations lead to different algorithms.","section":"Algorithm 1, Section 4.3"},{"comment":"The proof bounds ||u_t||_m but uses u_t = m_t / (sqrt(v_t) + epsilon) in the algorithm while the analysis omits epsilon; please define the exact normalized update used in the proof and justify the bound on its max-norm.","section":"Appendix J"},{"comment":"The text says experiments use '48 billion tokens' following the protocol of Liu et al. (2024), but the rest of Section 5.1 specifies 2B/8B/16B token budgets under the Chinchilla protocol; please state explicitly which runs produced Table 1 and how the token budgets relate to the rest of the experiments.","section":"Section 5.3"},{"comment":"The quantity C_X is introduced as a token-dependent sum of absolute input values and then treated as a constant upper bound; if LayerNorm is used, please state explicitly under what conditions C_X is uniformly bounded across tokens.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"I recommend major revision rather than rejection. The proposed optimizer is plausible and the empirical package is broad, but the central 'no degradation' claim depends on a single-run 0.11-point average gap and the convergence theorem does not cover Algorithm 1 as written. Both issues are addressable: add seeds/error bars and the missing small-batch/large-batch baselines, and either prove convergence for the actual algorithm or explicitly limit the theoretical claim to MERIT-W. The paper should also clarify the clipping semantics in Algorithm 1."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"MERIT is a useful incremental optimizer with a clear mechanism and a consistent empirical signal. The genuinely new pieces are the max-norm trust ratio and the element-wise row/column ratio, both simple modifications of LAMB that are easy to implement and clearly motivated by the attention-logit story. The ablations are more thorough than most optimizer papers: norm choice, clipping, weight-wise bound, and element-wise ratio each show a measurable effect. Validation losses are consistently lower than AdamW, LAMB, Lion, and Sophia-G across GPT-2 small/medium/large and two Llama sizes, with code released. That's real evidence the optimizer helps large-batch pretraining.\n\nNow the soft spots, in proportion. The headline claim — MERIT enables 6k batch without degradation versus 480 — is not actually tested. Table 1 compares MERIT at 6k/4k to AdamW at 480, so you cannot separate general optimizer improvement from batch-size invariance. The paper even shows in Section 5.2 that MERIT outperforms AdamW at the same large batch size, so some of the gain is simply a better optimizer. There are no seeds or error bars anywhere; a 0.11-point downstream average gap from a single run is well within typical noise for 355M models. The convergence proof analyzes only MERIT-W, the weight-wise max-norm variant, not the full algorithm with element-wise ratios and clipping; the claim that clipping makes this sufficient is not formal. The paper's own limitations section appropriately cautions against extrapolating beyond 770M, so the 6k claim should be read as a small-scale observation.\n\nNone of this kills the paper. The mechanism is plausible, the empirical package is consistent, and the public code makes it easy for others to reproduce. The central idea — use max-norm and local row/column structure in the trust ratio — is worth taking seriously. But the 'no degradation' claim is stronger than the data. I'd recommend sending to peer review with a request for multi-seed runs, an AdamW-at-large-batch baseline, a MERIT-at-small-batch baseline, and either a proof covering the actual algorithm or a much clearer statement that the theory is only for a simplified variant. For someone working on large-batch LLM training, this is worth reading and probably worth citing once the missing comparisons land.","headline":"Useful optimizer with an honest mechanism and consistent empirical gains, but the headline 'no degradation at 6k batch' is under-supported by the experimental design.","tokens_in":19154,"tokens_out":3116,"would_cite":true,"duration_ms":28260,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MERIT claims that replacing LAMB's layer-wise $l_2$ trust ratio with max-norm, row/column-local ratios lets GPT-2 Medium train at a 6k batch size with no quality loss versus the 480-batch baseline.","keywords":["large-batch training","language model pretraining","optimizer","trust ratio","max attention logit","LAMB","GPT-2","attention entropy collapse"],"falsifier":"Repeat the paper's 48B-token GPT-2 Medium comparison with, say, five seeds each for MERIT at batch 6K and AdamW at batch 480, and compute the five-task average with standard errors. If the MERIT average does not exceed the AdamW average, or their confidence intervals overlap broadly, the \"no performance degradation\" claim fails as stated. Separately, if the max attention logit in the 12th self-attention layer under MERIT grows to the level LAMB reaches at the same step count, the proposed mechanism is not what is doing the work.","tokens_in":18076,"feed_emoji":"⚡","tokens_out":6643,"duration_ms":56898,"temperature":0.7,"pith_summary":"This paper tries to establish that the performance collapse of large-batch language-model training is driven by runaway max attention logits in middle self-attention layers, and that an optimizer can fix it without changing the model or the data. It introduces MERIT, which rescales each parameter update by a trust ratio built from max-norms and row/column-local norms instead of LAMB's single layer-wide $l_2$-norm ratio. On GPT-2 models, the paper reports that MERIT beats AdamW, LAMB, Lion, and Sophia-G in validation loss, and that with 48B training tokens GPT-2 Medium can use a batch size of 6,000 with no measured generalization drop against AdamW's standard batch size of 480. A reader would care because, if true, an optimizer-only swap would cut wall-clock pretraining time by using many more GPUs in parallel without sacrificing model quality.","feed_headline":"An optimizer swap enables 6k-batch GPT-2 training with no quality loss","feed_subtitle":"MERIT rescales updates by local max-norm ratios, matching the 480-batch baseline at 6,000 samples per step.","key_machinery":"The load-bearing object is the element-wise trust ratio $s^{(i,j)}_t=\\max\\{r^{(i)}_t, c^{(j)}_t, b_t\\}$, where $r$ and $c$ are row-wise and column-wise max-norm ratios $\\|w^{(i)}\\|_m/\\|u^{(i)}+\\lambda w^{(i)}\\|_m$ and $\\|w^{(j)}\\|_m/\\|u^{(j)}+\\lambda w^{(j)}\\|_m$, and $b_t$ is the whole-matrix max-norm ratio. Rows are similar because attention is multi-headed, and columns are similar because of outlier dimensions, so a per-element ratio isolates local structure instead of letting one extreme value contaminate the whole layer's update scale. An element-wise clipping operation caps every update at magnitude 1. Together these pieces bound the max-norm of query and key weights, hence bound the max attention logit and prevent the one-hot attention collapse that the paper identifies as the large-batch failure mode.","core_discovery":"MERIT's central claim is that a finer-grained, max-norm-based version of LAMB's trust ratio removes the large-batch training gap for decoder-only language models. In LAMB, the trust ratio is $R=\\|w_t\\|/\\|u_t+\\lambda w_t\\|$ with the $l_2$ norm; MERIT computes a weight-wise max-norm ratio, row- and column-wise max-norm ratios, and sets each element's ratio to the maximum of its row ratio, column ratio, and the weight-wise ratio, then clips the update magnitude to 1. Because the attention logit is bounded by $\\sqrt{d}\\,M_Q M_K C_X^2$, where $M_Q$ and $M_K$ are the max-norms of the query and key weight matrices, controlling max-norm keeps the max attention logit from spiking in mid-depth layers where LAMB still fails. The paper reports that this lowers validation loss on GPT-2 Small, Medium, and Large at batch sizes 1K, 4K, and 8K, and that on GPT-2 Medium with 48B tokens, MERIT at batch 6K matches AdamW at batch 480 on five downstream tasks, averaging 46.70 versus 46.59.","pith_inferences":["If the max-attention-logit diagnosis is right, the same failure signature could be monitored online during pretraining: a rising max attention logit in mid-depth layers would signal that the trust-ratio granularity is too coarse, independent of validation-loss movement.","A testable extension is to apply MERIT at 1B-plus parameter scale with in-context-learning benchmarks; the paper evaluates up to 770M parameters and explicitly leaves larger-scale validation open.","The optimizer applies its element-wise ratio to all tensors, not just query and key weights, so ablating attention-only updates could reveal whether the gains come from constraining attention logits or from a more general update-scaling benefit.","The paper does not report a MERIT small-batch run that matches a LAMB small-batch run, so part of the advantage could be optimizer quality rather than batch-size invariance; comparing both optimizers at small and large batches would separate the two effects."],"forward_implications":["An optimizer-only change, with no architectural or data changes, can remove the large-batch generalization gap at the tested GPT-2 scales.","GPT-2 Medium can be trained with a batch size of 6,000 under a 48B-token budget and still match AdamW's 480-batch downstream average, implying about 12.5x more data parallelism at equal quality.","The gap between MERIT and LAMB grows with model size in the paper's runs, so the reported advantage is not confined to the smallest model.","MERIT's convergence point has a smaller top Hessian eigenvalue and trace than AdamW's, consistent with landing in a flatter region of the loss landscape.","Because only the optimizer changes, the speed-up composes with other large-batch techniques such as gradient accumulation and distributed data parallelism."],"supporting_citations":[{"why":"Supplies the LAMB trust-ratio design that MERIT modifies and the primary large-batch baseline it must beat.","marker":"You et al., 2020"},{"why":"Documents that transformer loss diverges when the max attention logit exceeds roughly 10^4, motivating the paper's target metric.","marker":"Wortsman et al., 2024"},{"why":"Identifies attention entropy collapse as a transformer training failure mode that large-batch training triggers.","marker":"Zhai et al., 2023"},{"why":"Provides the QK-norm baseline that stabilizes logits but degrades large-batch language-model performance, which MERIT is compared against.","marker":"Dehghani et al., 2023"},{"why":"Sets the Chinchilla token budgets and batch settings used to construct the training configurations.","marker":"Hoffmann et al., 2022"},{"why":"Supplies the Sophia-G baseline and the beta hyperparameter settings adopted for the comparison.","marker":"Liu et al., 2024"},{"why":"Provides the generalization-gap and sharp-minima interpretation used in the Hessian-curvature analysis.","marker":"Keskar et al., 2017"}],"fun_headline_variants":["Max-norm trust ratios unlock 6k-batch GPT-2 training","MERIT matches batch-480 quality at 6k samples per step","Element-wise max-norm scaling removes large-batch LLM penalty","MERIT: lossless large-batch training via max-norm ratios"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reported single-run numbers are representative: the headline \"no degradation\" for GPT-2 Medium rests on a 0.11-point average downstream difference (46.70 vs 46.59) with no seeds or variance estimate, so if that gap is run-to-run noise the central claim collapses.","fun_headline_variants_meta":{"raw":{"variants":["Max-norm trust ratios unlock 6k-batch GPT-2 training","MERIT matches batch-480 quality at 6k samples per step","Element-wise max-norm scaling removes large-batch LLM penalty","MERIT: lossless large-batch training via max-norm ratios"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000571,"raw_usage":{"total_tokens":2788,"prompt_tokens":1119,"completion_tokens":1669,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":735,"completion_tokens_details":{"reasoning_tokens":1590}},"tokens_in":735,"tokens_out":1669,"duration_ms":14071,"temperature":1.0,"reasoning_tokens":1590,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:43:16.159380+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the paper's 48B-token GPT-2 Medium comparison with, say, five seeds each for MERIT at batch 6K and AdamW at batch 480, and compute the five-task average with standard errors. If the MERIT average does not exceed the AdamW average, or their confidence intervals overlap broadly, the \"no performance degradation\" claim fails as stated. Separately, if the max attention logit in the 12th self-attention layer under MERIT grows to the level LAMB reaches at the same step count, the proposed mechanism is not what is doing the work.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Identifies attention entropy collapse as a transformer training failure mode that large-batch training triggers."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the QK-norm baseline that stabilizes logits but degrades large-batch language-model performance, which MERIT is compared against."},{"cited_title":"A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Vinyals, O., Rae, J","cited_arxiv_id":null,"evidence_quote":"Sets the Chinchilla token budgets and batch settings used to construct the training configurations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Sophia-G baseline and the beta hyperparameter settings adopted for the comparison."}],"review_version":2}