Scaling depth capacity via zero/one-layer model expansion

Zhiqi Bu

REVIEW 4 major objections 4 minor 2 cited by

Training a zero- or one-layer model for most of the run, then expanding its depth, matches full-depth loss at roughly one-fifth of the compute.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 23:33 UTC pith:NZ3U7IAK

load-bearing objection Reports a genuinely new zero/one-layer progressive-training recipe with a plausible but unproven 5x speedup: an internal batch-size inconsistency undercuts the mixing-time story, and the abstract overclaims LLaMA3/DeepSeekV3 evidence. the 4 major comments →

arxiv 2511.04981 v2 pith:NZ3U7IAK submitted 2025-11-07 cs.LG

Scaling depth capacity via zero/one-layer model expansion

Zhiqi Bu This is my paper

classification cs.LG

keywords progressive trainingmodel expansiondepth scalingzero-layer initializationmixing timeWSD learning rate schedulefeature learningLLM pretraining

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that the cheapest possible start to training a deep network — a model with zero or one transformer block — can be grown into a full-depth model with almost no loss in final quality, provided the expansion happens late and under the right learning-rate schedule. On GPT2-class models it reports an 80% compute saving, about a 5x speedup, with final validation loss within 0.5% for a 124M model and within 0.2% for a 60-layer 7B model. The supporting theory views progressive training as projected gradient descent plus a good initialization, and the key empirical phenomenon is 'mixing': after expansion, the grown model's loss curve quickly meets the fixed-size training curve and tracks it. The recipe prescribes training a zero/one-layer model under a warmup-stable-decay schedule, expanding depth at roughly 80% of the run, and calibrating the expansion time with two small early-stopped runs. The abstract also states scaling-law gains on LLaMA3 and DeepSeekV3; the body's detailed experiments cover GPT2, ResNet, and MoE.

Core claim

The paper's claim is that zero/one-layer progressive training can replace most of a fixed-size deep-model run with a tiny model. Starting from a zero-layer (residual stream only) or one-layer model, training proceeds under a warmup-stable-decay schedule, and at about 80% of the total horizon the remaining depth is inserted with random initialization (or by copying the one layer). After a transient loss spike, the grown model's loss 'mixes' with the fixed-size-from-scratch curve and remains essentially matched to the end: within 0.5% validation loss at 124M parameters and within 0.2% at 7B/60 layers. Because the source model costs about 2% of the target's FLOPs and expansion is late, total co

What carries the argument

The central mechanism is the depth expansion itself, analyzed as projected gradient descent: before expansion, training is interpreted as optimizing the deep model with the not-yet-added layers held at zero; at time τ those layers are 'teleported' to a good initialization (random or copied), and ordinary SGD continues. The carrying identity is the convergence-bound gap between progressive and fixed-size training, which splits into a weighted sum of the small-model versus large-model optimum gap plus an initialization-quality term. The empirical quantity that sets the schedule is the mixing time — the amount of data (tokens or images) after expansion until the grown model's loss meets the fix

Load-bearing premise

The recipe transfers the mixing time measured in two small early-stopped runs to the full-scale run; if the mixing time is not robust across model size, batch size, or data distribution, the chosen expansion time τ ≈ 0.8T could be mistimed and the advertised 5x speedup would not materialize.

What would settle it

Run a full-scale GPT2-class or larger training with zero/one-layer progressive training at τ = 0.8T under WSD, and check whether the grown model's validation loss converges to within about 0.2% of a fixed-size run within the calibrated mixing-time budget; a final gap larger than 0.2%, or a mixing time that grows noticeably with model size, would falsify the transfer claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

If the recipe holds at scale, a multilingual or multimodal model could spend roughly 80% of its training budget as a tiny skeleton model, with full depth inserted only near the end, without sacrificing final loss.
Mixing time is measured in data seen, not iterations, so practitioners can raise the batch size after expansion without degrading the loss-compute tradeoff.
Single-stage expansion from zero/one layer dominates multi-stage stacking, because each stage's mixing can be decomposed and multi-stage costs more for the same final loss.
Hyperparameters can be reused unchanged across expansion when new layers are randomly initialized or copied, since these satisfy the feature-learning scaling condition; zero initialization breaks it and should be avoided outside residual paths.
The phenomenon appears across dense GPT2, ResNet, and MoE models, suggesting it is a property of depth growth rather than of one architecture.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

A direct, cheap test of the recipe's universality would be to measure mixing time in tokens across several data distributions, batch sizes, and model widths; if it stays stable, the two-run calibration procedure could be treated as a general scheduling law for depth growth.
If the result extends to frontier-scale training, it suggests depth can be treated as a late-stage resource, which would change how compute is allocated across data quality, context length, and multimodal stages.
The paper's 'PGD + initialization + SGD' framing implies that width expansion or expert-count expansion might exhibit similar mixing behavior; checking that would show whether the phenomenon is about capacity growth generally rather than depth specifically.
The abstract's scaling-law claim on LLaMA3 and DeepSeekV3 is not backed by experimental detail in the supplied body, so treating it as established would require the full-scale runs or scaling-law fits to be examined separately.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Reports a genuinely new zero/one-layer progressive-training recipe with a plausible but unproven 5x speedup: an internal batch-size inconsistency undercuts the mixing-time story, and the abstract overclaims LLaMA3/DeepSeekV3 evidence.

read the letter

Two things to know. First, this is a real and useful idea: prior progressive training expands from 2+ layer sources; starting from zero/one layer and using WSD to time a single expansion is new, and the ablations (150 GPT2 runs, ResNet, MoE) make the core recipe credible. Second, the paper's headline claim — 5x speedup on 7B GPT2 — is not actually supported by its own mechanism. The abstract promises LLaMA3/DeepSeekV3 scaling-law evidence that never appears in the body, and the 7B experiments have a batch-size inconsistency that breaks the mixing-time transfer argument.

The stress-test concern holds up on reading. The mixing time is calibrated in Figure 5 using batch 512, giving ~16B tokens (30k iterations). Section C.6 says mixing time is measured in tokens, not iterations, and is largely batch-size agnostic. But the 7B runs in Figure 1 use batch 64, so the post-expansion budget of 120k iterations is ~8B tokens — half the calibrated tmix. So either the run didn't actually mix, contradicting the claimed <0.2% loss difference, or tmix is not transferable in tokens across batch sizes, contradicting C.6. The paper can't have it both ways. This is the load-bearing soft spot.

Other issues are smaller but still real: single seeds, no error bars, no code, and the convergence theory (Section 4) assumes W*=[w*,x*] without justification. These are fixable. The GPT2 124M results with batch 512 are self-consistent and the zero/one-layer advantage over 2/6-layer sources is shown convincingly in Figure 8.

Who this is for: people working on efficient LLM pretraining, progressive training, or WSD schedules. It deserves a serious referee: the recipe is simple, the experiments are broad, and the central claim is important if it holds. A referee should ask for the LLaMA/DeepSeek experiments, error bars, code, and a resolution of the batch-size discrepancy. My own verdict: the novelty is real, the evidence is incomplete, and the 7B speedup should be treated as unverified until the mixing-time accounting is fixed.

Referee Report

4 major / 4 minor

Summary. The paper proposes zero/one-layer progressive training for depth expansion of residual networks, claiming that training a 0- or 1-layer model for most of the run and then expanding depth at ~80% of the horizon yields ~5x compute savings with negligible loss degradation. The method is evaluated on GPT2 (up to 7B parameters), MoE, and ResNet, and is supported by a convergence analysis for convex Lipschitz losses plus ablations of initialization, learning-rate schedule, and expansion timing. The central empirical claims are that mixing time is transferable and that WSD schedules enable late expansion.

Significance. If the reported speedups hold, the paper would be a substantial contribution to efficient LLM pre-training: zero/one-layer expansion is simple, architecture-agnostic, and avoids ordering/initialization complexities of multi-layer growth. The strength is the breadth of ablations (150 runs in Figure 8) and the concrete, falsifiable recipe in Section 7, including the mixing-time-transfer rule. However, the headline claims rest on a single 7B run without error bars, the abstract advertises LLaMA3/DeepSeekV3 results that are absent from the body, and the theoretical bound relies on an unproved structural assumption about the global optimum. These need to be fixed before the contribution is fully credible.

major comments (4)

[Section B / Section 5.2 / Figure 1] The claim that the 7B progressive run mixes is inconsistent with the paper's own mixing-time calibration. Section B states the 7B runs in Figure 1 use batch size 64 (65,536 tokens/iter). At τ=480k of T=600k, the post-expansion budget is ~7.9B tokens. Section 5.2 reports tmix≈16B tokens for 1→12-layer expansion. Thus, under the paper's own token-based transfer, the 7B run cannot have mixed by the end, so the <0.2% final-loss gap is not explained by the mechanism. If the run did mix, then tmix is not transferable in tokens across batch size, contradicting Section C.6 and Figure 18. Please provide direct evidence for mixing in this setting or revise the speedup claim.
[Abstract / body] The abstract states that 'scaling laws on LLAMA3 and DeepSeekV3 models show a 3~5x improvement in compute efficiency'. The body contains no LLaMA-3 or DeepSeek-V3 experiments; the only mention is 'LLAMA-4' in the introduction. This unsupported claim must be removed or substantiated with actual scaling-law experiments.
[Section 4, Eq. (4.5)] The derivation after Eq. (4.4) assumes W*=[w*,x*], i.e., the global minimizer of the large model coincides with the small-model minimizer in the first coordinates. This is not justified for residual networks: the optimal first-layer weights of a deep network need not be the optimum of a 0/1-layer network. Without a proof or an explicit condition, the bound in Eq. (4.5) and the 'PGD + initialization' interpretation do not follow from the preceding inequalities.
[Section 2 / Figure 1] The headline results in Figure 1 are single runs with no error bars or repeated seeds. Since the claimed advantage is a <0.2% validation-loss difference, it is unclear whether this is within run-to-run variation. Please provide repeated runs or confidence intervals for the main comparisons, especially the 7B case.

minor comments (4)

[Section 1.1] Typo: 'progrssive' should be 'progressive'.
[Section 5.2] The sentence 'expanding at 80% horizon cannot mix well' refers to Figure 5, where the WSD decay is 20% (per Section B), whereas Figure 1 uses 10% decay and τ=0.8T. Please make the schedule dependence explicit so the apparent contradiction is resolved.
[Section 4] The notation W_t=[w_t,x_t] is used before being formally introduced. Define this decomposition explicitly before Eq. (4.5).
[Figure 1 caption] The caption says 'zero-layer (red, 39M or 0.15B)'; the units are ambiguous. Clarify whether 39M is the 0/1-layer model size and 0.15B is the 12-layer target, or provide a consistent notation.

Circularity Check

0 steps flagged

No significant circularity: central speedup is a measured comparison, the theory is derived, and self-citations are background only.

full rationale

The central claims are empirical comparisons: Figure 1 directly contrasts progressive and fixed-size GPT2 runs in loss-vs-FLOPs space, and the ~5x speedup is read from the measured curves, not from any fitted constant. The convergence bound (4.3)-(4.5) is derived from standard SGD inequalities; it does not assume progressive training succeeds, and the WSD/random-init recommendations are validated independently in Figures 2, 5, and 8. The expansion timing is calibrated by measuring a mixing time on small runs (Section 5.2, recipe step 4); this is a transfer assumption, not a circular reduction, because the full 7B run could in principle have failed to mix. The paper's self-citations (Boreiko et al. 2025; Bu et al. 2021; Bu and Xu 2024) are background for the optimizer and for the convex-dynamics analogy; they are not invoked to establish the main result. The abstract's LLaMA3/DeepSeekV3 scaling-law claim is unsupported in the body, but a missing-evidence gap is distinct from circularity. The batch-size difference between the calibration runs and the 7B run (Section B vs. C.6) is an empirical validity concern, not a definitional reduction. Thus no specific circular step can be exhibited.

Axiom & Free-Parameter Ledger

0 free parameters · 6 axioms · 0 invented entities

The paper's theoretical contributions rest on a convex approximation of non-convex deep-learning dynamics and on a stated decomposition of the large model's optimum. No new physical entities or forces are introduced. The empirical recipe itself has no fitted free constants; the expansion time is set by a measured mixing-time heuristic.

axioms (6)

domain assumption Training dynamics of deep networks are adequately modeled by convex G-Lipschitz optimization
Section 4's convergence bound assumes convexity and Lipschitz loss; the paper cites empirical analogies but does not prove them for transformers.
domain assumption Residual networks satisfy L([w,0]) = L(w): zero-initialized extra layers preserve the function
Used in Section 4's Jensen step; holds for residual nets but is a structural assumption about the model family.
ad hoc to paper The large model optimum decomposes as W* = [w*, x*]
Stated in Section 4 as a simplification; not guaranteed for non-convex losses or when extra layers interact with shared parameters.
domain assumption Zero-initialized new layers kill gradient flow
Section 3.1 states this without proof; it is load-bearing for rejecting zero init and for the PGD analogy.
domain assumption muP spectral scaling condition ensures feature learning and zero-shot hyperparameter transfer
Section 3.2 relies on muP theory from Yang et al.; the paper validates empirically at small scales.
standard math Standard SGD convergence inequality for convex Lipschitz functions (Eq. 4.1)
Used to derive the progressive-training bound; a classical result.

pith-pipeline@v1.3.0-alltime-deepseek · 18623 in / 18172 out tokens · 146743 ms · 2026-08-03T23:33:05.768049+00:00 · methodology

0 comments

read the original abstract

Model depth is a double-edged sword in deep learning: deeper models achieve higher accuracy but require higher computational cost. To efficiently train models at scale, progressive training (also known as model expansion) scales up model capacity during training and significantly reduces computation with little performance degradation. In this work, we study the depth expansion of large-scale models through the lens of optimization theory and feature learning, offering insights on the initialization of new layers, hyperparameter transfer, learning rate schedule, and timing of model expansion. Specifically, we propose zero/one-layer progressive training to achieve an optimal tradeoff between computation and loss, with a comprehensive ablations on our expansion strategy. For example, zero/one-layer progressive training on GPT2 can save $\approx 80\%$ compute, or equivalently achieve an $\approx 5\times$ acceleration, while attaining a loss comparable to that of a fully trained 60-layer model with 7B parameters, thus demonstrating a mixing behavior in terms of loss. Furthermore, scaling laws on LLAMA3 and DeepSeekV3 models show a $3\sim 5\times$ improvement in compute efficiency, with an increasing advantage at larger scales.

discussion (0)

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts
cs.LG 2026-04 unverdicted novelty 7.0

Expert upcycling duplicates experts in an existing MoE checkpoint and continues pre-training to match fixed-size baseline performance with 32% less compute.
Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts
cs.LG 2026-04 unverdicted novelty 6.0

Expert upcycling expands MoE models by duplicating experts and continuing pre-training, matching baseline performance while saving 32% GPU hours in 7B-13B experiments.

Reference graph

Works this paper leans on

7 extracted references · 7 linked inside Pith · cited by 1 Pith paper

[3]

Wide neural networks of any depth evolve as linear models under gradient descent.Advances in neural information processing systems, 32, 2019

6 Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent.Advances in neural information processing systems, 32, 2019. 6 Bei Li, Ziyang Wang, Hui Liu, Yufan Jiang, Quan Du, Tong Xiao, Huizhen Wang, and Jingbo Zhu...

Pith/arXiv arXiv 2019
[7]

Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer.arXiv preprint arXiv:2203.03466, 2022

4 Greg Yang, Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer.arXiv preprint arXiv:2203.03466, 2022. 4 Greg Yang, James B Simon, and Jeremy Bernstein. A spectral condition for feature learni...

Pith/arXiv arXiv 2022
[2018]

Openmoe: An early effort on open mixture-of-experts language models.arXiv preprint arXiv:2402.01739, 2024

2 Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou, and Yang You. Openmoe: An early effort on open mixture-of-experts language models.arXiv preprint arXiv:2402.01739, 2024. 10 Cheng Yang, Shengnan Wang, Chao Yang, Yuechuan Li, Ru He, and Jingqiao Zhang. Progressively stacking 2.0: A multi-stage layerwise training method for bert ...

Pith/arXiv arXiv 2024
[2020]

Sparse upcycling: Training mixture-of-experts from dense checkpoints

1 Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. Sparse upcycling: Training mixture-of-experts from dense checkpoints. InThe Eleventh International Conference on Learning Representations. 10 Guillaume Leclerc and Aleksander Madry. The two regimes of deep ...

Pith/arXiv arXiv 2002
[2023]

Openwebtext corpus.http://Skylion007.github

5, 19 Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. Openwebtext corpus.http://Skylion007.github. io/OpenWebTextCorpus, 2019. 3 Linyuan Gong, Di He, Zhuohan Li, Tao Qin, Liwei Wang, and Tieyan Liu. Efficient training of bert by progressively stacking. InInternational conference on machine learning, pages 2337–2346. PMLR, 2019. 2, 4, 5, 9...

Pith/arXiv arXiv 2019
[2024]

Llama pro: Progressive llama with block expansion.arXiv preprint arXiv:2401.02415, 2024

10 Chengyue Wu, Yukang Gan, Yixiao Ge, Zeyu Lu, Jiahao Wang, Ye Feng, Ying Shan, and Ping Luo. Llama pro: Progressive llama with block expansion.arXiv preprint arXiv:2401.02415, 2024. 4, 14 Chen Xing, Devansh Arpit, Christos Tsirigotis, and Yoshua Bengio. A walk with sgd.arXiv preprint arXiv:1802.08770,

Pith/arXiv arXiv 2024
[2025]

Staged training for transformer language models

6 Sheng Shen, Pete Walsh, Kurt Keutzer, Jesse Dodge, Matthew Peters, and Iz Beltagy. Staged training for transformer language models. InInternational Conference on Machine Learning, pages 19893–19908. PMLR, 2022. 2, 4, 5, 8, 14, 19 Zhen Tan, Daize Dong, Xinyu Zhao, Jie Peng, Yu Cheng, and Tianlong Chen. Dlo: Dynamic layer operation for efficient vertical ...

Pith/arXiv arXiv 2022

[1] [3]

Wide neural networks of any depth evolve as linear models under gradient descent.Advances in neural information processing systems, 32, 2019

6 Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent.Advances in neural information processing systems, 32, 2019. 6 Bei Li, Ziyang Wang, Hui Liu, Yufan Jiang, Quan Du, Tong Xiao, Huizhen Wang, and Jingbo Zhu...

Pith/arXiv arXiv 2019

[2] [7]

Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer.arXiv preprint arXiv:2203.03466, 2022

4 Greg Yang, Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer.arXiv preprint arXiv:2203.03466, 2022. 4 Greg Yang, James B Simon, and Jeremy Bernstein. A spectral condition for feature learni...

Pith/arXiv arXiv 2022

[3] [2018]

Openmoe: An early effort on open mixture-of-experts language models.arXiv preprint arXiv:2402.01739, 2024

2 Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou, and Yang You. Openmoe: An early effort on open mixture-of-experts language models.arXiv preprint arXiv:2402.01739, 2024. 10 Cheng Yang, Shengnan Wang, Chao Yang, Yuechuan Li, Ru He, and Jingqiao Zhang. Progressively stacking 2.0: A multi-stage layerwise training method for bert ...

Pith/arXiv arXiv 2024

[4] [2020]

Sparse upcycling: Training mixture-of-experts from dense checkpoints

1 Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. Sparse upcycling: Training mixture-of-experts from dense checkpoints. InThe Eleventh International Conference on Learning Representations. 10 Guillaume Leclerc and Aleksander Madry. The two regimes of deep ...

Pith/arXiv arXiv 2002

[5] [2023]

Openwebtext corpus.http://Skylion007.github

5, 19 Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. Openwebtext corpus.http://Skylion007.github. io/OpenWebTextCorpus, 2019. 3 Linyuan Gong, Di He, Zhuohan Li, Tao Qin, Liwei Wang, and Tieyan Liu. Efficient training of bert by progressively stacking. InInternational conference on machine learning, pages 2337–2346. PMLR, 2019. 2, 4, 5, 9...

Pith/arXiv arXiv 2019

[6] [2024]

Llama pro: Progressive llama with block expansion.arXiv preprint arXiv:2401.02415, 2024

10 Chengyue Wu, Yukang Gan, Yixiao Ge, Zeyu Lu, Jiahao Wang, Ye Feng, Ying Shan, and Ping Luo. Llama pro: Progressive llama with block expansion.arXiv preprint arXiv:2401.02415, 2024. 4, 14 Chen Xing, Devansh Arpit, Christos Tsirigotis, and Yoshua Bengio. A walk with sgd.arXiv preprint arXiv:1802.08770,

Pith/arXiv arXiv 2024

[7] [2025]

Staged training for transformer language models

6 Sheng Shen, Pete Walsh, Kurt Keutzer, Jesse Dodge, Matthew Peters, and Iz Beltagy. Staged training for transformer language models. InInternational Conference on Machine Learning, pages 19893–19908. PMLR, 2022. 2, 4, 5, 8, 14, 19 Zhen Tan, Daize Dong, Xinyu Zhao, Jie Peng, Yu Cheng, and Tianlong Chen. Dlo: Dynamic layer operation for efficient vertical ...

Pith/arXiv arXiv 2022