{"id":"fa3e4998-f3b5-4b4a-b295-d98f04add425","arxiv_id":"2511.04981","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Training GPT2 from a zero/one-layer model and expanding depth at 80% of the schedule reaches fixed-size loss with approximately 5x less compute.","lead":"This paper proposes training a language model starting from just an embedding layer (zero layers) or a single layer, then expanding it to the full deep network near the end of training. The authors report roughly five times less compute to reach a similar final loss on GPT2-scale models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Mixing-time transfer ignores the 7B run's 8x smaller batch: post-expansion budget is ~8B tokens, below the paper's own ~16B-token tmix, so the headline 5x speedup is not established.","rationale":"The reader's weakest assumption correctly identifies the transferability of tmix as load-bearing. I sharpen it: the transfer is not just across model size but across an 8x batch-size reduction. Section C.6's 'mixing needs data, not iterations' makes this a quantitative contradiction, not merely a robustness worry. The paper's own FLOPs axis in Figure 1 is consistent with a global batch of 64 sequences for the 7B run (total tokens ~39B), so the post-expansion budget is ~7.9B tokens. This is the single most concrete threat to the central claim: either the empirical 7B result is mis-calibrated, or the transfer recipe is internally inconsistent. I also agree with the reader that the abstract's LLaMA3/DeepSeekV3 scaling-law claim is absent from the provided body. I would keep the verdict conditional rather than reject outright because the raw empirical claim could still be true if the 7B mixing time is much shorter; the proposed test would settle this. No code or error bars are provided, so an independent reproduction of the 7B run would also be valuable.","tokens_in":18875,"tokens_out":15298,"duration_ms":129941,"concrete_test":"Run the small-scale 1-layer-to-12-layer calibration at batch size 64 (65,536 tokens/iteration, matching the 7B run) and measure tmix in tokens. If tmix remains ~16B tokens, then the 7B run's post-expansion budget of ~7.9B tokens is insufficient and the claimed <0.2% loss match at tau=0.8T is not reproducible; if tmix drops below ~7.9B tokens at this batch size, the transfer may hold. Alternatively, reproduce the Figure 1 7B run and extend post-expansion training until >=16B tokens; if the loss only matches fixed-size after ~244k post-expansion iterations (rather than 120k), the advertised 5x speedup is unsupported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The recipe's calibration step (Section 5.2, Takeaway 6) transfers a mixing time measured in small early-stopped runs to the full-scale 7B run. Section 5.2 states the 1-layer-to-12-layer mixing time is ~16B tokens (~30k iterations at the default 512-sequence batch), and Section C.6 explicitly says mixing time is measured in tokens, not iterations. However, Section B reports that the long Figure 1 7B runs use batch size 64 (sequences) rather than 512, i.e. 65,536 tokens/iteration instead of 524,288. With expansion at tau=0.8T (t=480k of 600k), the post-expansion budget is only 120k x 65,536 ~ 7.9B tokens — about half the 16B-token mixing threshold. If the mixing-time transfer is taken literally, the final progressive 7B run cannot have mixed with the fixed-size run by the end of training, so the claimed <0.2% final-loss difference and ~5x speedup are not explained by the paper's own mechanism. If instead the run did mix within 120k iterations, then tmix is not transferable in tokens across the batch-size change, contradicting Section C.6 and invalidating Section 7 step 4 as stated. The missing LLaMA3/DeepSeekV3 scaling-law evidence in the abstract is a separate support gap; the batch-size inconsistency is more directly load-bearing because it targets the headline GPT2 result.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes zero/one-layer progressive training for depth expansion of residual networks, claiming that training a 0- or 1-layer model for most of the run and then expanding depth at ~80% of the horizon yields ~5x compute savings with negligible loss degradation. The method is evaluated on GPT2 (up to 7B parameters), MoE, and ResNet, and is supported by a convergence analysis for convex Lipschitz losses plus ablations of initialization, learning-rate schedule, and expansion timing. The central empirical claims are that mixing time is transferable and that WSD schedules enable late expansion.","tokens_in":19285,"tokens_out":4729,"duration_ms":44343,"significance":"If the reported speedups hold, the paper would be a substantial contribution to efficient LLM pre-training: zero/one-layer expansion is simple, architecture-agnostic, and avoids ordering/initialization complexities of multi-layer growth. The strength is the breadth of ablations (150 runs in Figure 8) and the concrete, falsifiable recipe in Section 7, including the mixing-time-transfer rule. However, the headline claims rest on a single 7B run without error bars, the abstract advertises LLaMA3/DeepSeekV3 results that are absent from the body, and the theoretical bound relies on an unproved structural assumption about the global optimum. These need to be fixed before the contribution is fully credible.","major_comments":[{"comment":"The claim that the 7B progressive run mixes is inconsistent with the paper's own mixing-time calibration. Section B states the 7B runs in Figure 1 use batch size 64 (65,536 tokens/iter). At τ=480k of T=600k, the post-expansion budget is ~7.9B tokens. Section 5.2 reports tmix≈16B tokens for 1→12-layer expansion. Thus, under the paper's own token-based transfer, the 7B run cannot have mixed by the end, so the <0.2% final-loss gap is not explained by the mechanism. If the run did mix, then tmix is not transferable in tokens across batch size, contradicting Section C.6 and Figure 18. Please provide direct evidence for mixing in this setting or revise the speedup claim.","section":"Section B / Section 5.2 / Figure 1"},{"comment":"The abstract states that 'scaling laws on LLAMA3 and DeepSeekV3 models show a 3~5x improvement in compute efficiency'. The body contains no LLaMA-3 or DeepSeek-V3 experiments; the only mention is 'LLAMA-4' in the introduction. This unsupported claim must be removed or substantiated with actual scaling-law experiments.","section":"Abstract / body"},{"comment":"The derivation after Eq. (4.4) assumes W*=[w*,x*], i.e., the global minimizer of the large model coincides with the small-model minimizer in the first coordinates. This is not justified for residual networks: the optimal first-layer weights of a deep network need not be the optimum of a 0/1-layer network. Without a proof or an explicit condition, the bound in Eq. (4.5) and the 'PGD + initialization' interpretation do not follow from the preceding inequalities.","section":"Section 4, Eq. (4.5)"},{"comment":"The headline results in Figure 1 are single runs with no error bars or repeated seeds. Since the claimed advantage is a <0.2% validation-loss difference, it is unclear whether this is within run-to-run variation. Please provide repeated runs or confidence intervals for the main comparisons, especially the 7B case.","section":"Section 2 / Figure 1"}],"minor_comments":[{"comment":"Typo: 'progrssive' should be 'progressive'.","section":"Section 1.1"},{"comment":"The sentence 'expanding at 80% horizon cannot mix well' refers to Figure 5, where the WSD decay is 20% (per Section B), whereas Figure 1 uses 10% decay and τ=0.8T. Please make the schedule dependence explicit so the apparent contradiction is resolved.","section":"Section 5.2"},{"comment":"The notation W_t=[w_t,x_t] is used before being formally introduced. Define this decomposition explicitly before Eq. (4.5).","section":"Section 4"},{"comment":"The caption says 'zero-layer (red, 39M or 0.15B)'; the units are ambiguous. Clarify whether 39M is the 0/1-layer model size and 0.15B is the 12-layer target, or provide a consistent notation.","section":"Figure 1 caption"}],"recommendation":"major_revision","confidential_remarks":"The batch-size inconsistency in the 7B runs is the most serious issue: it directly undermines the central speedup claim. The LLaMA3/DeepSeekV3 statement in the abstract is an unsupported overclaim that should be removed. The theoretical assumption in Eq. (4.5) needs either a proof or an explicit caveat. If these are addressed, the paper could become publishable, but the current version requires substantial revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is a real and useful idea: prior progressive training expands from 2+ layer sources; starting from zero/one layer and using WSD to time a single expansion is new, and the ablations (150 GPT2 runs, ResNet, MoE) make the core recipe credible. Second, the paper's headline claim — 5x speedup on 7B GPT2 — is not actually supported by its own mechanism. The abstract promises LLaMA3/DeepSeekV3 scaling-law evidence that never appears in the body, and the 7B experiments have a batch-size inconsistency that breaks the mixing-time transfer argument.\n\nThe stress-test concern holds up on reading. The mixing time is calibrated in Figure 5 using batch 512, giving ~16B tokens (30k iterations). Section C.6 says mixing time is measured in tokens, not iterations, and is largely batch-size agnostic. But the 7B runs in Figure 1 use batch 64, so the post-expansion budget of 120k iterations is ~8B tokens — half the calibrated tmix. So either the run didn't actually mix, contradicting the claimed <0.2% loss difference, or tmix is not transferable in tokens across batch sizes, contradicting C.6. The paper can't have it both ways. This is the load-bearing soft spot.\n\nOther issues are smaller but still real: single seeds, no error bars, no code, and the convergence theory (Section 4) assumes W*=[w*,x*] without justification. These are fixable. The GPT2 124M results with batch 512 are self-consistent and the zero/one-layer advantage over 2/6-layer sources is shown convincingly in Figure 8.\n\nWho this is for: people working on efficient LLM pretraining, progressive training, or WSD schedules. It deserves a serious referee: the recipe is simple, the experiments are broad, and the central claim is important if it holds. A referee should ask for the LLaMA/DeepSeek experiments, error bars, code, and a resolution of the batch-size discrepancy. My own verdict: the novelty is real, the evidence is incomplete, and the 7B speedup should be treated as unverified until the mixing-time accounting is fixed.","headline":"Reports a genuinely new zero/one-layer progressive-training recipe with a plausible but unproven 5x speedup: an internal batch-size inconsistency undercuts the mixing-time story, and the abstract overclaims LLaMA3/DeepSeekV3 evidence.","tokens_in":19681,"tokens_out":3663,"would_cite":true,"duration_ms":32386,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training a zero- or one-layer model for most of the run, then expanding its depth, matches full-depth loss at roughly one-fifth of the compute.","keywords":["progressive training","model expansion","depth scaling","zero-layer initialization","mixing time","WSD learning rate schedule","feature learning","LLM pretraining"],"falsifier":"Run a full-scale GPT2-class or larger training with zero/one-layer progressive training at τ = 0.8T under WSD, and check whether the grown model's validation loss converges to within about 0.2% of a fixed-size run within the calibrated mixing-time budget; a final gap larger than 0.2%, or a mixing time that grows noticeably with model size, would falsify the transfer claim.","tokens_in":18800,"feed_emoji":"⚡","tokens_out":6938,"duration_ms":71414,"temperature":0.7,"pith_summary":"This paper argues that the cheapest possible start to training a deep network — a model with zero or one transformer block — can be grown into a full-depth model with almost no loss in final quality, provided the expansion happens late and under the right learning-rate schedule. On GPT2-class models it reports an 80% compute saving, about a 5x speedup, with final validation loss within 0.5% for a 124M model and within 0.2% for a 60-layer 7B model. The supporting theory views progressive training as projected gradient descent plus a good initialization, and the key empirical phenomenon is 'mixing': after expansion, the grown model's loss curve quickly meets the fixed-size training curve and tracks it. The recipe prescribes training a zero/one-layer model under a warmup-stable-decay schedule, expanding depth at roughly 80% of the run, and calibrating the expansion time with two small early-stopped runs. The abstract also states scaling-law gains on LLaMA3 and DeepSeekV3; the body's detailed experiments cover GPT2, ResNet, and MoE.","feed_headline":"Starting from zero layers saves 80% of GPT2 training compute","feed_subtitle":"Expand depth at 80% of the run and the grown model matches full-depth validation loss within 0.2%.","key_machinery":"The central mechanism is the depth expansion itself, analyzed as projected gradient descent: before expansion, training is interpreted as optimizing the deep model with the not-yet-added layers held at zero; at time τ those layers are 'teleported' to a good initialization (random or copied), and ordinary SGD continues. The carrying identity is the convergence-bound gap between progressive and fixed-size training, which splits into a weighted sum of the small-model versus large-model optimum gap plus an initialization-quality term. The empirical quantity that sets the schedule is the mixing time — the amount of data (tokens or images) after expansion until the grown model's loss meets the fix","core_discovery":"The paper's claim is that zero/one-layer progressive training can replace most of a fixed-size deep-model run with a tiny model. Starting from a zero-layer (residual stream only) or one-layer model, training proceeds under a warmup-stable-decay schedule, and at about 80% of the total horizon the remaining depth is inserted with random initialization (or by copying the one layer). After a transient loss spike, the grown model's loss 'mixes' with the fixed-size-from-scratch curve and remains essentially matched to the end: within 0.5% validation loss at 124M parameters and within 0.2% at 7B/60 layers. Because the source model costs about 2% of the target's FLOPs and expansion is late, total co","pith_inferences":["A direct, cheap test of the recipe's universality would be to measure mixing time in tokens across several data distributions, batch sizes, and model widths; if it stays stable, the two-run calibration procedure could be treated as a general scheduling law for depth growth.","If the result extends to frontier-scale training, it suggests depth can be treated as a late-stage resource, which would change how compute is allocated across data quality, context length, and multimodal stages.","The paper's 'PGD + initialization + SGD' framing implies that width expansion or expert-count expansion might exhibit similar mixing behavior; checking that would show whether the phenomenon is about capacity growth generally rather than depth specifically.","The abstract's scaling-law claim on LLaMA3 and DeepSeekV3 is not backed by experimental detail in the supplied body, so treating it as established would require the full-scale runs or scaling-law fits to be examined separately."],"forward_implications":["If the recipe holds at scale, a multilingual or multimodal model could spend roughly 80% of its training budget as a tiny skeleton model, with full depth inserted only near the end, without sacrificing final loss.","Mixing time is measured in data seen, not iterations, so practitioners can raise the batch size after expansion without degrading the loss-compute tradeoff.","Single-stage expansion from zero/one layer dominates multi-stage stacking, because each stage's mixing can be decomposed and multi-stage costs more for the same final loss.","Hyperparameters can be reused unchanged across expansion when new layers are randomly initialized or copied, since these satisfy the feature-learning scaling condition; zero initialization breaks it and should be avoided outside residual paths.","The phenomenon appears across dense GPT2, ResNet, and MoE models, suggesting it is a property of depth growth rather than of one architecture."],"fun_headline_variants":["Zero-layer start cuts GPT2 training compute by 80%","Grow depth late: 5x faster training, same loss","Start small, expand depth: 3-5x efficiency at scale","Zero/one-layer progressive training matches full-depth loss"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The recipe transfers the mixing time measured in two small early-stopped runs to the full-scale run; if the mixing time is not robust across model size, batch size, or data distribution, the chosen expansion time τ ≈ 0.8T could be mistimed and the advertised 5x speedup would not materialize.","fun_headline_variants_meta":{"raw":{"variants":["Zero-layer start cuts GPT2 training compute by 80%","Grow depth late: 5x faster training, same loss","Start small, expand depth: 3-5x efficiency at scale","Zero/one-layer progressive training matches full-depth loss"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000152,"raw_usage":{"total_tokens":1047,"prompt_tokens":756,"completion_tokens":291,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":500,"completion_tokens_details":{"reasoning_tokens":231}},"tokens_in":500,"tokens_out":291,"duration_ms":3192,"temperature":1.0,"reasoning_tokens":231,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T23:33:05.768049+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a full-scale GPT2-class or larger training with zero/one-layer progressive training at τ = 0.8T under WSD, and check whether the grown model's validation loss converges to within about 0.2% of a fixed-size run within the calibrated mixing-time budget; a final gap larger than 0.2%, or a mixing time that grows noticeably with model size, would falsify the transfer claim.","supporting_citations":[],"review_version":1}