{"id":"6f1d164f-a61d-4de2-9d08-2138087fa1c5","arxiv_id":"2509.06518","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"At 180M parameters and 5B tokens, all layer-wise scaling variants beat the paper's 18-layer uniform baseline, yet the 12-layer uniform baseline remains best.","lead":"This paper tests what happens when transformer layers get different sizes instead of all being the same. The exact size profile matters little, but the authors' own 12-layer uniform model beats all their new 18-layer designs.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Depth confound in baseline selection undermines 'heterogeneity helps' claim: the 12L isotropic baseline (PPL 4.962) beats every 18L LWS variant, so the 18L uniform baseline may be an unfair comparator.","rationale":"I read the paper as an honest, code-released ablation study of layer-wise scaling profiles at a fixed parameter budget. The experiments are clearly described, and the limitations section candidly notes single-run training and small scale. However, the strongest claim—that heterogeneous layer sizes improve over an isotropic baseline—rests on the 18-layer uniform model with FFN scale 2.5 and QKV scale 0.75 being a representative isotropic architecture at 180M parameters. The paper's own Table 3 undermines this: the original 12-layer isotropic baseline achieves PPL 4.962, better than every 18L LWS variant. The authors moved to 18 layers to smooth LWS interpolation, but in doing so they simultaneously changed depth and width of the baseline, creating a confound. Since the 12L vanilla LWS does not beat the 12L baseline, the only evidence for 'heterogeneity helps' comes from the poorly-performing 18L baseline. The reader's weakest assumption identifies exactly this issue, and I agree. A practical check—sweeping the 18L uniform baseline or comparing against the 12L baseline—would settle whether the heterogeneity claim is robust. Given the paper's honest reporting and the plausibility that a tuned 18L baseline could narrow or eliminate the gap, the appropriate verdict remains CONDITIONAL, as the reader recommended; no further adjustment is needed beyond their stated conditions.","tokens_in":11399,"tokens_out":6524,"duration_ms":69810,"concrete_test":"Run a small grid over uniform 18-layer configurations at ~180M params (e.g., FFN scale in {2.0, 2.5, 3.0, 3.5} and QKV scale in {0.5, 0.75, 1.0}) with the same 5B-token budget, and take the best validation PPL. Then compare it against the LWS variants in Table 3. Additionally, re-run the comparison using the already-reported 12-layer isotropic baseline (PPL 4.962) as the reference. If the best tuned 18-layer isotropic model or the 12-layer baseline matches or beats all LWS variants, the conclusion that heterogeneity helps is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that 'all LWS variants improve in validation perplexity over an isotropic baseline' is confounded by the choice of baseline. In Table 3, the original 12-layer isotropic model (with GQA) achieves PPL 4.962, which is better than all 18-layer LWS variants (Crown 5.057, Reverse 5.087, Vanilla 5.093, Framed 5.205). The authors switched from 12 to 18 layers mid-study (Section 4) and rebuilt the isotropic baseline as an untuned deep-narrow config (FFN scale 2.5, QKV scale 0.75), which performs markedly worse (PPL 5.400). Thus the observed 'heterogeneity helps' effect may be an artifact of degrading the isotropic baseline by making it deeper and narrower without tuning, rather than evidence that non-uniform allocation at fixed parameter budget is beneficial. Moreover, 12-layer vanilla LWS does not beat the 12-layer baseline (4.984 vs 4.962), so the depth change, not heterogeneity, is the variable associated with the outcome. The conclusion should be scoped to the 18-layer setting and requires a stronger isotropic baseline.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether non-uniform (layer-wise) parameter allocation can improve language model pre-training at a fixed parameter budget. Building on OpenELM's Layer-Wise Scaling (LWS), it introduces three variants -- Framed, Reverse, and Crown -- that redistribute FFN widths and attention-head counts via linear interpolation profiles, and compares them against an isotropic baseline. Experiments use a 180M-parameter model trained on 5B tokens, with validation perplexity as the main metric. The authors report that all 18-layer LWS variants outperform the 18-layer isotropic baseline, that the exact profile matters little, and that LWS alone does not reproduce OpenELM's claimed data-efficiency gains. The central evidence is Table 3 and Figures 4-5.","tokens_in":11788,"tokens_out":7044,"duration_ms":82586,"significance":"If the main claim were robust, this would be a useful contribution: it is the first controlled isolation of LWS from the other changes in OpenELM, and it suggests a cheap architectural improvement at fixed parameter count. The paper has genuine strengths: a fixed 180M parameter budget with parameter accounting in Table 2, a baseline repeated five times, explicit acknowledgment of limitations, and open-source code. However, the current evidence does not establish the headline claim because the isotropic comparator is confounded with depth and tuning. The central 'heterogeneity helps' conclusion should therefore be treated as conditional pending a fairer baseline and multi-seed evaluation.","major_comments":[{"comment":"The paper's central comparison is confounded by depth and baseline tuning. The original 12-layer isotropic baseline (PPL 4.962) beats every 18-layer LWS variant (Crown 5.057, Reverse 5.087, Vanilla 5.093, Framed 5.205), and 12-layer vanilla LWS (4.984) does not beat the 12-layer baseline. The 18-layer baseline was introduced only after the authors saw that 12-layer LWS did not help, and it is configured as a deeper, narrower uniform model (FFN 2.5, QKV 0.75) without reported tuning; its PPL is 5.400. The conclusion that 'heterogeneity helps' therefore rests on an isotropic comparator that under-performs the original 12-layer model. The authors should either tune the 18-layer baseline, include the 12-layer comparison in the central claim, or explicitly restrict the conclusion to the 18-layer deep-narrow regime.","section":"Section 4 / Table 3"},{"comment":"The claim that 'the exact shape matters little' is not supported by the reported statistics. Only the 18-layer baseline was run five times (std 0.0595 on final PPL, Table 3); each LWS variant was run once. Framed LWS (PPL 5.205) differs from Crown LWS (5.057) by about 2.5 baseline standard deviations, and Figure 5 shows a late-training PPL bump for Framed. Without per-variant error bars or a formal significance test, the apparent convergence among variants -- and some of the improvement over baseline -- could be run-to-run variation. The authors acknowledge the single-run limitation in Section 5.1, but this is precisely the point on which the 'shape does not matter' conclusion depends.","section":"Section 5 / Section 5.1"}],"minor_comments":[{"comment":"'Training steps 1 3000' appears to be a typo; the table should also state explicitly that it is for the 12-layer baseline, since later models use 18 layers.","section":"Table 1"},{"comment":"The index range '0 ≤ i ≤ N' should likely be '0 ≤ i < N' or the interpolation should be defined consistently for N layers; there is also a typo in Eq. (1) ('di ffn').","section":"Section 3.2, Eqs. (1)-(4)"},{"comment":"'Baseline 12L has 180M tokens instead of the 190M from olmo2-190M configuration' should read 'parameters', not 'tokens'.","section":"Section 4"},{"comment":"The heading 'Weakneseess' is misspelled. More substantively, the text acknowledges single-run training but frames it as future work rather than qualifying the abstract's and conclusion's definitive claims.","section":"Section 5.1"},{"comment":"Figures 4 and 5 would be clearer with shaded error bars or confidence bands for the baseline; the zoomed plots should state the exact x-axis range. The DOLMino citation is malformed and should be cleaned up.","section":"Figures 4-5 and references"}],"recommendation":"major_revision","confidential_remarks":"The depth/baseline confound is the decisive issue. If the authors add a properly tuned 18-layer isotropic baseline and at least a few seeds for the LWS variants, the paper could become sound. The current version over-states 'heterogeneity helps' and 'exact shape does not matter.' The paper is a small-scale empirical study; its fit depends on whether the venue values qualified negative/limited results about LWS. No concerns about novelty disclosure."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper does something useful: it runs the first fixed-budget ablation of layer-wise scaling profiles (Framed, Reverse, Crown) and isolates LWS from the rest of OpenELM's recipe. The negative result — that LWS by itself does not reproduce OpenELM's data-efficiency gains — is an honest, valuable finding, especially given the field's tendency to over-attribute gains to architectural tweaks. The authors also deserve credit for open-sourcing code and for a limitations section that is unusually candid.\n\nThe problem is the central claim. The paper concludes that 'all LWS variants improve in validation perplexity over an isotropic baseline' and that 'the presence of heterogeneity, not its exact shape, matters.' But the baseline used for that comparison is not fair. The authors switched from 12 to 18 layers mid-study after seeing that 12-layer vanilla LWS did not beat the 12-layer uniform model, then built an 18-layer uniform baseline with untuned FFN/QKV scalars (2.5/0.75). That baseline lands at PPL 5.400. The original 12-layer isotropic baseline gets 4.962 — better than every 18-layer LWS variant (which range from 5.057 to 5.205). So the headline comparison confounds depth with heterogeneity: the apparent win for LWS may simply reflect a poorly chosen deep-narrow uniform baseline.\n\nThere is also a statistics issue. The LWS variants are single runs; only the 18-layer baseline has repeated runs (five). That means differences among variants — and even the gap between LWS and baseline — cannot be assessed for significance. The authors acknowledge this, but it undermines the confident claim that the exact profile 'matters little.'\n\nThe paper is otherwise coherent. The scaling formulas are clear, parameter counts are close to the stated 180M, and the discussion of why perplexity is preferable to benchmarks at this scale is reasonable. Some citation formatting is sloppy but that's minor.\n\nNet: this is a serious, honest small-scale study with a load-bearing flaw in its baseline selection. It deserves peer review because the empirical question is important and the negative result on LWS is worth reporting. But it needs revision: scope the claims to the 18-layer setting, test a properly tuned 18-layer uniform baseline (or compare against the 12-layer baseline explicitly), and add repeated runs for at least the main variants. If those are fixed, this would be a useful contribution to small-model architecture design. I'd bring it to a reading group and would cite the negative result with caveats.","headline":"An honest small-scale ablation of LWS variants whose 'heterogeneity helps' conclusion is undercut by a depth-confounded baseline, though the negative result on LWS alone is a useful data point.","tokens_in":12214,"tokens_out":2253,"would_cite":true,"duration_ms":27863,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that any non-uniform distribution of layer sizes beats an equal-sized uniform model at equal parameter count, and that the exact shape of the distribution barely matters.","keywords":["Layer-Wise Scaling","heterogeneous architectures","LLM pre-training","validation perplexity","parameter allocation","grouped-query attention","layer importance","pruning-inspired design"],"falsifier":"Train the same 18-layer uniform model with the same per-layer compute but tuned width scalars (e.g., matching the average FLOPs of the LWS variants) and show that a well-tuned uniform model closes the perplexity gap; or scale the comparison to a 7B-parameter model trained on >=100B tokens and check whether the perplexity advantage of the crown, reverse, and vanilla profiles persists or reverses.","tokens_in":11343,"feed_emoji":"🧠","tokens_out":5302,"duration_ms":55905,"temperature":0.7,"pith_summary":"The paper isolates Layer-Wise Scaling (LWS) from other architectural changes and tests four ways to redistribute feed-forward widths and attention heads across layers at a fixed 180M-parameter budget and 5B training tokens. All four non-uniform profiles — linearly increasing, framed, reverse, and crown-shaped — reach validation perplexities meaningfully below an 18-layer isotropic baseline, with gains around 5–6%. The authors conclude that heterogeneity itself, not the specific profile, drives the improvement, and that LWS reallocates rather than creates representational capacity. They also find that LWS alone does not deliver the roughly 2x data-efficiency gain previously attributed to it, suggesting that earlier claims depend on other parts of the training recipe.","feed_headline":"Uneven layer sizes beat uniform ones in small LLM pre-training","feed_subtitle":"Every non-uniform profile tested — growing, framed, reverse, crown — beats the uniform baseline by about 5–6% at 180M parameters.","key_machinery":"The machinery is a per-layer linear interpolation of two architectural widths: the feed-forward expansion factor beta_i (FFN hidden size) and the query-head count alpha_i, with key/value heads fixed by grouped-query attention. Profiles are defined by the endpoints (and, for Crown, a middle point) of the interpolation, and all variants are built to the same total parameter budget. This device converts a qualitative hypothesis about differential layer importance into concrete architectures that can be trained and compared, and it is what allows the paper to attribute the observed perplexity gap to heterogeneity per se.","core_discovery":"On the paper's own terms, the central discovery is that pre-training a Transformer with non-uniform per-layer parameter allocation reliably improves validation perplexity over an equal-cost isotropic baseline, while the choice among heterogeneous profiles is a second-order concern. The authors compare an 18-layer uniform model with four LWS variants — Vanilla (linearly growing), Framed (maximum first and last layers), Reverse (largest early layers, framed), and Crown (peak in the middle, framed) — all at around 180M parameters trained on 5B tokens with grouped-query attention applied throughout. Every variant beats the uniform baseline, and the three best variants cluster within 0.04 perplex","pith_inferences":["If the heterogeneity effect survives scaling, it would let model builders trade a uniform architecture for one matched to the layer-importance profile discovered by pruning studies, potentially getting the same perplexity at fewer parameters or tokens.","The near-equivalence of all non-uniform profiles suggests a saturation or regularity phenomenon: once the network is released from exact width uniformity, the optimizer adapts to a variety of shapes. A testable extension would train the same profiles at 7B parameters and >=100B tokens to see whether the gap to isotropic models widens or collapses.","The paper's own perplexity values are unusually low (around 5) relative to typical 10–15 for larger models; if that reflects an artifact of the validation set, the relative ranking of variants could still hold while the absolute gains might not transfer."],"forward_implications":["Architectures that vary layer sizes at fixed parameter count can improve validation perplexity by roughly 5–6% over a uniform model of the same size, without slowing training.","The shape of the layer-wise profile is a second-order choice: Vanilla, Reverse, and Crown cluster together, so practitioners may pick a profile for hardware or pipeline convenience.","LWS alone does not reproduce the ~2x data-efficiency advantage claimed by OpenELM; that advantage likely arises from the interaction of LWS with other recipe components.","Because the gains are reallocative rather than additive, the practical value is in matching compute, not in increasing capacity."],"supporting_citations":[{"why":"Supplies the Layer-Wise Scaling method, the vanilla increasing profile, and the data-efficiency claim the paper re-tests.","marker":"(Mehta et al., 2024)"},{"why":"Provides the similarity-score analysis across layers that inspires the Crown and Reverse profiles.","marker":"(He et al., 2024)"},{"why":"Gives the theoretical argument that pruning early layers causes compounding reconstruction error, motivating the Reverse variant.","marker":"(Huang et al., 2025)"},{"why":"Introduces layer-wise adaptive pruning whose profile (keeping first/last layers) motivates the Framed and Reverse designs.","marker":"(Pan et al., 2025)"},{"why":"Reports that middle layers contribute most to performance, directly inspiring the Crown profile.","marker":"(Askari et al., 2025)"},{"why":"Provides the base model architecture and training hyperparameters used for all experiments.","marker":"(OLMo et al., 2025)"},{"why":"Guides the choice of batch size and learning rate and documents that small-scale evaluation via benchmarks is unreliable, justifying perplexity as the metric.","marker":"(Biderman et al., 2023)"},{"why":"Defines Grouped Query Attention, which the paper applies uniformly across all models.","marker":"(Ainslie et al., 2023)"}],"fun_headline_variants":["LLM pre-training: non-uniform layer sizes beat uniform baseline","All tested layer-shape variants outperform uniform LLM pre-training","Uneven LLM layers: every variant wins over uniform at 180M scale","At 180M params, layer shape matters: all non-uniform profiles win"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The 18-layer uniform baseline, with its particular width scalars (FFN scale 2.5, QKV scale 0.75), fairly represents isotropic architectures at the 180M budget; if a better-tuned uniform model were used, the all-variants-beat-baseline result could weaken.","fun_headline_variants_meta":{"raw":{"variants":["LLM pre-training: non-uniform layer sizes beat uniform baseline","All tested layer-shape variants outperform uniform LLM pre-training","Uneven LLM layers: every variant wins over uniform at 180M scale","At 180M params, layer shape matters: all non-uniform profiles win"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000199,"raw_usage":{"total_tokens":1179,"prompt_tokens":689,"completion_tokens":490,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":433,"completion_tokens_details":{"reasoning_tokens":421}},"tokens_in":433,"tokens_out":490,"duration_ms":6051,"temperature":1.0,"reasoning_tokens":421,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T23:28:58.763888+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same 18-layer uniform model with the same per-layer compute but tuned width scalars (e.g., matching the average FLOPs of the LWS variants) and show that a well-tuned uniform model closes the perplexity gap; or scale the comparison to a 7B-parameter model trained on >=100B tokens and check whether the perplexity advantage of the crown, reverse, and vanilla profiles persists or reverses.","supporting_citations":[{"cited_title":"Determining Layer-wise Sparsity for Large Language Models Through a Theoretical Perspective","cited_arxiv_id":"2502.14770","evidence_quote":"Gives the theoretical argument that pruning early layers causes compounding reconstruction error, motivating the Reverse variant."},{"cited_title":"Layerif: Estimating layer quality for large language models using influence functions, 2025","cited_arxiv_id":null,"evidence_quote":"Reports that middle layers contribute most to performance, directly inspiring the Crown profile."}],"review_version":1}