{"id":"c081bac2-3992-4199-a8fc-ba86e43e052f","arxiv_id":"2608.13277","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Independently trained layer blocks, trained inside a frozen aligner scaffold, can be recomposed into a 1.3B-parameter language model that reaches monolithic perplexity only when the scaffold is reused or the token budget is larger.","lead":"A method called Mixture of Training splits a language model into layer blocks, trains each block independently inside a frozen pretrained scaffold, then stitches the blocks back together. On a 1.3B-parameter model it shows the pieces can form a working language model, but parity with the monolithic baseline only comes with more total tokens, plus more compute unless the scaffold is reused across runs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Quality-parity claim rests on a single unseeded schedule fitted to the baseline PPL, with no compute-matched monolithic control; the reported PPL match is not yet established.","rationale":"The reader's weakest_assumption focuses on whether the aligner's interface mechanism transfers beyond this 4-layer C4 instance. That is a valid generalization concern, but the paper's central quantitative claim—quality parity at PPL 15.0—is more immediately threatened by the absence of controlled comparison in the reported experiment. The aligner concern affects external validity; the missing controls and single seed affect internal validity of the headline number. Section 4 explicitly states that compute-matched monolithic controls are not included, and Appendix E states that main results are individual runs. The 30k adaptation pass is long enough to substantially retrain the composed model, so the attribution of the final PPL to the independently trained blocks is not isolated. A concrete test with seed replication and an equal-budget monolithic baseline would settle whether the parity claim is real or an artifact of schedule fitting and seed luck. This does not overturn the paper's narrower composability result—cold composition at 19.3 with the aligner versus 38.9 without it is a clear behavioral demonstration—but it should be part of the conditional acceptance.","tokens_in":10920,"tokens_out":10302,"duration_ms":107331,"concrete_test":"Run the MoT quality-parity schedule (75k submodel updates + 30k adaptation) and the monolithic 128k-update baseline with 5 independent seeds each, and also train a monolithic baseline for the same aggregate token budget (47.1B tokens) and the same fully charged EFLOP budget (about 285 EF). If the MoT PPL distribution's 95% CI excludes 15.0, or if the equal-budget monolithic PPL is more than 0.2 below the MoT mean, the headline equality claim fails; if the intervals overlap and the equal-budget baseline is no better, the parity claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central evidence for the claim that MoT 'reaches the same reported perplexity' as the monolithic baseline is Table 1's quality-parity row: PPL 15.0 at 285.0 EFLOPs and 47.1B tokens. This row is a single unseeded run (Appendix E: 'The main results are individual runs'), and the 75k+30k schedule was evidently selected so that the final PPL lands on the baseline value. There is no monolithic checkpoint trained or tuned at the 285 EF / 47.1B budget, so we cannot tell whether MoT matches, beats, or loses to monolithic training at equal resources; Section 4 explicitly concedes that compute-matched controls are absent. Moreover, the 30k adaptation pass is 23% of the baseline's training steps and trains all layers end-to-end, so it is unclear how much of the PPL 15.0 is attributable to the scaffolded Stage-1 blocks versus the adaptation pass itself. Without seed variance or an equal-budget baseline, the 'quality parity' headline is a fitted point estimate, not an established equivalence. The cold-composition result (19.3) does demonstrate that independently trained blocks form a usable model, so the core composability claim survives; but the stronger parity claim that motivates the amortized-reuse interpretation is not yet supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Mixture of Training (MoT), a scaffolded modular pretraining procedure. A target Transformer is partitioned into K contiguous layer blocks; each block f_i is trained independently (no gradient exchange) inside a frozen pretrained 'aligner' scaffold S_i that sandwiches the block with shape-compatible aligner slices, after which the blocks are recomposed into F and optionally given a short end-to-end adaptation pass. Experiments on a 12-layer, 1.3B-parameter Gemma-style model on C4 report: cold composition reaches PPL 19.3 versus 15.0 for the 128k-step monolithic baseline (268.4 EFLOPs); a 15k-step adaptation pass reaches PPL 15.9 at 189.4 fully-charged EFLOPs; and a 'quality-parity' schedule (75k scaffolded updates plus 30k adaptation, 47.1B aggregate tokens, 255.3 train EFLOPs / 285.0 fully charged) reaches PPL 15.0. Ablations show the aligner is essential (no-aligner cold PPL 38.9), disjoint submodel data streams help under the tested aligner (19.3 vs 20.3), and K=4 trades quality for compute. The paper provides explicit FLOP, token, memory, and idealized critical-path accounting, gives an amortization formula (effective cost 255.3 + 29.7/R per run), and explicitly positions the work as a small-scale proof of mechanism rather than a general replacement for monolithic pretraining.","tokens_in":11203,"tokens_out":19682,"duration_ms":200153,"significance":"The paper's strengths are real. I checked the arithmetic in Appendices B and D; the FLOP, token, and critical-path numbers are internally consistent (e.g., 47.1B tokens = 75k×2×256×1024 + 30k×256×1024; 128k/(75×0.61 + 30) ≈ 1.7×). The cold-composition result together with the no-aligner control (19.3 vs 38.9) is a clean behavioral demonstration that independently trained depth slices can be recomposed when trained inside a shared scaffold. The accounting is unusually transparent: Section 4 explicitly concedes the absence of compute-matched controls, measured wall-clock times, downstream evaluations, and direct interface diagnostics, and Appendix E discloses that main results are individual runs. If the composability result survives seed variance and transfers beyond this single 1.3B/C4 instance, MoT opens a genuinely useful design space of parallel, restartable, and potentially reusable scaffolded sub-runs.","major_comments":[{"comment":"The quality-parity claim (PPL 15.0 in Table 1) rests on a single unseeded run, as Appendix E states ('The main results are individual runs'), and the paper does not document how the 75k + 30k schedule was chosen. If the schedule was selected so that the final perplexity lands on the baseline value of 15.0, the parity result is a fitted operating point rather than an independent prediction, and with typical run-to-run variance the 15.0-vs-15.0 equality could be within noise. This matters because both the abstract's headline claim ('a quality-parity schedule reaches the same reported perplexity as the monolithic baseline') and the amortization break-even (R≥3) rest on this single point. Please report at least three seeds for the quality-parity and cold-composition schedules with mean and spread, or explicitly re-label the result as a single achieved operating point and document the schedule-selection procedure.","section":"§3 Table 1; Appendix E"},{"comment":"No equal-resource monolithic control is reported. The fully charged MoT parity schedule (285.0 EFLOPs, 47.1B tokens) is compared only against the 128k-step baseline (268.4 EFLOPs, 33.6B tokens); a monolithic run at the MoT FLOP budget (~36B tokens, ≈136k steps) or at the MoT token budget (~375 EFLOPs) would determine whether MoT matches, beats, or loses to monolithic training at equal resources, and neither is reported. Section 4 concedes this. The omission is load-bearing because the R≥3 amortization argument is a direct function of the parity schedule's 255.3/285.0 EFLOP costs, which are not yet established as necessary-or-sufficient for parity. In addition, the 30k Stage-2 pass (23% of baseline steps, all layers trainable) closes most of the cold-composition gap (19.3 to 15.0); without a control such as a 30k-step end-to-end run from random initialization, the fraction of the parity result attributable to the scaffolded Stage-1 blocks is not quantified. I recommend adding at least one monolithic checkpoint at the ~285-EF budget and a random-init-plus-30k-adaptation control, or removing the parity framing from the abstract.","section":"§3 Table 1; §4"}],"minor_comments":[{"comment":"Under the paper's stated 6ND accounting rule, the 4-layer aligner's 29.7 EFLOPs at a 10B-token budget implies an effective non-embedding parameter count of about 0.50B, whereas Table 5 lists 0.4B parameters; the 6ND value computed from the listed count is 24 EFLOPs, about 19% below the reported figure. A sentence reconciling this (e.g., naming any extra terms included in the aligner FLOPs) would keep the accounting fully transparent.","section":"Appendix B; Table 5"},{"comment":"The disclosure that 'the main results are individual runs' appears only in Appendix E; since Table 1 presents the headline comparisons without variance information, this caveat should also appear in the main text near Table 1.","section":"Appendix E; §3"},{"comment":"The disjoint-vs-shared data effect (20.3 vs 19.3 at K=2) and the K=2-vs-K=4 comparisons are single unreplicated runs with small effect sizes; a sentence in the main text stating that these are unreplicated observations would prevent readers from over-weighting differences of about one perplexity point.","section":"Table 6; §F.1"},{"comment":"In the definition S_i = a_K ∘ ... ∘ a_{i+1} ∘ f_i ∘ a_{i-1} ∘ ... ∘ a_1, the degenerate cases i=1 and i=K are not spelled out; a one-line clarification that empty aligner prefixes or suffixes are dropped would remove ambiguity.","section":"§2"},{"comment":"The abstract and Section 1 describe the aligner as supplying a 'stable representational interface' that makes blocks compatible, while Section 4 correctly states that only behavioral evidence exists and that 'a fuller mechanistic account would require direct interface diagnostics'; the earlier phrasing should be hedged to match that stated evidentiary level.","section":"§1; §4"}],"recommendation":"major_revision","confidential_remarks":"The referee's assessment is that the core composability claim survives the skeptic's critique, but the quality-parity headline does not: it is a single fitted schedule with no seeds and no equal-budget monolithic control, and the paper's own Section 4 concedes the missing controls. The authors' transparency is a genuine strength and suggests they may be receptive to adding seeds or softening the abstract. Please ensure the revision either reports seed variance and at least one equal-resource control, or explicitly reframes the parity result as a single achieved operating point; as written, the schedule-selection procedure is not documented, which makes the parity claim unfalsifiable from the manuscript alone. On scope: relative to Deep Incubation, the incremental contribution is a single-setting demonstration for autoregressive LMs with careful accounting; that is appropriate for a small-scale venue but should be reflected in how prominently the parity claim is advertised."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is an honest, well-scoped proof-of-mechanism paper, and the core composability result holds up. The quality-parity headline, though, is a fitted point estimate, not an established equivalence, and the compute story is more modest than the abstract sometimes suggests.\n\nWhat is actually new: the paper adapts Deep Incubation's scaffolded block-training pattern to autoregressive LM pretraining, with flexible aligner depth, disjoint submodel data streams, end-to-end adaptation, and explicit amortization accounting. The cold-composition ablation is the real result: with a frozen 4-layer aligner, independently trained 6-layer blocks recompose to PPL 19.3 at 157.9 EFLOPs, versus 38.9 without the aligner. That is a clean demonstration that the scaffold supplies a working representational interface, and it goes beyond what the prior vision-focused literature had shown. The related work is also positioned fairly, with Deep Incubation, model stitching, growth methods, and DiffusionBlocks all properly credited.\n\nThe soft spots are in the parity and efficiency claims. The quality-parity row in Table 1 (PPL 15.0 at 285.0 EFLOPs, 47.1B tokens) comes from a single unseeded run, and the 75k+30k schedule was evidently selected so the final PPL lands on the baseline value of 15.0. There is no monolithic checkpoint trained at the same 189.4 or 285.0 EFLOP budget, so we cannot tell whether MoT matches, beats, or loses to monolithic training at equal resources. The paper itself concedes this, and I appreciate that transparency, but the abstract still says MoT \"reaches the same reported perplexity\" without the caveat that this is a fitted target. The 30k adaptation pass is also 23% of the baseline's training steps and trains all layers end-to-end, so a non-trivial part of the parity is just ordinary continued training. The FLOP accounting excludes the output projection and embedding costs, which is material with a 256k vocabulary, and the critical-path numbers are idealized layer-equivalent estimates, not wall-clock measurements. The amortization arithmetic (R>=3 to beat baseline) is fine as arithmetic, but it is not evidence that reuse actually works across runs.\n\nThese concerns are real but proportionate. They do not damage the central finding that independent blocks trained inside an aligner scaffold can be recomposed into a usable model. What they damage is the stronger claim that MoT is an efficiency win. The paper already frames itself as a small-scale study of reusable sub-runs, and if the parity row is reworded as a fitted demonstration rather than an equivalence, the claims match the evidence.\n\nWho this is for: anyone working on modular or small-scale LM training, scheduling, or compute-amortization schemes. It deserves a serious referee. With seed variance, an equal-budget monolithic control, and a more careful parity wording, it could be a solid contribution.","headline":"A genuinely useful proof-of-mechanism paper that shows independently trained depth slices can be recomposed into a working language model; the quality-parity headline, however, is a fitted point estimate with no equal-budget baseline and no seed variance, so the efficiency claims should be read as provisional.","tokens_in":11724,"tokens_out":1854,"would_cite":true,"duration_ms":22719,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A frozen aligner scaffold lets independently trained Transformer depth blocks be recomposed into a working language model, and a quality-parity schedule matches the monolithic baseline's reported perplexity.","keywords":["mixture of training","scaffolded pretraining","modular language models","model recomposition","aligner scaffold","depth slicing","pretraining efficiency","language model pretraining"],"falsifier":"Train a 12-layer target without any aligner on a different tokenizer, model width, or data stream and check whether cold-composed perplexity stays near the baseline; the paper reports 38.9 under shared data, so a configuration where no-aligner cold composition matches baseline would refute the claim that the aligner is essential. Alternatively, measure CKA or SVCCA similarity between the frozen aligner's hidden states and the trained slices' hidden states at each recomposition boundary; if similarity is no higher than between random slices, the aligner is not acting as a stable interface.","tokens_in":10707,"feed_emoji":"🧩","tokens_out":6559,"duration_ms":55769,"temperature":0.7,"pith_summary":"The paper asks whether language-model pretraining has to be a single monolithic run, or whether it can be split into smaller, independently trainable jobs that are later joined into the full model. It introduces Mixture of Training (MoT), which trains each contiguous block of a target Transformer inside a frozen pretrained aligner scaffold, then discards the scaffold and recomposes the blocks. On a 12-layer, 1.3B-parameter Gemma-style model trained on C4, the paper reports that the recomposed model reaches the same validation perplexity (15.0) as the monolithic baseline when submodel training is extended to 75k updates and followed by a 30k-step adaptation pass. The result is presented as a small-scale proof of mechanism, not a general replacement for monolithic pretraining.","feed_headline":"Sliced pretraining reassembles into a working language model","feed_subtitle":"A frozen aligner lets independently trained depth slices match monolithic perplexity after a short adaptation.","key_machinery":"The load-bearing object is the frozen pretrained aligner $A = a_K \\circ \\cdots \\circ a_1$, a shape-compatible Transformer of the same width, attention-head dimensionality, feed-forward width, token-embedding space, and output head as the target. It is sliced into $K$ blocks, and each target block $f_i$ is trained inside $S_i$, with all aligner slices frozen, so every block learns against the same representational context but no gradients are exchanged between target blocks. The aligner's role is to provide a stable interface during independent training; after Stage 1 it is discarded and the recomposed blocks are optionally adapted end-to-end.","core_discovery":"The central claim is that a target Transformer $F = f_K \\circ \\cdots \\circ f_1$ can be trained as $K$ independent submodels, each optimized inside a frozen aligner scaffold $S_i = a_K \\circ \\cdots \\circ a_{i+1} \\circ f_i \\circ a_{i-1} \\circ \\cdots \\circ a_1$, and that the trained blocks can then be recomposed into a recomposed model $\\hat{F}$ that behaves like a coherent language model. The paper shows that a quality-parity schedule reaches perplexity 15.0, identical to the monolithic baseline, after extending Stage-1 training to 75k updates and adding a 30k-update adaptation pass, while processing 47.1B aggregate tokens and costing 285.0 EFLOPs when the aligner is fully charged to a single run. Without the aligner, cold-composition perplexity degrades from 19.3 to 38.9.","pith_inferences":["If the aligner interface transfers to other architectures, depths, and data streams, pretraining could become a library of reusable depth blocks whose interfaces are fixed by a shared scaffold, enabling mix-and-match assembly of new models without joint training.","The fault-tolerance benefit the paper sketches is untested: because each scaffold job is independently restartable, a hardware failure would in principle only redo one slice, but recovery cost depends on how much of the aligner and data pipeline must be re-run.","A direct test of interface stability—measuring hidden-state similarity (e.g., CKA) across recomposition boundaries before and after adaptation—would likely predict which block pairs compose well without needing a full training run.","The same scaffolded scheme could be combined with data specialization: training different slices on different corpora or objectives (e.g., code vs. natural language) might yield recomposed models with controllable skill mixtures, if the aligner keeps the interface stable."],"forward_implications":["Cold composition alone produces a usable model (PPL 19.3) at 157.9 fully charged EFLOPs, about 41% below the baseline budget, but with a clear quality gap.","A short 15k-step adaptation pass closes most of the gap, reaching PPL 15.9 at 189.4 fully charged EFLOPs.","The quality-parity schedule matches the baseline PPL of 15.0, but only when the one-time aligner cost is amortized over at least three independent target-model runs ($R \\ge 3$); fully charged to one run it costs 285.0 EFLOPs versus 268.4 for the baseline.","Disjoint data streams for the submodels improve cold-composition quality when the aligner is present (PPL 20.3 to 19.3), but worsen it without an aligner (PPL 38.9 to 50.4).","Increasing the number of splits from 2 to 4 cuts Stage-1 compute by roughly 37% but degrades cold-composition perplexity (PPL 19.3 to 24.8 under the 4-layer aligner)."],"supporting_citations":[{"why":"Supplies the divide-and-conquer scaffolded module-training pattern that MoT adapts to autoregressive language-model pretraining.","marker":"Ni et al., 2023"},{"why":"Defines the width configuration, vocabulary size, and post-normalization design used by both the target model and the aligner.","marker":"Gemma Team, 2024"},{"why":"Gives the $6ND$ FLOP approximation and the $M = D/N = 20$ rule used in the paper's compute accounting.","marker":"Hoffmann et al., 2022"},{"why":"Establishes the representation-compatibility problem that the aligner is meant to solve without post-hoc stitching layers.","marker":"Bansal et al., 2021"},{"why":"Provides functional-similarity diagnostics for comparing representations across networks, relevant to the proposed interface measurements.","marker":"Hernandez et al., 2023"},{"why":"A contemporaneous independent-block training method that MoT contrasts to justify keeping the next-token objective and the aligner scaffold.","marker":"Shing et al., 2026"},{"why":"Represents the progressive-growth baseline that MoT contrasts as sequential rather than parallel training of depth slices.","marker":"Chen et al., 2022"}],"fun_headline_variants":["Sliced pretraining reassembles into a coherent LM","Independent depth slices recompose to match monolithic perplexity","Block-wise training with a frozen aligner achieves perplexity parity","Modular pretraining: recomposed slices hit monolithic perplexity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The frozen scaffold must give each independently trained slice a shared way of representing its inputs and outputs; if that shared interface stops working outside this one 4-layer, C4 setup, the slices will not snap back together.","fun_headline_variants_meta":{"raw":{"variants":["Sliced pretraining reassembles into a coherent LM","Independent depth slices recompose to match monolithic perplexity","Block-wise training with a frozen aligner achieves perplexity parity","Modular pretraining: recomposed slices hit monolithic perplexity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000271,"raw_usage":{"total_tokens":1624,"prompt_tokens":938,"completion_tokens":686,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":554,"completion_tokens_details":{"reasoning_tokens":618}},"tokens_in":554,"tokens_out":686,"duration_ms":7487,"temperature":1.0,"reasoning_tokens":618,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:45:38.033001+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a 12-layer target without any aligner on a different tokenizer, model width, or data stream and check whether cold-composed perplexity stays near the baseline; the paper reports 38.9 under shared data, so a configuration where no-aligner cold composition matches baseline would refute the claim that the aligner is essential. Alternatively, measure CKA or SVCCA similarity between the frozen aligner's hidden states and the trained slices' hidden states at each recomposition boundary; if similarity is no higher than between random slices, the aligner is not acting as a stable interface.","supporting_citations":[],"review_version":1}