REVIEW 4 major objections 4 minor 5 references
Progressive Depth Up-scaling via Optimal Transport
T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read The paper argues that aligning and fusing adjacent Transformer blocks with optimal transport before averaging their weights yields better-initialized new layers, making depth up-scaling both more accurate and cheaper, and that top-half inse
desk verdict Clean application of OT to depth up-scaling, but the key ablation is confounded by zero-initialization, so the OT claim is unproven. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Transport Matrix Flow (TMF) with two transport matrices: $T^{\text{in}}$, propagated within a layer to keep block-to-block correspondence, and $T^{\text{out}}$, computed by optimal transport via Sinkhorn-Knopp to align each block's neurons with the corresponding block of the next base layer. The work of TMF is to define which weight matrices need which alignment so that averaging two adjacent layers yields a coherent new layer rather than a scrambled mixture.
What would settle it
Measure, on a fixed corpus, the activation correlation of neurons paired by OpT-DeUS's transport matrix against same-index neurons in unaligned layers; if OT-paired neurons are not more correlated, the alignment mechanism is not doing the claimed work.
Extended reading notes
Core claim
The central discovery is that neuron permutation mismatch between adjacent base layers can be neutralized at initialization time by solving a block-wise optimal transport problem. For each block $b$ of a new layer placed between base layers $f_i$ and $f_{i+1}$, OpT-DeUS first aligns the block to its predecessor within $f_i$ using a transport matrix propagated by the Transport Matrix Flow, then computes a transport matrix $T^{\text{out}}$ between this aligned block and the corresponding block in $f_{i+1}$ by minimizing Euclidean transport cost with Sinkhorn-Knopp, and initializes the new block as the average of the aligned pair. Attention output and MLP down projections are zero-initialized,
Load-bearing premise
The load-bearing premise is that the optimal-transport correspondence between neurons of adjacent layers, taken from Transformer fusion work, remains functionally valid when those layers come from the same network and are being averaged to create a new intermediate layer; if the transport matches unrelated neurons, the benefit over plain averaging disappears.
Editorial extensions
If this is right
- Depth up-scaling can be made both more accurate and cheaper by spending a small one-time OT computation at initialization instead of relying on copy or average initializations.
- Placing new layers in the top half of the model shortens back-propagation through trainable layers, giving faster wall-clock training without sacrificing accuracy.
- The method's creation cost stays low at larger scales, unlike auxiliary-network initialization whose creation time grows sharply.
- OT alignment beats plain averaging on both tested model sizes and most benchmarks, indicating that neuron permutation mismatch, not averaging itself, is the limiting factor.
- The gains persist through supervised fine-tuning after continual pre-training, so the initialization benefit is not washed out by subsequent training.
Reading between the lines
- If the OT correspondence is genuinely functional, the same transport matrices could be reused for other weight-space operations, such as layer pruning or merging models of different depths, where neuron permutation mismatch also matters.
- The weight-based neuron distance ignores the input distribution; an activation-based cost could sharpen the alignment further, a variant the paper does not test.
- The top-half insertion result implies a continuous efficiency–performance trade-off in insertion depth; scanning fractional positions or per-layer insertion choices may find an optimum the paper's four strategies bracket.
- Because OpT-DeUS costs minutes at initialization, it makes depth up-scaling practical as a routine intermediate step during training, not just a one-off expansion.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. OpT-DeUS is a progressive depth up-scaling method that creates new Transformer layers by aligning adjacent base-layer blocks with optimal transport, averaging the aligned weights, and zero-initializing the attention output and MLP down projections. The inserted layers are the only trainable parameters. The paper reports continual pre-training and supervised fine-tuning results for Llama-3.1-8B expanded to 11.5B and Llama-3.2-1B expanded to 1.72B, compares against SOLAR, LLaMA PRO, LESA, and an averaging-only variant (Avg-DeUS), and analyzes interpolation position, creation time, and training time. The central claims are that OT alignment mitigates neuron permutation mismatch and thereby improves downstream performance and training efficiency.
Significance. Depth up-scaling is practically important, and the paper targets a genuine limitation of copy/average initialization: neuron indices are not semantically aligned across layers. The proposed initialization is concrete, well-specified, and cheap to construct; the creation-time advantage over LESA (37 min vs 4h52m at 11.5B) is a real practical asset; the code is released; and OpT-DeUS matches base perplexity after expansion, verifying its function-preservation property. If the mechanism were cleanly established, this would be a solid contribution to model expansion. At present, however, the experimental design does not isolate the OT alignment mechanism from zero-initialization, and the performance margins over the strongest baseline are small and unreplicated. The contribution is therefore better viewed as a promising recipe requiring further validation.
major comments (4)
- [§6.4 / §4.2–4.3] The OpT-DeUS vs. Avg-DeUS comparison is confounded. OpT-DeUS sets W'_O and W'_Down to zero (Algorithm 1, line 11), while §4.3 states that Avg-DeUS does not apply zero-initialization. Since zero-initializing output projections is itself a function-preserving and training-stabilizing intervention (LLaMA PRO), the consistent gains in Table 1 (e.g., 68.87 vs 68.39 CPT at 11.5B) do not establish that OT alignment drives the improvement. Please add a 2x2 ablation (OT/no-OT, zero-init/no-zero-init) or otherwise match the zero-init condition, and report initial perplexity for all variants. The §6.4 conclusion 'confirms that using OT for neuron alignment ... enhances downstream performance' is not supported as written.
- [Table 1 / §6.1] The headline performance advantage over LESA is within typical evaluation noise. Average deltas are 0.10 (11.5B CPT), 0.20 (11.5B SFT), 0.51 (1.72B CPT), and 0.29 (1.72B SFT), and on several individual tasks LESA is better. No multiple seeds, confidence intervals, or significance tests are reported. This is insufficient to support 'better overall performance' as a central claim. Please provide repeated runs or paired tests (e.g., across benchmarks) and report the corresponding variability.
- [§6.2 / Table 2] The Top interpolation position is selected after observing the same eight benchmarks used in the main comparison. The differences among positions are small (average 68.29–68.87), so this is vulnerable to selection on the evaluation set. The training-efficiency benefit of Top insertion is unaffected, but the performance-gain claim should be validated on a held-out split or by pre-registering the Top hypothesis.
- [§6.5 / Table 3] LESA's initial perplexity on Llama-3.2-1B is 871.50, while the base is 11.57; this is two orders of magnitude worse than the other 1B-scale baseline. The paper only hypothesizes auxiliary-network underfitting. Because Table 1's 1.72B comparison against LESA is part of the cross-scale argument, this baseline failure needs to be investigated and reported (e.g., whether the auxiliary network trains successfully, or whether the 871.50 value reflects an implementation issue).
minor comments (4)
- [§6.4 / Table 1] The text says OpT-DeUS 11.5B 'wins six out of eight benchmarks' against Avg-DeUS, but Table 1 shows it wins seven (all except ARC: 82.07 vs 82.15).
- [Algorithm 1] Line 3: the support function δ is never defined; give the exact weight-based δ used, rather than only citing Singh & Jaggi.
- [§3.1] The interpolation equation contains a stray '◦' after M′(x; θ′); it should be M′(x; θ′).
- [Tables 3 and 5] The asterisk/footnote markers are ambiguous in the rendered tables, making it hard to tell which entries are copied from prior work (Yang et al., 2025); please clarify.
Circularity Check
No significant circularity: OpT-DeUS is a constructive initialization method whose claims rest on external benchmarks, not on inputs by construction.
full rationale
I walked the paper's derivation chain and found no step where a prediction or first-principles result reduces to its inputs by definition, fit, or self-citation. OpT-DeUS is a constructive weight-initialization recipe (Algorithm 1): the new layer weights are computed by aligning adjacent base-layer blocks with an OT transport matrix and averaging them, plus zero-initializing W_O and W_Down. There is no fitted parameter that is later reported as a prediction; the Sinkhorn regularization (0.06) is taken from prior external work (Imfeld et al., 2024), and no constants are tuned to the evaluation benchmarks. The interpolation-position choice (Top) is selected after observing the ablation on the same benchmarks, which is a post-hoc selection / multiple-comparisons concern rather than circularity: the reported downstream numbers are not equivalent to the method's input weights by construction. The comparison against Avg-DeUS is confounded because OpT-DeUS uses zero-initialization for W_O/W_Down while Avg-DeUS does not (Section 4.3 vs. Algorithm 1 line 11), so the causal claim in Section 6.4 that OT alignment 'comprehensively enhances downstream performance' is not cleanly supported; however, this is an experimental-validity issue, not a circularity. There are no self-citations from the present authors, and the cited TMF/Sinkhorn results are external, independently published prior work rather than a self-citation chain. Neither the uniqueness nor ansatz-smuggling pattern applies. Therefore the paper is self-contained in the sense relevant to circularity: its empirical claims are evaluated against external benchmarks and are not forced by the initialization equations.
Assumptions & free parameters
free parameters (5)
- Sinkhorn regularization epsilon =
0.06
- Number of new layers k =
50% of base layers (16 for 8B, 8 for 1B)
- Interpolation position =
Top half (between f_{n/2} and f_n)
- Zero-initialized blocks =
W_O and W_down
- Learning rates =
5e-5 / 1e-4 (CPT), 5e-6 / 1e-5 (SFT)
assumptions (5)
- domain assumption Neuron permutation mismatch exists between layers of the same network and harms naive copying/averaging.
- domain assumption Adjacent layers in the base model are functionally similar enough that a fused initialization is beneficial.
- ad hoc to paper The TMF rules of Imfeld et al. (2024) transfer unchanged from transformer fusion to depth up-scaling.
- domain assumption Setting W_O and W_down to zero gives function preservation and is beneficial for progressive training.
- domain assumption Benchmark scores from a single run with no seeds are representative.
Cite this review
Pith. "Pith review of Progressive Depth Up-scaling via Optimal Transport." pith.science (2026). https://pith.science/paper/HBJBLL2Y
@misc{pith2026250808011,
author = {Pith},
title = {Pith review of: Progressive Depth Up-scaling via Optimal Transport},
year = {2026},
howpublished = {\url{https://pith.science/paper/HBJBLL2Y}},
note = {Machine review of arXiv:2508.08011}
}
read the original abstract
Scaling Large Language Models (LLMs) yields performance gains but incurs substantial training costs. Depth up-scaling offers training efficiency by adding new layers to pre-trained models. However, most existing methods copy or average weights from base layers, neglecting neuron permutation differences. This limitation can potentially cause misalignment that harms performance. Inspired by applying Optimal Transport (OT) for neuron alignment, we propose Optimal Transport Depth Up-Scaling (OpT-DeUS). OpT-DeUS aligns and fuses Transformer blocks in adjacent base layers via OT for new layer creation, to mitigate neuron permutation mismatch between layers. OpT-DeUS achieves better overall performance and offers improved training efficiency than existing methods for continual pre-training and supervised fine-tuning across different model sizes. To further evaluate the impact of interpolation positions, our extensive analysis shows that inserting new layers closer to the top results in higher training efficiency due to shorter back-propagation time while obtaining additional performance gains.
Figures
Reference graph
Works this paper leans on
-
[2018]
In International Conference on Learning Repre- sentations
Multi-level Residual Networks from Dynamical Sys- tems View. In International Conference on Learning Repre- sentations. Chen, C.; Yin, Y .; Shang, L.; Jiang, X.; Qin, Y .; Wang, F.; Wang, Z.; Chen, X.; Liu, Z.; and Liu, Q. 2022. bert2BERT: Towards Reusable Pretrained Language Models. In Mure- san, S.; Nakov, P.; and Villavicencio, A., eds., Proceedings of...
arXiv 2022
-
[2019]
Efficient Training of BERT by Progressively Stacking. In Chaudhuri, K.; and Salakhutdinov, R., eds., Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research , 2337–2346. PMLR. Grattafiori, A.; Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; Letman, A.; Mathur, A.; Schelten, A...
arXiv 2024
-
[2021]
WinoGrande: an adversarial winograd schema chal- lenge at scale. Commun. ACM, 64(9): 99–106. Samragh, M.; Mirzadeh, S. I.; Alizadeh-Vahid, K.; Faghri, F.; Cho, M.; Nabi, M.; Naik, D.; and Farajtabar, M
-
[2024]
Scaling Smart: Accelerating Large Language Model Pre-Training with Small Model Initialization. In Reza- gholizadeh, M.; Passban, P.; Samiee, S.; Partovi Nia, V .; Cheng, Y .; Deng, Y .; Liu, Q.; and Chen, B., eds.,Proceed- ings of The 4th NeurIPS Efficient Natural Language and Speech Processing Workshop, volume 262 of Proceedings of Machine Learning Resea...
arXiv 2022
-
[2936]
Clark, P.; Cowhey, I.; Etzioni, O.; Khot, T.; Sabharwal, A.; Schoenick, C.; and Tafjord, O
Minneapolis, Minnesota: Association for Computa- tional Linguistics. Clark, P.; Cowhey, I.; Etzioni, O.; Khot, T.; Sabharwal, A.; Schoenick, C.; and Tafjord, O. 2018. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Chal- lenge. arXiv:1803.05457. Dao, T. 2024. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitio...
arXiv 2018
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.