Mobius Learning: Cyclic Depth Folding in Transformers

Tongtian Zhu

REVIEW 3 major objections 5 minor 25 references

Cyclic depth folding beats fixed-order looping in Transformers.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 16:49 UTC pith:425IVT43

load-bearing objection A genuinely new cyclic-depth training idea with clean math, but the headline loss improvement rests on an unstated parameter-matching detail and unreplicated single runs. the 3 major comments →

arxiv 2607.17843 v1 pith:425IVT43 submitted 2026-07-20 cs.LG cs.CLcs.DC

Mobius Learning: Cyclic Depth Folding in Transformers

Tongtian Zhu This is my paper

classification cs.LG cs.CLcs.DC

keywords cyclic depth foldingdepth-role superpositionMöbius Learninglooped Transformersparameter sharingdistributed traininglanguage model pretraining

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to show that a Transformer block's depth role is not fixed by its position. In Möbius Learning, data streams from different workers enter at different block groups and then circle through all groups in cyclically shifted orders, so every block group is trained at every source-relative depth. Against fixed-order looped transformers, this construction reaches lower validation loss at loop depths 6, 10, and 15 under matched token budgets and block-application counts. If the result holds, it means shared parameters can usefully occupy multiple depth roles and that cyclic depth folding is a practical way to distribute a deep model with each worker holding only one block group.

Core claim

Möbius Learning replaces the single ordered Transformer pass with worker-indexed cyclic passes: for batches originating at worker s, the block groups are applied in order F_s, F_{s+1}, ..., then wrap around, for L complete passes. The same group F_i then appears at each of the P within-pass positions exactly once across the P data streams. The paper's four-worker experiments with a 124M-parameter GPT-2 variant on 2.5B FineWeb tokens report lower validation loss for Möbius Learning than for ordered-loop training at L=6, 10, and 15, with differences of -0.0084, -0.0059, and -0.0083 under equal token budgets and equal block-group application counts K=4L.

What carries the argument

The load-bearing construction is cyclic depth folding, defined by the map Φ_{s,K}=F_{⟨s+K-1⟩_P}∘...∘F_{s+1}∘F_s, with source-relative depth d_{s,i}=⟨i-s⟩_P. Since s↦d_{s,i} is a bijection over the P workers, every block group is trained at every depth role exactly once per set of originating workers, a property the paper calls depth-role superposition. This is what lets one set of shared parameters receive gradients from both shallow and deep contexts, while each worker stores only its own block group and local data.

Load-bearing premise

The comparison assumes that averaging validation loss across all P cyclic starting points is the right way to measure success; if a single fixed starting point were used instead, the advantage could shrink or reverse, and the reported losses differ by only about 0.006-0.008 with no variance reported.

What would settle it

Run the same four-worker, 2.5B-token setup with several seeds and compute the validation loss under the paper's equal-weighted average, and also under each individual starting point separately. If at L=10 the average no longer beats ordered-loop training across seeds, or if any single starting point underperforms the ordered-loop baseline by a clear margin, the central claim fails. A P=2 version, where each block group serves only two roles, would also test whether the benefit scales with role coverage.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

A block group's parameters need not be confined to one fixed depth; the same parameters can serve shallow and deep roles in the same model.
Cyclic order variation across data streams is a usable training signal, not just a source of noise: at larger loop depths it improves validation loss relative to a fixed order.
Four workers can realize a model four times as deep per worker's memory footprint, since each worker stores one block group rather than the full stack.
Across the P data streams, each block group receives balanced supervision at every source-relative depth, which the paper links to the improved validation loss.
The reported evaluation averages over all P cyclic starting points, giving equal weight to every induced order.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If depth-role superposition is the active mechanism, intermediate representations produced by the same block group should show signs of both shallow and deep functions depending on the path; this is directly probeable.
The benefit is reported for P=4; varying P (e.g., 2 or 8 workers) would map how the advantage depends on the number of depth roles each block group must cover.
Because the reported loss differences are small and no seed variance is shown, repeating the four-worker runs with several seeds would clarify whether the effect is robust; the paper itself flags this as a preliminary study.
A natural extension is to apply cyclic depth folding to decoder blocks in larger models with longer token budgets, or to other sequence-modeling architectures, to see whether the depth-role superposition benefit transfers.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

A genuinely new cyclic-depth training idea with clean math, but the headline loss improvement rests on an unstated parameter-matching detail and unreplicated single runs.

read the letter

The idea is the real thing: training a shared block group under cyclically shifted orders so each group sees both shallow and deep roles is a construction I have not seen before, and it is formalized carefully. The paper also deserves credit for reporting the negative results at L=1 and 2 rather than burying them, and for being explicit that this is a preliminary version. The LayerShuffle distinction is handled well: that work randomizes order after pretraining for robustness, whereas this trains from scratch with deterministic cyclic shifts and loop depth. If the result holds, it opens a design space for looped transformers and worker-local distributed training.

But the central empirical claim is not yet solid. The stress-test note lands: the paper never states whether the path-specific Es and Rs in Eq. 7 are shared copies or independent per-worker parameters. The ordered baseline uses a single shared embedding and head; the Möbius notation indexes them by s. If each worker has its own E and R, the Möbius model has P−1 extra embeddings and heads, which for GPT-2 small is tens of millions of extra parameters. That would make the comparison parametically unmatched and the reported loss gap a capacity artifact, not evidence for depth-role superposition. This is load-bearing, and the paper just does not clarify it.

Second, the numbers themselves are fragile: single runs, no seeds, no error bars, and differences of 0.006–0.008. These could easily be run-to-run noise at this scale. The evaluation also averages over all P cyclic starting points; the paper is transparent about this, but it is a nonstandard metric, and it would be useful to see the loss under the standard fixed order for the Möbius model. If that number is noticeably worse, the story shifts from 'better' to 'different but comparable on average.'

The distributed memory advantage is appealing but not measured — no wall-clock or memory numbers, just a structural argument. That is fine for a preliminary report, but it remains a motivating conjecture, not a demonstrated benefit.

The paper shows clear thinking and honest engagement, so I would not desk reject it in spirit, but the load-bearing ambiguity makes the empirical claim unverifiable as written. The right next step is for the authors to state explicitly whether Es/Rs are shared, and if they are shared, say so in the equations; if they are independent, rerun the baseline with the same parameter count or acknowledge the mismatch.

Bottom line: worth a serious referee, mostly because the idea is novel and the math is clean, but the referee would need to insist on the parameter clarification and seed variance before treating the loss improvement as real. I would read a revised version with those details; I would not cite the current empirical result in my own work until it is confirmed at larger scale.

Referee Report

3 major / 5 minor

Summary. The paper proposes Möbius Learning, a distributed training architecture in which P workers each hold one Transformer block group, and local data streams start at their own worker’s block group and then traverse the groups in cyclically shifted orders. The same block group is therefore applied at every source-relative depth across the P data streams, which the paper calls depth-role superposition. The central empirical claim is that, in four-worker experiments with a modded GPT-2 small (124M) model trained on 2.5B FineWeb tokens with Muon, Möbius Learning achieves lower validation loss than an ordered-loop Transformer baseline at loop depths L = 6, 10, and 15, while matching token budget and the number of block-group applications (K = 4L) within each row of Table 1. The paper also presents negative results at L = 1 and L = 2, which supports the claim that the effect is not universal.

Significance. The idea of depth-role superposition is genuinely novel and, if validated, would open a new design axis for Transformer depth and for memory-constrained distributed training. The mathematical formulation is clear, and the experiment design is honest in stating the token budget, the loop-depth definition, and the matched K = 4L block-group application counts. The negative results at L = 1 and L = 2 are a useful guard against a broad overclaim. However, the headline empirical result currently rests on a single unseeded run per configuration, with small loss differences (0.006–0.008), no code release, and an unresolved ambiguity about whether the per-path input embeddings and prediction heads are shared or independent. These issues are load-bearing: if the embeddings/heads are independent, the Möbius model has substantially more parameters than the ordered-loop baseline, and the reported advantage could be a capacity effect rather than evidence for depth-role superposition. With the evidence as presented, the central claim is plausible but not yet established at the standard expected for a journal publication.

major comments (3)

[§2.2, §2.4, Appendix A, Table 1] The manuscript never states whether Es and Rs in Eqs. (7)–(8) are shared copies of E and R or independent per-path parameters. Appendix A only says decoder-block parameters are not synchronized; it does not mention embeddings or prediction heads. If each of the P paths has its own embedding and head, the Möbius model has P−1 additional embeddings/heads compared to the ordered-loop baseline. At GPT-2 small scale, that is tens of millions of extra parameters per path, so the comparison in Table 1 is not parameter-matched and the reported Δ could reflect extra capacity or shard-specific adaptation rather than depth-role superposition. Please state explicitly whether Es and Rs are tied (Es=E and Rs=R for all s), and if they are not, run a controlled experiment with tied E/R or a baseline with equivalent extra capacity.
[§3.1, Table 1, Figure 1] All reported numbers are from a single run per configuration, with no seed variance. The differences at L=6,10,15 are only 0.0059–0.0084 in validation loss; at this scale and token budget these differences may easily lie within run-to-run noise. The paper also omits the Muon hyperparameters (learning rate, betas, schedule, global batch size), despite stating they are fixed. To support the headline claim, report mean ± standard deviation over at least three seeds for each configuration and provide full training hyperparameters and, ideally, code.
[§2.4, Eq. (8)] The reported validation loss is the equally weighted average over all P cyclic starting points. This is a defensible criterion, but it should be justified explicitly, and the paper should report the same metric for a single fixed starting order (e.g., s=0 only) or per-s values. If the advantage only appears under the averaged criterion, the practical meaning of the improvement for standard single-stream inference is unclear. The current text states the choice but does not discuss its implications.

minor comments (5)

[§3.1] The hyperparameter description is too vague: 'the Muon configuration, learning-rate schedule, and global batch are fixed across runs' is not sufficient for reproducibility. List all relevant hyperparameters and schedule details.
[Table 1] The L=15 row reads '60 3.2337' rather than '60 3.2337'—a missing space that makes the table hard to parse.
[Figure 1] The x-axis labels 1, 2, 6, 10, 15 are plotted as if equally spaced; since these values are not linear, consider using a proper categorical or logarithmic axis to avoid visual distortion.
[§5] The 'Citation suggestions' paragraph is unusual for a formal submission and should be moved to acknowledgments or removed.
[§2.4] The validation distributions Vs are not described in detail. State whether each worker uses a distinct shard of the FineWeb validation set and whether the shards are disjoint.

Circularity Check

0 steps flagged

No significant circularity: the claimed Möbius advantage is an empirical comparison against a matched ordered-loop baseline, not a quantity forced by definition or self-citation.

full rationale

The paper's central claim is an experimental result: at L=6, 10, and 15, Möbius Learning reaches lower validation loss than ordered-loop training. The derivation chain is Eqs. (1)-(8), which define two different training objectives; no parameter is fitted to the comparison outcome. The paper states the comparison is matched: 'Each table row holds L fixed: both constructions use the same token budget, perform K=4L block-group applications along each input path, and apply the input embedding and prediction map once.' The only construction-level identity is that for Möbius, every block group occupies every source-relative depth exactly once (Eq. 5 and the bijection s→d_{s,i}); this is a deliberate architectural property, not a predicted outcome. There are no self-citations; the references are external prior work. The potential concerns raised about parameter matching (whether E_s/R_s are shared or worker-specific) and lack of seed variance relate to experimental validity, not to circularity: even if the comparison were unfair, that would be a confound, not an equivalence-by-construction. The paper also reports negative results at L=1 and L=2, so the claim is not forced by the chosen metric. Score 0.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central result is an empirical comparison. It depends on experimental design choices (P, L, token budget), on the unstated Muon and architecture details, and on the domain assumption that shared parameters can serve multiple depth roles. No constants are fitted to data, and no new physical or mathematical entities are introduced.

free parameters (4)

P (number of workers/block groups) = 4
Chosen by hand. The depth-role coverage and the reported comparisons depend on P=4.
Loop depths L where the claim is made = 6, 10, 15
Chosen test points. The paper does not test all depths, and the advantage appears only at these larger values.
Token budget = 2.5B
Chosen following Chinchilla allocation; the result is specific to this budget.
Muon hyperparameters (learning rate, etc.)
Mentioned as fixed across runs, but values are not reported, yet the central comparison depends on them.

axioms (4)

standard math Cyclic modular arithmetic in Eq. (3) correctly represents the intended block-group circulation.
The recurrence uses modular indexing to define the cyclic order. This is a mathematical construction, not an empirical assumption.
domain assumption The same block-group parameters can be trained at multiple source-relative depths without catastrophic interference.
This is the core premise of depth-role superposition. The paper tests it empirically, but it is not proven.
domain assumption Worker-local data distributions are comparable, and averaging losses across workers is a valid evaluation.
Equations (6)–(8) assume equal weighting of all workers and that their local distributions are meaningful for language modeling. The paper does not describe the data partitioning.
domain assumption The 'modded GPT-2 small' architecture and Muon configuration are suitable and equivalent across methods.
The modded architecture and optimizer settings are not fully specified, so the comparison relies on unstated implementation details.

pith-pipeline@v1.3.0-alltime-deepseek · 3516 in / 8089 out tokens · 183041 ms · 2026-08-01T16:49:15.041329+00:00 · methodology

0 comments

read the original abstract

Transformer-based language models organize computation along an ordered depth axis, where shallow and deep blocks often develop distinct representational roles. We challenge the conventional view that these roles must remain tied to a block's position in the ordered sequence. We introduce Mobius Learning, a training architecture based on cyclic depth folding, in which different data streams follow cyclically shifted block orders. The same block group is therefore applied early in the block sequence for some data streams and late for others, so it is optimized in both shallow and deep roles, a phenomenon we call depth-role superposition. Surprisingly, in four-worker experiments with a modded GPT-2 small (124M) model trained on 2.5B FineWeb tokens using Muon, Mobius Learning achieves lower validation loss than a fixed-order looped Transformer at larger numbers of Transformer block-sequence passes. This counterintuitive result shows that a block group need not remain confined to one fixed shallow or deep role within the block sequence and opens a new design space based on cyclic depth folding. Crucially, this structure makes Mobius Learning particularly well suited to memory-constrained distributed training: raw training data remain local, while each worker stores one block group rather than the complete Transformer block stack.

Figures

Figures reproduced from arXiv: 2607.17843 by Tongtian Zhu.

**Figure 1.** Figure 1: Left: FineWeb validation loss after 2.5B training tokens for four-worker modded GPT-2 small (124M) runs trained with Muon. Ordered-loop training and Möbius Learning are compared at the same loop depth L, defined as the number of complete Transformer block-sequence passes. With P = 4 block groups, each setting therefore performs K = 4L block-group applications. At loop depths L = 6, 10, 15, Möbius Learning … view at source ↗

**Figure 2.** Figure 2: Conceptual comparison of three forms of parallelism. Replicated data parallelism retains [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Figure 3: Ordered-loop training and Möbius Learning for P = 4. Left: every worker-local batch follows the fixed order F0, F1, F2, F3 in each of L complete passes, and the resulting losses are averaged across workers. Right: the column indexed by originating worker s depicts that batch’s logical path, which begins at block group Fs and follows the worker-indexed cyclic order; the columns do not represent separate phy… view at source ↗

discussion (0)

Reference graph

Works this paper leans on

25 extracted references · 1 canonical work pages

[1]

Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look at? an analysis of BERT’s attention, 2019. URL https://arxiv.org/abs/1906. 04341

2019
[2]

Uni- versal transformers, 2019

Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Uni- versal transformers, 2019. URLhttps://arxiv.org/abs/1807.03819

Pith/arXiv arXiv 2019
[3]

Reducing transformer depth on demand with structured dropout

Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. InInternational Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=SylO2yStDr

2020
[4]

Layershuffle: Enhancing robustness in vision transformers by randomizing layer execution order, 2024

Matthias Freiberger, Peter Kun, Anders Sundnes Løvlie, and Sebastian Risi. Layershuffle: Enhancing robustness in vision transformers by randomizing layer execution order, 2024. URL https://arxiv.org/abs/2407.04513

Pith/arXiv arXiv 2024
[5]

Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein

Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test- time compute with latent reasoning: A recurrent depth approach, 2025. URL https: //arxiv.org/abs/2502.05171

Pith/arXiv arXiv 2025
[6]

Transformer feed-forward layers are key-value memories, 2021

Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories, 2021. URLhttps://arxiv.org/abs/2012.14913. 8 MÖBIUSLEARNING

Pith/arXiv arXiv 2021
[7]

Rae, and Laurent Sifre

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laurent Sifre...

2022
[8]

Le, Yonghui Wu, and Zhifeng Chen

Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Xu Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V . Le, Yonghui Wu, and Zhifeng Chen. GPipe: Efficient training of giant neural networks using pipeline paral- lelism. InAdvances in Neural Information Processing Systems, volume 32, pages 103–112, 2019. URL https://proceedings.neurips.cc/pa...

2019
[9]

ALBERT: A lite BERT for self-supervised learning of language representations, 2020

Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite BERT for self-supervised learning of language representations, 2020. URLhttps://arxiv.org/abs/1909.11942

Pith/arXiv arXiv 2020
[10]

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas

H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communication-efficient learning of deep networks from decentralized data, 2017. URLhttps://arxiv.org/abs/1602.05629

Pith/arXiv arXiv 2017
[11]

Devanur, Gregory R

Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. PipeDream: Generalized pipeline parallelism for DNN training. InProceedings of the 27th ACM Symposium on Operating Systems Principles, pages 1–15, 2019. doi: 10.1145/3341301.3359646. URL https://doi. org/10.1145/33...

arXiv 2019
[12]

The FineWeb datasets: Decanting the web for the finest text data at scale

Guilherme Penedo, Hynek Kydlíˇcek, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro von Werra, and Thomas Wolf. The FineWeb datasets: Decanting the web for the finest text data at scale. InAdvances in Neural Information Processing Systems, volume 37, pages 30811–30849, 2024. doi: 10.52202/079017-0970. URL https: //doi.org/10.52202...

doi:10.52202/079017-0970 2024
[13]

Hayden Prairie, Zachary Novack, Taylor Berg-Kirkpatrick, and Daniel Y . Fu. Parcae: Scaling laws for stable looped language models, 2026. URL https://arxiv.org/abs/2604. 12946

2026
[14]

Language models are unsupervised multitask learners

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI techni- cal report, 2019. URL https://cdn.openai.com/better-language-models/ language_models_are_unsupervised_multitask_learners.pdf

2019
[15]

Mixture-of-depths: Dynamically allocating compute in transformer-based language models, 2024

David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. Mixture-of-depths: Dynamically allocating compute in transformer-based language models, 2024. URLhttps://arxiv.org/abs/2404.02258. 9 MÖBIUSLEARNING

Pith/arXiv arXiv 2024
[16]

Subformer: Exploring weight sharing for parameter efficiency in generative transformers, 2021

Machel Reid, Edison Marrese-Taylor, and Yutaka Matsuo. Subformer: Exploring weight sharing for parameter efficiency in generative transformers, 2021. URL https://arxiv. org/abs/2101.00234

Pith/arXiv arXiv 2021
[17]

Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, and Sashank J. Reddi. Reasoning with latent thoughts: On the power of looped transformers, 2025. URL https://arxiv. org/abs/2502.17416

Pith/arXiv arXiv 2025
[18]

Lessons on parameter sharing across layers in transformers

Sho Takase and Shun Kiyono. Lessons on parameter sharing across layers in transformers. In Proceedings of the Fourth Workshop on Simple and Efficient Natural Language Processing (SustaiNLP), pages 78–90, Toronto, Canada (Hybrid), July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.sustainlp-1.5. URL https://aclanthology.org/ 2023.s...

work page doi:10.18653/v1/2023.sustainlp-1.5 2023
[19]

BERT rediscovers the classical NLP pipeline,

Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline,
[20]

Chandra Thapa, M. A. P. Chamikara, Seyit Camtepe, and Lichao Sun. SplitFed: When federated learning meets split learning, 2022. URLhttps://arxiv.org/abs/2004.12088

Pith/arXiv arXiv 2022
[21]

Gomez, Lukasz Kaiser, and Illia Polosukhin

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, pages 5998– 6008, 2017. URL https://proceedings.neurips.cc/paper/2017/hash/ 3f5ee243547dee91fbd053c1c4a845aa-Abstract.html

2017
[22]

Split learning for health: Distributed deep learning without sharing raw patient data, 2018

Praneeth Vepakomma, Otkrist Gupta, Tristan Swedish, and Ramesh Raskar. Split learning for health: Distributed deep learning without sharing raw patient data, 2018. URL https: //arxiv.org/abs/1812.00564

Pith/arXiv arXiv 2018
[23]

The bottom-up evolution of representations in the transformer: A study with machine translation and language modeling objectives, 2019

Elena V oita, Rico Sennrich, and Ivan Titov. The bottom-up evolution of representations in the transformer: A study with machine translation and language modeling objectives, 2019. URL https://arxiv.org/abs/1909.01380

Pith/arXiv arXiv 2019
[24]

Scaling latent reasoning via looped language models, 2025

Rui-Jie Zhu, Zixuan Wang, Kai Hua, Tianyu Zhang, Ziniu Li, et al. Scaling latent reasoning via looped language models, 2025. URLhttps://arxiv.org/abs/2510.25741. Appendix A. Implementation Notes The reported implementation follows the state recurrence in Equation (3) with P= 4 workers and assigns block group Fi to worker i. Forward computation sends hidde...

Pith/arXiv arXiv 2025
[2019]

URLhttps://arxiv.org/abs/1905.05950

Pith/arXiv arXiv 1905

[1] [1]

Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look at? an analysis of BERT’s attention, 2019. URL https://arxiv.org/abs/1906. 04341

2019

[2] [2]

Uni- versal transformers, 2019

Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Uni- versal transformers, 2019. URLhttps://arxiv.org/abs/1807.03819

Pith/arXiv arXiv 2019

[3] [3]

Reducing transformer depth on demand with structured dropout

Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. InInternational Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=SylO2yStDr

2020

[4] [4]

Layershuffle: Enhancing robustness in vision transformers by randomizing layer execution order, 2024

Matthias Freiberger, Peter Kun, Anders Sundnes Løvlie, and Sebastian Risi. Layershuffle: Enhancing robustness in vision transformers by randomizing layer execution order, 2024. URL https://arxiv.org/abs/2407.04513

Pith/arXiv arXiv 2024

[5] [5]

Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein

Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test- time compute with latent reasoning: A recurrent depth approach, 2025. URL https: //arxiv.org/abs/2502.05171

Pith/arXiv arXiv 2025

[6] [6]

Transformer feed-forward layers are key-value memories, 2021

Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories, 2021. URLhttps://arxiv.org/abs/2012.14913. 8 MÖBIUSLEARNING

Pith/arXiv arXiv 2021

[7] [7]

Rae, and Laurent Sifre

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laurent Sifre...

2022

[8] [8]

Le, Yonghui Wu, and Zhifeng Chen

Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Xu Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V . Le, Yonghui Wu, and Zhifeng Chen. GPipe: Efficient training of giant neural networks using pipeline paral- lelism. InAdvances in Neural Information Processing Systems, volume 32, pages 103–112, 2019. URL https://proceedings.neurips.cc/pa...

2019

[9] [9]

ALBERT: A lite BERT for self-supervised learning of language representations, 2020

Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite BERT for self-supervised learning of language representations, 2020. URLhttps://arxiv.org/abs/1909.11942

Pith/arXiv arXiv 2020

[10] [10]

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas

H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communication-efficient learning of deep networks from decentralized data, 2017. URLhttps://arxiv.org/abs/1602.05629

Pith/arXiv arXiv 2017

[11] [11]

Devanur, Gregory R

Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. PipeDream: Generalized pipeline parallelism for DNN training. InProceedings of the 27th ACM Symposium on Operating Systems Principles, pages 1–15, 2019. doi: 10.1145/3341301.3359646. URL https://doi. org/10.1145/33...

arXiv 2019

[12] [12]

The FineWeb datasets: Decanting the web for the finest text data at scale

Guilherme Penedo, Hynek Kydlíˇcek, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro von Werra, and Thomas Wolf. The FineWeb datasets: Decanting the web for the finest text data at scale. InAdvances in Neural Information Processing Systems, volume 37, pages 30811–30849, 2024. doi: 10.52202/079017-0970. URL https: //doi.org/10.52202...

doi:10.52202/079017-0970 2024

[13] [13]

Hayden Prairie, Zachary Novack, Taylor Berg-Kirkpatrick, and Daniel Y . Fu. Parcae: Scaling laws for stable looped language models, 2026. URL https://arxiv.org/abs/2604. 12946

2026

[14] [14]

Language models are unsupervised multitask learners

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI techni- cal report, 2019. URL https://cdn.openai.com/better-language-models/ language_models_are_unsupervised_multitask_learners.pdf

2019

[15] [15]

Mixture-of-depths: Dynamically allocating compute in transformer-based language models, 2024

David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. Mixture-of-depths: Dynamically allocating compute in transformer-based language models, 2024. URLhttps://arxiv.org/abs/2404.02258. 9 MÖBIUSLEARNING

Pith/arXiv arXiv 2024

[16] [16]

Subformer: Exploring weight sharing for parameter efficiency in generative transformers, 2021

Machel Reid, Edison Marrese-Taylor, and Yutaka Matsuo. Subformer: Exploring weight sharing for parameter efficiency in generative transformers, 2021. URL https://arxiv. org/abs/2101.00234

Pith/arXiv arXiv 2021

[17] [17]

Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, and Sashank J. Reddi. Reasoning with latent thoughts: On the power of looped transformers, 2025. URL https://arxiv. org/abs/2502.17416

Pith/arXiv arXiv 2025

[18] [18]

Lessons on parameter sharing across layers in transformers

Sho Takase and Shun Kiyono. Lessons on parameter sharing across layers in transformers. In Proceedings of the Fourth Workshop on Simple and Efficient Natural Language Processing (SustaiNLP), pages 78–90, Toronto, Canada (Hybrid), July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.sustainlp-1.5. URL https://aclanthology.org/ 2023.s...

work page doi:10.18653/v1/2023.sustainlp-1.5 2023

[19] [19]

BERT rediscovers the classical NLP pipeline,

Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline,

[20] [20]

Chandra Thapa, M. A. P. Chamikara, Seyit Camtepe, and Lichao Sun. SplitFed: When federated learning meets split learning, 2022. URLhttps://arxiv.org/abs/2004.12088

Pith/arXiv arXiv 2022

[21] [21]

Gomez, Lukasz Kaiser, and Illia Polosukhin

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, pages 5998– 6008, 2017. URL https://proceedings.neurips.cc/paper/2017/hash/ 3f5ee243547dee91fbd053c1c4a845aa-Abstract.html

2017

[22] [22]

Split learning for health: Distributed deep learning without sharing raw patient data, 2018

Praneeth Vepakomma, Otkrist Gupta, Tristan Swedish, and Ramesh Raskar. Split learning for health: Distributed deep learning without sharing raw patient data, 2018. URL https: //arxiv.org/abs/1812.00564

Pith/arXiv arXiv 2018

[23] [23]

The bottom-up evolution of representations in the transformer: A study with machine translation and language modeling objectives, 2019

Elena V oita, Rico Sennrich, and Ivan Titov. The bottom-up evolution of representations in the transformer: A study with machine translation and language modeling objectives, 2019. URL https://arxiv.org/abs/1909.01380

Pith/arXiv arXiv 2019

[24] [24]

Scaling latent reasoning via looped language models, 2025

Rui-Jie Zhu, Zixuan Wang, Kai Hua, Tianyu Zhang, Ziniu Li, et al. Scaling latent reasoning via looped language models, 2025. URLhttps://arxiv.org/abs/2510.25741. Appendix A. Implementation Notes The reported implementation follows the state recurrence in Equation (3) with P= 4 workers and assigns block group Fi to worker i. Forward computation sends hidde...

Pith/arXiv arXiv 2025

[25] [2019]

URLhttps://arxiv.org/abs/1905.05950

Pith/arXiv arXiv 1905