Pith. sign in

REVIEW 3 major objections 5 minor 36 references

StateFlow: Sequence Pipeline Parallelism for Long-Context Modeling with Linear Recurrence

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read StateFlow: chunking long sequences for pipeline parallelism cuts training memory 2.45x and raises throughput 2.22x.

desk verdict First SPP system for linear-recurrent models that passes boundary states between chunks; the scheduling idea is sound and the evaluation is broad, but the 'mathematically exact' claim needs a gradient-equivalence check before I'd fully trust the speedups. read the letter →

arxiv 2608.06838 v1 pith:AIHI4OQJ submitted 2026-08-07 cs.DC

Wenxuan Zhao , Yingfa Chen , Xu Han , Wenjing Han , Tianbo Huang , Zhiyu Li , Ao Sun , Jingheng Xu
show 2 more authors
Lin Gan Guangwen Yang
This is my paper · ORCID
classification cs.DC
keywords sequencepipelineparallelismlinearrecurrencelong-contexttrainingactivationmemoryreductionstatetransitionoverlaphybridattentionmodelsschedulingchunkedpartitioning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that long-context training of models with linear recurrence can be made much cheaper by splitting each training sequence into chunks and pipelining those chunks across GPUs, instead of pipelining whole sequences. The system, StateFlow, passes compact boundary states between chunks in the forward pass and boundary-state gradients in the backward pass, and releases each chunk's activations as soon as its local backward computation finishes. On Gated DeltaNet and Mamba-3 models up to 32B parameters at 256K context, the paper reports up to $2.22\times$ training throughput and $2.45\times$ peak-memory reduction compared with conventional pipeline parallelism, and it runs configurations that otherwise run out of memory. The paper asserts that the chunked schedule is mathematically exact, preserving the same arithmetic as full-sequence training.

What carries the argument

The central object is the chunk-level boundary state: for a partition $X=C_0\|\cdots\|C_{N-1}$, each chunk $C_i$ carries a compact input state $S_i^{\text{in}}$ and output state $S_i^{\text{out}}$, plus auxiliary local-convolution states. Forward correctness is the chain $S_{i+1}^{\text{in}}=S_i^{\text{out}}$ (Eq. 5), and backward correctness is the reverse chain $\bar{S}_i^{\text{out}}=\bar{S}_{i+1}^{\text{in}}$ (Eq. 7). These chains, together with the chunk-level partial order in Eq. 8, turn each chunk's forward and backward pass into an independently scheduled pipeline unit while preserving the arithmetic of the un-chunked recurrence. Supporting machinery includes a hybrid cost model $\mathrm{Cost}_i=(L_{\text{lin}}C_{\text{lin}}+C_{\text{dense}})s_i + L_{\text{soft}}C_{\text{soft}}(c_i s_i+s_i^2/2)$ whose equal-cost solution anchors a searched family of partitions $c_i(\alpha)=(1-\alpha)c_i^{\text{equal}}+\alpha c_i^{\text{hybrid}}$, and a split-and-overlap schedule that runs state transitions concurrently with surrounding dense kernels.

What would settle it

Train the same recurrent or hybrid model from the same checkpoint with StateFlow's chunked schedule and with ordinary full-sequence pipeline training, on identical input data and random seed, and compare the resulting per-layer gradients and loss curves. Any difference beyond floating-point tolerance in the gradients, particularly those flowing through the auxiliary $V^{\text{new}}$ representation, would falsify the exactness claim, and a loss divergence would show the speedups come from altered optimization dynamics.

Watch

Extended reading notes

Core claim

The paper's central claim is that sequence pipeline parallelism works for linear-recurrence and hybrid models exactly, not approximately, when the only cross-chunk dependency is a compact boundary state. StateFlow partitions each training sequence into $N$ chunks, schedules each chunk's forward and backward passes as pipeline units, propagates recurrent states forward and boundary-state gradients backward, and releases activations chunk by chunk. For hybrid models it adds profile-guided nonuniform chunking that balances the growing softmax-attention cost of later chunks, and it overlaps low-parallelism state transitions with surrounding dense kernels. On Gated DeltaNet and Mamba-3 models up to 32B parameters and 256K context, the paper reports up to $2.22\times$ throughput improvement and $2.45\times$ peak-memory reduction over conventional pipeline parallelism, making configurations feasible that otherwise run out of memory on the same hardware.

Load-bearing premise

The load-bearing premise is that chunking the sequence into pipeline units reproduces exactly the same forward and backward arithmetic as training on whole sequences, including the special delta-rule state update path; the paper asserts this without reporting a numerical check.

Editorial extensions

If this is right

  • Recurrent and hybrid models can be trained at 256K context on the same GPU counts where conventional pipeline parallelism runs out of memory, because per-chunk activation release removes whole-sequence retention.
  • Pipeline bubbles shrink as the chunk count $N$ grows, which makes small-batch long-context training substantially faster; the paper measures the largest throughput gains at the smallest global batch sizes.
  • The scheme composes with data, tensor, and sequence parallelism, so its memory and throughput benefits apply on top of other parallelization strategies rather than instead of them.
  • Overlapping state-transition kernels with surrounding dense computation hides the low-utilization portion of recurrent layers, which matters more at higher tensor-parallel degrees where launch grids are smaller.
  • Profile-guided nonuniform chunking yields better hybrid-model throughput than either equal-length or FLOP-balanced partitioning, since kernel efficiency and memory effects shift the optimum.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the exactness claim holds, the same boundary-state chunking idea transfers naturally to long-context inference and prefilling, where the recurrent state already behaves as a compact cache and the memory savings would apply analogously.
  • Editorial inference: the profile-guided search over chunk count $N$ and hybrid skew $\alpha$ suggests that the optimal partitioning depends on hardware and model shape, so an adaptive runtime that tunes these online could extend the reported gains beyond the fixed evaluated configurations.
  • Editorial inference: because the paper reports only throughput and memory, not loss curves or gradient comparisons, the practical speedups would be strengthened by a direct equivalence test; absent that, the results should be read as efficiency measurements of a system whose training dynamics are asserted, not demonstrated, to match the baseline.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. StateFlow is a sequence pipeline parallelism system for training linear-recurrence and hybrid long-context models. It partitions each training sequence into chunks, propagates compact recurrent boundary states and their gradients across chunks, and schedules chunk-level forward/backward units on pipeline stages. For hybrid models it adds profile-guided nonuniform chunking, and for the state-transition bottleneck it proposes grid-size tuning plus overlapping of state transitions with surrounding dense kernels. The paper reports end-to-end experiments on GDN and Mamba-3 models up to 32B parameters and 256K context length, claiming up to 2.22x throughput improvement and 2.45x peak-memory reduction over conventional pipeline parallelism, and asserts that StateFlow is mathematically exact and introduces no approximation.

Significance. If the exactness claim holds, StateFlow is a genuinely useful systems contribution: it extends sequence pipeline parallelism from softmax-attention models to recurrent and hybrid models, where the relevant cross-chunk dependency is a compact state rather than prefix key/value tensors, and it provides a plausible mechanism for training configurations that otherwise run out of memory. The paper's formalization of chunk-level forward/backward dependencies in Section 3.1 is clear, the scheduling constraints are standard, and the evaluation is broad in model scale, context length, and backend (Megatron and Swift). The cost model of Section 3.2 is used to generate partitioning candidates rather than to predict speedups, so the reported results are empirical rather than circular. The main risk is that the central 'mathematically exact' assertion is never verified experimentally; if the chunked backward pass is not gradient-equivalent for the delta-rule path or the local-convolution boundary states, the measured throughput and memory numbers describe a training loop that may not optimize the intended objective.

major comments (3)
  1. [§4.1 and §3.1, Eqs. (4)-(7)] The claim in §4.1 that 'StateFlow is mathematically exact and introduces no approximation' is load-bearing and is not verified anywhere in §4. The backward computation in Eq. (6) is correct only if Bwd reconstructs the same intra-chunk state trajectory that Fwd consumed, including the auxiliary V_new path of the delta rule in Eq. (2) and the short-convolution boundary states mentioned in the footnote to §3.1. The paper reports no loss curves, no gradient-norm comparison, and no convergence check against an unchunked reference implementation. I request such a verification for at least one GDN and one Mamba-3 configuration, comparing the chunked schedule's gradients with those of full-sequence training. Note also that chunking changes the accumulation order of the recurrence in Eq. (1) across chunk boundaries, so 'exact' is at minimum a floating-point associativity claim that should be stated precisely and tested, not asserted.
  2. [§4.1 and Tables 2-3] The reported speedups are selected over StateFlow's free parameters, but the native baselines are not given the same selection budget. In the main tables, StateFlow uses the fastest feasible chunk count N (and, for hybrids, the fastest alpha) while the Megatron and Swift baselines use the fixed configurations listed in §4.1 (PP8/TP1, PP4/TP4, PP4/TP8). Only Figure 3 sweeps PP/TP for one 3B configuration, and there the text says StateFlow 'selects PP8/TP1 in every case,' which is difficult to reconcile with the fixed topology stated in §4.1. Since the paper's headline claims are comparisons against these fixed baselines, the evaluation should either report the best baseline over a comparable sweep, or clearly state that the speedups are against the default PP configurations rather than the best possible PP baseline. Without this clarification, the 'up to 2.22x' figure may overstate the improvement attributable to StateFlow.
  3. [§3.3 and Tables 2-4] The memory-reduction argument in Eq. (13) assumes B_r is negligible for recurrent models and that hybrid softmax K/V tensors can be stored once in sequence-wide buffers. The latter assumption is asserted rather than measured, and the hybrid tables show substantial memory reductions whose decomposition is not reported. For at least one hybrid configuration, please provide a per-device memory breakdown (activations, boundary states, K/V buffers) before and after StateFlow, so that the claimed 'approximately 1/N reduction' can be checked against the additional K/V buffer cost. This is needed to substantiate the memory mechanism for the hybrid case rather than only the aggregate peak-memory number.
minor comments (5)
  1. [Abstract and §4.2] The abstract reports up to 2.45x memory reduction, while §4.2 reports up to 2.54x peak-memory reduction without activation recomputation; please reconcile these numbers or state that they refer to different settings.
  2. [Figure 3 and §4.1] The caption of Figure 3 says StateFlow 'selects PP8/TP1 in every case,' but §4.1 fixes PP/TP configurations per model size; please clarify whether StateFlow also searches over PP/TP topologies or only over N and alpha.
  3. [§3.2, Eq. (9)] The cost model uses C_lin, C_soft, and C_dense but does not specify how these per-token FLOP coefficients are obtained or whether they are measured or computed analytically; please define them precisely.
  4. [Table 1] The table description refers to 'green highlighting' that will not be visible in grayscale or to color-blind readers; please use textual markers such as check marks.
  5. [Figure 1] The schedule diagrams are dense and the state-passing arrows are hard to read at print size; a zoomed inset or a short pseudocode listing of the 1F1B-style issue policy would improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: StateFlow's throughput and memory claims are measured against external baselines; self-citations are background, not load-bearing.

full rationale

StateFlow's central claims are empirical: end-to-end throughput and peak memory are benchmarked against Megatron and Swift native pipelines (Tables 2-4) under identical model, recomputation, and parallelism configurations. The chunk schedule (Eqs. 4-7) and partial order (Eq. 8) define the proposed execution semantics rather than deriving a prediction from a fitted input; the efficiency analysis (Eqs. 12-15) is an explanatory model of the measured schedule, not a substitute for measurement. The hybrid chunk candidates are generated by the FLOP cost model (Eq. 9) and then selected by profiling measured latency, so the reported speedups are not predicted from the cost model. The paper cites works with overlapping authorship (Seq1F1B, Hybrid Linear Attention Done Right, StateX), but these are used as background and baselines and do not carry the load of the central claims. The unverified assertion in Section 4.1 that StateFlow 'is mathematically exact and introduces no approximation' is a validation gap (no loss curves or gradient-norm checks), but it is not circular: no equation in the paper reduces the claimed exactness to its own input or to a self-citation. The absence of convergence verification is a correctness and evidence risk, not a circularity defect.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The system depends on four tuning knobs (N, alpha, BV, split fraction) selected by on-hardware profiling, on the chunkwise decomposition of recurrent layers into Pre/ST/Out with fixed-size boundary states, on the numerical equivalence of the partial-order schedule to full-sequence BPTT, and on the approximate cost model used to generate hybrid chunking candidates. No new physical or mathematical entities are introduced beyond the 'boundary state' abstraction, which is a scheduling artifact rather than a novel modeling construct.

free parameters (4)
  • Sequence chunk count N = e.g., 8 for many 3B configs; varies by workload
    Profiled over {2,4,6,8,16,32}; the fastest feasible N is selected and reported.
  • Hybrid partition skew alpha = 0.75 for 3B; 0.5 to 1.0 for 15B/32B
    Profiled over {0,0.1,...,1.0,1.25}; the best alpha determines chunk boundaries for hybrid models.
  • State-transition tile size BV = 8 to 32 depending on model and pass
    Profiled over {8,16,32,64}; selected for lowest measured state-transition latency.
  • Split fraction for overlap = e.g., 0.7 or 0.5
    Profiled per model, pass, and sequence length; reported as f.3, f.5, f.7, f.9 in Table 4.
assumptions (4)
  • domain assumption A linear recurrence layer can be decomposed chunkwise into Pre, ST, and Out with a fixed-size boundary state carrying all cross-chunk information (Eq. 2).
    Invoked in Section 2.2 and used throughout Section 3.1; true for the Gated DeltaNet and Mamba-3 families, but not for arbitrary RNNs with unbounded state.
  • domain assumption Executing forward and backward chunks in any order satisfying the partial order in Eq. 8 yields numerically identical gradients to full-sequence training.
    Section 3.1 defines the schedule; Section 4.1 asserts exactness but provides no convergence or gradient-norm validation.
  • domain assumption The per-token FLOPs C_lin and C_dense are position-independent, and softmax cost is captured by the quadratic term in Eq. 9.
    Used to compute the FLOP-balanced partition (Eq. 10). The authors acknowledge this is inexact, which motivates the alpha profile search.
  • domain assumption Kernel-level chunking within a chunk (b_{i,j}) preserves the same arithmetic as token-level recurrence; the existing GDN/Mamba kernels are correct.
    Assumed in Section 3.1's bottom-level discussion and used in the overlap schedules of Section 3.4.

how reviews work

0 comments
Cite this review

Pith. "Pith review of StateFlow: Sequence Pipeline Parallelism for Long-Context Modeling with Linear Recurrence." pith.science (2026). https://pith.science/paper/AIHI4OQJ

@misc{pith2026260806838,
  author       = {Pith},
  title        = {Pith review of: StateFlow: Sequence Pipeline Parallelism for Long-Context Modeling with Linear Recurrence},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AIHI4OQJ}},
  note         = {Machine review of arXiv:2608.06838}
}
abstract

Long-context training is increasingly important for large language models, and linear attention and state space models have become popular for improving long-context efficiency. However, efficiently parallelizing long-sequence training for recurrent and hybrid models remains challenging. We present StateFlow, a sequence pipeline parallelism system for models with linear recurrence. StateFlow partitions each sequence into chunks and schedules their execution while propagating boundary states and gradients across chunks, thereby reducing activation lifetimes and improving training throughput. StateFlow further uses profile-guided nonuniform chunking to balance recurrence and softmax attention computation in hybrid models, and overlaps state transitions that expose limited parallelism with surrounding computation. Applying StateFlow to models with up to 32B parameters and 256K context length, we achieve up to \(2.22\times\) throughput improvements and \(2.45\times\) memory reduction compared to conventional pipeline parallelism, enabling otherwise infeasible configurations.

Figures

Figures reproduced from arXiv: 2608.06838 by the authors.

Figure 1
Figure 1. Illustration of StateFlow (ours) and conventional 1F1B, with four devices. (a) Conventional 1F1B PP schedule where a batch is split into 5 microbatches {M0, M1, · · · }, which are used as pipeline units. (b) StateFlow further partitions each microbatch M𝑎 into chunks {M𝑎C0, M𝑎C1, · · · }, where M𝑎C𝑖 denotes the 𝑖-th chunk of 𝑎-th microbatch. from microbatch 𝑎 on pipeline stage 𝑟. A legal StateFlow schedule must sati… view at source ↗
Figure 2
Figure 2. Forward and backward overlap schedules for GDN and Mamba-3. The SM utilization diagrams show how surrounding kernels use idle SMs and may co-reside with state transition kernels when resources permit. Forward Pass. As mentioned in Section 2.2, each chunk with explicit recurrence kernels can be split into three stages: preparation 𝑃𝑟𝑒, state transition 𝑆𝑇 , and output computation 𝑂𝑢𝑡. For each token/chunk, these stag… view at source ↗
Figure 3
Figure 3. Comparison with the best-tuned native baseline and sensitivity to the sequence chunk count 𝑁 for 3B linear recurrence models at 64K on eight GPUs. Throughput is normalized to native PP8/TP1. Dashed lines mark the best native configuration, circled points mark the best 𝑁, and annotations report speedup over the dashed baseline. 16, while [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: End-to-end throughput and peak memory at 𝐺𝐵𝑆 = 8 with activation recomputation. Light and dark bars show the native pipelines and StateFlow using the fastest feasible 𝑁. Annotations show throughput speedup and peak memory reduction. OOM denotes out of memory. 0 .25 .5 …
Figure 5
Figure 5. Figure 5: Sensitivity to the hybrid partition parameter 𝛼 on representative 128K context length. GBS and the chunk number 𝑁 are fixed as labeled. Bars and lines denote throughput and peak memory, respectively. The dark bar marks the highest￾throughput feasible 𝛼, annotated with …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 12 canonical work pages

  1. [1]

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. 2023. GQA: Training General- ized Multi-Query Transformer Models from Multi-Head Checkpoints. InProceedings of the 2023 Conference on Empirical Methods in Natu- ral Language Processing, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for C...

    2023

  2. [2]

    Yingfa Chen, Zhen Leng Thai, Zihan Zhou, Zhu Zhang, Xingyu Shen, Shuo Wang, Chaojun Xiao, Xu Han, and Zhiyuan Liu. 2026. Hybrid Linear Attention Done Right: Efficient Distillation and Effective Ar- chitectures for Extremely Long Contexts. arXiv:2601.22156 [cs.CL] https://arxiv.org/abs/2601.22156

    arXiv 2026

  3. [3]

    Yuhong CHOU, Zehao Liu, Rui-Jie Zhu, Xinyi Wan, Tianjian Li, Congying Chu, Qian Liu, Jibin Wu, and Zejun MA. 2025. ZeCO: Zero-Communication Overhead Sequence Parallelism for Linear Attention. InAdvances in Neural Information Processing Systems, D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghas- semi, and N. Chen (Eds.), Vol. 38. Curran Associ...

    work page 2025

  4. [4]

    Tri Dao and Albert Gu. 2024. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Du- ality. InProceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 235), Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Sca...

    work page 2024

  5. [5]

    Shiqing Fan, Yi Rong, Chen Meng, Zongyan Cao, Siyu Wang, Zhen Zheng, Chuan Wu, Guoping Long, Jun Yang, Lixue Xia, Lansong Diao, Xiaoyong Liu, and Wei Lin. 2021. DAPPLE: a pipelined data parallel approach for training large models. InProceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Paral- lel Programming(Virtual Event, Republic o...

    arXiv 2021

  6. [6]

    Jiarui Fang and Shangchun Zhao. 2024. USP: A Unified Se- quence Parallelism Approach for Long Context Generative AI. arXiv:2405.07719 [cs.LG]https://arxiv.org/abs/2405.07719

    arXiv 2024

  7. [7]

    Albert Gu and Tri Dao. 2024. Mamba: Linear-Time Sequence Modeling with Selective State Spaces. InFirst Conference on Language Modeling. https://openreview.net/forum?id=tEYskw1VY2

    2024

  8. [8]

    Diandian Gu, Peng Sun, Qinghao Hu, Ting Huang, Xun Chen, Yingtong Xiong, Guoteng Wang, Qiaoling Chen, Shangchun Zhao, Jiarui Fang, Yonggang Wen, Tianwei Zhang, Xin Jin, and Xuanzhe Liu. 2024. Loong- Train: Efficient Training of Long-Sequence LLMs with Head-Context Parallelism. arXiv:2406.18485 [cs.DC]https://arxiv.org/abs/2406.18485

    arXiv 2024

Show all 36 references
  1. [9]

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, and zhifeng Chen. 2019. GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism. InAdvances in Neural Informa- tion Processi...

    2019

  2. [10]

    Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. 2023. DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models. arXiv:2309.14509 [cs.LG] https://arxiv.org/a...

    2023 arXiv

  3. [11]

    Kimi Team. 2025. Kimi Linear: An Expressive, Efficient Attention Architecture. arXiv:2510.26692 [cs.CL]https://arxiv.org/abs/2510. 26692

    2025 arXiv

  4. [12]

    Kimi Team. 2026. Kimi K3: Open Frontier Intelligence. arXiv:2607.24653 [cs.CL]https://arxiv.org/abs/2607.24653

    2026

  5. [13]

    Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catan- zaro. 2023. Reducing Activation Recomputation in Large Trans- former Models. InProceedings of Machine Learning and Systems, D. Song, M. Carbin, and T. Chen...

    2023

  6. [14]

    Li, Berlin Chen, Caitlin Wang, Aviv Bick, J

    Aakash Lahoti, Kevin Y. Li, Berlin Chen, Caitlin Wang, Aviv Bick, J. Zico Kolter, Tri Dao, and Albert Gu. 2026. Mamba- 3: Improved Sequence Modeling using State Space Principles. arXiv:2603.15569 [cs.LG]https://arxiv.org/abs/2603.15569

    2026

  7. [15]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval- Augmented Generation for Knowledge-Intensive NLP Tasks. InAd- vances i...

    2020

  8. [16]

    Zhouyang Li, Yuliang Liu, Wei Zhang, Tailing Yuan, Bin Chen, and Chengru Song. 2025. SlimPipe: Memory-Thrifty and Efficient Pipeline Parallelism for Long-Context LLM Training. InProceedings of the In- ternational Conference for High Performance Computing, Networking, Storage a...

    2025

  9. [17]

    Zhuohan Li, Siyuan Zhuang, Shiyuan Guo, Danyang Zhuo, Hao Zhang, Dawn Song, and Ion Stoica. 2021. TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models. InProceedings of the 38th International Conference on Machine Learning (Proceedings of Machine...

    2021

  10. [18]

    Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, Omri Abend, Raz Alon, Tomer Asida, Amir Bergman, Roman Glozman, Michael Gokhman, Avashalom Manevich, Nir Ratner, Noam Rozen, Ere...

    2024 arXiv

  11. [19]

    Hao Liu, Matei Zaharia, and Pieter Abbeel. 2023. Ring Attention with Blockwise Transformers for Near-Infinite Context. InNeurIPS 2023 Foundation Models for Decision Making Workshop.https://openreview. net/forum?id=fXugVDtCQO

    2023

  12. [20]

    Devanur, Gregory R

    Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. 2019. PipeDream: generalized pipeline parallelism for DNN training. InProceedings of the 27th ACM Symposium on Operating Systems Princ...

    2019

  13. [21]

    Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGres- ley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. 2021. Efficient large-scale language model training on GPU cl...

    2021

  14. [22]

    NVIDIA. 2026. Nemotron 3 Ultra: Open, Efficient Mixture-of- Experts Hybrid Mamba-Transformer Model for Agentic Reasoning. arXiv:2606.15007 [cs.CL]https://arxiv.org/abs/2606.15007

    2026

  15. [23]

    Penghui Qi, Xinyi Wan, Nyamdavaa Amar, and Min Lin. 2024. Pipeline Parallelism with Controllable Memory. InAdvances in Neural Informa- tion Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37. Curran Associates,...

    2024 doi

  16. [24]

    Qwen Team. 2026. Qwen3.6-35B-A3B: Agentic Coding Power, Now Open to All. Qwen Technical Blog.https://qwen.ai/blog?id=qwen3.6- 35b-a3b

    2026

  17. [25]

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. ZeRO: Memory optimizations Toward Training Trillion Parameter Models. InSC20: International Conference for High Per- formance Computing, Networking, Storage and Analysis. IEEE, 1–16. doi:10.1109/sc41405.2020.00024

    2020 arXiv

  18. [26]

    Xingyu Shen, Yingfa Chen, Zhen Leng Thai, Xu Han, Zhiyuan Liu, and Maosong Sun. 2026. StateX: Enhancing RNN Recall via Post-training State Expansion. arXiv:2509.22630 [cs.CL]https://arxiv.org/abs/2509. 22630

    2026 arXiv

  19. [27]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2020. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv:1909.08053 [cs.CL]https://arxiv.org/abs/1909.08053

    2020 arXiv

  20. [28]

    Ao Sun, Weilin Zhao, Xu Han, Cheng Yang, Xinrong Zhang, Zhiyuan Liu, Chuan Shi, and Maosong Sun. 2025. Seq1F1B: Efficient Sequence- Level Pipeline Parallelism for Large Language Model Training. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the...

    2025

  21. [29]

    Weigao Sun, Disen Lan, Yiran Zhong, Xiaoye Qu, and Yu Cheng. 2025. LASP-2: Rethinking Sequence Parallelism for Linear Attention and Its Hybrid. arXiv:2502.07563 [cs.LG]https://arxiv.org/abs/2502.07563

    2025 arXiv

  22. [30]

    Zhenbo Sun, Shengqi Chen, Yuanwei Wang, Jian Sha, Guanyu Feng, and Wenguang Chen. 2025. MEPipe: Democratizing LLM Training with Memory-Efficient Slice-Level Pipeline Scheduling on Cost-Effective Accelerators. InProceedings of the Twentieth European Conference on Computer Syste...

    2025

  23. [31]

    Xindi Wang, Mahsa Salmani, Parsa Omidi, Xiangyu Ren, Mehdi Reza- gholizadeh, and Armaghan Eshaghi. 2024. Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models. arXiv:2402.02244 [cs.CL]https://arxiv.org/abs/2402.02244

    2024 arXiv

  24. [32]

    Songlin Yang, Jan Kautz, and Ali Hatamizadeh. 2025. Gated Delta Networks: Improving Mamba2 with Delta Rule. InThe Thirteenth In- ternational Conference on Learning Representations.https://openreview. net/forum?id=r8H7xhYPwz

    2025

  25. [33]

    Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. 2024. Gated Linear Attention Transformers with Hardware- Efficient Training. InProceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 235), Rusla...

    2024

  26. [34]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. InThe Eleventh International Conference on Learning Representations.https://openreview.net/forum?id=WE_ vluYUL-X

    2023

  27. [35]

    Yuze Zhao, Jintao Huang, Jinghan Hu, Xingjun Wang, Yunlin Mao, Daoze Zhang, Zeyinzi Jiang, Zhikai Wu, Baole Ai, Ang Wang, Wen- meng Zhou, and Yingda Chen. 2025. SWIFT: A Scalable Light- weight Infrastructure for Fine-Tuning.Proceedings of the AAAI Conference on Artificial Inte...

    2025 doi

  28. [4901]

    doi:10.18653/v1/2023.emnlp-main.298

    2023 doi

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.