Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

SlimPipe: Memory-Thrifty and Efficient Pipeline Parallelism for Long-Context LLM Training

T0 review · 3 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SlimPipe claims slice-level pipeline scheduling makes long-context activation memory shrink with pipeline size while context exchange removes load-imbalance bubbles.

desk verdict SlimPipe's slice-level 1F1B schedule is a real new idea with a plausible memory model, but the 1.57x MFU claims rest on unverifiable closed-source benchmarks, so the paper should be reviewed conditionally on code release. read the letter →

arxiv 2504.14519 v1 pith:R2M44POH submitted 2025-04-20 cs.LG cs.AI

Zhouyang Li , Yuliang Liu , Wei Zhang , Tailing Yuan , Bin Chen , Chengru Song , Di Zhang This is my paper · ORCID
classification cs.LGcs.AI
keywords pipelineparallelismlong-contextLLMtrainingactivationmemoryreductionuniformsequenceslicing1F1Bscheduleattentioncontextexchangecausalimbalancevocabulary
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SlimPipe aims to remove the activation-memory wall that pipeline parallelism hits in long-context LLM training. The paper's central claim is that splitting each input sequence into many equal slices and running the pipeline at slice granularity with a one-forward-one-backward schedule makes accumulated activation memory roughly one slice per device rather than one full microbatch per device, scaling as $M_a/p$ with the pipeline size $p$. Causal attention makes later slices heavier, and the paper says this imbalance is eliminated by exchanging part of the attention context among pipeline devices and merging results with online softmax, with communication volume bounded near $2LM_h$ per microbatch. It further balances the output layer by sharding the vocabulary GEMM across all pipeline devices. If these claims hold, long-context training can use more pipeline stages to reduce memory rather than suffer constant activation memory, while keeping idle pipeline bubbles small; the paper reports above 45% MFU for a 70B model at 2048K context on 256 80GB GPUs and up to 1.57x throughput over strong baselines at 512K.

What carries the argument

The central object is the slice: an equal-length segment of the input sequence that replaces the microbatch as the atomic unit of pipeline work. Uniform slicing keeps the accumulated activation footprint bounded and makes the 1F1B schedule memory-stable, but causal attention makes later slices cost more compute; the load-bearing mechanism that fixes this is attention context exchange, where a device with more key-value context sends a query and part of its keys and values to a device with less context, receives the partial attention output, and merges it with the local result via online softmax. Vocabulary parallelism then distributes the output-layer GEMM and embedding across pipeline devices so the last stage does not become a memory and compute hotspot. Together these make the pipeline both memory-scalable and nearly bubble-free.

What would settle it

Reproduce the 70B 2048K configuration on 256 80GB GPUs with identical model, batch, and parallelism settings, and log peak activation memory per stage and per-device idle time in the steady phase; the claim fails if peak memory does not track $(1+2(p-1)/n)M_a/p$ as $p$ grows, or if attention-context exchange leaves per-device idle time above the one-key-value-slice imbalance bound.

Watch

Extended reading notes

Core claim

On its own terms, SlimPipe establishes that pipeline-parallel activation memory need not be constant in the pipeline size. By chopping every sequence into $n$ uniform slices and scheduling forward and backward passes at slice granularity in 1F1B order, the warm-up phase accumulates at most $M_{\text{acc}} = (1+\delta)M_a/p$ with $\delta = 2(p-1)/n$, approaching $M_a/p$ as $n$ grows, and warm-up and cool-down bubbles shrink by about a factor of $n$. Because causal attention makes later slices more expensive, SlimPipe exchanges part of each device's attention context with a less-loaded device, computing partial attention remotely and merging via online softmax, leaving per-device workloads within one key-value slice of balance. It also parallelizes the output-layer GEMM along the vocabulary dimension across all pipeline devices. The paper argues that this combination delivers near-zero memory overhead and minimal pipeline bubbles simultaneously, supported by measured MFU gains across dense and MoE models.

Load-bearing premise

The memory and bubble formulas assume slicing a sequence into $n$ equal pieces does not change total useful compute and that the attention-context exchange can be overlapped or kept small enough that every pipeline device stays busy; if that overlap fails, the reported speedups shrink accordingly.

Editorial extensions

If this is right

  • Activation memory becomes an inverse function of pipeline size: with $n$ equal slices and $p$ pipeline ranks, accumulated activation memory is $(1 + 2(p-1)/n) M_a / p$, approaching $M_a/p$ as the number of slices grows, so adding pipeline stages directly buys activation headroom.
  • Warm-up and cool-down bubbles shrink by about a factor of $n$, and further by the number of interleaved stages $v$, so training can stay efficient with as few as two microbatches, the regime forced by long context and critical batch sizes.
  • The context-exchange volume is bounded by about $2 L M_h$ per microbatch, essentially independent of $p$ and $n$, so load balancing does not add communication that grows with pipeline size.
  • In the reported measurements, this translates to up to 1.57x MFU over the strongest pipeline baseline at 512K context, and over 45% MFU for a 70B model at 2048K on 256 80GB GPUs; with pipeline-aware offloading, contexts reach 4096K on a Mixtral-class model.
  • Because SlimPipe distributes activations as well as model states, it avoids full activation checkpointing in many long-context settings, keeping arithmetic intensity high.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension beyond the paper: apply the same slice-level context exchange to long-context inference prefill, where causal-attention imbalance also appears, and measure whether the one-key-value-slice balance bound holds at serving batch sizes.
  • The memory formula suggests a design rule of choosing $n$ proportional to $p$ (the paper uses values like $n=4p$ in its largest runs); an independent sweep of arithmetic intensity versus bubble fraction could sharpen this rule across different attention-to-MLP compute ratios.
  • If the overlap assumption on context exchange holds on clusters with slower interconnects, SlimPipe-style scheduling would let pipeline parallelism substitute for context parallelism as the primary long-context axis, an inference the paper does not directly test.
  • The reported 1.57x is on a 512K MoE configuration; a synthetic ablation that varies only the exchange-overlap ratio would isolate how much of the gain comes from memory savings versus bubble elimination, which the aggregate MFU comparison does not separate.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes SlimPipe, a fine-grained pipeline-parallelism scheme that splits each microbatch into uniform sequence slices and schedules them with a 1F1B schedule, reducing accumulated activation memory and warm-up bubble time. To counteract the workload imbalance caused by causal attention, SlimPipe introduces an attention context exchange mechanism that redistributes attention computation across pipeline devices, and it distributes the vocabulary/output layer across PP ranks. The paper derives formulas for activation memory scaling (Eq. 1) and context-exchange communication volume (Eq. 2), and reports empirical MFU gains over Megatron-LM and DeepSpeed, including a 1.57x speedup on Mixtral 8x7B at 512K context and 45.0% MFU on Llama 70B at 2048K context on 256 GPUs.

Significance. The core idea is timely and potentially significant: training long-context LLMs is memory-bound, and a slice-level schedule that makes activation memory scale inversely with PP size, while balancing causal-attention load, could be an important building block. The paper's strength is a clean theoretical model for activation memory (Eq. 1), supported by the memory experiment in Figure 10, and the context-exchange scheme is a novel mechanism for balancing causal-attention work. However, the headline throughput claims rest on a closed-source Megatron-LM fork with grid-searched configurations that are not disclosed, and the communication cost of context exchange is bounded only by a volume expression, not demonstrated to be overlapped. If the comparisons are made fair and the communication behavior is measured, the contribution would be solid and likely to be adopted by the community.

major comments (3)
  1. [Section 6.4, Figures 12 and Table 4] The central MFU claims (up to 1.57x over Megatron-LM and 45.0% on Llama 70B at 2048K) are not reproducible because no code, configuration files, raw logs, or per-benchmark winning grid-search configurations are provided. The baselines are not stock systems: Section 6.4 says activation-saving techniques from Section 5 are applied uniformly to both systems and that full/selective checkpointing is enabled on their demand, so the reported speedups could reflect baseline configuration deficiencies rather than the slice-level schedule itself. Please release the code and configuration files, or at minimum provide the winning hybrid-parallelism configuration for every benchmark point and the raw measured MFU for all configurations in the grid.
  2. [Sections 4.2.3 and 5 (Early Key-Value Exchange)] Equation (2) bounds the volume of context-exchange communication, but the paper asserts that this communication is overlapped with computation without providing any measured communication time, overlap ratio, or comparison of communication time to compute time. The theoretical bubble-fraction formulas in Section 4.1.3 and Table 2 do not include communication cost, so the claimed 'minimal pipeline bubbles' are established only under an implicit overlap assumption that is not validated. Please provide communication time measurements (e.g., fraction of step time, overlap efficiency) for the context exchange in the configurations of Figure 12, and clarify whether the formulas in Table 2 are intended to include communication overhead.
  3. [Section 6.6, Figures 13 and 14] The ZB-V and V-Half baselines are run with float16 instead of bfloat16 and with a full-checkpointing implementation that the authors state 'does not work properly' for ZB-V, causing early OOM. This does not establish that SlimPipe outperforms a properly configured state-of-the-art ZB-V or V-Half; the claim that SlimPipe is superior across all context lengths in Figure 13 is therefore not supported by these experiments. Please either fix the baseline implementations, use the same precision for all schemes, or remove/replace this comparison.
minor comments (6)
  1. [Figure 10] The markers and curves are very small; consider adding error bars or reporting the number of repeated runs, and fix the corrupted unicode labels on the y-axis (e.g., '//u1D45D').
  2. [Table 2] The bubble-fraction expressions are hard to interpret; please define all variables (n, m, v, p) in the caption and state explicitly whether communication time is included in the formulas.
  3. [Figure 12] The legend contains the typo 'No Con/f_iguration'; please clarify the meaning of the green triangle and red cross markers and ensure the legend matches the text description.
  4. [Section 6.6] The statement that float16 was used 'in place of bfloat16, for the latter is not yet supported' is surprising for Hopper GPUs, which support bfloat16; please clarify what is not supported and whether this affects the comparison.
  5. [Section 4.2.3] The inequality in Eq. (2) is asserted without derivation; please show how the floor terms lead to the stated upper bound, since the bound is used to argue that the communication volume is almost independent of p and n.
  6. [Section 5] The term 'Commutated Context Parallelism' is unclear; if it is a variant of the described query/output-exchanging CP, please rename it to avoid confusion with 'communicated' or 'commutative'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SlimPipe's memory and bubble claims are accounting identities tested empirically; benchmark concerns are reproducibility issues, not circularity.

full rationale

The core derivations are schedule-accounting identities, not fitted predictions. Equation (1), M_acc = (1+delta) M_a/p with delta = 2(p-1)/n, follows from counting how many slice activations each pipeline device retains under the 1F1B schedule with uniform slicing, and Equation (2) bounds the context-exchange volume by counting Q/O/K/V slices. Neither formula contains a parameter fitted to the claims it supports. Figure 10 tests Equation (1) against measured torch.cuda.max_memory_allocated values using a memory model taken from [19,48]; the measured alignment is empirical support, not a definitional restatement. The MFU comparisons in Figures 12-13 and Table 4 are end-to-end measurements; concerns that the custom Megatron-LM fork and closed configurations make them hard to reproduce are correctness and verifiability concerns, not circularity. Reference [48] is authored by overlapping Kuaishou authors and is used for the offloading and memory-modeling components, but those components are externally falsifiable techniques from a peer-reviewed ATC paper and do not carry the paper's central scheduling or memory-scaling derivation.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new physical entities are introduced. The free parameters are system-level knobs (slice count, offload ratio, checkpointing choices) that are tuned per experiment. The main axioms are assumptions about compute proportionality for causal attention and about communication overlap. The paper does not ship code, so the axiom that the implemented system actually realizes the idealized schedule cannot be checked.

free parameters (3)
  • number of slices n = typically 2p or 4p in benchmarks; the paper notes 'moderate' values are best
    The slice count is a hand-tuned knob. The paper itself shows (Figure 11) that too many slices hurt arithmetic intensity and too few hurt bubble reduction, so the value used in each benchmark is a tuned choice.
  • per-model offload ratio = 75%, 80%, 95%, 100% in Table 4
    The ultra-long-context experiments rely on offloading a tunable fraction of activations to host memory. The paper says the ratio is 'adaptive', but does not give a rule for choosing it, so it is a free parameter fitted per model and context.
  • selective checkpointing on demand = full or selective, decided per configuration
    The end-to-end comparison enables checkpointing 'on their demand', which is a per-benchmark choice that changes both memory and throughput. The choice is not reported per configuration.
assumptions (4)
  • domain assumption Uniform slicing keeps total useful compute constant, and per-slice compute is proportional to the number of attended key-value tokens under causal attention.
    Used throughout Section 4.2 to derive the imbalance and the redistribution benefit. It ignores the marginal overhead of shorter GEMMs and kernel launches, which the paper acknowledges in Figure 11 by saying arithmetic intensity drops.
  • domain assumption The online-softmax merge of remotely computed attention outputs is numerically equivalent to exact attention and can be done at negligible cost.
    Invoked in Section 4.2.2 for the context exchange. It assumes the merge and renormalization overhead is small enough not to change the bubble analysis.
  • domain assumption Communication of exchanged context can be overlapped with compute (early key-value exchange) so that the bounded communication volume in Equation 2 does not become the bottleneck.
    Section 5 and Equation 2 argue the volume is bounded by about 2L M_h, but the paper does not measure the exposed communication latency in the end-to-end numbers. The claim of 'minimal pipeline bubbles' depends on this overlap being effective.
  • ad hoc to paper The memory model from reference [48] (by overlapping authors) correctly predicts activation sizes without SlimPipe.
    Used in Section 6.2 to draw the theoretical curves M_t/p. This is a self-cited external baseline with no code or validation supplied here.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SlimPipe: Memory-Thrifty and Efficient Pipeline Parallelism for Long-Context LLM Training." pith.science (2026). https://pith.science/paper/R2M44POH

@misc{pith2026250414519,
  author       = {Pith},
  title        = {Pith review of: SlimPipe: Memory-Thrifty and Efficient Pipeline Parallelism for Long-Context LLM Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R2M44POH}},
  note         = {Machine review of arXiv:2504.14519}
}
abstract

Pipeline Parallelism (PP) serves as a crucial technique for training Large Language Models (LLMs), owing to its capability to alleviate memory pressure from model states with relatively low communication overhead. However, in long-context scenarios, existing pipeline parallelism methods fail to address the substantial activation memory pressure, primarily due to the peak memory consumption resulting from the accumulation of activations across multiple microbatches. Moreover, these approaches inevitably introduce considerable pipeline bubbles, further hindering efficiency. To tackle these challenges, we propose SlimPipe, a novel approach to fine-grained pipeline parallelism that employs uniform sequence slicing coupled with one-forward-one-backward (1F1B) schedule. It reduces the accumulated activations from several microbatches to just one, which is split into several slices. Although the slices are evenly partitioned, the computation cost is not equal across slices due to causal attention. We develop a sophisticated workload redistribution technique to address this load imbalance. SlimPipe achieves (1) near-zero memory overhead and (2) minimal pipeline bubbles simultaneously. The effectiveness of SlimPipe has been proven by thorough testing with diverse model architectures, context window sizes, and SlimPipe-specific configurations. For example, on the Llama 70B model, compared to state-of-the-art methods, SlimPipe significantly boosts the Model FLOPs Utilization (MFU) to up to $1.57\times$ for a context length of 512K. More notably, for a context length of 2048K, it maintains over 45% utilization on 256 NVIDIA Hopper 80GB GPUs, while other approaches either suffer significant performance drops or fail entirely due to memory constraints.

Figures

Figures reproduced from arXiv: 2504.14519 by the authors.

Figure 1
Figure 1. Comparison of GPU memory footprint between [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Theoretical bub￾ble fractions of different PP schemes (PP size 8) in train￾ing Llama 13B with 4 micro￾batches and a 256K context length. relatively low communication overhead. However, existing PP ap￾proaches exhibit significant limitations in handling the memory pressure caused by activations, which scales with context lengths. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p001_3.png] view at source ↗
Figure 4
Figure 4. The top figure shows the default 1F1B schedule. The bottom figure shows the SlimPipe schedule, where each microbatch [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figures from the paper (9 more)
Figure 5
Figure 5. Figure 5: SlimPipe in its interleaving form. Each device is assigned 2 stages. Dark colors show the first stage and light colors [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 8
Figure 8. Figure 8: The attention workloads are rebalanced by exchang [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]
Figure 7
Figure 7. Figure 7: A closer look at the SlimPipe timeline with imbal [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 9
Figure 9. Figure 9: A SlimPipe timeline segment containing the output [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]
Figure 11
Figure 11. Figure 11: MFU of training the Llama 13B model with dif [PITH_FULL_IMAGE:figures/full_fig_p008_11.png]
Figure 10
Figure 10. Figure 10: The memory reduced by the PP size 𝑝. The mea￾sured values align well with our theoretical model. We use a simple experiment to show how SlimPipe reduces the memory usage in training. We train the Llama 13B model with input sequence lengths of 32K, 64K and 96K. The TP …
Figure 12
Figure 12. Figure 12: System performance comparison between DeepSpeed, Megatron-LM, and SlimPipe across different models and [PITH_FULL_IMAGE:figures/full_fig_p009_12.png]
Figure 13
Figure 13. Figure 13: Comparison of MFU across different PP schemes. [PITH_FULL_IMAGE:figures/full_fig_p010_13.png]
Figure 14
Figure 14. Figure 14: Comparison of GPU memory usage across differ [PITH_FULL_IMAGE:figures/full_fig_p010_14.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. OctoPipe: Reducing Pipeline Bubbles for Heterogeneous Models via Co-Optimizing Partitioning, Placement, and Scheduling

    cs.DC 2025-09 reject novelty 6.0 of 10

    Co-optimizing model partition, placement, and workload scheduling for pipeline-parallel LLM training is claimed to improve throughput by 1.15 to 1.44x (abstract) or up to 2.14x (body).

Reference graph

Works this paper leans on

52 extracted references · 14 canonical work pages · cited by 1 Pith paper

  1. [1]

    Joshua Ainslie, James Lee-Thorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. 2023. Gqa: Training generalized multi-query trans- former models from multi-head checkpoints. arXiv preprint arXiv:2305.13245 (2023)

    arXiv 2023

  2. [2]

    Olivier Beaumont, Lionel Eyraud-Dubois, Julien Hermann, Alexis Joly, and Alena Shilova. 2019. Optimal checkpointing for heterogeneous chains: how to train deep neural networks with limited memory. arXiv preprint arXiv:1911.13214 (2019)

    arXiv 2019

  3. [3]

    William Brandon, Aniruddha Nrusimha, Kevin Qian, Zachary Ankner, Tian Jin, Zhiye Song, and Jonathan Ragan-Kelley. 2023. Striped attention: Faster ring attention for causal transformers. arXiv preprint arXiv:2311.09431 (2023)

    arXiv 2023

  4. [4]

    Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, J

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, J. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. Henighan, R. Child, A. Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Ma teusz Litwin, S. Gray, B....

    work page 2020

  5. [5]

    Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. 2016. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174 (2016)

    arXiv 2016

  6. [6]

    Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catanzaro, and Evan Shelhamer. 2014. cuDNN: Efficient primitives for deep learning. arXiv preprint arXiv:1410.0759 (2014)

    arXiv 2014

  7. [7]

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Flashat- tention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems 35 (2022), 16344–16359

    2022

  8. [8]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ah- mad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sra- vankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bet...

    work page

Show all 52 references
  1. [9]

    Shiqing Fan, Yi Rong, Chen Meng, Zongyan Cao, Siyu Wang, Zhen Zheng, Chuan Wu, Guoping Long, Jun Yang, Lixue Xia, et al. 2021. DAPPLE: A pipelined data parallel approach for training large models. In Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of P...

    2021

  2. [10]

    William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research 23, 120 (2022), 1–39

    2022

  3. [11]

    Trevor Gale, Deepak Narayanan, Cliff Young, and Matei Zaharia. 2023. Megablocks: Efficient sparse training with mixture-of-experts. Proceedings of Zhouyang Li et al. Machine Learning and Systems 5 (2023), 288–304

    2023

  4. [12]

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al . 2019. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems 32 (2019)

    2019

  5. [13]

    Changho Hwang, Wei Cui, Yifan Xiong, Ziyue Yang, Ze Liu, Han Hu, Zilong Wang, Rafael Salas, Jithin Jose, Prabhat Ram, et al. 2023. Tutel: Adaptive mixture- of-experts at scale. Proceedings of Machine Learning and Systems 5 (2023), 269– 287

    2023

  6. [14]

    Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuai- wen Leon Song, Samyam Rajbhandari, and Yuxiong He. 2023. Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models. arXiv preprint arXiv:2309.14509 (2023)

    2023 arXiv

  7. [15]

    Gonzalez

    Paras Jain, Ajay Jain, Aniruddha Nrusimha, Amir Gholami, Pieter Abbeel, Kurt Keutzer, Ion Stoica, and Joseph E. Gonzalez. 2020. Checkmate: Breaking the memory wall with optimal tensor rematerialization. In Proceedings of Machine Learning and Systems, Vol. 2. 497–511

    2020

  8. [16]

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al . 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088 (2024)

    2024 arXiv

  9. [17]

    Taebum Kim, Hyoungjoo Kim, Gyeong-In Yu, and Byung-Gon Chun. 2023. BPIPE: memory-balanced pipeline parallelism for training large language models. In International Conference on Machine Learning . 16639–16653

    2023

  10. [18]

    Marisa Kirisame, Steven Lyubomirsky, Altan Haan, Jennifer Brennan, Mike He, Jared Roesch, Tianqi Chen, and Zachary Tatlock. 2020. Dynamic Tensor Rematerialization. arXiv preprint arXiv:2006.09616 (2020)

    2020 arXiv

  11. [19]

    Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. 2023. Reducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems 5 (2023), 341–353

    2023

  12. [20]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Princip...

    2023

  13. [21]

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668 (2020)

    2020 arXiv

  14. [22]

    Zhuohan Li, Siyuan Zhuang, Shiyuan Guo, Danyang Zhuo, Hao Zhang, Dawn Song, and Ion Stoica. 2021. Terapipe: Token-level pipeline parallelism for training large-scale language models. In International Conference on Machine Learning . 6543–6552

    2021

  15. [23]

    Hao Liu, Matei Zaharia, and Pieter Abbeel. 2023. Ring attention with blockwise transformers for near-infinite context. arXiv preprint arXiv:2310.01889 (2023)

    2023 arXiv

  16. [24]

    Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAI Dota Team. 2018. An empirical model of large-batch training. arXiv preprint arXiv:1812.06162 (2018)

    2018 arXiv

  17. [25]

    Maxim Milakov and Natalia Gimelshein. 2018. Online normalizer calculation for softmax. arXiv preprint arXiv:1805.02867 (2018)

    2018 arXiv

  18. [26]

    Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, and Matei Zaharia. 2019. PipeDream: Generalized pipeline parallelism for DNN training. In Proceedings of the 27th ACM symposium on operating systems princip...

    2019

  19. [27]

    Deepak Narayanan, Amar Phanishayee, Kaiyu Shi, Xie Chen, and Matei Zaharia

  20. [28]

    Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. 2021. Efficient large-scale language model training on gpu clusters using megatron-lm. In Pro...

    2021

  21. [29]

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing system...

    2019

  22. [30]

    Patil, Paras Jain, Prabal Dutta, Ion Stoica, and Joseph E

    Shishir G. Patil, Paras Jain, Prabal Dutta, Ion Stoica, and Joseph E. Gonzalez

  23. [31]

    Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Brad- bury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. 2023. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems 5 (2023), 606–624

    2023

  24. [32]

    Ofir Press and Lior Wolf. 2016. Using the output embedding to improve language models. arXiv preprint arXiv:1608.05859 (2016)

    2016 arXiv

  25. [33]

    Penghui Qi, Xinyi Wan, Nyamdavaa Amar, and Min Lin. 2024. Pipeline Paral- lelism with Controllable Memory. arXiv:2405.15362 [cs.LG] https://arxiv.org/ abs/2405.15362

    2024 arXiv

  26. [34]

    Penghui Qi, Xinyi Wan, Guangxing Huang, and Min Lin. 2024. Zero Bubble (Almost) Pipeline Parallelism. InThe Twelfth International Conference on Learning Representations

    2024

  27. [35]

    Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. 2022. Deepspeed- moe: Advancing mixture-of-experts inference and training to power next- generation ai scale. InInternational conference on machine...

    2022

  28. [36]

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. In SC20: Inter- national Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 1–16

    2020

  29. [37]

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deep- speed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining ...

    2020

  30. [38]

    Minsoo Rhu, Natalia Gimelshein, Jason Clemons, Arslan Zulfiqar, and Stephen W Keckler. 2016. vDNN: Virtualized deep neural networks for scalable, memory- efficient neural network design. In 2016 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO) . IEEE, 1–13

    2016

  31. [39]

    David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. 1986. Learning representations by back-propagating errors. nature 323, 6088 (1986), 533–536

    1986

  32. [40]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-LM: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053 (2019)

    2019 arXiv

  33. [41]

    SB Shriram, Anshuj Garg, and Purushottam Kulkarni. 2019. Dynamic memory management for GPU-based training of deep neural networks. In 2019 IEEE International Parallel and Distributed Processing Symposium (IPDPS) . 200–209

    2019

  34. [42]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lil...

    2024 arXiv

  35. [43]

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupati- raju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295 (2024)

    2024 arXiv

  36. [44]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lam- ple. 2023. LLaMA: Open and efficient foundation ...

    2023 arXiv

  37. [45]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yas- mine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy F...

    2023 arXiv

  38. [46]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017)

    2017

  39. [47]

    Xing, Qirong Ho, Wei Dai, Jin Kyu Kim, Jinliang Wei, Seunghak Lee, Xun Zheng, Pengtao Xie, Abhimanu Kumar, and Yaoliang Yu

    Eric P. Xing, Qirong Ho, Wei Dai, Jin Kyu Kim, Jinliang Wei, Seunghak Lee, Xun Zheng, Pengtao Xie, Abhimanu Kumar, and Yaoliang Yu. 2015. Petuum: A New Platform for Distributed Machine Learning on Big Data. IEEE Transactions on Big Data 1, 2 (2015), 49–67. doi:10.1109/TBDATA.2...

    2015

  40. [48]

    Tailing Yuan, Yuliang Liu, Xucheng Ye, Shenglong Zhang, Jianchao Tan, Bin Chen, Chengru Song, and Di Zhang. 2024. Accelerating the Training of Large Language Models using Efficient Activation Rematerialization and Optimal Hy- brid Parallelism. In 2024 USENIX Annual Technical C...

    2024

  41. [49]

    Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. 2022. St-moe: Designing stable and transfer- able sparse expert models. arXiv preprint arXiv:2202.08906 (2022)

    2022 arXiv

  42. [2021]

    InInternational Conference on Machine Learning

    Memory-efficient pipeline-parallel dnn training. InInternational Conference on Machine Learning. PMLR, 7937–7947

  43. [2022]

    In Proceedings of the 39th International Conference on Machine Learning, Vol

    POET: Training Neural Networks on Tiny Devices with Integrated Rema- terialization and Paging. In Proceedings of the 39th International Conference on Machine Learning, Vol. 162. 17573–17583

  44. [2024]

    arXiv preprint arXiv:2407.21783 (2024)

    The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783 (2024)

    2024 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.