Pith. sign in

REVIEW 3 major objections 5 minor 62 references

SmartSwap: Swap-Based Memory Optimization for LLM Training under Varying Operator Sequences

T0 review · 3 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proposes Chameleon, the first swap-based memory optimizer for Eager Mode LLM training that adapts to varying operator sequences, enabling models up to 4x larger than device memory.

desk verdict Solid systems contribution that addresses a real gap—varying operator sequences in eager-mode swap—but the core timing assumption is validated only on Llama2 and needs broader stability data before I trust the generalization. read the letter →

arxiv 2509.11076 v2 pith:45RC636L submitted 2025-09-14 cs.DC

Zibo Wang , Yuhang Zhou , Zhibin Wang , Shipeng Li , Xinjing Huang , Chendong Cai , Bingxu Mu , Yuqing Sun
show 8 more authors
Zhiheng Hu Bin She Shu You Guanghuan Fang Rong Gu Wanchun Dou Guihai Chen Chen Tian
This is my paper
classification cs.DC
keywords swap-basedmemoryoptimizationLLMtrainingEagerModeoperatorsequencevariationonlineprofilingoffloadingpolicygenerationcross-streamsynchronization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Chameleon is a swap-based memory optimization system for LLM training in Eager Mode frameworks, aimed at the common situation where operator sequences change from iteration to iteration due to loss scaling, on-the-fly validation, or conditional branches. The paper claims that existing swap methods, which assume a static computation graph, break under such changes, and that Chameleon is the first to systematically address them. It does so with a lightweight online profiler that detects sequence changes with 0.9% steady-state overhead, a policy generator that estimates operator-group timings without per-operator profiling and simulates swap schedules, and an executor that matches operators and tensors across iterations and uses a custom cross-stream synchronization. If correct, Chameleon lets models up to 4x larger than hardware memory train without out-of-memory errors, replaces recomputation in many settings, and improves iteration time by up to 38.94% compared to recomputation or higher-degree parallelism.

What carries the argument

The central mechanism is the pairing of a lightweight online profiler with a logical-layer timing model and a simulator. The profiler encodes operator sequences as integer tensors and switches stages (WarmUp, GenPolicy, Stable) based on length and cosine-similarity thresholds. The policy generator evenly groups operators into logical layers—evenly sized groups of operators used as timing units—estimates each group's duration by T_group = (T_iter / N_iter) * N_group, builds a memory reduction list and candidate list to choose which tensors to swap, and runs a simulator to determine when to pre-trigger swap-ins and when swap-outs complete. The executor applies the resulting policy through mult

What would settle it

Run Chameleon on a model with heterogeneous group execution times (for example, a Mixture-of-Experts model where different experts are active per token) and compare actual swap-in completion times against the schedule predicted by the grouping-based timing estimate; if the coefficient of variation of group times is far from zero, swap-in pre-triggering will miss deadlines, producing compute stalls or OOM.

Watch

Extended reading notes

Core claim

The paper argues that swap-based memory optimization can be made reliable in Eager Mode by treating operator-sequence change as a first-class event. Concretely, Chameleon continuously monitors operator sequences with a low-overhead profiler, regenerates swap policies when the sequence shifts significantly, and applies those policies through fuzzy operator/tensor matching rather than relying on persistent unique identifiers. The central mechanism is a logical-layer timing model: the operator sequence is split into evenly sized groups, each group's duration is estimated from the per-iteration time, and a simulator schedules pre-triggered swap-ins and computes swap-out completion times from a g

Load-bearing premise

The policy generator assumes that evenly grouped operators have similar total execution times per group—validated only on a 32-layer Llama2 model and generalized by belief to other LLMs—so if group times are heterogeneous, pre-triggered swap-ins will be mis-scheduled, causing stalls or out-of-memory errors.

Editorial extensions

If this is right

  • Models exceeding hardware memory by up to 4x along batch size or sequence length can be trained on fewer accelerators, reducing communication overhead and hardware cost.
  • Swap can substitute for full activation recomputation, removing redundant forward computation from the critical path and improving iteration time.
  • Operator-sequence changes such as loss-scale updates, validation runs, or conditional branches no longer crash training or require manual policy regeneration.
  • The steady-state profiling overhead of 0.9% makes continuous online monitoring practical for production training.
  • The design's hook point at operator dispatch is portable to other Eager Mode frameworks that expose a similar dispatch mechanism.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: The logical-layer timing assumption could be tested directly on Mixture-of-Experts models, where expert routing creates uneven per-group execution times; if group-time variance is high, the policy generator would need per-group calibration to avoid mis-scheduled swap-ins.
  • Inference: The 4x scaling figures were demonstrated on a specific 64 GB HBM NPU with transformer-style models; extrapolation to other hardware or to heavy non-layer-structured models is plausible only insofar as the grouping regularity holds.
  • Inference: The fuzzy-matching and simulator components could generalize beyond training to serve as a runtime memory manager for dynamic inference workloads with variable control flow.
  • Inference: The claimed 84.25% profiling-overhead reduction is relative to a heavyweight built-in profiler; a comparison against a lean streaming profiler might yield a different number, but the online, non-stalling design remains the core contribution.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents Chameleon, a swap-based memory optimization system for training LLMs in Eager Mode, where operator sequences can vary across iterations. Chameleon consists of a lightweight online profiler that monitors operator sequences and detects changes, a policy generator that estimates group-level execution times and uses a simulator to schedule swap-in/swap-out operations, and an executor that applies policies via multi-feature fuzzy matching and a custom recordStream mechanism. The evaluation on Ascend 910B NPUs reports 0.9% steady-state profiling overhead, successful adaptation to sequence changes such as loss scaling and on-the-fly validation, 5,000-step correctness validation on Llama2, and the ability to train models up to 4× larger than hardware memory, with up to 38.94% speedup over recomputation or high-degree parallelism baselines.

Significance. If the results hold, Chameleon addresses a real and timely gap: existing swap-based memory optimizers assume fixed operator sequences, while Eager Mode training routinely encounters dynamic sequences. The system is substantial (8,700+ lines, deployed in production), and the paper contains several concrete strengths: profiling overhead is measured against the built-in profiler; the long-term stability experiment uses external ground truth (loss overlap with full recomputation) and explicitly includes loss scaling and on-the-fly validation; the custom recordStream is evaluated with a clear mechanism; and scalability is explored along multiple dimensions. The central correctness claim is supported for Llama2, but the load-bearing uniform-group-time assumption is validated only on a single architecture, which leaves the generality claim under-supported.

major comments (3)
  1. [§5.1, Eq. (1); §5.4.1; §5.4.2; Table 2] The policy generator's timing estimates rest on the assumption that evenly grouped operator sequences have similar total execution time per group, quantified by T_group = (T_iter / N_iter) * N_group. This is validated only on a 32-layer Llama2 (Fig. 4), and the paper states 'we believe this observation generalizes to other LLMs' without supporting data. The assumption is load-bearing: the simulator uses T_group to search backward for pre-triggered swap-in slots (§5.4.1) and to mark swap-out completion times (§5.4.2), which the custom recordStream (§6.2) uses to reclaim and reuse memory. On heterogeneous models such as Mixtral—which appears in Table 2 but is not tested for Eq. (1)—MoE routing or variable per-layer shapes can make group times heterogeneous. An underestimated group time can delay a required swap-in and stall computation; an overestimated group time can mark a swap-out compl
  2. [§7.4] The long-term stability experiment, which is the main evidence for 'no training errors,' is performed only on Llama2 for 5,000 steps on a single NPU. Llama2 is composed of repeated identical transformer layers, which is exactly the structure that makes Eq. (1) valid. The paper does not provide a correctness experiment for Mixtral or any heterogeneous model. Since the custom recordStream's memory reuse decision depends on simulated swap-out completion times, an architecture that violates the uniform-group-time assumption could fail silently. A targeted experiment varying group execution times (e.g., MoE, uneven layers, variable shapes) or measuring loss/step correctness on Mixtral would substantially strengthen the claim that Chameleon adapts to varying operator sequences without training errors.
  3. [§4, Algo. 1; §5.3, Eq. (2); §7.1] Several design parameters are empirically tuned on the Llama2 setup without sensitivity analysis: the stage transition thresholds m=2 and n=5 in Algo. 1, the 5%/95% change-detection thresholds, and the coefficient C in Eq. (2). These choices directly affect how quickly Chameleon reacts to sequence changes and how candidates are prioritized. The paper does not report how sensitive the measured benefits are to these values or how they should be set for a new model/framework. Adding ablations or at least a discussion of the parameter landscape would make the evaluation more robust and reproducible.
minor comments (5)
  1. [Figure 8] The figure labels are garbled: '/glyph1197umber', 'O i-PyT o ch', 'Ite ation time(s)', and 'Op /glyph1197um(×10000)' need to be fixed. The same issue appears in the caption references.
  2. [§7.2, Table 4] The text says 'up to 4×, 1.83×, 4×, and 1.24× along the three dimensions, respectively,' but four dimensions (batch size, layers, sequence length, hidden size) are listed. Also 'linear performance scaling to80/64 of the maximum' is unclear.
  3. [Throughout] There are numerous typos and formatting errors, e.g., 'Occurrs', 'reveales', 'profilng', 'naiverecordStream', 'aprofiling→ policy generation→ policy applicationworkflow'. A careful proofreading pass is needed.
  4. [Table 2] Performance benefits are reported without repetitions, standard deviations, or error bars. Given that the system runs in a production environment, reporting at least three runs for key configurations would increase confidence.
  5. [§7.1, n=5] The statement 'With n = 5, Chameleon generates five different policies and selects the one with the best runtime performance' suggests post-hoc selection over a small sample. This could overfit to the current iteration; please clarify whether the selection is based on a held-out criterion and whether it affects the reported benefits.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's headline results are empirical measurements against external baselines, and its swap-timing model is an explicit, stated assumption rather than a prediction derived from its own outputs.

full rationale

The derivation chain is self-contained. Profiling overhead (Table 1) is measured against the built-in PyTorch profiler; long-term correctness (Fig. 7) is checked against an external full-recompute loss curve; scalability and performance (Fig. 6, Tables 2-4) are measured against native PyTorch or recomputation baselines. The policy generator's timing model, Eq. (1) T_group = (T_iter / N_iter) * N_group, is an explicit assumption ('we believe this observation generalizes to other LLMs', §5.1) motivated by a Llama2 coefficient-of-variation measurement; it is not derived from the claimed results. The simulator uses this assumed timing to schedule swap pre-triggers and mark swap-out completion, and the custom recordStream trusts those marks, so an inaccurate T_group could cause stalls or premature memory reuse'on heterogeneous models such as Mixtral. That is an unvalidated premise and a correctness/robustness concern, not circularity: the paper does not redefine its output in terms of this assumption, and the assumption itself is stated as an input ('This assumption forms the foundation of our policy generator'). Hyperparameters (m, n, C, 5%/95% thresholds) are empirically set, and the n=5 best-policy selection tunes the system on the same workload family, which may inflate reported gains, but none of these is a fitted parameter renamed as a prediction. There are no load-bearing self-citations: cited works (Capuchin, MegTaiChi, SwapAdvisor, GMLake, etc.) are external. No step reduces, by the paper's own equations or by self-citation, to its own inputs, so no circular step can be quoted or exhibited.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The central design rests on a small set of empirical assumptions and tuned parameters. No new physical entities are postulated; 'logical layers' is a software abstraction. The main free parameters are the score coefficient C, stage thresholds m/n, and change-detection thresholds. The grouping assumption is the most consequential axiom.

free parameters (3)
  • C (candidate score coefficient in Eq. (2)) = not reported
    Balances normalized MRE coverage and tensor size when ranking swap candidates; affects which tensors are swapped and thus memory savings and overhead.
  • m and n (stage transition thresholds, Algo. 1) = m=2, n=5
    Determines how many stable iterations are needed before policy generation and before moving to Stable stage; empirically set in §7.1.
  • Sequence-change detection thresholds = 5% length diff, 95% cosine similarity
    Defines when an operator sequence change is significant enough to regenerate policy; empirically chosen in §4.
assumptions (6)
  • ad hoc to paper Evenly sized operator groups have low variance in total execution time, so average group time approximates individual group durations.
    Introduced in §5.1 as an insight from a 32-layer Llama2 profile; underpins Eq. (1) and all swap timing decisions.
  • ad hoc to paper Most modern LLMs are built by stacking similar structures, so the grouping observation generalizes.
    Stated in §5.1; extends the empirical finding beyond Llama2 without additional evidence.
  • domain assumption Operator sequences in Eager Mode change frequently in real training (loss scaling, validation, branches).
    Motivates the whole system; asserted in §2.3 with examples but no quantitative prevalence data.
  • domain assumption Host-device transfer bandwidth B is known and constant for swap time calculation (Eq. (3)).
    Used to compute T_swap = S / B; assumes stable PCIe bandwidth and that swap and compute overlap perfectly.
  • domain assumption PyTorch eager memory management semantics (host-side malloc/free, per-stream pools, ref counting) hold on the target platform.
    Basis of the memory reconstruction in §4 and the recordStream redesign in §6.2.
  • domain assumption GMLake memory pool can be used for defragmentation on the target NPU platform.
    Used in OOM handling (§6.3) without demonstrating compatibility on Ascend NPU.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SmartSwap: Swap-Based Memory Optimization for LLM Training under Varying Operator Sequences." pith.science (2026). https://pith.science/paper/45RC636L

@misc{pith2026250911076,
  author       = {Pith},
  title        = {Pith review of: SmartSwap: Swap-Based Memory Optimization for LLM Training under Varying Operator Sequences},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/45RC636L}},
  note         = {Machine review of arXiv:2509.11076}
}
read the original abstract

The increasing size of large language models (LLMs) has led to a surge in memory requirements during training, often exceeding the capacity of high-bandwidth memory (HBM). Swap-based memory optimization incurs neither accuracy loss nor additional end-to-end overhead when effectively overlapped, thus being an attractive solution. However, existing swap methods assume consistent operator sequences, which is impractical in Eager Mode, where operator sequences can vary during change. We propose Chameleon, which redesigns the end-to-end process of swap-based memory optimization and is the first work to consider varying operator sequences in Eager Mode. Chameleon (i) introduces a lightweight online profiler to enable continuous profiling for monitoring operator sequences, (ii) generates effective swap policies with limited operator information, and (iii) optimizes the policy execution module for accurate policy application and better performance. Experimental results demonstrate that Chameleon reduces profiling overhead by 84.25%, enables training models up to 4x larger than hardware memory while adapting to changes in operator sequences, improves performance by up to 38.94% compared to recomputation or high-degree parallelism.

Figures

Figures reproduced from arXiv: 2509.11076 by the authors.

Figure 1
Figure 1. Illustration of PyTorch Memory Management. host side. Each stream has its own memory pool, and mem￾ory cannot be reused across streams directly. This design leverages sequential execution within a stream to improve allocation efficiency. Although host and device progress asyn￾chronously, the device executes operators in the order dis￾patched by the host. This alignment eliminates the need for host-device synchroniza… view at source ↗
Figure 2
Figure 2. Overview of Chameleon and its workflow. Lightweight, which only monitors operator sequence, and Detailed, which collects comprehensive operator and tensor information for policy generation. In Eager Mode frame￾works, host and device operate at different paces, so col￾lecting the execution time for individual operators requires heavyweight hardware profiling. To reduce this overhead, we avoid capturing this informati… view at source ↗
Figure 3
Figure 3. Reconstruction of the actual memory usage. explain how we generate swap policy based solely on the op￾erator sequence and the duration of each training iteration, without relying on operator execution times. In addition to collecting operator- and tensor-related data, the profiler must also capture the amount of memory in use during each operator’s execution. Furthermore, when a swap operation occurs (whether it’s a… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: The relationship between the number of groups and (1) the CV of total execution time per group, and (2) the error of using the time calculated by Eq.(1) for each group. built by stacking similar structures, we believe this observa￾tion generalizes to other LLMs. This a…
Figure 5
Figure 5. Figure 5: (a), after dispatching a swap-out for T1, the pointer to the physical memory of T1 is changed from a device pointer to a host pointer, and the reference count of the device mem￾ory block drops to 0, freeing it. This block may be imme￾diately allocated to OP2 to store i…
Figure 6
Figure 6. Figure 6: Performance under batch size, sequence length, and hidden size scaling [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 8
Figure 8. Figure 8: Results of comparison experiment between the custom recordStream and the original recordStream. training. Using the Llama2 model, scaled to approximately 80GB of memory usage during training, we train the model for 5,000 steps on a single NPU with loss scaling and per￾…
Figure 7
Figure 7. Figure 7: Long-term stability experiment result. 7.4 Long-term Stability Experiment We conduct a long-term stability experiment to verify that Chameleon does not compromise the correctness of model 8     8  #   $ !      " " " !  $  #!"  "…
Figure 9
Figure 9. Figure 9: Illustration of optimized OOM Handling Process. including the very first. For the first iteration, Chameleon generates no swap policy, so FreeSwappingOutBlock() in line 3 frees nothing. Upon OOM recurrence, PassiveSwap() in line 7 is invoked to release enough space for…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

62 extracted references · 5 linked inside Pith

  1. [1]

    Murray, Benoit Steiner, Paul Tucker, Vijay Va- sudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng

    Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irv- ing, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Va- sudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: A system for L...

    2016

  2. [2]

    Tensorflow eager: A multi-stage, python-embedded dsl for machine learning

    Akshay Agrawal, Akshay Modi, Alexandre Passos, Allen Lavoie, Ashish Agarwal, Asim Shankar, Igor Ganichev, Josh Levenberg, Ming- sheng Hong, Rajat Monga, and Shanqing Cai. Tensorflow eager: A multi-stage, python-embedded dsl for machine learning. In A. Tal- walkar, V. Smith, and M. Zaharia, editors,Proceedings of Machine Learning and Systems, volume 1, pag...

    2019

  3. [3]

    Accessed: 2024-12

    Meta AI. Accessed: 2024-12. papers with code trends.https:// paperswithcode.com/trends

    2024

  4. [4]

    Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary DeVito, Elias Ellison, Will Feng, Jiong Gong, Michael Gschwind, Brian Hirsh, Sherlock Huang, Kshiteej Kalam- barkar, Laurent Kirsch, Michael...

    2024

  5. [5]

    Accessed: 2025-06

    Ascend. Accessed: 2025-06. ascend extension for pytorch > py- torch training > model migration and tuning > model migra- tion > model script migration > (recommended) automatic migra- tion.https://www.hiascend.com/document/detail/zh/Pytorch/700/ ptmoddevg/trainingmigrguide/PT_LMTMOG_0014.html

    2025

  6. [6]

    Accessed: 2025-06

    Ascend. Accessed: 2025-06. opcommand.cpp.https://gitee.com/ascend/ pytorch/blob/master/torch_npu/csrc/framework/OpCommand.cpp

    2025

  7. [7]

    Accessed: 2025-08

    Ascend. Accessed: 2025-08. ascendcl profiling api.https: //www.hiascend.com/document/detail/zh/canncommercial/82RC1/ devaids/Profiling/atlasprofiling_16_0042.html

    2025

  8. [8]

    B, Anshuj Garg, and Purushottam Kulkarni

    Shriram S. B, Anshuj Garg, and Purushottam Kulkarni. Dynamic mem- ory management for gpu-based training of deep neural networks. In 2019 IEEE International Parallel and Distributed Processing Symposium, IPDPS 2019, Rio de Janeiro, Brazil, May 20-24, 2019, pages 200–209. IEEE, 2019

    2019

Show all 62 references
  1. [9]

    Qwen2.5-vl technical report, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Han...

    2025

  2. [10]

    CSWAP: A self-tuning compression framework for accelerating tensor swapping in gpus

    Ping Chen, Shuibing He, Xuechen Zhang, Shuaiben Chen, Peiyi Hong, Yanlong Yin, Xian-He Sun, and Gang Chen. CSWAP: A self-tuning compression framework for accelerating tensor swapping in gpus. In IEEE International Conference on Cluster Computing, CLUSTER 2021, Portland, OR, US...

    2021

  3. [11]

    Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems.arXiv:1512.01274, 2015

    Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang. Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems.arXiv:1512.01274, 2015

    2015 arXiv

  4. [12]

    Training deep nets with sublinear memory cost.arXiv:1604.06174, 2016

    Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost.arXiv:1604.06174, 2016

    2016 arXiv

  5. [13]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei F...

    2025

  6. [14]

    Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei L...

    2025

  7. [15]

    Parallel training of pre-trained models via chunk-based dynamic memory management.IEEE Transactions on Parallel and Distributed Systems, 34(1):304–315, 2023

    Jiarui Fang, Zilin Zhu, Shenggui Li, Hui Su, Yang Yu, Jie Zhou, and Yang You. Parallel training of pre-trained models via chunk-based dynamic memory management.IEEE Transactions on Parallel and Distributed Systems, 34(1):304–315, 2023

    2023

  8. [16]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022. 13 Zibo Wang et al

    2022

  9. [17]

    En- abling parallelism hot switching for efficient training of large language models

    Hao Ge, Fangcheng Fu, Haoyang Li, Xuanyu Wang, Sheng Lin, Yujie Wang, Xiaonan Nie, Hailin Zhang, Xupeng Miao, and Bin Cui. En- abling parallelism hot switching for efficient training of large language models. InProceedings of the ACM SIGOPS 30th Symposium on Operat- ing System...

  10. [18]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravanku- mar, Artem Korenev, A...

    2024 arXiv

  11. [19]

    Gmlake: Efficient and transparent gpu memory defragmentation for large-scale dnn training with virtual memory stitching

    Cong Guo, Rui Zhang, Jiale Xu, Jingwen Leng, Zihan Liu, Ziyu Huang, Minyi Guo, Hao Wu, Shouren Zhao, Junping Zhao, and Ke Zhang. Gmlake: Efficient and transparent gpu memory defragmentation for large-scale dnn training with virtual memory stitching. InProceedings of the 29th A...

    2024

  12. [20]

    Song Han, Huizi Mao, and William J. Dally. Deep compression: Com- pressing deep neural networks with pruning, trained quantization and huffman coding, 2016

    2016

  13. [21]

    Transcending runtime-memory trade- offs in checkpointing by being fusion aware

    Horace He and Shangdi Yu. Transcending runtime-memory trade- offs in checkpointing by being fusion aware. In D. Song, M. Carbin, and T. Chen, editors,Proceedings of Machine Learning and Systems, volume 5, pages 414–427. Curan, 2023

    2023

  14. [22]

    Gpu memory usage optimization for backward propagation in deep network training.Journal of Parallel and Distributed Computing, 199:105053, 2025

    Ding-Yong Hong, Tzu-Hsien Tsai, Ning Wang, Pangfeng Liu, and Jan- Jan Wu. Gpu memory usage optimization for backward propagation in deep network training.Journal of Parallel and Distributed Computing, 199:105053, 2025

    2025

  15. [23]

    Meg- taichi: dynamic tensor-based memory management optimization for DNN training

    Zhongzhe Hu, Junmin Xiao, Zheye Deng, Mingyi Li, Kewei Zhang, Xiaoyang Zhang, Ke Meng, Ninghui Sun, and Guangming Tan. Meg- taichi: dynamic tensor-based memory management optimization for DNN training. In Lawrence Rauchwerger, Kirk W. Cameron, Dim- itrios S. Nikolopoulos, and ...

    2022

  16. [24]

    Swapadvisor: Pushing deep learning beyond the gpu memory limit via smart swapping

    Chien-Chin Huang, Gu Jin, and Jinyang Li. Swapadvisor: Pushing deep learning beyond the gpu memory limit via smart swapping. InProceedings of the Twenty-Fifth International Conference on Archi- tectural Support for Programming Languages and Operating Systems, ASPLOS ’20, page ...

    2020

  17. [25]

    Gpipe: Efficient training of giant neural networks using pipeline parallelism

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, and zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In H. Wallach, H. Larochelle, A. Beygelzimer, F. ...

    2019

  18. [26]

    Oobleck: Resilient distributed training of large models using pipeline templates

    Insu Jang, Zhenning Yang, Zhen Zhang, Xin Jin, and Mosharaf Chowd- hury. Oobleck: Resilient distributed training of large models using pipeline templates. InProceedings of the 29th Symposium on Operating Systems Principles, SOSP ’23, page 382–395, New York, NY, USA, 2023. Asso...

    2023

  19. [27]

    Caffe: Convolutional architecture for fast feature embedding

    Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature embedding. InProceedings of the 22nd ACM International Conference on Multimedia, MM ’14, page 675...

    2014

  20. [28]

    MegaScale: Scaling large language model training to more than 10,000 GPUs

    Ziheng Jiang, Haibin Lin, Yinmin Zhong, Qi Huang, Yangrui Chen, Zhi Zhang, Yanghua Peng, Xiang Li, Cong Xie, Shibiao Nong, Yulu Jia, Sun He, Hongmin Chen, Zhihao Bai, Qi Hou, Shipeng Yan, Ding Zhou, Yiyao Sheng, Zhuo Jiang, Haohan Xu, Haoran Wei, Zhang Zhang, Pengfei Nie, Leqi...

    2024

  21. [29]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv:2001.08361, 2020

    2001 arXiv

  22. [30]

    On model parallelization and scheduling strategies for distributed machine learning

    Seunghak Lee, Jin Kyu Kim, Xun Zheng, Qirong Ho, Garth A Gibson, and Eric P Xing. On model parallelization and scheduling strategies for distributed machine learning. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors,Advances in Neural Informat...

    2014

  23. [31]

    Cognitive Intelligence and Robotics

    Chen Lei.Deep Learning and Practice with MindSpore. Cognitive Intelligence and Robotics. Springer, 2021

    2021

  24. [32]

    Pytorch distributed: Experiences on accelerating data parallel training.Proc

    Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, and Soumith Chintala. Pytorch distributed: Experiences on accelerating data parallel training.Proc. VLDB Endow., 13(12):3005–3018, 2020

    2020

  25. [33]

    Parameter-efficient sparsity for large language models fine-tuning

    Yuchao Li, Fuli Luo, Chuanqi Tan, Mengdi Wang, Songfang Huang, Shen Li, and Junjie Bai. Parameter-efficient sparsity for large language models fine-tuning. In Luc De Raedt, editor,Proceedings of the Thirty- First International Joint Conference on Artificial Intelligence, IJCAI...

    2022

  26. [34]

    Model compression for deep neural networks: A survey.Computers, 12(3), 2023

    Zhuo Li, Hengyi Li, and Lin Meng. Model compression for deep neural networks: A survey.Computers, 12(3), 2023

    2023

  27. [35]

    Ascend: a scalable and unified architecture for ubiquitous deep neural network computing : Industry track paper

    Heng Liao, Jiajin Tu, Jing Xia, Hu Liu, Xiping Zhou, Honghui Yuan, and Yuxing Hu. Ascend: a scalable and unified architecture for ubiquitous deep neural network computing : Industry track paper. In2021 IEEE International Symposium on High-Performance Computer Architecture (HPC...

    2021

  28. [36]

    Diamos, Erich Elsen, David García, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu

    Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory F. Diamos, Erich Elsen, David García, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision train- ing. In6th International Conference on Learning Representations, ICLR 2018, V...

    2018

  29. [37]

    Devanur, Gregory R

    Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. Pipedream: generalized pipeline parallelism for dnn training. InProceedings of the 27th ACM Symposium on Operating Systems Prin- ciple...

    2019

  30. [38]

    Accessed: 2025-06

    NVIDIA. Accessed: 2025-06. nvidia nemo framework developer docs > optimizations > cpu offloading.https://docs.nvidia.com/nemo- framework/user-guide/latest/nemotoolkit/features/optimizations/ cpu_offloading.html

    2025

  31. [39]

    Accessed: 2025-08

    NVIDIA. Accessed: 2025-08. nvidia cuda profiling tools interface (cupti) - cuda toolkit.https://developer.nvidia.com/cupti

    2025

  32. [40]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jef...

    2024

  33. [41]

    Pytorch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chil- amkurthy, Benoit Steiner, L...

    2019

  34. [42]

    Capuchin: Tensor-based gpu memory management for deep learning

    Xuan Peng, Xuanhua Shi, Hulin Dai, Hai Jin, Weiliang Ma, Qian Xiong, Fan Yang, and Xuehai Qian. Capuchin: Tensor-based gpu memory management for deep learning. InProceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Ope...

    2020

  35. [43]

    Accessed: 2024-12

    PyTorch. Accessed: 2024-12. control flow - cond.https://pytorch.org/ docs/stable/cond.html

    2024

  36. [44]

    Accessed: 2024-12

    PyTorch. Accessed: 2024-12. torch.tensor.record_stream.https:// pytorch.org/docs/stable/generated/torch.Tensor.record_stream.html

    2024

  37. [45]

    Zero: Memory optimizations toward training trillion parameter mod- els

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter mod- els. InSC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16, 2020

    2020

  38. [46]

    Zero-infinity: breaking the gpu memory wall for extreme scale deep learning

    Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, and Yuxiong He. Zero-infinity: breaking the gpu memory wall for extreme scale deep learning. InProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’21, ...

    2021

  39. [47]

    Deepspeed: System optimizations enable training deep learning mod- els with over 100 billion parameters

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning mod- els with over 100 billion parameters. InProceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Min- ing, KDD ...

    2020

  40. [48]

    Sentinel: Efficient tensor migration and allocation on heteroge- neous memory systems for deep learning

    Jie Ren, Jiaolin Luo, Kai Wu, Minjia Zhang, Hyeran Jeon, and Dong Li. Sentinel: Efficient tensor migration and allocation on heteroge- neous memory systems for deep learning. In2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pages 598–611, 2021

    2021

  41. [49]

    ZeRO-Offload: Democratizing Billion-Scale model training

    Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. ZeRO-Offload: Democratizing Billion-Scale model training. In2021 USENIX Annual Technical Conference (USENIX ATC 21), pages 551–564. USENIX Association, ...

    2021

  42. [50]

    Minsoo Rhu, Natalia Gimelshein, Jason Clemons, Arslan Zulfiqar, and Stephen W. Keckler. vdnn: Virtualized deep neural networks for scalable, memory-efficient neural network design. In49th Annual IEEE/ACM International Symposium on Microarchitecture, MICRO 2016, Taipei, Taiwan,...

    2016

  43. [51]

    Minsoo Rhu, Mike O’Connor, Niladrish Chatterjee, Jeff Pool, Youngeun Kwon, and Stephen W. Keckler. Compressing DMA engine: Lever- aging activation sparsity for training deep neural networks. InIEEE International Symposium on High Performance Computer Architecture, HPCA 2018, V...

    2018

  44. [52]

    Cntk: Microsoft’s open-source deep- learning toolkit

    Frank Seide and Amit Agarwal. Cntk: Microsoft’s open-source deep- learning toolkit. InProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, page 2135, New York, NY, USA, 2016. Association for Computing Machinery

    2016

  45. [53]

    Neural machine translation of rare words with subword units, 2016

    Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units, 2016

    2016

  46. [54]

    Megatron-lm: Training multi-billion parameter language models using model parallelism

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGres- ley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv:1909.08053, 2019

    1909 arXiv

  47. [55]

    Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, Zhuofu Chen, Jialei Cui, Hao Ding, Mengnan Dong, Angang Du, Chen- zhuang Du, Dikang Du, Yulun Du, Yu Fan, Yichen Feng, Kelin Fu, Bofei Gao, Hongcheng ...

    2025

  48. [56]

    Bamboo: Making preemptible instances resilient for affordable training of large DNNs

    John Thorpe, Pengzhan Zhao, Jonathan Eyolfson, Yifan Qiao, Zhihao Jia, Minjia Zhang, Ravi Netravali, and Guoqing Harry Xu. Bamboo: Making preemptible instances resilient for affordable training of large DNNs. In20th USENIX Symposium on Networked Systems Design and Implementati...

    2023

  49. [57]

    Superneurons: dynamic gpu memory management for training deep neural networks

    Linnan Wang, Jinmian Ye, Yiyang Zhao, Wei Wu, Ang Li, Shuai- wen Leon Song, Zenglin Xu, and Tim Kraska. Superneurons: dynamic gpu memory management for training deep neural networks. In Proceedings of the 23rd ACM SIGPLAN Symposium on Principles and Practice of Parallel Progra...

    2018

  50. [58]

    Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebas- tian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models, 2022

    2022

  51. [59]

    Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Y. X. Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, and Wangding Zeng. Native sparse attention: Hardware-aligned and na- tively trainable sparse attention, 2025

    2025

  52. [60]

    Accelerating the training of large language models using efficient activation rematerialization and optimal hybrid parallelism

    Tailing Yuan, Yuliang Liu, Xucheng Ye, Shenglong Zhang, Jianchao Tan, Bin Chen, Chengru Song, and Di Zhang. Accelerating the training of large language models using efficient activation rematerialization and optimal hybrid parallelism. In2024 USENIX Annual Technical Conference...

  53. [61]

    Pytorch fsdp: Experiences on scaling fully sharded data parallel.Proc

    Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. Pytorch fsdp: Experiences on sc...

    2023

  54. [2024]

    Association for Computing Machinery

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.