Pith. sign in

REVIEW 4 major objections 5 minor 39 references

Cost-Efficient LLM Training with Lifetime-Aware Tensor Offloading via GPUDirect Storage

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TERAIO claims that lifetime-aware tensor offloading to PCIe SSDs lets LLM training run at 80.7% of ideal throughput while cutting hardware cost severalfold.

desk verdict Genuinely new integration of lifetime profiling, I/O-aware planning, and GDS offloading; plausible claims but the evaluation needs variance, direct SSD baselines, and an artifact. read the letter →

arxiv 2506.06472 v1 pith:67UAAU7H submitted 2025-06-06 cs.DC cs.AIcs.LGcs.PF

Ziqi Yuan , Haoyang Zhang , Yirui Eric Zhou , Apoorve Mohan , I-Hsin Chung , Seetharami Seelam , Jian Huang This is my paper
classification cs.DCcs.AIcs.LGcs.PF
keywords LLMtrainingtensoroffloadingGPUmemoryexpansionGPUDirectStorageSSDlifetimeprofilingmigrationplanningPyTorch
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that LLM training can be made far cheaper by treating commodity PCIe SSDs as a large, slow extension of GPU memory without paying the usual throughput penalty. It supports this with TERAIO, a framework that profiles the first few training iterations to learn each tensor's lifetime, then plans fine-grained offload and prefetch operations so that only the tiny fraction of tensors a kernel needs stay in GPU memory. On Llama and Granite models, TERAIO reports 1.47x higher throughput than ZeRO-Offload and ZeRO-Infinity, and reaches 80.7% of the throughput of an idealized machine with unlimited GPU memory. The payoff would be that institutions can train large models on machines with small amounts of HBM plus a few NVMe drives instead of buying extra GPU memory.

What carries the argument

The key mechanism is the lifetime-aware tensor migration plan: a profiler instrumenting PyTorch's automatic operator generator records each tensor's size and active/inactive intervals; a planning algorithm then iteratively selects the inactive tensor whose eviction gives the largest reduction in critical memory pressure per byte of I/O, schedules offload and prefetch with explicit deadlines computed from the measured kernel execution times and available SSD bandwidth, and emits the plan into the compiled training program; at runtime a migration engine executes these transfers directly between GPU memory and SSDs over GPUDirect Storage. The profiler's fixed assumption is that the first few iterations reveal the whole run's activity pattern.

What would settle it

Train a model with per-batch variable sequence lengths or data-dependent control flow so that kernel execution times drift from the profiled values; if TERAIO's throughput falls below ZeRO-Infinity's on that workload, the static-lifetime assumption is the reason.

Watch

Extended reading notes

Core claim

The central discovery is that in LLM training, the working set is tiny even when total tensor memory is enormous: across the models studied, active tensors use on average only 1.7% of requested GPU memory, and most inactive tensors stay unused for longer than it takes to move them to an SSD. This asymmetry means a scheduler that knows each tensor's lifetime can evict large tensors right after their last use and prefetch them just in time, overlapping migration with GPU compute. TERAIO captures the required lifetime information with a lightweight PyTorch profiler, turns it into an offload/prefetch plan that maximizes the ratio of memory-pressure relief to I/O cost, and executes the plan with GPUDirect Storage to bypass the host CPU. The result is claimed to be near-ideal training throughput at a fraction of the hardware cost.

Load-bearing premise

The profiler assumes that the first few training iterations faithfully represent the entire run, so the precomputed offload and prefetch deadlines never go stale.

Editorial extensions

If this is right

  • Trainers can run models that normally need 8-GPU H100 servers on 2-GPU machines with several NVMe SSDs, cutting server cost by about 5x.
  • Fine-grained, lifetime-aware offloading beats coarse layer-level offloading such as ZeRO-Infinity, making storage-based memory expansion practical for pre-training rather than just inference.
  • The reported 1.47x speedup over ZeRO baselines translates directly into cost efficiency because TERAIO uses less CPU memory and fewer SSDs to achieve it.
  • With per-GPU migration bandwidth above 32 to 48 GB/s (achievable by striping commodity SSDs), TERAIO approaches the ideal unlimited-memory roofline, suggesting diminishing returns from further offloading optimizations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the planner assumes static kernel durations, TERAIO's benefit could degrade in workloads with dynamic shapes or data-dependent kernels; adding an online re-profiling or adaptive adjustment layer would be a natural extension.
  • The same lifetime-aware planning concept could be applied to other GPU-memory-congested settings such as fine-tuning, inference batching, or mixture-of-experts routing, where tensor lifetimes are also predictable.
  • The 1.7% active-fraction observation implies that even moderately faster external memory (for example, CXL-attached memory) could be integrated with the same planner to push the achievable fraction of ideal performance closer to 100%.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents TERAIO, a tensor offloading framework that uses GPUDirect Storage to migrate inactive tensors between GPU memory and PCIe-attached SSDs during LLM training. The system profiles the first few training iterations to obtain per-tensor sizes, active periods, and kernel execution times, then runs a greedy benefit-to-cost algorithm (Algorithm 1) to generate offloading and prefetching plans, executing them at runtime through a GDS-based migration engine. The evaluation trains Llama3-8B, Granite-code-base-8B, and Llama3-70B on two H100 GPUs with up to eight SSDs, comparing against ZeRO-Offload, ZeRO-Infinity, and an idealized unlimited-memory baseline. The headline claims are a 1.47x average throughput improvement over the ZeRO systems and 80.7% of ideal performance, with cost-efficiency improvements of up to 5.41x relative to a pure-GPU setup.

Significance. If the central empirical claim holds, TERAIO would be a useful contribution: it demonstrates that commodity PCIe SSDs can serve as a large, low-cost expansion tier for GPU memory in multi-GPU LLM training, with only modest throughput loss. The paper's strengths include a concrete characterization of tensor activity patterns in LLM training, a roofline analysis quantifying the bandwidth needed for near-ideal performance, a clean system design built on PyTorch without requiring user code changes, and a direct comparison against two widely used baselines. The measured speedups and cost-efficiency numbers are internally consistent with the reported figures. However, the evidence is limited to a single two-GPU machine, and several evaluation details needed to substantiate the headline percentages are not reported.

major comments (4)
  1. [Section 3.1 / Section 3.2] The profiling-stability assumption is load-bearing but not validated. The paper states that "the training follows the same execution graph in subsequent iterations, the tensor activity patterns remain the same," yet it never reports kernel-time variance, tensor-lifetime drift, or memory-footprint changes over the course of training. This matters because the runtime engine can itself perturb execution: Section 3.2 admits that it will "stall kernel execution" and "wait for the tensors needed by the kernel to be migrated back," and GDS transfers contend for PCIe bandwidth. If kernel durations or lifetimes deviate from the profiled values, the precomputed prefetch deadlines can be missed, undermining the 80.7%-of-ideal claim. The authors should report run durations, the number of profiled versus evaluated iterations, measured stall counts, and throughput over the full training run rather than only the profiled window.
  2. [Section 4.2, Figure 7] The evaluation lacks run-to-run statistics and covers only one two-GPU machine. No error bars, standard deviations, or number of repeated runs are reported, so the 1.47x average figure could be within noise, especially for configurations where the reported differences between TeraIO-SSD and ZeRO-Infinity are small. Additionally, the paper does not compare against the most closely related SSD-offloading systems such as FlashNeuron, Smart-Infinity, or TBA, only against ZeRO-Offload and ZeRO-Infinity. Without these comparisons and without variance information, the claim that TERAIO is superior to state-of-the-art offloading is not fully supported.
  3. [Section 4.2, Ideal baseline] It is unclear how the "Ideal" baseline is obtained for models whose memory demand exceeds the available GPU memory. Llama3-70B requires approximately 940% of one GPU's capacity, so an unlimited-memory run cannot be executed on the two H100s; it must be simulated. The paper does not state whether the Ideal throughput comes from the kernel-time-based performance model of Section 2.2 or from a separate measurement, nor does it disclose the simulation assumptions. If the Ideal baseline assumes fixed kernel times and no migration stalls, then the 80.7% figure is an upper-bound estimate under those assumptions rather than an empirical comparison, and the paper should say so explicitly.
  4. [Algorithm 1] The lifetime-aware migration planner is a greedy benefit-to-cost search, but its quality is never evaluated. The paper does not compare the generated plan against an optimal or exhaustive plan, nor does it report sensitivity to the profiling window length, to the estimated I/O bandwidth, or to the order in which candidates are examined. Since the central novelty is precisely this planning algorithm, the authors should provide evidence that the greedy choice does not leave significant performance on the table and that the plan is robust to reasonable variation in its inputs.
minor comments (5)
  1. [Figure 1] The subfigure labels are inconsistent: both subplots in the first row are labeled "(a)" and several rows use "(d)" for different models, which makes it hard to identify which plot corresponds to which model and pipeline stage.
  2. [Section 3.1] There is a typo in the phrase "the proifler requires minimal code modifications to PyTorch"; it should be "profiler."
  3. [Throughout] The system name is spelled inconsistently as both "Teraio" and "TeraIO" (and "TERAIO" in the title). Please standardize the spelling.
  4. [Algorithm 1, line 15] The notation "tpref etched" appears to be a typo; it should presumably be "tprefetched" or "tprefetch."
  5. [Section 4.1] The paper cites Smart-Infinity [12] in the introduction but does not use it as a baseline in the evaluation; a sentence explaining why it was omitted would help the reader calibrate the comparison.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: TERAIO's profiled tensor lifetimes feed an execution plan whose throughput is measured independently, so the central 1.47x / 80.7%-of-ideal claims do not reduce to their inputs by construction.

full rationale

The paper's derivation chain is: (1) profile the first few training iterations to obtain per-tensor sizes, active/inactive periods, and kernel execution times; (2) run Algorithm 1 to select offload/prefetch decisions by benefit-to-cost ratio; (3) execute those decisions through a GPUDirect-storage migration engine; and (4) measure end-to-end training throughput against ZeRO-Offload, ZeRO-Infinity, and an Ideal baseline. No step makes the final throughput equal to an input by definition. The lifetime profiler's assumption that the execution graph and tensor activity remain stable across iterations (Section 3.1) is an empirical stationarity assumption; if it fails, the precomputed plan may miss deadlines, but that is a correctness/robustness risk, not a circularity. Algorithm 1 uses tensor sizes, kernel times, bandwidth states, and memory capacity as inputs, and its output is a migration plan, not the measured throughput; the evaluation is an independent experimental result. The Ideal baseline is used as a normalization upper bound, and while its measurement methodology for models exceeding GPU memory is not fully specified, that is a reproducibility concern, not a tautological reduction. The only self-citation involving a coauthor appears in reference [26], used to motivate GPUDirect storage over CPU-mediated copies; this is peripheral support rather than a load-bearing or uniqueness-derived premise. There is no fitted parameter later renamed as a prediction, no imported uniqueness theorem, and no ansatz smuggled in via self-citation. The central performance claims rest on measured system behavior and external baselines, so the appropriate verdict is no significant circularity.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The system uses measured hardware and profiled kernel behavior as inputs; nothing is fitted to the final throughput. The load-bearing premises are that training dataflow is iteration-stable, that the greedy planner is good enough, and that GDS provides the advertised direct transfer path. No new physical entities are introduced.

free parameters (1)
  • Profiling window (number of initial iterations used to build the migration plan) = unspecified; described as 'the first few iterations' (Sections 1 and 3.1)
    The plan's accuracy depends on how many iterations are profiled. The paper never states the number or validates that this window yields stable lifetime estimates.
assumptions (4)
  • domain assumption Tensor lifetimes and sizes are iteration-stable
    The profiler runs a small number of initial iterations and carries the plan forward (Section 3.1). Any data-dependent variation in tensor usage invalidates the schedule.
  • domain assumption GPU kernel execution times are representative across iterations
    The planner uses profiled kernel times tau_i to compute memory-pressure integrals and bandwidth feasibility (Algorithm 1, lines 4-8 and 16-17). Different kernel durations would shift offload deadlines.
  • ad hoc to paper The greedy benefit-to-cost search yields a plan close to optimal
    Algorithm 1 selects candidates by benefit/size ratio and stops when peak memory fits, but no optimality proof is given, while the abstract and Section 1 call the plan optimized or optimal.
  • domain assumption GPUDirect Storage provides direct GPU-to-SSD transfers without a CPU bottleneck
    Section 3.3 relies on GDS to bypass the host. The paper's hardware has eight SSDs at 6.5 GB/s each, but only two GPUs are evaluated, so the multi-GPU scalability assumption is untested.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Cost-Efficient LLM Training with Lifetime-Aware Tensor Offloading via GPUDirect Storage." pith.science (2026). https://pith.science/paper/67UAAU7H

@misc{pith2026250606472,
  author       = {Pith},
  title        = {Pith review of: Cost-Efficient LLM Training with Lifetime-Aware Tensor Offloading via GPUDirect Storage},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/67UAAU7H}},
  note         = {Machine review of arXiv:2506.06472}
}
read the original abstract

We present the design and implementation of a new lifetime-aware tensor offloading framework for GPU memory expansion using low-cost PCIe-based solid-state drives (SSDs). Our framework, TERAIO, is developed explicitly for large language model (LLM) training with multiple GPUs and multiple SSDs. Its design is driven by our observation that the active tensors take only a small fraction (1.7% on average) of allocated GPU memory in each LLM training iteration, the inactive tensors are usually large and will not be used for a long period of time, creating ample opportunities for offloading/prefetching tensors to/from slow SSDs without stalling the GPU training process. TERAIO accurately estimates the lifetime (active period of time in GPU memory) of each tensor with the profiling of the first few iterations in the training process. With the tensor lifetime analysis, TERAIO will generate an optimized tensor offloading/prefetching plan and integrate it into the compiled LLM program via PyTorch. TERAIO has a runtime tensor migration engine to execute the offloading/prefetching plan via GPUDirect storage, which allows direct tensor migration between GPUs and SSDs for alleviating the CPU bottleneck and maximizing the SSD bandwidth utilization. In comparison with state-of-the-art studies such as ZeRO-Offload and ZeRO-Infinity, we show that TERAIO improves the training performance of various LLMs by 1.47x on average, and achieves 80.7% of the ideal performance assuming unlimited GPU memory.

Figures

Figures reproduced from arXiv: 2506.06472 by the authors.

Figure 1
Figure 1. Memory consumption of all and active tensors (w.r.t. the GPU memory capacity) in one [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The distribution of inactive periods of tensors. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Roofline analysis with different migration bandwidths. The training performance is nor￾malized to the ideal case assuming GPU memory is infinite. Long inactive periods of inactive tensors. To understand how long the inactive tensors remain inactive and how much GPU memory they consume, we study the distribution of their inactive periods, as shown in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: System overview of Teraio. 3 Teraio Design and Implementation We show the system overview of Teraio in [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Teraio tracks and analyzes tensor activity patterns for the tensor migration algorithm. Analyzing tensor activity patterns [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Illustration examples to explain the key insights of lifetime-aware migration algorithm. [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Average training throughput of Llama3-8B, Granite-code-base-8B and Llama3-70B with [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: The average migration bandwidth utilization when training the Granite-code-base-8B model [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Training throughput as we vary the CPU memory capacity and the number of SSDs per GPU. These results validate our LLM charac￾terization study’s key observation: with lifetime-aware tensor offloading, we can achieve good performance even with lim￾ited hardware resources…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

39 extracted references · 22 canonical work pages

  1. [1]

    In19th USENIX Conference on File and Storage Technologies (FAST 21), pages 387–401, 2021

    Jonghyun Bae, Jongsung Lee, Yunho Jin, Sam Son, Shine Kim, Hakbeom Jang, Tae Jun Ham, and Jae W Lee.{FlashNeuron}:{SSD-Enabled}{Large-Batch} training of very deep neural networks. In19th USENIX Conference on File and Storage Technologies (FAST 21), pages 387–401, 2021

    work page 2021

  2. [2]

    Efficient combination of rematerialization and offloading for training dnns.Advances in Neural Information Processing Systems, 34:23844–23857, 2021

    Olivier Beaumont, Lionel Eyraud-Dubois, and Alena Shilova. Efficient combination of rematerialization and offloading for training dnns.Advances in Neural Information Processing Systems, 34:23844–23857, 2021

    work page 2021

  3. [3]

    Bert: Pre-trainingofdeep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805, 2018

    JacobDevlin,Ming-WeiChang,KentonLee,andKristinaToutanova. Bert: Pre-trainingofdeep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805, 2018

    arXiv 2018

  4. [4]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

  5. [5]

    Exxact.https://www.exxactcorp.com/, 2025

    EXXACT. Exxact.https://www.exxactcorp.com/, 2025

    work page 2025

  6. [6]

    T5 11b.https://huggingface.co/google-t5/t5-11b, 2025

    Google. T5 11b.https://huggingface.co/google-t5/t5-11b, 2025

    work page 2025

  7. [7]

    nvidia.com/blog/gpudirect-storage/

    GPUDirect Storage: A Direct Path Between Storage and GPU Memory.https://developer. nvidia.com/blog/gpudirect-storage/

    work page

  8. [8]

    Swapadvisor: Pushingdeeplearningbeyondthegpu memory limit via smart swapping

    Chien-ChinHuang,GuJin,andJinyangLi. Swapadvisor: Pushingdeeplearningbeyondthegpu memory limit via smart swapping. InProceedings of the Twenty-Fifth International Conference onArchitecturalSupportforProgrammingLanguagesandOperatingSystems,pages1341–1355, 2020

    work page 2020

Show all 39 references
  1. [9]

    Gpipe: Efficient training of giant neural networks using pipeline parallelism.Advances in neural information processing systems, 32, 2019

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, Hy- oukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism.Advances in neural information processing systems, 32, 2019

    2019

  2. [10]

    Ibm granite.https://huggingface.co/ibm-granite, 2025

    IBM. Ibm granite.https://huggingface.co/ibm-granite, 2025

    2025

  3. [11]

    Checkmate: Breaking the memory wall with optimal tensor rematerialization.Proceedings of Machine Learning and Systems, 2:497–511, 2020

    Paras Jain, Ajay Jain, Aniruddha Nrusimha, Amir Gholami, Pieter Abbeel, Joseph Gonzalez, Kurt Keutzer, and Ion Stoica. Checkmate: Breaking the memory wall with optimal tensor rematerialization.Proceedings of Machine Learning and Systems, 2:497–511, 2020. 10

    2020

  4. [12]

    Smart-infinity: Fastlargelanguagemodeltrainingusingnear-storageprocessingonarealsystem

    Hongsun Jang, Jaeyong Song, Jaewon Jung, Jaeyoung Park, Youngsok Kim, and Jinho Lee. Smart-infinity: Fastlargelanguagemodeltrainingusingnear-storageprocessingonarealsystem. In2024 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pages 345–360. IEEE, 2024

    2024

  5. [13]

    Deepum: Tensormigrationandprefetchinginunified memory

    JaehoonJung,JinpyoKim,andJaejinLee. Deepum: Tensormigrationandprefetchinginunified memory. InProceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, pages 207–221, 2023

    2023

  6. [14]

    Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  7. [15]

    Beyond the memory wall: A case for memory-centric hpc system for deep learning

    Youngeun Kwon and Minsoo Rhu. Beyond the memory wall: A case for memory-centric hpc system for deep learning. In2018 51st Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), pages 148–161. IEEE, 2018

    2018

  8. [16]

    Tflms: Large model support in tensorflow by graph rewriting.arXiv preprint arXiv:1807.02037, 2018

    Tung D Le, Haruki Imai, Yasushi Negishi, and Kiyokuni Kawachiya. Tflms: Large model support in tensorflow by graph rewriting.arXiv preprint arXiv:1807.02037, 2018

    2018 arXiv

  9. [17]

    Torchtitan: One-stop pytorch native solution for production ready llm pre-training.arXiv preprint arXiv:2410.06511, 2024

    Wanchao Liang, Tianyu Liu, Less Wright, Will Constable, Andrew Gu, Chien-Chin Huang, Iris Zhang, Wei Feng, Howard Huang, Junjie Wang, et al. Torchtitan: One-stop pytorch native solution for production ready llm pre-training.arXiv preprint arXiv:2410.06511, 2024

    arXiv

  10. [18]

    Adding nvme ssds to enable and accelerate 100b model fine-tuning on a single gpu.arXiv preprint arXiv:2403.06504, 2024

    Changyue Liao, Mo Sun, Zihan Yang, Kaiqi Chen, Binhang Yuan, Fei Wu, and Zeke Wang. Adding nvme ssds to enable and accelerate 100b model fine-tuning on a single gpu.arXiv preprint arXiv:2403.06504, 2024

    2024 arXiv

  11. [19]

    An empirical model of large-batch training.arXiv preprint arXiv:1812.06162, 2018

    Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAI Dota Team. An empirical model of large-batch training.arXiv preprint arXiv:1812.06162, 2018

    2018 arXiv

  12. [20]

    Mixed precision training.arXiv preprint arXiv:1710.03740, 2017

    PauliusMicikevicius,SharanNarang,JonahAlben,GregoryDiamos,ErichElsen,DavidGarcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training.arXiv preprint arXiv:1710.03740, 2017

    2017 arXiv

  13. [21]

    Pipedream: Generalized pipeline parallelism for dnn training

    Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, and Matei Zaharia. Pipedream: Generalized pipeline parallelism for dnn training. InProceedings of the 27th ACM symposium on operating systems principles, pa...

    2019

  14. [22]

    Angel-ptm: A scalable and economical large-scale pre-training system in tencent.arXiv preprint arXiv:2303.02868, 2023

    XiaonanNie, YiLiu, FangchengFu, JinbaoXue, DianJiao, XupengMiao, YangyuTao, andBin Cui. Angel-ptm: A scalable and economical large-scale pre-training system in tencent.arXiv preprint arXiv:2303.02868, 2023

    2023 arXiv

  15. [23]

    Automatic differentiation in pytorch

    Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. InNIPS-W, 2017

    2017

  16. [24]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195–4205, 2023

    2023

  17. [25]

    Training large neural networks with constant memory using a new execution algorithm.arXiv preprint arXiv:2002.05645, 2020

    Bharadwaj Pudipeddi, Maral Mesmakhosroshahi, Jinwen Xi, and Sujeeth Bharadwaj. Training large neural networks with constant memory using a new execution algorithm.arXiv preprint arXiv:2002.05645, 2020

    2002 arXiv

  18. [26]

    Gpu-initiated on-demand high-throughput storage access in the bam system architecture

    Zaid Qureshi, Vikram Sharma Mailthody, Isaac Gelado, Seungwon Min, Amna Masood, JeongminPark,JinjunXiong,C.J.Newburn,DmitriVainbrand,I-HsinChung,MichaelGarland, William Dally, and Wen-mei Hwu. Gpu-initiated on-demand high-throughput storage access in the bam system architectur...

    2023

  19. [27]

    Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019. 11

    2019

  20. [28]

    Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020

    2020

  21. [29]

    Zero- infinity: Breaking the gpu memory wall for extreme scale deep learning

    Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, and Yuxiong He. Zero- infinity: Breaking the gpu memory wall for extreme scale deep learning. InProceedings of the international conference for high performance computing, networking, storage and analysis, pages 1...

    2021

  22. [30]

    {Zero-offload}: Democratizing{billion-scale}model training

    Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, MinjiaZhang,DongLi,andYuxiongHe. {Zero-offload}: Democratizing{billion-scale}model training. In2021 USENIX Annual Technical Conference (USENIX ATC 21), pages 551–564, 2021

    2021

  23. [31]

    vdnn: Virtualizeddeepneuralnetworksforscalable,memory-efficientneuralnetworkdesign

    Minsoo Rhu, Natalia Gimelshein, Jason Clemons, Arslan Zulfiqar, and Stephen W Keckler. vdnn: Virtualizeddeepneuralnetworksforscalable,memory-efficientneuralnetworkdesign. In 2016 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), pages 1–13. IEEE, 2016

    2016

  24. [32]

    Building ai agents for autonomous clouds: Challenges and design principles

    Manish Shetty, Yinfang Chen, Gagan Somashekar, Minghua Ma, Yogesh Simmhan, Xuchao Zhang, Jonathan Mace, Dax Vandevoorde, Pedro Las-Casas, Shachee Mishra Gupta, et al. Building ai agents for autonomous clouds: Challenges and design principles. InProceedings of the 2024 ACM Symp...

    2024

  25. [33]

    Stronghold: fast and affordable billion-scale deep learning model training

    Xiaoyang Sun, Wei Wang, Shenghao Qiu, Renyu Yang, Songfang Huang, Jie Xu, and Zheng Wang. Stronghold: fast and affordable billion-scale deep learning model training. InSC22: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–17. ...

    2022

  26. [34]

    Superneurons: Dynamic gpu memory management for training deep neural networks

    Linnan Wang, Jinmian Ye, Yiyang Zhao, Wei Wu, Ang Li, Shuaiwen Leon Song, Zenglin Xu, and Tim Kraska. Superneurons: Dynamic gpu memory management for training deep neural networks. InProceedings of the 23rd ACM SIGPLAN symposium on principles and practice of parallel programmi...

    2018

  27. [35]

    Netllm: Adapting large language models for networking

    Duo Wu, Xianda Wang, Yaqi Qiao, Zhi Wang, Junchen Jiang, Shuguang Cui, and Fangxin Wang. Netllm: Adapting large language models for networking. InProceedings of the ACM SIGCOMM 2024 Conference, pages 661–678, 2024

    2024

  28. [36]

    Tba: Faster large language model training using ssd-based activation offloading.arXiv preprint arXiv:2408.10013, 2024

    Kun Wu, Jeongmin Brian Park, Xiaofan Zhang, Mert Hidayetoğlu, Vikram Sharma Mailthody, Sitao Huang, Steven Sam Lumetta, and Wen-mei Hwu. Tba: Faster large language model training using ssd-based activation offloading.arXiv preprint arXiv:2408.10013, 2024

    2024 arXiv

  29. [37]

    Acceleratingthetrainingoflargelanguagemodelsusingefficientactivation rematerializationandoptimalhybridparallelism

    Tailing Yuan, Yuliang Liu, Xucheng Ye, Shenglong Zhang, Jianchao Tan, Bin Chen, Chengru Song,andDiZhang. Acceleratingthetrainingoflargelanguagemodelsusingefficientactivation rematerializationandoptimalhybridparallelism. In2024USENIXAnnualTechnicalConference (USENIX ATC 24), pa...

    2024

  30. [38]

    Zng: Architecting gpu multi-processors with new flash for scalable data analysis

    Jie Zhang and Myoungsoo Jung. Zng: Architecting gpu multi-processors with new flash for scalable data analysis. In2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA), pages 1064–1075. IEEE, 2020

    2020

  31. [39]

    Flashgpu: Placing new flash next to gpu cores

    Jie Zhang, Miryeong Kwon, Hyojong Kim, Hyesoon Kim, and Myoungsoo Jung. Flashgpu: Placing new flash next to gpu cores. InProceedings of the 56th Annual Design Automation Conference 2019, pages 1–6, 2019. 12

    2019

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.