Pith. sign in

REVIEW 4 major objections 4 minor 56 references

Offloading optimizer state to NVMe plus a parallel file system speeds LLM pretraining up to 2.5x.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-05 11:34 UTC pith:NMAIBTIF

load-bearing objection A well-built offloading engine that earns its speedup on uncontended tiers; the headline 2.5x is real under the paper's stated conditions, but those conditions are not stress-tested. the 4 major comments →

arxiv 2509.02480 v1 pith:NMAIBTIF submitted 2025-09-02 cs.DC cs.AIcs.LG

MLP-Offload: Multi-Level, Multi-Path Offloading for LLM Pre-training to Break the GPU Memory Wall

Avinash Maurya , M. Mustafa Rafique , Franck Cappello , Bogdan Nicolae This is my paper
classification cs.DC cs.AIcs.LG
keywords LLM pretrainingoptimizer state offloadingmulti-tier storageparallel file systemNVMeZeRO-3I/O bottleneckasynchronous I/O
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Large language models are growing faster than GPU memory, so training them increasingly requires spilling optimizer state to disk. The paper's central claim is that this spill does not have to be a severe bottleneck: by combining the local NVMe drive with an underused parallel file system into a single virtual tier, and by controlling which process reads and writes which tier, the update phase's I/O can be cut dramatically. The authors report end-to-end iterations up to 2.5x faster than the standard ZeRO-3 offloading runtime on models from 40B to 280B parameters, with the backward pass up to 13.5x faster. A sympathetic reader would care because it points to a practical way to pretrain very large models on a small number of GPUs without waiting for bigger GPU memory.

Core claim

The authors establish that, when optimizer states are offloaded to storage, the update phase is dominated not by CPU arithmetic but by fetching and flushing full-precision subgroups; in their 20B-120B measurements, 99% of update time is disk I/O. They then show that this I/O can be attacked on four coordinated axes: (1) create a virtual third-level tier from node-local NVMe and a remote parallel file system, assigning subgroups to each tier in proportion to measured bandwidth so all tiers finish around the same time; (2) enforce node-local concurrency control so only one worker process accesses a given tier at a time, preserving full bandwidth per transfer; (3) alternate ascending and descen

What carries the argument

The central object is the virtual third-level tier: a pool of alternative storage (local NVMe, parallel file system, and by extension object stores) treated as one logical tier. Subgroups are allocated across its constituent storages by a bandwidth-proportional rule, assigning each tier a number of subgroups roughly proportional to its share of the total I/O bandwidth, so slow tiers get fewer subgroups and fetches and flushes complete at the same time. Around this, the update loop is reordered (ascending and descending subgroup IDs in alternate iterations) to reuse cached subgroups, gradient conversion is deferred from the backward pass to the update phase, and a process-exclusive, multi-thr

Load-bearing premise

The load-bearing premise is that the parallel file system has stable, uncontended bandwidth available during training; if other jobs saturate it, the multi-path speedup shrinks or disappears.

What would settle it

Run the same models with the parallel file system bandwidth artificially throttled to a small fraction of local NVMe bandwidth, or with concurrent jobs writing to the same PFS; if end-to-end iteration time over the baseline drops to near 1x, the central opportunity is not portable to shared production systems.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Training runs with hundreds of billions of parameters can use a small GPU allocation if a parallel file system is available, because idle PFS bandwidth becomes usable offload capacity.
  • The update phase stops being a hard I/O wall; with multi-path fetch and flush, iteration time can approach the compute and communication bound rather than the disk bound.
  • Because optimizer states are pre-staged on persistent storage, asynchronous checkpointing can overlap with forward and backward passes instead of stalling training.
  • The subgroup-reordering trick is independent of the tier split and should yield gains even when only local NVMe is available.
  • The bandwidth-proportional allocation rule gives a simple knob for splitting work between any two storage tiers, not just NVMe and PFS.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The speedup is contingent on shared parallel file system bandwidth remaining available; under heavy cross-job contention the adaptive allocation would shift subgroups back to NVMe, and the advantage over the baseline should shrink—an untested stress case the paper itself flags for future work.
  • The same bandwidth-proportional allocation could be reused for next-generation memory tiers such as CXL memory pools or object stores, as long as read and write bandwidth can be measured; the paper names this as a direction, not a demonstrated result.
  • A direct extension would be to apply the alternating update order and delayed gradient conversion to CPU-only offloading runtimes, where the cache-thrash and FP32-gradient fetch costs are smaller but still present.
  • The reported comparisons set subgroup size to 100M for all approaches; running with the runtime's default 1B subgroups could change the overlap and is a natural robustness check.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes MLP-Offload, a multi-level, multi-path offloading engine for LLM pre-training under GPU memory constraints. The key idea is to combine node-local NVMe with a parallel file system (PFS) into a single virtual third-level storage tier, allocate optimizer-state subgroups across these tiers proportionally to measured I/O bandwidth (§3.3, Eq. 1), and use cache-friendly subgroup reordering, delayed in-place FP16-to-FP32 gradient conversion, and node-local concurrency control to reduce I/O stalls. The implementation extends DeepSpeed's DeepNVMe engine and is open source. Evaluations on models from 40B to 280B parameters on up to 32 A100/H100 GPUs report end-to-end iteration speedups of about 2.5x over DeepSpeed ZeRO-3 with NVMe offloading, with a 1.6x speedup from the three non-PFS optimizations alone (Fig. 14) and an additional 1.6x from adding the PFS path (Fig. 15).

Significance. If the reported results hold, the paper makes a useful systems contribution: it identifies a real bottleneck in disk-offloaded LLM training, provides a clean design with several plausibly orthogonal optimizations, and validates each optimization through ablation. The gap analysis in §3.1 is convincing, and the open-source artifact is a strength. The central claim, however, is conditional on the assumption that the PFS provides stable, uncontended bandwidth during training (§3.2, §5). Because the additional gain from the PFS path is what lifts the speedup from 1.6x to 2.5x, this assumption is load-bearing for the headline result. The evaluation also lacks variance reporting and contains a few internal inconsistencies in the reported speedup values.

major comments (4)
  1. [§3.2, §4.6, §5] The headline 2.5x speedup depends critically on the PFS path: the ablation in Fig. 15 shows that without PFS-enabled multi-path I/O, the accumulated optimizations produce only about 1.6x (Fig. 14). The paper justifies PFS usage with the claim in §3.2 that the PFS is 'typically under-utilized during pretraining (save for occasional checkpoints),' but this is not stress-tested. The evaluation reports only uncontended PFS microbenchmarks (Table 1) and does not inject background PFS load or report contention from other jobs. The concurrency control in §3.2 is node-local, so cross-node/cross-job contention on a shared PFS is unmitigated. The paper itself acknowledges in §5 that 'a deeper study on the behavior of globally shared alternative storage tiers under I/O competition' is future work. This makes the 2.5x claim as stated too broad. Please either scope the abstract/conclusion to 'unconte
  2. [§4.1, 'Runtime Configurations'] The evaluation changes the subgroup size to 100M parameters for all approaches, whereas DeepSpeed's default is 1B. The text says 'subgroup sizes do not impact the iteration duration, convergence, or accuracy' but then immediately states that 'smaller subgroups achieve better I/O and compute overlap,' which is contradictory. This matters because the comparison changes a key parameter of the baseline. While using the same subgroup size for both approaches is a reasonable methodological choice, the paper should clarify why 100M is representative for DeepSpeed ZeRO-3, quantify sensitivity to subgroup size (e.g., 1B vs 500M vs 100M), and correct the contradictory sentence.
  3. [§4.2, §4.4, §5] There is no statistical variance reported for the end-to-end speedups. Each configuration is averaged over 8 iterations on a single testbed (Testbed-1 or Testbed-2) with no run-to-run variability, no multiple seeds, and no repeated measurements across different batches or node allocations. The central claim of 2.5x is a single point estimate. For a systems paper, this is acceptable only if the authors state that results are deterministic or if they provide at least a range. Please report the spread, e.g., error bars or min-max over repeated runs, for at least the headline configuration (40B/70B on Testbed-1 and 280B on Testbed-2).
  4. [§4.2 vs. Abstract/Conclusion] The reported speedup is internally inconsistent: §4.2 states 'overall 2.7× faster compared with DeepSpeed ZeRO-3,' while the abstract and §5 state '2.5× faster training iterations.' The weak-scaling results in §4.4 show up to 2× speedups. Please reconcile these numbers and state the range (e.g., 1.6x–2.7x depending on configuration, with 2.5x as a representative value) so the headline claim is supported by the data.
minor comments (4)
  1. [§3.1] In the 40B example, 0.6s out of 242s is 0.25%, not 0.02% as printed. Please correct the percentage.
  2. [§4.1] Typo: 'ZeRO-Inifinity' should be 'ZeRO-Infinity' (also appears in the Introduction).
  3. [§4.4] The sentence about cost-effectiveness is confusing: 'MLP-Offload is only 4.8× slower, thereby achieving a 5× slowdown while using 10× fewer GPUs– yielding a 2× improvement in cost-effectiveness.' Please rephrase to state the GPU-normalized throughput comparison cleanly.
  4. [Table 1] The description of Testbed-2 says 'through 160 (OSTs)'; this should read 'through 160 OSTs' or 'object storage targets'.

Circularity Check

0 steps flagged

No significant circularity: the 2.5x speedup is an end-to-end measurement; Eq. (1) is a load-balancing heuristic, not a fitted predictor, and the self-citations are contextual.

full rationale

The paper's central claim is empirical. MLP-Offload reports measured iteration times on 40B-280B models against DeepSpeed ZeRO-3 (§4). The performance model in Eq. (1) is a bandwidth-proportional allocation rule: T_i = M*B_i/sum(B_i), where B_i are measured/observed I/O bandwidths. This rule determines how many subgroups go to NVMe versus PFS, but it does not generate the reported 2.5x speedup; the speedup is obtained by direct timing of training iterations. There is no fitted parameter that is later renamed as a prediction of iteration time. The PFS is added as a second offload path based on an external observation that PFS is typically underutilized during pretraining (§3.2, citing [46]); while §5 acknowledges that a shared PFS under contention is not deeply studied, this is a scoping limitation for the speedup claim, not a circular argument. Self-citations to the authors' prior works ([24] DataStates-LLM, [25] I/O pattern study, [26] Deep Optimizer States) appear only as related work or as a note on integrating checkpointing; they are not load-bearing for the central 2.5x result, and no uniqueness theorem or ansatz is imported from them. The choice to use 100M subgroups for all compared approaches is a methodology decision that could affect fairness, but it is a correctness/comparison concern, not a circular derivation. Therefore no circular step can be exhibited.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

The central claim depends on standard systems assumptions (PFS availability, stable bandwidth, per-subgroup Adam independence) and on the chosen subgroup size and buffer allocation; these are configuration choices, not hidden fitted constants. No new entities are introduced.

free parameters (3)
  • subgroup_size (M) = 100 million parameters
    User-configurable; set to 100M for all experiments, smaller than DeepSpeed default of 1B, to improve I/O/compute overlap. This affects the granularity of caching and the number of asynchronous operations.
  • host_buffer_allocation = >90% memory utilization
    Pinned buffers pre-allocated to use maximum host memory for both approaches; exact fraction depends on model and runtime overheads.
  • NVMe:PFS allocation ratio = approximately 2:1
    Derived from measured bandwidths via Eq. 1 and observed in Figure 10; not an independent fit, but a computed configuration.
axioms (3)
  • domain assumption Adam optimizer updates are embarrassingly parallel per parameter subgroup, so processing order does not affect convergence.
    Section 3.2 (Cache-Friendly Ordering) relies on this to reverse subgroup order between iterations. True for standard Adam but not formally re-derived.
  • domain assumption The parallel file system is available and underutilized during LLM training, so its bandwidth can be used without interfering with other workloads.
    Section 3.2 states PFS is 'typically under-utilized during pretraining (save for occasional checkpoints)'; this is an empirical claim about HPC scheduling, not guaranteed.
  • domain assumption Each storage tier has a scalar bandwidth B_i that is stable and additive across tiers, making Equation 1 a valid load-balancing rule.
    Section 3.3 assumes a single B_i per tier; under cross-job PFS contention or NVMe interference this assumption degrades, acknowledged in Section 5.

pith-pipeline@v1.4.0-alltime-deepseek-medium · 21289 in / 10827 out tokens · 107989 ms · 2026-08-05T11:34:30.064751+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of MLP-Offload: Multi-Level, Multi-Path Offloading for LLM Pre-training to Break the GPU Memory Wall." pith.science (2026). https://pith.science/paper/NMAIBTIF

@misc{pith2026250902480,
  author       = {Pith},
  title        = {Pith review of: MLP-Offload: Multi-Level, Multi-Path Offloading for LLM Pre-training to Break the GPU Memory Wall},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NMAIBTIF}},
  note         = {Machine review of arXiv:2509.02480}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Training LLMs larger than the aggregated memory of multiple GPUs is increasingly necessary due to the faster growth of LLM sizes compared to GPU memory. To this end, multi-tier host memory or disk offloading techniques are proposed by state of art. Despite advanced asynchronous multi-tier read/write strategies, such offloading strategies result in significant I/O overheads in the critical path of training, resulting in slower iterations. To this end, we propose MLP-Offload, a novel multi-level, multi-path offloading engine specifically designed for optimizing LLM training on resource-constrained setups by mitigating I/O bottlenecks. We make several key observations that drive the design of MLP-Offload, such as I/O overheads during the update dominate the iteration time; I/O bandwidth of the third-level remote storage tier remains unutilized; and, contention due to concurrent offloading amplifies I/O bottlenecks. Driven by these insights, we design and implement MLP-Offload to offload the optimizer states across multiple tiers in a cache-efficient and concurrency-controlled fashion to mitigate I/O bottlenecks during the backward and update phases. Evaluations on models up to 280B parameters shows that MLP-Offload achieves 2.5$\times$ faster iterations compared to the state-of-the-art LLM training runtimes.

Figures

Figures reproduced from arXiv: 2509.02480 by Avinash Maurya, Bogdan Nicolae, Franck Cappello, M. Mustafa Rafique.

Figure 1
Figure 1. Figure 1: Model vs GPU memory growth. generations of devices (2×/2 years) [10, 25, 38]. This trend, illus￾trated in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: DeepSpeed ZeRO-3 showing (a) hybrid model and data parallelism; wherein (b) the model parameters, activations, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: I/O bandwidth of SSD (local) vs. parallel file system (remote) 20 40 60 80 Subgroup Number 1 2 3 4 5 6 Effective Read/Write Throughput (GB/s) Read (x̄= 3.68) Write (x̄= 1.44) [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Illustrative example: application of the design principles proposed by [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 9
Figure 9. Figure 9: Effective I/O throughputs for different model sizes. different amount of runtime-level buffers, such as for gradient accu￾mulation, all-reduce, etc. We refer the reader to ZeRO-Offload [32], ZeRO-Infinity [30], and DeepSpeed memory estimator [7] for the breakdown of GPU/host memory consumed by the runtime. Throughout our evaluations, we ensure that the aggregated GPU memory is sufficient to store the follo… view at source ↗
Figure 10
Figure 10. Figure 10: Distribution of optimizer states across different tiers. 40B [4] 70B [8] 100B [12] 130B [16] Model size [Num GPUs] 0 100 200 300 400 Average iteration time breakdown (s) DeepSpeed ZeRO-3 Our Approach Forward Backward Update 242.3 111.0 178.0 68.3 167.5 85.7 155.6 79.4 [PITH_FULL_IMAGE:figures/full_fig_p010_10.png] view at source ↗
Figure 14
Figure 14. Figure 14: Performance ablation on node￾local NVMe. 40B 70B 100B Model size (Billions of params) 0 200 400 600 Average iteration time breakdown (s) Multi-Path (with caching) MP Skip Grads Our Approach Forward Backward Update 166.3 108.5 95.8 244.3 157.8 144.4 404.8 272.8 241.4 [PITH_FULL_IMAGE:figures/full_fig_p010_14.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

56 extracted references · 37 canonical work pages · 1 internal anchor

  1. [1]

    Amazon Web Services. 2025. Amazon FSx for Lustre Pricing. https://aws.amazon. com/fsx/lustre/pricing/. Accessed: 2025-02-05

    work page 2025

  2. [2]

    Argonne Joint Laboratory for System Evaluation (JLSE). 2023. NVIDIA H100. https://www.jlse.anl.gov/nvidia-h100. Accessed: 2025-02-04

    work page 2023

  3. [3]

    Argonne Leadership Computing Facility. 2025. Polaris. https://www.alcf.anl.gov/ polaris

    work page 2025

  4. [4]

    Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, et al

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, et al

    work page

  5. [5]

    Zhenkun Cai, Xiao Yan, Kaihao Ma, Yidi Wu, Yuzhen Huang, James Cheng, Teng Su, and Fan Yu. 2021. Tensoropt: Exploring the tradeoffs in distributed dnn training with auto-parallelism. IEEE Transactions on Parallel and Distributed Systems 33, 8 (2021), 1967–1981

    2021

  6. [6]

    Xiaoyu Chu, Daniel Hofstätter, Shashikant Ilager, Sacheendra Talluri, Duncan Kampert, Damian Podareanu, Dmitry Duplyakin, Ivona Brandic, and Alexandru Iosup. 2024. Generic and ML Workloads in an HPC Datacenter: Node Energy, Job Failures, and Node-Job Analysis. In International Conference on Parallel and Distributed Systems (ICPADS). IEEE, 710–719

    work page 2024

  7. [7]

    DeepSpeed Team. n.d.. Memory Requirements- DeepSpeed Documentation. https://deepspeed.readthedocs.io/en/latest/memory.html. Accessed: 2025-04-09. SC ’25, November 16–21, 2025, St Louis, MO, USA Avinash Maurya et al

    work page 2025

  8. [8]

    Diego Didona, Jonas Pfefferle, Nikolas Ioannou, Bernard Metzler, and Animesh Trivedi. 2022. Understanding modern storage APIs: a systematic study of libaio, SPDK, and io_uring. In Intl. Conference on Systems and Storage (SYSTOR) . ACM, 120–127

    work page 2022

  9. [9]

    Jiangfei Duan, Shuo Zhang, Zerui Wang, Lijuan Jiang, Wenwen Qu, Qinghao Hu, Guoteng Wang, Qizhen Weng, Hang Yan, Xingcheng Zhang, Xipeng Qiu, Dahua Lin, Yonggang Wen, Xin Jin, Tianwei Zhang, and Peng Sun. 2024. Efficient Training of Large Language Models on Distributed Infrastructures: A Survey. arXiv:2407.20018 [cs.DC] https://arxiv.org/abs/2407.20018

    Pith/arXiv arXiv 2024

  10. [10]

    Amir Gholami, Zhewei Yao, Sehoon Kim, Coleman Hooper, Michael W Mahoney, and Kurt Keutzer. 2024. AI and memory wall. Micro 44, 3 (2024), 33–39

    work page 2024

  11. [11]

    Gossman, Bogdan Nicolae, and Jon C

    Mikaila J. Gossman, Bogdan Nicolae, and Jon C. Calhoun. 2024. Scalable I/O aggregation for asynchronous multi-level checkpointing. Future Generation Computer Systems 160 (2024), 420–432

    work page 2024

  12. [12]

    Matt Hamblen. 2023. Update: ChatGPT runs 10K Nvidia training GPUs with po- tential for thousands more. https://www.fierceelectronics.com/sensors/chatgpt- runs-10k-nvidia-training-gpus-potential-thousands-more

    work page 2023

  13. [13]

    Le, Yonghui Wu, and Zhifeng Chen

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia Xu Chen, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. 2019. GPipe: efficient training of giant neural networks using pipeline parallelism. Curran Associates Inc., Red Hook, NY, USA

    work page 2019

  14. [14]

    Hongsun Jang, Jaeyong Song, Jaewon Jung, Jaeyoung Park, Youngsok Kim, and Jinho Lee. 2024. Smart-Infinity: Fast Large Language Model Training using Near- Storage Processing on a Real System. In 2024 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE, 345–360

    work page 2024

  15. [15]

    Xuanlin Jiang, Yang Zhou, Shiyi Cao, Ion Stoica, and Minlan Yu. 2024. NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inference. arXiv:2411.01142 [cs.DC] https://arxiv.org/abs/2411.01142

    Pith/arXiv arXiv 2024

  16. [16]

    Yao Jiang, Xinyu Yan, Ge-Peng Ji, Keren Fu, Meijun Sun, Huan Xiong, Deng-Ping Fan, and Fahad Shahbaz Khan. 2024. Effectiveness assessment of recent large vision-language models. Visual Intelligence 2, 1 (2024), 17

    work page 2024

  17. [17]

    Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic opti- mization. arXiv preprint arXiv:1412.6980 (2014)

    Pith/arXiv arXiv 2014

  18. [18]

    Joonhyung Lee, Jeongin Bae, Byeongwook Kim, Se Jung Kwon, and Dongsoo Lee

    work page

  19. [19]

    Shenggui Li, Hongxin Liu, Zhengda Bian, Jiarui Fang, Haichen Huang, Yuliang Liu, Boxiang Wang, and Yang You. 2023. Colossal-ai: A unified deep learning system for large-scale parallel training. In Proceedings of the 52nd International Conference on Parallel Processing . 766–775

    work page 2023

  20. [20]

    Xiang Li, Yiqun Yao, Xin Jiang, Xuezhi Fang, Chao Wang, Xinzhang Liu, Zi- han Wang, Yu Zhao, Xin Wang, Yuyao Huang, Shuangyong Song, Yongxi- ang Li, Zheng Zhang, Bo Zhao, Aixin Sun, Yequan Wang, Zhongjiang He, Zhongyuan Wang, Xuelong Li, and Tiejun Huang. 2024. Tele-FLM Technical Report. arXiv:2404.16645

    Pith/arXiv arXiv 2024

  21. [22]

    Changyue Liao, Mo Sun, Zihan Yang, Jun Xie, Kaiqi Chen, Binhang Yuan, Fei Wu, and Zeke Wang. 2024. LoHan: Low-Cost High-Performance Framework to Fine-Tune 100B Model on a Consumer GPU. arXiv:2403.06504 [cs.DC] https: //arxiv.org/abs/2403.06504

    Pith/arXiv arXiv 2024

  22. [23]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 (2024)

    Pith/arXiv arXiv 2024

  23. [24]

    Mustafa Rafique, Franck Cappello, and Bogdan Nicolae

    Avinash Maurya, Robert Underwood, M. Mustafa Rafique, Franck Cappello, and Bogdan Nicolae. 2024. DataStates-LLM: Lazy Asynchronous Checkpointing for Large Language Models. In Proc. of the International Symposium on High- Performance Parallel and Distributed Computing (Pisa, Italy) (HPDC’24). 13 pages

    work page 2024

  24. [25]

    Mustafa Rafique, Franck Cappello, and Bogdan Nico- lae

    Avinash Maurya, Jue Ye, M. Mustafa Rafique, Franck Cappello, and Bogdan Nico- lae. 2024. Breaking the Memory Wall: A Study of I/O Patterns and GPU Memory Utilization for Hybrid CPU-GPU Offloaded Optimizers. In FlexScience’24: Work- shop on AI & Scientific Computing at Scale using Flexible Comp. Infrastructures

    work page 2024

  25. [26]

    Avinash Maurya, Jie Ye, M Mustafa Rafique, Franck Cappello, and Bogdan Nicolae

    work page

  26. [27]

    Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. 2018. Mixed Precision Training. arXiv:1710.03740 [cs.AI] https://arxiv.org/abs/1710.03740

    Pith/arXiv arXiv 2018

  27. [28]

    Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, et al

    Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, et al. 2022. Scal- ing Language Models: Methods, Analysis & Insights from Training Gopher. arXiv:2112.11446 [cs.CL] https://arxiv.org/abs/2112.11446

    Pith/arXiv arXiv 2022

  28. [29]

    In Proceedings of the 25th International Middleware Conference

    Deep optimizer states: Towards scalable training of transformer models using interleaved offloading. In Proceedings of the 25th International Middleware Conference. 404–416

    work page

  29. [30]

    Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, and Yuxiong He. 2021. ZeRO-infinity: breaking the GPU memory wall for extreme scale deep learning. In SC’21: The 2021 International Conference for High Performance Computing, Networking, Storage and Analysis. St. Louis, USA, Article 59, 14 pages

    work page 2021

  30. [31]

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deep- speed: System optimizations enable training deep learning models with over 100 billion parameters. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining . 3505–3506

    work page 2020

  31. [32]

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. ZeRO: Memory optimizations Toward Training Trillion Parameter Models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis. 1–16

    work page 2020

  32. [33]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-LM: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053 (2019)

    Pith/arXiv arXiv 2019

  33. [34]

    Shuaiwen Leon Song, Bonnie Kruft, Minjia Zhang, Conglong Li, Shiyang Chen, et al. 2023. DeepSpeed4Science Initiative: Enabling Large-Scale Scientific Discov- ery through Sophisticated AI System Technologies. arXiv:2310.04610 [cs]

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  34. [35]

    2021.{Zero-offload}: Democratizing{billion-scale} model training

    Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. 2021.{Zero-offload}: Democratizing{billion-scale} model training. In 2021 USENIX Annual Technical Conference (USENIX ATC 21). 551–564

    work page 2021

  35. [36]

    DeepSpeed Team. 2022. GitHub Issue: Delayed parameter update in ZeRO-Offload. https://github.com/deepspeedai/DeepSpeed/discussions/2461

    work page 2022

  36. [37]

    HPCAI Tech. 2025. TensorNVMe: A Python library for transferring PyTorch tensors between CPU and NVMe. https://github.com/hpcaitech/TensorNVMe. Accessed: 2025-02-06

    work page 2025

  37. [38]

    Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic

  38. [40]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, et al

  39. [41]

    Jonathan Vanian and Kif Leswing. [n. d.]. ChatGPT and generative AI are booming, but the costs can be extraordinary. https://www.cnbc.com/2023/03/13/ chatgpt-and-generative-ai-are-booming-but-at-a-very-expensive-price.html. Accessed: 2023-03-15

    work page 2023

  40. [42]

    Alan Thompson. [n. d.]. Inside Large Language Models. https://lifearchitect.ai/ models. Accessed: 2025-04-12

    work page 2025

  41. [43]

    Guanhua Wang, Masahiro Tanaka, Xiaoxia Wu, Lok Chand Koppaka, Samyam Ra- jbhandari, Olatunji Ruwase, and Yuxiong He. 2023. DeepSpeed ZeRO-Offload++: 6x Higher Training Throughput via Collaborative CPU/GPU Twin-Flow

    work page 2023

  42. [44]

    Zhuang Wang, Zhen Jia, Shuai Zheng, Zhen Zhang, Xinwei Fu, TS Eugene Ng, and Yida Wang. 2023. Gemini: Fast failure recovery in distributed training with in- memory checkpoints. In Proceedings of the 29th Symposium on Operating Systems Principles. 364–381

    work page 2023

  43. [45]

    BigScience Workshop. 2023. BLOOM: A 176B-Parameter Open-Access Multilin- gual Language Model. arXiv:2211.05100 [cs]

    Pith/arXiv arXiv 2023

  44. [46]

    Gang Xian, Wenxiang Yang, Yusong Tan, Jinghua Feng, Yuqi Li, Jian Zhang, and Jie Yu. 2024. Mobilizing underutilized storage nodes via job path: A job-aware file striping approach. Parallel Comput. 121 (2024), 103095

    work page 2024

  45. [47]

    VAST Data. 2023. The VAST Data Platform Whitepaper. https://www.vastdata. com/whitepaper. Accessed: 2025-04-12

    work page 2023

  46. [48]

    Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Peng Zhang, Yuxiao Dong, and Jie Tang

    work page

  47. [49]

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 (2022)

    Pith/arXiv arXiv 2022

  48. [50]

    Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay. 2019. Deep learning based recom- mender system: A survey and new perspectives. ACM computing surveys (CSUR) (2019)

    work page 2019

  49. [51]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223 (2023)

    Pith/arXiv arXiv 2023

  50. [53]

    Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. 2023. PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel. Proc. VLDB Endow. 16, 12 (Aug. 2023), 3848–3860

    work page 2023

  51. [54]

    arXiv:2210.02414 [cs.CL] https://arxiv.org/abs/2210.02414

    GLM-130B: An Open Bilingual Pre-trained Model. arXiv:2210.02414 [cs.CL] https://arxiv.org/abs/2210.02414

    Pith/arXiv arXiv

  52. [58]

    Xia Zhao, Limin Wang, Yufei Zhang, Xuming Han, Muhammet Deveci, and Milan Parmar. 2024. A review of convolutional neural networks in computer vision. Artificial Intelligence Review 57, 4 (2024), 99

    work page 2024

  53. [2020]

    Advances in neural information processing systems 33 (2020), 1877–1901

    Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901

    work page 2020

  54. [2022]

    arXiv:2211.09085 [cs.CL] https://arxiv.org/abs/2211.09085

    Galactica: A Large Language Model for Science. arXiv:2211.09085 [cs.CL] https://arxiv.org/abs/2211.09085

    Pith/arXiv arXiv

  55. [2023]

    arXiv:2307.09288

    Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv:2307.09288

    Pith/arXiv arXiv

  56. [2024]

    arXiv:2405.18710 [cs.LG] https://arxiv.org/abs/2405.18710

    To FP8 and Back Again: Quantifying the Effects of Reducing Precision on LLM Training Stability. arXiv:2405.18710 [cs.LG] https://arxiv.org/abs/2405.18710

    Pith/arXiv arXiv