An Efficient Heterogeneous Co-Design for Fine-Tuning on a Single GPU

RuiJia Yang; Zeyi Wen

REVIEW 3 major objections 6 minor 37 references

A sliding-window GPU co-design lets one consumer GPU fine-tune 123B+ language models by overlapping compute with CPU updates and multi-tier I/O while roughly halving memory use.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 23:46 UTC pith:ICJQ3TBJ

load-bearing objection Solid single-GPU co-design that actually ships code and multi-vendor numbers; the headline speedups are real under the overlap regime the paper itself measures, not a free lunch at every batch size. the 3 major comments →

arxiv 2603.16428 v2 pith:ICJQ3TBJ submitted 2026-03-17 cs.DC cs.AI

An Efficient Heterogeneous Co-Design for Fine-Tuning on a Single GPU

Ruijia Yang , Zeyi Wen This is my paper

classification cs.DC cs.AI

keywords LLM fine-tuningsingle-GPU trainingheterogeneous memory managementoffloadingasynchronous enginelayer slidingGPUDirect StorageTriton kernels

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Full-parameter fine-tuning of large language models usually needs more VRAM than a single GPU holds, so individuals without clusters are locked out. SlideFormer treats the GPU as a sliding window of active layers and runs a lightweight asynchronous engine that overlaps GPU backward passes with CPU optimizer updates and transfers across RAM and NVMe. Pre-allocated cache units and shared host buffers keep the memory footprint fixed and small; fused kernels remove the large-vocabulary loss bottleneck. The result is that models larger than 123B can be fine-tuned on a single RTX 4090, batch sizes grow up to eightfold, and throughput stays within a few percent of non-offload peak while cutting both GPU and CPU memory roughly in half versus prior offloaders. A sympathetic reader cares because the design turns ordinary high-end PCs into practical full-parameter fine-tuning machines instead of forcing reliance on clusters or parameter-efficient shortcuts.

Core claim

Holistic heterogeneous co-design—layer-sliding asynchronous engine, pre-allocated multi-tier memory, and fused kernels—makes full-parameter LLM fine-tuning practical on one GPU, supporting 123B+ models, up to 8 imes larger batches and 6 imes larger models, 1.40–6.27 imes higher throughput, and roughly half the CPU/GPU memory of prior offloaders while sustaining >95 % of non-offload peak on both NVIDIA and AMD hardware.

What carries the argument

Layer-Sliding Architecture: a fixed queue of pre-allocated GPU cache units holds only a single active layer at a time; a thread-based engine pipelines that layer’s backward compute with asynchronous gradient offload, CPU Adam update, and parameter prefetch so the GPU never waits for the host when the hiding factor η ≥ 1.

Load-bearing premise

Near-peak speed requires that a single layer’s backward time is long enough to hide the sum of gradient transfer and CPU update time at practical batch sizes; if the CPU or interconnect cannot keep up, the claimed utilization and speedups disappear.

What would settle it

On a given GPU–CPU–PCIe setup, measure wall-clock step time while increasing batch size until the reported hiding factor reaches 1; if throughput remains well below non-offload peak or GPU utilization stays under 90 % after that point, the overlap claim fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Solid single-GPU co-design that actually ships code and multi-vendor numbers; the headline speedups are real under the overlap regime the paper itself measures, not a free lunch at every batch size.

read the letter

This is a practical systems paper that does what it claims on the hardware it shows. The new piece is not another offload slogan; it is a tight single-GPU co-design: layer-granular sliding window with a pre-allocated cache-unit queue, a thread-based (not multi-process) async engine, host shared buffers for grads and type conversion, fused LCE, and optional GDS/NVMe for activations and optimizer states. They keep the active GPU footprint to roughly one layer, cut peak CPU/GPU memory roughly in half versus ZeRO-Offload/Infinity and ColossalAI, and push max model size past 123B on a 4090 when CPU RAM + NVMe are available. Code is public and they report both NVIDIA and AMD numbers, which is more than most of this literature does.

What works: the memory layout and the LCE kernel are the real enablers for large batch and large vocab. Table 1 and Figure 4 make the overlap condition explicit (η ≥ 1 once batch is large enough that T_bwd covers d2h + CPU Adam). Figures 7–9 and 11 give the expected curves: throughput climbs once the pipeline is compute-bound, CPU memory stays low because of the shared buffers, and NVMe scaling is near-linear with drive count. The comparison to LoHan under a standard GPT-2 loss is fairer than LoHan’s own MSE evaluation. Citation pattern is appropriate; they sit on ZeRO, LoHan, Stronghold, Gemini, FlashAttention, etc., without pretending to invent offloading.

Soft spots, in proportion: the stress-test is right that η < 1 at small batch (Table 1, BS=16 on 4090) and that throughput is measured on synthetic fixed-length data. That means the “>95% peak” and the upper end of the 1.4–6.27× range are regime-dependent, not universal. They do not hide this; the text and figures show the transition. Missing pieces for a full story are variance/error bars, one real-task quality check, and clearer baseline config dumps so others can re-measure. None of that breaks the central engineering claim.

Who it is for: people who actually need full-parameter fine-tuning on one consumer or single-server GPU and care about memory envelopes more than asymptotic theory. It deserves a serious referee. I would bring it to reading group and I would cite the system numbers if I were writing on single-device training. Accept for review; ask for variance, one real workload, and a short note on when η drops below 1.

Referee Report

3 major / 6 minor

Summary. SlideFormer is a single-GPU full-parameter fine-tuning system that co-designs a layer-sliding asynchronous engine (CUDA streams + CPU threads), pre-allocated GPU cache-unit queues with host shared gradient/convert buffers, sliding activation checkpointing, Layer-Adam, GPUDirect Storage NVMe tiering for activations/optimizer states, and fused Triton kernels (notably LinearCrossEntropy). The central claim is that this co-design enables fine-tuning of 123B+ models on one RTX 4090, supports up to 8× larger batches and 6× larger models than baselines, delivers 1.40–6.27× higher throughput while roughly halving CPU/GPU memory versus ZeRO-Offload/Infinity, ColossalAI, and LoHan, and sustains >95% of non-offload peak TFLOPS on NVIDIA and AMD GPUs. Evidence includes multi-platform measurements (RTX 4090, A100, RX 7900XT), multi-model scaling (Llama-3.1-8B, Qwen2.5 3B–72B, Mistral up to 123B), timeline breakdowns (Table 1), memory/batch curves, and NVMe scaling (Figure 11).

Significance. If the results hold under realistic workloads, the paper is a substantial systems contribution: it turns the widening CPU–GPU memory gap into a practical single-GPU full-parameter fine-tuning path and substantially raises the model/batch size frontier on consumer hardware. Strengths that raise the bar include open code, multi-vendor GPU results (NVIDIA + AMD), explicit overlap analysis (η, critical batch size), and concrete memory co-design (shared buffers, pre-allocated units, fused LCE) rather than pure scheduling. The work is complementary to PEFT and multi-GPU systems and is of clear interest to the systems and LLM-training communities.

major comments (3)

§4.1 and Figures 7–8, 10: All throughput and “>95% peak” claims are measured on a synthetic dataset with fixed sequence length 1024 and “stable effective length.” Real fine-tuning corpora have variable packing, padding, and loss-mask patterns that change activation traffic, CrossEntropy behavior, and effective T_bwd. Without at least one real-corpus (or packed) ablation at the same sequence length, the headline 1.40–6.27× and >95% peak numbers remain under-supported for the practical regimes the abstract advertises.
§3.1, Table 1, Figure 4: The lossless-overlap condition T_bwd ≥ T_grad_d2h + T_update (η ≥ 1) is load-bearing for the near-peak claim, yet Table 1 reports η = 0.66 at BS=16 on RTX 4090 for Qwen2.5-14B (update-bound). The paper correctly notes the compute-bound transition near BS=32, but the abstract and §4.2 still present >95% peak and large speedups without systematically reporting throughput/η at the small-to-medium batch sizes users often run under memory pressure. A short sensitivity table (η and tokens/s vs. batch size and layer size, including NVMe optimizer offload) is needed so the claimed regime is unambiguous.
§4.2–4.6 baselines: Fairness of the comparison is not fully documented. It is unclear whether ZeRO-Offload/Infinity, ColossalAI, and LoHan were given the same fused LCE / Triton kernels, identical activation-checkpointing policy, and the same host-memory layout optimizations, or whether part of the 1.4–6.3× gap is kernel/memory-layout advantage rather than the layer-sliding engine. A short “equal-kernel” or component ablation (engine only vs. kernels only vs. full system) would isolate the co-design contribution that the title emphasizes.

minor comments (6)

Abstract vs. body: abstract states 1.40×–6.27×; §4.2 text cites 1.39×, 2.82×, 6.34× for Llama-3.1-8B. Align the numbers.
Figure 3 caption/labels: “40%” improvement annotation is not derived in the text; either compute it from the timelines or remove the hard-coded percentage.
§3.2: “reduces the gradient footprint on CPU memory (2N bytes) to 1/num_layers” is slightly imprecise (peak shared buffer is one layer, not a permanent 1/n reduction of the logical gradient store). Clarify peak vs. logical footprint.
§2.3 / §4.6: LoHan comparison is valuable but limited to GPT-2 + non-standard MSE in the original work; state more clearly that the re-run uses standard CE so readers do not over-generalize.
Typos/notation: “Param Group” vs. param-group; occasional missing spaces before citations; “LCE” introduced before full expansion in one place.
Figure 11: SSD-count scaling is useful; add error bars or repeat counts if available, and state NVMe model/bandwidth so the linear scaling claim is reproducible.

Circularity Check

0 steps flagged

No circularity: empirical systems paper whose performance claims are measured wall-clock results, not algebraic restatements of fitted inputs or self-definitional identities.

full rationale

SlideFormer is a systems/engineering paper. Its central claims (throughput multipliers, memory reductions, max model/batch sizes, >95% of non-offload peak, η values) are obtained by running the implemented engine against baselines on concrete hardware and reporting measured tokens/s, TFLOPS, and peak memory. The Layer-Sliding schedule, pre-allocated cache units, shared host buffers, GDS/NVMe tiering, and fused LCE kernel are design choices whose correctness is demonstrated by experiment, not derived from equations that reduce to their own premises. The overlap condition T_bwd ≥ T_grad_d2h + T_update and the hiding factor η are definitions used to interpret timelines (Table 1, Figure 4); they are not fitted parameters later re-labeled as predictions, nor are they justified solely by self-citation. No uniqueness theorem, ansatz, or prior result by the same authors is load-bearing for the performance numbers. Consequently the derivation chain contains no self-definitional, fitted-as-prediction, or self-citation circular steps.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 3 invented entities

Load-bearing content is systems design plus empirical measurement, not a closed-form derivation. Free parameters are engineering knobs (batch size for overlap, offload fractions, single-layer window). Axioms are standard LLM training facts and the paper’s design premises. Invented entities are architectural components introduced to realize the co-design; none are physical particles or forces, and independent evidence is the open-source implementation plus reported benchmarks rather than external physical prediction.

free parameters (3)

critical batch size for full backward overlap
Chosen/observed per model and GPU so that T_bwd masks d2h+update (Figure 4, Table 1); the >95% peak claim depends on operating at or above this regime.
optimizer-state offload fraction (0 / 50% / 100%)
User-selected NVMe offload fraction that trades CPU memory for throughput (Figure 11); not derived from first principles.
active GPU window size = one layer
Design choice argued as optimal for memory-constrained single-GPU settings; multi-layer windows are rejected without exhaustive search over window sizes.

axioms (4)

domain assumption Transformer blocks are the natural repeating unit; serial layer compute makes multi-layer GPU windows waste VRAM without throughput gain.
Section 3.1 rationale for layer granularity versus param-group designs in ZeRO/LoHan.
domain assumption Mixed-precision Adam memory model Mem ≈ 2N+2N+4N+8N + O(n·h·s·b) (Eq. 1) correctly dominates single-GPU pressure.
Section 2.1; standard but load-bearing for why optimizer/activation offload is the right target.
ad hoc to paper Parameters should not be placed on NVMe because PCIe-bound h2d/d2h is required for compute overlap; only activations and optimizer states may use the NVMe tier.
Section 3.3 'Why Not Offload Parameters'; design axiom that caps max model size relative to host RAM unless activations/OS absorb the rest.
ad hoc to paper Thread-based CUDA-stream + CPU-thread engine has lower overhead than multi-process IPC optimizers for single-GPU async updates.
Section 3.1 contrast with LoHan; underpins the lightweight-engine novelty claim.

invented entities (3)

Layer-Sliding Architecture / sliding GPU cache-unit queue independent evidence
purpose: Keep a fixed, fragment-free GPU footprint while streaming layers for compute and offload.
Core scheduling abstraction of the paper; evidence is implementation and memory curves, not external theory.
Layer-Adam with host shared grad/convert buffers independent evidence
purpose: Update one layer at a time on CPU while cutting peak host memory for gradients and BF16/FP16 copies.
Custom optimizer layout claimed to cut CPU memory ~25–40% versus prior layouts.
Sliding activation checkpointing to CPU/NVMe independent evidence
purpose: Bound activation VRAM to a small window via async offload/prefetch.
Modified checkpointing path integrated with the same pipeline.

pith-pipeline@v1.1.0-grok45 · 17778 in / 3525 out tokens · 44652 ms · 2026-07-13T23:46:51.075451+00:00 · methodology

0 comments

read the original abstract

Fine-tuning Large Language Models (LLMs) has become essential for domain adaptation, but its memory-intensive property exceeds the capabilities of most GPUs. To address this challenge and democratize LLM fine-tuning, we present SlideFormer, a novel system designed for single-GPU environments. Our innovations are: (1) A lightweight asynchronous engine that treats the GPU as a sliding window and overlaps GPU computation with CPU updates and multi-tier I/O. (2) A highly efficient heterogeneous memory management scheme significantly reduces peak memory usage. (3) Optimized Triton kernels to solve key bottlenecks and integrated advanced I/O. This collaborative design enables fine-tuning of the latest 123B+ models on a single RTX 4090, supporting up to 8x larger batch sizes and 6x larger models. In evaluations, SlideFormer achieves 1.40x to 6.27x higher throughput while roughly halving CPU/GPU memory usage compared to baselines, sustaining >95% peak performance on both NVIDIA and AMD GPUs.The code is available at https://github.com/RegiaYoung/SlideFormer.

Figures

Figures reproduced from arXiv: 2603.16428 by RuiJia Yang, Zeyi Wen.

**Figure 4.** Figure 4: Critical batch size for achieving full backward over [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗

**Figure 5.** Figure 5: Computation-communication overlap during back [PITH_FULL_IMAGE:figures/full_fig_p003_5.png] view at source ↗

**Figure 6.** Figure 6: Memory usage and execution time comparison be [PITH_FULL_IMAGE:figures/full_fig_p004_6.png] view at source ↗

**Figure 8.** Figure 8: Throughput and CPU memory comparison between SlideFormer and baselines for various sizes of Qwen2.5 on RTX4090. 4 8 16 32 64 Batch Size 8 12 16 20 GPU Allocated Memory (GB) ColossalAI ZeRO-Offload SlideFormer [PITH_FULL_IMAGE:figures/full_fig_p005_8.png] view at source ↗

**Figure 10.** Figure 10: The fine-tuning throughput of Qwen2.5 in various [PITH_FULL_IMAGE:figures/full_fig_p005_10.png] view at source ↗

**Figure 12.** Figure 12: Maximum trainable model size For example, an individual with a PC with 128GB CPU memory can fine-tune the Llama-3.1-8B model on a single RTX 4080 GPU. This is achievable on a single GPU without resorting to NVMe offloading, while maintaining nearly lossless throughput compared to non-offloaded training. This capability is a cornerstone of our goal to democratize access to large model fine-tuning. 4.4 Anal… view at source ↗

**Figure 13.** Figure 13: Throughput and memory comparison between [PITH_FULL_IMAGE:figures/full_fig_p006_13.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

37 extracted references · 1 canonical work pages

[1]

Aaron Grattafiori

et al. Aaron Grattafiori. 2024. The Llama 3 Herd of Models. arXiv:2407.21783 [cs.AI] https://arxiv.org/abs/2407.21783

Pith/arXiv arXiv 2024
[2]

Mistral AI. 2024. Mistral-Large-Instruct-2411. https://huggingface.co/mistralai/ Mistral-Large-Instruct-2411

2024
[3]

Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. 2016. Training deep nets with sublinear memory cost.arXiv preprint arXiv:1604.06174(2016)

Pith/arXiv arXiv 2016
[4]

NVIDIA Corporation. 2021. NVIDIA GPUDirect Storage: Benchmarking and Configuration Guide. https://docs.nvidia.com/gpudirect-storage/

2021
[5]

NVIDIA Corporation. 2025. CUDA Runtime API: Stream Management. https: //docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__STREAM.html

2025
[6]

Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Flashat- tention: Fast and memory-efficient exact attention with io-awareness.Advances in neural information processing systems35 (2022), 16344–16359

2022
[7]

Jiarui Fang and Yang You. 2022. Meet Gemini: The Heterogeneous Memory Manager of Colossal-AI. https://colossalai.org/docs/advanced_tutorials/meet_ gemini/

2022
[8]

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. InInternational conference on machine learning. PMLR, 2790–2799

2019
[9]

Pin-Lun Hsu, Yun Dai, Vignesh Kothapalli, Qingquan Song, Shao Tang, Siyu Zhu, Steven Shimizu, Shivam Sahni, Haowen Ning, and Yanning Chen. 2024. Liger Kernel: Efficient Triton Kernels for LLM Training.arXiv preprint arXiv:2410.10989 (2024). arXiv:2410.10989 [cs.LG] https://arxiv.org/abs/2410.10989

Pith/arXiv arXiv 2024
[10]

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large language models.ICLR1, 2 (2022), 3

2022
[11]

Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al . 2019. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems32 (2019)

2019
[12]

Hongsun Jang, Jaeyong Song, Jaewon Jung, Jaeyoung Park, Youngsok Kim, and Jinho Lee. 2024. Smart-infinity: Fast large language model training using near- storage processing on a real system. In2024 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE, 345–360

2024
[13]

Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic opti- mization.arXiv preprint arXiv:1412.6980(2014)

Pith/arXiv arXiv 2014
[14]

Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, Daniel Haziza, Luca Wehrstedt, Jeremy Reizenstein, and Grigory Sizov
[15]

https: //github.com/facebookresearch/xformers

xFormers: A modular and hackable Transformer modelling library. https: //github.com/facebookresearch/xformers
[16]

Shenggui Li, Hongxin Liu, Zhengda Bian, Jiarui Fang, Haichen Huang, Yuliang Liu, Boxiang Wang, and Yang You. 2023. Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training. InProceedings of the 52nd International Conference on Parallel Processing(Salt Lake City, UT, USA)(ICPP ’23). Association for Computing Machinery, New York, NY, US...

work page doi:10.1145/3605573 2023
[17]

Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, et al. 2020. Pytorch distributed: Experiences on accelerating data parallel training.arXiv preprint arXiv:2006.15704(2020)

Pith/arXiv arXiv 2020
[18]

Changyue Liao, Mo Sun, Zihan Yang, Jun Xie, Kaiqi Chen, Binhang Yuan, Fei Wu, and Zeke Wang. 2024. LoHan: Low-Cost High-Performance Framework to Fine-Tune 100B Model on a Consumer GPU. arXiv:2403.06504 [cs.DC] https: //arxiv.org/abs/2403.06504

Pith/arXiv arXiv 2024
[19]

Qijun Luo, Hengxu Yu, and Xiao Li. 2024. BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Models. InAdvances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37. Curran Associates, Inc., 24926–24958. https://proceedings.neurips.cc/paper_f...

2024
[20]

Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. 2022. PEFT: State-of-the-art Parameter-Efficient Fine-Tuning methods. https://github.com/huggingface/peft

2022
[21]

Ben Mann, N Ryder, M Subbiah, J Kaplan, P Dhariwal, A Neelakantan, P Shyam, G Sastry, A Askell, S Agarwal, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.141651 (2020), 3

Pith/arXiv arXiv 2020
[22]

Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. 2017. Mixed precision training.arXiv preprint arXiv:1710.03740 (2017)

Pith/arXiv arXiv 2017
[23]

Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, and Matei Zaharia. 2019. PipeDream: Generalized pipeline parallelism for DNN training. InProceedings of the 27th ACM symposium on operating systems principles. 1–15

2019
[24]

NVIDIA. [n. d.]. NVIDIA/NeMo: A Scalable Generative AI Framework Built for Researchers and Developers Working on Large Language Models, Multimodal, and Speech AI (Automatic Speech Recognition and Text-to-Speech). https: //github.com/NVIDIA/NeMo. Accessed: May 15, 2025, n.d

2025
[25]

NVIDIA. 2024. Transformer Engine: A Library for Accelerating Transformer Models on NVIDIA GPUs. https://github.com/NVIDIA/TransformerEngine. Ver- sion 2.1.0, accessed on 2025-04-23

2024
[26]

Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li,...

Pith/arXiv arXiv 2025
[27]

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog1, 8 (2019), 9

2019
[28]

Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. InSC20: Inter- national Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 1–16

2020
[29]

Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, and Yuxiong He. 2021. Zero-infinity: Breaking the gpu memory wall for extreme scale deep learning. InProceedings of the international conference for high performance computing, networking, storage and analysis. 1–14

2021
[30]

Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. 2021. Zero-offload: Democratizing Billion-Scale Model Training. In2021 USENIX Annual Technical Conference (USENIX ATC 21). 551–564

2021
[31]

Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism.arXiv preprint arXiv:1909.08053(2019)

Pith/arXiv arXiv 2019
[32]

Reece Shuttleworth, Jacob Andreas, Antonio Torralba, and Pratyusha Sharma. 2025. LoRA vs Full Fine-tuning: An Illusion of Equivalence. arXiv:2410.21228 [cs.LG] https://arxiv.org/abs/2410.21228

arXiv 2025
[33]

Xiaoyang Sun, Wei Wang, Shenghao Qiu, Renyu Yang, Songfang Huang, Jie Xu, and Zheng Wang. 2022. Stronghold: fast and affordable billion-scale deep learning model training. InSC22: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 1–17

2022
[34]

Philippe Tillet, H. T. Kung, and David Cox. 2019. Triton: an intermediate lan- guage and compiler for tiled neural network computations. InProceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Program- ming Languages(Phoenix, AZ, USA)(MAPL 2019). Association for Computing Machinery, New York, NY, USA, 10–19. doi:10.1145/3315508.3329973

doi:10.1145/3315508.3329973 2019
[35]

Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners.arXiv preprint arXiv:2109.01652(2021)

Pith/arXiv arXiv 2021
[36]

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement De- langue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of...

2020
[37]

Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. 2024. Galore: Memory-efficient llm training by gradient low-rank projection.arXiv preprint arXiv:2403.03507(2024)

Pith/arXiv arXiv 2024

[1] [1]

Aaron Grattafiori

et al. Aaron Grattafiori. 2024. The Llama 3 Herd of Models. arXiv:2407.21783 [cs.AI] https://arxiv.org/abs/2407.21783

Pith/arXiv arXiv 2024

[2] [2]

Mistral AI. 2024. Mistral-Large-Instruct-2411. https://huggingface.co/mistralai/ Mistral-Large-Instruct-2411

2024

[3] [3]

Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. 2016. Training deep nets with sublinear memory cost.arXiv preprint arXiv:1604.06174(2016)

Pith/arXiv arXiv 2016

[4] [4]

NVIDIA Corporation. 2021. NVIDIA GPUDirect Storage: Benchmarking and Configuration Guide. https://docs.nvidia.com/gpudirect-storage/

2021

[5] [5]

NVIDIA Corporation. 2025. CUDA Runtime API: Stream Management. https: //docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__STREAM.html

2025

[6] [6]

Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Flashat- tention: Fast and memory-efficient exact attention with io-awareness.Advances in neural information processing systems35 (2022), 16344–16359

2022

[7] [7]

Jiarui Fang and Yang You. 2022. Meet Gemini: The Heterogeneous Memory Manager of Colossal-AI. https://colossalai.org/docs/advanced_tutorials/meet_ gemini/

2022

[8] [8]

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. InInternational conference on machine learning. PMLR, 2790–2799

2019

[9] [9]

Pin-Lun Hsu, Yun Dai, Vignesh Kothapalli, Qingquan Song, Shao Tang, Siyu Zhu, Steven Shimizu, Shivam Sahni, Haowen Ning, and Yanning Chen. 2024. Liger Kernel: Efficient Triton Kernels for LLM Training.arXiv preprint arXiv:2410.10989 (2024). arXiv:2410.10989 [cs.LG] https://arxiv.org/abs/2410.10989

Pith/arXiv arXiv 2024

[10] [10]

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large language models.ICLR1, 2 (2022), 3

2022

[11] [11]

Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al . 2019. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems32 (2019)

2019

[12] [12]

Hongsun Jang, Jaeyong Song, Jaewon Jung, Jaeyoung Park, Youngsok Kim, and Jinho Lee. 2024. Smart-infinity: Fast large language model training using near- storage processing on a real system. In2024 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE, 345–360

2024

[13] [13]

Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic opti- mization.arXiv preprint arXiv:1412.6980(2014)

Pith/arXiv arXiv 2014

[14] [14]

Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, Daniel Haziza, Luca Wehrstedt, Jeremy Reizenstein, and Grigory Sizov

[15] [15]

https: //github.com/facebookresearch/xformers

xFormers: A modular and hackable Transformer modelling library. https: //github.com/facebookresearch/xformers

[16] [16]

Shenggui Li, Hongxin Liu, Zhengda Bian, Jiarui Fang, Haichen Huang, Yuliang Liu, Boxiang Wang, and Yang You. 2023. Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training. InProceedings of the 52nd International Conference on Parallel Processing(Salt Lake City, UT, USA)(ICPP ’23). Association for Computing Machinery, New York, NY, US...

work page doi:10.1145/3605573 2023

[17] [17]

Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, et al. 2020. Pytorch distributed: Experiences on accelerating data parallel training.arXiv preprint arXiv:2006.15704(2020)

Pith/arXiv arXiv 2020

[18] [18]

Changyue Liao, Mo Sun, Zihan Yang, Jun Xie, Kaiqi Chen, Binhang Yuan, Fei Wu, and Zeke Wang. 2024. LoHan: Low-Cost High-Performance Framework to Fine-Tune 100B Model on a Consumer GPU. arXiv:2403.06504 [cs.DC] https: //arxiv.org/abs/2403.06504

Pith/arXiv arXiv 2024

[19] [19]

Qijun Luo, Hengxu Yu, and Xiao Li. 2024. BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Models. InAdvances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37. Curran Associates, Inc., 24926–24958. https://proceedings.neurips.cc/paper_f...

2024

[20] [20]

Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. 2022. PEFT: State-of-the-art Parameter-Efficient Fine-Tuning methods. https://github.com/huggingface/peft

2022

[21] [21]

Ben Mann, N Ryder, M Subbiah, J Kaplan, P Dhariwal, A Neelakantan, P Shyam, G Sastry, A Askell, S Agarwal, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.141651 (2020), 3

Pith/arXiv arXiv 2020

[22] [22]

Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. 2017. Mixed precision training.arXiv preprint arXiv:1710.03740 (2017)

Pith/arXiv arXiv 2017

[23] [23]

Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, and Matei Zaharia. 2019. PipeDream: Generalized pipeline parallelism for DNN training. InProceedings of the 27th ACM symposium on operating systems principles. 1–15

2019

[24] [24]

NVIDIA. [n. d.]. NVIDIA/NeMo: A Scalable Generative AI Framework Built for Researchers and Developers Working on Large Language Models, Multimodal, and Speech AI (Automatic Speech Recognition and Text-to-Speech). https: //github.com/NVIDIA/NeMo. Accessed: May 15, 2025, n.d

2025

[25] [25]

NVIDIA. 2024. Transformer Engine: A Library for Accelerating Transformer Models on NVIDIA GPUs. https://github.com/NVIDIA/TransformerEngine. Ver- sion 2.1.0, accessed on 2025-04-23

2024

[26] [26]

Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li,...

Pith/arXiv arXiv 2025

[27] [27]

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog1, 8 (2019), 9

2019

[28] [28]

Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. InSC20: Inter- national Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 1–16

2020

[29] [29]

Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, and Yuxiong He. 2021. Zero-infinity: Breaking the gpu memory wall for extreme scale deep learning. InProceedings of the international conference for high performance computing, networking, storage and analysis. 1–14

2021

[30] [30]

Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. 2021. Zero-offload: Democratizing Billion-Scale Model Training. In2021 USENIX Annual Technical Conference (USENIX ATC 21). 551–564

2021

[31] [31]

Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism.arXiv preprint arXiv:1909.08053(2019)

Pith/arXiv arXiv 2019

[32] [32]

Reece Shuttleworth, Jacob Andreas, Antonio Torralba, and Pratyusha Sharma. 2025. LoRA vs Full Fine-tuning: An Illusion of Equivalence. arXiv:2410.21228 [cs.LG] https://arxiv.org/abs/2410.21228

arXiv 2025

[33] [33]

Xiaoyang Sun, Wei Wang, Shenghao Qiu, Renyu Yang, Songfang Huang, Jie Xu, and Zheng Wang. 2022. Stronghold: fast and affordable billion-scale deep learning model training. InSC22: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 1–17

2022

[34] [34]

Philippe Tillet, H. T. Kung, and David Cox. 2019. Triton: an intermediate lan- guage and compiler for tiled neural network computations. InProceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Program- ming Languages(Phoenix, AZ, USA)(MAPL 2019). Association for Computing Machinery, New York, NY, USA, 10–19. doi:10.1145/3315508.3329973

doi:10.1145/3315508.3329973 2019

[35] [35]

Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners.arXiv preprint arXiv:2109.01652(2021)

Pith/arXiv arXiv 2021

[36] [36]

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement De- langue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of...

2020

[37] [37]

Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. 2024. Galore: Memory-efficient llm training by gradient low-rank projection.arXiv preprint arXiv:2403.03507(2024)

Pith/arXiv arXiv 2024