REVIEW 3 major objections 5 minor 21 references
NNTile: a machine learning framework capable of training extremely large GPT language models on a single node
T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read A task-based scheduler lets a single eight-GPU node train a 49.9-billion-parameter GPT2 model.
desk verdict A promising systems idea with open code, but the headline claim outstrips the experiments: the 49.9B result is for wide, shallow custom GPT2-style models, and the paper never shows a loss curve. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the tile-wise task graph with a sequential task flow: split each tensor along all axes into tiles, express every layer as tasks that read and write a few tiles, and let the runtime's scheduler map ready tasks onto any device while asynchronously moving tiles between GPU and CPU memory. The scheduler replaces the human decisions of parameter sharding and communication timing. The argument travels through this mechanism: the same tile-wise implementations are given for embedding, linear, layer norm, softmax, attention, cross-entropy, and Adam/SGD updates, so the entire training loop is a single graph of small memory-bound tasks.
What would settle it
Run the framework on a GPT2 with 12 or more transformer blocks at the same 49.9-billion-parameter scale on one eight-GPU node; if the training step does not complete at comparable throughput, the broad claim about extremely large GPT models rests on the shallow, wide configurations only.
Extended reading notes
Core claim
The central discovery is an application: the tile-based task-parallel paradigm, previously used for dense linear algebra, can carry transformer training, and doing so extends single-node model size far beyond the aggregate GPU memory. For an 8-layer GPT2 with varying embedding sizes on eight A100s, NNTile reaches 49.9 billion parameters before memory exhaustion, versus 10.6 billion for the sharded data-parallel baseline; for 4-layer models on four A100s the numbers are 25.5 and 6.7 billion. The paper traces this advantage to automatic data placement: tiles remain on a GPU while needed, are copied asynchronously between GPUs, and are offloaded to CPU RAM when idle, with the scheduler choosing where each task runs. Performance at a given model size is comparable to the baseline; the difference is capacity, not per-step speed.
Load-bearing premise
The 49.9-billion-parameter result was produced with custom GPT2 models that have only a few transformer blocks and very large embedding dimensions, because standard pretrained GPT2 configurations did not scale from one to eight GPUs in this framework.
Editorial extensions
If this is right
- Models larger than the aggregate GPU memory of a node can be trained without hand-written offload schedules, because tile tasks are placed and migrated automatically.
- The practical ceiling on single-node training moves from total GPU memory to scheduler quality; the paper reports that performance degrades as the number of transformer layers grows.
- The same tile-wise treatment of forward and backward passes covers a full list of standard components, so the approach could be assembled into other architectures, not just GPT2.
- Within the demonstrated configuration, a four-layer model reaches 25.5 billion parameters on four GPUs, indicating that the memory advantage appears at smaller node counts too.
Reading between the lines
- The paper reports that scheduling quality degrades as layers are added; the natural next step, which the authors leave implicit, is that a scheduler that plans across the whole training iteration rather than greedily is the direct lever for extending the single-node record to standard-depth models.
- Because the demonstrated advantage is the automatic use of CPU RAM, a fairer capacity comparison would enable CPU offload in the sharded data-parallel baseline as well; that comparison is not run here, so the 49.9-versus-10.6-billion factor is specific to the default offload-free baseline.
- The same tile-wise, scheduler-driven memory management should transfer to other memory-bound phases such as fine-tuning and long-context inference, though the paper only demonstrates training.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript introduces NNTile, a machine learning framework built on the StarPU task-based parallel programming library, and claims it can train extremely large GPT-style language models on a single multi-GPU node by using CPU RAM offloading and dynamic task scheduling. The central experimental result is that NNTile can train custom GPT2 configurations with up to 49.9 billion parameters on one node with 8 A100 GPUs, whereas PyTorch FSDP is limited to about 10.6 billion parameters in the same setting. The paper describes the task-based paradigm, the tile-based implementations of transformer layers, and reports throughput measurements rather than training dynamics.
Significance. The potential significance is substantial: if NNTile genuinely enabled training of very large GPT models on a single node, it would lower the hardware barrier for LLM research and demonstrate a novel application of task-based scheduling to neural network training. The open-source release and the idea of automatically using CPU RAM as a memory extension are positive contributions. However, the experiments cover only custom architectures with very few transformer blocks and very large embedding dimensions, and the absence of any convergence evidence means the headline claim is not yet established.
major comments (3)
- [Section 3, Section 4, Figures 1 and 2] The reported 49.9B-parameter result is obtained with custom GPT2 models containing only 8 transformer layers and very large embedding sizes. The manuscript states in Section 3 that standard pretrained GPT2 models did not scale from one to eight GPUs, and in Section 4 that 'the more layers are in a neural network, the worse performance we get with the NNTile.' These statements directly undermine the claim that NNTile can train 'extremely large GPT language models,' because the tested models are not representative of the GPT2 family's depth. Please either demonstrate training with a standard-depth GPT2 configuration (e.g., 12 or 24 layers) or re-scope the title and abstract to claim capability for wide, shallow transformer models with large embedding dimensions.
- [Section 4, Figures 1 and 2] The paper reports only throughput in Tflops/s and never shows loss curves, perplexity, or any convergence metric. 'Training' is inferred solely from executing forward and backward passes; no evidence demonstrates that the gradients are correct or that optimization actually reduces the loss. Please add at least one full training run with a loss curve and a comparison of the training dynamics against PyTorch FSDP to validate that the framework truly trains models rather than merely fitting them in memory.
- [Section 4] The experimental setup is under-specified. For each model size, the manuscript does not report sequence length, batch size, vocabulary size, number of attention heads, tile sizes, optimizer settings, or the exact model configuration behind each data point. Without these details, the parameter counts cannot be verified, the comparison with PyTorch FSDP is not reproducible, and the assertion that the models are 'similar' is not checkable. A table listing all hyperparameters and model architectures for every point in Figures 1 and 2 is needed.
minor comments (5)
- [Section 4] There is a typo in the sentence about the 8-layer model: 'dmdasd' appears after 'scheduling policy.'
- [Abstract] The abstract describes training in 'heterogeneous clusters,' but all experiments are conducted on a single node; please clarify that the reported results are single-node only.
- [References] Reference [1] is incomplete; it lacks a year, publisher, and access date, making it unverifiable.
- [Figures 1 and 2] The x-axis model sizes are not evenly spaced, which can visually distort scaling behavior; consider presenting the exact configurations in a table and using a log scale or annotated points.
- [Section 2] The memory-management description would benefit from a concrete example of how tiles are retained on a GPU across successive tasks, since this mechanism is central to the claimed advantage over standard offloading approaches.
Circularity Check
No circularity: the central claim is an empirical benchmark against the external PyTorch FSDP baseline; the single self-citation is background only and not load-bearing.
full rationale
This is an empirical systems paper, not a mathematical derivation. The headline result (NNTile training a 49.9B-parameter GPT2 model on one 8xA100 node versus 10.6B with PyTorch FSDP, Figures 1 and 2) is a direct measurement on fixed hardware and TensorFloat32 precision, compared against an independent external baseline. No parameter is fitted to the headline number, no predicted quantity is defined in terms of the model it is supposed to validate, and no uniqueness or self-citation theorem is invoked to force a design choice. The only self-citation is reference [13], a survey co-authored by some of the present authors, cited for background context ('A comprehensive survey on the main approaches to distributed training of large models is presented in [13]'), and it does not enter the experimental logic or support any quantitative claim. The paper's own limitation statements—that standard pretrained GPT2 models 'did not scale from a single GPU to 8 GPUs' and that 'the more layers are in a neural network, the worse performance we get with the NNTile'—are concerns about external validity and architecture coverage, not circularity. No self-definitional step, fitted input renamed as prediction, or load-bearing self-citation is present, so the paper is self-contained against the external benchmark it reports.
Assumptions & free parameters
free parameters (1)
- Tile size configuration =
not reported
assumptions (4)
- ad hoc to paper Custom GPT2 models with few transformer blocks and large embedding sizes are representative of the GPT2 family.
- domain assumption StarPU greedy scheduling policies are sufficiently effective for transformer training.
- domain assumption Throughput in Tflops/s is an adequate proxy for the ability to train a model.
- domain assumption The PyTorch FSDP configuration used for comparison is fair and properly tuned.
Cite this review
Pith. "Pith review of NNTile: a machine learning framework capable of training extremely large GPT language models on a single node." pith.science (2026). https://pith.science/paper/TYRL7RZU
@misc{pith2026250413236,
author = {Pith},
title = {Pith review of: NNTile: a machine learning framework capable of training extremely large GPT language models on a single node},
year = {2026},
howpublished = {\url{https://pith.science/paper/TYRL7RZU}},
note = {Machine review of arXiv:2504.13236}
}
read the original abstract
This study presents an NNTile framework for training large deep neural networks in heterogeneous clusters. The NNTile is based on a StarPU library, which implements task-based parallelism and schedules all provided tasks onto all available processing units (CPUs and GPUs). It means that a particular operation, necessary to train a large neural network, can be performed on any of the CPU cores or GPU devices, depending on automatic scheduling decisions. Such an approach shifts the burden of deciding where to compute and when to communicate from a human being to an automatic decision maker, whether a simple greedy heuristic or a complex AI-based software. The performance of the presented tool for training large language models is demonstrated in extensive numerical experiments.
Figures
Reference graph
Works this paper leans on
-
[1]
Artificial intelligence (ai) market by offering (hardware, software), technology (ml (deep learning (llm, trans- formers (gpt 1, 2, 3, 4)), nlp, computer vision), business function, vertical, and region - global forecast to 2030
-
[2]
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017
work page 2017
- [3]
-
[4]
J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L.A. Hendricks, J. Welbl, A. Clark, et al. An empirical analysis of compute-optimal large language model training. Advances in Neural Information Processing Systems, 35:30016–30030, 2022
work page 2022
-
[5]
J. Rasley, S. Rajbhandari, O. Ruwase, and Y . He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3505–3506, 2020
work page 2020
-
[6]
Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale
Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, et al. Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale. In SC22: International Conference for High Performance Comput- ing, Networking, Storage and Analysis, pa...
work page 2022
-
[7]
M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019
arXiv 1909
- [8]
Show all 21 references
-
[9]
Korthikanti, J
V .A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro. Reducing activa- tion recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5, 2023
2023
-
[10]
Jiang, H
Z. Jiang, H. Lin, Y . Zhong, Q. Huang, Y . Chen, Z. Zhang, Y . Peng, X. Li, C. Xie, S. Nong, et al. Megascale: Scaling large language model training to more than 10,000 gpus. In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), pages 745–760, 2024
2024
-
[11]
Pytorch: An imperative style, high-performance deep learning library
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32,...
2019
-
[12]
Y . Zhao, A. Gu, R. Varma, L. Luo, C.-C. Huang, M. Xu, L. Wright, H. Shojanazeri, M. Ott, S. Shleifer, et al. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.11277, 2023
2023 arXiv
-
[13]
Survey on efficient training of large neural networks
Julia Gusak, Daria Cherniuk, Alena Shilova, Alexandr Katrutsa, Daniel Bershatsky, Xunyi Zhao, Lionel Eyraud- Dubois, Oleh Shliazhko, Denis Dimitrov, Ivan Oseledets, et al. Survey on efficient training of large neural networks. In Proceedings of the 31st International Joint Con...
2022
-
[14]
Almazrouei, H
E. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, ´E. Goffinet, D. Hesslow, J. Launay, Q. Malartic, et al. The falcon series of open language models. arXiv preprint arXiv:2311.16867 , 2023
2023 arXiv
-
[15]
StarPU: A Unified Plat- form for Task Scheduling on Heterogeneous Multicore Architectures
C ´edric Augonnet, Samuel Thibault, Raymond Namyst, and Pierre-Andr ´e Wacrenier. StarPU: A Unified Plat- form for Task Scheduling on Heterogeneous Multicore Architectures. CCPE - Concurrency and Computation: Practice and Experience, Special Issue: Euro-Par 2009, 23:187–198, 2011
2009
-
[16]
A hybridization methodology for high-performance linear algebra software for gpus
Emmanuel Agullo, C ´edric Augonnet, Jack Dongarra, Hatem Ltaief, Raymond Namyst, Samuel Thibault, and Stanimire Tomov. A hybridization methodology for high-performance linear algebra software for gpus. In GPU Computing Gems Jade Edition, pages 473–484. Elsevier, 2012
2012
-
[17]
Layer normalization
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016
2016 arXiv
-
[18]
On layer normalization in the transformer architecture
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. InInternational Conference on Machine Learning, pages 10524–10533. PMLR, 2020
2020
-
[19]
Accelerated optimization for machine learning
Zhouchen Lin, Huan Li, and Cong Fang. Accelerated optimization for machine learning. Nature Singapore: Springer, 2020
2020
-
[20]
Adam: A method for stochastic optimization
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014
2014 arXiv
-
[21]
Fixing weight decay regularization in Adam
Ilya Loshchilov, Frank Hutter, et al. Fixing weight decay regularization in Adam. arXiv preprint arXiv:1711.05101, 5, 2017. 6
2017 arXiv
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.