{"id":"e3428f1e-c14b-4c69-90e4-9576189445a8","arxiv_id":"2504.13236","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"NNTile trains a custom 49.9-billion-parameter GPT-2-style model on a single 8-GPU node, versus about 10.6 billion with PyTorch FSDP.","lead":"The paper presents NNTile, a software framework that uses task-based scheduling to train very large GPT-style language models on a single computer with multiple GPUs, while also using the computer's main memory. A smart generalist might read it to learn that a 50-billion-parameter model can fit on one node, while a standard PyTorch approach runs out of memory around 10 billion parameters.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 49.9B-parameter result uses custom 8-layer, huge-embedding GPT2 models; standard deep GPT2 failed to scale, so the headline claim is not established for realistic LLM architectures.","rationale":"The reader's weakest assumption matches the main soft spot: the benchmark swaps standard deep GPT2 for custom shallow models, so the headline claim is not validated for realistic large language models. The paper itself provides the decisive admission in Section 3, and Section 4 reports worsening performance with layer count. This is not an internal inconsistency, but an empirical gap between the claim and the experimental configuration. The proposed concrete test directly closes that gap by requiring a standard-depth model with actual loss reduction. Since the reader already recommends a conditional verdict, my stress-test does not move the verdict; it reinforces the condition with a specific check. I also credit the open-source code and the task-based parallelism idea as plausible engineering contributions, but the central claim needs the additional evidence before acceptance at face value.","tokens_in":6608,"tokens_out":3428,"duration_ms":31184,"concrete_test":"Run a standard-depth GPT2 model (e.g., 24 layers, hidden size 1024, 16 heads, ~1.5B parameters) on the same 8xA100 node with NNTile and with PyTorch FSDP for 100 optimizer steps on a fixed corpus, recording training loss and tokens/sec. If NNTile either cannot run the standard-depth model, runs slower than FSDP, or fails to reduce loss as expected, the headline claim should be downgraded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline result (49.9B parameters trained on one 8xA100 node, Section 4, Figure 2) is obtained with custom GPT2 models that have only 8 transformer layers and extremely large embedding dimensions. The authors explicitly state in Section 3 that standard pretrained GPT2 models \"did not scale from a single GPU to 8 GPUs\" because gradient reduction became a bottleneck, and in Section 4 that \"the more layers are in a neural network, the worse performance we get with NNTile.\" Thus the experiments demonstrate memory capacity and throughput for wide, shallow models where tensor parallelism across the embedding dimension can hide the framework's weakness on deep stacks, not that NNTile can train deep, realistic GPT-scale architectures. In addition, no loss curve or convergence evidence is reported, so \"training\" is only inferred from throughput runs. Together, these gaps leave the central claim that NNTile is \"capable of training extremely large GPT language models\" unsupported by the reported evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces NNTile, a machine learning framework built on the StarPU task-based parallel programming library, and claims it can train extremely large GPT-style language models on a single multi-GPU node by using CPU RAM offloading and dynamic task scheduling. The central experimental result is that NNTile can train custom GPT2 configurations with up to 49.9 billion parameters on one node with 8 A100 GPUs, whereas PyTorch FSDP is limited to about 10.6 billion parameters in the same setting. The paper describes the task-based paradigm, the tile-based implementations of transformer layers, and reports throughput measurements rather than training dynamics.","tokens_in":6814,"tokens_out":3384,"duration_ms":31931,"significance":"The potential significance is substantial: if NNTile genuinely enabled training of very large GPT models on a single node, it would lower the hardware barrier for LLM research and demonstrate a novel application of task-based scheduling to neural network training. The open-source release and the idea of automatically using CPU RAM as a memory extension are positive contributions. However, the experiments cover only custom architectures with very few transformer blocks and very large embedding dimensions, and the absence of any convergence evidence means the headline claim is not yet established.","major_comments":[{"comment":"The reported 49.9B-parameter result is obtained with custom GPT2 models containing only 8 transformer layers and very large embedding sizes. The manuscript states in Section 3 that standard pretrained GPT2 models did not scale from one to eight GPUs, and in Section 4 that 'the more layers are in a neural network, the worse performance we get with the NNTile.' These statements directly undermine the claim that NNTile can train 'extremely large GPT language models,' because the tested models are not representative of the GPT2 family's depth. Please either demonstrate training with a standard-depth GPT2 configuration (e.g., 12 or 24 layers) or re-scope the title and abstract to claim capability for wide, shallow transformer models with large embedding dimensions.","section":"Section 3, Section 4, Figures 1 and 2"},{"comment":"The paper reports only throughput in Tflops/s and never shows loss curves, perplexity, or any convergence metric. 'Training' is inferred solely from executing forward and backward passes; no evidence demonstrates that the gradients are correct or that optimization actually reduces the loss. Please add at least one full training run with a loss curve and a comparison of the training dynamics against PyTorch FSDP to validate that the framework truly trains models rather than merely fitting them in memory.","section":"Section 4, Figures 1 and 2"},{"comment":"The experimental setup is under-specified. For each model size, the manuscript does not report sequence length, batch size, vocabulary size, number of attention heads, tile sizes, optimizer settings, or the exact model configuration behind each data point. Without these details, the parameter counts cannot be verified, the comparison with PyTorch FSDP is not reproducible, and the assertion that the models are 'similar' is not checkable. A table listing all hyperparameters and model architectures for every point in Figures 1 and 2 is needed.","section":"Section 4"}],"minor_comments":[{"comment":"There is a typo in the sentence about the 8-layer model: 'dmdasd' appears after 'scheduling policy.'","section":"Section 4"},{"comment":"The abstract describes training in 'heterogeneous clusters,' but all experiments are conducted on a single node; please clarify that the reported results are single-node only.","section":"Abstract"},{"comment":"Reference [1] is incomplete; it lacks a year, publisher, and access date, making it unverifiable.","section":"References"},{"comment":"The x-axis model sizes are not evenly spaced, which can visually distort scaling behavior; consider presenting the exact configurations in a table and using a log scale or annotated points.","section":"Figures 1 and 2"},{"comment":"The memory-management description would benefit from a concrete example of how tiles are retained on a GPU across successive tasks, since this mechanism is central to the claimed advantage over standard offloading approaches.","section":"Section 2"}],"recommendation":"major_revision","confidential_remarks":"The paper's central claim is significantly overstated relative to the evidence. The use of custom, very shallow GPT2 variants with enormous embedding dimensions appears to inflate parameter counts without testing realistic transformer depth, and the paper itself admits that performance degrades with more layers. In addition, the complete absence of loss curves means the word 'training' is not supported. I would require the authors to either re-scope the claims to match the actual experiments or provide standard-depth GPT2 results and convergence evidence before considering publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this is a genuine systems contribution with open code, but the headline claim—training extremely large GPT language models on a single node—is not supported by the experiments as written. The stress-test is correct. The 49.9B result comes from custom 8-layer GPT2 models with enormous embeddings, and the authors themselves say standard GPT2 models didn't scale from one GPU to eight and that more layers make NNTile slower. So what is actually demonstrated is memory capacity for wide, shallow transformer-like models, where tensor parallelism over the embedding dimension can hide the framework's weakness on deep stacks. That's an interesting engineering data point, but it is not extremely large GPT language models in any ordinary sense.\n\nWhat is genuinely new: this is the first application of task-based parallel programming with StarPU to transformer training, as far as I know. The framework is open source, and the throughput and capacity comparison against PyTorch FSDP on identical hardware and precision (tf32) is a straightforward, reproducible benchmark. That part is useful. The paper also gives a clear, if brief, overview of how each GPT2 component maps to tile-based tasks.\n\nSoft spots, in proportion. The missing convergence evidence is the biggest one. There is no loss curve anywhere. Training is inferred from throughput runs. For all the paper shows, the optimizer could be diverging; at minimum a small-scale run with a decreasing loss would establish that the implementation is actually training, not just moving tensors. Second, the baseline is only FSDP. DeepSpeed ZeRO-Offload is the obvious competitor for a system whose whole point is using CPU RAM alongside GPU VRAM, and its absence makes the 49.9B-versus-10.6B comparison look curated. Third, tile-size configuration is a free parameter and no sensitivity analysis is given. Fourth, the architecture switch from standard GPT2 to custom shallow models is a red flag; the paper acknowledges it but does not justify why the framework should be expected to work on deep models later.\n\nNone of this kills the idea. Task-based scheduling with asynchronous CPU offload is plausible and worth exploring. But the claims need to be scaled back and the evidence strengthened. A serious referee should ask for: a convergence test on a real, even small, GPT2 model; a deep-stack scaling experiment; a DeepSpeed offload baseline; and a headline that matches the experiments.\n\nVerdict: worth a serious review, not a desk reject. I'd bring it to reading group, and I'd cite it as related work if I were working on single-node large-model training. But as is, I would not trust the 49.9B number as evidence that NNTile can train realistic GPT-scale models.","headline":"A promising systems idea with open code, but the headline claim outstrips the experiments: the 49.9B result is for wide, shallow custom GPT2-style models, and the paper never shows a loss curve.","tokens_in":7324,"tokens_out":2451,"would_cite":true,"duration_ms":23349,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A task-based scheduler lets a single eight-GPU node train a 49.9-billion-parameter GPT2 model.","keywords":["NNTile","task-based parallelism","large language models","GPT2","heterogeneous computing","memory offloading","tensor parallelism","single-node training"],"falsifier":"Run the framework on a GPT2 with 12 or more transformer blocks at the same 49.9-billion-parameter scale on one eight-GPU node; if the training step does not complete at comparable throughput, the broad claim about extremely large GPT models rests on the shallow, wide configurations only.","tokens_in":6441,"feed_emoji":"🧠","tokens_out":8456,"duration_ms":73791,"temperature":0.7,"pith_summary":"This paper argues that task-based parallelism removes the memory wall that limits large-language-model training on a single server. Instead of sharding the model by hand and synchronizing gradients on a fixed schedule, every tensor is split into tiles and each operation becomes a small task that a runtime scheduler can place on any CPU core or GPU, moving tiles between GPU memory and CPU RAM asynchronously. On one node with eight A100 GPUs, the resulting NNTile framework trains a custom GPT2 model with 49.9 billion parameters, while the fully sharded data-parallel baseline fits only 10.6 billion at the same TensorFloat32 precision. The claim matters because it points to a different route to very large models: use all the memory and compute in a node automatically, rather than adding more GPUs.","feed_headline":"Task-based scheduling trains a 50B-parameter GPT2 on one node","feed_subtitle":"One 8-GPU server outruns the FSDP baseline's 10.6B-parameter ceiling by treating CPU RAM as memory.","key_machinery":"The load-bearing mechanism is the tile-wise task graph with a sequential task flow: split each tensor along all axes into tiles, express every layer as tasks that read and write a few tiles, and let the runtime's scheduler map ready tasks onto any device while asynchronously moving tiles between GPU and CPU memory. The scheduler replaces the human decisions of parameter sharding and communication timing. The argument travels through this mechanism: the same tile-wise implementations are given for embedding, linear, layer norm, softmax, attention, cross-entropy, and Adam/SGD updates, so the entire training loop is a single graph of small memory-bound tasks.","core_discovery":"The central discovery is an application: the tile-based task-parallel paradigm, previously used for dense linear algebra, can carry transformer training, and doing so extends single-node model size far beyond the aggregate GPU memory. For an 8-layer GPT2 with varying embedding sizes on eight A100s, NNTile reaches 49.9 billion parameters before memory exhaustion, versus 10.6 billion for the sharded data-parallel baseline; for 4-layer models on four A100s the numbers are 25.5 and 6.7 billion. The paper traces this advantage to automatic data placement: tiles remain on a GPU while needed, are copied asynchronously between GPUs, and are offloaded to CPU RAM when idle, with the scheduler choosing where each task runs. Performance at a given model size is comparable to the baseline; the difference is capacity, not per-step speed.","pith_inferences":["The paper reports that scheduling quality degrades as layers are added; the natural next step, which the authors leave implicit, is that a scheduler that plans across the whole training iteration rather than greedily is the direct lever for extending the single-node record to standard-depth models.","Because the demonstrated advantage is the automatic use of CPU RAM, a fairer capacity comparison would enable CPU offload in the sharded data-parallel baseline as well; that comparison is not run here, so the 49.9-versus-10.6-billion factor is specific to the default offload-free baseline.","The same tile-wise, scheduler-driven memory management should transfer to other memory-bound phases such as fine-tuning and long-context inference, though the paper only demonstrates training."],"forward_implications":["Models larger than the aggregate GPU memory of a node can be trained without hand-written offload schedules, because tile tasks are placed and migrated automatically.","The practical ceiling on single-node training moves from total GPU memory to scheduler quality; the paper reports that performance degrades as the number of transformer layers grows.","The same tile-wise treatment of forward and backward passes covers a full list of standard components, so the approach could be assembled into other architectures, not just GPT2.","Within the demonstrated configuration, a four-layer model reaches 25.5 billion parameters on four GPUs, indicating that the memory advantage appears at smaller node counts too."],"supporting_citations":[{"why":"Supplies the task-based runtime that provides task submission, tile memory management, and the scheduler that places each operation on CPUs or GPUs.","marker":"[15]"},{"why":"Defines the fully sharded data-parallel baseline that the paper compares against on identical hardware; the baseline's memory ceiling is the number to beat.","marker":"[12]"},{"why":"Provides the deep learning implementation used to run the baseline comparison.","marker":"[11]"},{"why":"Representative of popular distributed training solutions whose aggregate-GPU-memory limit motivates the use of CPU RAM.","marker":"[5]"},{"why":"Introduces the transformer architecture whose GPT2 instantiation is the model class being trained, including the attention layer re-implemented tile-wise.","marker":"[2]"}],"fun_headline_variants":["Tile-based parallelism trains 50B GPT2 on a single node","Single node trains 50B-parameter GPT2 via task scheduling","NNTile: 50B-param GPT2 on one node, not many","Task scheduler turns CPU RAM into GPU memory for 50B GPT2","One 8-GPU node beats FSDP: 50B vs 10B parameters"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 49.9-billion-parameter result was produced with custom GPT2 models that have only a few transformer blocks and very large embedding dimensions, because standard pretrained GPT2 configurations did not scale from one to eight GPUs in this framework.","fun_headline_variants_meta":{"raw":{"variants":["Tile-based parallelism trains 50B GPT2 on a single node","Single node trains 50B-parameter GPT2 via task scheduling","NNTile: 50B-param GPT2 on one node, not many","Task scheduler turns CPU RAM into GPU memory for 50B GPT2","One 8-GPU node beats FSDP: 50B vs 10B parameters"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000641,"raw_usage":{"total_tokens":2902,"prompt_tokens":849,"completion_tokens":2053,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":465,"completion_tokens_details":{"reasoning_tokens":1952}},"tokens_in":465,"tokens_out":2053,"duration_ms":14497,"temperature":1.0,"reasoning_tokens":1952,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:14:26.595288+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the framework on a GPT2 with 12 or more transformer blocks at the same 49.9-billion-parameter scale on one eight-GPU node; if the training step does not complete at comparable throughput, the broad claim about extremely large GPT models rests on the shallow, wide configurations only.","supporting_citations":[{"cited_title":"StarPU: A Unified Plat- form for Task Scheduling on Heterogeneous Multicore Architectures","cited_arxiv_id":null,"evidence_quote":"Supplies the task-based runtime that provides task submission, tile memory management, and the scheduler that places each operation on CPUs or GPUs."},{"cited_title":"Pytorch: An imperative style, high-performance deep learning library","cited_arxiv_id":null,"evidence_quote":"Provides the deep learning implementation used to run the baseline comparison."},{"cited_title":"Rasley, S","cited_arxiv_id":null,"evidence_quote":"Representative of popular distributed training solutions whose aggregate-GPU-memory limit motivates the use of CPU RAM."},{"cited_title":"Vaswani, N","cited_arxiv_id":null,"evidence_quote":"Introduces the transformer architecture whose GPT2 instantiation is the model class being trained, including the attention layer re-implemented tile-wise."}],"review_version":1}