{"id":"3762f3fd-758b-4fdb-9686-2213762de4b7","arxiv_id":"2506.06472","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"TERAIO speeds up LLM training by offloading inactive tensors to SSDs with a lifetime-aware, GPUDirect Storage based migration plan, reaching 1.47x over ZeRO-Offload/Infinity and 80.7% of ideal.","lead":"This paper builds TERAIO, a system that offloads rarely-used tensors from GPUs to cheap SSDs during LLM training, using a migration plan learned from the first few training iterations. It reports 1.47x faster training than ZeRO-based offloading and 80.7% of the speed of an idealized unlimited-memory GPU.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Profiling stability and runtime feedback are asserted, not demonstrated; if kernel times or memory pressure drift after profiling, the planned deadlines miss and the 80.7%-of-ideal claim may not persist.","rationale":"The stress-test pass identifies the same load-bearing concern as the reader: the entire system is predicated on the stability of tensor lifetimes and kernel times after a short profiling phase. This is not a matter of consensus but of internal consistency: Algorithm 1 plans migrations using timings collected from the unmodified run, yet the runtime may introduce stalls and contention that change exactly those timings. Without a feedback loop or empirical validation, the 80.7%-of-ideal number is not protected against drift. I agree with the conditional verdict: the idea is plausible and the evaluation is coherent on its face, but the conditions (artifact, error bars, longer runs, stall statistics) are necessary. I do not see a basis to reject outright, because the measured 1.47x speedup over ZeRO baselines is a real comparison that would hold even if the absolute ideal reference is soft. The concrete test above would settle whether the central assumption fails.","tokens_in":13945,"tokens_out":12016,"duration_ms":126886,"concrete_test":"Release the code and rerun a representative configuration (e.g., Llama3-8B bs=64 seq=2048, and Llama3-70B bs=16 seq=4096) for at least 10,000 iterations or a fixed token budget. Instrument the runtime to record (a) per-kernel execution times against profiled values, (b) number of urgent/blocking stalls due to memory-capacity or prefetch misses, (c) migration deadline misses. Also rerun the profiler at iteration 100 and compare the resulting migration plans and predicted throughput. If the plan changes >5% or stall cycles grow after the first few iterations, the static-profiling assumption is violated and the 80.7%-of-ideal average is not representative of training at scale.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim relies on Section 3.1's assumption that 'the training follows the same execution graph in subsequent iterations, the tensor activity patterns remain the same.' The migration plan in Algorithm 1 is computed once from profiling the unmodified run; it assumes kernel execution times and tensor lifetimes are fixed. But at runtime, the migration engine can perturb execution: §3.2 admits it will 'stall kernel execution to wait for more inactive tensors to be offloaded' and 'wait for the tensors needed by the kernel to be migrated back.' These stalls, and any HBM/PCIe bandwidth contention from GDS transfers, change kernel times and thus invalidate the precomputed slack. The paper reports no stall counts, no kernel-time variance, and no run duration; the reported throughput could have been measured over the profiled window only. For Llama-70B and other models that exceed 2-GPU memory, the 'Ideal' baseline cannot be measured directly and must be simulated, so the 80.7% value is not an independently reproducible reference. The lack of a released artifact makes it impossible to check any of this.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents TERAIO, a tensor offloading framework that uses GPUDirect Storage to migrate inactive tensors between GPU memory and PCIe-attached SSDs during LLM training. The system profiles the first few training iterations to obtain per-tensor sizes, active periods, and kernel execution times, then runs a greedy benefit-to-cost algorithm (Algorithm 1) to generate offloading and prefetching plans, executing them at runtime through a GDS-based migration engine. The evaluation trains Llama3-8B, Granite-code-base-8B, and Llama3-70B on two H100 GPUs with up to eight SSDs, comparing against ZeRO-Offload, ZeRO-Infinity, and an idealized unlimited-memory baseline. The headline claims are a 1.47x average throughput improvement over the ZeRO systems and 80.7% of ideal performance, with cost-efficiency improvements of up to 5.41x relative to a pure-GPU setup.","tokens_in":14180,"tokens_out":3617,"duration_ms":36077,"significance":"If the central empirical claim holds, TERAIO would be a useful contribution: it demonstrates that commodity PCIe SSDs can serve as a large, low-cost expansion tier for GPU memory in multi-GPU LLM training, with only modest throughput loss. The paper's strengths include a concrete characterization of tensor activity patterns in LLM training, a roofline analysis quantifying the bandwidth needed for near-ideal performance, a clean system design built on PyTorch without requiring user code changes, and a direct comparison against two widely used baselines. The measured speedups and cost-efficiency numbers are internally consistent with the reported figures. However, the evidence is limited to a single two-GPU machine, and several evaluation details needed to substantiate the headline percentages are not reported.","major_comments":[{"comment":"The profiling-stability assumption is load-bearing but not validated. The paper states that \"the training follows the same execution graph in subsequent iterations, the tensor activity patterns remain the same,\" yet it never reports kernel-time variance, tensor-lifetime drift, or memory-footprint changes over the course of training. This matters because the runtime engine can itself perturb execution: Section 3.2 admits that it will \"stall kernel execution\" and \"wait for the tensors needed by the kernel to be migrated back,\" and GDS transfers contend for PCIe bandwidth. If kernel durations or lifetimes deviate from the profiled values, the precomputed prefetch deadlines can be missed, undermining the 80.7%-of-ideal claim. The authors should report run durations, the number of profiled versus evaluated iterations, measured stall counts, and throughput over the full training run rather than only the profiled window.","section":"Section 3.1 / Section 3.2"},{"comment":"The evaluation lacks run-to-run statistics and covers only one two-GPU machine. No error bars, standard deviations, or number of repeated runs are reported, so the 1.47x average figure could be within noise, especially for configurations where the reported differences between TeraIO-SSD and ZeRO-Infinity are small. Additionally, the paper does not compare against the most closely related SSD-offloading systems such as FlashNeuron, Smart-Infinity, or TBA, only against ZeRO-Offload and ZeRO-Infinity. Without these comparisons and without variance information, the claim that TERAIO is superior to state-of-the-art offloading is not fully supported.","section":"Section 4.2, Figure 7"},{"comment":"It is unclear how the \"Ideal\" baseline is obtained for models whose memory demand exceeds the available GPU memory. Llama3-70B requires approximately 940% of one GPU's capacity, so an unlimited-memory run cannot be executed on the two H100s; it must be simulated. The paper does not state whether the Ideal throughput comes from the kernel-time-based performance model of Section 2.2 or from a separate measurement, nor does it disclose the simulation assumptions. If the Ideal baseline assumes fixed kernel times and no migration stalls, then the 80.7% figure is an upper-bound estimate under those assumptions rather than an empirical comparison, and the paper should say so explicitly.","section":"Section 4.2, Ideal baseline"},{"comment":"The lifetime-aware migration planner is a greedy benefit-to-cost search, but its quality is never evaluated. The paper does not compare the generated plan against an optimal or exhaustive plan, nor does it report sensitivity to the profiling window length, to the estimated I/O bandwidth, or to the order in which candidates are examined. Since the central novelty is precisely this planning algorithm, the authors should provide evidence that the greedy choice does not leave significant performance on the table and that the plan is robust to reasonable variation in its inputs.","section":"Algorithm 1"}],"minor_comments":[{"comment":"The subfigure labels are inconsistent: both subplots in the first row are labeled \"(a)\" and several rows use \"(d)\" for different models, which makes it hard to identify which plot corresponds to which model and pipeline stage.","section":"Figure 1"},{"comment":"There is a typo in the phrase \"the proifler requires minimal code modifications to PyTorch\"; it should be \"profiler.\"","section":"Section 3.1"},{"comment":"The system name is spelled inconsistently as both \"Teraio\" and \"TeraIO\" (and \"TERAIO\" in the title). Please standardize the spelling.","section":"Throughout"},{"comment":"The notation \"tpref etched\" appears to be a typo; it should presumably be \"tprefetched\" or \"tprefetch.\"","section":"Algorithm 1, line 15"},{"comment":"The paper cites Smart-Infinity [12] in the introduction but does not use it as a baseline in the evaluation; a sentence explaining why it was omitted would help the reader calibrate the comparison.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper would benefit substantially from releasing the artifact, since the reproducibility of the throughput claims currently depends on undocumented details of the ideal baseline and the measurement window. I would also encourage the editor to weigh whether the single-machine evaluation scope is sufficient for a general claim about multi-GPU systems; the authors should either extend the evaluation or temper the generality of the conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper is a genuine systems contribution: it combines fine-grained tensor lifetime profiling via PyTorch's operator generator, an I/O-aware migration planner, and GPUDirect Storage for multi-GPU LLM training. That combination is new relative to ZeRO-Offload, ZeRO-Infinity, FlashNeuron, Smart-Infinity, and TBA. The measured 1.47x average speedup and 80.7% of ideal are internally consistent with the reported memory and bandwidth traces, and the cost analysis is straightforward and fair.\n\nThe soft spots are real but not fatal. The evaluation runs on a single two-GPU machine with no error bars or run counts, and the closest SSD-based systems are cited but not compared. The paper calls Algorithm 1 'optimal' in two places, but it is a greedy benefit-to-cost heuristic; that overstates what is proven. The stress-test concern about profiling stability is legitimate: Section 3.1 assumes the execution graph and tensor activity stay the same after the first iterations, and Section 3.2 admits the runtime engine can stall kernels, which perturbs the very timings the plan was computed from. The paper reports no stall counts or kernel-time variance, so we cannot tell how much this matters. In my reading this is not a load-bearing flaw—LLM training with fixed shapes is highly regular—but the authors should measure it or bound it explicitly.\n\nThe paper is solid enough for a serious referee. I would send it to a systems conference or journal with requests for the artifact, variance data, a run-length check, and at least one direct SSD-baseline comparison. If those hold up, the hardware-cost reduction is real and useful.","headline":"Genuinely new integration of lifetime profiling, I/O-aware planning, and GDS offloading; plausible claims but the evaluation needs variance, direct SSD baselines, and an artifact.","tokens_in":14736,"tokens_out":2361,"would_cite":true,"duration_ms":21690,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TERAIO claims that lifetime-aware tensor offloading to PCIe SSDs lets LLM training run at 80.7% of ideal throughput while cutting hardware cost severalfold.","keywords":["LLM training","tensor offloading","GPU memory expansion","GPUDirect Storage","SSD","tensor lifetime profiling","migration planning","PyTorch"],"falsifier":"Train a model with per-batch variable sequence lengths or data-dependent control flow so that kernel execution times drift from the profiled values; if TERAIO's throughput falls below ZeRO-Infinity's on that workload, the static-lifetime assumption is the reason.","tokens_in":13771,"feed_emoji":"⚡","tokens_out":3598,"duration_ms":31900,"temperature":0.7,"pith_summary":"The paper claims that LLM training can be made far cheaper by treating commodity PCIe SSDs as a large, slow extension of GPU memory without paying the usual throughput penalty. It supports this with TERAIO, a framework that profiles the first few training iterations to learn each tensor's lifetime, then plans fine-grained offload and prefetch operations so that only the tiny fraction of tensors a kernel needs stay in GPU memory. On Llama and Granite models, TERAIO reports 1.47x higher throughput than ZeRO-Offload and ZeRO-Infinity, and reaches 80.7% of the throughput of an idealized machine with unlimited GPU memory. The payoff would be that institutions can train large models on machines with small amounts of HBM plus a few NVMe drives instead of buying extra GPU memory.","feed_headline":"SSD offloading brings LLM training to 80.7% of ideal","feed_subtitle":"TERAIO's fine-grained tensor migration beats ZeRO offloading by 1.47x while slashing memory cost.","key_machinery":"The key mechanism is the lifetime-aware tensor migration plan: a profiler instrumenting PyTorch's automatic operator generator records each tensor's size and active/inactive intervals; a planning algorithm then iteratively selects the inactive tensor whose eviction gives the largest reduction in critical memory pressure per byte of I/O, schedules offload and prefetch with explicit deadlines computed from the measured kernel execution times and available SSD bandwidth, and emits the plan into the compiled training program; at runtime a migration engine executes these transfers directly between GPU memory and SSDs over GPUDirect Storage. The profiler's fixed assumption is that the first few iterations reveal the whole run's activity pattern.","core_discovery":"The central discovery is that in LLM training, the working set is tiny even when total tensor memory is enormous: across the models studied, active tensors use on average only 1.7% of requested GPU memory, and most inactive tensors stay unused for longer than it takes to move them to an SSD. This asymmetry means a scheduler that knows each tensor's lifetime can evict large tensors right after their last use and prefetch them just in time, overlapping migration with GPU compute. TERAIO captures the required lifetime information with a lightweight PyTorch profiler, turns it into an offload/prefetch plan that maximizes the ratio of memory-pressure relief to I/O cost, and executes the plan with GPUDirect Storage to bypass the host CPU. The result is claimed to be near-ideal training throughput at a fraction of the hardware cost.","pith_inferences":["Because the planner assumes static kernel durations, TERAIO's benefit could degrade in workloads with dynamic shapes or data-dependent kernels; adding an online re-profiling or adaptive adjustment layer would be a natural extension.","The same lifetime-aware planning concept could be applied to other GPU-memory-congested settings such as fine-tuning, inference batching, or mixture-of-experts routing, where tensor lifetimes are also predictable.","The 1.7% active-fraction observation implies that even moderately faster external memory (for example, CXL-attached memory) could be integrated with the same planner to push the achievable fraction of ideal performance closer to 100%."],"forward_implications":["Trainers can run models that normally need 8-GPU H100 servers on 2-GPU machines with several NVMe SSDs, cutting server cost by about 5x.","Fine-grained, lifetime-aware offloading beats coarse layer-level offloading such as ZeRO-Infinity, making storage-based memory expansion practical for pre-training rather than just inference.","The reported 1.47x speedup over ZeRO baselines translates directly into cost efficiency because TERAIO uses less CPU memory and fewer SSDs to achieve it.","With per-GPU migration bandwidth above 32 to 48 GB/s (achievable by striping commodity SSDs), TERAIO approaches the ideal unlimited-memory roofline, suggesting diminishing returns from further offloading optimizations."],"supporting_citations":[{"why":"ZeRO-Offload is the CPU-memory offloading baseline that TERAIO must beat; its 65.9% of ideal throughput for Llama3-8B is the comparison point.","marker":"[30]"},{"why":"ZeRO-Infinity is the SSD-plus-CPU offloading baseline; its coarse-grained, layer-level offloading is what TERAIO's fine-grained lifetime-aware plan is designed to outperform.","marker":"[29]"},{"why":"GPUDirect Storage is the mechanism that lets TERAIO move tensors directly between GPU memory and SSDs without copying through the host CPU.","marker":"[7]"},{"why":"TorchTitan is the training framework TERAIO integrates with to run Llama and Granite models in the evaluation.","marker":"[17]"},{"why":"PyTorch is the framework whose automatic operator generator TERAIO instruments to track tensor sizes, lifetimes, and kernel execution times.","marker":"[23]"},{"why":"PipeDream's 1F1B pipeline parallelism is the scheduling scheme used in the multi-GPU setup that shapes the tensor activity patterns TERAIO profiles.","marker":"[21]"}],"fun_headline_variants":["Lifetime-aware SSD offload boosts LLM training 1.47x","TERAIO reaches 80.7% of ideal LLM speed via SSD offload","Offload idle tensors: LLM training at 1.47x faster with SSDs","Tensor lifetimes guide GPU-to-SSD migration for cost-efficient LLM training","GPUDirect Storage plus lifetime-aware offload trims LLM memory costs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The profiler assumes that the first few training iterations faithfully represent the entire run, so the precomputed offload and prefetch deadlines never go stale.","fun_headline_variants_meta":{"raw":{"variants":["Lifetime-aware SSD offload boosts LLM training 1.47x","TERAIO reaches 80.7% of ideal LLM speed via SSD offload","Offload idle tensors: LLM training at 1.47x faster with SSDs","Tensor lifetimes guide GPU-to-SSD migration for cost-efficient LLM training","GPUDirect Storage plus lifetime-aware offload trims LLM memory costs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000866,"raw_usage":{"total_tokens":3787,"prompt_tokens":1011,"completion_tokens":2776,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":627,"completion_tokens_details":{"reasoning_tokens":2669}},"tokens_in":627,"tokens_out":2776,"duration_ms":20337,"temperature":1.0,"reasoning_tokens":2669,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:55:55.018799+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a model with per-batch variable sequence lengths or data-dependent control flow so that kernel execution times drift from the profiled values; if TERAIO's throughput falls below ZeRO-Infinity's on that workload, the static-lifetime assumption is the reason.","supporting_citations":[{"cited_title":"{Zero-offload}: Democratizing{billion-scale}model training","cited_arxiv_id":null,"evidence_quote":"ZeRO-Offload is the CPU-memory offloading baseline that TERAIO must beat; its 65.9% of ideal throughput for Llama3-8B is the comparison point."},{"cited_title":"Zero- infinity: Breaking the gpu memory wall for extreme scale deep learning","cited_arxiv_id":null,"evidence_quote":"ZeRO-Infinity is the SSD-plus-CPU offloading baseline; its coarse-grained, layer-level offloading is what TERAIO's fine-grained lifetime-aware plan is designed to outperform."},{"cited_title":"nvidia.com/blog/gpudirect-storage/","cited_arxiv_id":null,"evidence_quote":"GPUDirect Storage is the mechanism that lets TERAIO move tensors directly between GPU memory and SSDs without copying through the host CPU."}],"review_version":1}