{"id":"985e2d0d-b0ba-46c9-a3a0-b2215759fb45","arxiv_id":"2504.13266","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Pre-propagation GNNs, whose training is bottlenecked by data loading, can be sped up roughly 15x with custom loaders, GPU double buffering, chunk reshuffling, and direct storage access, beating sampling-based GNNs by up to two orders of magnitude in throughput at comparable accuracy.","lead":"This paper finds that pre-propagation graph neural networks, a class of GNNs that precompute neighbor features, are slowed mainly by how training data is loaded, not by computation. The authors build an optimized data-loading pipeline that makes PP-GNN training about 15 times faster and up to 100 times faster than sampling-based GNNs at similar or better accuracy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 9.9x claim compares PP-GNNs to MP-GNN systems that never combine the SOTA LABOR sampler with GPU feature caching; a GNNLab+LABOR baseline could substantially narrow the headline speedup.","rationale":"The paper is a solid systems contribution: the characterization of data loading as the PP-GNN bottleneck and input expansion as the scalability challenge is useful, the optimizations are well motivated, and the medium-graph accuracy-efficiency study uses 400 epochs and 5 runs. Public code is a plus, though the artifact only fully reproduces ogbn-products. The central risk is not internal inconsistency but external validity of the headline speedup: the empirical comparison omits the natural combination of the two best MP-GNN ingredients (LABOR sampling and GPU feature caching). The reader identified baseline fairness as the weakest assumption; I agree with that direction and make it concrete, while placing less weight on the vanilla PP-GNN data-loader concern because it does not affect the 9.9x versus MP-GNN claim. The igb-large accuracy result (3 epochs, 1 run) is also weak, but the throughput number is the more load-bearing part of the central claim, so I focus there. If the GNNLab+LABOR experiment shows only modest erosion, the conclusion survives; if it reduces the geometric mean substantially, the headline requires adjustment. The reader's CONDITIONAL verdict remains appropriate, hence UNCHANGED.","tokens_in":25423,"tokens_out":19293,"duration_ms":177683,"concrete_test":"Modify GNNLab to use the LABOR sampler (or equivalently add GNNLab-style GPU feature caching to DGL's LABOR pipeline) and rerun Tables 3 and 4 on ogbn-papers100M and igb-medium for 2, 3, and 4 layers, with at least 5 runs per configuration. Recompute the geometric-mean training-throughput advantage of SIGN/HOGA over this combined baseline. If the mean falls below the claimed 9.9x, revise the headline to report the speedup against the stronger baseline; if the 2-orders-of-magnitude cell disappears, remove or qualify that claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline speedup is a geometric mean over per-setting ratios in Tables 3-5. The two ingredients that make MP-GNNs fast are (i) a low-expansion sampler such as LABOR and (ii) GPU-side feature caching as in GNNLab/SALIENT++. No baseline in the paper combines both. DGL uses LABOR but without GPU feature caching (ogbn-papers100M: 0.12/0.04/0.02 epoch/s for 2/3/4 layers); GNNLab has GPU caching but uses a hardcoded neighbor sampler that the paper itself says produces larger subgraphs than LABOR (0.72/0.19/0.06 epoch/s). A GNNLab-style system with LABOR sampling is a natural 'SOTA sampler + SOTA system' baseline, and it is absent. Because LABOR reduces the number of sampled nodes, integrating it into GNNLab could materially raise the MP-GNN throughput: the 2-layer papers100M ratio could drop from about 4x toward 2x, and the 4-layer ratio from about 48x toward well under 20x. The paper also reports no throughput variance, so the stability of the 9.9x mean is unclear. This does not invalidate the systems contribution, but the strongest quantitative claim is not yet tested against the strongest MP-GNN configuration that its own components would permit.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a systematic empirical comparison of pre-propagation GNNs (PP-GNNs: SGC, SIGN, HOGA) against message-passing GNNs (MP-GNNs) trained with sampling-based systems on graphs ranging from roughly 2M to 111M nodes. The authors identify data loading as the dominant cost in vanilla PP-GNN training and identify the 'input expansion problem' as a scalability bottleneck. They propose a customized data loader, double-buffer GPU-side prefetching, chunk reshuffling, and GPU direct storage access, together with an automated data-placement policy. On medium graphs the optimizations yield about 15x throughput improvement over a PyTorch DataLoader baseline; on three large graphs the paper reports an average 9.9x and up to two orders of magnitude higher throughput than the MP-GNN systems evaluated, with comparable or higher accuracy.","tokens_in":25715,"tokens_out":10795,"duration_ms":84091,"significance":"If the quantitative claims hold, this paper would make a strong practical case for PP-GNNs as a default approach for large-scale node classification, countering the common view that PP-GNNs are only a theoretical alternative with limited systems support. The study is valuable as a first systematic separation of model accuracy and systems efficiency for PP-GNNs, and the ablation cleanly attributes the end-to-end speedup to each proposed optimization. The public artifact, the identification of the input-expansion problem, and the careful treatment of preprocessing amortization are concrete strengths. However, the headline speedup is not tested against the strongest MP-GNN configuration that the paper's own components would permit, throughput variability is not reported, and the 9.9x number is not reproducible from the tables as stated; these issues currently overstate the strength of the central claim.","major_comments":[{"comment":"The headline claim of an average 9.9x and up to two orders of magnitude speedup over 'MP-GNN models with state-of-the-art graph samplers' is not tested against the strongest MP-GNN configuration that the paper's own components would permit. DGL is run with LABOR but without GPU-side feature caching, while GNNLab and SALIENT++ provide GPU feature caching but use hardcoded neighbor samplers that the paper itself states produce larger sampled subgraphs than LABOR. A GNNLab-style system using LABOR sampling is a natural 'SOTA sampler + SOTA system' baseline, and it is absent from Tables 3 and 4. Because LABOR reduces the number of sampled nodes, integrating it into GNNLab could materially raise MP-GNN throughput; for example, the 4-layer ogbn-papers100M ratio of SIGN over GNNLab is about 48x, and a reduction in sampled subgraph size would shrink this substantially. The quantitative claim in the abstract and Section 6.4 therefore needs either this baseline or a more restrictive wording, such as 'compared to DGL with LABOR and to GNNLab/SALIENT++ with their default samplers.'","section":"§6.4, Tables 3–4"},{"comment":"The derivation of the 9.9x average is not specified, and I could not reproduce it from the tables. Using the per-setting ratios of SIGN and HOGA to DGL in Tables 3–5 gives a geometric mean of about 23x; using the best MP-GNN system in each row gives about 8.9x. The text does not state which ratios are included (which model, which layer/hop count, which GPU count, which baseline system, and whether the geometric mean is over settings or over datasets). The claim should be made reproducible by reporting the exact set of ratios and the aggregation formula, or by presenting the ratios in a dedicated table.","section":"Abstract and §6.4 (Tables 3–5)"},{"comment":"No throughput variance is reported anywhere in the paper. Tables 3–5 give single epoch/sec values, and the 15x ablation in Fig. 9 is a single geometric mean over datasets, models, and hops. The statement that the vanilla PP-GNN baseline uses PyTorch DataLoader with pin_memory and 2 workers 'to achieve optimal performance' is not supported by a worker-count sweep; a stronger data-loading baseline (more workers, DALI, or a CUDA-aware loader) could reduce the 15x figure. For a systems paper whose central claims are throughput ratios, mean ± std over repeated runs should be reported at least for the headline settings, and the baseline configuration should be justified by a sweep.","section":"§6 'Baselines' and Fig. 9"},{"comment":"The claim of 'maintaining superior accuracy' on igb-large rests on test accuracies measured after only 3 epochs and a single run. At 3 epochs the DGL and Ginex baselines are far from convergence, so this comparison does not establish that PP-GNNs are more accurate on this dataset; it only compares very early-training behavior. Either the MP-GNN baselines should be run to a comparable convergence point, or the claim should be restricted to 'higher accuracy at the same early training budget.' The one-run no-error-bar reporting also makes the 64.41% versus 63.07% gap unreliable.","section":"Table 5 and §6.4"}],"minor_comments":[{"comment":"There are several typos and inconsistent acronyms: 'Practial' in §3.3, 'ognb-papers100M' and 'IGB-meduim' in Table 2, 'Pre-propgation' in §2.5, 'SDG-RR' in Table 4, and 'UV A' should be 'UVA'.","section":"Throughout"},{"comment":"The table uses red and blue coloring to distinguish feature propagation from feature transformation; this will not survive grayscale printing, so the legend should use textual markers or symbols instead of color alone.","section":"Table 1"},{"comment":"Figure 6 is very dense; the sub-figure labels, buffer diagrams, and stream dependencies are hard to read at normal size and should be enlarged or simplified.","section":"Figure 6"},{"comment":"The text says pre-processing overhead is 'usually much smaller than the time required for a single training run,' but for ogbn-papers100M the preprocessing time is 90% of a single run; this exception should be acknowledged in the main text near the claim.","section":"§3.5 and Table 7"},{"comment":"The acronym 'SGD-CR' is used without definition; it should be defined as 'stochastic gradient descent with chunk reshuffling' at first use.","section":"Appendix E"},{"comment":"The phrase 'The configure system defaults to SGD-RR' is missing the final 'd' in 'configured' and should read 'The configured system defaults to SGD-RR.'","section":"§5"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the main risk is the missing LABOR-plus-GPU-caching baseline, which is the natural strongest MP-GNN configuration permitted by the paper's own components. If the authors add that baseline, report throughput variance, make the 9.9x aggregation reproducible, and temper the igb-large accuracy claim, I would view the paper as publishable. The paper is within scope for MLSys, the artifact appears genuine, and I see no novelty or circularity concern."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper is worth reading and worth refereeing. It gives the first honest characterization of PP-GNN training costs, and the system work is real. But the 9.9x headline is a comparison against existing systems, not against the strongest possible MP-GNN system; treat the number as conditional on baselines that could be pushed further.\n\nWhat's new: the paper identifies data loading as the dominant training bottleneck for PP-GNNs, and the input expansion problem that makes large-graph PP-GNN inputs exceed host memory. Both are useful observations that prior work left implicit. The individual optimizations—indexed batch assembly, double buffering, chunk reshuffling, GDS—are known, but the integration is careful and the ablation cleanly separates the contributions: 3.3x from host-side assembly, 1.9x from double buffering, 2.4x from chunk reshuffling. That is a legitimate systems contribution, and the experiments are extensive: six datasets, multiple MP-GNN systems, multiple PP-GNN models, plus a data-transfer analysis that supports the direction of the claims.\n\nThe main soft spot is the comparison baseline. No MP-GNN system in the paper combines the LABOR sampler with GPU feature caching. DGL uses LABOR but without GPU caching; GNNLab has caching but uses a hardcoded sampler that the paper itself says produces larger subgraphs than LABOR. A GNNLab+LABOR baseline could plausibly narrow the speedup on ogbn-papers100M and igb-medium. That does not invalidate the systems contribution—comparing against shipped systems is the norm—but the abstract's claim of 9.9x \"compared to MP-GNN models with state-of-the-art graph samplers\" overstates what was actually tested.\n\nOther soft spots are minor but real. The vanilla PP-GNN baseline uses a 2-worker PyTorch DataLoader and calls it optimal without a sweep; a stronger loader could shrink the 15x internal speedup. Throughput values have no error bars, and igb-large accuracy is reported from 3 epochs and a single run. The artifact is public but only fully covers ogbn-products, with the DOI still TBD. None of these are load-bearing flaws.\n\nThe central argument holds: PP-GNNs match MP-GNN accuracy on these benchmarks and, once data loading is engineered properly, become throughput-competitive or better. The paper is a serious piece of systems work, and a serious referee should engage with it. I would recommend minor-to-major revision to address the missing strongest baseline or at least discuss it explicitly, add throughput variance, and strengthen the igb-large evaluation.","headline":"A solid systems paper with a real contribution, but the headline 9.9x speedup is only as strong as the MP-GNN baselines, and the paper never tests the strongest combination of sampler and caching.","tokens_in":26235,"tokens_out":2766,"would_cite":true,"duration_ms":26816,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Optimized pre-propagation GNNs train on average 9.9x faster than sampling-based GNNs on large graphs.","keywords":["pre-propagation GNNs","graph sampling","data loading","chunk reshuffling","GNN training systems","large-scale graph learning","node classification","input expansion problem"],"falsifier":"Run the identical benchmarks with a vanilla pre-propagation baseline that uses many data-loader workers or a CUDA-aware loader, and with sampling-based baselines whose sampler, fanout, and batch size are tuned per dataset; if the optimized pipeline's throughput lead over both falls well below the reported 9.9x average, the central claim would need to be qualified to specific system configurations.","tokens_in":25217,"feed_emoji":"⚡","tokens_out":5246,"duration_ms":47409,"temperature":0.7,"pith_summary":"This paper sets out to show that pre-propagation GNNs—models that move neighbor feature aggregation into a one-time preprocessing step so training is a dense model on precomputed hop features—can be made practically fast on graphs with up to 100 million nodes. The authors argue that the reason these models have looked slow is not computation but data loading: the lightweight dense training step is dominated by assembling and transferring batches, and the precomputed features can expand beyond host memory. They characterize the bottleneck, then engineer around it with a custom batch assembler, double-buffer prefetching, chunk-level reshuffling that moves assembly to the GPU, and direct storage access. Their central result is that optimized pre-propagation GNNs train on average 9.9x faster than sampling-based message-passing GNNs on large graph benchmarks, with speedups reaching two orders of magnitude and accuracy matching or exceeding the baselines. If true, this converts a niche theoretical idea into a practical default for large-scale node classification.","feed_headline":"Optimized pre-propagation GNNs train 9.9x faster than sampled GNNs","feed_subtitle":"Data-loading fixes turn a niche model class into a practical choice for graphs with 100 million nodes.","key_machinery":"The argument rides on treating the precomputed hop-feature matrices as a dense training set and making their movement the object of optimization. The load-bearing pieces are: a custom batch assembler that gathers scattered node vectors with a single index operation; a double-buffer prefetching scheme on the GPU that overlaps data transfer with compute using separate streams; and chunk reshuffling, which shuffles contiguous chunks of node indices rather than individual nodes so that entire chunks move over the bus and batch assembly happens on the GPU. Chunk reshuffling also enables direct storage access for data too large for host memory. The one-time preprocessing cost, multiplying operator matrices by input features, is amortized over training runs and hyperparameter searches.","core_discovery":"Pre-propagation GNNs match sampling-based message-passing GNNs in accuracy while their training time is dominated by data loading, not compute. After applying efficient host-side batch assembly through a single index operation, double-buffer prefetching that pipelines data movement with GPU computation, and chunk reshuffling that lets batches be assembled on the GPU from contiguous chunks, the authors report a 15x average throughput improvement over vanilla implementations. On three large graphs, optimized pre-propagation GNNs achieve on average 9.9x and up to two orders of magnitude higher training throughput than message-passing GNNs run with state-of-the-art samplers, while reaching higher test accuracy in the reported settings. The gain grows with receptive-field size because pre-propagation training cost grows sublinearly with hops, and the proposed storage access path handles input sizes that exceed host memory.","pith_inferences":["The data-loading optimizations are not GNN-specific: any dense training pipeline that gathers batches from large precomputed tensors could reuse chunk reshuffling and double-buffer prefetching with similar gains.","If the accuracy trend on the largest benchmark persists with more epochs and multiple seeds, the case for pre-propagation GNNs strengthens beyond systems into modeling; this can be tested by running the models to convergence.","Chunk granularity creates a tradeoff between shuffling quality and transfer efficiency, and on non-homophilous graphs or tasks that need strict batch randomness, larger chunks may matter more than the reported datasets suggest.","The complexity analysis predicts even larger relative speedups when input feature dimension is high, so high-dimensional-feature graphs are a natural next test of the central claim."],"forward_implications":["For node classification on graphs with tens to hundreds of millions of nodes, pre-propagation GNNs become a credible default: they deliver comparable or better accuracy with far higher throughput.","As the receptive field grows, the pre-propagation advantage increases because training cost grows sublinearly with hops, making deeper propagation affordable.","Inputs too large for host memory can be trained on from storage with modest slowdown, removing a major scalability barrier.","Data transfer volume during training is one to two orders of magnitude smaller for pre-propagation GNNs than for sampling-based message-passing GNNs, a decisive advantage on bandwidth-limited systems.","A one-time preprocessing cost that is smaller than a single training run becomes negligible across many runs, so the comparison favors pre-propagation GNNs in realistic tuning workflows."],"supporting_citations":[{"why":"Defines SGC, the simplest pre-propagation GNN whose training reduces to one linear layer on precomputed features.","marker":"(Wu et al., 2019)"},{"why":"Introduces SIGN, the pre-propagation architecture used as a main efficiency benchmark.","marker":"(Frasca et al., 2020)"},{"why":"Introduces HOGA, the more expressive pre-propagation model used in the accuracy and throughput comparisons.","marker":"(Deng et al., 2024)"},{"why":"Defines the neighbor-explosion problem and GraphSAGE, the message-passing backbone used as a baseline.","marker":"(Hamilton et al., 2017)"},{"why":"Provides the LABOR sampler, the state-of-the-art sampling baseline the paper compares against.","marker":"(Balin & Çatalyürek, 2024)"},{"why":"Supplies GraphSAINT, a graph-wise sampling baseline used in the accuracy comparisons.","marker":"(Zeng et al., 2020)"},{"why":"Provides the IGB datasets, including igb-large whose expanded preprocessed input drives the input-expansion scenario.","marker":"(Khatua et al., 2023)"},{"why":"Provides the OGB datasets, including ogbn-products and ogbn-papers100M used for accuracy and scalability evaluation.","marker":"(Hu et al., 2020)"}],"fun_headline_variants":["Pre-propagation GNNs get 9.9x speedup with data-loading fixes","Data-loading fixes make pre-propagation GNNs 9.9x faster","Pre-propagation GNNs: 9.9x faster than sampling with same accuracy","Optimized pre-propagation GNNs match accuracy, train 9.9x faster","Pre-propagation GNNs scale: 9.9x training speedup over sampling"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline speedups assume the comparison systems—the vanilla pre-propagation baseline and the sampling-based message-passing systems—are each configured at a representative level of effort; a stronger data-loading baseline or differently tuned samplers could shrink the reported margins.","fun_headline_variants_meta":{"raw":{"variants":["Pre-propagation GNNs get 9.9x speedup with data-loading fixes","Data-loading fixes make pre-propagation GNNs 9.9x faster","Pre-propagation GNNs: 9.9x faster than sampling with same accuracy","Optimized pre-propagation GNNs match accuracy, train 9.9x faster","Pre-propagation GNNs scale: 9.9x training speedup over sampling"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00085,"raw_usage":{"total_tokens":3708,"prompt_tokens":967,"completion_tokens":2741,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":583,"completion_tokens_details":{"reasoning_tokens":2625}},"tokens_in":583,"tokens_out":2741,"duration_ms":17427,"temperature":1.0,"reasoning_tokens":2625,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:12:13.730453+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical benchmarks with a vanilla pre-propagation baseline that uses many data-loader workers or a CUDA-aware loader, and with sampling-based baselines whose sampler, fanout, and batch size are tuned per dataset; if the optimized pipeline's throughput lead over both falls well below the reported 9.9x average, the central claim would need to be qualified to specific system configurations.","supporting_citations":[{"cited_title":"Less is More: Hop-Wise Graph Attention for Scalable and Generalizable Learning on Circuits","cited_arxiv_id":null,"evidence_quote":"Introduces HOGA, the more expressive pre-propagation model used in the accuracy and throughput comparisons."},{"cited_title":"Inductive Representation Learning on Large Graphs","cited_arxiv_id":null,"evidence_quote":"Defines the neighbor-explosion problem and GraphSAGE, the message-passing backbone used as a baseline."},{"cited_title":"u rek, \\","cited_arxiv_id":null,"evidence_quote":"Provides the LABOR sampler, the state-of-the-art sampling baseline the paper compares against."},{"cited_title":"GraphSAINT: Graph Sampling Based Inductive Learning Method","cited_arxiv_id":null,"evidence_quote":"Supplies GraphSAINT, a graph-wise sampling baseline used in the accuracy comparisons."},{"cited_title":"S., Taleka, B., Ma, T., Song, X., and Hwu, W.-m","cited_arxiv_id":null,"evidence_quote":"Provides the IGB datasets, including igb-large whose expanded preprocessed input drives the input-expansion scenario."}],"review_version":1}