{"id":"20061ecc-de19-4e9d-b989-5483f0f14f42","arxiv_id":"2508.03854","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":2,"one_line_summary":"A two-dimensional sparse data-and-model parallel scheme, plus a momentum-scaled row-wise AdaGrad optimizer, is reported to scale DLRM training to 4,000 GPUs with model quality parity.","lead":"This paper proposes a two-dimensional sparse parallelism approach for training large recommendation models, splitting embedding tables across GPUs in two dimensions rather than fully sharding them. It reports near-linear training speed scaling up to 4,000 GPUs while keeping model performance roughly unchanged.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Attribution of near-linear scaling to the 2D layout is the weakest link; the abstract does not rule out the optimizer change or hardware/network conditions as the actual cause.","rationale":"The paper's headline is that a new 2D sparse parallelism scheme achieves near-linear scaling to 4K GPUs with performance parity. For this claim to hold, the 2D layout itself must be the cause of the observed speedup and the parity must persist under the new optimizer. The abstract simultaneously introduces two changes: the 2D layout and the momentum-scaled row-wise AdaGrad algorithm. Since no ablations are visible in the abstract, the argument is vulnerable to a confound: the scaling gain or the parity could be driven by the optimizer change or by cluster-specific hardware and network conditions. This is not an allegation of misconduct; it is a standard attribution problem in systems papers. A second issue is memory: adding data parallelism on top of model parallelism appears, at face value, to introduce replication, so the claimed peak-memory reduction needs a concrete breakdown rather than an abstract assertion. The reader's weakest assumption already identified exactly this attribution concern, and I agree with it. Because the full text, data, and code are unavailable, the paper remains unverdictable; the stress-test does not move the reader's verdict, so UNCHANGED is appropriate. The proposed concrete test is an ablation that isolates the 2D layout from the optimizer and hardware confounds and would settle whether the central scaling claim is real.","tokens_in":704,"tokens_out":3049,"duration_ms":34573,"concrete_test":"In the full paper, locate the main scaling experiment and its baseline. Run one ablation: keep the momentum-scaled row-wise AdaGrad optimizer, the same global batch size, and the same hardware, but replace the 2D sparse layout with the standard fully-sharded model-parallel layout. If the throughput-vs-GPU curve remains within a few percent of the 2D-layout curve, then the 2D layout is not the causal source of the claimed near-linear scaling; if the final evaluation metric changes, the parity claim is also tied to the optimizer rather than the layout.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is an empirical systems claim: 2D sparse parallelism yields near-linear throughput scaling to 4K GPUs while preserving model quality. The load-bearing assumption is that the layout itself, not the momentum-scaled row-wise AdaGrad optimizer introduced in the same work, is what produces the scaling and parity. The abstract gives no ablation, so the optimizer change is a confound. A second confound is infrastructure: near-linear weak scaling on a specific cluster (e.g., 4K GPUs with a given NVLink/InfiniBand topology) may not generalize, and if the baseline uses full sharding with a different communication schedule, the comparison could be unfair. Additionally, 'reduces peak memory' is non-obvious because adding data parallelism on top of model parallelism replicates tables across data-parallel replicas; without a memory breakdown the claimed memory benefit is unsupported. These are not internal contradictions; they are unverified empirical attributions.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-dimensional sparse parallelism strategy for large-scale deep learning recommendation model training. Instead of fully sharding embedding tables across all GPUs, the method adds data parallelism on top of model parallelism, aiming to improve all-to-all communication efficiency and reduce peak memory consumption. The abstract also introduces a momentum-scaled row-wise AdaGrad optimizer to mitigate quality loss from the shift in training paradigm. The empirical claims are nearly linear training speed scaling up to 4K GPUs and model performance parity compared with existing approaches.","tokens_in":891,"tokens_out":3331,"duration_ms":35142,"significance":"If the empirical claims are substantiated, the work would address a recognized bottleneck in industrial-scale recommendation model training: the high communication and memory cost of sparse embeddings. The two-dimensional layout and the accompanying optimizer variant are plausible practical contributions, and the scale of the reported experiments (up to 4K GPUs) is ambitious. However, the significance cannot be assessed from the abstract alone. The central claims of near-linear scaling and performance parity require detailed experimental evidence, including baseline comparisons, datasets, and the full system architecture. The paper's value is conditional on the missing evidence.","major_comments":[{"comment":"The abstract states 'achieves nearly linear training speed scaling up to 4K GPUs' without specifying whether the scaling is weak or strong, the hardware and network topology (e.g., NVLink, InfiniBand), the baseline distributed training configuration, or the number of runs and variance. As written, this claim cannot be reproduced or independently assessed.","section":"Abstract, scaling claim"},{"comment":"The momentum-scaled row-wise AdaGrad algorithm is introduced in the same abstract as the layout change and is described as mitigating 'performance losses associated with the shift in training paradigms', yet no ablation is reported that separates the effect of the 2D layout from the effect of the new optimizer. Without such an ablation, the attribution of the scaling and parity results to the 2D layout is not established.","section":"Abstract, optimizer confound"},{"comment":"The claim that adding data parallelism on top of model parallelism 'reduces peak memory consumption' is non-obvious because data parallelism replicates embedding tables across the data-parallel dimension. A rigorous memory accounting that includes embedding tables, activations, gradients, and optimizer states is needed to support this claim, and the abstract provides no such breakdown.","section":"Abstract, memory claim"},{"comment":"The assertion of 'maintaining model performance parity' is stated without defining the evaluation metric (e.g., AUC, NDCG, loss), the baseline systems used for comparison, or the statistical significance of the parity claim. 'Parity' is a strong claim and requires a clear experimental protocol that is not visible in the abstract.","section":"Abstract, performance parity"}],"minor_comments":[{"comment":"The phrase 'fully parallelism strategies' should be 'fully parallel strategies', and 'post significant scalability challenges' should be 'pose significant scalability challenges'.","section":"Abstract, language"},{"comment":"The abstract uses 'two-dimensional sparse parallelism' without precisely defining the two dimensions; a figure or formal algorithm description in the full text would help readers understand the proposed layout.","section":"Abstract, definition of 2D parallelism"},{"comment":"The statement 'setting a new state-of-the-art benchmark for recommendation model training' is unsupported by any named comparison systems or public benchmarks in the abstract.","section":"Abstract, state-of-the-art claim"},{"comment":"The abstract does not mention prior work on large-scale DLRM training, such as Facebook's DLRM, HugeCTR, or embedding table sharding techniques; the full paper should provide these references to position the contribution.","section":"Abstract, related work"}],"recommendation":"uncertain","confidential_remarks":"The manuscript is only available as an abstract at the time of review, so I cannot verify the empirical claims. The central claims of near-linear scaling to 4K GPUs and performance parity are plausible but unsupported by the abstract alone. The main risk is that the optimizer change, rather than the 2D layout, may be responsible for the reported gains, and the memory benefit is argued without a quantitative model. I recommend obtaining the full text before any decision; if the full text contains the missing ablations and experiment details, the paper may be suitable for publication in a systems venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is an abstract-only read, so keep that in mind. The headline claim—near-linear scaling to 4K GPUs with performance parity—is exactly the kind of thing that moves the needle in recommendation training, and the proposed 2D sparse parallelism is a sensible response to a real bottleneck: fully sharded embedding tables create stragglers, heavy all-to-all traffic, and activation memory pressure. The idea of adding data parallelism on top of model parallelism to cut communication and peak memory is not obviously wrong, and the momentum-scaled row-wise AdaGrad is a reasonable attempt to keep quality when the training paradigm shifts. Credit where due: the problem is well-motivated, the combination is new to my knowledge, and the scale alone makes it worth a serious look.\n\nBut the soft spots are real, and they are the ones the stress-test flags. The abstract gives no ablation separating the 2D layout from the optimizer change. If most of the speedup or parity comes from the optimizer, or from the specific cluster's network, then the central attribution to the parallelism scheme does not hold. The memory claim is also undersupported: data-parallel replicas of tables cost memory, so 'reduces peak memory' is not self-evident without a breakdown. None of this is an internal contradiction—it is missing evidence, which is expected in an abstract. But it means the paper's key claims are unverified.\n\nI would send this to peer review, because the result is important and the approach is credible. The referees should demand ablations, memory/communication breakdowns, baseline comparisons, and ideally code or detailed reproducibility info. Without those, the scaling claim stays a claim. Also note this is a systems paper, not a theoretical one; the bar is empirical clarity, not formal proof. My reading-group verdict is 'maybe'—I'd be curious to see the full paper, but the abstract alone wouldn't drive a session.\n\nBottom line: the paper deserves a serious referee despite the abstract's overreach. If the full text delivers the ablations, this is a strong industrial contribution. If not, it is a well-written but unsubstantiated claim.","headline":"Plausible and potentially important industrial systems result, but the abstract alone cannot support the scaling claim; the optimizer confound is real and should be checked in review.","tokens_in":1417,"tokens_out":999,"would_cite":false,"duration_ms":13822,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A two-dimensional layout for embedding tables lets recommendation training scale to 4,000 GPUs.","keywords":["two-dimensional sparse parallelism","deep learning recommendation models","embedding tables","model parallelism","data parallelism","all-to-all communication","AdaGrad","large-scale GPU training"],"falsifier":"Run the same 2D layout with the standard (non-momentum) AdaGrad and the 1D fully-sharded layout with momentum-scaled AdaGrad; if the speedup and memory reduction track the optimizer rather than the layout, the central scalability claim fails. A simpler check: profile per-component time and memory (all-to-all, embedding lookup, optimizer update) on the 2D layout; if the all-to-all cost does not drop relative to the 1D baseline, the mechanism is not doing the claimed work.","tokens_in":533,"feed_emoji":"⚡","tokens_out":4226,"duration_ms":44156,"temperature":0.7,"pith_summary":"This paper claims that the bottleneck in training large recommendation models is how embedding tables are spread across GPUs, and that a two-dimensional arrangement fixes it. Rather than sharding every embedding table across all GPUs, the method shards within groups and replicates across groups, so communication stays inside smaller groups and peak memory drops. The paper pairs this layout with a modified AdaGrad optimizer to keep model quality unchanged. If the claim holds, training a trillion-parameter recommendation model can run on thousands of GPUs with near-linear speedup instead of choking on communication and memory imbalance.","feed_headline":"2D sparse parallelism scales recommendation training to 4K GPUs","feed_subtitle":"Sharding embeddings within groups and replicating across groups cuts communication and memory while keeping model quality.","key_machinery":"The central object is the two-dimensional parallelism layout of the embedding tables. It replaces the one-dimensional shard-everything-everywhere scheme with a grid: shard within a group, replicate across groups, so each GPU's all-to-all exchange involves only its group. The momentum-scaled row-wise AdaGrad is the second mechanism; it adjusts per-row adaptive learning rates with momentum to keep convergence on replicated rows close to the fully sharded baseline. Together they carry the paper's scalability and quality claims.","core_discovery":"The paper proposes two-dimensional sparse parallelism as a replacement for fully sharded embedding tables in distributed recommendation model training. In this scheme, the GPU cluster is partitioned into groups: embedding tables are sharded across GPUs within each group (model parallelism), while the same sharded tables are replicated across groups (data parallelism). This keeps the all-to-all lookup communication within each group and removes the need for a global all-to-all, lowering both communication volume and peak activation memory. To prevent the replication from degrading convergence, the paper introduces momentum-scaled row-wise AdaGrad, which scales per-row adaptive learning rates by a momentum term. Across experiments, the method reaches near-linear scaling on up to 4,000 GPUs while matching the model quality of baseline training.","pith_inferences":["By extension, the same two-dimensional trick may apply to other models with large sparse tables, such as graph neural networks or language models with massive vocabularies, wherever a global all-to-all is the bottleneck.","If momentum-scaled AdaGrad is what preserves convergence, the layout could also be paired with other optimizers, but the optimizer's contribution would need separate verification.","A direct comparison of per-GPU memory and communication breakdowns between 2D and 1D layouts, with the optimizer held fixed, would settle whether the 2D layout or the optimizer drives the gains."],"forward_implications":["Training runs on industrial-scale recommendation models can reach near-linear speedup with GPU count instead of being limited by the all-to-all communication bottleneck.","The same quality of model is achievable with the 2D layout plus momentum-scaled AdaGrad, so practitioners do not have to sacrifice accuracy for speed.","Peak memory per GPU is reduced, which allows larger embedding tables or bigger batch sizes to fit on the same hardware."],"supporting_citations":[],"fun_headline_variants":["2D sparse parallelism scales DLRM to 4K GPUs with quality","Communication-light 2D sparse parallelism for DLRM at 4K GPUs","Near-linear scaling to 4K GPUs via 2D sparse parallelism","Cutting memory and communication: 2D sparse parallelism for DLRM","Momentum-scaled AdaGrad unlocks 2D sparse parallelism for DLRM"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim rests on the assumption that the two-dimensional layout itself, rather than the new optimizer or the particular network hardware, is what reduces communication and memory costs, and that this reduction does not simply move the bottleneck elsewhere, such as to embedding activation memory or stragglers.","fun_headline_variants_meta":{"raw":{"variants":["2D sparse parallelism scales DLRM to 4K GPUs with quality","Communication-light 2D sparse parallelism for DLRM at 4K GPUs","Near-linear scaling to 4K GPUs via 2D sparse parallelism","Cutting memory and communication: 2D sparse parallelism for DLRM","Momentum-scaled AdaGrad unlocks 2D sparse parallelism for DLRM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000822,"raw_usage":{"total_tokens":3577,"prompt_tokens":907,"completion_tokens":2670,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":2562}},"tokens_in":523,"tokens_out":2670,"duration_ms":20822,"temperature":1.0,"reasoning_tokens":2562,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T04:11:48.324894+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 2D layout with the standard (non-momentum) AdaGrad and the 1D fully-sharded layout with momentum-scaled AdaGrad; if the speedup and memory reduction track the optimizer rather than the layout, the central scalability claim fails. A simpler check: profile per-component time and memory (all-to-all, embedding lookup, optimizer update) on the 2D layout; if the all-to-all cost does not drop relative to the 1D baseline, the mechanism is not doing the claimed work.","supporting_citations":[],"review_version":1}