{"id":"7ef35bb7-6b50-4d1d-a54b-a41793796d9b","arxiv_id":"2502.07563","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LASP-2 trains linear-attention transformers with long sequences by exchanging GPU memory states in one all-gather step, improving throughput over prior sequence-parallel methods.","lead":"This paper describes a faster way to train neural networks that use linear attention on very long text inputs. It rearranges how partial attention states are shared between GPUs so that only one group communication step is needed, instead of many sequential ones.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 3.4's claimed W-1 communication-traffic reduction for LASP-2 is a counting artifact: a W-way AllGather of W memory states moves (W-1)BHd^2 bytes per rank, exactly what LASP-1's ring moves, so the advantage must rest on latency/overlap, not traffic.","rationale":"The central claim is that LASP-2 reduces communication cost by replacing LASP-1's ring with a single AllGather. The weakest point is the cost model: Section 3.4 counts one AllGather as one message of size BHd^2, but a W-way AllGather inherently moves (W-1) of those messages into every rank. Correcting this removes the factor-(W-1) traffic advantage, so the theoretical support for 'communication improvements' collapses to a latency/overlap argument. This is load-bearing because the abstract and Section 3.4 explicitly advertise the traffic reduction, and the paper's scalability story (Fig. 4) is built on communication cost being sequence-length-independent. I do not think this refutes the method: eliminating LASP-1's sequential ring dependency is a real computational-parallelism gain, and the 15.2% over LASP-1 is plausible. But the paper must re-derive the model and re-benchmark with profiled byte counts and an apples-to-apples Ring Attention baseline (which Section 4.1 currently is not, since it omits the right-product kernel trick). The 2048K/64-GPU abstract claim also conflicts with Table 6's OOM at that configuration; this is likely a typo but should be fixed. These are correctable, so the verdict stays CONDITIONAL rather than moving further.","tokens_in":19228,"tokens_out":15182,"duration_ms":137946,"concrete_test":"Insert NCCL instrumentation (e.g., torch.profiler with record_shapes=True and NCCL traces) into the released code and record bytes sent and received per rank for one forward+backward iteration of LASP-1 and LASP-2, with identical B,H,d,W and no compute overlap. If LASP-2's per-rank byte count is approximately equal to LASP-1's (i.e., 4(W-1)BHd^2 for both, not 4BHd^2 for LASP-2), then Section 3.4's factor-(W-1) traffic reduction is a counting artifact and the paper's communication-advantage claim must be restated in terms of latency and overlap.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.4 lists the per-step traffic as BHd^2 for both LASP-1 and LASP-2, then multiplies by 2(W-1) steps for LASP-1 and 2 for LASP-2, concluding total traffic of 2(W-1)IBHd^2 vs 2IBHd^2. This is inconsistent: the LASP-2 AllGather is one collective, but it is not one message. To gather all W memory states (each [B,H,d,d]), every rank must receive (W-1)BHd^2 bytes, and in a bidirectional accounting it sends the same volume; over forward+backward this is 4(W-1)BHd^2 per rank. LASP-1's ring, with W-1 send+receive steps, moves exactly the same 4(W-1)BHd^2 per rank. Therefore the 'reduced by a factor of W-1' conclusion in Section 3.4 does not follow from byte volume. The genuine algorithmic benefit is different: one collective launch instead of W-1 P2P launches, and removal of LASP-1's sequential inter-chunk dependency so all ranks compute their inter-chunk outputs concurrently. That is a latency/parallelism benefit, not a traffic reduction, and it is the load-bearing part of the communication-advantage claim. The empirical speedups may still be real, but they need to be explained and benchmarked under this corrected model; the current abstract attributes them to reduced communication.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"LASP-2 proposes a new sequence-parallelism scheme for training linear-attention and hybrid linear/standard-attention transformers. Each rank computes its chunk-local memory state M_t = K_t^T V_t (shape [B,H,d,d]), performs a single AllGather to replicate all W states, and then computes its output Q_t M_{1:T} locally; the masked autoregressive variant separates intra-chunk (left-product with causal mask) and inter-chunk (gathered prefix-sum) computations, and the backward passes use an analogous AllGather on gradients dM_t. The paper claims that this reorganization reduces communication traffic by a factor of W-1 relative to the authors' earlier LASP-1, improves computation and communication parallelism, and reports throughput gains of 15.2% over LASP-1 and 36.6% over Ring Attention at sequence length 2048K on 64 A100 GPUs using a Linear-Llama3-1B model, together with scalability measurements up to 4096K on 128 GPUs and convergence results on several linear-attention variants. An extension, LASP-2H, applies the same allgather-based design to standard attention layers for hybrid models.","tokens_in":19558,"tokens_out":16305,"duration_ms":137820,"significance":"The algorithmic idea is clean and directly derived from the associativity of the right-product form of linear attention; there are no fitted parameters, the forward/backward algorithms are fully specified, code is released, and the paper reports concrete per-configuration throughput and memory numbers (Table 6) that are falsifiable. The single-AllGather design has genuine operational virtues over the ring-style P2P exchange in LASP-1: one collective launch instead of W-1 send/receive pairs, removal of the serial inter-chunk dependency so that all ranks compute their inter-chunk outputs concurrently, and straightforward overlap with intra-chunk computation in the masked case. If the measured speedups survive a corrected communication-accounting analysis, this is a practically useful, low-risk improvement for very-long-sequence training. The significance is substantially tempered, however, by the finding that the paper's headline traffic-reduction argument (Section 3.4) is a step-counting artifact: byte-for-byte, the AllGather moves the same volume as LASP-1's ring.","major_comments":[{"comment":"Section 3.4, communication-traffic formulas (2(W-1)IBHd^2 for LASP-1 vs 2IBHd^2 for LASP-2): the model counts each AllGather as one message of BHd^2 bytes, but an AllGather that replicates all W memory states moves (W-1)*BHd^2 bytes into each rank and the same volume out of each rank; including forward and backward passes this is 4(W-1)*BHd^2 per rank per iteration, exactly the volume of LASP-1's 2(W-1) send/receive steps. The claim that LASP-2 reduces communication traffic by a factor of W-1 is therefore a step-counting artifact and is not supported by byte accounting. The genuine benefits are fewer collective launches, the removal of LASP-1's serial inter-chunk dependency, and overlap of the AllGather with intra-chunk computation; Sections 1 and 3.3 and the abstract should be revised to attribute the measured speedups to these effects rather than to reduced traffic.","section":"Section 3.4"},{"comment":"The headline result, training speed improvements of 15.2% over LASP and 36.6% over Ring Attention at sequence length 2048K across 64 GPUs, is contradicted by Table 6, which reports OOM for LASP-2 at 2048K with 64 GPUs, and Figure 3's caption states that the speed comparisons used 64 GPUs with batch size 1. The configuration that produced the 2048K speedup numbers (for example a different batch size, a different SP size, or an error in Table 6) must be identified and reported consistently before the abstract's central claim can be verified.","section":"Abstract / Section 4.2 / Table 6"},{"comment":"The baselines Megatron-SP and Ring Attention are evaluated on linear-attention layers without the right-product kernel trick, keeping each method's original standard-attention computational manner. The reported 36.6% gain over Ring Attention at 2048K therefore conflates the quadratic-to-linear compute saving with the sequence-parallelism communication advantage. The comparison against LASP-1 is the fair right-product-aware baseline; the Ring Attention comparison should either be re-run with a right-product-first linear kernel inside the ring communication pattern or explicitly re-scoped, and the paper's central empirical claim should rest on the LASP-1 comparison.","section":"Section 4.1"}],"minor_comments":[{"comment":"Algorithm 3, line 5: the notation 'dM1:T = Sum([dM]^T_{t+1})' reads as a suffix sum over chunks t+1 through T, but the unmasked backward pass requires dM_{1:T} to be the sum of dM_i over all chunks i = 1..T; please correct the subscript and state explicitly how this quantity differs from the SuffixSum used in Algorithm 4.","section":"Algorithm 3 (Appendix A.1)"},{"comment":"Typos and grammar: 'Ring Attentoin' (Section 4.1), 'decices' (Section 3.3), 'lenghth' (Appendix A.5.3), 'striked a balance' (Section 4.4), and 'each memory state will has' (Section 3.4) should be corrected.","section":"Section 4.1"},{"comment":"Table 6 reports identical per-GPU memory (25.6 GB) for sequence lengths 2K through 16K at every fixed GPU count; please clarify whether this plateau reflects a memory floor from model weights and optimizer states, since otherwise the activation-memory scaling shown in Figure 4 is not visible at these settings.","section":"Table 6"},{"comment":"Figure 1's caption ('Colored chunks represent inter-chunks') does not make clear which regions are intra-chunk versus inter-chunk, and Figure 2's AG/RS and AG/No labels are too small to read at print resolution; both figures should be redrawn for clarity.","section":"Figures 1 and 2"},{"comment":"The statement that the communication cost for the all-gather operation is independent of the sequence or chunk length is true per message, but the total gathered volume is W*BHd^2 and grows with the SP size; the abstract's phrasing that the communicated tensors' sizes are independent of the sequence length should be qualified so that readers do not infer constant communication as the device count grows.","section":"Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"The comparison against LASP-1 is a same-group baseline, and the novelty relative to LASP-1 is the main incremental claim; the paper's own traffic-reduction argument in Section 3.4 is the part most likely to be challenged, and the 2048K/64-GPU OOM inconsistency in Table 6 should be resolved before the headline numbers are publicized. The release of code and the full algorithm listings (including the masked/unmasked backward passes) is a strength, as is the concrete per-configuration scaling table. The fit of the submission is appropriate for a systems/ML venue, provided the cost model and baseline comparison are corrected."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"LASP-2 is a genuine, incremental step in sequence parallelism for linear attention: replacing LASP-1's ring of P2P sends with a single AllGather of the per-chunk memory states (d×d per head), both forward and backward, and giving the same treatment to standard attention in hybrid models. The algorithms are correct, the code is released, and the 512K/128-GPU scaling results look credible. That part deserves credit.\n\nThe paper's main weakness is that Section 3.4 mis-states its own communication advantage. It counts an AllGather as one message and concludes a W−1 reduction in traffic. But an AllGather of W memory states moves (W−1)·BHd² bytes per rank, exactly the same volume LASP-1's ring moves. The real advantages are lower launch overhead, no sequential dependency between chunks, and better overlap with intra-chunk compute. Those may be worth the reported 15–36%, but the paper doesn't actually benchmark traffic or latency separately.\n\nThere's also a clear mismatch between the abstract and Table 6. The abstract claims the 15.2%/36.6% speedups at 2048K across 64 GPUs, but Table 6 lists OOM for 2048K with 64 GPUs (only 128 GPUs succeed). That is an internal inconsistency, not a minor typo, and it needs to be resolved.\n\nThe baseline comparison is also apples-to-oranges: LASP-2 uses a right-product kernel, while the Ring Attention and Megatron-SP baselines do not. That means part of the speedup may come from the kernel, not the sequence-parallelism design. The right fix is to benchmark all methods with the same right-product kernel, or at least report both variants.\n\nNone of these are load-bearing flaws. The algorithmic core is sound, and the paper is honest about its scope (batch size 1, fixed sequence lengths). It is a solid systems paper for the long-context linear-attention audience. I'd send it to peer review with a request to fix the cost model, reconcile the abstract with Table 6, and equalize the kernel in the baselines. The results should then speak for themselves.","headline":"A real algorithmic improvement for sequence-parallel linear attention, but the paper's own cost model overstates the traffic reduction and the abstract conflicts with its own OOM table.","tokens_in":20105,"tokens_out":3756,"would_cite":true,"duration_ms":29118,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that sequence-parallel linear attention can be reorganized around a single AllGather of chunk memory states, giving 15.2% and 36.6% training-speed gains over LASP and Ring Attention at 2048K sequence length.","keywords":["sequence parallelism","linear attention","AllGather collective","memory states","long sequence training","hybrid attention models","distributed training"],"falsifier":"Run LASP-2 and LASP-1 on the same cluster and sequence length, profile only the communication phases, and record both wall-clock time and per-device bytes; if the AllGather transfers $(W-1)BHd^2$ bytes per device (same as the ring) and the total communication time scales with bytes rather than with step count, then the claimed $W-1$ traffic reduction is not real and the measured speedup must be attributed to launch overhead and overlap.","tokens_in":19003,"feed_emoji":"⚡","tokens_out":11061,"duration_ms":92571,"temperature":0.7,"pith_summary":"The paper introduces LASP-2, a sequence-parallel training scheme for linear attention transformers on very long inputs. Because each attention chunk can be summarized by a $d\\times d$ memory state $M_t=K_t^\\top V_t$, one AllGather can place all chunk states on every device, after which prefix sums and a local multiplication produce the output. The paper argues this replaces the chained ring-style sends of the earlier LASP method with a single well-optimized collective, raising both communication and computation parallelism, and reports 15.2% higher throughput than LASP and 36.6% higher than Ring Attention at sequence length 2048K across 64 GPUs. A hybrid extension, LASP-2H, applies the same AllGather pattern to standard attention layers so mixed linear/softmax models share one SP design. If the claim holds, multi-million-token training of linear and hybrid transformers becomes faster while communication overhead stays independent of sequence length.","feed_headline":"36% faster: one AllGather replaces ring sends in linear attention","feed_subtitle":"LASP-2's single collective on d×d memory states trains 2M-token sequences faster than Ring Attention.","key_machinery":"The central object is the memory state $M_t = K_t^\\top V_t \\in \\mathbb{R}^{d\\times d}$, computed per chunk from that chunk's keys and values, plus its all-gathered prefix sums. Because linear attention satisfies $O = Q(K^\\top V) = QM$, each chunk's contribution to every later position is fully summarized by its $d\\times d$ state; LASP-2 uses one AllGather to collect all these state matrices, then prefix/suffix sums to build $M_{1:t-1}$ for the inter-chunk output and a masked intra-chunk term for causality. This object is what makes the communication volume independent of sequence length and what lets one collective replace a ring of sends.","core_discovery":"The paper's central claim is that in LASP-2 the forward pass needs exactly one AllGather of the memory states $[M_t]_1^T$, each of shape $d\\times d$, followed by a sum or prefix-sum to get $M_{1:T}$; the backward pass needs one more AllGather on the gradients $dM_t$. With causal masking, the computation is decomposed into a parallel intra-chunk term $O_{t,\\mathrm{intra}}=[(Q_tK_t^\\top)\\odot\\Psi]V_t$ and an inter-chunk term $O_{t,\\mathrm{inter}}=Q_tM_{1:t-1}$ built from gathered states. Thus the paper's step-counting model gives two communication steps per iteration instead of $2(W-1)$ for LASP-1, and the per-step traffic $BHd^2$ is independent of sequence length. LASP-2H transfers the same single-collective principle to standard attention by AllGathering $K_t,V_t$, and experiments on Linear-Llama3 up to 2048K tokens support the efficiency claim.","pith_inferences":["The Section 3.4 cost model counts steps, not bytes; an AllGather of all $W$ states moves $(W-1)BHd^2$ bytes per device, the same volume as LASP-1's ring, so the real advantage is likely fewer launches, lower latency, and easier overlap rather than reduced data movement.","The same chunk-state-then-AllGather pattern should transfer to other linear RNNs with matrix-valued additive states, such as SSM-style recurrences, since their recurrence has the same sum-of-states form; the paper does not make this claim.","LASP-2H's AllGather-based standard attention trades higher latency for mask flexibility, suggesting it will combine naturally with variable-length packing and document masks, which ring-based strategies handle less easily.","On very fast interconnects the throughput gap over ring methods should narrow because collective latency dominates less; a direct byte-level comparison would quantify how much of the reported speedup is overlap versus raw traffic."],"forward_implications":["At sequence length 2048K across 64 A100 GPUs, LASP-2 reports 15.2% higher throughput than LASP and 36.6% higher than Ring Attention.","Communication per step stays at $BHd^2$ bytes regardless of sequence length, so the relative cost shrinks as sequences get longer.","Each iteration uses only two AllGather collectives (forward and backward), versus $2(W-1)$ point-to-point hops for LASP-1.","For hybrid models, LASP-2H gives both linear and standard attention layers the same AllGather-based design, gathering $K_t,V_t$ for standard layers and $M_t$ for linear layers.","With 128 GPUs, the method trains a Linear-Llama3-1B model on a 2048K-token sequence without running out of memory, keeping per-GPU usage around 57.8 GB."],"supporting_citations":[{"why":"Prior sequence-parallel method for linear attention; the ring-style P2P baseline that LASP-2 reorganizes.","marker":"Sun et al., 2024a"},{"why":"Introduces linear attention and the right-product associativity O=Q(K^T V) that underlies chunk-local memory states.","marker":"Katharopoulos et al., 2020"},{"why":"Ring Attention baseline; the existing SP strategy with chained point-to-point KV transfers that LASP-2 is compared against.","marker":"Liu et al., 2023"},{"why":"Megatron-SP baseline; the block-based sequence parallelism method LASP-2 outperforms in speed tests.","marker":"Korthikanti et al., 2022"},{"why":"Gated linear attention; supplies the causal-mask computation decomposition into intra- and inter-chunk terms used with masking.","marker":"Yang et al., 2023"},{"why":"Llama3 architecture and the AllGather-based context-parallelism practice that LASP-2H adopts for standard attention layers.","marker":"Dubey et al., 2024"}],"fun_headline_variants":["One AllGather per pass: LASP-2 redefines linear attention SP","LASP-2: single AllGather on memory states cuts comms, 36% faster","Rethinking sequence parallelism: one AllGather replaces ring sends","Two AllGathers per step: LASP-2 scales to 2M tokens faster"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that replacing the ring of point-to-point sends with one AllGather is genuinely cheaper in wall-clock time; the paper's cost model counts communication steps rather than bytes, and an AllGather of all $W$ states moves the same per-device bytes as the ring it replaces.","fun_headline_variants_meta":{"raw":{"variants":["One AllGather per pass: LASP-2 redefines linear attention SP","LASP-2: single AllGather on memory states cuts comms, 36% faster","Rethinking sequence parallelism: one AllGather replaces ring sends","Two AllGathers per step: LASP-2 scales to 2M tokens faster"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000324,"raw_usage":{"total_tokens":1892,"prompt_tokens":1090,"completion_tokens":802,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":706,"completion_tokens_details":{"reasoning_tokens":711}},"tokens_in":706,"tokens_out":802,"duration_ms":7245,"temperature":1.0,"reasoning_tokens":711,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T12:17:44.072260+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run LASP-2 and LASP-1 on the same cluster and sequence length, profile only the communication phases, and record both wall-clock time and per-device bytes; if the AllGather transfers $(W-1)BHd^2$ bytes per device (same as the ring) and the total communication time scales with bytes rather than with step count, then the claimed $W-1$ traffic reduction is not real and the measured speedup must be attributed to launch overhead and overlap.","supporting_citations":[],"review_version":1}