{"id":"ea12e7d3-1e42-4214-8ab3-14a11fb2afb2","arxiv_id":"2507.01004","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"ZeCO sequence parallelism for linear attention uses a pipelined All-Scan collective to cut communication volume and time, achieving near-linear scaling up to 256 GPUs with 8M-token sequences.","lead":"ZeCO is a new way to split long input sequences across many GPUs for linear attention models, using a pipelined All-Scan step that moves only the minimal state between neighboring devices. The authors report near-linear throughput scaling up to 256 GPUs and 8M tokens, and about 60 percent higher throughput than the prior best sequence-parallel method.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'zero overhead' claim assumes All-Scan is fully hidden by the diagonal-attention compute and extra I/O is negligible, but Tables 2-4 show All-Scan latency grows with P and ZeCO's overhead over the DP baseline reaches ~50% at 128 GPUs, so Eq. 13-14 are unsupported.","rationale":"The central claim is that ZeCO attains the minimum possible SP runtime (Eq. 13-14) by overlapping All-Scan with local computation and keeping extra compute/I/O negligible. Both pillars are load-bearing. The paper's own numbers can test them. Table 2 shows All-Scan growing from 0.138 ms (P=8) to 0.604 ms (P=256), so communication is not asymptotically vanishing. The overlappable work is only the diagonal P computation, which is O(L·C) per head and, under the reported settings, is orders of magnitude smaller than the All-Scan latency at scale; thus overlap cannot hide the communication. Tables 3-4 show ZeCO minus DP baseline growing from ~0.9 ms (P=8) to ~3.3 ms (P=128) on a ~6.4 ms baseline, a 50% overhead, far above the ~0.5 ms All-Scan time; the excess must come from the extra two-pass HBM traffic and kernel launches in Algorithm 1, contradicting the claimed '<1%' extra cost. These two failures mean Eq. 13-14 do not follow from the stated assumptions, so the 'zero communication overhead' headline is overstated. The method may still be a useful faster SP alternative (ZeCO beats LASP-1/LASP-2 on throughput), which is why the verdict remains CONDITIONAL rather than REJECT, but the theoretical optimality proof and the 'zero overhead' terminology need to be scoped to the regime where the overlap holds, with sensitivity data.","tokens_in":15383,"tokens_out":14299,"duration_ms":267284,"concrete_test":"On the same 128-GPU cluster, instrument Algorithm 1 with CUDA events around the All-Scan kernel (stream 1) and the diagonal P-computation kernel (stream 2) for L=16k/device, H=16, dk=128. Record start/end times on both streams: if the P-computation window does not cover the All-Scan window (overlap < 90%), then T_All_Scan - Toverlaped_comp ≈ T_All_Scan and Eq. 13's cancellation is invalid. Then disable the All-Scan (replace S_recv with a zero tensor) and re-measure ZeCO vs the DP baseline; the remaining gap isolates ε. If that gap exceeds 5% of the DP runtime, the paper's 'less than 1%' statement is contradicted. Report both numbers across P=8, 64, and 256.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 claims T_ZeCO ≈ T_ideal - Toverlapped + τ(S) (Eqs. 13-14), which requires two load-bearing assumptions: (i) All-Scan is almost fully overlapped with the local diagonal-attention computation, and (ii) ε, the extra compute/I/O of Algorithm 1's two-pass schedule, is negligible. The paper provides no direct overlap measurement, and its own data contradict both assumptions. Table 2 shows the standalone All-Scan time growing from 0.138 ms at P=8 to 0.604 ms at P=256, so the communication term does not shrink with P. Tables 3-4 show ZeCO's operator runtime exceeding the DP baseline by ~0.9 ms at P=8 and ~3.3 ms at P=128 (≈50% over a ~6.4 ms baseline). Since the standalone All-Scan is only ~0.14-0.5 ms in that range, the residual overhead is several times larger than the total communication time; therefore Toverlapped cannot be recovering the All-Scan cost, and ε cannot be <1% as claimed. The gap must come from the second pass over Q, K, V and the extra HBM state store/load (Algorithm 1 lines 17-32). Thus Eq. 13-14 are not established, and the 'zero communication overhead' claim is stronger than the evidence supports.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ZeCO, a sequence parallelism (SP) method for linear attention models based on a new collective primitive called All-Scan. All-Scan passes the accumulated state tensor from device to device in a pipelined, blockwise fashion, aiming to achieve the minimal communication volume (one state tensor per device) and to overlap communication with local diagonal attention computation. The authors provide a theoretical optimality analysis (Section 3.3) claiming near-zero overhead and near-ideal linear scaling, and report experiments on up to 256 H100 GPUs showing speedups over LASP1, LASP2, and All-Gather/Megatron-style SP at the communication, operator, and model levels.","tokens_in":15676,"tokens_out":7258,"duration_ms":75515,"significance":"If the central claims were fully supported, ZeCO would be a valuable contribution to long-context training of linear attention models: the communication-volume lower bound is correct for the forward pass, the pipelined scan idea is a natural and likely effective way to reduce the latency of state passing, and the multi-level empirical comparison (communication, operator, model) is a strength. The paper also makes a useful observation that existing SP methods have communication volume or serial dependencies that scale with the number of devices, while ZeCO's communication volume is constant per device. However, the paper's own measurements reveal that the residual overhead over a data-parallel baseline is several times larger than the entire All-Scan communication time, so the 'zero communication overhead' and 'negligible extra cost' claims are not established as stated.","major_comments":[{"comment":"The predicted total time T_ZeCO(P L) ≈ T1_ideal-SP(L) − T_overlapped + τ(d_k d_v) with negligible ε is contradicted by the paper's own measurements. For the 16k per-device setting, Table 3 shows ZeCO operator runtime exceeds the GLA/DP baseline by 0.93 ms at P=8 and 3.33 ms at P=128, while Table 2 shows the standalone All-Scan latency is only 0.1375 ms at P=8 and 0.50084 ms at P=128. The residual overhead is several times larger than the total communication time, so the assumptions that All-Scan is fully overlapped and that the extra compute/I/O ε is negligible cannot both hold. The authors should provide a direct overlap measurement (e.g., a timeline of communication and compute streams) and a breakdown of the 3.3 ms overhead into the second pass over Q, K, V (Algorithm 1 lines 17–32) and the extra HBM state store/load traffic.","section":"Section 3.3, Eqs. (13)–(14); Tables 2–4"},{"comment":"The boundary term (P−1)τ(d_k d_v)/K does not vanish for any finite K, and K is bounded by the head dimension d_k (typically 64–128 in the experiments). The statement 'when K becomes sufficiently large, the boundary overhead approaches zero' is misleading because K is not a free parameter in practice; it cannot exceed d_k. Consequently, Eq. (12) itself predicts that T_All_Scan grows with P, which is consistent with Table 2 but incompatible with the interpretation of Eq. (14) as 'zero communication overhead.' The analysis should explicitly discuss the achievable K range and the resulting residual boundary cost at realistic head dimensions.","section":"Section 3.3, Eq. (12)"},{"comment":"The optimality argument is an assertion, not a proof, of full overlap and negligible ε. No evidence is given that the local diagonal-attention computation is long enough to hide the full pipelined transfer, nor that the extra HBM store/load operations in the two-pass schedule (Algorithm 1 lines 6–12 and 17–32) incur less than 1% overhead as claimed. The experimental data in Tables 3–4 contradict the ≤1% claim, since the measured overhead over the DP baseline is 15–50% of the baseline runtime (e.g., 3.33 ms / 6.55 ms at P=128 in Table 3). The theoretical section should be revised to state explicit sufficient conditions for the overhead to vanish and should be validated against the measurements, or the claims should be weakened to reflect the observed residual cost.","section":"Section 3.3, 'Optimality Analysis'"},{"comment":"The pseudo-code appears to contain a bug. In the intermediate-device branch (lines 15–17), the algorithm computes S^k_send = S^k_local + (γ*1)⊙S^k_recv but then sends S_local to recv_rank on line 17 instead of S^k_send. If implemented literally, the accumulated state from earlier devices would not propagate beyond the first hop, producing incorrect outputs. Please correct the pseudo-code and confirm that the implementation matches the intended pipeline. This is a load-bearing point because the correctness of All-Scan is the foundation of the whole method.","section":"Algorithm 2 (All-Scan)"},{"comment":"The phrase 'Zero Communication Overhead' is used in the paper to mean minimal communication volume plus overlapped communication, but the title and abstract claim a stronger property. The paper's own Table 2 shows All-Scan latency growing from 0.1375 ms at P=8 to 0.60405 ms at P=256, which is not 'zero' communication time, and Tables 3–4 show operator-level overhead growing with P. The claims should be reworded to 'minimal communication volume' and 'near-full overlap,' with the residual overhead explicitly quantified, to avoid overstating the result.","section":"Title and Section 1 (Abstract)"}],"minor_comments":[{"comment":"There is a typo in the abstract: 'optimaity' should be 'optimality.' Also, '16k sequence' should be '16k-token sequence' for clarity.","section":"Abstract and Section 1"},{"comment":"'Sevaral works' should be 'Several works.'","section":"Section 2.1"},{"comment":"The text uses 'Toverlaped_compare' in one place where 'Toverlaped_comp' is meant, and 'formularized' should be 'formulated.'","section":"Section 3.3"},{"comment":"The algorithm computes \\bar K[n] = K[n] / Λ[n], whereas Section 2.1 defines \\bar K[n] = K[n] ⊙ Γ[n]. The relationship between Γ and Λ (they are not simple inverses) should be clarified to avoid confusion.","section":"Algorithm 1, line 20"},{"comment":"Line 17 reads 'Load ,∈ R^{dk×dv}, from HBM to SRAM' with a missing variable name, and line 30 uses 'in parallel do' for what should be a sequential loop over n. These should be fixed.","section":"Algorithm 3"},{"comment":"'memory-out occurred' should be 'an out-of-memory error occurred.'","section":"Section 4.1"},{"comment":"The left two panels are described as showing theoretical values of calculation speed and communication volume, but the axes are not labeled, making it difficult to verify the claims from the figure.","section":"Figure 3"},{"comment":"The experimental section reports average times over 50 runs but gives no standard deviations or error bars. Given the claim of '<1% overhead,' a sensitivity analysis over repeated runs and different model configurations would strengthen the paper.","section":"Section 4.2 and Appendix A.3"}],"recommendation":"major_revision","confidential_remarks":"The core algorithmic idea — a pipelined All-Scan for linear attention sequence parallelism with constant per-device communication volume — is plausible and the reported speedups over LASP1/LASP2 are likely real. However, the paper's central theoretical claim of 'zero communication overhead' and 'negligible extra cost' is contradicted by its own tables: the operator-level overhead over the DP baseline is several times larger than the entire All-Scan communication time. This is not a local presentation issue; it affects how the contribution should be framed. I would be willing to consider a revised version that tempers the claims, provides a direct overlap measurement and a decomposition of the residual overhead, and corrects the All-Scan pseudo-code bug. I also note that no code is released, which is a concern for a systems paper that introduces a new collective primitive."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this is a real systems contribution wrapped in an overclaimed theoretical narrative. The All-Scan primitive—pipelined block-wise state pass with the scan overlapped against diagonal attention—is a genuine improvement over LASP1's serial P2P and LASP2's All-Gather, and the reported speedups are consistent across Tables 3-6. If the numbers hold up, this is a useful tool for long-sequence linear attention training. The paper deserves referee time.\n\nWhat's actually new: All-Scan partitions the state tensor along the head dimension and pipelines K blocks across devices, so the boundary term in latency shrinks as K grows. Overlapping that pipeline with local diagonal attention is a natural but non-trivial scheduling trick. The authors also correctly identify that the minimum communication volume for SP is one state tensor per device, not P copies. The empirical comparisons against LASP1/LASP2 are fair and the trends are internally consistent.\n\nThe soft spots are in the \"zero communication overhead\" and \"optimality\" claims. Table 2 shows All-Scan latency growing from 0.14 ms at P=8 to 0.60 ms at P=256, so the communication term does not vanish. Tables 3-4 show ZeCO's operator runtime exceeding the DP baseline by ~0.9 ms at P=8 and ~3.3 ms at P=128—several times larger than the entire All-Scan cost. That gap has to come from the two-pass schedule's extra HBM traffic, which the paper dismisses as <1% without measuring it. Equations 13-14 therefore rest on an unverified overlap assumption, not on proof. The bound K is also limited by dk, so the boundary term only goes to zero in the limit, not in practice. The \"theoretically optimal\" language should be scoped to communication volume, not wall-clock overhead.\n\nNo code or data is provided, so there is no independent check of the speedups or of numerical correctness. That is a meaningful gap for a systems paper, but not a fatal one.\n\nWho this is for: anyone implementing or benchmarking sequence parallelism for linear attention. It is a solid, citeable contribution if the claims are slimmed down. I would send it to a serious referee with a request to verify the overlap measurement and release the implementation.\n\nRecommendation: accept for peer review, with major revision likely on the framing.","headline":"A real systems contribution with an overclaimed 'zero overhead' framing: All-Scan is a genuine improvement over LASP1/LASP2, but the optimality proof rests on unverified overlap assumptions and the paper's own tables contradict the headline claim.","tokens_in":16246,"tokens_out":1809,"would_cite":true,"duration_ms":19966,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new sequence-parallel method for linear attention communicates only the theoretically minimal state tensor and hides that transfer behind local computation, yielding near-linear scaling to 256 GPUs.","keywords":["sequence parallelism","linear attention","All-Scan collective communication","communication-computation overlap","long-context training","gated linear attention","near-linear scaling"],"falsifier":"Run ZeCO at fixed per-device sequence length while shrinking the state size $d_k\\times d_v$ relative to local FLOPs, or across a slower interconnect, and record the runtime gap versus an ideal single-stream data-parallel operator; if the gap grows with $P$, the boundary term is not negligible. A direct instrument-level check is to timestamp All-Scan's transfer kernel and the diagonal-attention kernel on the same device and compute the fraction of All-Scan time that does not overlap; Equation (11) predicts this fraction should be near zero.","tokens_in":15179,"feed_emoji":"🚀","tokens_out":9373,"duration_ms":99382,"temperature":0.7,"pith_summary":"Linear attention layers keep a fixed-size matrix of key–value information instead of a growing cache, so the state is small enough to pass between devices. Sequence parallelism is what lets a long sequence be split across many GPUs, but today's pass-the-state schemes either force devices to run one after another or make every device collect everyone else's state, so communication grows with the number of devices. ZeCO claims a third option: each device sends exactly one state matrix to its neighbor, which is the least any correct scheme could send, and the transfer is pipelined and overlapped with the device's own attention computation. If that is right, sequence parallelism for linear attention would no longer throttle training: a 1M-token context on 64 devices should cost about the same as a 16k-token context on one device, with throughput scaling almost linearly.","feed_headline":"All-Scan cuts sequence-parallel traffic to one tensor per device","feed_subtitle":"Hiding state transfer behind local compute lets 8M-token training scale near-linearly to 256 GPUs.","key_machinery":"The central object is All-Scan, a pipelined collective communication primitive. All-Scan partitions the state tensor $S$ along the $d_k$ dimension into $K$ contiguous blocks; each device receives a block from its predecessor, applies the chunk-local state update $S^{(k)}_{pL} = (\\tilde\\gamma^{(k)\\top}_N \\mathbf{1})\\odot S^{(k)}_{(p-1)L} + S^{(k)}_{[N]}$, and forwards it to the next device. Its defining property is that it moves exactly one state tensor per device regardless of $P$, while the block-wise pipeline lets downstream devices begin their scan incrementally. ZeCO also relies on a linear-decomposition identity for the gated linear attention state: the global state equals the cumulative decay applied to the incoming global state plus the locally computed residual state, which is what lets local computation proceed without waiting for the full remote state.","core_discovery":"The paper's central claim is that ZeCO achieves the optimal sequence-parallel strategy for linear attention. Because the gated linear attention recurrence updates a fixed state $S\\in\\mathbb{R}^{d_k\\times d_v}$, the only information that must cross a device boundary is this state; the paper argues that no correct SP algorithm can communicate less. All-Scan realizes that lower bound, $V_{\\mathrm{ZeCO}}=d_k\\times d_v$ per device, by splitting the state into $K$ blocks along $d_k$, updating each block with the local cumulative decay, and forwarding it to the next device in a pipeline. The pipeline lets neighboring devices start consuming early blocks before the full state is ready, and ZeCO runs All-Scan on a separate stream while the device computes the diagonal (intra-chunk) attention terms. The resulting time is $T^P_{\\mathrm{ZeCO}}(PL)\\approx T^1_{\\mathrm{ideal\\text{-}SP}}(L)-T_{\\mathrm{overlapped}}+\\tau(d_k\\,d_v)$, with the boundary term $(P-1)\\tau(d_k\\,d_v/K)$ argued to shrink as the block count $K$ grows. In the paper's measurements, the ZeCO operator stays within about 3 ms of the ideal data-parallel operator on 128 GPUs, and at 256 GPUs with 8M-token sequences it reports a 60% throughput gain over the strongest baseline.","pith_inferences":["The paper leaves implicit that All-Scan's advantage should be largest when local per-device compute is long relative to state-transfer time; a future sweep over per-device sequence length, state size, and interconnect speed would make that trade-off concrete, since the current experiments fix $d_k\\times d_v$ and vary only $P$.","The same pipelined receive-update-send pattern should carry over to other linear recurrent layers with fixed-size states, but for matrix-valued (non-diagonal) transitions the per-tensor volume would grow, so the regime where ZeCO beats all-gather methods would narrow.","A tree-structured or hierarchical version of All-Scan could trade the remaining boundary term for extra depth; the paper lists tree-like implementations as future work but does not analyze them."],"forward_implications":["Per-GPU runtime becomes nearly flat in $P$: in the reported operator benchmarks, ZeCO's forward-plus-backward time grows by only about 2.5 ms from 8 to 128 GPUs at 16k tokens per device, while the serial and all-gather baselines grow by tens of milliseconds.","At 256 GPUs with 8M-token sequences, ZeCO reports a 60% model-level throughput gain over the strongest SP baseline, and the All-Scan collective is about 4× faster than an all-gather-based scheme in the paper's 256-GPU communication benchmark.","Because the backward pass can reuse the global state already obtained in the forward pass, the zero-extra-communication argument applies to full training iterations, not only to a forward pass.","ZeCO's optimality claim is specific to linear attention and kindred fixed-state recurrent layers; for standard softmax attention there is no fixed-size state matrix to pass, so the same lower-bound argument does not apply."],"supporting_citations":[{"why":"Defines linear attention as a fixed-size state recurrence, the algorithmic basis that ZeCO parallelizes.","marker":"Katharopoulos et al. 2020"},{"why":"Gives the gated linear attention (GLA) chunkwise recurrence that ZeCO reformulates and whose state $S$ it communicates.","marker":"S. Yang, Bailin Wang, Shen, et al. 2024"},{"why":"LASP-1, the serial state-passing baseline that ZeCO must beat; its $P\\times$ serial overhead motivates the pipeline.","marker":"Weigao Sun, Qin, et al. 2025"},{"why":"LASP-2, the all-gather-based baseline whose $(P-1)\\times$ communication volume is the comparison target for the lower-bound argument.","marker":"Weigao Sun, Lan, et al. 2025"},{"why":"Another LASP-2-style all-gather baseline used in the same communication-volume comparison.","marker":"A. Li et al. 2025"},{"why":"Flash Linear Attention implementation that ZeCO adapts for its operator-level experiments, grounding the empirical speedups in a standard reference implementation.","marker":"S. Yang and Y. Zhang 2024"},{"why":"Lingua training framework used for the model-level throughput measurements.","marker":"Videau et al. 2024"}],"fun_headline_variants":["ZeCO: sequence parallelism with zero communication overhead","All-Scan: one tensor per device, zero comm bottleneck","ZeCO: hide state transfer to erase SP communication","Near-linear scaling for 1M-token linear attention via ZeCO"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The near-zero-overhead proof assumes the pipelined All-Scan transfer is fully hidden behind a device's local diagonal-attention computation; if local work is too short or the communication kernel contends with compute for memory or network bandwidth, the residual boundary term grows and the zero-overhead claim weakens.","fun_headline_variants_meta":{"raw":{"variants":["ZeCO: sequence parallelism with zero communication overhead","All-Scan: one tensor per device, zero comm bottleneck","ZeCO: hide state transfer to erase SP communication","Near-linear scaling for 1M-token linear attention via ZeCO"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000778,"raw_usage":{"total_tokens":3525,"prompt_tokens":1117,"completion_tokens":2408,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":733,"completion_tokens_details":{"reasoning_tokens":2340}},"tokens_in":733,"tokens_out":2408,"duration_ms":21392,"temperature":1.0,"reasoning_tokens":2340,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:00:17.416217+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ZeCO at fixed per-device sequence length while shrinking the state size $d_k\\times d_v$ relative to local FLOPs, or across a slower interconnect, and record the runtime gap versus an ideal single-stream data-parallel operator; if the gap grows with $P$, the boundary term is not negligible. A direct instrument-level check is to timestamp All-Scan's transfer kernel and the diagonal-attention kernel on the same device and compute the fraction of All-Scan time that does not overlap; Equation (11) predicts this fraction should be near zero.","supporting_citations":[],"review_version":1}