Pith. sign in

REVIEW 5 major objections 8 minor 3 references

ZeCO: Zero Communication Overhead Sequence Parallelism for Linear Attention

T0 review · 5 major / 8 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A new sequence-parallel method for linear attention communicates only the theoretically minimal state tensor and hides that transfer behind local computation, yielding near-linear scaling to 256 GPUs.

desk verdict A real systems contribution with an overclaimed 'zero overhead' framing: All-Scan is a genuine improvement over LASP1/LASP2, but the optimality proof rests on unverified overlap assumptions and the paper's own tables contradict the headline claim. read the letter →

arxiv 2507.01004 v2 pith:3MRRII4V submitted 2025-07-01 cs.LG

Yuhong Chou , Zehao Liu , Ruijie Zhu , Xinyi Wan , Tianjian Li , Congying Chu , Qian Liu , Jibin Wu
show 1 more author
Zejun Ma
This is my paper · ORCID
classification cs.LG
keywords sequenceparallelismlinearattentionAll-Scancollectivecommunicationcommunication-computationoverlaplong-contexttraininggatednear-linearscaling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Linear attention layers keep a fixed-size matrix of key–value information instead of a growing cache, so the state is small enough to pass between devices. Sequence parallelism is what lets a long sequence be split across many GPUs, but today's pass-the-state schemes either force devices to run one after another or make every device collect everyone else's state, so communication grows with the number of devices. ZeCO claims a third option: each device sends exactly one state matrix to its neighbor, which is the least any correct scheme could send, and the transfer is pipelined and overlapped with the device's own attention computation. If that is right, sequence parallelism for linear attention would no longer throttle training: a 1M-token context on 64 devices should cost about the same as a 16k-token context on one device, with throughput scaling almost linearly.

What carries the argument

The central object is All-Scan, a pipelined collective communication primitive. All-Scan partitions the state tensor $S$ along the $d_k$ dimension into $K$ contiguous blocks; each device receives a block from its predecessor, applies the chunk-local state update $S^{(k)}_{pL} = (\tilde\gamma^{(k)\top}_N \mathbf{1})\odot S^{(k)}_{(p-1)L} + S^{(k)}_{[N]}$, and forwards it to the next device. Its defining property is that it moves exactly one state tensor per device regardless of $P$, while the block-wise pipeline lets downstream devices begin their scan incrementally. ZeCO also relies on a linear-decomposition identity for the gated linear attention state: the global state equals the cumulative decay applied to the incoming global state plus the locally computed residual state, which is what lets local computation proceed without waiting for the full remote state.

What would settle it

Run ZeCO at fixed per-device sequence length while shrinking the state size $d_k\times d_v$ relative to local FLOPs, or across a slower interconnect, and record the runtime gap versus an ideal single-stream data-parallel operator; if the gap grows with $P$, the boundary term is not negligible. A direct instrument-level check is to timestamp All-Scan's transfer kernel and the diagonal-attention kernel on the same device and compute the fraction of All-Scan time that does not overlap; Equation (11) predicts this fraction should be near zero.

Watch

Extended reading notes

Core claim

The paper's central claim is that ZeCO achieves the optimal sequence-parallel strategy for linear attention. Because the gated linear attention recurrence updates a fixed state $S\in\mathbb{R}^{d_k\times d_v}$, the only information that must cross a device boundary is this state; the paper argues that no correct SP algorithm can communicate less. All-Scan realizes that lower bound, $V_{\mathrm{ZeCO}}=d_k\times d_v$ per device, by splitting the state into $K$ blocks along $d_k$, updating each block with the local cumulative decay, and forwarding it to the next device in a pipeline. The pipeline lets neighboring devices start consuming early blocks before the full state is ready, and ZeCO runs All-Scan on a separate stream while the device computes the diagonal (intra-chunk) attention terms. The resulting time is $T^P_{\mathrm{ZeCO}}(PL)\approx T^1_{\mathrm{ideal\text{-}SP}}(L)-T_{\mathrm{overlapped}}+\tau(d_k\,d_v)$, with the boundary term $(P-1)\tau(d_k\,d_v/K)$ argued to shrink as the block count $K$ grows. In the paper's measurements, the ZeCO operator stays within about 3 ms of the ideal data-parallel operator on 128 GPUs, and at 256 GPUs with 8M-token sequences it reports a 60% throughput gain over the strongest baseline.

Load-bearing premise

The near-zero-overhead proof assumes the pipelined All-Scan transfer is fully hidden behind a device's local diagonal-attention computation; if local work is too short or the communication kernel contends with compute for memory or network bandwidth, the residual boundary term grows and the zero-overhead claim weakens.

Editorial extensions

If this is right

  • Per-GPU runtime becomes nearly flat in $P$: in the reported operator benchmarks, ZeCO's forward-plus-backward time grows by only about 2.5 ms from 8 to 128 GPUs at 16k tokens per device, while the serial and all-gather baselines grow by tens of milliseconds.
  • At 256 GPUs with 8M-token sequences, ZeCO reports a 60% model-level throughput gain over the strongest SP baseline, and the All-Scan collective is about 4× faster than an all-gather-based scheme in the paper's 256-GPU communication benchmark.
  • Because the backward pass can reuse the global state already obtained in the forward pass, the zero-extra-communication argument applies to full training iterations, not only to a forward pass.
  • ZeCO's optimality claim is specific to linear attention and kindred fixed-state recurrent layers; for standard softmax attention there is no fixed-size state matrix to pass, so the same lower-bound argument does not apply.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that All-Scan's advantage should be largest when local per-device compute is long relative to state-transfer time; a future sweep over per-device sequence length, state size, and interconnect speed would make that trade-off concrete, since the current experiments fix $d_k\times d_v$ and vary only $P$.
  • The same pipelined receive-update-send pattern should carry over to other linear recurrent layers with fixed-size states, but for matrix-valued (non-diagonal) transitions the per-tensor volume would grow, so the regime where ZeCO beats all-gather methods would narrow.
  • A tree-structured or hierarchical version of All-Scan could trade the remaining boundary term for extra depth; the paper lists tree-like implementations as future work but does not analyze them.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 8 minor

Summary. The paper proposes ZeCO, a sequence parallelism (SP) method for linear attention models based on a new collective primitive called All-Scan. All-Scan passes the accumulated state tensor from device to device in a pipelined, blockwise fashion, aiming to achieve the minimal communication volume (one state tensor per device) and to overlap communication with local diagonal attention computation. The authors provide a theoretical optimality analysis (Section 3.3) claiming near-zero overhead and near-ideal linear scaling, and report experiments on up to 256 H100 GPUs showing speedups over LASP1, LASP2, and All-Gather/Megatron-style SP at the communication, operator, and model levels.

Significance. If the central claims were fully supported, ZeCO would be a valuable contribution to long-context training of linear attention models: the communication-volume lower bound is correct for the forward pass, the pipelined scan idea is a natural and likely effective way to reduce the latency of state passing, and the multi-level empirical comparison (communication, operator, model) is a strength. The paper also makes a useful observation that existing SP methods have communication volume or serial dependencies that scale with the number of devices, while ZeCO's communication volume is constant per device. However, the paper's own measurements reveal that the residual overhead over a data-parallel baseline is several times larger than the entire All-Scan communication time, so the 'zero communication overhead' and 'negligible extra cost' claims are not established as stated.

major comments (5)
  1. [Section 3.3, Eqs. (13)–(14); Tables 2–4] The predicted total time T_ZeCO(P L) ≈ T1_ideal-SP(L) − T_overlapped + τ(d_k d_v) with negligible ε is contradicted by the paper's own measurements. For the 16k per-device setting, Table 3 shows ZeCO operator runtime exceeds the GLA/DP baseline by 0.93 ms at P=8 and 3.33 ms at P=128, while Table 2 shows the standalone All-Scan latency is only 0.1375 ms at P=8 and 0.50084 ms at P=128. The residual overhead is several times larger than the total communication time, so the assumptions that All-Scan is fully overlapped and that the extra compute/I/O ε is negligible cannot both hold. The authors should provide a direct overlap measurement (e.g., a timeline of communication and compute streams) and a breakdown of the 3.3 ms overhead into the second pass over Q, K, V (Algorithm 1 lines 17–32) and the extra HBM state store/load traffic.
  2. [Section 3.3, Eq. (12)] The boundary term (P−1)τ(d_k d_v)/K does not vanish for any finite K, and K is bounded by the head dimension d_k (typically 64–128 in the experiments). The statement 'when K becomes sufficiently large, the boundary overhead approaches zero' is misleading because K is not a free parameter in practice; it cannot exceed d_k. Consequently, Eq. (12) itself predicts that T_All_Scan grows with P, which is consistent with Table 2 but incompatible with the interpretation of Eq. (14) as 'zero communication overhead.' The analysis should explicitly discuss the achievable K range and the resulting residual boundary cost at realistic head dimensions.
  3. [Section 3.3, 'Optimality Analysis'] The optimality argument is an assertion, not a proof, of full overlap and negligible ε. No evidence is given that the local diagonal-attention computation is long enough to hide the full pipelined transfer, nor that the extra HBM store/load operations in the two-pass schedule (Algorithm 1 lines 6–12 and 17–32) incur less than 1% overhead as claimed. The experimental data in Tables 3–4 contradict the ≤1% claim, since the measured overhead over the DP baseline is 15–50% of the baseline runtime (e.g., 3.33 ms / 6.55 ms at P=128 in Table 3). The theoretical section should be revised to state explicit sufficient conditions for the overhead to vanish and should be validated against the measurements, or the claims should be weakened to reflect the observed residual cost.
  4. [Algorithm 2 (All-Scan)] The pseudo-code appears to contain a bug. In the intermediate-device branch (lines 15–17), the algorithm computes S^k_send = S^k_local + (γ*1)⊙S^k_recv but then sends S_local to recv_rank on line 17 instead of S^k_send. If implemented literally, the accumulated state from earlier devices would not propagate beyond the first hop, producing incorrect outputs. Please correct the pseudo-code and confirm that the implementation matches the intended pipeline. This is a load-bearing point because the correctness of All-Scan is the foundation of the whole method.
  5. [Title and Section 1 (Abstract)] The phrase 'Zero Communication Overhead' is used in the paper to mean minimal communication volume plus overlapped communication, but the title and abstract claim a stronger property. The paper's own Table 2 shows All-Scan latency growing from 0.1375 ms at P=8 to 0.60405 ms at P=256, which is not 'zero' communication time, and Tables 3–4 show operator-level overhead growing with P. The claims should be reworded to 'minimal communication volume' and 'near-full overlap,' with the residual overhead explicitly quantified, to avoid overstating the result.
minor comments (8)
  1. [Abstract and Section 1] There is a typo in the abstract: 'optimaity' should be 'optimality.' Also, '16k sequence' should be '16k-token sequence' for clarity.
  2. [Section 2.1] 'Sevaral works' should be 'Several works.'
  3. [Section 3.3] The text uses 'Toverlaped_compare' in one place where 'Toverlaped_comp' is meant, and 'formularized' should be 'formulated.'
  4. [Algorithm 1, line 20] The algorithm computes \bar K[n] = K[n] / Λ[n], whereas Section 2.1 defines \bar K[n] = K[n] ⊙ Γ[n]. The relationship between Γ and Λ (they are not simple inverses) should be clarified to avoid confusion.
  5. [Algorithm 3] Line 17 reads 'Load ,∈ R^{dk×dv}, from HBM to SRAM' with a missing variable name, and line 30 uses 'in parallel do' for what should be a sequential loop over n. These should be fixed.
  6. [Section 4.1] 'memory-out occurred' should be 'an out-of-memory error occurred.'
  7. [Figure 3] The left two panels are described as showing theoretical values of calculation speed and communication volume, but the axes are not labeled, making it difficult to verify the claims from the figure.
  8. [Section 4.2 and Appendix A.3] The experimental section reports average times over 50 runs but gives no standard deviations or error bars. Given the claim of '<1% overhead,' a sensitivity analysis over repeated runs and different model configurations would strengthen the paper.

Circularity Check

1 steps flagged · score 4.0 of 10

The communication-volume optimality claim is definitional: S is declared the minimal message, then |S| is declared the lower bound. The latency accounting and empirical benchmarks are independent, with no fitted-parameter or load-bearing self-citation circularity.

  1. self definitional [Section 3.3, 'Zero Communication Overhead' and Equations (8)-(9)]
    "We identify two necessary and sufficient conditions for optimality and prove that ZeCO satisfies both: 1. Zero Communication Overhead: Each device transmits and receives only the minimal essential size of information (data). ... Let S∈ Rdk×dv denote the accumulated state tensor. According to linear attention output Equation(3), this state represents the minimal information that must be communicated between chunks. ... V(p) ZeCO= ∣S∣= dk× dv. (9) This represents the theoretical lower bound."

    The lower-bound proof is a stipulation: the paper defines the 'minimal essential' message to be the state tensor S and then concludes that |S| is the theoretical lower bound. Equation (3) shows only that S is sufficient to compute chunk outputs; it does not prove that no alternative, smaller, or differently structured message (e.g., factorized state or partial Q/K/V products) can cross the device boundary. Consequently V_ZeCO = |S| = 'theoretical lower bound' holds by construction rather than by an adversarial lower-bound argument, so this part of the optimality theorem reduces to its own premise.

full rationale

ZeCO's derivation is mostly self-contained: Eq. (12) is a latency model, Eqs. (13)-(14) are an accounting identity built on an overlap assumption that is not measured (a correctness risk, not a circularity), and no parameter is fitted and then renamed a prediction. The only circular step is the communication-volume optimality proof, where the 'minimal information' is asserted to be S and the lower bound is then read off as |S|. That is a definitional move, not a derivation over alternative message types. The paper's citations to GLA, FLA, LASP, etc. are external and not load-bearing self-citations, so no self-citation circularity is present. Given that the central theoretical optimality claim partially reduces by definition, but the algorithm's measured speedups against DP and LASP baselines provide independent evidence, the appropriate score is moderate rather than zero.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central claim rests on system-level assumptions about what must be communicated, whether overlap is free, and whether K is unbounded, plus standard GLA recurrence algebra. It does not rely on fitted constants. The only hand-tuned knob affecting the theory is K, whose value is not reported in the paper.

free parameters (1)
  • K (number of pipeline blocks) = not stated; bounded by dk / block size
    The theoretical boundary cost (P-1)tau(dk dv)/K in Eq. 12 is driven toward zero by assuming K can be large, but no selected values or sensitivity study are reported, so the optimality claim depends on this hand-chosen knob.
assumptions (5)
  • domain assumption The accumulated state S in R^{dk x dv} is the minimal information that must cross device boundaries for linear attention sequence parallelism.
    Asserted in Section 3.3 as 'this state represents the minimal information that must be communicated between chunks'. If additional tensors such as normalization or decay metadata must be exchanged, the lower bound is understated.
  • domain assumption All-Scan communication can be fully overlapped with local diagonal attention computation on another stream.
    Algorithm 1 places All-Scan and local causal-mask attention in parallel streams; Eq. 11 allows a residual, but the 'zero overhead' conclusion assumes the residual is negligible.
  • domain assumption Splitting the state into K blocks introduces only a (P-1)tau/K boundary term, and K can be made sufficiently large.
    Eq. 12 contains this term. In practice K is limited by the dk dimension, so for large P the boundary term does not vanish as the proof requires.
  • standard math The local GLA recurrence can be computed from a zero initial state and later corrected by a cumulative decay factor.
    Appendix A.1 proves the decomposition algebraically from the GLA recurrence; this is consistent standard math.
  • domain assumption In the backward pass, recomputing S[n] needs no All-Scan because the global initial state is already held from the forward pass.
    Appendix A.4 states this without a full derivation or memory accounting for the stored global state.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ZeCO: Zero Communication Overhead Sequence Parallelism for Linear Attention." pith.science (2026). https://pith.science/paper/3MRRII4V

@misc{pith2026250701004,
  author       = {Pith},
  title        = {Pith review of: ZeCO: Zero Communication Overhead Sequence Parallelism for Linear Attention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3MRRII4V}},
  note         = {Machine review of arXiv:2507.01004}
}
read the original abstract

Linear attention mechanisms deliver significant advantages for Large Language Models (LLMs) by providing linear computational complexity, enabling efficient processing of ultra-long sequences (e.g., 1M context). However, existing Sequence Parallelism (SP) methods, essential for distributing these workloads across devices, become the primary bottleneck due to substantial communication overhead. In this paper, we introduce ZeCO (Zero Communication Overhead) sequence parallelism for linear attention models, a new SP method designed to overcome these limitations and achieve end-to-end near-linear scalability for long sequence training. For example, training a model with a 1M sequence length across 64 devices using ZeCO takes roughly the same time as training with an 16k sequence on a single device. At the heart of ZeCO lies All-Scan, a new collective communication primitive. All-Scan provides each SP rank with precisely the initial operator state it requires while maintaining a minimal communication footprint, effectively eliminating communication overhead. Theoretically, we prove the optimaity of ZeCO, showing that it introduces only negligible time and space overhead. Empirically, we compare the communication costs of different sequence parallelism strategies and demonstrate that All-Scan achieves the fastest communication in SP scenarios. Specifically, on 256 GPUs with an 8M sequence length, ZeCO achieves a 60\% speedup compared to the current state-of-the-art (SOTA) SP method. We believe ZeCO establishes a clear path toward efficiently training next-generation LLMs on previously intractable sequence lengths.

Figures

Figures reproduced from arXiv: 2507.01004 by the authors.

Figure 1
Figure 1. (Left) ZeCO demonstrates near-linear scaling efficiency when scaling sequence length proportionally with device count, approaching the theoretical upper bound. (Right) ZeCO substantially outperforms SOTA SP methods across three performance metrics: communication time, runtime extra cost, and per-GPU throughput. Metrics obtained using 256 GPUs (8M sequences) for communication / throughput and 128 GPUs (4M sequences) … view at source ↗
Figure 2
Figure 2. Illustration of ZeCO. ZeCO highlights its strengths in three dimensions: (1) Parallel Scalability: achieving efficiency comparable to DP (sub-figure: Algorithm Comparison); (2) Operator-Level Computation: enabling overlap of communication and local computation for maximal resource utilization (sub-figure: ZeCO Computation); and (3) Communication Pattern: utilizing a customized pipelined All-Scan Communication patter… view at source ↗
Figure 3
Figure 3. ZeCO has the lowest communication time while satisfying the lowest communication volume. The left two figures show the theoretical values of the algorithm calculation speed and communication volume, and the right figure shows the actual communication time. where ϵ represents a negligible computation and I/O cost. In contrast, existing methods like LASP have a strictly serial dependency across devices, resulting in (… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Scalability evaluation of LASP on SP operator runtime(top half) and Scalability evaluation of LASP on Throughput(bottom half). In the comparison test of 16k and 32k sequence length per GPU, ZeCO algorithm shows the same stable time as the DP algorithm. In both 16k and …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

3 extracted references · 2 canonical work pages

  1. [1]

    Phi-4 Technical Report

    Abdin, Marah I et al. (2024). “Phi-4 Technical Report”. In:CoRR abs/2412.08905. d oi: 10.48550/ARXIV. 2412.08905. arXiv:2412.08905. u r l: https://doi.org/10.48550/arXiv.2412.08905. Brandon, William et al. (2023). “Striped attention: Faster ring attention for causal transformers”. In:arXiv preprint arXiv:2311.09431. Brown, Tom B et al. (2020). “Language m...

    arXiv doi:10.48550/arxiv.2412.08905 2024

  2. [2]

    The experimental setup with 5 rounds of warm-up and reported the average of 50 rounds of experiment, see in Table 3, Table

    16 ZeCO: Zero Communication Overhead Sequence Parallelism for Linear Attention In experiment Section 4.2, In the experiment of algorithm run time, we test the GLA-attention algorithm equipped with different SP methods, record the time of 1 iteration of FWD and BWD.H is 16, the tensor size of each chunk of segmentation is 16384, the hidden dimensiond is 20...

    work page

  3. [4]

    The experimental setup with 5 rounds of warm-up reported the average of 100 steps of the experiment, see in Table 5, Table

    In the experiment of Model throughput, we test the GLA-1B Model equipped with different SP methods, and record the throughput in the training stage.H is 32, the tensor size of each chunk of segmentation is 16384, the number of model layers is 20, the hidden dimension d is 2048, and the sequence length per deviceL is 16384 and 32768. The experimental setup...

    work page

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.