REVIEW 4 major objections 6 minor 19 references
TokenRing: An Efficient Parallelism Framework for Infinite-Context LLMs via Bidirectional Communication
T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Bidirectional ring scheduling can cut the dominant communication cost of sequence-parallel attention by about half.
desk verdict TokenRing has a legitimate scheduling idea, but the central latency claim rests on a single 4-GPU PCIe profile, not the full-mesh interconnect the design targets. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is a bidirectional ring dataflow: while Q blocks move in one direction, block_out and block_lse move in the other, and the two transfers are overlapped with the flash-attention kernel that computes the next block. On a fully connected mesh this uses the send and receive halves of the same links instead of leaving one direction idle. The update rule $\text{out} \leftarrow \text{out} - \sigma(\text{block\_lse} - \text{lse}) \cdot (\text{out} - \text{block\_out})$ and $\text{lse} \leftarrow \text{lse} - \ln(\sigma(\text{lse} - \text{block\_lse}))$ stitches the partial outputs into exact attention without recomputing.
What would settle it
On an eight-GPU full-mesh NVLink or OAM node, run TokenRing and Ring Attention on the same long sequence, such as 24,000 tokens, with the same model configuration and profile per-step attention time. If per-step time does not drop below Ring Attention's or the profiler shows Q and block_out transmissions not overlapping with computation, the central communication-reduction claim is refuted.
Extended reading notes
Core claim
The central claim is that a ring of GPUs can compute blockwise attention with much less communication stretch than Ring Attention by moving queries and partial results in opposite directions at the same time. Each rank keeps one K/V block locally, passes Q blocks around the ring, and after each flash-attention kernel sends block_out and block_lse backward to the rank that owns the corresponding output rows; the output rows are then refined with the sigmoid/log-sum-exp update rule. The paper reports that this cuts per-step attention time from about 7.6 ms to about 3.5 ms at sequence length 24,000 on four A10 GPUs, with the overlap forming a larger fraction of the schedule as the ring grows.
Load-bearing premise
The projected benefit assumes that bidirectional overlap behaves on full-mesh NVLink or OAM hardware as it does on the PCIe (PIX/PXB) A10 testbed, and the paper's Section 4.1 explicitly states that testing on hardware optimally configured for TokenRing was not feasible.
Editorial extensions
If this is right
- At sequence length 24,000 on four GPUs, per-step attention time falls from roughly 7.6 ms for Ring Attention to about 3.5 ms for TokenRing, and the fraction of steps that gain from bidirectional overlap grows with the number of GPUs.
- Sequence parallelism is no longer capped by the number of attention heads, so models using grouped or multi-query attention can still use many sequence-parallel ranks.
- TokenRing composes with Flash Attention and with the zigzag causal partition, balancing causal attention workloads and cutting redundant transfers during LLM inference.
- A hybrid scheme that uses TokenRing inside a node and ordinary Ring Attention across nodes extends the method beyond a single full-mesh node.
- Operation on full-mesh NVLink or OAM/HCCS interconnects reduces dependence on NVSwitch, lowering the hardware cost of long-context serving.
Reading between the lines
- The same bidirectional schedule could be adapted to decode-phase KV-cache chunking, where causal masks make the return path asymmetric; the paper only evaluates prefill-style forward passes.
- On interconnects whose effective send and receive bandwidth is shared rather than full duplex, the overlap gain would shrink, so a controlled half-duplex test would delimit when TokenRing wins.
- The multi-node hybrid is described but not benchmarked, so whether intra-node TokenRing plus inter-node Ring Attention preserves the latency gain at scale remains an open quantitative question the paper does not settle.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. TokenRing is a sequence-parallel attention scheme. It partitions Q, K, V along the sequence/token dimension, keeps the KV block local at each GPU, and circulates query blocks around a ring while concurrently sending block_out and block_lse in the opposite direction to accumulate the final attention output using an online-softmax (log-sum-exp) merge. The claimed contribution is that this bidirectional P2P communication overlaps computation and communication and better utilizes full-mesh interconnects such as NVLink and OAM. The evaluation reports an Nsight profiling comparison on four A10 GPUs connected by PCIe PIX/PXB links at sequence length 24,000, showing about 3.5 ms per TokenRing step versus 7.6 ms for Ring Attention, and claims scalability to full-mesh topologies and hybrid multi-node systems.
Significance. The proposal is coherent and the online-softmax merge in Section 3.1 is mathematically correct. The paper is not circular: performance is measured rather than predicted from a model that already contains the answer, no free parameters are fitted, and the code is publicly available. If the bidirectional-overlap mechanism were demonstrated on a full-mesh interconnect, TokenRing would be a useful low-cost alternative to NVSwitch for long-context attention. As it stands, however, the central quantitative claim rests on a platform where the key mechanism is untested, and no end-to-end comparison against existing sequence-parallel systems is provided. The contribution is therefore plausible but not yet established.
major comments (4)
- [§4.1 and §4.2] The central latency claim (3.5 ms versus 7.6 ms in Figure 6) is obtained on four A10 GPUs on PIX/PXB PCIe links, while the claimed advantage relies on concurrent opposite-direction P2P traffic saturating independent full-mesh links. The paper itself states in §4.1 that testing on hardware optimally configured for TokenRing was not feasible; on PCIe with shared bridges and root complexes, simultaneous sends in opposite directions can contend, so the observed difference cannot be attributed to full-duplex bandwidth. Please provide measurements on NVLink or OAM full-mesh hardware, or at minimum a link-level bidirectional-bandwidth microbenchmark that supports the full-duplex assumption, and compare end-to-end latency and throughput with Ring Attention, Ulysses, USP, LoongTrain, and Striped Attention.
- [Algorithm 1 (§3.2)] The pseudocode is not self-contained: the variable 'step' in the line t ← (j − step + 1) mod N is never defined, and the conditions 'if i > 1' and 'if i != 1' are not explained in the text; the prose description in §3.3.1 suggests that only one step performs the Out update, which conflicts with the pseudocode's loop and final send. Please define all variables, fix the off-by-one conditions, and give a concrete communication schedule for N = 4 and N = 8 that matches Figure 3b. This is needed for reproducibility.
- [§3.1] The paper should state explicitly that for MHA the communication volume is not reduced: Q, K, V, and block_out have the same shape (N, H, D), so the only source of improvement is concurrent use of opposite directions on each link. Since §3.1 concedes that block_out and block_lse offset the savings from sending Q instead of KV, the claimed reduction in communication overhead rests entirely on the bidirectional-overlap mechanism that is unmeasured on the target topology. Please quantify per-step communication bytes for Q, KV, block_out, and block_lse in the tested configuration and discuss how link contention is avoided.
- [§4.2 and §5] The abstract and conclusion claim improved throughput, but the only quantitative result is a per-attention-step latency profile from Nsight Systems at a single sequence length; no end-to-end training or inference throughput, no scaling study beyond four GPUs, and no load-balancing measurement is reported. Please provide end-to-end measurements on the target topology or temper the throughput claims accordingly.
minor comments (6)
- [Abstract] The phrase 'rendering their effectiveness' in the abstract is grammatically incomplete and should be rewritten.
- [Figure 3 caption] The caption says 'Partition Query equally across all GPUs' while Algorithm 1 partitions Q, K, and V; the caption and the figure labels should be aligned with the text.
- [§3.3.1] There is a dangling cross-reference 'Figure??' that should be fixed or removed.
- [§3.3.2] The zigzag strategy is adopted for causal attention, but no pseudo-code or load-balancing data is given; a reference to the exact implementation used would help reproducibility.
- [Algorithm 1] The final line 'send block out, block lse to rank j − N + 1' is ambiguous because no modulo operation is shown and for j = 0 the target rank is negative; please clarify.
- [§5] The mention of 'two independent channels' in the NCCL backend is not explained; adding a sentence on what these channels are would make the discussion of observed latency easier to interpret.
Circularity Check
No circularity: TokenRing's performance claims are measured rather than derived from their own assumptions, and the only mathematical content is the standard online-softmax merge; the full-mesh extrapolation is an external-validity gap, not a circular step.
full rationale
I walked the claimed derivation chain and found no circular step. The paper's central quantitative claims (3.5 ms vs. 7.6 ms attention time) are obtained from Nsight Systems profiling measurements on a 4-GPU A10 platform (Sections 4.1 and 4.2), not from a model that already encodes the conclusion. No parameter is fitted to a subset of data and then called a prediction. The only mathematical derivation in the paper is the flash-attention merge update in Section 3.1, which correctly follows from the log-sum-exp identity; it is not used to conclude that TokenRing is faster. The design choice to transmit Q concurrently with block_out/block_lse is presented as an implementation mechanism, and the paper explicitly concedes that the extra block_out/block_lse volume 'offsets the expected benefits of transmitting smaller tensors like Q,' so the claimed benefit is not a tautology but an empirical claim about bidirectional bandwidth utilization. There are no load-bearing self-citations and no imported uniqueness theorems. The limitation stated in Section 4.1 ('testing on hardware optimally configured for TokenRing is not feasible for us') is a genuine external-validity concern for full-mesh NVLink/OAM deployments, but that is a correctness/validity issue, not circularity. Accordingly, the circularity score is 0.
Assumptions & free parameters
assumptions (3)
- domain assumption For sequence-parallel attention, computation time per step decreases quadratically while communication volume per step decreases only linearly as the number of devices grows (Section 3.1, citing Gu et al. 2024).
- domain assumption Two simultaneous P2P transfers in opposite ring directions can be performed without contention and with full bidirectional bandwidth (Section 3.2, Figure 3b).
- standard math The online softmax merge rule using block_lse and block_out is mathematically correct and exactly reproduces standard attention (Section 3.1, equations for out and lse).
Cite this review
Pith. "Pith review of TokenRing: An Efficient Parallelism Framework for Infinite-Context LLMs via Bidirectional Communication." pith.science (2026). https://pith.science/paper/AEQFSUK4
@misc{pith2026241220501,
author = {Pith},
title = {Pith review of: TokenRing: An Efficient Parallelism Framework for Infinite-Context LLMs via Bidirectional Communication},
year = {2026},
howpublished = {\url{https://pith.science/paper/AEQFSUK4}},
note = {Machine review of arXiv:2412.20501}
}
abstract
Efficient parallelization of Large Language Models (LLMs) with long sequences is essential but challenging due to their significant computational and memory demands, particularly stemming from communication bottlenecks in attention mechanisms. While sequence parallelism (SP) has been introduced as a potential solution, existing methods often suffer from limited scalability or inefficiency, rendering their effectiveness. Ring-Attention demonstrates the potential for scaling sequence processing but faces significant limitations due to its reliance on peer-to-peer (P2P) communication and inefficient utilization of network resources. As the degree of SP increases, the quadratic decrease in computation time per step contrasts sharply with the linear reduction in communication volume, exacerbating communication bottlenecks. To address these challenges, we propose TokenRing, a fine-grained parallel framework that leverages bidirectional P2P communication to effectively overlap computation and data transmission. By partitioning the attention block and concurrently transmitting Query and block outputs (i.e., $block\_out$ and $block\_lse$) within a fully connected mesh topology, TokenRing achieves significant reductions in communication overhead and better load balancing. These innovations improve the scalability and efficiency of distributed Transformer models, particularly for long-context sequences. Experimental results demonstrate that TokenRing enhances throughput and reduces communication latency. Moreover, its design adapts seamlessly to various multi-GPU interconnect solutions, such as Huawei Ascend, ensuring broad compatibility and cost-effectiveness for distributed LLM inference and training. The code is available at: \url{https://github.com/ACA-Lab-SJTU/token-ring}.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Agrawal, A., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B. S., and Ramjee, R. Sarathi: Efficient llm inference by piggybacking decodes with chunked prefills. arXiv preprint arXiv:2308.16369,
-
[3]
Striped attention: Faster ring attention for causal transformers
Brandon, W., Nrusimha, A., Qian, K., Ankner, Z., Jin, T., Song, Z., and Ragan-Kelley, J. Striped attention: Faster ring attention for causal transformers. arXiv preprint arXiv:2311.09431,
-
[4]
D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems , 33: 1877–1901,
1901
-
[8]
xdit: an inference engine for diffusion transformers (dits) with massive parallelism
Fang, J., Pan, J., Sun, X., Li, A., and Wang, J. xdit: an inference engine for diffusion transformers (dits) with massive parallelism. arXiv preprint arXiv:2411.01738,
-
[9]
Loong- train: Efficient training of long-sequence llms with head- context parallelism
8 Submission and Formatting Instructions for ICML 2025 Gu, D., Sun, P., Hu, Q., Huang, T., Chen, X., Xiong, Y ., Wang, G., Chen, Q., Zhao, S., Fang, J., et al. Loong- train: Efficient training of long-sequence llms with head- context parallelism. arXiv preprint arXiv:2406.18485 ,
arXiv 2025
-
[12]
Ren, J., Rajbhandari, S., Aminabadi, R
URL https:// arxiv.org/abs/2407.00079. Ren, J., Rajbhandari, S., Aminabadi, R. Y ., Ruwase, O., Yang, S., Zhang, M., Li, D., and He, Y .{Zero-offload}: Democratizing {billion-scale} model training. In 2021 USENIX Annual Technical Conference (USENIX ATC 21), pp. 551–564,
arXiv 2021
-
[14]
Tree attention: Topology-aware decoding for long-context attention on gpu clusters
Shyam, V ., Pilault, J., Shepperd, E., Anthony, Q., and Mil- lidge, B. Tree attention: Topology-aware decoding for long-context attention on gpu clusters. arXiv preprint arXiv:2408.04093,
-
[15]
Sun, X., Fang, J., Li, A., and Pan, J. Unveiling redundancy in diffusion transformers (dits): A systematic study.arXiv preprint arXiv:2411.13588,
Show all 19 references
-
[16]
Llama: Open and efficient foundation lan- guage models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi`ere, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302.13971, 2023a. Touvron, H., Martin, L., Stone, K., Albert, P...
-
[18]
Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving
Zhong, Y ., Liu, S., Chen, J., Hu, J., Zhu, Y ., Liu, X., Jin, X., and Zhang, H. Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving. arXiv preprint arXiv:2401.09670,
-
[19]
You can have an appendix here
9 Submission and Formatting Instructions for ICML 2025 A. You can have an appendix here. You can have as much text here as you want. The main body must be at most 8 pages long. For the final version, one more page can be added. If you want, you can use an appendix like this on...
2025
-
[2012]
The llama 3 herd of models
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,
-
[2014]
Sequence parallelism: Long sequence training from system perspec- tive
Li, S., Xue, F., Baranwal, C., Li, Y ., and You, Y . Sequence parallelism: Long sequence training from system perspec- tive. arXiv preprint arXiv:2105.13120,
-
[2017]
Context parallelism for scalable million- token inference
Yang, J., Ibrahim, A., Xie, X., Tang, B., Sizov, G., Park, J., Huang, J., et al. Context parallelism for scalable million- token inference. arXiv preprint arXiv:2411.01783,
-
[2019]
A., Tanaka, M., Zhang, C., Zhang, M., Song, S
Jacobs, S. A., Tanaka, M., Zhang, C., Zhang, M., Song, S. L., Rajbhandari, S., and He, Y . Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models. arXiv preprint arXiv:2309.14509,
-
[2020]
Internevo: Efficient long-sequence large language model training via hybrid parallelism and redundant sharding
Chen, Q., Gu, D., Wang, G., Chen, X., Xiong, Y ., Huang, T., Hu, Q., Jin, X., Wen, Y ., Zhang, T., et al. Internevo: Efficient long-sequence large language model training via hybrid parallelism and redundant sharding. arXiv preprint arXiv:2401.09149,
-
[2021]
Fast transformer decoding: One write-head is all you need
Shazeer, N. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150,
1911 arXiv
-
[2023]
Gqa: Training generalized multi-query transformer models from multi-head check- points
Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y ., Lebr´on, F., and Sanghai, S. Gqa: Training generalized multi-query transformer models from multi-head check- points. arXiv preprint arXiv:2305.13245,
-
[2024]
and Zhao, S
Fang, J. and Zhao, S. A unified sequence parallelism ap- proach for long context generative ai. arXiv preprint arXiv:2405.07719,
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.