Pith. sign in

REVIEW 1 major objections 5 minor 66 references

In low-batch LLM decode, All-Reduce time is mostly barriers, not data movement; SiFAR removes those barriers and raises token throughput by up to 18.6% at eight GPUs.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 01:17 UTC pith:OEVV2BUR

load-bearing objection Solid systems paper: co-designed All-Reduce that actually moves the needle on low-batch TP decode, with measured 13–19% e2e gains and honest ablations. the 1 major comments →

arxiv 2607.08973 v1 pith:OEVV2BUR submitted 2026-07-09 cs.DC

SiFAR: Synchronization-Free All-Reduce for Low-Latency LLM Inference

Hritvik Taneja , Anish Saxena , Abhishek Revinipati , Jae Hyung Ju , Neal C. Crago , Moinuddin Qureshi This is my paper
classification cs.DC
keywords low-latency LLM inferenceAll-Reducetensor parallelismin-switch reductionspeculative reductiondual bufferingsynchronization overhead
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Reasoning models and agentic systems generate intermediate tokens that no human reads, so every microsecond of per-token latency compounds into end-to-end delay. With tiny batches, decoding is bandwidth-bound, and tensor parallelism shards weights across GPUs to raise aggregate bandwidth—but All-Reduce then becomes a large share of time. The paper shows that for the small payloads of this regime, the barriers around oneshot and twoshot All-Reduce already cost 32–62% of the collective. SiFAR co-designs communication with model execution: dual buffering drops the oneshot bottom barrier, redundant pull uses the switch to reduce the full payload for each GPU, and speculative reduction starts transfer early and checks a reduced validation flag. Measured on H200 GPUs, SiFAR cuts All-Reduce latency by as much as 52% and lifts end-to-end decode throughput by 18.6% for an 8B dense model and 13.1% for a large MoE model at TP=8.

Core claim

For low-latency tensor-parallel LLM inference, synchronization barriers—not raw data transfer—dominate All-Reduce latency on the small payloads that appear at batch size one. By co-designing communication with model execution, SiFAR eliminates the oneshot bottom barrier with dual buffering, keeps data-transfer cost low via full-payload in-switch reduction (redundant pull), and largely removes top-barrier flag-exchange cost with speculative reduction that starts early and validates with a reduced flag, producing large measured cuts in collective latency and end-to-end token throughput.

What carries the argument

Synchronization-Free All-Reduce (SiFAR): dual buffering to remove the oneshot bottom barrier, redundant pull (each GPU issues switch ld_reduce over the entire payload), and speculative reduction (start transfer without the top barrier, verify via a reduced flag, retry on mismatch).

Load-bearing premise

Speculative reduction is safe only if a GPU that is ahead when it reads another GPU’s payload is still ahead when it later reads the validation flag; relative progress is assumed not to reverse between those two loads.

What would settle it

Force preemption or deliberately staggered kernel starts that invert relative GPU order between the payload read and the validation-flag read; if incorrect reductions pass validation (or silent mismatches vs a barrier All-Reduce appear under multi-tenant interference), the safety claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Once non-communication overheads are fused away, All-Reduce is the main remaining limiter of low-batch tensor-parallel decode.
  • Higher tensor-parallel degrees become more attractive for latency-sensitive serving because communication no longer scales as badly with GPU count.
  • Decode steps that issue multiple All-Reduces can treat GPUs as tightly synchronized after the first collective, so top-barrier cost is mostly flag exchange.
  • Direct collective optimization beats compute–communication overlap for the batch-size-one regime, where there is too little compute to hide traffic.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Speculate-and-validate over a reduced flag could apply to other collectives inside tightly synchronized GPU graphs, not only All-Reduce.
  • As in-switch reduction becomes common, full-payload oneshot-style collectives may stay competitive with twoshot even at higher GPU counts for small messages.
  • Long chains of unconsumed reasoning tokens would multiply these per-token gains into large wall-clock response-time savings.
  • Under preemption-heavy multi-tenant runtimes the retry path, not the happy path, would become the reliability story that needs stress-testing.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 5 minor

Summary. The paper argues that All-Reduce synchronization barriers dominate latency in low-batch, tensor-parallel LLM decode, and presents SiFAR to reduce them. It combines three co-designed techniques: dual buffering to remove the oneshot bottom (WAW) barrier by alternating payload buffers with model execution; redundant pull, which uses NVSwitch multimem.ld_reduce so each GPU reduces the full small payload in-switch rather than K×(N−1) peer data; and speculative reduction that starts transfer without the top barrier and validates via a reduced flag (with retry). Integrated into Megakernels on H200s, SiFAR cuts All-Reduce latency by up to 52% for 8–32 KB payloads and raises end-to-end decode throughput by 18.6% (Llama-3.1-8B) and 13.1% (Qwen3.5-397B-17B) at TP=8 versus the best of oneshot/twoshot, with ablations, NCCL/TRT-LLM/Lamport comparisons, and sensitivity to ISL and batch size.

Significance. If the measured gains hold, the work is a timely systems contribution for reasoning/agentic inference, where TPOT compounds and microbatching/fusion cannot hide communication. The three techniques are complementary, hardware-aware (in-switch reduction, dual buffering co-designed with memory managers), and evaluated with strong empirical discipline: microbenchmarks, full-model megakernel runs, per-optimization ablations (Fig. 22), mis-speculation cost isolation (Figs. 23–25), tail percentiles, and comparisons to oneshot, twoshot, TRT-LLM, Lamport, and NCCL 2.30. The 43% headroom from removing All-Reduce and the 18.6%/13.1% realized gains at TP=8 are concrete and actionable for low-latency serving stacks.

major comments (1)
  1. §8.1: Correctness of speculative reduction rests on a monotonicity assumption (relative GPU progress does not invert between payload read and flag read). The paper mitigates this with dual-buffered validation, L1-bypass reloads, and 100k-iteration exact match to twoshot, and Figs. 23–25 show mis-speculation costs ≤1.6% even at ~32% rate. Dual buffering and redundant pull do not depend on the assumption, so the central latency/throughput claims remain supported. Still, the manuscript should state more clearly the hardware/runtime conditions under which the assumption is expected to hold (CUDA Graphs/Megakernels, run-to-completion, no preemption) and what would be required for a stronger guarantee (e.g., hardware fence or ordered flag protocol) so readers can assess portability beyond the evaluated stack.
minor comments (5)
  1. Figure 1 caption and §3.1: Clarify that the twoshot baseline is the in-network (ld_reduce) variant throughout, so readers do not compare against a pure software twoshot.
  2. §4.2 dual-buffering implementation: The "pass previous buffer as unused argument" trick is clever but framework-specific; a short note on whether it generalizes cleanly to non-PyTorch memory managers would help.
  3. Fig. 15 vs. Fig. 16: Normalize NCCL and SiFAR to the same baseline scale in one place so relative ranking is immediate without cross-figure arithmetic.
  4. Abstract and §6: "up to 52%" is clear from Fig. 15; stating the payload/TP cell that achieves the peak would make the claim easier to cite.
  5. Related work §7: A one-sentence contrast with PRESERVE on MoE (expert selection depends on All-Reduce output) is already present; ensuring it is not buried would help readers who care about MoE-specific prefetch limits.

Circularity Check

0 steps flagged

No significant circularity: SiFAR's claims are empirical hardware measurements of engineered All-Reduce optimizations, not predictions forced by fitted parameters or self-definitional identities.

full rationale

This is a systems paper whose central claims (up to 52% lower All-Reduce latency; 18.6% / 13.1% end-to-end decode throughput gains at TP=8) are obtained by implementing dual buffering, redundant pull (multimem.ld_reduce over the full payload), and speculative reduction with flag validation, then measuring them on H200 GPUs inside a Megakernel against oneshot/twoshot, TRT-LLM, Lamport, and NCCL baselines. There is no derivation chain in which a quantity is defined from data and then re-presented as a prediction of that same data; no uniqueness theorem imported from the authors; and no ansatz smuggled via self-citation that forces the reported speedups. Self-citations are to standard libraries and prior collective algorithms (NCCL, vLLM, Megakernels, oneshot/twoshot) used as baselines or integration targets, not as load-bearing premises that make the measured gains true by construction. The monotonicity assumption for speculative-reduction correctness (§8.1) is an empirical runtime assumption checked by 100k-iteration equivalence to twoshot and by ablations of mis-speculation cost; it is not a circular definition of the latency numbers. Score 0 is therefore the correct outcome.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 2 invented entities

The central performance claims rest on standard GPU/NVSwitch behavior, the empirical observation that GPUs stay synchronized after the first All-Reduce of a decode step, and the monotonicity assumption needed for the validation flag. No free parameters are fitted to produce the speedups; the invented constructs are engineering techniques rather than new physical entities.

axioms (4)
  • domain assumption After the first All-Reduce of a forward pass launched via CUDA Graph or megakernel, GPUs remain tightly synchronized for subsequent identical All-Reduces, so top-barrier cost is dominated by flag exchange rather than divergence (§3.5, Fig. 12).
    Load-bearing for the claim that speculative reduction can safely skip the top barrier most of the time.
  • domain assumption Relative GPU progress is monotonic: if GPU A is ahead of GPU B when reading the payload, it remains ahead when later reading the validation flag (§8.1).
    Required for the validation mechanism to detect all mis-speculations; argued from run-to-completion kernels but not formally proved.
  • domain assumption NVIDIA multimem.ld_reduce performs correct in-switch reduction of the full symmetric payload and returns only K bytes to the requesting GPU (§3.4).
    Hardware primitive on which redundant pull is built; treated as given by the PTX ISA.
  • domain assumption Dual buffering of 4–8 KB per token is negligible relative to multi-GB model weights and can be forced under automatic memory managers by retaining a live reference to the previous buffer (§4.2).
    Enables removal of the bottom barrier without framework-internal changes.
invented entities (2)
  • Redundant pull independent evidence
    purpose: Each GPU issues ld_reduce over the entire K-byte payload so the switch returns a fully reduced K-byte result, combining oneshot’s single round-trip with twoshot-like data volume.
    Engineering construct; independent evidence is the measured latency curves in Figs. 9–11.
  • Speculative reduction with validation buffer independent evidence
    purpose: Start data transfer before the top barrier and detect mis-speculation by reducing a per-threadblock flag that must equal N×flag.
    Engineering construct; independent evidence is the mis-speculation rates and retry-cost measurements in Figs. 23–25.

pith-pipeline@v1.1.0-grok45 · 28000 in / 2965 out tokens · 36186 ms · 2026-07-13T01:17:53.973965+00:00 · methodology

0 comments
read the original abstract

The rise of reasoning models and agentic systems has made LLM token-generation latency a key bottleneck. Unlike chatbots, whose latency gains saturate at human reading speed, these systems generate intermediate reasoning tokens not consumed by humans. Thus, per-token latency directly determines end-to-end response time. Low-latency inference uses minimal batching, making token generation bandwidth-bound. Tensor Parallelism addresses this by sharding model weights across GPUs and loading them in parallel. However, scaling to more GPUs introduces All-Reduce overheads that grow with GPU count. Removing All-Reduce improves token throughput by 43% for Llama-3.1-8B on 8 H200 GPUs. We propose Synchronization-Free All-Reduce (SiFAR), which reduces synchronization overhead during low-latency inference. Existing oneshot and twoshot algorithms incur overheads from barriers before and after communication. First, we find that the bottom barrier in oneshot enforces a WAW dependency and eliminate it by co-designing communication and model execution to enable dual buffering. However, oneshot scales poorly with GPU count. Twoshot performs better at higher TP degrees but incurs an unavoidable bottom barrier. To overcome this, we leverage in-switch reduction in modern switches. We propose redundant pull, where each GPU reduces the full All-Reduce payload at the switch. This improves oneshot scalability while retaining its no-bottom-barrier advantage. Finally, to reduce top-barrier overhead, we observe that each decode step issues multiple All-Reduce operations, keeping GPUs tightly synchronized after the first. We therefore propose speculative reduction, which initiates data transfer before the top barrier and ensures correctness via lightweight validation. SiFAR reduces All-Reduce latency by up to 52% and improves end-to-end throughput by 18.6% for Llama-3.1-8B and 13.1% for Qwen3.5-397B-17B at TP=8.

Figures

Figures reproduced from arXiv: 2607.08973 by Abhishek Revinipati, Anish Saxena, Hritvik Taneja, Jae Hyung Ju, Moinuddin Qureshi, Neal C. Crago.

Figure 1
Figure 1. Figure 1: (a) Oneshot All-Reduce with top and bottom barriers around data transfer; these barriers account for up to 50% of [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of TP and All-Reduce: TP adds two All [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Impact of All-Reduce on TPOT across different TP degrees for Llama-3.1-8B and Qwen3.5-397B-17B. At higher TP, removing All-Reduce leads to larger gains, indicating that All-Reduce becomes an increasingly dominant bottleneck. we feed the input of the All-Reduce directly to the next opera￾tion instead). For Llama-3.1-8B, removing All-Reduce improves performance by 18% at TP=2 and 43% at TP=8. For Qwen3.5-397… view at source ↗
Figure 5
Figure 5. Figure 5: Oneshot All-Reduce: each GPU pulls the entire [PITH_FULL_IMAGE:figures/full_fig_p004_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Breakdown of All-Reduce latency on H200 GPUs. [PITH_FULL_IMAGE:figures/full_fig_p005_7.png] view at source ↗
Figure 6
Figure 6. Figure 6: Twoshot All-Reduce: each GPU reduces a portion [PITH_FULL_IMAGE:figures/full_fig_p005_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: (left) Application-agnostic All-Reduce requires a [PITH_FULL_IMAGE:figures/full_fig_p005_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Latency of reduce-scatter, the first step of twoshot [PITH_FULL_IMAGE:figures/full_fig_p006_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Redundant Pull: each GPU issues ld_reduce over the entire 𝐾 bytes. The switch reduces data from all peers and returns the result, reducing data received per GPU from 𝐾 × (𝑁 − 1) to 𝐾. Latency Comparison [PITH_FULL_IMAGE:figures/full_fig_p006_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Latency comparison of oneshot, twoshot, oneshot [PITH_FULL_IMAGE:figures/full_fig_p006_11.png] view at source ↗
Figure 13
Figure 13. Figure 13: Overview of SiFAR: (left) Redundant pull replaces [PITH_FULL_IMAGE:figures/full_fig_p007_13.png] view at source ↗
Figure 12
Figure 12. Figure 12: Representative Nsight profiler output: the cost of [PITH_FULL_IMAGE:figures/full_fig_p007_12.png] view at source ↗
Figure 14
Figure 14. Figure 14: Verification Logic to ensure correctness of specula [PITH_FULL_IMAGE:figures/full_fig_p007_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: All-Reduce latency across five implementations for TP=2, 4, and 8 with payloads from 8 to 32 KB, normalized to the [PITH_FULL_IMAGE:figures/full_fig_p009_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: All-Reduce latency with NCCL 2.30 auto-selection for TP=2, 4, and 8 across 8–32 KB payloads. NCCL is competi￾tive with the best of oneshot and twoshot, but SiFAR achieves the lowest latency across all configurations. 6.2 Impact on End-to-End LLM Inference [PITH_FULL_IMAGE:figures/full_fig_p009_16.png] view at source ↗
Figure 19
Figure 19. Figure 19: Batch-size sensitivity for Llama-3.1-8B at TP=8. SiFAR improves throughput by 18.6% at BS=1, and the gain shrinks to 10.2% at BS=4 as larger batches make All-Reduce a smaller fraction of TPOT. Comparison with Other All-Reduce Algorithms [PITH_FULL_IMAGE:figures/full_fig_p010_19.png] view at source ↗
Figure 17
Figure 17. Figure 17: End-to-end throughput (tokens/s) for Llama-3.1- [PITH_FULL_IMAGE:figures/full_fig_p010_17.png] view at source ↗
Figure 20
Figure 20. Figure 20: End-to-end throughput comparison across All￾Reduce implementations, normalized to SiFAR . SiFAR out￾performs the TRT-LLM oneshot variant and the Lamport￾style push-based All-Reduce. Impact of SiFAR on Tail Latency [PITH_FULL_IMAGE:figures/full_fig_p010_20.png] view at source ↗
Figure 18
Figure 18. Figure 18: End-to-end throughput (tokens/s) for Llama-3.1- 8B and Qwen3.5-397B-17B at TP=8 across input sequence lengths and output sequence length = 1000. Improvement decreases with longer sequences as attention dominates, but SiFAR still provides 14.6% and 10.2% improvement at ISL=16K. Impact of Batch Size [PITH_FULL_IMAGE:figures/full_fig_p010_18.png] view at source ↗
Figure 21
Figure 21. Figure 21: End-to-end TPOT percentiles for Llama-3.1 and Qwen3.5 at TP=8, normalized to the baseline p50 latency. The speedup from SiFAR holds through the tail, reducing p99 la￾tency to 84.0% for Llama-3.1-8B and 87.8% for Qwen3.5-397B￾17B, so mis-speculation retries do not inflate tail latency. 10 [PITH_FULL_IMAGE:figures/full_fig_p010_21.png] view at source ↗
Figure 24
Figure 24. Figure 24: Mis-speculation rate for Llama-3.1-8B and [PITH_FULL_IMAGE:figures/full_fig_p011_24.png] view at source ↗
Figure 25
Figure 25. Figure 25: Retry latencies of SiFAR. Even with retries, the average SiFAR latency stays below the baseline. 6.4 Comparison with Compute-Comm. Fusion [PITH_FULL_IMAGE:figures/full_fig_p011_25.png] view at source ↗
Figure 23
Figure 23. Figure 23: SiFAR compared against idealized configurations: [PITH_FULL_IMAGE:figures/full_fig_p011_23.png] view at source ↗
Figure 26
Figure 26. Figure 26: compares fine-grained compute-communication fusion from ParallelKittens [49] against an unfused implementation for the down-projection GEMM of Llama-3.1-8B at TP=8. At small batch sizes, the fused GEMM has similar or worse latency, because there is not enough computation to overlap with communication. Fusion helps only at very large batch sizes, such as BS=2048, where the GEMM has enough work to hide comm… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

66 extracted references · 17 linked inside Pith

  1. [1]

    Deepak Bhaskar Acharya, Karthigeyan Kuppan, and B Divya. 2025. Agentic ai: Autonomous intelligence for complex goals–a comprehensive survey.IEEe Access (2025)

    2025

  2. [2]

    Matthew Joseph Adiletta, Gu-Yeon Wei, and David Brooks. 2026. RPU–A Reason- ing Processing Unit. In2026 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE, 1–17

    2026

  3. [3]

    Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. 2025. gpt-oss-120b & gpt-oss-20b model card.arXiv preprint arXiv:2508.10925(2025)

    Pith/arXiv arXiv 2025

  4. [4]

    Alham Fikri Aji and Kenneth Heafield. 2017. Sparse communication for dis- tributed gradient descent.arXiv preprint arXiv:1704.05021(2017)

    Pith/arXiv arXiv 2017

  5. [5]

    Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. 2017. QSGD: Communication-efficient SGD via gradient quantization and encoding. Advances in neural information processing systems30 (2017)

    2017

  6. [6]

    Marc Brysbaert. 2019. How many words do we read per minute? A review and meta-analysis of reading rate.Journal of memory and language109 (2019), 104047

    2019

  7. [7]

    Milind Chabbi, Wim Lavrijsen, Wibe de Jong, Koushik Sen, John Mellor-Crummey, and Costin Iancu. 2015. Barrier elision for production parallel programs.ACM SIGPLAN Notices50, 8 (2015), 109–119

    2015

  8. [8]

    K Mani Chandy and Leslie Lamport. 1985. Distributed snapshots: Determining global states of distributed systems.ACM Transactions on Computer Systems (TOCS)3, 1 (1985), 63–75

    1985

  9. [9]

    Li-Wen Chang, Wenlei Bao, Qi Hou, Chengquan Jiang, Ningxin Zheng, Yinmin Zhong, Xuanrun Zhang, Zuquan Song, Ziheng Jiang, Haibin Lin, Xin Jin, and Xin Liu. 2024. FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusion. arXiv:2406.06858 [cs.LG]

    Pith/arXiv arXiv 2024

  10. [10]

    Michael Davies, Neal Crago, Karthikeyan Sankaralingam, and Stephen W Keck- ler. 2025. Kitsune: Enabling Dataflow Execution on GPUs.arXiv preprint arXiv:2502.18403(2025)

    Pith/arXiv arXiv 2025

  11. [11]

    Michael Davies, Neal Crago, Karthikeyan Sankaralingam, and Christos Kozyrakis

  12. [12]

    Efficient llm inference: Bandwidth, compute, synchronization, and capacity are all you need.arXiv preprint arXiv:2507.14397(2025)

    arXiv 2025

  13. [13]

    William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research23, 120 (2022), 1–39

    2022

  14. [14]

    Jiawei Fei, Chen-Yu Ho, Atal N Sahu, Marco Canini, and Amedeo Sapio. 2021. Efficient sparse collective communication and its application to accelerate dis- tributed deep learning. InProceedings of the 2021 ACM SIGCOMM 2021 Conference. 676–691

    2021

  15. [15]

    Luigi Fusco, Mikhail Khalilov, Marcin Chrapek, Giridhar Chukkapalli, Thomas Schulthess, and Torsten Hoefler. 2024. Understanding data movement in tightly coupled heterogeneous systems: A case study with the Grace Hopper superchip. arXiv preprint arXiv:2408.11556(2024)

    Pith/arXiv arXiv 2024

  16. [16]

    Raja Gond, Nipun Kwatra, and Ramachandran Ramjee. 2025. TokenWeave: Efficient Compute-Communication Overlap for Distributed LLM Inference. https: //arxiv.org/abs/2505.11329

    Pith/arXiv arXiv 2025

  17. [17]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models.arXiv preprint arXiv:2407.21783 (2024)

    Pith/arXiv arXiv 2024

  18. [18]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al . 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948(2025)

    Pith/arXiv arXiv 2025

  19. [19]

    Maurice Herlihy and J Eliot B Moss. 1993. Transactional memory: Architec- tural support for lock-free data structures. InProceedings of the 20th annual international symposium on Computer architecture. 289–300

    1993

  20. [20]

    Zhiyi Hu, Siyuan Shen, Tommaso Bonato, Sylvain Jeaugey, Cedell Alexander, Eric Spada, James Dinan, Jeff Hammond, and Torsten Hoefler. 2025. Demystifying NCCL: An in-depth analysis of GPU communication protocols and algorithms. In2025 IEEE Symposium on High-Performance Interconnects (HOTI). IEEE, 48–59

    2025

  21. [21]

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. 2019. Gpipe: Efficient training of giant neural networks using pipeline parallelism.Advances in neural information processing systems32 (2019)

    2019

  22. [22]

    Zhixian Jin, Christopher Rocca, Jiho Kim, Hans Kasan, Minsoo Rhu, Ali Bakhoda, Tor M Aamodt, and John Kim. 2024. Uncovering real gpu noc characteristics: Implications on interconnect architecture. In2024 57th IEEE/ACM International Symposium on Microarchitecture (MICRO). IEEE, 885–898

    2024

  23. [23]

    Hans Kasan, Dennis Abts, Jungwook Choi, and John Kim. 2025. SkipReduce:(Interconnection) Network Sparsity to Accelerate Distributed Machine Learning. InProceedings of the 58th IEEE/ACM International Symposium on Microarchitecture®. 643–658

    2025

  24. [24]

    Benjamin Klenk, Nan Jiang, Greg Thorson, and Larry Dennison. 2020. An in- network architecture for accelerating shared-memory multiprocessor collectives. In2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA). IEEE, 996–1009

    2020

  25. [25]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Mem- ory Management for Large Language Model Serving with PagedAttention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles

    2023

  26. [26]

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast Inference from Transformers via Speculative Decoding. InProceedings of the 40th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 202). PMLR, 19274–19286

    2023

  27. [27]

    Dacheng Li, Shiyi Cao, Chengkun Cao, Xiuyu Li, Shangyin Tan, Kurt Keutzer, Jiarong Xing, Joseph E Gonzalez, and Ion Stoica. 2025. S*: Test time scaling for code generation.arXiv preprint arXiv:2502.14382(2025)

    Pith/arXiv arXiv 2025

  28. [28]

    Minghao Li, Ran Ben Basat, Shay Vargaftik, ChonLam Lao, Kevin Xu, Michael Mitzenmacher, and Minlan Yu. 2024. {THC}: Accelerating distributed deep learning using tensor homomorphic compression. In21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24). 1191–1211

    2024

  29. [29]

    Shigang Li and Torsten Hoefler. 2022. Near-optimal sparse allreduce for dis- tributed deep learning. InProceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming. 135–149

    2022

  30. [30]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437(2024)

    Pith/arXiv arXiv 2024

  31. [31]

    Jose F Martinez and Josep Torrellas. 2002. Speculative synchronization: Applying thread-level speculation to explicitly parallel applications.ACM SIGOPS Operating Systems Review36, 5 (2002), 18–29

    2002

  32. [32]

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. 2025. s1: Simple test-time scaling.arXiv preprint arXiv:2501.19393 (2025)

    Pith/arXiv arXiv 2025

  33. [33]

    NVIDIA. 2016. Pascal Tuning Guide: Compute Preemption. https://docs.nvidia. com/cuda/pascal-tuning-guide/index.html#preemption

    2016

  34. [34]

    NVIDIA. 2025. Blackwell Breaks the 1,000 TPS/User Barrier With Meta’s Llama 4 Maverick. https://developer.nvidia.com/blog/blackwell-breaks-the-1000-tps- user-barrier-with-metas-llama-4-maverick/

    2025

  35. [35]

    NVIDIA. 2025. Programmatic Dependent Launch and Synchroniza- tion. https://docs.nvidia.com/cuda/cuda-c-programming-guide/#programmatic- dependent-launch-and-synchronization Online; accessed 2025-11-17

    2025

  36. [36]

    Nvidia. 2025. TensorRT-LLM: A TensorRT Toolbox for Optimized Large Language Model Inference. https://github.com/NVIDIA/TensorRT-LLM

    2025

  37. [37]

    NVIDIA Corporation. 2025. NCCL: NVIDIA Collective Communication Library. https://github.com/NVIDIA/nccl. Accessed 2025-11-17

    2025

  38. [38]

    NVIDIA Corporation. 2025. Parallel Thread Execution ISA Version 8.7. https: //docs.nvidia.com/cuda/parallel-thread-execution/index.html Accessed: March 2026

    2025

  39. [39]

    Retrieved November 16, 2025

    NVIDIA Corporation. Retrieved November 16, 2025. NVIDIA Nsight Compute. https://developer.nvidia.com/nsight-compute

    2025

  40. [40]

    NVIDIA Developer Blog. 2026. Inside NVIDIA Groq 3 LPX: The Low-Latency Inference Accelerator for the NVIDIA Vera Rubin Plat- form. https://developer.nvidia.com/blog/inside-nvidia-groq-3-lpx-the-low- latency-inference-accelerator-for-the-nvidia-vera-rubin-platform/. Accessed: 2026-03-27

    2026

  41. [41]

    Manuel Pedrero, Eladio Gutierrez, and Oscar Plata. 2018. TMbarrier: Speculative barriers using hardware transactional memory. In2018 26th Euromicro Interna- tional Conference on Parallel, Distributed and Network-based Processing (PDP). IEEE, 214–221

    2018

  42. [42]

    Qwen Team. 2026. Qwen3.5: Towards Native Multimodal Agents. https: //qwen.ai/blog?id=qwen3.5

    2026

  43. [43]

    Ravi Rajwar and James R Goodman. 2001. Speculative lock elision: Enabling highly concurrent multithreaded execution. InProceedings. 34th ACM/IEEE Inter- national Symposium on Microarchitecture. MICRO-34. IEEE, 294–305

    2001

  44. [44]

    Ravi Rajwar and James R Goodman. 2002. Transactional lock-free execution of lock-based programs.ACM SIGOPS Operating Systems Review36, 5 (2002), 5–17

    2002

  45. [45]

    Cèdric Renggli, Saleh Ashkboos, Mehdi Aghagolzadeh, Dan Alistarh, and Torsten Hoefler. 2019. SparCML: High-performance sparse communication for machine learning. InProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. 1–15

    2019

  46. [46]

    Amedeo Sapio, Marco Canini, Chen-Yu Ho, Jacob Nelson, Panos Kalnis, Changhoon Kim, Arvind Krishnamurthy, Masoud Moshref, Dan Ports, and Pe- ter Richtárik. 2021. Scaling distributed machine learning with {In-Network} 13 aggregation. In18th USENIX Symposium on Networked Systems Design and Imple- mentation (NSDI 21). 785–808

    2021

  47. [47]

    SemiAnalysis. 2026. InferenceX: Open-Source Continuous AI Inference Bench- mark. https://inferencex.semianalysis.com/. Accessed: 2026-04-05

    2026

  48. [48]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-LM: Training multi-billion parameter language models using model parallelism.arXiv preprint arXiv:1909.08053(2019)

    Pith/arXiv arXiv 2019

  49. [49]

    Benjamin Spector, Jordan Juravsky, Stuart Sul, Owen Dugan, Dylan Lim, Dan Fu, Simran Arora, and Chris Ré. 2025. Look Ma, No Bubbles! Designing a Low- Latency Megakernel for Llama-1B. https://hazyresearch.stanford.edu/blog/2025- 05-27-no-bubbles. Blog post

    2025

  50. [50]

    Stuart H Sul, Simran Arora, Benjamin F Spector, and Christopher Ré. 2025. Par- allelKittens: Systematic and Practical Simplification of Multi-GPU AI Kernels. arXiv preprint arXiv:2511.13940(2025)

    arXiv 2025

  51. [51]

    Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al . 2025. Kimi k2: Open agentic intelligence.arXiv preprint arXiv:2507.20534(2025)

    Pith/arXiv arXiv 2025

  52. [52]

    Amit Tiwari and V Krishna Nandivada. 2025. Unleashing Parallelism with Elastic- Barriers.ACM Transactions on Architecture and Code Optimization22, 2 (2025), 1–26

    2025

  53. [53]

    Retrieved November 16, 2025

    vLLM. Retrieved November 16, 2025. vLLM Custom All-Reduce Implementation. https://github.com/vllm-project/vllm/blob/main/csrc/custom_all_reduce.cuh

    2025

  54. [54]

    Thijs Vogels, Sai Praneeth Karimireddy, and Martin Jaggi. 2019. PowerSGD: Practical low-rank gradient compression for distributed optimization.Advances in Neural Information Processing Systems32 (2019)

    2019

  55. [55]

    Yifu Wang, Horace He, and Luca Wehrstedt. 2025. PyTorch SymmetricMem- ory: Harnessing NVLink Programmability with Ease. PyTorch Developer Blog. https://dev-discuss.pytorch.org/t/pytorch-symmetricmemory-harnessing- nvlink-programmability-with-ease/2798 Accessed: March 2026

    2025

  56. [56]

    Mengdi Wu, Xinhao Cheng, Shengyu Liu, Chunan Shi, Jianan Ji, Kit Ao, Praveen Velliengiri, Xupeng Miao, Oded Padon, and Zhihao Jia. 2025. Mirage: A Multi- Level Superoptimizer for Tensor Programs. In19th USENIX Symposium on Oper- ating Systems Design and Implementation (OSDI 25). USENIX Association, Boston, MA

    2025

  57. [57]

    Mengdi Wu, Xinhao Cheng, Shengyu Liu, Chunan Shi, Jianan Ji, Man Kit Ao, Praveen Velliengiri, Xupeng Miao, Oded Padon, and Zhihao Jia. 2025. Mirage: A {Multi-Level} Superoptimizer for Tensor Programs. In19th USENIX Symposium on Operating Systems Design and Implementation (OSDI 25). 21–38

    2025

  58. [58]

    Pooria M Yaghini, George Michelogiannakis, and Paul V Gratz. 2019. SpecLock: Speculative lock forwarding. In2019 IEEE 37th International Conference on Com- puter Design (ICCD). IEEE, 273–282

    2019

  59. [59]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report.arXiv preprint arXiv:2505.09388(2025)

    Pith/arXiv arXiv 2025

  60. [60]

    Ahmet Caner Yüzügüler, Jiawei Zhuang, and Lukas Cavigelli. 2025. PRESERVE: Prefetching Model Weights and KV-Cache in Distributed LLM Serving.arXiv preprint arXiv:2501.08192(2025)

    Pith/arXiv arXiv 2025

  61. [61]

    Chen Zhang, Qijun Zhang, Zhuoshan Zhou, Yijia Diao, Haibo Wang, Zhe Zhou, Zhipeng Tu, Zhiyao Li, Guangyu Sun, Zhuoran Song, et al . 2026. Towards Compute-Aware In-Switch Computing for LLMs Tensor-Parallelism on Multi- GPU Systems. In2026 IEEE International Symposium on High Performance Com- puter Architecture (HPCA). IEEE, 1–15

    2026

  62. [62]

    Chenggang Zhao, Shangyan Zhou, Liyue Zhang, Chengqi Deng, Zhean Xu, Yuxuan Liu, Kuai Yu, Jiashi Li, and Liang Zhao. 2025. DeepEP: an efficient expert-parallel communication library. https://github.com/deepseek-ai/DeepEP

    2025

  63. [63]

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. 2024. Sglang: Efficient execution of structured language model programs. Advances in neural information processing systems37 (2024), 62557–62583

    2024

  64. [64]

    Size Zheng, Wenlei Bao, Qi Hou, Xuegui Zheng, Jin Fang, Chenhui Huang, Tianqi Li, Haojie Duanmu, Renze Chen, Ruifan Xu, Yifan Guo, Ningxin Zheng, Ziheng Jiang, Xinyi Di, Dongyang Wang, Jianxi Ye, Haibin Lin, Li-Wen Chang, Liqiang Lu, Yun Liang, Jidong Zhai, and Xin Liu. 2025. Triton-distributed: Program- ming Overlapping Kernels on Distributed AI Systems ...

    Pith/arXiv arXiv 2025

  65. [65]

    Size Zheng, Jin Fang, Xuegui Zheng, Qi Hou, Wenlei Bao, Ningxin Zheng, Ziheng Jiang, Dongyang Wang, Jianxi Ye, Haibin Lin, et al. 2025. Tilelink: Generating ef- ficient compute-communication overlapping kernels using tile-centric primitives. arXiv preprint arXiv:2503.20313(2025)

    Pith/arXiv arXiv 2025

  66. [66]

    Kan Zhu, Yufei Gao, Yilong Zhao, Liangyu Zhao, Gefei Zuo, Yile Gu, Dedong Xie, Zihao Ye, Keisuke Kamahori, Chien-Yu Lin, et al. 2025. {NanoFlow}: Towards optimal large language model serving throughput. In19th USENIX Symposium on Operating Systems Design and Implementation (OSDI 25). 749–765. 14

    2025