Pith. sign in

REVIEW 2 major objections 6 minor 2 cited by

Flash Communication: Reducing Tensor Parallelization Bottleneck for Fast Large Language Model Inference

T0 review · 2 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Low-bit quantized All-Reduce cuts tensor-parallel LLM inference communication by up to 3x and first-token latency by 2x.

desk verdict A credible systems paper that overclaims in the abstract: the headline 2x TTFT uses INT4 communication where accuracy drops by ~1.5 points, so the 'nearly no sacrifice' line should be revised, but the method itself is a solid, publishable contribution. read the letter →

arxiv 2412.04964 v2 pith:CSM4UHHK submitted 2024-12-06 cs.AI

Qingyuan Li , Bo Zhang , Liang Ye , Yifan Zhang , Wei Wu , Yerui Sun , Lin Ma , Yuchen Xie This is my paper · ORCID
classification cs.AI
keywords FlashAll-ReducetensorparallelismactivationquantizationLLMinferencetime-to-first-tokencommunicationcompressionfine-grained
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Flash Communication attacks the communication bottleneck that tensor parallelism creates during LLM inference: on bandwidth-limited GPUs such as the L40, all-reduce can take more than half of prefill time. The paper claims that by quantizing the activations exchanged between tensor-parallel ranks to 4-8 bits with group size 128, and restructuring All-Reduce into a two-step quantized exchange, communication volume shrinks enough to cut intra-node all-reduce latency by up to 3.18x and time-to-first-token by up to 2.06x while keeping benchmark accuracy nearly unchanged. The method is implemented as a fused CUDA kernel called Flash All-Reduce, using INT4 for the reduce stage and INT8 for the gather stage. A sympathetic reader would care because it offers a practical way to speed up prefill on low-bandwidth inference servers without changing model weights.

What carries the argument

The load-bearing object is the Flash All-Reduce kernel, a fused CUDA implementation of a two-step All-Reduce. Instead of Ring All-Reduce's N-1 reduce-scatter and N-1 all-gather steps with N quantize/dequantize cycles, it divides each rank's activation into chunks, quantizes each chunk with asymmetric group-128 scaling, exchanges them via one All2All over peer-to-peer GPU memory access, sums locally, re-quantizes the partial sums to INT8, and finishes with one All-Gather and dequantization. The kernel's speed comes from limiting quantization round trips to two, using bit-packing tricks for fast INT4-to-FP16 dequantization, and tuning the number of streaming multiprocessors to the communication volume.

What would settle it

Apply Flash Communication's INT4 All-Reduce to LLaMA-3-70B with FP16 weights at 32k-token input on 4x L40 and compare C4 perplexity against FP16 communication; if the gap exceeds the roughly 0.3-point margin reported at 1024 tokens, the accuracy-preservation claim fails for long contexts.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that tensor-parallel communication is a first-order cost in LLM inference, not just training, and that it can be reduced without re-architecting the model. The central discovery is that activations in the output and down projections can be fine-grained quantized per 128-element group and exchanged through a two-step All-Reduce: first an All2All of quantized chunks followed by local reduction, then a second quantized All-Gather, so that only two quantization/dequantization round trips occur instead of N. The hard tensor is the down projection, whose quantization MSE is far above the output projection; the paper controls it with asymmetric INT4/INT8 grouping and shows that on LLaMA-2 and LLaMA-3 models, INT6 and INT4 perplexity stays within a small margin of near-lossless communication. The speed payoff is demonstrated on L40 and A100 GPUs: a 3.18x faster All-Reduce kernel at 1GB volume with INT4, TTFT speedups up to 2.06x for 8-bit weight-quantized LLaMA-3-8B on L40, and a smaller 1.19x gain for LLaMA-3-70B on A100.

Load-bearing premise

The accuracy-preserving premise is that fine-grained 4- to 8-bit quantization of activations, including the hard-to-quantize down-projection tensor, introduces errors that stay small enough over dozens of layers that downstream task accuracy is unaffected.

Editorial extensions

If this is right

  • On bandwidth-limited L40 nodes, INT4 communication cuts All-Reduce kernel latency by up to 3.18x at large volumes, shifting prefill cost away from communication toward compute.
  • For 8-bit weight-quantized LLaMA-3-8B, INT4 communication gives a 2.06x time-to-first-token speedup at batch size 64 with tensor parallelism of 4 on L40.
  • On A100 with tensor parallelism of 8, LLaMA-3-70B still gains 1.19x in time-to-first-token with INT4, so the method remains useful on high-bandwidth hardware but with smaller returns.
  • The INT6 configuration (INT4 reduce, INT8 gather) keeps C4 perplexity within roughly 0.3 points of near-lossless communication while retaining most of the latency gain.
  • On PIQA, ARC, HellaSwag, and WinoGrande, INT6 essentially matches FP16 communication and INT4 stays within about one point for most tested models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the speedup grows with communication volume, the same quantized two-step pattern should transfer to other collective-heavy inference settings such as expert parallelism and sequence parallelism, which the paper does not claim to cover.
  • A natural extension the paper leaves implicit is applying rotation-based outlier suppression to the down-projection tensor only, since the paper's own ablation shows rotation quantizes well at coarse granularity but loses its edge at group size 128.
  • The paper's INT6 configuration is effectively a tunable accuracy-latency dial; a deployment could choose bit widths per layer based on measured down-projection MSE rather than using one global setting.
  • The A100 results suggest a practical rule: use Flash Communication when a cost breakdown shows communication above a few tens of percent of prefill time, and keep FP16 communication when bandwidth is abundant.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper introduces Flash Communication, a low-bit activation compression scheme that replaces the NCCL Ring All-Reduce in tensor-parallel LLM inference with a two-step quantized All2All plus All-Gather strategy, implemented as a fused CUDA kernel called Flash All-Reduce. The authors report kernel latency reductions up to 3.18x on L40 GPUs and time-to-first-token (TTFT) reductions up to 2.06x for an 8-bit weight-quantized LLaMA-3-8B on L40 using INT4 communication, with smaller gains on A100. Accuracy is evaluated through C4 and WikiText perplexity and five commonsense reasoning benchmarks across LLaMA-2/3 models from 7B to 70B parameters, under both FP16 and INT8 weight quantization.

Significance. If the kernel and TTFT measurements are correct, this is a practical optimization for prefill latency on bandwidth-limited inference GPUs, a setting where communication can dominate cost (Figure 1). The paper's strengths include detailed communication profiling, a non-trivial fused kernel design with fine-grained asymmetric quantization and fast INT4 dequantization, and systematic ablations of quantization granularity, bit width, INT versus FP formats, and SM count. The comparisons against an external NCCL baseline and standard benchmarks make the speedup results concrete and falsifiable. The principal weakness is the accuracy claim attached to the headline INT4 configuration, which the paper's own tables do not support.

major comments (2)
  1. [Section 4.3, Fig. 9; Tables 3 and 7]
  2. [Section 3.2.1, Fig. 5]
minor comments (6)
  1. [Table 4] The table lists 'INT4 Asym' twice, with different numbers in each row; one row is likely INT4 Sym or a different granularity. Please correct the label or add the missing configuration.
  2. [References] There are duplicate citation keys: 'NVIDIA, 2024a' appears for both Megatron-LM and Collective operations, and 'NVIDIA, 2024b' for both Nsight Systems and Context parallelism. Please renumber these entries.
  3. [Algorithm 1] In the dequantization line, 'Dequantize(M_q_ij, z_ij, ij)' should likely be 'Dequantize(M_q_ij, z_ij, s_ij)' with the scale variable defined; the current subscript is ambiguous.
  4. [Figure 8] The figure contains garbled tokens such as '0x40x6' and 'W10x0'. Please redraw the figure and ensure all mask and subtraction constants are legible and correct.
  5. [Section 3.2.3] The term 'INT6' denotes a mixed scheme (INT4 for Reduce-Scatter, INT8 for All-Gather), not a uniform 6-bit format. Please state this explicitly in the experimental tables so readers do not misinterpret the effective communication bit width.
  6. [General] The paper does not include an artifact or code availability statement. Releasing the kernel and benchmark scripts would strengthen reproducibility, given that the main results are systems measurements.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the speed and accuracy claims are measured against external baselines and benchmarks, and the self-citations are motivational rather than load-bearing.

full rationale

The paper's central claims are empirical rather than derived from their own outputs. The kernel speedup (up to 3.18x for INT4 Flash All-Reduce in Fig. 10) is measured directly against NCCL Ring All-Reduce with nccl-test, and the TTFT speedups (up to 2.06x in Fig. 9) are measured end-to-end against an FP16-communication baseline on L40 and A100 GPUs. The accuracy claims are evaluated on external benchmarks (PIQA, ARC, HellaSwag, WinoGrande, C4, WikiText-2) against an FP16-communication baseline (Tables 2, 3, 6, 7), so the 'nearly no sacrifice' claim is testable externally and is not manufactured by construction. The quantization algorithm in Alg. 1 is a concrete procedure whose bit-widths and group sizes are selected by ablation experiments (Figs. 4 and 5), not fitted to the headline numbers. The authors cite their own FPTQ work for the observation that down-projection activations are harder to quantize, but this observation is also attributed to Ashkboos et al. 2023 and Yu et al. 2024, and it motivates the design rather than establishing the claimed speedups. LMDeploy is cited only for an INT4 packing layout, an implementation detail. There is no invoked uniqueness theorem, no fitted parameter renamed as a prediction, and no equation that reduces the result to its inputs. Even if the accuracy drop at the 2x TTFT setting (e.g., LLaMA-3-8B average accuracy 72.44 to 70.93, C4 perplexity 9.01 to 9.85) undermines the abstract's 'nearly no sacrifice' phrasing, that is a correctness or support concern, not circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method's main knobs are the quantization group size (128), the SM count (48), and the mixed-precision schedule (INT4 for reduce-scatter, INT8 for all-gather). These were selected from experiments rather than predicted from theory. The kernel also relies on standard FP16/quantization definitions and on the empirical fact that All2All plus local reduction is faster than NCCL ring on these nodes.

free parameters (4)
  • Quantization group size g = 128
    Chosen in Fig. 4 as the finest granularity tested that keeps C4 perplexity acceptable while limiting kernel overhead.
  • SM count = 48
    Tuned from Fig. 11 as a balance between kernel launch overhead and compute capacity.
  • Communication bit-widths for reduce and gather stages = INT4 for All2All/ReduceSum, INT8 for All-Gather (called INT6)
    Selected because quantization before All-Gather accumulates errors (Fig. 5 right); no theoretical derivation.
  • SmoothQuant alpha for INT8 weight baselines = 0.85 (0.9 for LLaMA2-70B)
    Reused from SmoothQuant for the INT8-weight experiments; not a contribution of this paper.
assumptions (4)
  • domain assumption Asymmetric quantization with a group size of 128 preserves LLM output quality well enough for the reported tasks
    Validated on C4, WikiText, and five accuracy benchmarks, but not guaranteed for other models or tasks; Section 3.2.1 and Tables 2, 3, 6, 7.
  • domain assumption All2All via CUDA peer-to-peer memory is faster than NCCL Ring All-Reduce on the tested L40/A100 nodes
    Measured in Fig. 10; may not hold for inter-node or NVSwitch topologies.
  • domain assumption Tensor-parallel All-Reduce is the dominant prefill cost on L40
    Measured with NSys in Fig. 1 and Fig. 2; motivates the entire method.
  • standard math IEEE 754 FP16 arithmetic and the standard min/max quantization formulas in Eq. (1)-(4) are correct
    Standard definitions, no proof needed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Flash Communication: Reducing Tensor Parallelization Bottleneck for Fast Large Language Model Inference." pith.science (2026). https://pith.science/paper/CSM4UHHK

@misc{pith2026241204964,
  author       = {Pith},
  title        = {Pith review of: Flash Communication: Reducing Tensor Parallelization Bottleneck for Fast Large Language Model Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CSM4UHHK}},
  note         = {Machine review of arXiv:2412.04964}
}
read the original abstract

The ever-increasing sizes of large language models necessitate distributed solutions for fast inference that exploit multi-dimensional parallelism, where computational loads are split across various accelerators such as GPU clusters. However, this approach often introduces significant communication overhead, especially on devices with limited bandwidth. In this paper, we introduce Flash Communication, a novel low-bit compression technique designed to alleviate the tensor-parallelism communication bottleneck during inference. Our method substantially boosts intra-node communication speed by more than 3x and reduces the time-to-first-token by 2x, with nearly no sacrifice in model accuracy. Extensive experiments on various up-to-date LLMs demonstrate the effectiveness of our approach.

Figures

Figures reproduced from arXiv: 2412.04964 by the authors.

Figure 1
Figure 1. Prefill cost breakdown of LLaMA-3-70B operations with and without Flash Communication, as measured by NSys (NVIDIA, 2024b). Tested on 4×L40/A100 GPUs (TP=4) with a batch size of 8, each with 1024 input and 64 output tokens. NCCL (NVIDIA, 2024d)’s Ring All-Reduce is applied. The notion of x-ticks (e.g. L40 FP16/FP16) denotes GPU type, model weight precision, and communication precision, respectively. rameters. In the… view at source ↗
Figure 2
Figure 2. Prefill cost breakdown of LLaMA-3-70B operations at various sequence lengths. Tested on 4×L40 GPUs (TP=4) with a batch size of 8. For communication optimization, one can think of overlap￾ping the communication with computation to hide overhead but it requires sophisticated design with scheduling which is harder to implement in any given inference framework, for which reason NanoFlow (Zhu et al., 2024) invents a new … view at source ↗
Figure 4
Figure 4. Activation quantization with various block sizes of LLaMA-3-8B on C4. Starting from 4096 (the length of hidden dimension), the granularity becomes finer till 128. activation quantization in this scenario. To investigate the quantization sensitivity, we calculate the layerwise mean squared errors (MSE) before and after activation quantiza￾tion on LLaMA-3-8B, as depicted in [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (10 more)
Figure 5
Figure 5. Figure 5: Left: Comparison of oproj and dproj All-Reduce Quan￾tization MSE. Right: MSE of quantization before Reduce-Scatter (RS) vs. All-Gather (AG). Besides, an All-Reduce comprises a pair of Reduce-Scatter and All-Gather operations, where the quantization corre￾sponding to ea…
Figure 6
Figure 6. Figure 6: Flash communication’s two-step All-Reduce. The communication volume is quantized and dequantized only twice. 3.2.2. ALGORITHM Considering the above issues, we design a two-step quanti￾zation strategy to replace vanilla Ring All-Reduce, as por￾trayed in [PITH_FULL_IMAG…
Figure 7
Figure 7. Figure 7: Thread mapping of fast fine-grained quantization. time API (NVIDIA Corporation, 2024) to transmit quan￾tized volume, where we can directly fetch data from differ￾ent ranks, substantially boosting the communication speed. Fast Dequantization. Once quantization volumes a…
Figure 8
Figure 8. Figure 8: Given that two 32-bit unsigned integers U0 and U1 holding 4 INT4-quantized activations (each stored in the lower 4 bits out of 8 bits) to transmit, we first perform the right shift by 12 bits, and then apply bitwise OR to itself. Later we select the target bits from th…
Figure 9
Figure 9. Figure 9: TTFT speed-up ratio of 8-bit LLaMA-3-8B under various communication quantization bit widths on L40 with TP=4 (left), and 8-bit LLaMA-3-70B on A100 with TP=8 (right). INT8 Weights. As model weights are quantized, the im￾pact of communication overhead becomes more pronou…
Figure 11
Figure 11. Figure 11: The number of SMs affects the communication latency at different sizes of communication volume. SM=48 is similar to SM=128 in larger volumes. 6. Conclusion Our work presents a novel technique to reduce the commu￾nication volume associated with tensor parallelism while…
Figure 10
Figure 10. Figure 10: Flash Communication’s All-Reduce (Flash AR) Perfor￾mance compared with NCCL’s ring version. NCCL’s latency is tested with nccl-test (NVIDIA, 2016-2024). Assembling several boosting techniques, the speed of our Flash All-Reduce kernel surpasses that of Ring All-Reduce …
Figure 12
Figure 12. Figure 12: Physical topology of two NVIDIA 8× L40 GPU nodes connected for inference. Each node has 8 GPUs interconnected with PCI Switches and 2 NUMA nodes. For simplicity, NIC is shown to only connect with the last PCI. For high-performance large-scale training, high-end GPUs l…
Figure 13
Figure 13. Figure 13: TTFT speed-up ratio of LLaMA-3-8B (INT8 vs. FP16) under various communication quantization bit widths on L40 with TP=2. 8 16 32 64 Batch Size 0.5 0.6 0.7 0.8 0.9 1.0 1.1 TTFT Speed-up Ratio 1.06 1.091.11 1.06 1.081.10 1.05 1.071.09 1.04 1.071.08 LLaMA-3-70B (INT8) on …
Figure 14
Figure 14. Figure 14: TTFT speed-up ratio of LLaMA-3-70B (INT8 vs. FP16) under various communication quantization bit widths on A100 with TP=4. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_14.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. HAP: Hybrid Adaptive Parallelism for Efficient Mixture-of-Experts Inference

    cs.DC 2025-08 conditional novelty 6.0 of 10

    HAP uses ILP over module-specific hybrid parallel choices to speed up MoE inference, reporting up to 1.77x versus tensor parallelism on tested GPUs.

  2. TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inference

    cs.LG 2025-08 conditional novelty 5.0 of 10

    TPLA splits the latent KV cache across tensor-parallel GPUs while keeping every head's full view, yielding 1.79x and 1.93x decode speedups on DeepSeek-V3 and Kimi-K2 at 32K context with modest accuracy loss.

Reference graph

Works this paper leans on

74 extracted references · 30 canonical work pages · cited by 2 Pith papers

  1. [1]

    Aji, A. F. and Heafield, K. Sparse communication for distributed gradient descent. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2017. doi:10.18653/v1/d17-1045. URL http://dx.doi.org/10.18653/v1/D17-1045

    doi:10.18653/v1/d17-1045 2017

  2. [2]

    Towards end-to-end 4-bit inference on generative large language models

    Ashkboos, S., Markov, I., Frantar, E., Zhong, T., Wang, X., Ren, J., Hoefler, T., and Alistarh, D. Towards end-to-end 4-bit inference on generative large language models. arXiv preprint arXiv:2310.09259, 2023

    arXiv 2023

  3. [3]

    L., Li, B., Cameron, P., Jaggi, M., Alistarh, D., Hoefler, T., and Hensman, J

    Ashkboos, S., Mohtashami, A., Croci, M. L., Li, B., Cameron, P., Jaggi, M., Alistarh, D., Hoefler, T., and Hensman, J. Quarot: Outlier-free 4-bit inference in rotated llms. arXiv preprint arXiv:2404.00456, 2024

    arXiv 2024

  4. [4]

    baidu-allreduce: A C++ library demonstrating ring allreduce and ring allgather techniques

    Baidu Research . baidu-allreduce: A C++ library demonstrating ring allreduce and ring allgather techniques . GitHub repository , 2024 . URL https://github.com/baidu-research/baidu-allreduce

    work page 2024

  5. [5]

    and Hoefler, T

    Ben-Nun, T. and Hoefler, T. Demystifying parallel and distributed deep learning: An in-depth concurrency analysis. ACM Computing Surveys (CSUR), 52 0 (4): 0 1--43, 2019

    work page 2019

  6. [6]

    Piqa: Reasoning about physical commonsense in natural language

    Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp.\ 7432--7439, 2020

    2020

  7. [7]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. 2020

    work page 2020

  8. [8]

    Shortcut-connected expert parallelism for accelerating mixture-of-experts, 2024

    Cai, W., Jiang, J., Qin, L., Cui, J., Kim, S., and Huang, J. Shortcut-connected expert parallelism for accelerating mixture-of-experts, 2024. URL https://arxiv.org/abs/2404.05019

    arXiv 2024

Show all 74 references
  1. [9]

    Flux: Fast software-based communication overlap on gpus through kernel fusion, 2024

    Chang, L.-W., Bao, W., Hou, Q., Jiang, C., Zheng, N., Zhong, Y., Zhang, X., Song, Z., Yao, C., Jiang, Z., Lin, H., Jin, X., and Liu, X. Flux: Fast software-based communication overlap on gpus through kernel fusion, 2024. URL https://arxiv.org/abs/2406.06858

    2024 arXiv

  2. [10]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  3. [11]

    Flashattention-2: Faster attention with better parallelism and work partitioning

    Dao, T. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691, 2023

    2023 arXiv

  4. [12]

    Large scale distributed deep networks

    Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P., Yang, K., et al. Large scale distributed deep networks. Advances in neural information processing systems, 25, 2012

    2012

  5. [13]

    Llm.int8(): 8-bit matrix multiplication for transformers at scale, 2022

    Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. Llm.int8(): 8-bit matrix multiplication for transformers at scale, 2022. URL https://arxiv.org/abs/2208.07339

    2022 arXiv

  6. [14]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  7. [15]

    Gloo: Collective communications library with various primitives for multi-machine training

    Facebook. Gloo: Collective communications library with various primitives for multi-machine training. https://github.com/facebookincubator/gloo, 2024. Accessed: 2024-11-23

    2024

  8. [16]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity

    Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23 0 (120): 0 1--39, 2022

    2022

  9. [17]

    Pipedream: Fast and efficient pipeline parallel dnn training

    Harlap, A., Narayanan, D., Phanishayee, A., Seshadri, V., Devanur, N., Ganger, G., and Gibbons, P. Pipedream: Fast and efficient pipeline parallel dnn training. arXiv preprint arXiv:1806.03377, 2018

    2018 arXiv

  10. [18]

    Distributed gemm, 2024

    Hassani, A., Isaev, M., McDonald, N., Ren, J., Thakkar, V., Wu, H., and Shi, H. Distributed gemm, 2024. URL https://blog.shi-labs.com/distributed-gemm-88be6a481e2b. Accessed: 2024-12-04

    2024

  11. [19]

    G., Slaughter, E., Surana, P., Hwu, W.-m., Gropp, W., and Aiken, A

    Hidayetoglu, M., de Gonzalo, S. G., Slaughter, E., Surana, P., Hwu, W.-m., Gropp, W., and Aiken, A. Hiccl: A hierarchical collective communication library. arXiv preprint arXiv:2408.05962, 2024

    2024 arXiv

  12. [20]

    V., Wu, Y., et al

    Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, D., Chen, M., Lee, H., Ngiam, J., Le, Q. V., Wu, Y., et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019

    2019

  13. [21]

    IEEE Standard for Floating-Point Arithmetic

    IEEE . IEEE Standard for Floating-Point Arithmetic . Technical Report IEEE 754-1985 , Institute of Electrical and Electronics Engineers , New York, NY , 1985. URL https://standards.ieee.org/ieee/754/6210/

    1985

  14. [22]

    A., Tanaka, M., Zhang, C., Zhang, M., Song, S

    Jacobs, S. A., Tanaka, M., Zhang, C., Zhang, M., Song, S. L., Rajbhandari, S., and He, Y. Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models. arXiv preprint arXiv:2309.14509, 2023

    2023 arXiv

  15. [23]

    Highly scalable deep learning training system with mixed-precision: Training imagenet in four minutes

    Jia, X., Song, S., He, W., Wang, Y., Rong, H., Zhou, F., Xie, L., Guo, Z., Yang, Y., Yu, L., et al. Highly scalable deep learning training system with mixed-precision: Training imagenet in four minutes. arXiv preprint arXiv:1807.11205, 2018

    2018 arXiv

  16. [24]

    Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. l., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024

    2024 arXiv

  17. [25]

    J., Henry, R., Fahim, R., and Awadalla, H

    Kim, Y. J., Henry, R., Fahim, R., and Awadalla, H. H. Who says elephants can't run: Bringing large scale moe models into cloud scale production. arXiv preprint arXiv:2211.10017, 2022

    2022 arXiv

  18. [26]

    A., Casper, J., Lym, S., McAfee, L., Andersch, M., Shoeybi, M., and Catanzaro, B

    Korthikanti, V. A., Casper, J., Lym, S., McAfee, L., Andersch, M., Shoeybi, M., and Catanzaro, B. Reducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5: 0 341--353, 2023

    2023

  19. [27]

    H., Gonzalez, J

    Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023

    2023

  20. [28]

    Fptq: Fine-grained post-training quantization for large language models

    Li, Q., Zhang, Y., Li, L., Yao, P., Zhang, B., Chu, X., Sun, Y., Du, L., and Xie, Y. Fptq: Fine-grained post-training quantization for large language models. arXiv preprint arXiv:2308.15987, 2023

    2023 arXiv

  21. [29]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model

    Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., Dengr, C., Ruan, C., Dai, D., Guo, D., et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, 2024

    2024 arXiv

  22. [30]

    and Abbeel, P

    Liu, H. and Abbeel, P. Blockwise parallel transformers for large context models. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36, pp.\ 8828--8844. Curran Associates, Inc., 2023

    2023

  23. [31]

    Ring attention with blockwise transformers for near-infinite context

    Liu, H., Zaharia, M., and Abbeel, P. Ring attention with blockwise transformers for near-infinite context. arXiv preprint arXiv:2310.01889, 2023

    2023 arXiv

  24. [32]

    Lmdeploy: A toolkit for compressing, deploying, and serving llm

    LMDeploy. Lmdeploy: A toolkit for compressing, deploying, and serving llm. https://github.com/InternLM/lmdeploy, 2023

    2023

  25. [33]

    Quantized distributed training of large models with convergence guarantees, 2023

    Markov, I., Vladu, A., Guo, Q., and Alistarh, D. Quantized distributed training of large models with convergence guarantees, 2023. URL https://arxiv.org/abs/2302.02390

    2023 arXiv

  26. [34]

    Pointer sentinel mixture models

    Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016

    2016 arXiv

  27. [35]

    Fp8 formats for deep learning

    Micikevicius, P., Stosic, D., Burgess, N., Cornea, M., Dubey, P., Grisenthwaite, R., Ha, S., Heinecke, A., Judd, P., Kamalu, J., et al. Fp8 formats for deep learning. arXiv preprint arXiv:2209.05433, 2022

    2022 arXiv

  28. [36]

    DeepSpeed: Deep learning optimization library

    Microsoft . DeepSpeed: Deep learning optimization library . GitHub repository , 2024 . URL https://github.com/microsoft/DeepSpeed

    2024

  29. [37]

    Microsoft collective communication library (msccl)

    Microsoft. Microsoft collective communication library (msccl). https://github.com/microsoft/msccl, 2024. Accessed: 2024-11-23

    2024

  30. [38]

    Massively distributed sgd: Imagenet/resnet-50 training in a flash

    Mikami, H., Suganuma, H., Tanaka, Y., Kageyama, Y., et al. Massively distributed sgd: Imagenet/resnet-50 training in a flash. arXiv preprint arXiv:1811.05233, 2018

    2018 arXiv

  31. [39]

    Efficient large-scale language model training on gpu clusters using megatron-lm

    Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Patwary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Confere...

    2021

  32. [40]

    Hetumoe: An efficient trillion-scale mixture-of-expert distributed training system, 2022

    Nie, X., Zhao, P., Miao, X., Zhao, T., and Cui, B. Hetumoe: An efficient trillion-scale mixture-of-expert distributed training system, 2022. URL https://arxiv.org/abs/2203.14685

    2022 arXiv

  33. [41]

    NCCL Tests

    NVIDIA . NCCL Tests . https://github.com/NVIDIA/nccl-tests , 2016-2024 . Accessed: 2024-12-06

    2016

  34. [42]

    Massively scale your deep learning training with nccl 2.4

    NVIDIA. Massively scale your deep learning training with nccl 2.4. https://developer.nvidia.com/blog/massively-scale-deep-learning-training-nccl-2-4/, 2019. Accessed: 2024-11-23

    2019

  35. [43]

    NVIDIA Scalable Hierarchical Aggregation and Reduction Protocol (SHARP) v2.6.1 Release Notes

    NVIDIA . NVIDIA Scalable Hierarchical Aggregation and Reduction Protocol (SHARP) v2.6.1 Release Notes . Technical Report Revision 2.6.1 , NVIDIA , 2023. URL https://docs.nvidia.com/networking/display/sharpv261/release+notes . Last updated on May 23, 2023

    2023

  36. [44]

    TensorRT-LLM

    NVIDIA . TensorRT-LLM . GitHub repository , 2023 . URL https://github.com/NVIDIA/TensorRT-LLM

    2023

  37. [45]

    Megatron-LM: Ongoing research training transformer models at scale

    NVIDIA . Megatron-LM: Ongoing research training transformer models at scale . GitHub repository , 2024 a . URL https://github.com/NVIDIA/Megatron-LM

    2024

  38. [46]

    NVIDIA Nsight Systems

    NVIDIA . NVIDIA Nsight Systems . Web Page , 2024 b . URL https://developer.nvidia.com/nsight-systems

    2024

  39. [47]

    Collective operations

    NVIDIA. Collective operations. https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/collectives.html, 2024 a . Accessed: 2024-11-23

    2024

  40. [48]

    Context parallelism overview

    NVIDIA. Context parallelism overview. https://docs.nvidia.com/megatron-core/developer-guide/latest/api-guide/context_parallel.html, 2024 b . Accessed: 2024-11-23

    2024

  41. [49]

    Nvidia l40: Delivering unprecedented visual computing performance for the data center

    NVIDIA. Nvidia l40: Delivering unprecedented visual computing performance for the data center. https://images.nvidia.cn/content/Solutions/data-center/vgpu-L40-datasheet.pdf, 2024 c . Accessed: 2024-11-23

    2024

  42. [50]

    Optimized primitives for collective multi-gpu communication

    NVIDIA. Optimized primitives for collective multi-gpu communication. https://github.com/NVIDIA/nccl, 2024 d . Accessed: 2024-11-23

    2024

  43. [51]

    Nvlink & nvswitch for advanced multi-gpu communication

    NVIDIA. Nvlink & nvswitch for advanced multi-gpu communication. https://www.nvidia.com/en-us/data-center/nvlink/, 2024 e . Accessed: 2024-11-26

    2024

  44. [52]

    Cuda templates for linear algebra subroutines, 2024 f

    NVIDIA. Cuda templates for linear algebra subroutines, 2024 f . URL https://github.com/NVIDIA/cutlass. Accessed: 2024-12-04

    2024

  45. [53]

    Peer Device Memory Access

    NVIDIA Corporation . Peer Device Memory Access . Technical report, NVIDIA , 2024 . URL https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__PEER.html . Accessed: 2024-12-04

    2024

  46. [54]

    Disco: Distributed inference with sparse communications

    Qin, M., Sun, C., Hofmann, J., and Vucinic, D. Disco: Distributed inference with sparse communications. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp.\ 2432--2440, 2024

    2024

  47. [55]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21 0 (140): 0 1--67, 2020

    2020

  48. [56]

    Zero: Memory optimizations toward training trillion parameter models

    Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp.\ 1--16. IEEE, 2020

    2020

  49. [57]

    Y., Awan, A

    Rajbhandari, S., Li, C., Yao, Z., Zhang, M., Aminabadi, R. Y., Awan, A. A., Rasley, J., and He, Y. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale. In International conference on machine learning, pp.\ 18332--18346. PMLR, 2022

    2022

  50. [58]

    D., Zhao, R., More, A., Hall, M., Khodamoradi, A., Deng, S., Choudhary, D., Cornea, M., Dellinger, E., Denolf, K., et al

    Rouhani, B. D., Zhao, R., More, A., Hall, M., Khodamoradi, A., Deng, S., Choudhary, D., Cornea, M., Dellinger, E., Denolf, K., et al. Microscaling data formats for deep learning. arXiv preprint arXiv:2310.10537, 2023

    2023 arXiv

  51. [59]

    L., Bhagavatula, C., and Choi, Y

    Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64 0 (9): 0 99--106, 2021

    2021

  52. [60]

    Sanders, P., Speck, J., and Tr \"a ff, J. L. Two-tree algorithms for full bandwidth broadcast, reduction and scan. Parallel Computing, 35 0 (12): 0 581--594, 2009

    2009

  53. [61]

    and Balso, M

    Sergeev, A. and Balso, M. D. Horovod: fast and easy distributed deep learning in TensorFlow . arXiv preprint arXiv:1802.05799, 2018

    2018 arXiv

  54. [62]

    SGLang: A Fast Serving Framework for Large Language Models and Vision Language Models

    sgl-project . SGLang: A Fast Serving Framework for Large Language Models and Vision Language Models . GitHub repository , 2024. URL https://github.com/sgl-project/sglang . Accessed: 2024-12-01

    2024

  55. [63]

    Megatron-lm: Training multi-billion parameter language models using model parallelism

    Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019

    1909 arXiv

  56. [64]

    Qwen1.5-moe: Matching 7b model performance with 1/3 activated parameters", February 2024

    Team, Q. Qwen1.5-moe: Matching 7b model performance with 1/3 activated parameters", February 2024. URL https://qwenlm.github.io/blog/qwen-moe/

    2024

  57. [65]

    A., Holmes, C., Rajbhandari, S., Ruwase, O., Yan, F., Yang, L., and He, Y

    Wang, G., Qin, H., Jacobs, S. A., Holmes, C., Rajbhandari, S., Ruwase, O., Yan, F., Yang, L., and He, Y. Zero++: Extremely efficient collective communication for giant model training. arXiv preprint arXiv:2306.10209, 2023

    2023 arXiv

  58. [66]

    Domino: Eliminating communication in llm training via generic tensor slicing and overlapping, 2024

    Wang, G., Zhang, C., Shen, Z., Li, A., and Ruwase, O. Domino: Eliminating communication in llm training via generic tensor slicing and overlapping, 2024. URL https://arxiv.org/abs/2409.15241

    2024 arXiv

  59. [67]

    Smoothquant: Accurate and efficient post-training quantization for large language models, 2024

    Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models, 2024. URL https://arxiv.org/abs/2211.10438

    2024 arXiv

  60. [68]

    Image classification at supercomputer scale

    Ying, C., Kumar, S., Chen, D., Wang, T., and Cheng, Y. Image classification at supercomputer scale. arXiv preprint arXiv:1811.06992, 2018

    2018 arXiv

  61. [69]

    The super weight in large language models

    Yu, M., Wang, D., Shan, Q., and Wan, A. The super weight in large language models. arXiv preprint arXiv:2411.07191, 2024

    2024 arXiv

  62. [70]

    Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    1905 arXiv

  63. [71]

    Zhang, T., Lin, Z., Yang, G., and Sa, C. D. Qpytorch: A low-precision arithmetic simulation framework, 2019

    2019

  64. [72]

    Pytorch fsdp: experiences on scaling fully sharded data parallel

    Zhao, Y., Gu, A., Varma, R., Luo, L., Huang, C.-C., Xu, M., Wright, L., Shojanazeri, H., Ott, M., Shleifer, S., et al. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.11277, 2023

    2023 arXiv

  65. [73]

    Nanoflow: Towards optimal large language model serving throughput, 2024

    Zhu, K., Zhao, Y., Zhao, L., Zuo, G., Gu, Y., Xie, D., Gao, Y., Xu, Q., Tang, T., Ye, Z., Kamahori, K., Lin, C.-Y., Wang, S., Krishnamurthy, A., and Kasikci, B. Nanoflow: Towards optimal large language model serving throughput, 2024. URL https://arxiv.org/abs/2408.12757

    2024 arXiv

  66. [74]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.