Pith. sign in

REVIEW 3 major objections 4 minor 34 references

EQuARX: Efficient Quantized AllReduce in XLA for Distributed Machine Learning Acceleration

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read EQuARX, a native XLA int8 AllReduce on TPUs, cuts collective time to about 55 percent of BF16 and speeds Gemma 3 prefill by up to 1.28 times.

desk verdict Solid TPU compiler engineering with internally consistent speedups, but the paper's own significance test contradicts the 'negligible quality impact' claim that anchors the trade-off. read the letter →

arxiv 2506.17615 v1 pith:HO75WA3B submitted 2025-06-21 cs.LG

Ibrahim Ahmed , Clemens Schaefer , Gil Tabak , Denis Vnukov , Zenong Zhang , Felix chern , Anatoliy Yevtushenko , Andy Davis This is my paper · ORCID
classification cs.LG
keywords quantizedAllReduceblock-wisequantizationXLATPUreduce-scatterdeeppipeliningLLMinferenceint8
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper seeks to establish that AllReduce, the distributed collective that sums tensors across devices, can be quantized to int8 on TPUs without the overflow and error accumulation that make naive low-precision reduction dangerous. The authors present EQuARX, an XLA-native implementation that sends block-wise dynamically quantized int8 shards over the interconnect, dequantizes every received shard to FP32 before adding, and hides the extra compute by deep pipelining at the microshard level. They report roughly 55 percent of baseline BF16 AllReduce runtime (about a 1.8x speedup), close to the ideal gain from halving transferred bytes, and prefill speedups of up to 1.28x for Gemma 3 27B and 1.1x for 12B with small to negligible accuracy changes. The paper's contribution is a practical co-design of quantization and the ring collective, not a formal accuracy guarantee.

What carries the argument

The load-bearing mechanism is dynamic block-wise symmetric int8 quantization fused into a ring-based reduce-scatter. For each shard, EQuARX computes 8x128 scale factors by an absolute-value maximum reduction over VPU-friendly 2D chunks (one FP32 scale per 64 entries for the default block size), then quantizes in two phases: Qp1 computes the metadata and Qp2 scales the data. Receivers dequantize each incoming microshard to FP32 before adding, so all arithmetic on partial sums stays in high precision and only the wire bytes are int8. Deep pipelining splits shards into microshards, sends metadata first, and overlaps dequantize-add of one microshard with reception of the next; the full-loop ring is bandwidth-optimal (N-1 hops) and the semi-loop ring uses both directions to halve the number of quantization hops at some bandwidth cost.

What would settle it

Run EQuARX int8 with block size 64 on real prefill activations recorded from a third model family and compare the AllReduce output to the BF16 baseline: if the mean squared error rises materially above the 0.0014 reported for standard-normal tensors, or if downstream accuracy drops beyond the HellaSwag and AGIEval margins on 27B, the scheme's accuracy claim fails.

Watch

Extended reading notes

Core claim

On its own terms, EQuARX claims that the reduce-scatter half of AllReduce can run in int8 while preserving the quality of the sum, as long as each transmitted shard is dynamically block-wise quantized and every received shard is dequantized to FP32 before the partial-sum addition. Shards are divided into 8x128 chunks that match the TPU's vector registers, one FP32 scale factor is computed per chunk (block size 64 in the example), and metadata is sent before the data so receivers can dequantize immediately. Microshard pipelining overlaps quantization and dequantization with network transfers, removing most of the compute overhead. With both reduce-scatter and all-gather quantized, full-loop EQuARX reaches about 55 percent of BF16 AllReduce runtime (a 1.8x speedup), within 10 percent of the ideal for halving data volume, and the semi-loop variant trades some throughput for lower accumulated error.

Load-bearing premise

The accuracy claims rest on the untested-in-general assumption that one FP32 scale per 64 values keeps quantized inter-device sums close enough to the BF16 result for arbitrary LLM activation distributions, since the paper gives no formal error bound and measures error only on N(0,1) tensors and two Gemma models.

Editorial extensions

If this is right

  • For tensors above roughly 2 MiB, EQuARX reduces AllReduce time to about 55 percent of the BF16 baseline—a 1.8x speedup—while tensors below that size remain latency-bound and gain little.
  • Quantizing both the reduce-scatter and all-gather stages yields the largest speedup with a mean squared error of 0.0014 on standard-normal data, about two orders of magnitude lower than a naive FP8 AllReduce at comparable speed.
  • The semi-loop variant lowers accumulated error (MSE 0.001 versus 0.0014) at a reduced 1.6x speedup, and quantizing only the all-gather gives the smallest error (0.0003) at 1.3x, giving users an explicit accuracy-throughput knob.
  • On Gemma 3, EQuARX speeds the prefill stage by up to 1.28x for the 27B model and 1.1x for the 12B model, with most benchmark differences within noise; the two statistically significant 27B drops are HellaSwag and AGIEval.
  • Because the decode-stage AllReduce is latency-bound, EQuARX is applied only to prefill, so its gains are specific to bandwidth-bound collectives with large tensors.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same microshard-pipelined, VPU-shaped block quantization could be applied to other bandwidth-bound collectives, such as the reduce-scatter used in data-parallel training, but training convergence under repeated quantized reductions is not demonstrated in this paper.
  • The block size of 64 is tied to TPU register geometry rather than to activation statistics, so an adaptive per-layer or per-tensor block size is a natural testable extension that could reclaim accuracy where HellaSwag and AGIEval showed significant drops.
  • If future TPU generations raise interconnect bandwidth faster than vector-unit throughput, the quantization compute that is currently hidden behind communication may become the bottleneck, requiring deeper pipelining or cheaper metadata computation.
  • The near-roofline efficiency of the scheme implies most of the gain comes from halving wire bytes, so comparable speedups should appear on any bandwidth-bound torus topology, not only the 2x2 and 4x4 configurations tested.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents EQuARX, a native XLA implementation of block-wise dynamic int8 quantization for AllReduce on TPUs. It contributes deep pipelining of quantization/dequantization with communication, a VPU-friendly 8x128 block-wise scaling scheme, and two ring variants (full-loop and semi-loop) with options to quantize the reduce-scatter stage, the all-gather stage, or both. Microbenchmarks on TPU v5e report that int8 EQuARX reduces AllReduce execution time to about 55% of a BF16 baseline for large tensors, close to the roofline for halving transferred data, with measured speedups of about 1.8x and an MSE of 0.0014 on a synthetic N(0,1) tensor. On Gemma 3, the paper reports 1.1x prefill speedup for the 12B model and up to 1.28x for the 27B model, while claiming small to negligible quality impact. The speedup evidence is internally consistent, but the quality claim is weakened by the paper's own statistically significant drops on HellaSwag and AGIEval for the 27B model.

Significance. If the results hold after revision, EQuARX is a practically valuable contribution: it is a compiler-integrated, topology-aware collective that converts part of the AllReduce bandwidth win into measured end-to-end prefill speedups on production-scale models, with transparent comparison to a roofline. The paper reports measured outcomes rather than fitting constants, uses standard benchmarks, and includes statistical significance checks; the microbenchmark speedup claims are plausible and well explained by the pipelining design. The main weakness is the accuracy side of the trade-off: the synthetic error measurement and the statistically significant task regressions do not support the blanket "small to negligible" quality claim. With a recalibrated quality assessment and a stronger error characterization, the contribution would be solid.

major comments (3)
  1. [V, Table I; Abstract; Conclusion] The qualitative claim that EQuARX has 'small to negligible' quality impact is contradicted by the paper's own significance results. For Gemma 3 27B, HellaSwag drops from 83.26 to 81.46 (-1.80 points) and AGIEval from 67.44 to 64.86 (-2.58 points), and Section V states that both are statistically significant at p=0.05. These are task-dependent, statistically significant regressions on the two largest evaluation sets (N=10042 and N=2340), so the blanket characterization in the abstract and conclusion is too strong. Because quality degradation is the cost side of the speedup/accuracy trade-off, the central claim as stated needs either softened wording or additional evidence, such as per-task confidence intervals, an aggregate quality analysis, or results on additional model sizes and sharding configurations, showing that the drops are acceptable in context.
  2. [IV-A, Fig. 7] The microbenchmark error characterization rests on a single tensor sampled from N(0,1) on one topology, with no confidence intervals or repeated trials. The paper reports MSE 0.0014 for the fastest EQuARX flavor but provides no distributional evidence that quantization errors at actual AllReduce boundaries in Gemma prefill resemble this synthetic case, and no formal error bound is derived for the block-wise dynamic scheme. Since the end-to-end quality impact is the load-bearing cost of the method, the authors should measure error on realistic activation distributions from the evaluated models or provide an analytical error analysis, and report variability across trials.
  3. [V, Table I] The claim that 'EQuARX results in small to negligible accuracy drop across all tasks' is not supported by the table even setting statistical significance aside: the 12B model shows -2.80 on MBPP and -2.60 on MedQA with full-loop, and the 27B model shows -1.80 on MedQA and -2.58 on AGIEval. The authors do report that the smaller-n tasks are not statistically significant, but no confidence intervals are given for any metric, so the reader cannot assess the precision of the reported drops. Please report confidence intervals or standard errors for the accuracy metrics and avoid 'negligible' for point estimates of this size.
minor comments (4)
  1. [III-C] Please clarify the derivation of the bandwidth lower bounds. As written, the full-loop bound (N-1)D/(2NB) appears to differ from the standard ring reduce-scatter bound by a factor of two, and the precise role of B as per-direction bandwidth in the bidirectional semi-loop variant should be defined more carefully.
  2. [Fig. 7] The axis labels contain typos: 'Mean Sqaured Error' and 'basline' should be 'Mean Squared Error' and 'baseline'.
  3. [III-B] Minor wording issues: 'to to determine' and 'result in under utilizing' should be cleaned up.
  4. [V] The end-to-end evaluation covers only two model sizes and one sharding configuration each (4x4 for 27B, 2x2 for 12B). Please discuss the extent to which the reported quality and speedup results generalize to other topologies and model families.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: EQuARX's speedup and accuracy claims are measured against independent BF16 baselines and external benchmarks, with no fitted parameter renamed as a prediction.

full rationale

EQuARX's central claims are empirical rather than derived from fitted constants. Fig. 6 reports measured normalized AllReduce time for int8 EQuARX versus a BF16 baseline, and Fig. 7 plots measured speedup against measured MSE on tensors sampled from N(0,1). The 'ideal' 2x-compression reference is a roofline (halving transferred bytes with no quantization compute and no metadata), not a quantity fitted from EQuARX's own parameters, so comparing against it is not circular. The block size of 64 is a design choice, but the paper does not select it to force a target result; it exposes full-loop/semi-loop and RS/AG-only variants and reports the resulting trade-offs. End-to-end quality numbers are measured on Gemma 3 12B/27B against a BF16 AllReduce baseline and external benchmarks (HellaSwag, AGIEval, etc.), with no fitted constants entering the accuracy computation. The paper's own significance statement ('for 27B the HellaSwag and AGIEval were (N=10042 and 2340, respectively)') undercuts the abstract's 'small to negligible' phrasing, but that is an overgeneralization in the accuracy claim, not a circular derivation: the accuracy numbers are not defined in terms of the conclusion. No load-bearing self-citation chain appears; citations to prior collective-communication and quantization work are contextual. Hence no step reduces by construction to its inputs.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. Its design choices are tunable parameters and implementation details, not new postulates. The main burden is the empirical assumption that block-wise int8 error is acceptable, which is tested on a narrow distribution and two models.

free parameters (3)
  • block size (entries per scale factor) = 64 entries (8x128 block with m=1)
    The paper sets the scale-factor granularity to 8x128 TPU VPU vectors, which gives a block size of 64 entries per scale factor for the shown example. This choice is driven by hardware efficiency and is not derived from an error model.
  • microshard count u = 2 in the example
    The pipelining example uses u=2 microshards; the optimal u is a tunable performance knob, not derived from first principles.
  • minishard count m = 1 in the example
    m controls the block size and metadata volume; the paper says users can tune m and u for quality and performance trade-offs.
assumptions (3)
  • domain assumption Symmetric int8 quantization with per-block scale factors preserves AllReduce accuracy for the tested distributions.
    The method assumes that block-wise scaling and dequantization before every addition keeps accumulated error acceptable; the paper validates this empirically only on N(0,1) tensors and on two Gemma models.
  • standard math Ring-based reduce-scatter followed by all-gather is bandwidth-optimal and remains the right algorithmic skeleton.
    The paper cites Chan et al. for bandwidth-optimality of ring collectives on torus topologies.
  • domain assumption TPU VPU operations on 8x128 chunks are representative of the cost model for quantization overhead.
    The design assumes that avoiding reshapes and using 8x128 native vectors is the right way to minimize quantization overhead on TPU v5e.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EQuARX: Efficient Quantized AllReduce in XLA for Distributed Machine Learning Acceleration." pith.science (2026). https://pith.science/paper/HO75WA3B

@misc{pith2026250617615,
  author       = {Pith},
  title        = {Pith review of: EQuARX: Efficient Quantized AllReduce in XLA for Distributed Machine Learning Acceleration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HO75WA3B}},
  note         = {Machine review of arXiv:2506.17615}
}
read the original abstract

While Large Language Models (LLMs) have become highly influential, their enormous scale presents significant deployment challenges. Efficiently serving these models typically requires distributing them across numerous accelerator devices, which introduces substantial performance overhead from inter-device communication (collectives). While model quantization has been widely adopted to reduce the memory and compute requirements of LLM weights and activations with minimal quality impact, applying quantization directly to collectives like AllReduce is inherently difficult due to the inter-device summation involved, which can lead to numerical instability or significant error accumulation. In this work, we present a native dynamic block-wise efficient quantized AllReduce within the XLA compiler for TPUs (EQuARX). By using TPU-friendly quantization and deep pipelining of communication and compute, EQuARX with int8 precision achieves a 1.8X speedup over baseline BF16 AllReduce across various network topologies. Furthermore, EQuARX accelerates the prefill stage of Gemma 3 27B by 1.25X and Gemma 3 12B by 1.1X, respectively, with small to negligible impact on quality.

Figures

Figures reproduced from arXiv: 2506.17615 by the authors.

Figure 1
Figure 1. Three iterations of a ring-based reduce-scatter algo [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Baseline and pipelined execution timeline of a single [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 4
Figure 4. Full-loop adder chain to calculate shard 0 result. [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Semi-loop adder chain to calculate shard 0 result. [PITH_FULL_IMAGE:figures/full_fig_p003_5.png]
Figure 6
Figure 6. Figure 6: Speedup of Full-loop EQuARX across different tensor [PITH_FULL_IMAGE:figures/full_fig_p004_6.png]
Figure 7
Figure 7. Figure 7: Speedup vs error of EQuARX (with a block size of [PITH_FULL_IMAGE:figures/full_fig_p004_7.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 8 canonical work pages

  1. [1]

    Program synthesis with large language models,

    J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le et al., “Program synthesis with large language models,” arXiv preprint arXiv:2108.07732 , 2021

    arXiv 2021

  2. [2]

    Language models are few-shot learners,

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Am...

    2020

  3. [3]

    Collective Communication: Theory, Practice, and Experience: Research Articles,

    E. Chan, M. Heimlich, A. Purkayastha, and R. van de Geijn, “Collective Communication: Theory, Practice, and Experience: Research Articles,” Concurr. Comput. : Pract. Exper. , vol. 19, no. 13, p. 1749–1783, Sep. 2007

    work page 2007

  4. [4]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Her...

    2021

  5. [5]

    Boolq: Exploring the surprising difficulty of natural yes/no questions,

    C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova, “Boolq: Exploring the surprising difficulty of natural yes/no questions,” arXiv preprint arXiv:1905.10044 , 2019

    arXiv 1905

  6. [6]

    A Survey of Quantization Methods for Efficient Neural Network Inference,

    A. Gholami, S. Kim, Z. Dong, Z. Yao, M. W. Mahoney, and K. Keutzer, “A Survey of Quantization Methods for Efficient Neural Network Inference,” 2021. [Online]. Available: https://arxiv.org/abs/2103.13630

    arXiv 2021

  7. [7]

    Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour,

    P. Goyal, P. Doll ´ar, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y . Jia, and K. He, “Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour,” 2018. [Online]. Available: https://arxiv.org/abs/1706.02677

    arXiv 2018

  8. [8]

    Measuring massive multitask language understanding,

    D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,” arXiv preprint arXiv:2009.03300 , 2020

    arXiv 2009

Show all 34 references
  1. [9]

    gZCCL: Compression-Accelerated Collective Communication Framework for GPU Clusters,

    J. Huang, S. Di, X. Yu, Y . Zhai, J. Liu, Y . Huang, K. Raffenetti, H. Zhou, K. Zhao, X. Lu, Z. Chen, F. Cappello, Y . Guo, and R. Thakur, “gZCCL: Compression-Accelerated Collective Communication Framework for GPU Clusters,” in Proceedings of the 38th ACM International Confere...

    2024

  2. [10]

    Huang, Y

    Y . Huang, Y . Cheng, A. Bapna, O. Firat, M. X. Chen, D. Chen, H. Lee, J. Ngiam, Q. V . Le, Y . Wu, and Z. Chen, GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism . Red Hook, NY , USA: Curran Associates Inc., 2019

    2019

  3. [11]

    Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference,

    B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko, “Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference,”

  4. [12]

    Optimal Bucket Algorithms for Large MPI Collectives on Torus Interconnects,

    N. Jain and Y . Sabharwal, “Optimal Bucket Algorithms for Large MPI Collectives on Torus Interconnects,” in Proceedings of the 24th ACM International Conference on Supercomputing, ser. ICS ’10. New York, NY , USA: Association for Computing Machinery, 2010, p. 27–36. [Online]. ...

  5. [13]

    SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Training,

    J. Jia, C. Xie, H. Lu, D. Wang, H. Feng, C. Zhang, B. Sun, H. Lin, Z. Zhang, X. Liu et al. , “SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Training,” arXiv preprint arXiv:2410.15526, 2024

    arXiv

  6. [14]

    What disease does this patient have? a large-scale open domain question answering dataset from medical exams,

    D. Jin, E. Pan, N. Oufattole, W.-H. Weng, H. Fang, and P. Szolovits, “What disease does this patient have? a large-scale open domain question answering dataset from medical exams,” Applied Sciences , vol. 11, no. 14, p. 6421, 2021

    2021

  7. [15]

    In-Datacenter Performance Analysis of a Tensor Processing Unit,

    N. P. Joppi, C. Young, N. Patil, D. Patterson, G. Agrawal, R. Bajwa, S. Bates, S. Bhatia, N. Boden, A. Borchers, R. Boyle, P.-l. Cantin, C. Chao, C. Clark, J. Coriell, M. Daley, M. Dau, J. Dean, B. Gelb, T. V . Ghaemmaghami, R. Gottipati, W. Gulland, R. Hagmann, C. R. Ho, D. H...

    2017

  8. [16]

    Triviaqa: A large scale distantly supervised challenge dataset for reading comprehen- sion,

    M. Joshi, E. Choi, D. S. Weld, and L. Zettlemoyer, “Triviaqa: A large scale distantly supervised challenge dataset for reading comprehen- sion,” in Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics . Vancouver, Canada: Association for Comp...

    2017

  9. [17]

    TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings,

    N. Jouppi, G. Kurian, S. Li, P. Ma, R. Nagarajan, L. Nai, N. Patil, S. Subramanian, A. Swing, B. Towles, C. Young, X. Zhou, Z. Zhou, and D. A. Patterson, “TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings,” ser. ISCA ’2...

  10. [18]

    A domain-specific Supercomputer for Training Deep Neural Networks,

    N. P. Jouppi, D. H. Yoon, G. Kurian, S. Li, N. Patil, J. Laudon, C. Young, and D. Patterson, “A domain-specific Supercomputer for Training Deep Neural Networks,” Commun. ACM , vol. 63, no. 7, p. 67–78, Jun. 2020. [Online]. Available: https://doi.org/10.1145/3360307

    2020 doi

  11. [19]

    Scaling Laws for Neural Language Models,

    J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling Laws for Neural Language Models,” 2020. [Online]. Available: https://arxiv.org/abs/2001.08361

    2020 arXiv

  12. [20]

    Evaluating Modern GPU Interconnect: PCIe, NVLink, NV- SLI, NVSwitch and GPUDirect,

    A. Li, S. L. Song, J. Chen, J. Li, X. Liu, N. R. Tallent, and K. J. Barker, “Evaluating Modern GPU Interconnect: PCIe, NVLink, NV- SLI, NVSwitch and GPUDirect,” IEEE Transactions on Parallel and Distributed Systems, vol. 31, no. 1, pp. 94–110, 2020

    2020

  13. [21]

    Quantized Distributed Training of Large Models with Convergence Guarantees,

    I. Markov, A. Vladu, Q. Guo, and D. Alistarh, “Quantized Distributed Training of Large Models with Convergence Guarantees,” in Interna- tional Conference on Machine Learning . PMLR, 2023, pp. 24 020– 24 044

    2023

  14. [22]

    PipeDream: Generalized Pipeline Parallelism for DNN Training,

    D. Narayanan, A. Harlap, A. Phanishayee, V . Seshadri, N. R. Devanur, G. R. Ganger, P. B. Gibbons, and M. Zaharia, “PipeDream: Generalized Pipeline Parallelism for DNN Training,” in Proceedings of the 27th ACM Symposium on Operating Systems Principles , ser. SOSP ’19. New York...

    2019

  15. [23]

    Efficiently Scaling Transformer Inference,

    R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, A. Levskaya, J. Heek, K. Xiao, S. Agrawal, and J. Dean, “Efficiently Scaling Transformer Inference,” 2022. [Online]. Available: https://arxiv.org/abs/2211.05102

    2022 arXiv

  16. [24]

    Gpqa: A graduate-level google-proof q&a benchmark,

    D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y . Pang, J. Dirani, J. Michael, and S. R. Bowman, “Gpqa: A graduate-level google-proof q&a benchmark,” in First Conference on Language Modeling , 2024

    2024

  17. [25]

    Winogrande: An adversarial winograd schema challenge at scale,

    K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y . Choi, “Winogrande: An adversarial winograd schema challenge at scale,” Communications of the ACM , vol. 64, no. 9, pp. 99–106, 2021

    2021

  18. [26]

    Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism,

    M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro, “Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism,” 2020. [Online]. Available: https://arxiv.org/abs/1909.08053

    2020 arXiv

  19. [27]

    M. Snir, S. W. Otto, D. W. Walker, J. Dongarra, and S. Huss-Lederman, MPI: The Complete Reference . Cambridge, MA, USA: MIT Press, 1995

    1995

  20. [28]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Proceedings of the 31st International Conference on Neural Information Processing Systems, ser. NIPS’17. Red Hook, NY , USA: Curran Associates ...

    2017

  21. [29]

    Zero++: Extremely Efficient Col- lective Communication for Giant Model Training,

    G. Wang, H. Qin, S. A. Jacobs, C. Holmes, S. Rajbhandari, O. Ruwase, F. Yan, L. Yang, and Y . He, “Zero++: Extremely Efficient Col- lective Communication for Giant Model Training,” arXiv preprint arXiv:2306.10209, 2023

    2023 arXiv

  22. [30]

    Integer Quantization for Deep Learning Inference: Principles and Empirical evaluation,

    H. Wu, P. Judd, X. Zhang, M. Isaev, and P. Micikevicius, “Integer Quantization for Deep Learning Inference: Principles and Empirical evaluation,” arXiv preprint arXiv:2004.09602 , 2020

    2004 arXiv

  23. [31]

    Hel- laswag: Can a machine really finish your sentence?

    R. Zellers, A. Holtzman, Y . Bisk, A. Farhadi, and Y . Choi, “Hel- laswag: Can a machine really finish your sentence?” arXiv preprint arXiv:1905.07830, 2019

    1905 arXiv

  24. [32]

    Agieval: A human-centric benchmark for evaluating foundation models,

    W. Zhong, R. Cui, Y . Guo, Y . Liang, S. Lu, Y . Wang, A. Saied, W. Chen, and N. Duan, “Agieval: A human-centric benchmark for evaluating foundation models,” arXiv preprint arXiv:2304.06364 , 2023. 6

    2023 arXiv

  25. [2017]

    Available: https://arxiv.org/abs/1712.05877

    [Online]. Available: https://arxiv.org/abs/1712.05877

    arXiv

  26. [2023]

    Available: https://doi.org/10.1145/3579371.3589350

    [Online]. Available: https://doi.org/10.1145/3579371.3589350

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.