Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

DynamiQ claims multi-hop gradient all-reduce can be compressed to about 5 bits per coordinate while keeping final LLM accuracy at 99.9% of the uncompressed baseline.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 03:05 UTC pith:GI7GZABL

load-bearing objection Solid systems contribution; the 5-bit near-baseline claim looks real in the tested settings, but the bit-allocation optimality argument has a gap worth fixing. the 3 major comments →

arxiv 2602.08923 v3 pith:GI7GZABL submitted 2026-02-09 cs.LG cs.DCcs.NI

DynamiQ: Accelerating Gradient Synchronization using Compressed Multi-hop All-reduce

Wenchen Han , Shay Vargaftik , Michael Mitzenmacher , Ran Ben Basat This is my paper
classification cs.LG cs.DCcs.NI
keywords gradient compressionall-reducequantizationdistributed traininglarge language modelsvariable bitwidthfused kernelstime-to-accuracy
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Distributed training of large models stalls on the network when workers must repeatedly aggregate gradients, and compression schemes built for single-hop parameter servers lose accuracy when sums are re-quantized at every hop. DynamiQ is designed around multi-hop topology: before the main all-reduce, workers exchange lightweight per-super-group statistics, use them to assign different bit widths to different gradient regions according to their magnitude, then reorder data so a fused decompress-accumulate-recompress kernel processes contiguous uniform-bitwidth streams. At an average of 5 bits per coordinate, it reports 2.5-3x lower quantization error than MXFP8, up to 34.2% better time-to-accuracy than the best prior baseline, and final accuracy within 0.1% of BF16 across BERT, LLaMA, and Gemma workloads. It is the only evaluated method that consistently reaches near-baseline accuracy while still accelerating training. If this holds, bandwidth-limited LLM training can be substantially faster without changing final model quality.

Core claim

On the paper's own terms, the discovery is that the accuracy cost of gradient compression in multi-hop all-reduce is not inevitable: it can be made negligible by allocating bits per super-group based on global squared norms, rather than applying a fixed format to every coordinate. The pipeline works because a small initial all-reduce of means and norm sums lets every worker agree on a bit allocation that stays fixed along the aggregation path, and because quantization is unbiased at each hop, with correlated rounding across workers and hierarchical scaling of group factors. The reported result is that with 5 bits per coordinate, DynamiQ keeps final accuracy at 99.9% of the BF16 baseline acro

What carries the argument

The carrying mechanism is the super-group: 256 consecutive gradient entries, subdivided into 16 groups of 16, sharing per-super-group mean and squared-norm metadata and a per-group scale. A lightweight all-reduce makes these statistics global, and a threshold rule converts squared norms into per-super-group bitwidths chosen from {2,4,8}; the thresholds are locked together by a per-bit benefit formula derived from a worst-case scalar analysis, namely each added bit reduces MSE by about 4x, yielding fixed ratios such as T_{1,2}=5/32 T_{2,4} and T_{2,4}=17/512 T_{4,8}. This lets a single scalar (the smallest threshold) select the whole allocation under any bandwidth budget. The computation is m

Load-bearing premise

The bit allocation assumes quantization error scales with each super-group's squared norm and that each added bit cuts worst-case MSE by roughly 4x; if real gradient blocks violate those relations, the chosen widths are no longer near-optimal.

What would settle it

Run DynamiQ's allocation on real LLM gradients and compare its vNMSE against an exhaustive search over all allocations with the same bit budget: if some other allocation achieves materially lower error at the same bandwidth, the threshold formula is not doing the claimed work. Similarly, measure the actual per-bit MSE reduction on real super-group distributions and check whether it is close to the assumed 4x.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Multi-hop gradient sync at ~5 bits per coordinate preserves final model accuracy within 0.1% of BF16 in the tested LLM workloads.
  • Training wall-clock time to a target quality improves by up to 34.2% over the best prior compression baseline and about 40.8% over BF16.
  • The advantage grows when the network is shared with other jobs, e.g., from 34.5% to 40.2% on LLaMA 1B MMLU, because the bandwidth savings are more valuable under contention.
  • A butterfly all-reduce topology reduces compression error relative to ring due to fewer re-quantization hops, extending the same bit budget to larger worker counts.
  • Because the fused kernel keeps memory traffic near that of fixed-format compression, the accuracy gains do not trade away compute efficiency.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same threshold-ratio logic could be applied to per-round adaptive budgets, letting a training run spend more bits during early high-variance rounds and fewer later; the paper does not explore this.
  • If the empirical 4x-per-bit MSE rule is close to right, the technique should also benefit federated averaging or other protocols where the same partial sums are repeatedly quantized; verifying that would require experiments outside the paper's LLM all-reduce setting.
  • The reported O(n^2) vs O(n^3) MSE upper bounds for butterfly vs ring suggest topology choice and bit allocation should be co-optimized; the paper stops at intuition, so a formal treatment remains open.
  • An adversarial extension would feed DynamiQ gradients whose entries are randomly permuted; if the accuracy gap to BF16 widens, spatial locality is doing much of the work, and the method would need reordering or different grouping to stay effective.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. DynamiQ is a gradient-compression framework for multi-hop all-reduce in distributed LLM training. It first gathers per-super-group statistics via a lightweight all-reduce, then assigns variable bitwidths based on a threshold heuristic derived from a worst-case per-bit benefit model. Each worker normalizes by the global super-group mean, reorders super-groups by bitwidth, and performs the main all-reduce with fused decompress-accumulate-recompress kernels; the receiver reconstructs the gradient by unnormalizing. The framework also uses non-uniform stochastic quantization, hierarchical UINT8 group scaling, and correlated rounding across workers. The authors implement DynamiQ on PyTorch DDP with NCCL P2P and evaluate on four LLM fine-tuning workloads, comparing with BF16, MXFP4/6/8, OmniReduce, and THC under ring and butterfly topologies. They report TTA improvements up to 34.2% over the best baseline and about 40.8% over BF16, with final accuracy within 0.1% of BF16 using 5 bits per coordinate.

Significance. If the results hold, DynamiQ is a strong practical contribution: it demonstrates that a multi-hop-aware, variable-bitwidth quantization scheme can preserve BF16-level accuracy at roughly 5 bits per coordinate while accelerating training. The paper provides useful structural contributions, including an unbiasedness proof for hierarchical scale quantization, a broad evaluation covering TTA, vNMSE, throughput, ablations, and scaling to 64 workers, and a clear description of the fused kernel design. The claims are falsifiable and the methodology is largely reproducible from the text, although the code is only promised for future release.

major comments (3)
  1. [Section 3.2 / Section 3.1 / Section 3.3] The threshold relations T_{1,2}=5/32*T_{2,4}, T_{2,4}=17/512*T_{4,8}, etc. are derived from a scalar worst-case model in which super-group MSE is proportional to F_j and each extra bit cuts worst-case MSE by 4x. In the actual pipeline, F_j is computed before the global mean mu_j is subtracted (Section 3.1), so the quantity actually quantized has squared norm F_j - n*S*mu_j^2, not F_j. In addition, the UINT8 hierarchical scaling factors (Section 3.3) introduce a bitwidth-independent error floor that is not captured by the 4^{-b} model. The text labels the scheme a "fast heuristic," and the Table 6 ablation shows that variable bitwidths help relative to uniform quantization, but it does not establish that this particular allocation is near-optimal. Given that the central 5-bit near-baseline claim depends on the allocation being sensible, I request either an oracle comparison (e.g., greedil
  2. [Section 5, Figures 4-9 and Tables 3-5] No repeated-seed results are reported anywhere in the evaluation. The TTA curves, final accuracies, vNMSE numbers, and throughput values appear to come from a single run per configuration. On an 8-GPU/4-worker testbed with stochastic quantization and standard training noise, the claim that DynamiQ "consistently" reaches near-baseline accuracy is not yet supported. Please provide at least three seeds for the main TTA and final-accuracy results, with error bars or a variance table, and state whether the reported curves are representative runs or averages.
  3. [Section 5, first paragraph and Figures 4/5] MXFP4 and MXFP6 are not actually executed end-to-end; their TTA is a "best-case lower-bound estimate" obtained by decoupling software-based accuracy from timing without performing the corresponding compute. The abstract and Section 5 present these as evaluated baselines (e.g., "up to 34.2% over ... MXFP4, MXFP6"). Since these are estimates, not full implementations, the comparison is not apples-to-apples, and the claim that DynamiQ is "the only evaluated method" to reach near-baseline accuracy should be qualified. The direction of bias is favorable to DynamiQ if the lower-bound is optimistic, but the text should clearly distinguish fully implemented baselines from estimated ones in the abstract, the main results, and the headline numbers.
minor comments (4)
  1. [Section 3.2] The phrase "the per-bit benefit of lowering T_{4,8} is 255/49" appears to contain a typo; from the stated formula the value should be 255/4^9 (i.e., 255/262144), not 255/49.
  2. [Appendix A] The expression q_j = 2*clamp([1,3], ...) uses undefined notation. Please clarify that the clamp returns an integer in {1,2,3} and that q_j is then in {2,4,8}.
  3. [Appendix B] The MSE bound uses epsilon_S without a precise definition. Please state what epsilon_S represents in terms of the quantization parameters.
  4. [Figure 14] Several subfigure labels are misaligned or duplicated (e.g., repeated "(b)" labels), which makes the zoomed-out TTA plots harder to read.

Circularity Check

0 steps flagged

No significant circularity; the central threshold heuristic is explicitly assumption-based and validated empirically, and self-citations are background/baselines.

full rationale

DynamiQ's claimed derivation chain is not circular. The variable-bitwidth allocation (Section 3.2) is an explicit worst-case heuristic: it assumes MSE is proportional to the super-group squared norm F_j and that each added bit reduces MSE by ~4x, then equalizes per-bit benefit to obtain threshold ratios and chooses the remaining degree of freedom from a bandwidth constraint. This is a stated modeling assumption, not a quantity fitted to the accuracy it later claims to predict; the paper's near-baseline-accuracy claim is an end-to-end measurement against BF16 and external baselines (MXFP, THC, OmniReduce), not a consequence of the allocation rule by construction. The 5-bit budget is selected post-hoc in an ablation (Figure 7); that is model selection, which can overfit but does not reduce the reported TTA/vNMSE numbers to the choice of budget. The paper uses several self-citations ([18-21], [36], [49]), but none is load-bearing: [36] is background on compression utility, [49] is a baseline, and [18-20] are examples of shared randomness while the actual correlated-rounding algorithm is taken from external work [63]. No uniqueness theorem or prior author result is invoked to force the design. The reviewer's concern that mean subtraction and UINT8 hierarchical scales may violate the MSE model is a robustness/correctness risk, not circularity: the derivation does not define its target in terms of its inputs. Overall: independent empirical evaluation with minor, non-load-bearing self-citations.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

The central claim rests on hand-picked configuration constants (group size, super-group size, bit budget, epsilon) and two modeling assumptions about gradient statistics and quantization error. No invented physical or mathematical entities. The bit budget b=5 is the most consequential free parameter because it was selected by ablation on the evaluation workloads and then featured in the headline.

free parameters (4)
  • group_size_s = 16
    Chosen by hand; sets quantization granularity and metadata overhead (Section 5, DynamiQ configuration).
  • super_group_size_S = 256
    Chosen by hand; determines metadata volume and reordering granularity (Section 5, DynamiQ configuration).
  • bit_budget_b = 5 bits/coordinate
    Selected via ablation on LLaMA 1B MMLU to optimize TTA (Figure 7, Table 4), then used as a headline configuration.
  • epsilon_nonuniform
    Shapes the non-uniform quantization values Q = {f(epsilon,r)} (Section 3.3); the paper never states the value used in experiments.
axioms (6)
  • domain assumption MSE of quantizing a set is proportional to its squared norm; F_j serves as a proxy for expected error of super-group j.
    Invoked in Section 3.2 to justify variable bitwidth allocation based on F_j. If violated by real gradient blocks, the allocation is not error-optimal.
  • domain assumption Each additional bit reduces worst-case MSE by roughly 4x, leading to threshold ratios T_{a,b}.
    Derived from scalar worst-case uniform quantization with midpoint insertion (Section 3.2) and applied to non-uniform quantization and real partial sums without a formal guarantee.
  • domain assumption Gradients exhibit spatial locality and skewness, so variable bitwidths and reordering help.
    Motivated by Figure 1 on two workloads; if gradients were flat or shuffled, the reordering and bit-allocation gains would shrink.
  • domain assumption In butterfly analysis, gradients on different workers follow the same distribution and partial-sum magnitudes scale with subtree size.
    Used in Section B to claim butterfly all-reduce reduces error by a factor of n vs. ring; heuristic, not proven for real data.
  • domain assumption The initial metadata all-reduce is lightweight (<1% of gradient volume) and does not become a bottleneck.
    Assumed in Section 3 and used to justify the two-phase design; metadata precision is not fully specified.
  • standard math Unbiased stochastic quantization preserves convergence (QSGD-type argument).
    Implicitly relied upon in Sections 2.1 and 3.3 via reference to Alistarh et al. [16].

pith-pipeline@v1.3.0-alltime-deepseek · 26429 in / 12255 out tokens · 117846 ms · 2026-08-03T03:05:30.109607+00:00 · methodology

0 comments
read the original abstract

Multi-hop all-reduce is the de facto backbone of large model training. As the training scale increases, the network often becomes a bottleneck, motivating the reduction of the volume of transmitted data. Accordingly, recent systems have demonstrated significant acceleration of the training process using gradient quantization. However, these systems are not optimized for multi-hop aggregation, where entries are partially summed multiple times along their aggregation topology. We present DynamiQ, a quantization framework that bridges the gap between quantization best practices and multi-hop aggregation. DynamiQ introduces novel techniques to better represent partial sums, codesigned with a decompress accumulate recompress fused kernel to facilitate fast execution. We extend PyTorch DDP to support DynamiQ over NCCL P2P, and across different LLMs, tasks, and scales, we demonstrate consistent improvement of up to 34.2% over the best among state-of-the-art methods such as Omni-Reduce, THC, and emerging standards such as MXFP4, MXFP6, and MXFP8. Further, DynamiQ is the only evaluated method that consistently reaches near-baseline accuracy (e.g., 99.9\% of the BF16 baseline) and does so while significantly accelerating the training.

Figures

Figures reproduced from arXiv: 2602.08923 by Michael Mitzenmacher, Ran Ben Basat, Shay Vargaftik, Wenchen Han.

Figure 1
Figure 1. Figure 1: ℓ2 norm distributions of the gradients and their random shuffle for groups of size 16 and super-groups of size 256. The detailed experimental setups appear in Section 5. too many small groups inflates the required bandwidth, de￾feating the purpose of quantization. To reduce the encoding overheads, one can use super-groups (e.g., of 16 consecutive groups each) to share some of the metadata [9]. We conduct a… view at source ↗
Figure 2
Figure 2. Figure 2: The DynamiQ workflow: (a) workers first compute the metadata (mean and ℓ2 norm) for each of their super-groups; (b) a lightweight all-reduce call aggregated the metadata such that all workers know that global super-group means and sum of ℓ2 norms; (c) based on the aggregated metadata, each worker normalizes each super-group by subtracting its global mean and reorders the super-groups based on their bit wid… view at source ↗
Figure 3
Figure 3. Figure 3: The CDF distribution of 𝐹𝑗 , summed ℓ2 squared norm per super-group across workers. The vertical dashed lines are thresholds for our variable bitwidth allocation algorithms, where super-groups with larger ℓ2 norms are assigned more bits in one of 2, 4, or 8 bits. same (e.g., 𝑇1,2 · 3/16 =𝑇2,4 · 15/256), yielding: 𝑇1,2 = 5/32 ·𝑇2,4, 𝑇2,4 = 17/512 ·𝑇4,8, 𝑇4,8 = 257/2 17 ·𝑇8,16. Notice that this gives us |𝑊 |… view at source ↗
Figure 10
Figure 10. Figure 10: Scalability evaluation on the LLaMA + MMLU task [PITH_FULL_IMAGE:figures/full_fig_p011_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Scalability evaluation on the TinyBERT + GLUE [PITH_FULL_IMAGE:figures/full_fig_p011_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Compression error comparisons measured in vN [PITH_FULL_IMAGE:figures/full_fig_p016_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Butterfly all-reduce topology for a specific gradient [PITH_FULL_IMAGE:figures/full_fig_p016_13.png] view at source ↗
Figure 17
Figure 17. Figure 17: Bandwidth usage over time for the LLaMA 1B [PITH_FULL_IMAGE:figures/full_fig_p017_17.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Quantizing With Randomized Hadamard Transforms: Efficient Heuristic Now Proven

    cs.LG 2026-05 unverdicted novelty 7.0

    Two randomized Hadamard transforms suffice to make coordinate marginals O(d^{-1/2})-close to Gaussian for most quantization methods, with three needed for vector quantization to match uniform random rotations asymptotically.

Reference graph

Works this paper leans on

82 extracted references · 16 linked inside Pith · cited by 1 Pith paper

  1. [1]

    https://github.com/baidu-research/baidu-allreduce, 2017

    Ring all reduce. https://github.com/baidu-research/baidu-allreduce, 2017

    2017

  2. [2]

    https://cloud

    Bfloat16: The secret to high performance on cloud tpus. https://cloud. google.com/blog/products/ai-machine-learning/bfloat16-the-secre t-to-high-performance-on-cloud-tpus, 2019

    2019

  3. [3]

    https://developer.nvidia.com/blog/accelerating-hpc-applica tions-with-nsight-compute-roofline-analysis, 2020

    Accelerating hpc applications with nvidia nsight compute roofline analysis. https://developer.nvidia.com/blog/accelerating-hpc-applica tions-with-nsight-compute-roofline-analysis, 2020

    2020

  4. [4]

    =https://docs.jax.dev/en/latest/index.html, 2020

    Jax: High performance array computing. =https://docs.jax.dev/en/latest/index.html, 2020

    2020

  5. [5]

    https://developer

    Mastering llm techniques: Inference optimization. https://developer. nvidia.com/blog/mastering-llm-techniques-inference-optimization/, 2023

    2023

  6. [6]

    https://docs.nvidia.com/deeplear ning/performance/dl-performance-memory-limited/index.html, 2023

    Memory-limited layers user’s guide. https://docs.nvidia.com/deeplear ning/performance/dl-performance-memory-limited/index.html, 2023

    2023

  7. [7]

    https://www.opencomp ute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final -pdf, 2023

    Ocp microscaling formats (mx) specification. https://www.opencomp ute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final -pdf, 2023

    2023

  8. [8]

    https://docs.nvidia.com/deeplearning/n ccl/user-guide/docs/usage/collectives.html, 2024

    Collective operations in nccl. https://docs.nvidia.com/deeplearning/n ccl/user-guide/docs/usage/collectives.html, 2024

    2024

  9. [9]

    https://github.com/ggml-org/ggml/blob/master/docs/g guf.md, 2024

    Gguf format. https://github.com/ggml-org/ggml/blob/master/docs/g guf.md, 2024

    2024

  10. [10]

    https://engineering

    How meta trains large language models at scale. https://engineering. fb.com/2024/06/12/data-infrastructure/training-large-language-m odels-at-scale-meta/, 2024

    2024

  11. [11]

    https://developer.nv idia.com/nccl, 2024

    Nvidia collective communications library (nccl). https://developer.nv idia.com/nccl, 2024

    2024

  12. [12]

    https://developer.nvidia.com/blog/advanced-nvidia-cuda-kerne l-optimization-techniques-handwritten-ptx/, 2025

    Advanced nvidia cuda kernel optimization techniques: Handwritten ptx. https://developer.nvidia.com/blog/advanced-nvidia-cuda-kerne l-optimization-techniques-handwritten-ptx/, 2025

    2025

  13. [13]

    https://developer.nvidia.com/blog/nvfp4-trains-with-precision-of-1 6-bit-and-speed-and-efficiency-of-4-bit/, 2025

    Nvfp4 trains with precision of 16-bit and speed and efficiency of 4-bit. https://developer.nvidia.com/blog/nvfp4-trains-with-precision-of-1 6-bit-and-speed-and-efficiency-of-4-bit/, 2025

    2025

  14. [14]

    Nvidia blackwell architecture technical brief, 2025

    2025

  15. [15]

    On the utility of gradient compression in dis- tributed training systems.Proceedings of Machine Learning and Systems, 4:652–672, 2022

    Saurabh Agarwal, Hongyi Wang, Shivaram Venkataraman, and Dim- itris Papailiopoulos. On the utility of gradient compression in dis- tributed training systems.Proceedings of Machine Learning and Systems, 4:652–672, 2022

    2022

  16. [16]

    Qsgd: Communication-efficient sgd via gradient quantization and encoding.Advances in neural information processing systems, 30, 2017

    Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. Qsgd: Communication-efficient sgd via gradient quantization and encoding.Advances in neural information processing systems, 30, 2017

    2017

  17. [17]

    Gradient compression supercharged high-performance data parallel dnn training

    Youhui Bai, Cheng Li, Quan Zhou, Jun Yi, Ping Gong, Feng Yan, Ruichuan Chen, and Yinlong Xu. Gradient compression supercharged high-performance data parallel dnn training. InProceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles, pages 359–375, 2021

    2021

  18. [18]

    Optimal and approximate adaptive stochastic quantization

    Ran Ben-Basat, Yaniv Ben-Itzhak, Michael Mitzenmacher, and Shay Vargaftik. Optimal and approximate adaptive stochastic quantization. Advances in Neural Information Processing Systems, 37:94265–94291, 2024

    2024

  19. [19]

    Better than optimal: Improving adaptive stochastic quantiza- tion using shared randomness.Proceedings of the ACM on Measurement and Analysis of Computing Systems, 9(3):1–44, 2025

    Ran Ben Basat, Yaniv Ben-Itzhak, Michael Mitzenmacher, and Shay Vargaftik. Better than optimal: Improving adaptive stochastic quantiza- tion using shared randomness.Proceedings of the ACM on Measurement and Analysis of Computing Systems, 9(3):1–44, 2025

    2025

  20. [20]

    How to send a real number using a single bit (and some shared random- ness)

    Ran Ben Basat, Michael Mitzenmacher, and Shay Vargaftik. How to send a real number using a single bit (and some shared random- ness). In48th International Colloquium on Automata, Languages, and Programming (ICALP 2021), 2021

    2021

  21. [21]

    Accelerating federated learning with quick distributed mean estimation

    Ran Ben-Basat, Amit Portnoy, Gil Einziger, Yaniv Ben-Itzhak, and Michael Mitzenmacher. Accelerating federated learning with quick distributed mean estimation. InICML, 2024

    2024

  22. [22]

    signsgd: Compressed optimisation for non- convex problems

    Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and An- imashree Anandkumar. signsgd: Compressed optimisation for non- convex problems. InInternational Conference on Machine Learning, pages 560–569. PMLR, 2018

    2018

  23. [23]

    Crux: Gpu-efficient communication scheduling for deep learning training

    Jiamin Cao, Yu Guan, Kun Qian, Jiaqi Gao, Wencong Xiao, Jianbo Dong, Binzhang Fu, Dennis Cai, and Ennan Zhai. Crux: Gpu-efficient communication scheduling for deep learning training. InProceedings of the ACM SIGCOMM 2024 Conference, pages 1–15, 2024

    2024

  24. [24]

    Quartet: Native fp4 training can be optimal for large language models.arXiv preprint arXiv:2505.14669, 2025

    Roberto L Castro, Andrei Panferov, Soroush Tabesh, Oliver Sieberling, Jiale Chen, Mahdi Nikdan, Saleh Ashkboos, and Dan Alistarh. Quartet: Native fp4 training can be optimal for large language models.arXiv preprint arXiv:2505.14669, 2025

    arXiv 2025

  25. [25]

    Efficientqat: Efficient quantization-aware training for large language models.arXiv preprint arXiv:2407.11062, 2024

    Mengzhao Chen, Wenqi Shao, Peng Xu, Jiahao Wang, Peng Gao, Kaipeng Zhang, Yu Qiao, and Ping Luo. Efficientqat: Efficient quantization-aware training for large language models.arXiv preprint arXiv:2407.11062, 2024

    Pith/arXiv arXiv 2024

  26. [26]

    When ML Training Cuts Through Congestion: Just-in-Time Gradient Compression via Packet Trimming

    Xiaoqi Chen, Shay Vargaftik, and Ran Ben-Basat. When ML Training Cuts Through Congestion: Just-in-Time Gradient Compression via Packet Trimming. InHotnets, 2024

    2024

  27. [27]

    Large scale distributed deep networks

    Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Marc'aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, Quoc Le, and Andrew Ng. Large scale distributed deep networks. In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012

    2012

  28. [28]

    Bert: Pre-training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805, 2018

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805, 2018

    Pith/arXiv arXiv 2018

  29. [29]

    Enhancing chat language models by scaling high-quality instructional conversations

    Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.14233, 2023

    Pith/arXiv arXiv 2023

  30. [30]

    Docofl: Downlink compression for cross-device federated learning

    Ron Dorfman, Shay Vargaftik, Yaniv Ben-Itzhak, and Kfir Yehuda Levy. Docofl: Downlink compression for cross-device federated learning. In International Conference on Machine Learning, pages 8356–8388. PMLR, 2023

    2023

  31. [31]

    Ice buckets: Improved counter estimation for network measurement

    Gil Einziger, Benny Fellman, Roy Friedman, and Yaron Kassner. Ice buckets: Improved counter estimation for network measurement. IEEE/ACM Transactions on Networking, 26(3):1165–1178, 2018

    2018

  32. [32]

    The llama 3 herd of models, 2024

    Grattafiori et al. The llama 3 herd of models, 2024

    2024

  33. [33]

    Efficient sparse collective communication and its application to accelerate distributed deep learning

    Jiawei Fei, Chen-Yu Ho, Atal N Sahu, Marco Canini, and Amedeo Sapio. Efficient sparse collective communication and its application to accelerate distributed deep learning. InProceedings of the 2021 ACM SIGCOMM 2021 Conference, pages 676–691, 2021. 13 arXiv, 2026 Wenchen Han, Shay Vargaftik, Michael Mitzenmacher, and Ran Ben Basat

    2021

  34. [34]

    Gherghescu, Vlad-Andrei Bădoiu, Alexandru Agache, Mihai-Valentin Dumitru, Iuliu Vasilescu, Radu Mantu, and Costin Raiciu

    Alexandru M. Gherghescu, Vlad-Andrei Bădoiu, Alexandru Agache, Mihai-Valentin Dumitru, Iuliu Vasilescu, Radu Mantu, and Costin Raiciu. I’ve got 99 problems but flops ain’t one. InProceedings of the 23rd ACM Workshop on Hot Topics in Networks, HotNets ’24, page 195–204, New York, NY, USA, 2024. Association for Computing Ma- chinery

    2024

  35. [35]

    Ai and memory wall.IEEE Micro, 44(3):33– 39, 2024

    Amir Gholami, Zhewei Yao, Sehoon Kim, Coleman Hooper, Michael W Mahoney, and Kurt Keutzer. Ai and memory wall.IEEE Micro, 44(3):33– 39, 2024

    2024

  36. [36]

    Beyond throughput and compression ratios: To- wards high end-to-end utility of gradient compression

    Wenchen Han, Shay Vargaftik, Michael Mitzenmacher, Brad Karp, and Ran Ben Basat. Beyond throughput and compression ratios: To- wards high end-to-end utility of gradient compression. InProceedings of the 23rd ACM Workshop on Hot Topics in Networks, HotNets ’24, page 186–194, New York, NY, USA, 2024. Association for Computing Machinery

    2024

  37. [37]

    Hadamard matrices and their applications.The annals of statistics, pages 1184–1238, 1978

    A Hedayat and Walter Dennis Wallis. Hadamard matrices and their applications.The annals of statistics, pages 1184–1238, 1978

    1978

  38. [38]

    Measuring massive multi- task language understanding.arXiv preprint arXiv:2009.03300, 2020

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multi- task language understanding.arXiv preprint arXiv:2009.03300, 2020

    Pith/arXiv arXiv 2009

  39. [39]

    Hammingmesh: a network topology for large-scale deep learning

    Torsten Hoefler, Tommaso Bonato, Daniele De Sensi, Salvatore Di Giro- lamo, Shigang Li, Marco Heddes, Jon Belk, Deepak Goel, Miguel Castro, and Steve Scott. Hammingmesh: a network topology for large-scale deep learning. InProceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, SC ’22. IEEE Press, 2022

    2022

  40. [40]

    Elastic resource sharing for distributed deep learn- ing

    Changho Hwang, Taehyun Kim, Sunghyun Kim, Jinwoo Shin, and KyoungSoo Park. Elastic resource sharing for distributed deep learn- ing. In18th USENIX Symposium on Networked Systems Design and Implementation (NSDI 21), pages 721–739, 2021

    2021

  41. [41]

    A unified architecture for accelerating distributed{DNN} training in heterogeneous{GPU/CPU} clusters

    Yimin Jiang, Yibo Zhu, Chang Lan, Bairen Yi, Yong Cui, and Chuanx- iong Guo. A unified architecture for accelerating distributed{DNN} training in heterogeneous{GPU/CPU} clusters. In14th USENIX Sym- posium on Operating Systems Design and Implementation (OSDI 20), pages 463–479, 2020

    2020

  42. [42]

    Tinybert: Distilling bert for natural language understanding.arXiv preprint arXiv:1909.10351, 2019

    Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding.arXiv preprint arXiv:1909.10351, 2019

    Pith/arXiv arXiv 1909

  43. [43]

    Error feedback fixes signsgd and other gradient compression schemes

    Sai Praneeth Karimireddy, Quentin Rebjock, Sebastian Stich, and Mar- tin Jaggi. Error feedback fixes signsgd and other gradient compression schemes. InInternational Conference on Machine Learning, pages 3252–

  44. [44]

    Parallax: Sparsity-aware data parallel training of deep neural networks

    Soojeong Kim, Gyeong-In Yu, Hojin Park, Sungwoo Cho, Eunji Jeong, Hyeonmin Ha, Sanha Lee, Joo Seong Jeong, and Byung-Gon Chun. Parallax: Sparsity-aware data parallel training of deep neural networks. InProceedings of the Fourteenth EuroSys Conference 2019, pages 1–15, 2019

    2019

  45. [45]

    A method for stochastic opti- mization

    Diederik Kinga, Jimmy Ba Adam, et al. A method for stochastic opti- mization. InInternational conference on learning representations (ICLR), volume 5. California;, 2015

    2015

  46. [46]

    To fp8 and back again: Quantifying reduced precision effects on llm training stability.arXiv preprint arXiv:2405.18710, 2024

    Joonhyung Lee, Jeongin Bae, Byeongwook Kim, Se Jung Kwon, and Dongsoo Lee. To fp8 and back again: Quantifying reduced precision effects on llm training stability.arXiv preprint arXiv:2405.18710, 2024

    Pith/arXiv arXiv 2024

  47. [47]

    Training with mixed- precision floating-point assignments.arXiv preprint arXiv:2301.13464, 2023

    Wonyeol Lee, Rahul Sharma, and Alex Aiken. Training with mixed- precision floating-point assignments.arXiv preprint arXiv:2301.13464, 2023

    Pith/arXiv arXiv 2023

  48. [48]

    Accelerating distributed deep learning using lossless homomorphic compression.arXiv preprint arXiv:2402.07529, 2024

    Haoyu Li, Yuchen Xu, Jiayi Chen, Rohit Dwivedula, Wenfei Wu, Ke- qiang He, Aditya Akella, and Daehyeok Kim. Accelerating distributed deep learning using lossless homomorphic compression.arXiv preprint arXiv:2402.07529, 2024

    Pith/arXiv arXiv 2024

  49. [49]

    In21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), pages 1191–1211, 2024

    Minghao Li, Ran Ben Basat, Shay Vargaftik, ChonLam Lao, Kevin Xu, Michael Mitzenmacher, and Minlan Yu.{THC}: Accelerating dis- tributed deep learning using tensor homomorphic compression. In21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), pages 1191–1211, 2024

    2024

  50. [50]

    Scaling distributed machine learning with the parameter server

    Mu Li, David G Andersen, Jun Woo Park, Alexander J Smola, Amr Ahmed, Vanja Josifovski, James Long, Eugene J Shekita, and Bor-Yiing Su. Scaling distributed machine learning with the parameter server. In11th USENIX Symposium on operating systems design and implemen- tation (OSDI 14), pages 583–598, 2014

    2014

  51. [51]

    Pytorch distributed: Experiences on accelerating data parallel training.arXiv preprint arXiv:2006.15704, 2020

    Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, et al. Pytorch distributed: Experiences on accelerating data parallel training.arXiv preprint arXiv:2006.15704, 2020

    Pith/arXiv arXiv 2006

  52. [52]

    Decoupled weight decay regulariza- tion.arXiv preprint arXiv:1711.05101, 2017

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regulariza- tion.arXiv preprint arXiv:1711.05101, 2017

    Pith/arXiv arXiv 2017

  53. [53]

    An efficient statistical-based gradient compression technique for distributed training systems.Proceedings of Machine Learning and Systems, 3:297–322, 2021

    Ahmed M Abdelmoniem, Ahmed Elzanaty, Mohamed-Slim Alouini, and Marco Canini. An efficient statistical-based gradient compression technique for distributed training systems.Proceedings of Machine Learning and Systems, 3:297–322, 2021

    2021

  54. [54]

    Pointer sentinel mixture models.arXiv preprint arXiv:1609.07843, 2016

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models.arXiv preprint arXiv:1609.07843, 2016

    Pith/arXiv arXiv 2016

  55. [55]

    Quest: Training accurate llms over highly- compressed weights and activation

    Andrei Panferov, Jiale Chen, Soroush Tabesh, Roberto L Castro, Mahdi Nikdan, and Dan Alistarh. Quest: Training accurate llms over highly- compressed weights and activation. InSparsity in LLMs (SLLM): Deep Dive into Mixture of Experts, Quantization, Hardware, and Inference

  56. [56]

    Bandwidth optimal all-reduce algo- rithms for clusters of workstations.Journal of Parallel and Distributed Computing, 69(2):117–124, 2009

    Pitch Patarasuk and Xin Yuan. Bandwidth optimal all-reduce algo- rithms for clusters of workstations.Journal of Parallel and Distributed Computing, 69(2):117–124, 2009

    2009

  57. [57]

    Fp8-lm: Training fp8 large language models.arXiv preprint arXiv:2310.18313, 2023

    Houwen Peng, Kan Wu, Yixuan Wei, Guoshuai Zhao, Yuxiang Yang, Ze Liu, Yifan Xiong, Ziyue Yang, Bolin Ni, Jingcheng Hu, et al. Fp8-lm: Training fp8 large language models.arXiv preprint arXiv:2310.18313, 2023

    Pith/arXiv arXiv 2023

  58. [58]

    Zero: Memory optimizations toward training trillion parameter models

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. InSC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE, 2020

    2020

  59. [59]

    Microscaling data formats for deep learning.arXiv preprint arXiv:2310.10537, 2023

    Bita Darvish Rouhani, Ritchie Zhao, Ankit More, Mathew Hall, Alireza Khodamoradi, Summer Deng, Dhruv Choudhary, Marius Cornea, Eric Dellinger, Kristof Denolf, et al. Microscaling data formats for deep learning.arXiv preprint arXiv:2310.10537, 2023

    Pith/arXiv arXiv 2023

  60. [60]

    Scaling distributed machine learning with {In- Network} aggregation

    Amedeo Sapio, Marco Canini, Chen-Yu Ho, Jacob Nelson, Panos Kalnis, Changhoon Kim, Arvind Krishnamurthy, Masoud Moshref, Dan Ports, and Peter Richtárik. Scaling distributed machine learning with {In- Network} aggregation. In18th USENIX Symposium on Networked Systems Design and Implementation (NSDI 21), pages 785–808, 2021

    2021

  61. [61]

    Swing: Short-cutting rings for higher bandwidth allreduce

    Daniele De Sensi, Tommaso Bonato, David Saam, and Torsten Hoefler. Swing: Short-cutting rings for higher bandwidth allreduce. In21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), pages 1445–1462, Santa Clara, CA, April 2024. USENIX Association

    2024

  62. [62]

    Spar- sified sgd with memory.Advances in neural information processing systems, 31, 2018

    Sebastian U Stich, Jean-Baptiste Cordonnier, and Martin Jaggi. Spar- sified sgd with memory.Advances in neural information processing systems, 31, 2018

    2018

  63. [63]

    Corre- lated quantization for distributed mean estimation and optimization

    Ananda Theertha Suresh, Ziteng Sun, Jae Ro, and Felix Yu. Corre- lated quantization for distributed mean estimation and optimization. InInternational Conference on Machine Learning, pages 20856–20876. PMLR, 2022

    2022

  64. [64]

    Dreamddp: Accelerating data parallel distributed llm training with layer-wise scheduled partial synchronization, 2025

    Zhenheng Tang, Zichen Tang, Junlin Huang, Xinglin Pan, Rudan Yan, Yuxin Wang, Amelie Chi Zhou, Shaohuai Shi, Xiaowen Chu, and Bo Li. Dreamddp: Accelerating data parallel distributed llm training with layer-wise scheduled partial synchronization, 2025. 14 DynamiQ arXiv, 2026

    2025

  65. [65]

    Gemma 3 technical report.arXiv preprint arXiv:2503.19786, 2025

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexan- dre Ramé, Morgane Rivière, et al. Gemma 3 technical report.arXiv preprint arXiv:2503.19786, 2025

    Pith/arXiv arXiv 2025

  66. [66]

    Optimization of collective communication operations in mpich.The International Journal of High Performance Computing Applications, 19(1):49–66, 2005

    Rajeev Thakur, Rolf Rabenseifner, and William Gropp. Optimization of collective communication operations in mpich.The International Journal of High Performance Computing Applications, 19(1):49–66, 2005

    2005

  67. [67]

    Training llms with mxfp4

    Albert Tseng, Tao Yu, and Youngsuk Park. Training llms with mxfp4. In Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, pages 1630–1638, 2025

    2025

  68. [68]

    Eden: Communication- efficient and robust distributed mean estimation for federated learning

    Shay Vargaftik, Ran Ben Basat, Amit Portnoy, Gal Mendelson, Yaniv Ben Itzhak, and Michael Mitzenmacher. Eden: Communication- efficient and robust distributed mean estimation for federated learning. InInternational Conference on Machine Learning, pages 21984–22014. PMLR, 2022

    2022

  69. [69]

    Drive: One-bit distributed mean estimation.Advances in Neural Information Processing Systems, 34:362–377, 2021

    Shay Vargaftik, Ran Ben-Basat, Amit Portnoy, Gal Mendelson, Yaniv Ben-Itzhak, and Michael Mitzenmacher. Drive: One-bit distributed mean estimation.Advances in Neural Information Processing Systems, 34:362–377, 2021

    2021

  70. [70]

    Powersgd: Practical low-rank gradient compression for distributed optimization

    Thijs Vogels, Sai Praneeth Karimireddy, and Martin Jaggi. Powersgd: Practical low-rank gradient compression for distributed optimization. Advances in Neural Information Processing Systems, 32, 2019

    2019

  71. [71]

    Glue: A multi-task benchmark and anal- ysis platform for natural language understanding.arXiv preprint arXiv:1804.07461, 2018

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and anal- ysis platform for natural language understanding.arXiv preprint arXiv:1804.07461, 2018

    Pith/arXiv arXiv 2018

  72. [72]

    Atomo: Communication-efficient learning via atomic sparsification.Advances in neural information processing systems, 31, 2018

    Hongyi Wang, Scott Sievert, Shengchao Liu, Zachary Charles, Dimitris Papailiopoulos, and Stephen Wright. Atomo: Communication-efficient learning via atomic sparsification.Advances in neural information processing systems, 31, 2018

    2018

  73. [73]

    Optimizing large language model training using fp4 quantization

    Ruizhe Wang, Yeyun Gong, Xiao Liu, Guoshuai Zhao, Ziyue Yang, Bain- ing Guo, Zhengjun Zha, and Peng Cheng. Optimizing large language model training using fp4 quantization. InForty-second International Conference on Machine Learning, 2025

    2025

  74. [74]

    Hi-speed dnn training with espresso: Unleashing the full potential of gradient compression with near-optimal usage strategies

    Zhuang Wang, Haibin Lin, Yibo Zhu, and TS Eugene Ng. Hi-speed dnn training with espresso: Unleashing the full potential of gradient compression with near-optimal usage strategies. InProceedings of the Eighteenth European Conference on Computer Systems, pages 867–882, 2023

    2023

  75. [75]

    Cupcake: A compression optimizer for scalable communication- efficient distributed training

    Zhuang Wang, Xinyu Crystal Wu, Zhaozhuo Xu, and TS Eugene Ng. Cupcake: A compression optimizer for scalable communication- efficient distributed training. InProceedings of the Sixth Conference on Machine Learning and Systems (MLSys’ 23). Proceedings of the Sixth Conference on Machine Learning and Systems (MLSys’ 23), 2023

    2023

  76. [76]

    Optinic: A resilient and tail-optimal rdma nic for distributed ml workloads.arXiv preprint arXiv:2512.22743, 2025

    Ertza Warraich, Ali Imran, Annus Zulfiqar, Shay Vargaftik, Sonia Fahmy, and Muhammad Shahbaz. Optinic: A resilient and tail-optimal rdma nic for distributed ml workloads.arXiv preprint arXiv:2512.22743, 2025

    arXiv 2025

  77. [77]

    {OptiReduce}: Resilient and {Tail-Optimal}{ AllReduce} for dis- tributed deep learning in the cloud

    Ertza Warraich, Omer Shabtai, Khalid Manaa, Shay Vargaftik, Yonatan Piasetzky, Matty Kadosh, Lalith Suresh, and Muhammad Shahbaz. {OptiReduce}: Resilient and {Tail-Optimal}{ AllReduce} for dis- tributed deep learning in the cloud. In22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI 25), pages 685–703, 2025

    2025

  78. [78]

    Terngrad: Ternary gradients to reduce communi- cation in distributed deep learning.Advances in neural information processing systems, 30, 2017

    Wei Wen, Cong Xu, Feng Yan, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Terngrad: Ternary gradients to reduce communi- cation in distributed deep learning.Advances in neural information processing systems, 30, 2017

    2017

  79. [79]

    Hack: Homomorphic acceleration via compression of the key-value cache for disaggregated llm inference

    Zeyu Zhang, Haiying Shen, Shay Vargaftik, Ran Ben Basat, Michael Mitzenmacher, and Minlan Yu. Hack: Homomorphic acceleration via compression of the key-value cache for disaggregated llm inference. In Proceedings of the ACM SIGCOMM 2025 Conference, pages 1245–1247, 2025

    2025

  80. [80]

    Galore: Memory-efficient LLM training by gradient low-rank projection

    Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient LLM training by gradient low-rank projection. InForty-first International Conference on Machine Learning, 2024

    2024

Showing first 80 references.