REVIEW 5 minor 49 references
Limited-precision stochastic rounding recovers the sqrt(n) error growth of exact SR while matching what commercial chips already ship.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-15 14:01 UTC pith:JXZ6CDS4
load-bearing objection Solid, timely update survey that organizes limited-precision SR and the 2022–2026 hardware/theory wave; useful synthesis, not a new theorem paper.
What is New in Stochastic Rounding: a Survey on Theory, Hardware, and Applications
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Limited-precision stochastic rounding—defined by first rounding an exact value to precision p+r and then applying ordinary SR—is the practically relevant new form of the method. When r is chosen near ceil((log2 n)/2), the probabilistic O(sqrt(n) u) error bounds that hold for exact SR continue to hold for recursive summation, inner products, Horner’s method and pairwise summation, while the random-bit cost stays modest enough for real hardware. Commercial devices already implement concrete instances of this limited-precision rule, and the survey collates their bit-widths, standardisation proposals and application successes.
What carries the argument
Limited-precision SR: SR_p^r(x) = SR_p(fl_{p+r}(x)). The outer SR uses the usual distance-based probabilities; the inner deterministic rounding fl_{p+r} fixes the random-number precision to r bits. The construction is what lets analysis recover the concentration bounds of exact SR and what matches the conversion instructions already documented by major vendors.
Load-bearing premise
The hardware claims rest on public ISA documents, patents and white-papers rather than independent measurements of the actual silicon behaviour.
What would settle it
Measure the actual random-bit widths and rounding bias of the SR conversion instructions on AMD MI300, NVIDIA Blackwell and Graphcore devices; if the observed bit counts or bias diverge systematically from the ISA tables, the claim that commercial hardware realises the analysed limited-precision model fails.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript is an update survey of stochastic rounding (SR) covering roughly 2022–2026. It organizes recent progress around limited-precision SR, defined as SR_p^r(x) = SR_p(fl_{p+r}(x)), which approximates the ideal probabilities of exact SR while remaining implementable. The paper reviews probabilistic error analyses that recover O(√n u) bounds for summation and related algorithms when r is chosen near ⌈(log2 n)/2⌉, summarizes commercial hardware realizations (Graphcore, AMD MI300, NVIDIA PTX/Blackwell, Tesla, Google TPU) and research/patent activity, and surveys applications in mixed-precision ML training, neuromorphic computing, climate simulation, and scientific computing. It also covers the IEEE P3109 interim variants (StochasticA/B/C) and software emulation packages.
Significance. As a timely literature synthesis the paper is valuable. Limited-precision SR is the practically relevant variant now appearing in vendor ISAs and ML training stacks; collecting the random-bit widths (Table 1), the P3109 bias-complexity trade-offs, the martingale/variance analyses, and the application evidence into one place is useful for both numerical analysts and hardware designers. The survey does not claim original theorems or silicon measurements; its contribution is organizational and bibliographic, which is appropriate for the genre and for an update to Croci et al. (2022). Strengths include the clear exact-vs-limited-precision distinction (Fig. 1 and Eq. (1)), the concrete hardware table, and the breadth of application coverage.
minor comments (5)
- Section 5 / Table 1: the caption and body correctly note that the bit-widths come from public ISA documents and patents. A single explicit sentence that these are secondary sources (no independent silicon measurements) would make the inherent limitation of the genre fully transparent to readers.
- Section 4: the heuristic r ≈ ⌈(log2 n)/2⌉ is attributed to El Arar et al. [16,17]. A one-sentence pointer to the precise statement (or theorem number) in those papers would help readers locate the supporting analysis without hunting.
- Scattered typographical issues: missing spaces after commas/periods in several places (e.g., “round-to-nearest(RN)”, “fl +r(x)” rendering), and a few incomplete sentences near the end of Section 5 (Huawei/Google paragraphs). These are purely presentational.
- References [17] is listed as “in preparation”; if it remains unpublished at acceptance, consider citing the arXiv version or noting the status more prominently so readers know the supporting analysis is not yet peer-reviewed.
- Section 9 (ML): the discussion of 1-D vs 2-D scaling and double quantization is dense. A short clarifying sentence on why SR is preferred only for gradients (and not forward activations) would improve accessibility for non-ML readers.
Circularity Check
No significant circularity: pure literature survey with no original derivations, fitted predictions, or load-bearing self-citation chains that reduce claims to inputs.
full rationale
This is an update survey (explicitly framed as covering progress since Croci et al. 2022) whose central organizational claim is that limited-precision SR (defined as SR_p^r(x) = SR_p(fl_{p+r}(x))) is the practically relevant recent variant. The paper reports, rather than derives, probabilistic error bounds of order O(sqrt(n)u) and the heuristic r ~ ceil((log2 n)/2) from the authors' own prior SIAM papers [16,17] and from independent martingale/variance analyses [7,10,12]. Those citations are peer-reviewed external publications whose results are independently checkable; the survey does not re-derive them or treat them as axioms that force a new prediction. Hardware claims rest on public ISA documents, patents and white-papers (Graphcore, AMD MI300, NVIDIA PTX, IEEE P3109, Tesla, etc.) that are third-party sources. No equation, uniqueness theorem, ansatz, or fitted parameter of the present manuscript is used to manufacture a result that is true by construction. Self-citations are present but non-load-bearing for any claimed novelty; the genre is synthesis, not first-principles derivation. Score 0 is therefore the correct, proportionate finding.
Axiom & Free-Parameter Ledger
axioms (4)
- standard math Normalized floating-point system F with p digits; every real x between the extreme finite values has two adjacent candidates floor and ceil.
- domain assumption Exact SR returns the upper candidate with probability q(x) = (x - floor)/(ceil - floor) and is unbiased: E[SR(x)] = x.
- domain assumption Limited-precision SR is obtained by first rounding to p+r bits then applying exact SR; the resulting bias and variance are controlled by r.
- ad hoc to paper Public vendor ISA documents and patents correctly describe the random-bit widths and addition-of-random-bits logic actually present in silicon.
invented entities (1)
-
limited-precision stochastic rounding (SR_p^r)
independent evidence
read the original abstract
Stochastic rounding (SR) is a probabilistic method used to round numbers to floating-point and fixed-point representations. In length $n$ summation, the worst-case error of SR grows as $\sqrt{n}$ with high probability, unlike for standard modes, like round-to-nearest (RN), which grows as $n$. For this reason, the former is increasingly employed in large-scale, low-precision computations as an RN alternative. Additionally, SR alleviates stagnation, whereby relatively small summands are completely rounded off and do not contribute to the sum. We provide an update to [Croci et al., Roy. Soc. Open Sci. 9.3 (2022), pp. 1-25], a survey which discusses the development and use of SR between 1949 and 2022, citing over 100 references. Since then, there has been a surge of new research, and this update covers almost four years of further progress in applying, analysing, and implementing SR. Our main focus is limited-precision stochastic rounding, a new variant that fixes the precision of the random numbers used. We provide insights into industrial and numerical analysis activities surrounding SR, highlighting the next possible steps in making this rounding mode more widely available in hardware.
Reference graph
Works this paper leans on
-
[1]
Stochastic Rounding 2.0, with a View towards Complexity Analysis
P.DrineasandI.C.F.Ipsen.StochasticRounding2.0,withaViewtowardsComplexityAnalysis. arXiv:2410.10517 [math.NA]. 2024.���:10.48550/arXiv.2410.10517. [3]Interim Report on Binary Floating-point Formats for Machine Learning. Tech. rep. Version 3.2.1. 2025.���:https : / / github . com / P3109 / Public / blob / main / IEEE % 20WG % 20P3109%20Interim%20Report%20v3.2.1.pdf
work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2410.10517 2024
-
[2]
OnStochasticRoundingwithFewRandomBits
A.FitzgibbonandS.Felix.“OnStochasticRoundingwithFewRandomBits”.In:Proceedingsof the32ndSymposiumonComputerArithmetic.2025,pp.133–140.���:10.1109/ARITH64983. 2025.00029
-
[3]
���:https://digitalassets.tesla.com/tesla- contents/image/upload/tesla- dojo-technology.pdf
TeslaDojoTechnology.AGuidetoTesla’sConfigurableFloatingPointFormats&Arithmetic. ���:https://digitalassets.tesla.com/tesla- contents/image/upload/tesla- dojo-technology.pdf
-
[4]
The Positive Effects of Stochastic Rounding in Numerical Algorithms
E.-M. El Arar et al. “The Positive Effects of Stochastic Rounding in Numerical Algorithms”. In:Proceedings of the 29th IEEE Symposium on Computer Arithmetic. 2022, pp. 58–65.���: 10.1109/arith54963.2022.00018
-
[5]
Bounds on Non-Linear Errors for Variance Computation with Stochastic Rounding
E.-M. El Arar et al. “Bounds on Non-Linear Errors for Variance Computation with Stochastic Rounding”. In:SIAM J. Sci. Comput.46.5 (2024), B579–B599.���:10.1137/23M1563001
-
[6]
StochasticRoundingVarianceandProbabilisticBounds:ANewApproach
E.-M.ElAraretal.“StochasticRoundingVarianceandProbabilisticBounds:ANewApproach”. In:SIAM J. Sci. Comput.45.5 (2023), pp. C255–C275.���:10.1137/22m1510819
-
[7]
Stochastic Models for the Evaluation of Numerical Errors
E.-M. El Arar. “Stochastic Models for the Evaluation of Numerical Errors”. PhD thesis. Uni- versité Paris-Saclay, 2023
2023
-
[8]
ErrorAnalysisofSum-ProductAlgorithmsunderStochasticRound- ing
P.deOliveiraCastroetal.“ErrorAnalysisofSum-ProductAlgorithmsunderStochasticRound- ing”. In:SIAM J. Sci. Comput.47.6 (2025), B1481–B1502.���:10.1137/24M1710966
-
[9]
Precision-Aware Deterministic and Probabilistic Error Bounds For Floating Point Summation
E. Hallman and I. C. Ipsen. “Precision-Aware Deterministic and Probabilistic Error Bounds For Floating Point Summation”. In:Numer. Math.155.1 (2023), pp. 83–119.���:10.1007/ s00211-023-01370-y. Stochastic Rounding 11
2023
-
[10]
Stochastic Rounding and Its Probabilistic Backward Error Analysis
M. P. Connolly et al. “Stochastic Rounding and Its Probabilistic Backward Error Analysis”. In: SIAM J. Sci. Comput.43.1 (2021), A566–A585.���:10.1137/20m1334796
-
[11]
On Stochastic Roundoff Errors in Gradient Descent with Low-Precision Compu- tation
L. Xia et al. “On Stochastic Roundoff Errors in Gradient Descent with Low-Precision Compu- tation”. In:J. Optim. Theory Appl.200.2 (2023), pp. 634–668.���:10.1007/s10957-023- 02345-7
-
[12]
L.Xiaetal.“OntheConvergenceoftheGradientDescentMethodwithStochasticFixed-Point Rounding Errors Under the Polyak–Łojasiewicz Inequality”. In:Comput. Optim. Appl.90.3 (2025), pp. 753–799.���:10.1007/s10589-025-00656-1
-
[13]
StochasticRoundingforLLMTraining:TheoryandPractice
K.Ozkaraetal.“StochasticRoundingforLLMTraining:TheoryandPractice”.In:Proceedings of the 28th International Conference on Artificial Intelligence and Statistics. Ed. by Y. Li et al. Vol. 258. Proceedings of Machine Learning Research. PMLR, 2025, pp. 4402–4410.���: https://proceedings.mlr.press/v258/ozkara25b.html
2025
-
[14]
Probabilistic Error Analysis of Limited-Precision Stochastic Rounding
E.-M. El Arar et al. “Probabilistic Error Analysis of Limited-Precision Stochastic Rounding”. In:SIAM J. Sci. Comput.47.5 (2025), B1227–B1249.���:10.1137/24M1681458
-
[15]
Probabilistic Error Analysis of Limited-Precision Stochastic Rounding: Horner’s Algorithm and Pairwise Summation
E.-M. El Arar et al. “Probabilistic Error Analysis of Limited-Precision Stochastic Rounding: Horner’s Algorithm and Pairwise Summation”. In: (2026). In preparation
2026
-
[16]
Stochastic Rounding: Implementation, Error Analysis and Applications
M. Croci et al. “Stochastic Rounding: Implementation, Error Analysis and Applications”. In: Roy. Soc. Open Sci.9.3 (2022), pp. 1–25.���:10.1098/rsos.211631
-
[17]
Graphcore Limited.Tile Vertex ISA. IPU21. 2023.���:https://docs.graphcore.ai/ projects/isa/en/latest/_static/TileVertexISA-IPU21-1.3.1.pdf
2023
-
[18]
NVIDIA et al.Pretraining Large Language Models with NVFP4. arXiv:2509.25149 [cs.CL]. 2025.���:10.48550/arXiv.2509.25149
-
[19]
2024.���: https://www.amd.com/content/dam/amd/en/documents/instinct- tech- docs/ instruction-set-architectures/amd-instinct-mi300-cdna3-instruction-set- architecture.pdf
Advanced Micro Devices.AMD Instinct MI300 Instruction Set Architecture. 2024.���: https://www.amd.com/content/dam/amd/en/documents/instinct- tech- docs/ instruction-set-architectures/amd-instinct-mi300-cdna3-instruction-set- architecture.pdf
2024
-
[20]
Mellempudi et al.Supporting 8-Bit Floating Point Format Operands in a Computing Ar- chitecture
N. Mellempudi et al.Supporting 8-Bit Floating Point Format Operands in a Computing Ar- chitecture. Patent Status: Pending. 2025.���:https://patents.google.com/patent/ US20250321736A1
2025
-
[21]
Micikevicius et al.OCP 8-bit Floating Point Specification (OFP8)
P. Micikevicius et al.OCP 8-bit Floating Point Specification (OFP8). Version 1.0. 2023. ���:https : / / www . opencompute . org / documents / ocp - 8 - bit - floating - point - specification-ofp8-revision-1-0-2023-12-01-pdf-1
2023
-
[22]
���:https://docs.nvidia.com/cuda/pdf/ptx_isa_9.1.pdf
NVIDIACorp.ParallelThreadExecution(PTX)ISAVersion9.1.Tech.rep.Release9.1.2026. ���:https://docs.nvidia.com/cuda/pdf/ptx_isa_9.1.pdf
2026
-
[23]
B. D. Rouhani et al.OCP Microscaling Formats (MX) Specification. Version 1.0. 2023.���: https://www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0- spec-final-pdf
2023
-
[24]
Luo et al.Ascend HiFloat8 Format for Deep Learning
Y. Luo et al.Ascend HiFloat8 Format for Deep Learning. arXiv:2409.16626 [cs.LG]. 2024. ���:10.48550/ARXIV.2409.16626
-
[25]
S.-E. Chang et al. “ESRU: Extremely Low-Bit and Hardware-Efficient Stochastic Rounding Unit Design for Low-Bit DNN Training”. In:Proceedings of the 2023 Design, Automation & Test in Europe Conference & Exhibition. 2023, pp. 1–6.���:10.23919/DATE56975.2023. 10137222
-
[26]
A Stochastic Rounding-Enabled Low-Precision Floating-Point MAC for DNN Training
S. B. Ali et al. “A Stochastic Rounding-Enabled Low-Precision Floating-Point MAC for DNN Training”. In:Proceedings of the 2024 Design, Automation & Test in Europe Conference & Exhibition. 2024, pp. 1–6.���:10.23919/DATE58400.2024.10546735
-
[27]
YouAlreadyHaveIt:AGenerator-FreeLow-PrecisionDNNTrainingFramework UsingStochasticRounding
G.Yuanetal.“YouAlreadyHaveIt:AGenerator-FreeLow-PrecisionDNNTrainingFramework UsingStochasticRounding”.In:ComputerVision–ECCV2022.Ed.byS.Avidanetal.Cham: Springer-Verlag, 2022, pp. 34–51.���:10.1007/978-3-031-19775-8_3
-
[28]
V. Mishra et al. “Dual-Mode Rounding Algorithms and Hardware for Posit-Based DNN Train- ing:TheFutureofMixedPrecisionFrameworks”.In:ACMTrans.Embed.Comput.Syst.(2025). Just Accepted.���:10.1145/3772092
-
[29]
J. E. Parra Osorio et al.Supporting 8-Bit Floating Point Format for Parallel Computing And StochasticRoundingOperationsinaGraphicsArchitecture.PatentStatus:Pending.2023.���: https://patents.google.com/patent/US20250110741A1. 12 El-Mehdi El Arar, Massimiliano Fasi, Silviu-Ioan Filip, and Mantas Mikaitis
2023
-
[30]
Fu et al.Hardware-Accelerated Random Number Generation
F. Fu et al.Hardware-Accelerated Random Number Generation. Patent Status: Pending. 2025. ���:https://patents.google.com/patent/DE102025104339A1/en
2025
-
[31]
Shah et al.Stochastic Rounding Circuit
S. Shah et al.Stochastic Rounding Circuit. Patent Status: Pending. 2023.���:https : / / patents.google.com/patent/US20250130769A1
2023
-
[32]
���:https://patents.google.com/patent/DE102024129914A1/en
Q.ChangandS.YU.ParallelDeterministicStochasticRounding.PatentStatus:Pending.2024. ���:https://patents.google.com/patent/DE102024129914A1/en
2024
-
[33]
R. L. Leshem et al.Reproducible Stochastic Floating-Point Rounding. Patent Status: Pending. 2025.���:https://patents.google.com/patent/DE102025109725A1/en
2025
-
[34]
Oltchik et al.Reproducible Stochastic Rounding for Network-Internal Computing
Y. Oltchik et al.Reproducible Stochastic Rounding for Network-Internal Computing. Patent Status:Pending.2025.���:https://patents.google.com/patent/DE102025119312A1/ en
2025
-
[35]
A.Tsengetal.TrainingLLMswithMXFP4.arXiv:2502.20586[cs.LG].2025.���:10.48550/ arXiv.2502.20586
Pith/arXiv arXiv 2025
-
[36]
Micikevicius et al.Mixed Precision Training
P. Micikevicius et al.Mixed Precision Training. arXiv:1710.03740 [cs.AI]. 2017.���:10 . 48550/arXiv.1710.03740
Pith/arXiv arXiv 2017
-
[37]
FP4 All the Way: Fully Quantized Training of Large Language Models
B. Chmiel et al. “FP4 All the Way: Fully Quantized Training of Large Language Models”. In: Proceedings of the 39th Annual Conference on Neural Information Processing Systems. 2025. ���:https://openreview.net/forum?id=kuzye4EPLR
2025
-
[38]
Y. Chen et al.TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscil- lation Suppression and Outlier Control. arXiv:2510.27527 [cs.LG]. 2025.���:10.48550/ arXiv.2510.27527
Pith/arXiv arXiv 2025
-
[39]
Oscillation-Reduced MXFP4 Training for Vision Transformers
Y. Chen et al. “Oscillation-Reduced MXFP4 Training for Vision Transformers”. In:Pro- ceedings of the 42nd International Conference on Machine Learning. 2025.���:https : //openreview.net/forum?id=LUFPNGiCUw
2025
-
[40]
HyperBlockFloatingPoint:GeneralisedQuantizationScheme for Gradient and Inference Computation
M.GennaridoNascimentoetal.“HyperBlockFloatingPoint:GeneralisedQuantizationScheme for Gradient and Inference Computation”. In:Proceedings of the 2023 IEEE/CVF Winter Con- ferenceonApplicationsofComputerVision.2023,pp.6353–6362.���:10.1109/WACV56688. 2023.00630
-
[41]
TrainingwithFewerBits:UnlockingEdgeLLMsTrainingwithStochasticRound- ing
T.Liuetal.“TrainingwithFewerBits:UnlockingEdgeLLMsTrainingwithStochasticRound- ing”.In:FindingsoftheAssociationforComputationalLinguistics.Ed.byC.Christodoulopou- los et al. Suzhou, China: Association for Computational Linguistics, 2025, pp. 14531–14546. ���:10.18653/v1/2025.findings-emnlp.784
-
[42]
Multi-Timescale Synaptic Plasticity on Analog Neuromorphic Hardware
A. Atoui et al. “Multi-Timescale Synaptic Plasticity on Analog Neuromorphic Hardware”. In:2025 Neuro Inspired Computational Elements (NICE). 2025, pp. 1–9.���:10 . 1109 / NICE65350.2025.11065914
arXiv 2025
-
[43]
A Hardware-Efficient Spiking Predictive Coding Architecture with Online STDP Learning
J.-G. Kim et al. “A Hardware-Efficient Spiking Predictive Coding Architecture with Online STDP Learning”. In:SSRN Electron. J.(2025).���:10.2139/ssrn.5699160
-
[44]
StochasticRoundingforMemory-EfficientDigitalSimulationofSynap- ticPlasticityUsing8-BitFloating-Point
P.Urbizagasteguietal.“StochasticRoundingforMemory-EfficientDigitalSimulationofSynap- ticPlasticityUsing8-BitFloating-Point”.In:Neuromorph.Compt.Eng.5.3(2025),p.034014. ���:10.1088/2634-4386/ae01d2
-
[45]
PeriodicOrbitsinChaoticSystemsSimulatedatLowPrecision
M.Klöweretal.“PeriodicOrbitsinChaoticSystemsSimulatedatLowPrecision”.In:Sci.Rep. 13.1 (2023), p. 11410.���:10.1038/s41598-023-37004-4
-
[46]
Climate-Change Modelling at Reduced Floating-Point Precision With Stochastic Rounding
T. Kimpson et al. “Climate-Change Modelling at Reduced Floating-Point Precision With Stochastic Rounding”. In:Q. J. R. Meteorol. Soc.149.752 (2023), pp. 843–855.���:10 . 1002/qj.4435
2023
-
[47]
T.Creavin.TowardsStochasticRoundingforScientificApplications.MasterThesis.2025.���: https://www.creavin.ie/data/towards_sr_thesis.pdf
2025
-
[48]
StochasticRoundingImplicitlyRegularizesTall-and-ThinMatrices
G.Dexteretal.“StochasticRoundingImplicitlyRegularizesTall-and-ThinMatrices”.In:SIAM J. Matrix Anal. Appl.46.1 (2025), pp. 341–369.���:10.1137/24M1647679
-
[49]
PyTracer: Automatically Profiling Numerical Instabilities in Python
Y. Chatelain et al. “PyTracer: Automatically Profiling Numerical Instabilities in Python”. In: IEEE Trans. Comput.72.6 (2023), pp. 1792–1803.���:10.1109/TC.2022.3224377
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.