REVIEW 4 major objections 5 minor 1 cited by
Statistical Quality and Reproducibility of Pseudorandom Number Generators in Machine Learning technologies
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper claims that PRNGs labeled 'crush-resistant'—PCG and Philox—fail some BigCrush tests in about 30% of randomly seeded streams, and that their integrations in NumPy, TensorFlow, and PyTorch can fail different tests than the…
desk verdict The headline claim is statistical noise; the real value is the cross-framework anomaly detection. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing instrument is the TestU01 BigCrush battery, 106 empirical statistical tests run on 896 independent, randomly spaced seeds per generator. The comparison is organized around failure profiles: how many tests fail per stream, which named tests fail, and whether failures survive a conservative threshold of $p < 10^{-15}$. The paper also uses a multiple-testing argument—under the null, about 27.4% of 160-test runs should show a suspicious p-value by chance—to separate systematic weaknesses from expected fluctuations. The 'headroom' concept, borrowed from the PCG report, is used to rule out small-state explanations, since all tested generators have at least 64-bit states.
What would settle it
Run the exact 896 seeds through each framework's public API and through the reference C code while recording raw 32-bit outputs; if TensorFlow's Philox, when driven with keys identical to the reference, no longer fails SerialOver tests 1 and 2, the framework-specific failure is an artifact of seeding or wrapper code rather than of the Philox algorithm. Alternatively, show that the reference Philox32 fails SerialOver tests under any seed, which would overturn the paper's integration-difference claim.
Extended reading notes
Core claim
The central claim is that statistical quality claims attached to PRNGs do not survive contact with large-scale empirical testing or with framework integration. On the paper's evidence, PCG32 and Philox32—both advertised as BigCrush-resistant—each fail at least one BigCrush test in roughly 30% of the 896 streams tested, with individual streams failing as many as four tests. The paper also claims that the placement of failures differs between the reference C code and the framework versions: TensorFlow's Philox fails SerialOver tests 1 and 2 while reference and NumPy Philox do not, and NumPy's MT19937 systematically fails only one of the two LinearComp tests that reference MT fails consistently. These differences are read as evidence of implementation-specific deviations introduced during integration, with consequences for reproducibility across ML platforms.
Load-bearing premise
All generators were compared under identical conditions—same output word size, same BigCrush configuration, same amount of data per stream, and seeds initialized through each framework's official seeding API—so the observed differences are attributed to the implementations themselves, not to the testing setup.
Editorial extensions
If this is right
- A 'passes BigCrush' label should be understood as a property of a specific implementation and seed path, not of an algorithm family.
- ML framework maintainers can use this comparison protocol—896 streams through BigCrush—as a regression test when changing PRNG code or seeding behavior.
- Users who care about cross-framework reproducibility cannot assume NumPy, PyTorch, and TensorFlow generate equivalent randomness from the same algorithm; the differences are detectable by statistical tests even if they rarely affect training.
- At the conservative p-value threshold, reference PCG32 and Philox32 show zero failures, so the practical quality of the algorithms remains high; the actionable risk is implementation fidelity, not algorithmic weakness.
Reading between the lines
- Beyond the paper's claims, the same protocol could be applied to other language bindings or GPU libraries where a PRNG is re-implemented or wrapped; the TensorFlow SerialOver anomaly suggests every such wrapper deserves its own BigCrush run.
- The NumPy MT result—failing only one of two linear-complexity tests—would be cheap to probe directly by comparing raw output words and seed-to-state expansion against the reference; if NumPy truncates or skips output, that would explain the asymmetry.
- A testable extension would be to measure whether the framework-level failures actually correlate with training outcomes such as dropout masks or initialization, since the paper only tests the raw number streams, not model behavior.
- The multiple-testing estimate implies that one should expect any long-run test campaign to turn up occasional failures; a more useful reproducibility metric may be the stability of the failing-test set across many seeds rather than the count of failures.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper compares the statistical quality of three pseudorandom number generators (MT19937, PCG, Philox) as implemented in PyTorch, TensorFlow, and NumPy against their C reference implementations, using the TestU01 BigCrush suite on 896 independently seeded streams per generator. The authors report average numbers of failed tests per stream, observe that roughly 30% of PCG and Philox streams fail at least one BigCrush test, and identify implementation-specific anomalies, notably TensorFlow's Philox failing SerialOver tests and NumPy's MT19937 failing only one of the two linear-complexity tests that the reference and PyTorch implementations both fail.
Significance. The paper addresses a relevant and understudied topic: the statistical quality and reproducibility of PRNGs inside ML frameworks. Its strengths are the large number of tested streams (896 per generator), the public availability of the data on GitHub, and the direct comparison of framework integrations with reference C implementations. If the implementation-specific anomalies were verified, they would be a useful contribution to the reproducibility literature. However, the central claim that 'crush-resistant' generators PCG and Philox fail BigCrush tests is not supported by the paper's own conservative analysis, and the main conclusions are currently overstated because the raw failure counts are interpreted without proper multiple-testing correction.
major comments (4)
- [Section 4, Table 1, and Figures 1–3] The claim that PCG and Philox fail BigCrush in roughly 30% of streams contradicts the manuscript's own multiple-testing discussion and is not evidence against these generators' crush-resistance. Under the null hypothesis, with 160 independent p-values and a suspicion threshold of 0.001 or 0.999, the probability of at least one false failure is 1-(1-0.002)^160 ≈ 27.4%, so the observed ~30% rate is consistent with the null. The abstract's statement that these generators 'may fail certain statistical tests' is therefore misleading; Table 2, using the conservative p < 1e-15 filter, shows zero average failures for Pcg32, numpyPcg, NumpyPhilox, and Philox32. The authors should revise the abstract, Section 4, and the conclusion to base their claims on the conservative filter, and should not use the unfiltered failure counts to question the generators' advertised properties.
- [Section 4 (multiple-testing paragraph) and Section 1] The number of statistical tests in BigCrush is stated inconsistently: Section 1 and parts of Section 4 say '106 tests,' while the multiple-testing calculation uses 'approximately 160 individual tests.' If only 106 tests are run, the expected false-failure probability is 1-(1-0.002)^106 ≈ 19.1%, not 27.4%. The authors must report the exact number of independent p-values produced by their BigCrush configuration and recompute the null expectation accordingly; otherwise the claimed match between observed and expected failure rates is not established.
- [Section 3 and 4, test configuration] The manuscript does not report the exact TestU01 configuration, including the output bit-width (e.g., 32-bit vs 64-bit samples), the amount of data consumed per stream relative to BigCrush's nominal requirement of slightly more than 2^38 numbers, the seeding procedures for the reference implementations, and the precise versions of PyTorch, TensorFlow, NumPy, and TestU01. Without this information, the observed differences between framework and reference implementations (TensorFlow's SerialOver failures, NumPy-MT's single LinearComp failure) cannot be attributed to implementation differences rather than to differences in the testing harness, output formatting, or seed-stream derivation. This information is essential for the paper's reproducibility claims and for any reader to interpret the comparison.
- [Section 4, Table 2 and the implementation-difference discussion] After the conservative p < 1e-15 filter, the only surviving candidate anomalies are the TensorFlow SerialOver failures and the NumPy-MT behavior of failing only one of the two LinearComp tests. These are presented without confidence intervals, effect sizes, or a formal comparison of failure rates across implementations. Because these anomalies are now the load-bearing evidence for the paper's main contribution, the authors should provide additional evidence: replicate the runs over more streams, report the distribution of p-values for the specific tests, and rule out the possibility that the anomalies are artifacts of output bit-width or of the seeding transformation used by the frameworks.
minor comments (5)
- [Section 3] The word 'independant' is a typo and should be 'independent.'
- [Figure 5 caption] Figure 5's caption reads 'Number of test failures per random streams distribution, for MT implementations,' but the surrounding text discusses Philox implementations; the caption should be corrected to refer to Philox.
- [Table 1 and Table 2] The column header 'Total files' is unclear; it should be 'Total streams' to match the text.
- [Section 4, Table 2 paragraph] The phrase 'reported by BigCrush as 1-eps' needs a brief explanation that TestU01 represents extremely small p-values in that form, and the threshold selection criterion (p < 1e-15) should be stated precisely so readers understand how the filter was applied.
- [Section 5, Conclusion] The sentence mentioning 'PCG64DXSM' and weaknesses relies on a GitHub issue and a URL as citations; these should be formally cited or removed so that the statement is verifiable.
Circularity Check
No circularity: the analysis relies on the external TestU01 BigCrush benchmark and reference C implementations, with no fitted parameters, self-referential definitions, or load-bearing self-citations.
full rationale
The paper's empirical comparisons are self-contained with respect to circularity concerns. All generator evaluations are measured against TestU01 BigCrush, an external benchmark, and against original C reference implementations; no parameter is fitted to the claimed conclusion, and no result is defined in terms of another result it is supposed to predict. The central claim that some 'crush-resistant' generators fail certain tests is based on counting p-values outside conventional thresholds, but the paper explicitly acknowledges the multiple-testing null in Section 4, computing 1-(1-0.002)^160 ≈ 27.4% expected chance of at least one suspicious p-value across 160 tests. It then applies a conservative p<10^-15 filter in Table 2, where PCG32, Philox32, numpyPcg, and NumpyPhilox all show zero average failures per stream. This acknowledged qualification means the strong 'failure' interpretation is a statistical-inference concern rather than a circular derivation; it does not make the outcome equivalent to the input by construction. The only self-citation, [31], is used to extrapolate that more streams would eventually fail every BigCrush test at least once; that prior study is an external empirical result, not a uniqueness theorem or fitted input, and it is not load-bearing for the main PCG/Philox framework comparison. No circular step is present, so the appropriate score is 0.
Assumptions & free parameters
free parameters (1)
- extreme p-value filtering threshold =
10^-15
assumptions (4)
- standard math TestU01 BigCrush is a valid and reliable statistical test suite for PRNG evaluation.
- domain assumption The 896 streams per generator, seeded by values from Python's built-in random, are independent and representative of each generator's behavior.
- domain assumption The framework implementations (PyTorch MT, NumPy MT/PCG/Philox, TensorFlow Philox) are faithful to the listed algorithms, so differences in failure profiles indicate implementation deviations.
- standard math Multiple testing corrections, such as the 27.4% chance of a false failure over 160 tests, are correctly applied.
Cite this review
Pith. "Pith review of Statistical Quality and Reproducibility of Pseudorandom Number Generators in Machine Learning technologies." pith.science (2026). https://pith.science/paper/MTZIZTXR
@misc{pith2026250703007,
author = {Pith},
title = {Pith review of: Statistical Quality and Reproducibility of Pseudorandom Number Generators in Machine Learning technologies},
year = {2026},
howpublished = {\url{https://pith.science/paper/MTZIZTXR}},
note = {Machine review of arXiv:2507.03007}
}
read the original abstract
Machine learning (ML) frameworks rely heavily on pseudorandom number generators (PRNGs) for tasks such as data shuffling, weight initialization, dropout, and optimization. Yet, the statistical quality and reproducibility of these generators-particularly when integrated into frameworks like PyTorch, TensorFlow, and NumPy-are underexplored. In this paper, we compare the statistical quality of PRNGs used in ML frameworks (Mersenne Twister, PCG, and Philox) against their original C implementations. Using the rigorous TestU01 BigCrush test suite, we evaluate 896 independent random streams for each generator. Our findings challenge claims of statistical robustness, revealing that even generators labeled ''crush-resistant'' (e.g., PCG, Philox) may fail certain statistical tests. Surprisingly, we can observe some differences in failure profiles between the native and framework-integrated versions of the same algorithm, highlighting some implementation differences that may exist.
Figures
Figures from the paper (4 more)
Forward citations
Cited by 1 Pith paper
-
Pseudorandom Streams within Diffusion Models Act as Learnable Inputs That Affect Generation Quality
A diffusion model's training loss and output quality depend measurably on which pseudorandom orbit supplies its randomness, even after marginal-statistics control.
Reference graph
Works this paper leans on
-
[1]
Mersenne Twister: A 623-dimensionally equidistributed uniform pseudo-random number generator,
M. Matsumoto, and T. Nishimura, “Mersenne Twister: A 623-dimensionally equidistributed uniform pseudo-random number generator,” ACM Transactions on Modeling and Computer Simulation , vol. 8, no 1, pp. 3–30, 1998
work page 1998
-
[2]
Parallel random numbers: As easy as 1, 2, 3
J. K. Salmon, M. A. Moraes, R. O. Dror, and D. E. Shaw, “Parallel random numbers: As easy as 1, 2, 3”, in International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1–12, 2011
work page 2011
-
[3]
M. E. O’neill, “PCG: A family of simple fast space-efficient statistically good algorithms for random number generation,” ACM Transactions on Mathematical Software. Claremont, CA: Harvey Mudd College. https://www.cs.hmc.edu/tr/hmc-cs-2014-0905.pdf, 2014
work page 2014
-
[4]
TestU01: AC library for empirical testing of random number generators,
P. L’Ecuyer, and R. Simard, “TestU01: AC library for empirical testing of random number generators,” ACM Transactions on Mathematical Software, vol. 33, no 4, pp. 1–40, 2007
work page 2007
-
[5]
SIMD-oriented fast Mersenne Twister: A 128-bit pseudorandom number generator,
M. Saito, and M. Matsumoto, “SIMD-oriented fast Mersenne Twister: A 128-bit pseudorandom number generator,” in Monte Carlo and Quasi-Monte Carlo Methods, pp. 607–622, 2006
work page 2006
-
[6]
A. Rukhin, J. Soto, J. Nechvatal, M. Smid, E. Barker, S. Leigh, and A. Heckert, “A statistical test suite for random and pseudorandom number generators for cryptographic applications,” National Institute of Standards and Technology, USA, 2001
work page 2001
-
[7]
Comparing search algorithms on the retrosynthesis problem,
M. Roucairol and T. Cazenave, “Comparing search algorithms on the retrosynthesis problem,” Molecular Informatics, vol. 43, p. e202300259, 2024
work page 2024
-
[8]
Replicability is not reproducibility: Nor is it good science,
C. Drummond, “Replicability is not reproducibility: Nor is it good science,” in Proc. Evaluation Methods for Machine Learning Workshop, pp. 1–4, 2009
work page 2009
Show all 32 references
-
[9]
Trust not verify? The critical need for data curation standards in materials informatics,
M. Hart, K. Idanwekhai, V . M. Alves, A. J. Miller, J. L. Dempsey, J. F. Cahoon, and A. Tropsha, “Trust not verify? The critical need for data curation standards in materials informatics,” Chemistry of Materials, vol. 36, pp. 9046–9055, 2024
2024
-
[10]
Reproducibility, replicability and repeatability: A survey of reproducible research with a focus on high performance computing,
B. Antunes and D. R. C. Hill, “Reproducibility, replicability and repeatability: A survey of reproducible research with a focus on high performance computing,” Computer Science Review, vol. 53, p. 100655, 2024
2024
-
[11]
Random number generators in training of contextual neural networks,
M. Huk, K. Shin, T. Kuboyama, and T. Hashimoto, “Random number generators in training of contextual neural networks,” in Asian Conf. on Intelligent Information and Database Systems , pp. 717–730, 2021
2021
-
[12]
Quality of randomness and node dropout regularization for fitting neural networks,
A. Koivu, J. P. Kakko, S. Mäntyniemi, and M. Sairanen, “Quality of randomness and node dropout regularization for fitting neural networks,” Expert Systems with Applications , vol. 207, p. 117938, 2022
2022
-
[13]
A general analysis of example-selection for stochastic gradient descent,
Y . Lu, S. Y . Meng, and C. De Sa, “A general analysis of example-selection for stochastic gradient descent,” in International Conf. on Learning Representations, p. 44, 2022
2022
-
[14]
Depth uncertainty in neural networks,
J. Antorán, J. Allingham, and J. M. Hernández-Lobato, “Depth uncertainty in neural networks,” Advances in Neural Information Processing Systems, vol. 33, pp. 10620–10634, 2020
2020
-
[15]
Data augmentation: A comprehensive survey of modern approaches,
A. Mumuni and F. Mumuni, “Data augmentation: A comprehensive survey of modern approaches,” Array, vol. 16, p. 100258, 2022
2022
-
[16]
Generalizability of machine learning models: Quantitative evaluation of three methodological pitfalls,
F. Maleki, K. Ovens, R. Gupta, C. Reinhold, A. Spatz, and R. Forghani, “Generalizability of machine learning models: Quantitative evaluation of three methodological pitfalls,” Radiology: Artificial Intelligence, vol. 5, no. 1, p. e220028, 2022
2022
-
[17]
Bootstrapping the out-of-sample predictions for efficient and accurate cross-validation,
I. Tsamardinos, E. Greasidou, and G. Borboudakis, “Bootstrapping the out-of-sample predictions for efficient and accurate cross-validation,” Machine Learning, vol. 107, pp. 1895–1922, 2018
1922
-
[18]
A survey of stochastic computing neural networks for machine learning applications,
Y . Liu, S. Liu, Y . Wang, F. Lombardi, and J. Han, “A survey of stochastic computing neural networks for machine learning applications,” IEEE Trans. Neural Netw. Learn. Syst ., vol. 32, no. 7, pp. 2809– 2824, 2020
2020
-
[19]
Bayesian learning for neural networks: An algorithmic survey,
M. Magris and A. Iosifidis, “Bayesian learning for neural networks: An algorithmic survey,” Artificial Intelligence Review, vol. 56, no. 10, pp. 11773–11823, 2023
2023
-
[20]
Recent advances in variational autoencoders with representation learning for biomedical informatics: A survey,
R. Wei and A. Mahmood, “Recent advances in variational autoencoders with representation learning for biomedical informatics: A survey,” IEEE Access, vol. 9, pp. 4939–4956, 2020
2020
-
[21]
Exploration in deep reinforcement learning: A survey,
P. Ladosz, L. Weng, M. Kim, and H. Oh, “Exploration in deep reinforcement learning: A survey,” Information Fusion, vol. 85, pp. 1–22, 2022
2022
-
[22]
Noise optimization in artificial neural networks,
L. Xiao, Z. Zhang, K. Huang, and Y . Peng, “Noise optimization in artificial neural networks,” in IEEE Int. Conf. on Automation Science and Engineering, pp. 1595–1600, 2022
2022
-
[23]
Dynamic energy-accuracy trade-off using stochastic computing in deep neural networks,
K. Kim, J. Kim, J. Yu, J. Seo, J. Lee, and K. Choi, “Dynamic energy-accuracy trade-off using stochastic computing in deep neural networks,” in Annual Design Automation Conf., pp. 1–6, 2016
2016
-
[24]
An energy efficient online learning stochastic computational deep belief network,
Y . Liu, Y . Wang, F. Lombardi, and J. Han, “An energy efficient online learning stochastic computational deep belief network,” IEEE Journal Emerging and Selected Topics in Circuits and Systems, vol. 8, no. 3, pp. 454–465, 2018
2018
-
[25]
Transformer-based generative adversarial networks in computer vision: A comprehensive survey,
S. R. Dubey and S. K. Singh, “Transformer-based generative adversarial networks in computer vision: A comprehensive survey,” IEEE Trans. Artificial Intelligence , vol. 5, no. 10, pp. 4851–4867, 2024
2024
-
[26]
Machine learning needs better randomness standards: Randomised smoothing and PRNG-based attacks,
D. Pranav, I. Shumailov, and R. Anderson, “Machine learning needs better randomness standards: Randomised smoothing and PRNG-based attacks,” arXiv preprint, arXiv:2306.14043, 2023
2023 arXiv
-
[27]
From local pseudorandom generators to hardness of learning,
A. Daniely and G. Vardi, “From local pseudorandom generators to hardness of learning,” in Conf. on Learning Theory, pp. 1358–1394, 2021
2021
-
[28]
Explainable AI models for predicting drop coalescence in microfluidics device,
J. Hu, K. Zhu, S. Cheng, N. M. Kovalchuk, A. Soulsby, M. J. H. Simmons, and R. Arcucci, “Explainable AI models for predicting drop coalescence in microfluidics device,” Chemical Engineering Journal, vol. 481, p. 148465, 2024
2024
-
[29]
Analyzing drop coalescence in microfluidic devices with a deep learning generative model,
K. Zhu, S. Cheng, N. Kovalchuk, M. Simmons, Y . K. Guo, O. K. Matar, and R. Arcucci, “Analyzing drop coalescence in microfluidic devices with a deep learning generative model,” Physical Chemistry Chemical Physics, vol. 25, no. 23, pp. 15744–15755, 2023
2023
-
[30]
Sources of irreproducibility in machine learning: A review,
O. E. Gundersen, K. Coakley, C. Kirkpatrick, and Y . Gil, “Sources of irreproducibility in machine learning: A review,” arXiv preprint, arXiv:2204.07610, 2022
2022 arXiv
-
[31]
Identifying quality Mersenne Twister streams for parallel stochastic simulations,
B. Antunes, C. Mazel, and D. R. C. Hill, “Identifying quality Mersenne Twister streams for parallel stochastic simulations,” in Winter Simulation Conf. (WSC), pp. 2801–2812, 2023
2023
-
[32]
Scrambled linear pseudorandom number generators,
D. Blackman and S. Vigna, “Scrambled linear pseudorandom number generators,” ACM Trans. Math. Softw., vol. 47, no. 4, pp. 1–32, 2021
2021
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.