REVIEW 3 major objections 6 minor 1 cited by
Online Random Sampling with Real Probabilities
T0 review · 3 major / 6 minor · reviewed 2026-08-02 · deepseek-v4-flash
Pith's one-line read An online sampler can draw exact samples from any changing discrete distribution while losing only O(log n) bits of entropy after n draws, using O(log n) bits of persistent memory.
desk verdict New local-state recycling scheme gets O(log n) entropy loss with O(log n) persistent space; the main caveat is that the runtime/space bounds rest on 'efficient oracle' assumptions that deserve a precise statement in the theorem headers. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the discrete uniform state (Z, M), with Z uniformly distributed in {0,...,M-1}, maintained across rounds. The Recycle subroutine re-encodes the relative position of the uniform variate Q inside the interval [F(X-1), F(X)] into a fresh pair (Z',M'), with M' bounded by a parameter Delta_i that grows as i log i up to a cap. The analysis is carried by the telescoping identity E[T - lg M' | M, X] <= lg(1/f(X)) - lg M + O(lg Delta / Delta), which, summed over rounds, yields the final entropy-loss bound.
What would settle it
Run the sampler on a long stream of Bernoulli(p) draws with an irrational p (e.g. p = 1/pi), with epsilon = 0, and measure total coin flips minus n H(p). The theorem predicts this excess stays O(log n); if it grows linearly with n for any irrational p, the central entropy-loss claim fails.
Extended reading notes
Core claim
For every sequence of discrete distributions with computable-real CDF access, exact online sampling can be achieved with total entropy loss at most epsilon n + O(log n) for any epsilon >= 0, using O(log min{n,1/epsilon}) bits of persistent state; with epsilon = 0 the loss is O(log n). Concretely, the paper proves L_n <= epsilon n + 11 lg(n+2), with persistent space O(log min{n,1/epsilon}), expected temporary space O(log i + E[log X_i]), and oracle calls O(E[log X_i]) per sample; under efficiency assumptions, the runtime per sample is within a constant factor of the oracle cost at the sampled index. The key is that the algorithm recycles randomness locally: after sampling X, the interval [F(X
Load-bearing premise
The algorithm relies on a CDF-oracle model in which every distribution is queried through rational approximations with error < 2^-k, and on the ability to discover a lower bound on each positive atom f(x); if the oracle cannot resolve arbitrarily small atoms at the needed accuracy with controlled expected cost, the O(log n) entropy and space bounds do not follow.
Editorial extensions
If this is right
- Random-bit consumption per sample approaches the information-theoretic minimum (the Shannon entropy of the target) within an additive O(log n) total, for arbitrary non-stationary distributions.
- Persistent memory shrinks from linear in n (global interval methods) to logarithmic, so long-running samplers on memory-constrained devices can maintain near-optimal entropy use.
- The method removes the rational-denominator restriction: distributions with irrational probabilities and countably infinite supports are handled exactly, not approximately.
- The epsilon-tradeoff yields a practical spectrum: set epsilon small to save entropy at the cost of O(log(1/epsilon)) bits of space, or set epsilon = 0 for pure logarithmic loss.
- The space lower bound in the paper shows O(log n + E[max log X_i]) is necessary, so the algorithm is optimal up to constants when the means of the target distributions are polynomially bounded in n.
Reading between the lines
- The same local-recycling trick could be paired with other sampling primitives beyond dichotomous CDF search, potentially reducing entropy loss in statistical computing libraries that currently use rejection or inversion sampling.
- The space-entropy tradeoff suggests an interactive-protocol reading: each sample emits a compact certificate of unused randomness, which could be useful in secure multiparty settings where wasting random bits is costly.
- The telescoping potential-function argument may extend to batching or to streams with a known total length N, where the cap Delta could be tuned to the horizon to shave the log-factor in the entropy loss.
- A testable extension is to implement the oracle for common families (Poisson, discrete Gaussian) and measure wall-clock entropy savings against alias-table and rejection samplers for long streams.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies online exact sampling of a sequence of discrete distributions F_i from an i.i.d. fair-coin source, in a Word RAM model where each CDF is accessed through rational-approximation oracles. It replaces the global arithmetic-decoding interval of Han--Hoshi with a local discrete uniform state (Z_i, M_i), where Z_i | M_i is uniform on {0,...,M_i-1}. Algorithm 3 (IntervalSample plus Recycle) maintains this invariant and extracts the next state from the scaled interval of the sampled atom. Theorem 4.2 gives entropy loss L_n <= eps n + 11 lg(n+2) and persistent space O(log min{1/eps,n}); Proposition 6.1 telescopes the single-step entropy bound. Runtime bounds are derived in Theorems 5.8--5.10 and Corollary 6.2 under accuracy/index-efficiency assumptions (7)--(8) and a uniform moment bound. Section 7 gives a lower bound of Omega(log n + E[max_i log X_i]) for any CDF-oracle sampler. The paper also compares with Han--Hoshi, Kozen--Soloviev, and Draper--Saad.
Significance. If the results hold as stated, this is a substantial advance: it reduces the persistent state from O(n) bits (the global interval method) to O(log n) bits while retaining near-optimal entropy loss, and it extends randomness recycling beyond finite rational distributions with bounded denominator to arbitrary real probabilities and countably infinite supports. The key invariant and the telescoping entropy accounting are elegant and appear sound. The paper is self-contained and explicit: constants such as 5000 and the Delta schedule are concrete, with no fitted parameters, and the lower-bound argument in Section 7 is an independent contribution. The main caveat is that the algorithmic efficiency claims are conditional on oracle-efficiency and moment assumptions that deserve more prominent statement in the main theorems.
major comments (3)
- [§3, Thm 1.2, Cor 6.2, Prop 6.3] The polynomial-runtime and O(log n) overall-space claims are conditional on the accuracy/index-efficiency assumptions (7)--(8) and on a uniform moment bound E[X_i | F_i] <= mu_n. These conditions are not stated in Theorem 1.1/1.2 or Theorem 4.2, and no countably-infinite-support oracle is actually proved to satisfy (7)--(8); Poisson and discretized Gaussian are only named. If C_F(x,nu) is super-polynomial in log nu, or if E[log X_i] is unbounded with n, then Proposition 5.11 and Corollary 6.2 do not give the advertised polynomial runtime/O(log n) space. Please state the conditions in the main theorems and give at least one concrete efficient infinite-support oracle, or explicitly delimit the theorem's scope.
- [§7, Lemma 7.2] The proof writes Pr(tilde Y = z) = prod c_i prod (1-c_i), which requires the outputs Y_i to be conditionally independent. The online-sampling correctness condition—conditional exactness given previous outputs—does imply this, but the manuscript does not say so; as written, the factorization is not justified for an arbitrary sampler satisfying only marginal correctness. Please add an explicit derivation of independence from the online-sampling definition. Also, clarify why an oracle query with precision parameter j forces Omega(j) bits of space; this should be tied to the oracle's return representation (a rational with denominator 2^j) in Section 3.
- [§5.2, Prop 5.11, Cor 6.2] The temporary-space tail bound in Proposition 5.11 is O(log(X + Delta/f(X))), whose expectation is finite only when E[log X] is finite. Theorem 1.2's statement 'expected temporary space is O(log i + E[log X_i])' is vacuous when E[log X_i] = infinity. The abstract's claim of O(log n) expected overall space under 'standard assumptions' is therefore only meaningful under a finite-moment assumption. Please state this assumption explicitly in the theorem statements and clarify that Corollary 6.2's mu_n is an a.s. uniform bound, not merely a mean bound in the usual sense.
minor comments (6)
- [§3, (7)--(8)] Give a compact formal definition of 'efficient oracle' in the text and state that all asymptotic notation in later theorems may depend on the polynomial g in (7)--(8). The current remark at the bottom of page 6 is helpful but should be reflected in the theorem statements.
- [Algorithm 3, lines 27--28] The ceil and floor operations produce U^- and U^+ from rational bounds. It may help readers to note explicitly that these are exact integer computations over O(s + lg M) bits, and that the oracle bounds are dyadic rationals with denominator 2^{s+ceil(lg M)}.
- [Theorem 5.6, Eq. (27)] The expression for E[T - lg M' | M,X] omits the explicit M'=1 case on recycle failure. The formula is correct because lg M'=0 there, but stating this will prevent confusion.
- [Section 7, Theorem 7.3] The lower bound Omega(E[max_i log X_i]) counts the output length as part of the sampler's space usage. This is a modeling choice; please state it explicitly, since some definitions of internal memory exclude the output tape.
- [Table 1] The table notes that the [HH97] persistent-space bound holds in expectation while Algorithm 3's is deterministic; this should be a table footnote or stated next to the relevant rows for emphasis.
- [General] Minor typos and formatting: 'Theorem 4.2 (Restated)' is redundant; the variable 't' is used both for the number of samples and for the time index in Algorithm 2; please disambiguate. Also, the references to [DS26a,b,c] are self-citations and should be accompanied by a short description of their exact relation to this work.
Circularity Check
No significant circularity: the entropy and space theorems are derived from explicit uniform-state invariants; self-citations are comparative or support optimality claims and are not load-bearing for the core derivation.
full rationale
The main derivation chain is self-contained: Proposition 5.1 establishes the interval invariants, Proposition 5.3 derives exactness from uniformity of Q, Proposition 5.5 bounds the tail of the search depth, and Theorem 5.6 combines these to bound E[T - lg M'] in terms of lg(1/f(X)) and an explicit O(lg Delta / Delta) error. Proposition 6.1 and Theorem 4.2 then sum this bound using the explicit schedule Delta_i = 5000 i ceil(lg(i+2)); no parameter is fitted to data or to the targeted entropy loss. The runtime bounds are conditional on the stated oracle-efficiency assumptions (7)-(8), which are model assumptions, not imported results. The only same-author citations are used for comparison ([DS26b]) and for an optimality lower bound in the epsilon-tradeoff regime ([DS26a]); neither is needed to prove correctness, entropy loss, or the O(log n) space/entropy claims of Algorithm 2/3, and Section 7 supplies an independent lower bound for the main logarithmic-space regime. I find no step where a claimed prediction is equivalent by construction to an input, fitted value, or self-citation chain.
Assumptions & free parameters
free parameters (2)
- Delta growth constant (5000 t ceil(lg(t+2))) =
5000 t ceil(lg(t+2))
- Stabilization threshold (m = ceil(0.1/epsilon)) =
ceil(0.1/epsilon)
assumptions (6)
- domain assumption Target distributions are accessed through a CDF oracle phi(i,k) with |F(i) - phi(i,k)| < 2^-k and dyadic denominator 2^k.
- domain assumption The entropy source produces an infinite stream of i.i.d. fair coin flips.
- domain assumption The CDF oracle is efficient and satisfies the accuracy-efficient/index-efficient conditions in (7) and (8).
- domain assumption The target distributions have support N and finite mean when runtime bounds are stated.
- standard math Word RAM arithmetic with word size w; big integers stored in w-bit limbs.
- standard math Standard properties of Shannon entropy, conditional uniformity, and the data-processing inequality.
invented entities (2)
-
Local discrete uniform state (Z_i, M_i) with Z_i | M_i ~ Unif{0,...,M_i - 1}
independent evidence
-
CDF bound oracles F^- and F^+
independent evidence
Cite this review
Pith. "Pith review of Online Random Sampling with Real Probabilities." pith.science (2026). https://pith.science/paper/ZFMMEWK6
@misc{pith2026260713828,
author = {Pith},
title = {Pith review of: Online Random Sampling with Real Probabilities},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZFMMEWK6}},
note = {Machine review of arXiv:2607.13828}
}
abstract
We develop an efficient online algorithm to sample a sequence of discrete random variables using an entropy source of i.i.d. fair coin flips, in a standard model of real computation where real-valued probabilities are represented by rational approximations. For any sequence $F_1, F_2, \dots$ of probability distributions, our sampler generates $n$ outputs $X_1 \sim F_1, \dots, X_n \sim F_n$ using at most $\mathbb{E}\left[H(F_1) +\dots + H(F_n)\right] + O(\log n)$ coin flips in expectation while carrying $O(\log n)$ bits of persistent space, where $H$ is the Shannon entropy. Under standard assumptions, we prove that the space used by our sampler to achieve this information-theoretically optimal entropy rate is asymptotically optimal. The key idea is to replace the global arithmetic-decoding sampling scheme of Han and Hoshi (1997) with a local discrete uniform state, yielding an exponential reduction in space for a given entropy loss. Our approach applies to distributions with irrational probabilities and countably infinite supports, generalizing recent randomness-recycling methods beyond finite rational distributions with bounded denominator.
Figures
Forward citations
Cited by 1 Pith paper
-
Space-Entropy Lower Bounds for Random Sampling
Exact entropy-efficient random sampling with per-sample entropy loss ε requires Ω(log(1/ε)) bits of persistent space, with sharp constants for Bernoulli(1/3) and almost all k-outcome distributions.
Reference graph
Works this paper leans on
-
[1]
Probabilistic
Gill, John Thomas , school =. Probabilistic. 1972 , chapter =
1972
-
[2]
Journal für die reine und angewandte Mathematik (Crelles Journal) , doi =
Theorie der einfachen Ungleichungen , author =. Journal für die reine und angewandte Mathematik (Crelles Journal) , doi =
-
[3]
Information and Computation , volume =
The Functions Erf and Erfc Computed with Arbitrary Precision and Explicit Error Bounds , author =. Information and Computation , volume =. 2012 , publisher =
2012
-
[4]
Proc.\ International Conference on Computational Science and Its Applications (ICCSA) , pages =
Computing the Incomplete Gamma Function to Arbitrary Precision , author =. Proc.\ International Conference on Computational Science and Its Applications (ICCSA) , pages =. 2003 , doi =
2003
-
[5]
, title =
Shannon, Claude E. , title =. The Bell System Technical Journal , volume =. 1948 , publisher =
1948
-
[6]
1953 , publisher =
Statistical Tables for Biological, Agricultural And Medical Research , author =. 1953 , publisher =
1953
-
[7]
Communications of the ACM , volume =
Durstenfeld, Richard , title =. Communications of the ACM , volume =. 1964 , publisher =
1964
-
[8]
Annals of Mathematical Statistics , volume =
The Efficient Construction of an Unbiased Random Sequence , author =. Annals of Mathematical Statistics , volume =. 1972 , publisher =. doi:10.1214/aoms/1177692552 , address =
arXiv 1972
Show all 64 references
-
[9]
Algorithms and Complexity: New Directions and Recent Results , pages =
The Complexity of Nonuniform Random Number Generation , author =. Algorithms and Complexity: New Directions and Recent Results , pages =. 1976 , publisher =
1976
-
[10]
, title =
Walker, Alastair J. , title =. ACM Transactions on Mathematical Software , volume =. 1977 , publisher =
1977
-
[11]
1986 , address =
Non-Uniform Random Variate Generation , author =. 1986 , address =
1986
-
[12]
and Zuckerman, D
Impagliazzo, R. and Zuckerman, D. , title =. Proc.\ 30th Annual Symposium on Foundations of Computer Science (FOCS) , year =
-
[13]
IEEE Transactions on Software Engineering , volume =
A Linear Algorithm for Generating Random Numbers with a Given Distribution , author =. IEEE Transactions on Software Engineering , volume =. 1991 , publisher =
1991
-
[14]
IEEE Transactions on Information Theory , volume =
Interval Algorithm for Random Number Generation , author =. IEEE Transactions on Information Theory , volume =. 1997 , publisher =
1997
-
[15]
Journal of Complexity , volume =
Tomoyuki Yamakami , title =. Journal of Complexity , volume =. 1999 , doi =
1999
-
[16]
and Huber, Mark , booktitle =
Fill, James A. and Huber, Mark , booktitle =. The Randomness Recycler: A New Technique For Perfect Sampling , year =
-
[17]
and Jakobsson, M
Juels, A. and Jakobsson, M. and Shriver, E. and Hillyer, B.K. , journal =. How To Turn Loaded Dice Into Fair Coins , year =
-
[18]
David J. C. MacKay , title =
-
[19]
2004 , month = feb, howpublished =
Willekens, Jacques , title =. 2004 , month = feb, howpublished =
2004
-
[20]
arXiv , volume =
Bit Recycling For Scaling Random Number Generators , author =. arXiv , volume =. 2010 , doi =
2010
-
[21]
An Irregular Mind: Szemer
Trevisan, Luca , title =. An Irregular Mind: Szemer. 2010 , pages =
2010
-
[22]
arXiv , volume =
Optimal Discrete Uniform Generation from Coin Flips, and Applications , author =. arXiv , volume =. 2013 , doi =
2013
-
[23]
Algorithmica , volume =
Bringmann, Karl and Panagiotou, Konstantinos , title =. Algorithmica , volume =. 2017 , publisher =
2017
-
[24]
and Vercauteren, Frederik and Verbauwhede, Ingrid , booktitle =
Roy, Sinha S. and Vercauteren, Frederik and Verbauwhede, Ingrid , booktitle =. High Precision Discrete. 2013 , doi =
2013
-
[25]
ACM Transactions on Modeling and Computer Simulation , volume =
Lemire, Daniel , title =. ACM Transactions on Modeling and Computer Simulation , volume =. 2019 , publisher =
2019
-
[26]
1502.02539v6 , month = nov, year =
Random Variate Generation Using Only Finitely Many Unbiased, Independently and Identically Distributed Random Bits , author =. 1502.02539v6 , month = nov, year =
-
[27]
Proceedings of the ACM on Programming Languages , volume =
Optimal Approximate Sampling from Discrete Probability Distributions , author =. Proceedings of the ACM on Programming Languages , volume =. 2020 , publisher =. doi:10.1145/3371104 , address =
2020 doi
-
[28]
Proc.\ 23rd International Conference on Artificial Intelligence and Statistics (AISTATS) , pages =
The Fast Loaded Dice Roller: A Near-Optimal Exact Sampler for Discrete Probability Distributions , author =. Proc.\ 23rd International Conference on Artificial Intelligence and Statistics (AISTATS) , pages =
-
[29]
Journal of Logical and Algebraic Methods in Programming , volume =
Coalgebraic Tools for Randomness-Conserving Protocols , author =. Journal of Logical and Algebraic Methods in Programming , volume =. 2022 , doi =
2022
-
[30]
arXiv , volume =
Optimal Rolling of Fair Dice using Fair Coins , author =. arXiv , volume =. 2024 , doi =
2024
-
[31]
Software: Practice and Experience , volume =
Brackett-Rozinsky, Nevin and Lemire, Daniel , title =. Software: Practice and Experience , volume =. 2025 , doi =
2025
-
[32]
and Lee, Wonyeol , title =
Saad, Feras A. and Lee, Wonyeol , title =. Proceedings of the ACM on Programming Languages , volume =. 2025 , publisher =
2025
-
[33]
and Saad, Feras A
Draper, Thomas L. and Saad, Feras A. , title =. Proc.\ 2026 Annual ACM-SIAM Symposium on Discrete Algorithms (SODA) , pages =. 2026 , doi =
2026
-
[34]
2507.00915 , month = jul, year =
MichelangeRoll: Sculpting Rational Distributions Exactly and Efficiently , author =. 2507.00915 , month = jul, year =
-
[35]
2020 , publisher =
Foundations of Probabilistic Programming , editor =. 2020 , publisher =
2020
-
[36]
The Discrete
Canonne, Cl\'. The Discrete. Proc.\ 34th International Conference on Neural Information Processing Systems (NeurIPS) ,
-
[37]
Entropy , volume =
Chattopadhyay, Eshan , title =. Entropy , volume =. 2022 , doi =
2022
-
[38]
The Intel® Random Number Generator , author =
-
[39]
Quantum Random Number Generators , author =. Rev. Mod. Phys. , volume =. 2017 , publisher =
2017
-
[40]
and Koch, Julian and Antkowiak, Philipp L
Meiser, Linda C. and Koch, Julian and Antkowiak, Philipp L. and Stark, Wendelin J. and Heckel, Reinhard and Grass, Robert N. , title =. Nature Communications , volume = 11, eid = 5869, month = nov, year =
-
[41]
Monte Carlo Method , editor =
Various Techniques Used in Connection with Random Digits , author =. Monte Carlo Method , editor =. 1951 , publisher =
1951
-
[42]
Iterating von
Peres, Yuval , journal =. Iterating von. 1992 , publisher =
1992
-
[43]
Entropy , volume =
Prasitsupparote, Amonrat and Konno, Norio and Shikata, Junji , title =. Entropy , volume =. 2018 , doi =
2018
-
[44]
A Generalization of
Pae, Sung-Il , journal =. A Generalization of. 2015 , doi =
2015
-
[45]
Randomizing Functions: Simulation of a Discrete Probability Distribution Using a Source of Unknown Distribution , journal =
Pae,. Randomizing Functions: Simulation of a Discrete Probability Distribution Using a Source of Unknown Distribution , journal =. 2006 , volume =
2006
-
[46]
IEEE Transactions on Information Theory , volume =
Pae, Sung-Il , title =. IEEE Transactions on Information Theory , volume =. 2020 , pages =. doi:10.1109/TIT.2019.2962480 , address =
2020
-
[47]
Combinatorica , volume =
Blum, Manuel , title =. Combinatorica , volume =. 1986 , doi =
1986
-
[48]
ACM Transactions on Mathematical Software , volume =
Sampling Exactly from the Normal Distribution , author =. ACM Transactions on Mathematical Software , volume =. 2016 , month = jan, publisher =
2016
-
[49]
Proc.\ 40th International Colloquium on Automata, Languages, and Programming (ICALP) , pages =
Exact and Efficient Generation of Geometric Random Variates and Random Graphs , author =. Proc.\ 40th International Colloquium on Automata, Languages, and Programming (ICALP) , pages =. 2013 ,
2013
-
[50]
and Luk, Wayne , booktitle =
Thomas, David B. and Luk, Wayne , booktitle =. Sampling from the Exponential Distribution Using Independent Bernoulli Variates , year =
-
[51]
Scientific Reports , volume =
True Randomness from Big Data , author =. Scientific Reports , volume =. 2016 , publisher =
2016
-
[52]
Annals of Mathematics , volume =
Integer Multiplication in Time O(n n) , author =. Annals of Mathematics , volume =. 2021 , doi =
2021
-
[53]
2005 , doi =
Randomness Requirements for Security , author =. 2005 , doi =
2005
-
[54]
Granlund, Torbj. GNU
-
[55]
IEEE Transactions on Information Theory , volume =
Efficient Rejection Sampling in the Entropy-Optimal Range , author =. IEEE Transactions on Information Theory , volume =. 2026 , doi =
2026
-
[56]
Generating Random Floating-Point Numbers by Dividing Integers: A Case Study , booktitle =
Goualard, Fr. Generating Random Floating-Point Numbers by Dividing Integers: A Case Study , booktitle =. 2020 , doi =
2020
-
[57]
IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences , volume =
Two Algorithms for Random Number Generation Implemented by Using Arithmetic of Limited Precision , author =. IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences , volume =. 2003 , month = oct, publisher =
2003
-
[58]
Communications in Statistics -- Simulation and Computation , volume =
Mendo, Luis , title =. Communications in Statistics -- Simulation and Computation , volume =. 2025 , publisher =
2025
-
[59]
Information and Computation , volume =
Feldman, David and Impagliazzo, Russell and Naor, Moni and Nisan, Noam and Rudich, Steven and Shamir, Adi , title =. Information and Computation , volume =. 1993 , doi =
1993
-
[60]
, journal =
Böcherer, Georg and Geiger, Bernhard C. , journal =. Optimal Quantization for Distribution Synthesis , pages =. 2016 , doi =
2016
-
[61]
IEEE Transactions on Information Theory , volume =
Approximation Theory of Output Statistics , author =. IEEE Transactions on Information Theory , volume =. 1993 , publisher =
1993
-
[62]
Theory of Computing Systems , volume =
Matias, Yossi and Vitter, Jeffrey Scott and Ni, Wen-Chun , title =. Theory of Computing Systems , volume =. 2003 , publisher =
2003
-
[63]
1991 , doi =
Ko, Ker-I , title =. 1991 , doi =
1991
-
[64]
and Saad, Feras , title =
Draper, Thomas L. and Saad, Feras , title =. 2026 , fnote =
2026
Reviewed August 2, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.