{"id":"4836f06b-d14a-4054-b11f-8f35d01a2ea4","arxiv_id":"2504.20634","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"Stochastic rounding with few random bits can be systematically biased; the paper derives the bias and proposes a correction that keeps the rounding unbiased.","lead":"Low-precision computer arithmetic often uses stochastic rounding, which adds a little randomness when rounding numbers. This paper shows that when only a few random bits are used, some natural implementations add a hidden systematic error that can make machine learning training diverge, and offers a simple fix.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Bias claims and the SRC correction are defined only under the uniform-interval prior of Eq. (6); a fixed input can show large pointwise bias for SRF and SRC, so transfer to real nonuniform ML inputs is not established.","rationale":"Re-examined §III and Appendix A. The SRFF, SRF, and SRC bias derivations are internally consistent: SRFF's infinite-precision bias −2^{-(N+1)} follows by direct summation, and the finite-precision bounds match the empirical values in Figure 2. SRC's reduction to the N=D' case is valid under Eq. (6), and the paper carefully defines its test distribution family. The weakest point is that the entire quantitative separation among SRFF, SRF, and SRC is an expectation over U(f, succ(f)) (or the finite-precision discrete analog); it is not pointwise unbiasedness, and real ML inputs are not uniform over each interval. A concrete example with δ=0.4, N=2 shows both SRF and SRC have pointwise bias +0.10, so the 'unbiased' label is an aggregate property. This is a scope risk for the central practical claim, not a mathematical error; hence the existing CONDITIONAL verdict remains appropriate, but the proposed real-gradient test would tell whether the concern actually lands in the intended application domain.","tokens_in":10753,"tokens_out":17454,"duration_ms":191683,"concrete_test":"Use the §IV nanoGPT setup (or a small synthetic proxy) to record the actual BFloat16 gradient fractional positions δ before rounding to Binary8P4 over training. Compute the empirical mean of x - round(x) under SRF and SRC with N=3, and compare to the uniform-prior predictions (zero under Eq. (6) for both). If SRC's empirical mean deviates from zero by more than the standard error while SRF's behavior remains close to its predicted bound, the uniform-interval prior is not the operative distribution and the practical claim needs re-scoping; if both match the paper's predictions, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is the unbiasedness criterion in §III, Eq. (6): all bias formulas and the claim that SRC 'corrects' FBSR bias are expectations over U(f, succ(f)), the uniform distribution between adjacent target floats. This is not the prior encountered in QAT. The paper's finite-precision analysis even uses a different object: a discrete uniform over the 2^D source-grid points in an interval, with a no-binade-wrap caveat (§III-E). For a fixed input the aggregate statements fail: with N=2 and fractional position δ=0.4, SRFF has pointwise mean error −0.15, SRF has +0.10, and SRC has +0.10; only after averaging δ over [0,1) do SRF and SRC become 'unbiased'. Thus Eq. (6) does the whole work in separating SRFF/SRF/SRC. If real gradient or weight values are concentrated near tie points, binade boundaries, or other nonuniform regions, the reported bounds and the claimed exactness of SRC do not transfer. The paper states the criterion explicitly, so this is a scope limitation rather than an internal contradiction, but it is the premise on which the practical ML conclusions rest.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies few-bit stochastic rounding (FBSR), in which N random bits are used to round a value from a source format with precision Q = P + D to a target format with precision P, with N < D. It analyzes two natural implementations: SRFF (adding n × 2^{-N}) and SRF (adding (n + 1/2) × 2^{-N}), under the interval-uniform unbiasedness criterion of Eq. (6). It derives exact bias expressions for SRFF and SRF for infinite-precision inputs and bounds for finite-precision inputs, and proposes a correction, SRC, which first rounds to precision P + N and then applies SRFF, claiming exact unbiasedness. Experiments with nanoGPT show that SRFF can diverge in quantized language-model training while SRF and SRC converge.","tokens_in":10970,"tokens_out":27228,"duration_ms":244999,"significance":"The derivations are correct and the experiments reproduce the predicted bias values, which gives the paper a solid core. The practical message is useful: with few random bits, natural stochastic-rounding implementations can carry a systematic bias large enough to affect low-precision training. The proposed SRC correction is simple and plausible. The code and symbolic-algebra derivations are valuable for reproducibility. However, the quantitative claims are tied to the uniform-interval prior of Eq. (6), and the paper would be substantially stronger if it clearly scoped that limitation and supplied a formal proof of SRC's unbiasedness.","major_comments":[{"comment":"The unbiasedness criterion in Eq. (6) is a chosen prior (uniform between adjacent target floats), not pointwise unbiasedness. For a fixed input the paper's statements do not hold: with N = 2 and fractional position δ = 0.4, SRFF has mean error -0.15, SRF +0.10, and SRC +0.10, and only after averaging δ over [0,1) do SRF and SRC become unbiased under this criterion. The language-model experiments use non-uniform weight and gradient distributions, so the theoretical bias values do not transfer directly and the claim that SRC corrects the bias in QAT is not established by the analysis. Please either state explicitly that the bias results are prior-dependent and not worst-case, or provide pointwise worst-case bias expressions or experiments with measured input distributions to support the ML conclusions.","section":"§III, Eq. (6); §IV"},{"comment":"The exactness of SRC is asserted rather than proved, and the phrase 'Round is any unbiased rounding scheme' is too broad. For finite-precision source values on a uniform grid, round-to-nearest with ties-to-even or ties-to-odd is unbiased over the grid, but ties-away-from-zero is not. The paper should include a formal derivation of the unbiasedness of the first rounding in Eq. (8) over the discrete source grid (for example, by pairing the tie contributions in each block of 2^{D-N} source points) and restrict 'Round' to tie-breaking rules that satisfy this discrete unbiasedness.","section":"§III-F, Eq. (8); Appendix A"}],"minor_comments":[{"comment":"The caption says '100,000 samples are rounded with 2 bits of randomness,' but the figure labels and the text use 3 bits; this is inconsistent.","section":"Figure 2 caption"},{"comment":"The panel label 'SR' is inconsistent with the caption and the text, which call this method SRC.","section":"Figure 2, right panel"},{"comment":"The caption contains the apparent debugging artifact 'P+srnumbits=6 =?= 8'; this should be removed.","section":"Figure 2 caption"},{"comment":"The SRF finite-precision bound is tight for N < D, not for N ≤ D; for N = D the bias is zero, so the parenthetical in §III-E should be corrected.","section":"§III-E and Appendix A-D"},{"comment":"The paper should explicitly note in the introduction or at Eq. (6) that its use of 'unbiased' differs from the common pointwise definition of unbiased stochastic rounding, in order to avoid misleading readers who may not notice the interval-uniform prior.","section":"§III and Eq. (6)"}],"recommendation":"major_revision","confidential_remarks":"The paper is essentially sound; the main revision is to scope the claims properly and add the missing proof for SRC. The uniform-prior point is the central issue for the journal audience: the unqualified use of 'bias' and 'unbiased' without a clear statement of the prior is risky, especially because the ML experiments are the main motivation. The minor issues (caption inconsistencies and the tightness claim) are easy to fix. The SRC proof is also straightforward to supply and would remove the main technical gap."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this paper is a solid, small analysis of a real implementation bug in few-bit stochastic rounding, with a simple fix. It deserves a serious referee and likely publication after minor cleanup.\n\nThe new things are: the finite-precision bias bound for SRF, the SRC two-step correction (round to P+N then apply SRFF), and a small QAT demonstration where biased SRFF diverges while SRF/SRC converge. The derivations in §III and Appendix A are correct; I checked the arithmetic and the formulas match the empirical mean values in Figs 1–2. The paper is honest that the infinite-precision SRFF bias was already in El Arar et al.; it frames its contribution as the finite-precision analysis and the mitigation, which is fair.\n\nSoft spots are real but mostly minor. The unbiasedness definition is uniform-between-adjacent-floats; the stress-test is right that a fixed input can show pointwise bias for all three schemes, so the exact numbers don't transfer to arbitrary nonuniform ML tensors. But the paper states the criterion explicitly and doesn't hide it, and the ML experiment is a demonstration, not a claim of universal transfer. The experiments are single-seed, small scale, and the authors say so in the limitations. There are also two annoying inconsistencies: the code availability statement in the abstract says code is available while the conclusion says it will be made available, and Figure 2's caption says \"2 bits\" where the text says 3. These should be fixed but are not substantive.\n\nThe weakness that matters most: the SRC correction's practical value is only as strong as the uniform-interval criterion. If a hardware vendor cares about worst-case bias on adversarial inputs, this paper doesn't answer that. But that's a scope question, not a flaw in the math.\n\nWho should read it: anyone implementing stochastic rounding in hardware or quantized training, and researchers working on low-precision numerics. I'd bring it to reading group. I'd cite it if I write about SR implementations.\n\nFor peer review: send it. The theory is checkable, the correction is new, and the negative result about SRFF is useful even if the ML evidence is modest.","headline":"A correct and useful analysis of few-bit stochastic rounding bias, with a simple fix; the practical transfer rests on a prior the paper states clearly.","tokens_in":11535,"tokens_out":2400,"would_cite":true,"duration_ms":21728,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["65G50","65Y04"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows that few-bit stochastic rounding implementations carry a systematic bias unless corrected, and that a two-step scheme—round to intermediate precision $P+N$, then apply SRFF—restores exact unbiasedness under the paper's…","keywords":["stochastic rounding","few-bit stochastic rounding","floating-point arithmetic","rounding bias","quantization-aware training","low-precision machine learning","random bits","mixed precision arithmetic"],"falsifier":"Generate a large sample of inputs uniformly distributed between two adjacent floats of a target format, round them with SRFF using $N<D$ random bits, and compare the average of $x-\\text{round}(x)$ with the predicted values: $-2^{-(N+1)}$ for exact real inputs, and $(2^{-D}-2^{-N})/2$ for inputs that are $P+D$-bit floats when $N\\le D$. Matching those values to sampling accuracy confirms the bias calculation; a systematic mismatch would show the analysis depends on the assumed input distribution.","tokens_in":10520,"feed_emoji":"🎲","tokens_out":16864,"duration_ms":154156,"temperature":0.7,"pith_summary":"Stochastic rounding rounds a high-precision value up or down randomly, and this paper examines what happens when the supply of random bits is small. The paper proves that if the number $N$ of random bits is less than the precision gap $D$ between the source and target formats, the most natural implementation (SRFF) is systematically biased: with exact real inputs its mean error is $-2^{-(N+1)}$, and with inputs at finite precision its bias is exactly $(2^{-D}-2^{-N})/2$ for $N\\le D$. A common variant (SRF) is unbiased for exact inputs but retains a bias of at most $2^{-(D+1)}$ for finite-precision inputs. The paper's correction, SRC, first rounds the input to an intermediate precision $P+N$ and then applies SRFF, which makes the expected error exactly zero under the paper's uniform-between-floats criterion. The practical stake is that biased few-bit rounding can make quantized language-model training diverge where an unbiased scheme converges.","feed_headline":"Few random bits can bias stochastic rounding and derail training","feed_subtitle":"With N random bits smaller than the precision gap, natural schemes drift; the paper's SRC scheme restores unbiasedness.","key_machinery":"The load-bearing object is the rounding predicate $R(\\delta,n)$, the indicator that $\\delta$ plus a random noise term is at least 1; here $\\delta$ is the fractional position of the input between two adjacent target floats and $n$ is a uniformly random integer with $0\\le n<2^N$. SRFF uses noise $n2^{-N}$, SRF uses $(n+\\tfrac12)2^{-N}$, and SRC applies SRFF after deterministically rounding the input to $P+N$ bits. The analysis reduces unbiasedness to the discrete identity that the average over $n$ of the integral of $R(\\delta,n)$ over $\\delta\\in[0,1]$ equals $1/2$; the paper evaluates that average exactly for each scheme, producing the bias formulas and the tight bounds for $N\\le D$. SRC works because rounding the input to $P+N$ bits makes the effective precision gap equal to $N$, the case in which SRFF's bias is zero.","core_discovery":"Under the paper's uniform-between-floats unbiasedness criterion, the central discovery is that finite randomness, not finite input precision, is what breaks unbiasedness in natural stochastic rounding. SRFF, which rounds when $\\delta + n2^{-N}\\ge 1$, has non-positive pointwise bias: for every fractional position $\\delta$, the expected rounded value is at most $\\delta$. Integrated over a binade, this gives bias $-2^{-(N+1)}$ for infinite-precision inputs and exactly $(2^{-D}-2^{-N})/2$ for $P+D$-bit inputs when $N\\le D$. SRF uses the shifted noise $(n+\\tfrac12)2^{-N}$, which cancels the infinite-precision bias but leaves a finite-precision bias of $2^{-(D+1)}$. The proposed SRC scheme removes the bias structurally: deterministic rounding to $P+N$ bits reduces the problem to the $N=D$ case, for which SRFF is exactly unbiased. The paper demonstrates the practical effect in low-precision language-model training, where SRFF's bias leads to divergence while SRF and SRC converge.","pith_inferences":["A consequence the authors leave implicit: because SRFF's pointwise bias is non-positive on every interval, the direction of its drift does not depend on the input distribution even though the exact magnitude does; any training setup that rounds magnitudes with SRFF should expect a systematic downward pull on weight magnitudes.","The same interval-wise accounting could be applied to other rounding rules, such as ties-to-even stochastic rounding, block-floating-point formats, or non-uniform noise distributions, to test whether the 'round to $P+N$ first' construction remains exactly unbiased in those settings.","A larger-scale test of the thesis would compare SRFF and SRC in a training run while logging per-step sums of weight updates; the bias should appear as a systematic difference between the two update sums that grows with the number of roundings."],"forward_implications":["Designers of low-precision hardware can use the formulas as a design rule: for SRFF, setting the number of random bits equal to the precision gap $N=D$ eliminates the bias, and for $N<D$ the bias is known in advance.","Training runs that quantize weights with biased few-bit rounding can show divergence that is a rounding artifact rather than an optimization failure; comparing SRFF against SRF or SRC in the same setup isolates the effect.","Unbiased few-bit rounding is achievable at any $N$ by adding one deterministic rounding step to $P+N$ bits before applying SRFF, so the cost of unbiasedness is a rounding operation rather than more random bits.","Because SRFF's bias is downward at every fractional position, its effect over a training run is a systematic drift that accumulates across billions of roundings, making it a concern in large-scale training even when the per-rounding bias is tiny."],"supporting_citations":[{"why":"Supplies the theory-and-applications survey of stochastic rounding that frames the few-bit regime and the unbiasedness target.","marker":"[6]"},{"why":"Recent theoretical analysis of stochastic rounding variance and bounds that the paper extends into the few-bit bias setting.","marker":"[7]"},{"why":"Analysis of limited-precision stochastic rounding that motivates the paper's finite-randomness bias calculations.","marker":"[8]"},{"why":"Earlier work that introduces bias into stochastic rounding to reduce variance, contrasted with the paper's bias-removal goal.","marker":"[9]"},{"why":"Defines the quantization-aware training recipe whose experimental setup the paper uses to test practical impact.","marker":"[11]"},{"why":"Provides the small language-model training codebase used in the divergence-versus-convergence experiments.","marker":"[12]"},{"why":"Supplies the library used to implement the SRFF, SRF, and SRC rounding variants in the experiments.","marker":"[13]"},{"why":"Specifies the 8-bit target format with 4-bit precision used in the training experiments and examples.","marker":"[14]"}],"fun_headline_variants":["Few random bits skew stochastic rounding, SRC fixes it","Stochastic rounding bias emerges from finite random bits","SRC scheme restores unbiased rounding with few bits","Natural few-bit rounding biased; SRC converges","Finite randomness, not precision, biases stochastic rounding"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that 'unbiased' means zero mean error under the uniform distribution between each adjacent pair of floats in the target format; if real input distributions are not uniform between adjacent floats, the numeric bias values and the exact unbiasedness of SRC do not transfer directly.","fun_headline_variants_meta":{"raw":{"variants":["Few random bits skew stochastic rounding, SRC fixes it","Stochastic rounding bias emerges from finite random bits","SRC scheme restores unbiased rounding with few bits","Natural few-bit rounding biased; SRC converges","Finite randomness, not precision, biases stochastic rounding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000626,"raw_usage":{"total_tokens":2916,"prompt_tokens":981,"completion_tokens":1935,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":597,"completion_tokens_details":{"reasoning_tokens":1860}},"tokens_in":597,"tokens_out":1935,"duration_ms":15176,"temperature":1.0,"reasoning_tokens":1860,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:25:30.085999+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate a large sample of inputs uniformly distributed between two adjacent floats of a target format, round them with SRFF using $N<D$ random bits, and compare the average of $x-\\text{round}(x)$ with the predicted values: $-2^{-(N+1)}$ for exact real inputs, and $(2^{-D}-2^{-N})/2$ for inputs that are $P+D$-bit floats when $N\\le D$. Matching those values to sampling accuracy confirms the bias calculation; a systematic mismatch would show the analysis depends on the assumed input distribution.","supporting_citations":[{"cited_title":"Stochastic rounding: implementation, error analysis and applications,","cited_arxiv_id":null,"evidence_quote":"Supplies the theory-and-applications survey of stochastic rounding that frames the few-bit regime and the unbiasedness target."},{"cited_title":"Stochastic rounding variance and probabilistic bounds: A new approach,","cited_arxiv_id":null,"evidence_quote":"Recent theoretical analysis of stochastic rounding variance and bounds that the paper extends into the few-bit bias setting."},{"cited_title":"Probabilistic error analysis of limited-precision stochastic rounding","cited_arxiv_id":"2408.03069","evidence_quote":"Analysis of limited-precision stochastic rounding that motivates the paper's finite-randomness bias calculations."},{"cited_title":"Quantization-aware training for large language models with PyTorch,","cited_arxiv_id":null,"evidence_quote":"Defines the quantization-aware training recipe whose experimental setup the paper uses to test practical impact."},{"cited_title":"NanoGPT,","cited_arxiv_id":null,"evidence_quote":"Provides the small language-model training codebase used in the divergence-versus-convergence experiments."},{"cited_title":"GFloat: Generic floating point formats in Python,","cited_arxiv_id":null,"evidence_quote":"Supplies the library used to implement the SRFF, SRF, and SRC rounding variants in the experiments."},{"cited_title":"Interim report on binary floating-point formats for machine learning,","cited_arxiv_id":null,"evidence_quote":"Specifies the 8-bit target format with 4-bit precision used in the training experiments and examples."}],"review_version":1}