{"id":"0769be3f-c24b-4572-9545-8587b851198a","arxiv_id":"2505.11170","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Using a rounded Gaussian noise distribution for pseudo-quantization makes low-precision FP weight training stable and cheap, matching or approaching BF16 baseline loss in LLM pretraining.","lead":"The paper introduces a new noise distribution for pseudo-quantization training, letting large language models train with fake low-precision floating-point weights while keeping most of the accuracy of standard BF16 training. It reports overhead as low as 1.4% on A100 GPUs and stable pretraining for models up to 1B parameters and 275 to 300 billion tokens.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Bitwise R distribution is asserted but unverified; FP6/9-bit bounds depend on it.","rationale":"The paper is clear and the experiments are substantial. The reader's conditional verdict is appropriate. I examined the analytical core rather than the empirical extrapolation: the rounded-normal noise distribution is what makes the method FP-friendly, and all of the FP6/FP8 and 9-bit claims are derived from tau = 0 and p ~ 0.717. The bitwise generator in Section 3.4 is the only place where that distribution is linked to an implementation, and it is not specified precisely enough to verify. This is not a claim that the authors are being deceptive; it is a normal preprint gap. The proposed histogram test would quickly resolve it. If the test passes, the concern disappears and the main remaining issue is the preliminary Llama2-1B and FP6/FP8 validation already noted by the reader. Since my concern does not strengthen beyond CONDITIONAL, the reader's verdict stands unchanged.","tokens_in":15925,"tokens_out":18404,"duration_ms":185521,"concrete_test":"Obtain the Triton kernel or exact Boolean circuit, then generate 10^8 R values using the same Philox/Romu path and compute the empirical histogram. Run a chi-square test against Eq. 10 and check that the smallest observed nonzero magnitude is exactly 1; also compute lag-1 and byte-level correlations of the consumed 32-bit words. If the empirical Pr(0) differs from 0.717 by more than about 1e-3, or if any nonzero magnitude below 1 appears, re-derive Lemma 1 and Proposition 3 with the measured tau and p and see whether the FP6/FP8 and 9-bit claims survive.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.4 asserts, without a derivation or pseudocode, that a bitwise circuit over PRNG bits produces the rounded-normal distribution of Eq. 10, and that this holds 'assuming each bit ... independently random.' Every quantitative downstream claim uses the two facts that make this distribution FP-friendly: tau = 0 (min nonzero |R| = 1) and Pr(R = 0) ~ 0.717. Lemma 1's bt < 9 bound for BF16 and Table C.1's FP6/FP8 compatibility are direct consequences. If the implemented Triton kernel consumes 32-bit Philox/Romu words in a way that makes output bits correlated, or if the actual Boolean construction yields a different p (Eq. 10's probabilities are not shown to be realizable with the stated 4-bit packing), then the effective tau or p shifts; the exponent/mantissa requirements in Proposition 3 and the annealing probabilities in Proposition 4 change. The paper reports no histogram, no bit-correlation test, and no released code, so the central theoretical foundation is currently an unsupported assertion.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes Gaussian weight sampling (GaussWS), a pseudo-quantization training (PQT) method in which a rounded-normal noise distribution R is added to weights before floating-point casting, with a learned blockwise bitwidth. The central claims are that the method (1) supports low-precision FP parameters down to FP6 and noise precision up to 9-bit when using BF16 operators, (2) adds only about 1.40% training throughput overhead on A100 GPUs while storing 2 bytes per parameter, and (3) provides stable pretraining that closely follows or outperforms a BF16 baseline on GPT2-124M, Llama2-134M, and Llama2-1B models up to roughly 300B tokens. The paper derives underflow bounds (Lemmas 1 and 2), an FP exponent-cutoff proposition (Proposition 3), and a stochastic precision annealing result (Proposition 4) for the proposed R, and implements R generation with bitwise operations on PRNG outputs.","tokens_in":16127,"tokens_out":6277,"duration_ms":65383,"significance":"If the results hold, GaussWS would be a useful contribution to low-precision LLM training. The theoretical bounds in Section 3.3 are parameter-free given the proposed R, and the bitwise generation scheme is clever and shows large speedups over a PyTorch baseline in unit benchmarks. The stability improvement over DiffQ on GPT2 and Llama2-134M is interesting and empirically meaningful. However, the central empirical validation is incomplete: experiments use BF16 GEMM with FP32 accumulation rather than actual FP6/FP8 datatypes, the main Llama2-1B configuration degrades relative to the BF16 baseline, and the bitwise construction that underpins all theoretical claims is asserted but not validated or released. These gaps prevent the paper from currently supporting its advertised claims at the stated level.","major_comments":[{"comment":"The bitwise construction that is claimed to generate the distribution in Eq. (10) is asserted without pseudocode, a Boolean circuit, or a derivation. The distribution's two key properties, tau = 0 (min nonzero |R| = 1) and Pr(R = 0) ≈ 0.717, directly feed Lemmas 1 and 2, Propositions 3 and 4, the b_t < 9 bound, and the FP6/FP8 compatibility table. If the actual Triton kernel consumes Philox or Romu bits in a way that makes them correlated, or if the realizable probabilities differ from Eq. (10), these bounds shift. The paper should provide the exact construction and validate it empirically with histograms and bit-correlation tests over a large number of samples.","section":"Section 3.4, Eq. (10)"},{"comment":"All reported training runs use BF16 GEMM with FP32 accumulation; no model is trained with actual FP6 or FP8 weight storage and arithmetic. The claim that GaussWS 'supports low-precision FP parameters down to FP6' is therefore an analytical inference from Table C.1, not a demonstrated capability. The authors should either train with a real FP6/FP8 compute path, or clearly restrict the empirical claim to what is actually measured.","section":"Section 4 (experimental setup) and Table C.1"},{"comment":"The Llama2-1B result with the default hyperparameters (b_init = 6, b_target = 4) degrades relative to the BF16 baseline in both average and maximum loss, contradicting the abstract's 'stable' claim. Appendix F reports only a preliminary run with higher bitwidths and states that 'results for GaussWS and DiffQ will be prepared before July 2025.' This is an incomplete experiment in a submitted manuscript; the central stability and scalability claims for 1B-scale models are not yet supported. The authors should complete these runs and incorporate them into the main evaluation, or substantially qualify the claims.","section":"Section 4.1, Figure 4, and Appendix F"},{"comment":"The independence assumption on individual PRNG bits is load-bearing for the theoretical guarantees but is never checked. Philox and Romu are not automatically guaranteed to produce independent low-order bits, and the paper neither cites evidence nor reports a test. Concretely, if the low-order bits are correlated, the effective value of p in Proposition 4 and the effective value of tau in Lemmas 1 and 2 change, and the Table C.1 datatype assignments are no longer justified. A simple statistical test on the generated R values should be included.","section":"Section 3.4 and Eq. (5)"}],"minor_comments":[{"comment":"The 70B† row uses only 4 of 80 model layers, and the geometric-mean overhead reported in the text includes this row. The paper should report the geometric mean for the fully trained configurations separately, or at least state clearly that the 70B† numbers are not a full-model measurement.","section":"Section 4.2, Table 1"},{"comment":"The abstract's phrase 'requiring 2 bytes per parameter in GPU memory' should clarify that this is additional memory for storing the sampled weight w_hat, not the total memory footprint per parameter.","section":"Abstract and Section 3.5"},{"comment":"The distribution in Eq. (10) is introduced as an approximation to floor(N(0,1)/2), but the approximation error is not quantified. The paper should state which distribution, exact or approximate, is assumed in Lemmas 1 and 2 and Propositions 3 and 4.","section":"Section 3.3, Eq. (10)"},{"comment":"The statement that additional results 'will be prepared before July 2025' is not appropriate for a submitted manuscript and should be either completed or removed.","section":"Appendix F"},{"comment":"No code or kernel source is released. Given that the correctness of Section 3.4 depends on the exact bit operations, releasing the Triton implementation would substantially improve verifiability.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is promising, but the two biggest risks are the unverified bitwise distribution in Section 3.4 and the incomplete empirical support for the FP6 and 1B-scale claims. The Llama2-1B default run degrades, and the 'will be prepared before July 2025' sentence in Appendix F suggests the manuscript was submitted before the experimental work was finished. If the authors can provide the exact kernel implementation, validate the distribution of R, and complete the 1B-scale experiments with a config that actually matches the advertised bitwidths, the paper could become acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know up front. First, this is a genuine advance in pseudo-quantization training: the rounded normal distribution R=floor(N(0,1)/2), the underflow analysis that ties required exponent bits to the noise distribution, and the bitwise generation trick are all new relative to DiffQ and NIPQ. The paper is also careful about forward-backward consistency and MX compatibility. Second, the central distribution claim is currently unverified, and the empirical evidence does not yet back the headline FP6/FP8 claims.\n\nThe paper does several things well. The theoretical lemmas are parameter-free and self-consistent given R. The stochastic precision annealing view of Pr(R=0) is a nice explanation for why the method stabilizes training. The experiments are large (300B tokens for GPT2 and Llama2-134M), and the paper is honest about the Llama2-1B degradation, labeling it preliminary and showing a promising higher-bitwidth run in Appendix F. The throughput overhead numbers (1.4-5.3% on the tested sizes) are credible, and the design decisions are reasoned.\n\nThe soft spots are real but fixable. Section 3.4 asserts, without derivation or pseudocode, that a bitwise circuit over PRNG bits produces Eq. 10's distribution. All the FP-friendly properties—tau=0, Pr(0)~0.717, the bt<9 bound, the FP6/FP8 compatibility—hang on that distribution. The paper reports no histogram, no bit-correlation test, and no released code, so the theoretical foundation is currently an assertion. That needs to be pinned down, either with a precise Boolean construction and a proof, or with an empirical distribution check on Philox/Romu.\n\nSecond, the experiments use BF16 GEMM with FP32 accumulation; they never train with actual FP6 or FP8 weights. So \"supports FP6\" is derived, not demonstrated. The authors should either train with those datatypes or soften the language. Also, the loss curves appear to be single-seed with no error bars—standard for runs this large, but worth flagging.\n\nBottom line: this is a serious paper that deserves a real referee. The central idea is plausible and the theory is mostly self-consistent; the gaps are missing evidence, not missing sense. I'd send it out and ask for the distribution verification, an FP6/FP8 training run (or a clear statement that support is analysis-only), and multi-seed or completed higher-bitwidth results. If those land, it's a solid contribution.","headline":"A genuinely new PQT noise distribution and analysis, with a load-bearing unverified bitwise generation claim and no direct FP6/FP8 training—worth refereeing, not yet fully proven.","tokens_in":16711,"tokens_out":2759,"would_cite":false,"duration_ms":25692,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pseudo-quantization training becomes both FP-friendly and efficient when the added noise is rounded normal: the method matches or beats BF16 pre-training of GPT2-124M and Llama2-134M/1B up to 300B tokens while adding as little as 1.4%…","keywords":["pseudo-quantization training","rounded normal noise","low-precision floating-point","stochastic precision annealing","large language model pre-training","bitwise noise generation","FP6","GaussWS"],"falsifier":"Run the bitwise noise generator and empirically measure the distribution of R over a large sample: if Pr(R=0) differs from $\\approx 0.717$ or the nonzero values are not $\\pm1$ and $\\pm2$, then the stochastic-precision-annealing bound in Proposition 4 is broken. A complementary test is to pre-train GPT2-124M twice, once with the bitwise R and once with an exact rounded normal sampled via the standard transform, and check whether the loss curves differ systematically.","tokens_in":15674,"feed_emoji":"🎲","tokens_out":14626,"duration_ms":127256,"temperature":0.7,"pith_summary":"Fully quantized training (FQT) promises cheaper LLM training but suffers from forward/backward inconsistencies and an exponential search over bitwidth assignments, each case requiring over 200 billion tokens to validate. Pseudo-quantization training (PQT) avoids the search by adding noise instead of rounding, yet existing PQT noise is not friendly to floating-point arithmetic. This paper proposes using rounded normal noise, $R = \\lfloor N(0,1)/2 \\rceil$, as the fake-quantization distribution, which leaves the weight untouched with probability about 0.717 and has minimum nonzero magnitude 1. The paper proves that these two properties support bitwidths up to 9-bit with BF16 operators and weights down to FP6, and it shows empirically that the resulting method, GaussWS, stabilizes pre-training of GPT2-124M and Llama2-{134M,1B} up to 300B tokens, matching or beating BF16 baselines with as little as 1.40% training throughput overhead on A100 GPUs.","feed_headline":"Rounded-Gaussian noise lets low-precision FP training beat BF16","feed_subtitle":"A 71.7% zero-probability noise plus bitwise generation keeps overhead to ~1.4% on A100 GPUs.","key_machinery":"The central object is the rounded normal noise distribution $R = \\lfloor N(0,1)/2 \\rceil$ (nearest integer to a standard normal divided by 2), and the mechanism it drives is the FP casting of the fake-quantized weight $\\hat{w} = w + R \\odot \\mathrm{broadcast}_{bl}(\\max_{bl}(|w|)\\cdot 2^{1-b_t})$. The paper reduces the design problem to two parameters: the minimum nonzero magnitude $2^\\tau$ and the zero probability $p$. Lemmas 1 and 2 give inequalities showing that $\\tau$ must be small enough to keep near-zero weights from underflowing and large enough (coupled with $m$, the mantissa bits) to keep the noise itself from underflowing; Proposition 3 converts those bounds into a required exponent-bit count for $w$ and $\\hat{w}$, and Proposition 4 shows that a nonzero $p$ implements stochastic precision annealing. The rounded normal has $\\tau=0$ and $p \\approx 0.717$, which resolves the tension, supports $b_t$ up to 8 with BF16 operators and FP6 at $b_t \\le 4$. Efficient generation replaces the standard normal-sampling transform with bitwise AND/OR combinations of PRNG bits, packed 8 samples to a 32-bit register in a sign-mantissa format.","core_discovery":"The paper claims that pseudo-quantization training (PQT) — replacing quantized weights with a high-precision weight plus differentiable noise — becomes practical for low-precision floating-point formats once the noise is drawn from a rounded normal distribution, $R = \\lfloor N(0,1)/2 \\rceil$, i.e., the nearest integer to a standard normal divided by two. The key insight is that the FP casting of $w + \\text{noise}$ underflows the smaller of the two, so the noise must have a minimum nonzero magnitude that is large enough to survive (Lemma 1) yet small enough to preserve near-zero weights (Lemma 2). A distribution with a high probability $p$ of $R = 0$ resolves the conflict via stochastic precision annealing (Proposition 4): with probability $p$ the weight is kept exact, and with probability $1-p$, weights below a threshold are masked to zero, training the model to tolerate a reduced dynamic range. The rounded normal has $\\tau = 0$ (minimum nonzero magnitude 1) and $p \\approx 0.717$, which the paper proves yields FP exponent and mantissa requirements down to FP6_e3m2 for bitwidth 4 and up to 9-bit noise with BF16 operators. The paper then shows the distribution can be generated by bitwise AND/OR of PRNG bits (8 samples packed in a 32-bit register), avoiding normal-sampling arithmetic, and demonstrates stable pre-training of GPT2-124M and Llama2-134M/1B up to 300B tokens, matching or beating BF16 with as little as 1.40% throughput overhead on A100 GPUs.","pith_inferences":["The same underflow analysis could be carried over to activations and gradients: if a similar rounded distribution works there, weight-only GaussWS would generalize to a full FQT pipeline with a fixed noise family, removing the per-tensor case-search the paper highlights.","The bitwise approximation trades exact normality for speed; a direct head-to-head of the bitwise R against an exact rounded normal (sampled via the standard transform) on the same training runs would quantify how much of the reported behavior depends on the exact distribution shape versus the $\\tau=0$, $p\\approx0.717$ parameters.","The $\\approx 71.7\\%$ zero probability makes GaussWS act like a stochastic regularizer in weight space; if this intuition holds, the method may also improve generalization of low-precision models, a claim the paper does not test.","The paper's own scaling-law discussion implies $b_{target}$ should grow with model size and token count; an immediately testable extension is to anneal $b_{target}$ during training (e.g., from $b_{init}$ to a schedule) rather than fixing it, which could close the Llama2-1B gap at higher token budgets."],"forward_implications":["Bitwidth becomes a trainable hyperparameter rather than a search point: with GaussWS, each 32×32 block has its own $b_t$, updated by gradient, reducing the exponential FQT search to O(1) and removing the need to validate each configuration with 200B-token runs.","The underflow bounds give a direct recipe for picking FP datatypes: for a target bitwidth $b_t$, weights require $\\lceil \\log_2(b_t+1) \\rceil$ exponent bits and sampled weights $\\lceil \\log_2(b_t+3) \\rceil$, which for $b_t=4$ is FP6_e3m2 and for $b_t=9$ is FP16_e5m10.","GaussWS stabilizes pre-training in the two regimes where BF16 diverges: the attention layers around 30B tokens and the feed-forward layers around 200B tokens, and it works with both AdamW and Adam-mini optimizers.","Efficiency is not bought at the cost of accuracy: across GPT2-124M and Llama2-134M/1B, GaussWS matches or exceeds the BF16 loss curve (with the 1B model at $b_{target}=4$ slightly behind, and a preliminary run at $b_{target}=6$ closing the gap), while adding only 2 bytes per parameter of GPU memory and a geometric-mean 3.14% throughput overhead versus 22.34% for uniform-noise PQT."],"supporting_citations":[{"why":"Defines pseudo-quantization training with uniform noise; GaussWS is identical except for the noise distribution R, making DiffQ the central comparison baseline.","marker":"[6]"},{"why":"NIPQ is the other PQT method the paper targets for FP-friendliness and efficiency; the discussion contrasts its overhead and design choices.","marker":"[25]"},{"why":"Documents the 200B-token validation cost and the feed-forward-layer instability that the stability analysis of GaussWS builds on.","marker":"[7]"},{"why":"Normal-sampling transform is the baseline against which the bitwise generator is benchmarked.","marker":"[2]"},{"why":"Philox is the PRNG whose random bits feed the bitwise construction; the assumed bit independence is a property of its output.","marker":"[13]"},{"why":"Romu is the alternative PRNG considered for the bitwise generator; the independence assumption also applies to its output bits.","marker":"[19]"},{"why":"Microscaling (MX) formats define the square-block quantization structure and the FP datatype constraints that the proposed method is designed to satisfy.","marker":"[24]"},{"why":"Stochastic rounding motivates pseudo-quantization noise as a differentiable generalization; the paper positions GaussWS relative to this lineage.","marker":"[29]"}],"fun_headline_variants":["Rounded Gaussian noise makes pseudo-quantization training practical","Low-precision FP training with Gaussian weight sampling beats BF16","Fake rounding: Gaussian noise for efficient FP training","Pseudo-quantization with rounded Gaussian weight sampling","FP training down to 6 bits with Gaussian weight sampling"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The bitwise construction of the noise distribution assumes that every bit of the pseudo-random number generator's output is independent and fair; if bits are correlated, the claimed 71.7% chance of zero and the underflow bounds do not hold.","fun_headline_variants_meta":{"raw":{"variants":["Rounded Gaussian noise makes pseudo-quantization training practical","Low-precision FP training with Gaussian weight sampling beats BF16","Fake rounding: Gaussian noise for efficient FP training","Pseudo-quantization with rounded Gaussian weight sampling","FP training down to 6 bits with Gaussian weight sampling"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000495,"raw_usage":{"total_tokens":2539,"prompt_tokens":1169,"completion_tokens":1370,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":785,"completion_tokens_details":{"reasoning_tokens":1290}},"tokens_in":785,"tokens_out":1370,"duration_ms":12103,"temperature":1.0,"reasoning_tokens":1290,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:56:04.951174+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the bitwise noise generator and empirically measure the distribution of R over a large sample: if Pr(R=0) differs from $\\approx 0.717$ or the nonzero values are not $\\pm1$ and $\\pm2$, then the stochastic-precision-annealing bound in Proposition 4 is broken. A complementary test is to pre-train GPT2-124M twice, once with the bitwise R and once with an exact rounded normal sampled via the standard transform, and check whether the loss curves differ systematically.","supporting_citations":[{"cited_title":"NIPQ: Noise proxy-based Integrated Pseudo-Quantization","cited_arxiv_id":"2206.00820","evidence_quote":"NIPQ is the other PQT method the paper targets for FP-friendliness and efficiency; the discussion contrasts its overhead and design choices."},{"cited_title":"Fishman, B","cited_arxiv_id":null,"evidence_quote":"Documents the 200B-token validation cost and the feed-forward-layer instability that the stability analysis of GaussWS builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Romu is the alternative PRNG considered for the bitwise generator; the independence assumption also applies to its output bits."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Microscaling (MX) formats define the square-block quantization structure and the FP datatype constraints that the proposed method is designed to satisfy."}],"review_version":1}