Local Stability and Gaussian Smoothing of Quantized Neural Networks

Anton Makarov; Oleg Granichin; Sergey Salishev

REVIEW 2 major objections 6 minor 13 references

Gaussian averaging converts a discontinuous quantized network into an infinitely smooth surrogate, with a local, dimension-dependent error bound whenever the network's output oscillation is controlled.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 10:35 UTC pith:XGIZZHFP

load-bearing objection A clean, honest local error bound for Gaussian smoothing of quantized networks, under a zero-extension caveat that limits its practical reach. the 2 major comments →

arxiv 2607.20153 v1 pith:XGIZZHFP submitted 2026-07-22 cs.LG cs.SYeess.SYmath.OC

Local Stability and Gaussian Smoothing of Quantized Neural Networks

Sergey Salishev , Anton Makarov , Oleg Granichin This is my paper

classification cs.LG cs.SYeess.SYmath.OC MSC 68T0760F05

keywords quantized neural networkslow-bit neural networksGaussian smoothingbounded local oscillationSteklov averagesSobolev convolutionbinary perceptroncentral limit theorem

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper aims to show that a quantized neural network—discontinuous and often piecewise-constant—can be replaced, for analysis and training, by its Gaussian average g(x,s)=E[f(x+ξ)], which is C∞ and has explicit derivative formulas. The main theorem bounds the difference |f−g| locally under a weak regularity condition called bounded local oscillation: if the network's output changes by less than ε2 at scale ε1 near x and is bounded by C, then the error is at most ε2 plus a rapidly decaying Gaussian tail. Because low-bit networks in control and estimation are nonsmooth and block standard stability certificates and gradient tools, this smoothed surrogate offers a route to Lyapunov-style analysis and smooth surrogate gradients without assuming the original model is smooth. The paper also gives closed-form Gaussian averages of ReLU and sign activations and shows, in a high-dimensional binary perceptron, how coordinatewise quantization residuals aggregate through the central limit theorem into a Gaussian envelope with variance Δ²/12.

Core claim

The paper establishes that Gaussian convolution supplies smoothness for free, while bounded local oscillation controls fidelity. For d≥3, if f is bounded by C, has (ε1,ε2)-bounded local oscillation at x, and B(x,ε1) lies inside the domain D, then |f(x)−g(x,s)| ≤ ε2 + 2C·P(|ξ|≥ε1), with the tail probability bounded explicitly by a dimension-dependent expression involving γ^d exp(−γ²/2)/(γ²−d+2) whenever γ²=ε1²/s ≥ d−1. The authors also derive exact averages for ReLU, g(x,s)=xΦ(x/√s)+√sφ(x/√s), and for sign, g(x,s)=2Φ(x/√s)−1, and prove that in a binary perceptron with an i.i.d. additive-noise quantization surrogate, the preactivation perturbation converges to N(0,Δ²/12). This makes Δ²/12 a pr

What carries the argument

The central object is Gaussian averaging g(x,s)=E[f(x+ξ)] with ξ∼N(0,sI), a mollifier that turns any integrable f into an infinitely differentiable function while preserving explicit integral representations of derivatives. The load-bearing argument is Theorem 3's split of the error integral at radius ε1: inside that ball, bounded local oscillation limits the error to ε2; outside, boundedness and a dimension-dependent Gaussian-tail estimate contribute a decaying tail term. The closed-form ReLU and sign averages come from direct Gaussian integration, and the perceptron example uses the classical central limit theorem to aggregate many independent quantization residuals into a Gaussian envelop

Load-bearing premise

The theorem assumes the model is extended by zero outside a bounded domain D and that the oscillation bound holds on a whole ball around the evaluation point; real quantized networks have arbitrary outputs outside the trusted region, so the estimate is local and cannot be read as a global certificate.

What would settle it

At a point x whose Gaussian smoothing cloud has nontrivial probability of leaving the trusted domain D, compute g(x,s) once with the paper's zero-extension and once with the real quantized network's outputs outside D; if the difference is large enough to exceed the Theorem 3 bound, the zero-extension premise is the limiting factor, showing the guarantee is local to D, not global.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

Quantized networks admit a C∞ surrogate with explicit derivatives, so standard smooth-optimization and Lyapunov-style arguments become available on operating regions where bounded local oscillation holds.
Theorem 3 gives a local, dimension-dependent certificate: reducing the output jump scale ε2 tightens the surrogate error, while increasing the smoothing variance s shrinks the tail term only once γ²≥d−1.
Closed-form smoothed ReLU and sign activations provide cheap analytic surrogates for inference-time smoothing and training-time gradient proxies, avoiding explicit Gaussian sampling.
In the binary perceptron, the layer preactivation perturbation is asymptotically N(0,Δ²/12), giving a principled variance scale for Gaussian smoothing in both inference and training.
For multilayer networks the CLT mechanism is only a coarse-grained surrogate, not an exact identity, so its use in deeper models requires empirical validation or a more detailed analysis of latent geometry.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If the same local bound transfers to adversarial or distributional smoothing, then a per-region robustness certificate could be obtained directly from empirical estimates of ε1 and ε2, without retraining; this is an extension the paper does not explicitly pursue.
The CLT mechanism suggests that quantization-noise variance Δ²/12 could be used as a principled, non-heuristic schedule for noise in training surrogates, potentially replacing empirically tuned noise scales in low-bit training; whether this holds in deep networks is an open extrapolation.
Because Theorem 3 requires d≥3, low-dimensional quantized controllers fall outside the promised bound; deriving a dimension-2 analogue would be a useful, concrete test of the proof technique.
The paper's framing implies that tail events—large preactivation excursions caused by quantization—matter more than average quantization error for stability, so measuring tail probabilities of preactivations may be more informative than reporting average signal-to-noise ratios in low-bit inference.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

A clean, honest local error bound for Gaussian smoothing of quantized networks, under a zero-extension caveat that limits its practical reach.

read the letter

Colleague, here's my take on arXiv:2607.20153.

The paper does what it says: it gives a dimension-dependent local error bound for Gaussian smoothing of discontinuous functions under bounded local oscillation, derives closed-form Gaussian averages for ReLU and sign, and adds a CLT story for a binary perceptron. The proof of Theorem 3 is standard – split the integral, bound the near-field by ε2 and the tail by 2C times a chi-square tail, then use a simple majorant for the tail integral. That majorant is correct in the stated regime. The closed forms are textbook. Theorem 9 is an application of the Lindeberg–Lévy CLT to a surrogate noise model. Nothing here is groundbreaking, but it is careful and honest.

The real strength: the paper is explicit about its limits. Remark 4 says the result is local, not global. The model section warns that "one still needs control of off-support behavior." That's more than many such papers do.

The soft spot is the one the stress-test identified: Theorem 3 is stated for the zero-extension of f. The smoothed function g is the convolution of that extension, not the Gaussian average of the actual network. If the network has large outputs outside the trusted domain D, the tail term 2C·P(|ξ|≥ε1) is not valid for the actual network. The paper acknowledges this, but the consequence is that the practical claim—Gaussian smoothing faithfully approximates the quantized model—holds only under an off-support smallness condition that is not verified or even formalized. That's a real gap, but the paper doesn't pretend otherwise.

A second, minor issue: at γ² = d−1, the tail majorant in (7) is (d−1)^{d/2}e^{−(d−1)/2}, which is far larger than 1 for large d, so the bound is vacuous in exactly the high-dimensional regime the perceptron example emphasizes. The bound becomes useful only for γ ≫ √d. This weakens the advertised "dimension-dependent" guarantee.

The CLT is also conditional on an i.i.d. additive-noise surrogate for quantization residuals. The authors call it "explicit" and "consistent with classical quantization model," but it's a modeling assumption, not a theorem about actual quantization. They concede the multilayer version is empirical.

Overall: a decent theoretical building block, no hidden circularity, no inflated claims. The main theorem is correct as a statement about the zero-extension. If the intended use is control certificates on a trusted region where off-support values are controlled, the result is useful. I'd send this to peer review, but I'd ask the authors to either prove a version with a controlled off-support term or state clearly that the theorem only controls the zero-extension. It deserves referee time, though not a strong accept.

Referee Report

2 major / 6 minor

Summary. The paper studies Gaussian averaging as a smooth surrogate for quantized neural networks. The main theoretical result (Theorem 3) is a local, dimension-dependent bound on |f(x)-g(x,s)| under a bounded-local-oscillation condition, for an integrable function extended by zero outside a bounded domain D. The paper also computes the Gaussian averages of ReLU and sign activations in closed form (Theorems 6 and 7) and, for a high-dimensional binary perceptron, proves a CLT for the preactivation under an i.i.d. uniform quantization-noise surrogate (Theorem 9). The authors explicitly position the results as supporting inference-side smoothing and training-side smooth surrogate gradients, with several remarks acknowledging the local and coarse-grained nature of the surrogates.

Significance. If the main bound were directly applicable to actual quantized networks, it would provide a clean, parameter-free connection between Gaussian smoothing and local stability analysis under a weak regularity condition (bounded local oscillation) that does not require continuity. The closed-form ReLU/sign averages are useful elementary identities, and the CLT in Theorem 9 gives a plausible endogenous mechanism for a Gaussian envelope in layer preactivations. The proofs are concise, standard, and appear correct. However, the significance is substantially tempered by the fact that Theorem 3 is stated for a zero-extended function, not for a network defined on all of R^d; this gap makes the central practical claim conditional. The paper is honest about the limitation, but the main theorem as stated does not yet deliver the advertised link to quantized neural networks without additional, unverified assumptions.

major comments (2)

[Section 2 and Theorem 3] Theorem 3 is stated for a function f extended by zero outside the bounded domain D. Consequently, g(x,s) in (1) is the Gaussian average of the zero-extension, not of the actual network, which is defined on all of R^d and can take arbitrarily large values outside D. The proof's tail bound 2C·P(|ξ|≥ε1) in (3) is valid only because f(x+ξ)=0 for |ξ|≥ε1 outside D. For the actual network, the tail integral is ∫_{|ξ|≥ε1} |f(x)-f(x+ξ)| φ_s(ξ) dξ and cannot be bounded by 2C without global control of f. The paper acknowledges this in Remark 4 and Section 2 ('one still needs control of off-support behavior'), but the central claim of the abstract and conclusion that Gaussian smoothing faithfully approximates the discrete model is not supported by the theorem as stated. The fix is straightforward: assume f is globally bounded by C (which holds for typical bounded activations/network outputs) and rep
[Theorem 3, bound at γ²=d−1] The tail bound in (2) is not useful at the minimal admissible γ=√(d−1). Substituting γ²=d−1 gives denominator γ²−d+2=1, and the factor γ^d e^{-γ²/2} / (2^{d/2−1}Γ(d/2)) grows like O(√d) for large d. Hence the bound can exceed the trivial bound ε2+2C, which is always available since |f(x)−f(x+ξ)|≤2C. For the high-dimensional setting (d large), the theorem's condition γ²≥d−1 is therefore insufficient to guarantee a non-vacuous tail estimate; meaningful control requires γ² to exceed d−1 by a positive fraction. The authors should either add a condition of the form γ² ≥ c d with c>1 (or γ²−d sufficiently large) or explicitly note that the bound should be read as min(ε2 + 2C·majorant, ε2+2C). This does not invalidate the proof, but it weakens the claimed 'dimension-dependent guarantee' in the intended high-dimensional regime.

minor comments (6)

[Eq. (14)] The notation in the perceptron definition is malformed: σ(w_j^T x / √n − θ_j) should be written with parentheses or a displayed fraction to avoid ambiguity. The current 'σ( w⊤ j x√n −θ j )' is confusing.
[Lemma 2] The strict inequality in Definition 1 forces the statement '|f(x_i)-f(x_j)| ≤ η+2δ' to be non-strict; the paper notes that one should replace η+2δ by any larger constant for the strict convention. This is correct, but the wording 'replace by any larger constant' could be tightened by defining ε2 = η+2δ+ε' for an arbitrary ε'>0.
[Theorem 9 / Remark 10] The i.i.d. uniform quantization-noise surrogate is an ad hoc model; the paper acknowledges this in Remark 10, but the paragraph 'For any actual fixed input the residuals are deterministic' is important. It would be helpful to explicitly state that Theorem 9 is a probabilistic surrogate statement, not a statement about deterministic quantization error, directly in the theorem statement or immediately after its proof to avoid misreading.
[Numerical validation] The manuscript provides no numerical experiments. Given the practical focus (control, quantized neural networks), a simple experiment on a small quantized network verifying that the local oscillation constants and the bound in Theorem 3 are non-vacuous would substantially increase confidence in the paper's claims. This is not required for mathematical correctness but would strengthen the applied relevance.
[Figures] Figure captions appear as 'Fig. 1. Smoothing of ReLU activation function' and 'Fig. 2. Smoothing of sign activation function' with no reference in the text to these figures; the captions could indicate the choice of s and the comparison. Also, the figures themselves are not visible in the manuscript text, but that is a formatting issue.
[General notation] The paper uses ε1, ε2 in Definition 1 and ϵ1, ϵ2 in Theorem 3 (the same symbols with different glyphs). This is a minor inconsistency that should be unified.

Circularity Check

0 steps flagged

No significant circularity: Theorem 3 is a self-contained approximation bound and Theorem 9 is an explicit surrogate assumption with an independent CLT derivation.

full rationale

The derivation chain is self-contained. Theorem 3 proves a bound on |f−g| directly from the definition of g as a Gaussian convolution (Eq. 1), the (ε1,ε2)-oscillation assumption, and boundedness; the tail estimate is obtained by standard chi-square tail integration (Eqs. 4–7). No fitted parameter is renamed as a prediction, and the bound was not used to define g. The closed-form averages for ReLU and sign (Theorems 6–7) are direct integral evaluations. Theorem 9 invokes the classical CLT after explicitly postulating an i.i.d. uniform additive-noise quantization surrogate; its conclusion is the asymptotic distribution of that postulated surrogate, and the paper repeatedly calls it a 'coarse-grained surrogate architecture, not an exact equality' (Corollary 11, Remark 10). Self-citations to Salishev et al. 2026 and GDNSQ are used as motivation or empirical support ('supported by...', 'provides empirical support'), but they are not load-bearing for Theorem 3 or the CLT. The zero-extension/off-support issue raised by the skeptic is a scoping limitation, acknowledged in Remark 4 ('should not be read as a generic global property'), not a circular reduction: it concerns whether the theorem applies to the original network, not whether its assumptions entail its conclusion. Hence no circular step is present.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 2 invented entities

The paper's main theorem has few free parameters: the oscillation scale and bound are assumed, and the smoothing variance is a user-chosen parameter. The perceptron example introduces an explicit i.i.d. noise surrogate, which is the main ad hoc modeling assumption.

free parameters (3)

ε1, ε2, C = not fitted
These are assumed constants characterizing the function's oscillation and boundedness. They are not fitted to data; they define the class of functions considered.
s (smoothing variance) = s = Δ²/12 in Theorem 9
In the CLT example, the Gaussian envelope variance is set to the variance of the uniform quantization noise, which is a derived quantity (Δ²/12), not a free fitting knob.
Δ (quantization step) = not fitted
It is a hardware parameter of the quantizer model.

axioms (4)

domain assumption f is integrable and bounded by C on D, extended by zero outside D
This is required for the convolution integral to be finite and for the tail bound. It is an assumption, not derived.
domain assumption Bounded local oscillation (Definition 1) holds at the point of interest.
This is the central regularity premise of Theorem 3.
ad hoc to paper In Theorem 9, quantization residuals are modeled as i.i.d. uniform on [−Δ/2, Δ/2].
The i.i.d. additive-noise surrogate is a modeling choice, explicitly acknowledged in the paper. It is not exact for deterministic quantization; it is a surrogate.
standard math Slutsky's theorem and the classical CLT are used without proof.
These are standard results, cited by van der Vaart.

invented entities (2)

i.i.d. additive quantization-noise surrogate no independent evidence
purpose: To derive the Gaussian envelope for layer preactivations via the CLT.
The paper states explicitly this is a surrogate model, not an exact description of deterministic quantization. It is not validated against measured residual statistics.
Gaussian envelope no independent evidence
purpose: Connects the CLT limit to the smoothed activations g(x,s).
It is a mathematical object constructed from the surrogate noise model; no new physical or algorithmic entity with external predictions.

pith-pipeline@v1.3.0-alltime-deepseek · 7899 in / 2997 out tokens · 28248 ms · 2026-08-01T10:35:28.119957+00:00 · methodology

0 comments

read the original abstract

We study Gaussian averaging as a smooth surrogate for quantized neural models. Under bounded local oscillation, we derive a local dimension-dependent bound on |f-g|, linking Gaussian smoothing to the stability analysis of discontinuous networks. We compute closed-form Gaussian averages of the rectified linear unit (ReLU) and sign activation functions, and illustrate the mechanism on a high-dimensional binary perceptron, where layer-preactivation aggregation under an explicit quantization-noise surrogate yields the Gaussian envelope used in inference-side smoothing and training-side smooth surrogate gradients.

Figures

Figures reproduced from arXiv: 2607.20153 by Anton Makarov, Oleg Granichin, Sergey Salishev.

discussion (0)

Reference graph

Works this paper leans on

13 extracted references · 1 canonical work pages

[1]

Bengio, Y., L´ eonard, N., and Courville, A. (2013). Es- timating or propagating gradients through stochas- tic neurons for conditional computation. URL https://arxiv.org/abs/1308.3432

Pith/arXiv arXiv 2013
[2]

Cohen, J., Rosenfeld, E., and Kolter, Z. (2019). Certified adversarial robustness via randomized smoothing. In Proceedings of the 36th International Conference on Machine Learning (ICML), volume 97 of Proceedings of Machine Learning Research, 1310–1320. PMLR

2019
[3]

Esser, S.K., McKinstry, J.L., Bablani, D., Appuswamy, R., and Modha, D.S. (2020). Learned step size quan- tization. In International Conference on Learning Representations (ICLR). ArXiv:1902.08153

Pith/arXiv arXiv 2020
[4]

Granichin, O., Volkovich, Z.V., and Toledano-Kitai, D. (2015). Randomized Algorithms in Automatic Control and Data Mining, volume 67 of Intelligent Systems Reference Library. Springer, Berlin, Heidelberg. doi: 10.1007/978-3-642-54786-7

work page doi:10.1007/978-3-642-54786-7 2015
[5]

and Gimpel, K

Hendrycks, D. and Gimpel, K. (2016). Gaus- sian error linear units (gelus). URL https://arxiv.org/abs/1606.08415

Pith/arXiv arXiv 2016
[6]

Lipshitz, S.P., Wannamaker, R.A., and Vanderkooy, J. (1992). Quantization and dither: A theoretical survey. Journal of the audio engineering society, 40(5), 355–375

1992
[7]

and Akhremchik, I

Salishev, S. and Akhremchik, I. (2025). Gdnsq: Gradual differentiable noise scale quantization for low-bit neural networks. URLhttps://arxiv.org/abs/2508.14004

arXiv 2025
[8]

Granichin, O. (2026). Spsa view on the straight-through estimator in neural network quantization. IEEE Access, 1–1. doi:10.1109/ACCESS.2026.3683267

arXiv 2026
[9]

and Li, W

Slotine, J.J.E. and Li, W. (1991). Applied Nonlinear Control. Prentice Hall

1991
[10]

Spall, J.C. (1992). Multivariate stochastic approximation using a simultaneous perturbation gradient approxima- tion. IEEE Transactions on Automatic Control, 37(3), 332–341. doi:10.1109/9.119632

doi:10.1109/9.119632 1992
[11]

Torop, M., Masoomi, A., Hill, D., Kose, K., Ioannidis, S., and Dy, J. (2023). Smoothhess: ReLU network feature interactions via stein’s lemma. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, 50697–50729. Curran

2023
[12]

van der Vaart, A.W

Associates, Inc. van der Vaart, A.W. (1998). Asymptotic Statistics, volume 3 of Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University

1998
[13]

doi:10.1017/CBO9780511802256

Press, Cambridge. doi:10.1017/CBO9780511802256

doi:10.1017/cbo9780511802256

[1] [1]

Bengio, Y., L´ eonard, N., and Courville, A. (2013). Es- timating or propagating gradients through stochas- tic neurons for conditional computation. URL https://arxiv.org/abs/1308.3432

Pith/arXiv arXiv 2013

[2] [2]

Cohen, J., Rosenfeld, E., and Kolter, Z. (2019). Certified adversarial robustness via randomized smoothing. In Proceedings of the 36th International Conference on Machine Learning (ICML), volume 97 of Proceedings of Machine Learning Research, 1310–1320. PMLR

2019

[3] [3]

Esser, S.K., McKinstry, J.L., Bablani, D., Appuswamy, R., and Modha, D.S. (2020). Learned step size quan- tization. In International Conference on Learning Representations (ICLR). ArXiv:1902.08153

Pith/arXiv arXiv 2020

[4] [4]

Granichin, O., Volkovich, Z.V., and Toledano-Kitai, D. (2015). Randomized Algorithms in Automatic Control and Data Mining, volume 67 of Intelligent Systems Reference Library. Springer, Berlin, Heidelberg. doi: 10.1007/978-3-642-54786-7

work page doi:10.1007/978-3-642-54786-7 2015

[5] [5]

and Gimpel, K

Hendrycks, D. and Gimpel, K. (2016). Gaus- sian error linear units (gelus). URL https://arxiv.org/abs/1606.08415

Pith/arXiv arXiv 2016

[6] [6]

Lipshitz, S.P., Wannamaker, R.A., and Vanderkooy, J. (1992). Quantization and dither: A theoretical survey. Journal of the audio engineering society, 40(5), 355–375

1992

[7] [7]

and Akhremchik, I

Salishev, S. and Akhremchik, I. (2025). Gdnsq: Gradual differentiable noise scale quantization for low-bit neural networks. URLhttps://arxiv.org/abs/2508.14004

arXiv 2025

[8] [8]

Granichin, O. (2026). Spsa view on the straight-through estimator in neural network quantization. IEEE Access, 1–1. doi:10.1109/ACCESS.2026.3683267

arXiv 2026

[9] [9]

and Li, W

Slotine, J.J.E. and Li, W. (1991). Applied Nonlinear Control. Prentice Hall

1991

[10] [10]

Spall, J.C. (1992). Multivariate stochastic approximation using a simultaneous perturbation gradient approxima- tion. IEEE Transactions on Automatic Control, 37(3), 332–341. doi:10.1109/9.119632

doi:10.1109/9.119632 1992

[11] [11]

Torop, M., Masoomi, A., Hill, D., Kose, K., Ioannidis, S., and Dy, J. (2023). Smoothhess: ReLU network feature interactions via stein’s lemma. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, 50697–50729. Curran

2023

[12] [12]

van der Vaart, A.W

Associates, Inc. van der Vaart, A.W. (1998). Asymptotic Statistics, volume 3 of Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University

1998

[13] [13]

doi:10.1017/CBO9780511802256

Press, Cambridge. doi:10.1017/CBO9780511802256

doi:10.1017/cbo9780511802256