{"id":"aaee60e3-3f75-4f67-8f9a-76ffe5640e36","arxiv_id":"2505.10689","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An input-adaptive 8-bit quantization scheme estimates activation ranges via a lightweight probabilistic surrogate, achieving near-dynamic accuracy with static-like memory overhead.","lead":"The paper proposes a method that predicts a neural network layer's output range from its input statistics before the layer runs, so 8-bit quantization can adapt to each input with little extra memory. It could make input-adaptive quantization practical on microcontrollers and other memory-limited devices.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (12) as printed is not a variance: it squares Var[y_ijv] and omits the 1/(HWp) normalization. This makes the estimated interval width scale quadratically with input norm, an input-dependent bias that fixed alpha,beta cannot absorb; the released code must be checked against the printed formula.","rationale":"The reader's weakest_assumption is the i.i.d. normal weight assumption, which the paper explicitly acknowledges is not provably true and then mitigates with an extensive empirical evaluation across five tasks. That concern is real but is a modeling assumption whose failure can be partially absorbed by calibrating alpha,beta. The variance aggregation in Eq. (12) is a concrete internal error: even if the weights were exactly i.i.d. normal, the printed formula would not compute a variance. Because the interval endpoints depend on sigma_y, the wrong formula introduces an input-dependent scaling error that calibration cannot absorb. This is more load-bearing than the i.i.d. assumption because it directly undermines the probabilistic derivation in the method section, independent of any empirical validation. The reader did flag Eq. (12) in the rationale as 'looks wrong as written' but did not elevate it to the central load-bearing concern. Since the released code may implement the correct formula, the appropriate verdict remains CONDITIONAL rather than REJECT: the paper must be corrected or the code verified before acceptance. The single concrete test of inspecting the implementation resolves the ambiguity.","tokens_in":13549,"tokens_out":11191,"duration_ms":122024,"concrete_test":"Inspect the released PyTorch implementation of the aggregation step in Section 4.1 and compare it against Eq. (12). If the code uses (1/N) * sum(Var[y_ijv] + (E[y_ijv] - E[y])^2), the printed equation is a typo and the concern is resolved. If the code matches the printed formula, run the ResNet50/ImageNet per-tensor quantization experiment with gamma=1 and check whether top-1 accuracy reproduces the reported 0.7764; also feed a brightness-scaled input and verify whether the estimated interval width grows linearly rather than quadratically with input norm. Quadratic growth would confirm an input-dependent bias that fixed alpha,beta cannot fix.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 4.1, Eq. (12) defines the aggregated variance as sum over v,i,j of (Var[y_ijv]^2 + (E[y_ijv] - E[y])^2). This is not a variance: correct pooling of per-location Gaussian moments is (1/N) * sum over v,i,j of (Var[y_ijv] + (E[y_ijv] - E[y])^2). The printed formula has two errors: it omits the 1/N normalization and squares Var[y_ijv] instead of leaving it linear. The missing 1/N is a per-layer constant scale that calibrated alpha,beta could absorb, but the squaring is not constant. Under Eqs. (10)-(11), Var[y_ijv] scales with the local sum of squared inputs; hence the printed sigma_y = sqrt(Var[y]) scales quadratically with input magnitude, while the true pre-activation range scales approximately linearly. An input twice as bright yields an interval roughly four times as wide, over-quantizing and reducing resolution; a dimmer input yields too narrow an interval and clipping. Because alpha,beta are fixed after calibration, they cannot correct this input-dependent scaling error. The central accuracy results in Tables 1 and 2 rely on this estimator, so unless the released code silently corrects Eq. (12), the probabilistic surrogate is not computing the quantity the paper claims. This concern is independent of whether the i.i.d. normal weight assumption holds; even under exact normality, Eq. (12) as written is incorrect.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a probabilistic surrogate model for the pre-activations of linear and convolutional layers, using the input tensor's sum and sum-of-squares to estimate the mean and variance of the output before the layer executes. These estimates define an asymmetric quantization interval I(α,β) whose parameters α and β are tuned on a calibration set, enabling per-input adaptive quantization with a memory overhead similar to static quantization. The method is evaluated on object detection, segmentation, pose estimation, oriented bounding boxes, and image classification, and is implemented in CMSIS-NN on an STM32L476RG board, with code released for PyTorch and embedded C.","tokens_in":13927,"tokens_out":8758,"duration_ms":75958,"significance":"If the method works as claimed, it addresses a real deployment bottleneck: dynamic quantization's O(HW) output-tensor buffering prevents its use on memory-constrained devices. The paper's strengths include a released implementation for PyTorch and embedded C, on-device latency measurements, and a broad task coverage with out-of-domain evaluation. However, the central estimator contains a clear mathematical error (Eq. (12)), and the method's core assumption of i.i.d. Gaussian weights is unvalidated, so the empirical results must be interpreted with caution.","major_comments":[{"comment":"The aggregated variance as printed is not a variance: it omits the 1/(HWp) normalization and squares the per-entry variance term. The correct pooling of per-location Gaussian moments is (1/(HWp)) Σ_{v,i,j} [Var[y_ijv] + (E[y_ijv] − E[y])²]. Because Var[y_ijv] in Eq. (11) scales with the local sum of squared inputs, the printed σ_y would scale quadratically with input magnitude, a bias that the fixed α,β cannot absorb. Since the results in Tables 1 and 2 rely on this estimator, the formula must be corrected and the released code must be checked against the corrected version.","section":"Section 4.1, Eq. (12)"},{"comment":"The derivation of the surrogate rests on the assumption that 'neural network weights are i.i.d. and normally distributed,' which the paper itself concedes does not provably hold for trained networks. No evidence is provided that the trained weights used in the benchmarks satisfy this assumption (e.g., weight histograms, correlation statistics). A concrete test would be to compare the estimated μ_y and σ_y against the actual pre-activation statistics on the evaluation checkpoints; if the surrogate is biased, the fixed α,β cannot fully absorb the bias.","section":"Section 4.1"},{"comment":"The classification rows mix units: the FP32 column reports top-1 accuracy as a percentage (e.g., 80.858) while all quantized entries are reported as proportions (e.g., 0.7764). This obscures the reported degradations. Moreover, the claim 'Our method is always the second-best' is contradicted by Table 1: for ResNet50 per-tensor, static quantization achieves 0.7797 while the proposed method achieves 0.7764. Please correct the units and the statement.","section":"Tables 1 and 2, Section 6.2"},{"comment":"The tuning procedure for α and β is not described: the target coverage percentage, the search algorithm, and whether coverage is evaluated on actual pre-activations or on the surrogate's predictions are all unspecified. Since α and β are fixed after calibration and directly control the quantization range, the paper should specify this procedure for reproducibility and for judging how sensitive the results are to this choice.","section":"Section 4.1, Eq. (13)"},{"comment":"No error bars or multiple-seed runs are reported for the performance results. Given the sensitivity of quantization to the calibration set (see Fig. 5), the claims of 'negligible loss' and 'best tradeoff' would be considerably strengthened by reporting mean and standard deviation over several independent runs, especially because the reported differences between methods are often within about 0.5%.","section":"Section 6.2, Tables 1 and 2"}],"minor_comments":[{"comment":"The sentence 'Contrary to what we observed in Eqs. (8) and (11)' should likely refer to Eqs. (8) and (9), since the comparison is with the linear layer's variance formula.","section":"Section 4.1, after Eq. (11)"},{"comment":"The color assignments in the caption ('estimation' red, 'convolution' green, 'total' blue) are not consistent with the body text, which describes the overall latency 'in green' and the proposed strategy 'in blue'; please align the caption with the legend.","section":"Figure 3"},{"comment":"The x-axis of Fig. 4 shows a continuous range up to 20, but the text states γ ∈ {1,4,8,16,32}; make the axis discrete or correct the description to match the actual experimental values.","section":"Section 5.3, Figure 4"},{"comment":"The definition of γ ('0 < γ ≤ max((H,W))') is typographically awkward, and the text calls it 'a fraction of the output resolution' although it acts as a sampling stride; please rephrase to avoid confusion.","section":"Section 4.2"},{"comment":"Reference [19] is about LLM post-training and does not clearly support the sentence on domain distribution shifts; consider replacing it with a more directly relevant reference on covariate shift or domain adaptation.","section":"References"},{"comment":"The average degradation figures are reported as 'mAP50-95' percentages but appear to be percentage-point differences; please clarify whether these are relative drops or absolute differences.","section":"Section 6.2"}],"recommendation":"major_revision","confidential_remarks":"The Eq. (12) issue is the main technical blocker: if the released code follows the printed formula, the empirical results in Tables 1 and 2 are invalid. I would ask the authors to correct the formula and confirm the implementation, specify the α,β tuning procedure, and add error bars. The paper's topic fits the journal, but the current form requires substantial revision before it can be considered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the core idea is genuinely useful: they estimate each layer's pre-activation range from a lightweight mean/variance surrogate before the layer runs, which turns dynamic quantization's O(HW) memory overhead into a constant. Second, Eq. (12) as printed is not a variance. It omits the 1/(HWp) normalization and squares Var[y_ijv] instead of keeping it linear. If the released code reproduces that formula, the estimated interval width scales quadratically with input norm, and the calibration-tuned alpha,beta can't fix that input-dependent bias. The stress-test note is correct on this point.\n\nWhat's genuinely new: using the NNGP-style Gaussian propagation as an online per-input quantizer-range estimator, rather than just a theoretical initialization tool. That's a practical twist I haven't seen. They also embed the method in CMSIS-NN, measure actual latency on an STM32 with an oscilloscope, and evaluate across detection, segmentation, pose, OBB, and classification with several models. That's solid engineering work. The code is released.\n\nSoft spots, in proportion. The Eq. (12) issue is load-bearing; the central accuracy tables rely on that estimator. A missing 1/N could be absorbed by alpha,beta, but the squaring cannot. So the first thing to do is check the repository code against the printed formula. If the code is correct, the paper just has a typo; if not, the results need re-running. Second, there are no error bars anywhere, and several numbers in Table 1 look like they mix units (e.g., ResNet50's 80.858% next to 0.7764). Third, the most relevant adaptive-quantization baseline, Liu et al.'s instance-aware dynamic quantization, is cited but not compared. Fourth, the abstract says 'negligible loss' but the reported degradation is up to 1.6% mAP; small, yes, but not negligible. The i.i.d.-Gaussian weight assumption is clearly concessioned, and the empirical evaluation is the right way to support it, so I don't count that against them.\n\nBottom line: this is a promising method with a concrete, fixable issue. Worth a serious referee; I'd send it to review with a request to verify Eq. (12) against the code, add error bars, and add the missing baseline. If the formula is just a typo, the paper is likely acceptable after minor revision.","headline":"A practical input-adaptive quantization scheme with a solid empirical eval, but Eq. (12) as printed is not a variance and needs a code check before trusting the numbers.","tokens_in":14400,"tokens_out":3328,"would_cite":true,"duration_ms":30597,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that per-layer quantization parameters can be estimated before the layer runs from a probabilistic surrogate of pre-activations, making dynamic 8-bit quantization feasible at static-quantization memory cost.","keywords":["Neural network quantization","Dynamic quantization","Probabilistic surrogate","Pre-activation statistics","Memory-efficient inference","Embedded machine learning","Computer vision benchmarks","8-bit inference"],"falsifier":"Take a trained model whose weights are deliberately correlated—for example a low-rank or group-convolution network—compute the surrogate's predicted pre-activation interval for a batch of inputs, and compare it with the actual min and max of the output tensor; if the predicted interval is systematically too narrow or off-center by an amount the calibration parameters $\\alpha, \\beta$ cannot absorb, per-channel accuracy should fall below the static-quantization baseline, which would pin the failure on the Gaussian i.i.d. weight assumption.","tokens_in":13356,"feed_emoji":"🧮","tokens_out":12679,"duration_ms":105094,"temperature":0.7,"pith_summary":"Dynamic quantization is more accurate than static quantization because it measures each input's true activation range, but that measurement forces the device to store an entire layer output, which small embedded systems often cannot afford. The paper claims that the range can instead be predicted before the layer executes, by treating the layer's weights as independent Gaussians and propagating the input through closed-form mean and variance equations for the pre-activations. A short calibration pass fixes an asymmetric interval around those predicted moments, and that interval supplies the scale and zero-point for standard affine quantization. In 8-bit experiments across five vision tasks, the predicted ranges stay within a small margin of full dynamic quantization while using constant memory, and they degrade less than static quantization under input corruption.","feed_headline":"Surrogate statistics bring dynamic quantization to microcontrollers","feed_subtitle":"Predicting layer output ranges keeps accuracy close to dynamic quantization at static-quantization memory cost.","key_machinery":"Pre-activation surrogate: closed-form first and second moments of a layer's output derived from the assumption that weights are i.i.d. normal ($W_{ij} \\sim \\mathcal{N}(\\mu_W, \\sigma_W^2)$). For a linear layer, $\\mathbb{E}[y_j] = \\mu_W \\sum_i x_i$ and $\\mathrm{Var}[y_j] = \\sigma_W^2 \\sum_i x_i^2$; for a convolution, the sums run over the kernel window and input channels. A calibration set is used only to fix $\\alpha$ and $\\beta$, the edges of an asymmetric interval $[\\mu_y - \\alpha\\sigma_y, \\mu_y + \\beta\\sigma_y]$ around the predicted moments; the interval is then treated as the dynamic range in the standard affine quantization formula for scale and zero-point. The sampling stride $\\gamma$ subsamples output positions to reduce estimation cost quadratically.","core_discovery":"The paper's central claim is that the costly part of dynamic quantization—buffering an entire output tensor to measure its actual min and max—can be replaced by a prediction. For a linear layer $y = Wx$ whose weights are modeled as i.i.d. Gaussian, the surrogate gives $\\mathbb{E}[y_j] = \\mu_W \\sum_i x_i$ and $\\mathrm{Var}[y_j] = \\sigma_W^2 \\sum_i x_i^2$, and the same reasoning is extended to convolutions. These two numbers are computed per input before the layer executes, then rescaled by calibrated asymmetry parameters $\\alpha, \\beta$ into a quantization interval. In the authors' 8-bit experiments across detection, segmentation, pose estimation, oriented boxes, and classification, the surrogate-based method is consistently the second-best strategy: average in-domain degradation is $0.88\\%$ per-channel (vs. $0.54\\%$ for full dynamic quantization) and $1.64\\%$ per-tensor (vs. $1.09\\%$), while memory overhead is constant instead of growing with output resolution.","pith_inferences":["The surrogate is cheaper than the layer itself for linear layers because its cost is $O(d)$ and independent of output size; this suggests the same prediction could be reused to pick per-input, per-layer bit-widths, giving input-adaptive mixed precision without extra buffering.","The Gaussian ansatz is a plug-in choice. Because the framework only consumes first and second moments of pre-activations, a sparsity-aware or heavy-tailed model of ReLU outputs could replace Eqs. (8)-(11) and potentially close more of the gap to true dynamic quantization.","The paper demonstrates the trade-off on one microcontroller; a natural extension is a device sweep that maps latency, memory, and accuracy together to show where the method overtakes static or dynamic quantization.","Since the calibration interval is fixed on clean data, an online update of $\\alpha, \\beta$ from the incoming stream of observed pre-activations is a plausible way to recover the remaining out-of-domain gap to dynamic quantization."],"forward_implications":["Dynamic quantization becomes usable on microcontrollers and other working-memory-limited hardware, since the estimation adds a constant $2b'$-bit overhead instead of an $O(HW)$ output buffer.","The quantization latency can be tuned with the sampling stride $\\gamma$ after deployment; in the authors' per-channel tests, accuracy stays nearly flat even when the estimate is subsampled by a factor of 1024.","Calibration is light: on ImageNet with ResNet50, 16 calibration images were enough to reach near-peak per-channel performance.","Under corrupted inputs, the method is more robust than static quantization, with average out-of-domain degradations of $2.58\\%$ per-tensor and $0.99\\%$ per-channel against $5.62\\%$ and $12.6\\%$ for static quantization.","Because the linear-layer estimation cost is $O(d)$ regardless of output length, the method scales to layers with very large output tensors."],"supporting_citations":[{"why":"Supplies the i.i.d. normal weight assumption from which the whole pre-activation surrogate is derived.","marker":"[21]"},{"why":"Establishes the uniform affine quantization scheme and the static/dynamic distinction the paper optimizes.","marker":"[18]"},{"why":"Provides the standard scale/zero-point quantization formulation and the static quantization baseline.","marker":"[30]"},{"why":"Prior instance-aware dynamic quantization that motivates per-input range estimation and serves as the dynamic baseline.","marker":"[28]"},{"why":"Supplies the microcontroller inference kernels used for the on-device latency measurements.","marker":"[20]"},{"why":"Provides the YOLO11 checkpoints used across detection, segmentation, pose, and oriented-box benchmarks.","marker":"[17]"},{"why":"Provides the ResNet50 pretrained model for ImageNet classification.","marker":"[14]"},{"why":"Provides the MobileNetV2 pretrained model as a second classification task.","marker":"[37]"}],"fun_headline_variants":["Surrogate predicts ranges to slash dynamic quant memory","Dynamic quant accuracy at static quant memory cost","Probabilistic surrogate enables per-input dynamic quant","Gaussian surrogate predicts activations for cheap quant","Per-input range prediction cuts quant memory overhead"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"A trained network's weights are close enough to independent and identically distributed Gaussians that each layer's pre-activation mean and variance can be predicted from the input alone; the paper concedes that this does not provably hold for trained networks.","fun_headline_variants_meta":{"raw":{"variants":["Surrogate predicts ranges to slash dynamic quant memory","Dynamic quant accuracy at static quant memory cost","Probabilistic surrogate enables per-input dynamic quant","Gaussian surrogate predicts activations for cheap quant","Per-input range prediction cuts quant memory overhead"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000816,"raw_usage":{"total_tokens":3521,"prompt_tokens":834,"completion_tokens":2687,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":450,"completion_tokens_details":{"reasoning_tokens":2618}},"tokens_in":450,"tokens_out":2687,"duration_ms":20158,"temperature":1.0,"reasoning_tokens":2618,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:05:20.702554+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a trained model whose weights are deliberately correlated—for example a low-rank or group-convolution network—compute the surrogate's predicted pre-activation interval for a batch of inputs, and compare it with the actual min and max of the output tensor; if the predicted interval is systematically too narrow or off-center by an amount the calibration parameters $\\alpha, \\beta$ cannot absorb, per-channel accuracy should fall below the static-quantization baseline, which would pin the failure on the Gaussian i.i.d. weight assumption.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the i.i.d. normal weight assumption from which the whole pre-activation surrogate is derived."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior instance-aware dynamic quantization that motivates per-input range estimation and serves as the dynamic baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the YOLO11 checkpoints used across detection, segmentation, pose, and oriented-box benchmarks."}],"review_version":1}