{"id":"55815e88-a322-40ab-902c-505f9336ad3b","arxiv_id":"2607.25494","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"noisefloat integrates CESTAC stochastic validation into NumPy/PyTorch/JAX/TensorFlow and reports significant-digit losses per operator, detecting deliberately unstable operators in controlled image and text benchmarks.","lead":"This paper introduces noisefloat, a Python tool that uses CESTAC stochastic arithmetic to estimate how many significant digits survive in deep-learning operator outputs during training and inference. It is a debugging aid for finding numerically unstable layers in reduced- or mixed-precision models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GEMM surrogate in §3.4 misses kernel-internal accumulation rounding; empirical tests never cover this, so the central detection claim is unproven.","rationale":"The paper's central claim is twofold: (i) first integration of CESTAC into DL deployment, and (ii) effective detection of unstable operators. Claim (i) is plausible given the related-work survey; claim (ii) is the one that requires empirical support. The most load-bearing assumption is that the GEMM surrogate in §3.4 preserves CESTAC's diagnostic power for the most common DL operators (matmul, linear, convolutions). The surrogate only perturbs inputs at magnitude 2^{-p}; it does not randomize rounding inside the kernel. For operations whose instability stems from accumulation rounding in long reductions, the input-perturbation variability is orders of magnitude smaller than the true rounding-error variability, so the tool will report false stability. The paper's own Theorem 3 bounds the gap between surrogate and ideal arithmetic-level CESTAC, but that bound is small precisely when the internal perturbations Δ_p are small; it cannot certify detection of errors that input perturbation does not excite. Section 3.6 explicitly limits the scope to Python-visible operations, yet Section 6 concludes 'effectively detect numerically unstable operators' without this qualifier. All experiments in §4.2–4.4 inject hazards at operator boundaries (cancellation via large offsets, near-constant normalization, near-tie attention, exp overflow), where the instability is governed by input sensitivity, not internal accumulation order. The proposed concrete test—a long positive dot product where n·eps ≫ 2^{-p}—would reveal whether the surrogate detects accumulation rounding; if it does not, the central claim is overstated. This does not invalidate the framework's value as an operator-level diagnostic for Python-visible arithmetic, and it could be fixed by either instrumenting the accumulation or by adding an internal-error bound to the surrogate. The reader's CONDITIONAL verdict appropriately captures this: the core idea is sound, but the detection claim needs this gap closed or explicitly scoped.","tokens_in":26248,"tokens_out":8855,"duration_ms":98848,"concrete_test":"Construct a linear layer C = W x with K = 10,000, W_ij and x_j drawn from U(0,1) (all positive), so the exact output is well-conditioned but FP32 accumulation error is about n·eps ≈ 10^{-3} relative, far exceeding the input-perturbation scale η=2^{-24}. Run noisefloat's arithmetic-level mode, which uses the GEMM surrogate Eq. (16)–(17), and compare reported significant digits with a reference obtained by implementing the same dot product with true CESTAC (random directed rounding after each addition, e.g., using the paper's NFloat in a scalar loop). If noisefloat reports ≥3 digits while the true CESTAC reports near 0 digits, the surrogate fails to detect kernel-internal accumulation instability, falsifying the 'effectively detect' claim for GEMM-like operators.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The weakest point is the GEMM surrogate in §3.4. Equations (15)–(17) perturb inputs by η=2^{-p} when samples are insufficiently separated, but never apply directed rounding inside the GEMM kernel. This misses a broad class of instabilities: accumulation rounding in a long dot product. For a reduction of length n with all-positive terms, the backward error of the internal summation can be as large as γ_n Σ|a_i| ≈ n·eps·Σ|a_i| (Higham 2002), while input perturbation of magnitude η only changes the exact sum by ≤ η·Σ|a_i|. When n is large (e.g., n=10^6, eps=η=2^{-24}), the former is ~0.1 relative error, the latter ~10^{-7}. The three stochastic samples will therefore be nearly identical and noisefloat will report high digits, even though true CESTAC inside the kernel would expose the instability. Theorem 3 bounds the gap between the surrogate and an ideal arithmetic-level execution, but it depends on Δ_p, the perturbation induced by Eq. (16); it cannot detect errors that are invisible to input perturbations. Section 3.6 concedes that fused kernels are not instrumented, and the empirical evaluation (§4.2–4.4) never places a hazard inside a GEMM kernel—all injected instabilities are Python-visible (cancellation, near-constant normalization, near-tie attention, overflow). Thus the central claim 'effectively detect numerically unstable operators' (Section 6) is unsupported for the most common DL kernel family.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes 'noisefloat', a Python framework that wraps NumPy/PyTorch/JAX/TensorFlow tensor programs with CESTAC-style stochastic arithmetic. It represents each value by three synchronous samples with randomized directed rounding, estimates significant decimal digits, and reports diagnostics per operator or per intercepted primitive. For GEMM-like operators it does not instrument inside the kernel; instead it perturbs inputs by eta=2^{-p} when samples are insufficiently separated (Eqs. 15-17). The authors evaluate the method on classical numerical examples, controlled stable/unstable operator pairs, injected pathology studies on Fashion-MNIST/CIFAR-10/AG News, and activation/normalization reliability on Fashion-MNIST, reporting perfect separation of the controlled pairs and localization of injected hazards.","tokens_in":26692,"tokens_out":6271,"duration_ms":66168,"significance":"If the central claim holds, the paper contributes a practical, open-source diagnostic layer for mixed-precision deep learning: backend-native stochastic quantization, straight-through estimator compatibility, operator-level reports, and explicit error bounds (Theorems 2-3) that quantify the gap between the GEMM surrogate and a finer stochastic execution. The controlled S/U pairs separate cleanly and the theoretical results are internally consistent. However, the evidence does not yet establish the stated claim for fused GEMM-like kernels, and the evaluation relies on hand-set detection thresholds and single-run training studies. With narrowed claims or targeted additional experiments, the tool would be a useful complement to CADNA/Verificarlo in deep-learning deployment.","major_comments":[{"comment":"The 'trusted GEMM-like operator' path does not apply randomized directed rounding inside matrix multiplication, linear layers, or convolutions lowered to matmul; it only perturbs inputs by eta=2^{-p} when samples are insufficiently separated. For a reduction of length n, internal accumulation rounding can grow as gamma_n * sum|a_i| ~ n*eps*sum|a_i|, whereas the input perturbation changes the exact sum by O(eta*sum|a_i|). With n large and eta=eps, the former can exceed the latter by orders of magnitude, so three samples can remain nearly identical and Eq. (8) reports high digits despite kernel-internal instability. Theorem 3 bounds the effect only of the data perturbations already injected (Delta_p), not of rounding inside the kernel. Since §4.2-4.4 place all injected hazards at Python-visible boundaries, the central claim of §6 that noisefloat detects unstable operators is not demonstrat","section":"§3.4, Eqs. (15)-(17), and Theorem 3"},{"comment":"The reported perfect classification (accuracy=precision=recall=1.000) uses hand-set, precision-dependent thresholds gamma_23=3 and gamma_52=10. The paper does not justify these values or report how detection accuracy varies with them; the thresholds are free parameters and no selection procedure is described. Since the unstable constructions are deliberately extreme (about 0 digits vs. about 3.5+ in FP32), the benchmark demonstrates ranking/separation, not a robust automatic detection criterion. A principled threshold procedure or a sensitivity/ROC-style analysis is needed to support the 'effectively detect' claim without user calibration.","section":"Eq. (53) and Table 3"},{"comment":"The end-to-end studies use a single seed, 2,048 training examples, at most 32 optimizer steps per epoch, and 3 epochs. The conclusions about localization and about activation/normalization ordering are based on one trajectory each; no confidence intervals, repeated seeds, or ablations are provided. Moreover, the injected pathologies are all of the cancellation/overflow/near-zero-variance types that the CADNA-style source counters are designed to flag, so success on these controlled cases is weaker evidence for detecting unknown instability modes. The experiments support the tool's utility as an operator-level diagnostic, but they are not yet strong enough for the general detection claim in §6.","section":"§4.3-4.6, Figs. 9-13"}],"minor_comments":[{"comment":"The paper alternates between 'noisefloat' and 'noisyfloat' (title/abstract vs. §3); standardize the software name.","section":"Throughout"},{"comment":"Typo: 'deep learnin models' in the Fuzzy PyTorch paragraph.","section":"§2"},{"comment":"The session enumeration is garbled: 'the second session ... the second and third sessions'; clarify which experiments correspond to which session.","section":"§4.1"},{"comment":"The 'first work' claim should be positioned more carefully with respect to Fuzzy PyTorch [49] and Verificarlo-based stochastic arithmetic; the distinction from CESTAC is clear but should be stated explicitly.","section":"§1"},{"comment":"The notation gamma_23 and gamma_52 is not defined; state explicitly that these correspond to simulated FP32 and FP64 significand precisions.","section":"Eq. (53)"},{"comment":"Figure 12 panel (b) is labeled 'deterministic batch accuracy' while the text discusses representative accuracy; clarify which quantity is plotted. Figure 13 has duplicate panel labels for pre-normalization digits.","section":"Figs. 12-13"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a reasonable software/tools paper, but its broad conclusion overreaches the evidence. The GEMM surrogate gap is the main load-bearing issue: the tool cannot detect kernel-internal accumulation instability, and the empirical evaluation never exercises that case. I would be willing to see a revision that either adds a GEMM-internal hazard experiment or explicitly scopes the central claim to operator-boundary/Python-visible instabilities. The hand-set thresholds and single-seed training runs should also be addressed. The controlled S/U benchmark and Theorems 2-3 are solid enough to warrant revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real software contribution—a working CESTAC-style stochastic validation layer for PyTorch/JAX/TensorFlow/NumPy with operator-level reporting and STE compatibility. The controlled experiments are honest and the lemmas in Section 3 are correct. But the tool's detection power is only demonstrated for instabilities visible at Python boundaries; the GEMM surrogate in §3.4 cannot catch accumulation rounding inside fused kernels, and the evaluation never tests it. So the conclusion 'effectively detect numerically unstable operators' is too broad as written.\n\nWhat's new and good: the integration is the contribution. Wrapping existing DL operators with backend-native randomized rounding, batching three stochastic samples, and emitting per-operator digit reports over training iterations is genuinely useful for mixed-precision debugging. The paper is also unusually candid about boundaries—Section 3.6 explicitly says fused kernels are not instrumented. The Hilbert and Rump controls are sensible calibration checks, and the matched stable/unstable operator pairs (softmax, LayerNorm, attention, linear, reduction) cleanly separate, with perfect classification on those constructed cases. Theorems 2–3 are correct but modest: they bound the impact of the surrogate perturbation on the digit estimate, not the fidelity of the surrogate to true kernel-internal rounding.\n\nThe soft spots: the GEMM surrogate is the main one. As the stress-test notes, perturbing inputs by eta=2^{-p} does not expose accumulated rounding in a long dot product: for n large, n*eps*Σ|a_i| can be much larger than eta*Σ|a_i|. The paper's own Lemma 1 and Theorem 3 only bound the effect of the surrogate's perturbation; they don't certify detection of kernel-internal error. The empirical work never places a hazard inside a GEMM, so the central claim is unproven for the most common DL kernel family. The detection thresholds (gamma=3 in FP32, gamma=10 in FP64) are hand-set; the pathology experiments use small subsets (2,048 examples, three epochs) and single runs; and there's no comparison to existing variability tools like Verificarlo or Fuzzy PyTorch. These aren't fatal, but they cap the strength of the claims.\n\nBottom line: this deserves a serious referee. The tool is likely useful to practitioners, and the GEMM gap is addressable—either by instrumenting inside kernels where feasible or by phrasing claims as operator-boundary sensitivity rather than full kernel-level detection. I'd ask the authors to add a GEMM-specific failure case and to soften the conclusion.","headline":"A genuinely useful CESTAC-in-DL integration whose detection claim is only supported for Python-visible instabilities, not the fused GEMM kernels where many real problems live.","tokens_in":27078,"tokens_out":3441,"would_cite":true,"duration_ms":35459,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["65G50","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a CESTAC-based software tool, noisefloat, can detect numerically unstable deep-learning operators by estimating significant digits from synchronized stochastic samples during training and inference.","keywords":["numerical stability","CESTAC","stochastic arithmetic","deep learning operators","significant digits","rounding error","mixed precision","operator-level diagnostics"],"falsifier":"Construct a neural network whose only instability is inside a fused matrix-multiply kernel's internal accumulation order—for instance, an inner product that sums alternating large and near-equal terms so that the true result is small—and ask noisefloat's operator-level report to flag that operator. If the report stays high because the kernel is treated as trusted, the detection claim is limited to Python-visible operations.","tokens_in":26158,"feed_emoji":"🧮","tokens_out":4256,"duration_ms":50255,"temperature":0.7,"pith_summary":"noisefloat tries to make numerical-stability analysis a routine part of deep-learning development. It applies CESTAC-style stochastic arithmetic, where each tensor is evaluated as three synchronized samples under randomized directed rounding, and reports the estimated number of significant digits at operator outputs. The paper shows that unstable formulations—cancellation, overflow, near-zero normalization, near-tie attention—drop to near-zero digits, while stable counterparts stay high, and that these reports localize an injected hazard in otherwise ordinary classifiers and sequence models. If the approach works as claimed, developers gain a per-operator, per-iteration reliability reading without altering the optimization trajectory.","feed_headline":"Counting reliable digits exposes unstable neural operators","feed_subtitle":"Three sample runs per tensor estimate reliable digits, pinpointing cancellation and overflow before training breaks.","key_machinery":"The central mechanism is CESTAC-style stochastic arithmetic lifted to deep-learning operators: each tensor is carried as a batch of s stochastic samples (default 3), backend-native randomized directed rounding is applied at instrumented boundaries, and the significant-digit estimate C_Y = log10( sqrt(s) |mean| / (tau_beta * sigma) ) is computed elementwise. Operator-boundary wrappers quantize only operator outputs; arithmetic-level propagation rounds after Python-visible primitives. GEMM-like operators (matmul, linear, convolution lowered to matmul) are treated as trusted: when stochastic operands are insufficiently separated, inputs are perturbed at the unit-roundoff scale eta = 2^{-p} befo","core_discovery":"The author's central claim is that noisefloat is the first software tool to integrate CESTAC into the numerical validation of deep-learning deployment, and that it can effectively detect numerically unstable operators. The method represents every deterministic value as several synchronized stochastic samples, quantizes each operation with randomly directed rounding, and computes an estimated number of significant decimal digits from the sample mean and dispersion. Unstable computations—catastrophic cancellation, overflow in softmax, near-zero variance in normalization, near-tie attention logits—report very low or zero digits, while algebraically stabilized variants retain high digits. A sync","pith_inferences":["The same per-operator digit estimates could be used as a precision-tuning signal: operators with chronically low C_Y values are natural candidates for higher precision or stable reformulation, and this could be tested by correlating digit drops with actual reduced-precision training failures.","Because the tool does not instrument arithmetic inside fused GEMM-like kernels, instabilities that live entirely in internal accumulation order—such as a matmul whose inner products cancel—may escape detection; extending CESTAC to kernel internals or using randomized rounding inside the kernel would close that gap.","The reported overhead (roughly three sample evaluations plus quantization, potentially far more in practice) suggests that scalable monitoring will likely use calibration subsets or sparse iteration sampling rather than full-data tracing; a sampling strategy that preserves the localization power could be a natural follow-up.","The observed decoupling of low operator digits from high decision-level agreement points toward a stability-aware margin criterion: a network could be considered numerically safe when its minimum digit level stays above a threshold that depends on downstream logit margins."],"forward_implications":["Developers can localize numerically unstable operators during training or inference without changing the model's optimization trajectory.","The significant-digit reports separate stable and unstable formulations of the same operation—for example, shifted versus naive softmax, or rationalized versus cancellation-prone expressions.","Operator-level digit rankings provide a concrete signal for where to increase precision, reformulate an expression, or replace a numerically fragile operator.","Local loss of digits does not necessarily change the final decision; the paper shows that downstream margins can absorb operator-level instability, so the diagnostics are per-operator reliability measures rather than end-to-end failure predictions.","GEMM-like kernels are deliberately treated as trusted; their reports are data-perturbation estimates whose reliability depends on the cited probabilistic validation hypotheses."],"fun_headline_variants":["Stochastic sampling exposes unstable deep learning ops","CESTAC integration flags numerical fragility in neural nets","Digit counting detects operator instability before training","New tool pinpoints unreliable digits in DL operators","Catch numerical blowups early with stochastic digit estimates"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The central claim depends on treating GEMM-like kernels as trusted operators, so instabilities inside their fused accumulation are only probed by perturbing insufficiently separated inputs; if the perturbation surrogate misses an internal hazard, the tool will not flag it.","fun_headline_variants_meta":{"raw":{"variants":["Stochastic sampling exposes unstable deep learning ops","CESTAC integration flags numerical fragility in neural nets","Digit counting detects operator instability before training","New tool pinpoints unreliable digits in DL operators","Catch numerical blowups early with stochastic digit estimates"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000128,"raw_usage":{"total_tokens":883,"prompt_tokens":603,"completion_tokens":280,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":347,"completion_tokens_details":{"reasoning_tokens":210}},"tokens_in":347,"tokens_out":280,"duration_ms":3787,"temperature":1.0,"reasoning_tokens":210,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T02:15:11.918590+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a neural network whose only instability is inside a fused matrix-multiply kernel's internal accumulation order—for instance, an inner product that sums alternating large and near-equal terms so that the true result is small—and ask noisefloat's operator-level report to flag that operator. If the report stays high because the kernel is treated as trusted, the detection claim is limited to Python-visible operations.","supporting_citations":[],"review_version":1}