{"id":"3b7d3b52-eeef-4b9b-b8ca-c9fb13f217c2","arxiv_id":"2505.11589","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Boundary loss plus selective gradient clipping lets polynomial neural networks train stably at high degrees and match ReLU accuracy on seven datasets.","lead":"The authors propose two training modifications, a boundary penalty and a gradient-clipping rule that skips batch-normalization parameters, to stabilize polynomial neural networks. The approach matters because polynomial networks can be evaluated under homomorphic encryption, a step toward private machine learning on encrypted data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract's degree-22 claim is supported by one undocumented sentence; evidence covers only degrees 2, 8, and scattered 12, and the conclusion itself limits to degree 8.","rationale":"The reader's CONDITIONAL verdict already captures the need for repeated seeds, error bars, and code; I do not move the verdict. I set agreement_with_reader to 'partial' because the reader's weakest_assumption concerns the boundary-loss balance in Eq. (1), while I regard the unsupported degree-22 claim as the more decisive and more easily testable weakness. Both concerns point to the same broad problem—the paper's confidence exceeds its evidence—but the concrete test I propose addresses the headline claim directly: it either supplies the missing degree-22 evidence or forces the claim to be downgraded to the range actually reported (degree 8, with degree 12 explored). No formal proof of Eq. (1)'s stability is provided, and §7.1 acknowledges the empirical focus, but lack of proof is not by itself an internal inconsistency; the absence of any degree-22 result is a concrete omission.","tokens_in":9612,"tokens_out":5997,"duration_ms":61910,"concrete_test":"Reproduce the §5.2 degree-22 Speech Commands experiment after specifying the omitted hyperparameters (B, α, λ, c, dropout, architecture, seeds); report test accuracy and training curves for at least 5 seeds, and run the same configuration with the three §5.1 baselines (no modifications, boundary-loss only, clipping-only). If the degree-22 run cannot be reproduced, is not reported, or does not beat those baselines, the abstract and contributions should be revised to claim stability only up to degree 8/12, matching Table 1 and §8.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline claim—that PNNs 'exhibit stable training and strong performance with polynomial degrees up to 22, where standard methods typically fail'—rests on a single undocumented sentence in §5.2: 'we also successfully trained polynomial degrees as high as 22 for the Speech Commands dataset.' No accuracy, architecture, B, α, λ, clipping threshold, seed count, or baseline comparison is reported for that run. Table 1 contains only degrees 2 and 8; Figure 4 adds degree 12 on CIFAR-10 and UCI-HAR; and the conclusion (§8) limits the claim to 'degrees up to 8 across a diverse set of benchmarks.' The abstract and contributions therefore assert a capability the paper's own results do not document. This is the most load-bearing gap because the practical value of the framework for HE is tied to the high-degree regime, and as published the degree-22 result cannot be checked. The authors' own limitation note (§7.1) concedes the validation is empirical, so the missing reproducibility is not a minor omission.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a training framework for Polynomial Neural Networks (PNNs) aimed at homomorphic-encryption-compatible inference. The two proposed components are a Boundary Loss that exponentially penalizes activation inputs outside a predefined range, and Selective Gradient Clipping that clips gradients of non-BatchNorm parameters while leaving BatchNorm parameters unclipped. The authors evaluate the framework on seven image, audio, and human-activity datasets using ResNet-18 and MLP architectures with polynomial degrees 2 and 8, report ablations on CIFAR-10 and CIFAR-100, study the effect of the fitting boundary B and of excluding BatchNorm from clipping, and report one encrypted-inference result on a 5-layer MLP. The abstract claims stable training up to polynomial degree 22, but the only supporting evidence is a single undocumented sentence in Section 5.2.","tokens_in":9929,"tokens_out":4231,"duration_ms":44004,"significance":"If the results hold, the framework is a simple and potentially practical contribution to HE-friendly neural network training: it makes no exotic assumptions, uses only standard layers, and openly reports all hyperparameters (B, alpha, lambda, c), which is useful for reproducibility. The main strengths are the breadth of the seven-dataset evaluation, the explicit ablation of the two proposed components, and the honest admission in Section 7.1 that the validation is empirical. However, the significance is currently dampened by three gaps: the headline degree-22 claim is not backed by any reported experiment; Table 1 reports single accuracies without error bars; and there is no quantitative comparison against prior PNN stabilization methods. These gaps must be addressed before the central claims about 'stable training with high-degree polynomials' and 'standard methods typically fail' can be accepted as demonstrated.","major_comments":[{"comment":"The claim that PNNs 'exhibit stable training and strong performance with polynomial degrees up to 22' rests on a single undocumented sentence in Section 5.2 ('we also successfully trained polynomial degrees as high as 22 for the Speech Commands dataset'), with no accuracy, architecture, B, alpha, lambda, clipping threshold, number of seeds, or baseline comparison reported, and Table 1 reports only degrees 2 and 8. The Conclusion (Section 8) itself restricts the validated claim to 'degrees up to 8 across a diverse set of benchmarks,' so the abstract and contributions overstate the evidence. This is load-bearing because the abstract uses degree 22 as a headline capability; the sentence should either be removed from the abstract and contributions or replaced with a fully documented experimental result.","section":"Abstract and Section 5.2"},{"comment":"Table 1 reports a single accuracy number per condition with no standard deviation, seed count, or number of runs, which makes it impossible to assess whether the reported parity with ReLU baselines is significant or stable; the same issue affects Figures 3-5, and the only statistical statement in Section 5.1 ('p < 0.01, two-sided Z-test') does not describe the number of trials or what is counted as a success. The paper should report mean and variance over at least several seeds for the central results, and for the ablation comparisons it should state the success-rate definition and trial counts.","section":"Table 1 and Figures 3-5"},{"comment":"No quantitative comparison is made to existing PNN stabilization methods cited in Section 2.2 (e.g., Goyal et al., Aboulatta et al., Gottemukkula, and AESPA), so the claim that 'standard methods typically fail' is not supported by any baseline experiment. At minimum, the paper should train one or two prior stabilization baselines under the same architectures and report their accuracy and divergence rates, or alternatively soften the claim to say that the proposed method succeeds without such comparisons.","section":"Section 2.2 and Section 5"},{"comment":"The framework's stability mechanism depends on the exponential boundary penalty being strong enough to dominate the classification loss for inputs just above alpha*B, but for high degree d the post-activation magnitude can grow like (alpha*B)^d, and no analysis of this balance is provided; Section 5.2 itself concedes that setting B too low can lead to an exploding boundary loss. The paper's own limitation note (Section 7.1) states the validation is empirical; given that the central claim is stability, the authors should either provide a formal stability argument or at least report failure rates across a grid of B values and initialization seeds to quantify the claimed stability.","section":"Equations (1)-(2) and Section 7.1"},{"comment":"The HE evaluation is limited to one 5-layer MLP on UCI-HAR with an inference time of 177.33 seconds per sample; no HE evaluation of the ResNet-18 architectures used for the image and audio tasks is reported, so the abstract's statement that the framework 'confirm[s] the HE-compatibility of the trained models' is demonstrated only for a simple case. The authors should either scope the HE claim to the evaluated 5-layer MLP or add an HE evaluation for at least one convolutional model.","section":"Section 6"}],"minor_comments":[{"comment":"The PNN Accuracy column appears to contain a formatting problem: for MNIST the row lists '0.994' and '0.9944' in the same cell, which is confusing; the table layout should be clarified.","section":"Table 1"},{"comment":"There is a typo: 'differnt' should be 'different'. Also, Figure 4's caption says 'Each dot represents a training run' but the number of runs per configuration is not stated.","section":"Section 5.2"},{"comment":"The Z-test sentence should specify the number of runs, the definition of a successful run, and whether the test compares convergence success or final accuracy, so that the reader can interpret the p-value.","section":"Section 5.1"},{"comment":"The number m of sampling points used for polynomial fitting is not reported; this matters because the fit quality depends on m and the sampling range.","section":"Section 3.1.1"},{"comment":"The caption states that the activation function was fitted for the range [-35,35], but the text says B varies by degree and dataset; please clarify that this is one example configuration.","section":"Figure 2 caption"},{"comment":"The reference formatting is inconsistent, for example the first reference cites 'HIP' rather than the full act name, and several entries use 'et al.' in nonstandard ways; this should be normalized to the journal style.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is honest about its empirical focus, but the abstract and contributions are broader than the evidence presented. The main risk is overclaiming: the degree-22 statement appears in the abstract but is unsupported by any reported experiment, and the stability claims lack error bars and failure-rate analysis. I would ask the authors to reconcile the abstract with the results, add variance reporting, and provide at least one quantitative comparison with prior stabilization methods before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort version: this is a real, useful empirical contribution, but the abstract oversells one result that the paper itself cannot document. The core of the work—Boundary Loss plus Selective Gradient Clipping, with BatchNorm parameters excluded from clipping—is a sensible combination that appears to beat both baseline and partial ablations on CIFAR-10/100 and a few other datasets. The authors report degree-2 and degree-8 accuracies across seven datasets that hover near ReLU baselines, and they show a degree-12 B-sweep in Figure 4. That is genuine evidence for the central practical claim: you can train PNNs to degree 8 without the usual divergence. The ablation study showing that clipping BatchNorm causes collapse, while excluding it works, is a concretely useful finding. The HE runtime demo with CKKS and a degree-2 activation is a nice sanity check, even if the 177 seconds per sample is not a performance claim.\n\nNow the soft spots, in proportion. The abstract claims stable training \"with polynomial degrees up to 22.\" The entire support for that is one sentence in Section 5.2: \"we also successfully trained polynomial degrees as high as 22 for the Speech Commands dataset.\" No accuracy, no architecture detail, no hyperparameters, no seed count, no baseline. Table 1 only shows degrees 2 and 8. The conclusion itself retreats to \"degrees up to 8.\" That is a mismatch between headline and evidence, and it matters because the high-degree regime is exactly what would be interesting for HE, where multiplicative depth is a bottleneck. Also, Table 1 reports single accuracies with no error bars or repeated seeds; for a method whose whole pitch is stability, run-to-run variance is directly relevant. The paper acknowledges the B-balance instability (Section 5.1) but does not quantify how often it diverges or give a robustness measure. Finally, no comparison to prior PNN stabilization methods—the input-stabilization work of Aboulatta et al. 2019 is cited but never quantitatively compared. That is a real gap, though not a fatal one.\n\nCirculatory burden is low: the hyperparameters are reported, there are no invented quantities, and the results are trained accuracies, so no fitting disguised as prediction. The math is simple enough—the boundary loss is a per-layer penalty on pre-activation magnitude—and it is clearly described. What is missing is reproducibility: no code, no seeds, no error bars.\n\nWho is this for? Researchers working on HE-friendly ML or on stabilizing polynomial activations will get a useful starting point and a clear set of design choices to test. It deserves a serious referee: the central idea is plausible, the ablation is informative, and the degree-8 result is worth checking. But the referee should ask for code, repeated seeds, and a rewrite that de-emphasizes the degree-22 claim unless actual numbers are provided.\n\nI would send it to review, with a request for major revision.\n\nCandidly,\n[You]","headline":"A plausible, well-scoped empirical framework for stable PNN training, but the degree-22 headline overreaches and the evaluation needs error bars and code before the strong claims can be taken at face value.","tokens_in":10322,"tokens_out":759,"would_cite":false,"duration_ms":9249,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Polynomial neural networks can be trained stably at degrees up to 22, and reach near-ReLU accuracy on seven datasets, through a boundary loss plus selective gradient clipping.","keywords":["polynomial neural networks","homomorphic encryption","boundary loss","selective gradient clipping","training stability","polynomial activation functions","batch normalization"],"falsifier":"On CIFAR-10 with degree-8 activations, set $B=35$, $\\alpha=0.5$, and $\\lambda=1000$, and record the maximum $|X_{i,j}|$ for every layer at every step of the first epoch. If that maximum grows toward or beyond $B$ despite the boundary loss, the presumed balance fails; likewise, if the full method and the no-clipping baseline attain equal accuracy, the claimed synergy between boundary loss and selective clipping is not load-bearing.","tokens_in":9392,"feed_emoji":"🔐","tokens_out":13529,"duration_ms":111340,"temperature":0.7,"pith_summary":"Polynomial neural networks replace ReLU with polynomial activations, which makes them compatible with homomorphic encryption but hard to train once the degree rises. This paper claims that two techniques together remove that difficulty: a Boundary Loss that exponentially penalizes any activation input leaving a preset safe range, and Selective Gradient Clipping that clips large gradients everywhere except BatchNorm parameters. With both in place, the paper reports stable training for polynomial degrees 2, 4, and 8 across seven image, audio, and activity-recognition datasets, with accuracy close to ReLU baselines, and stable degree-22 training on Speech Commands. If true, this means privacy-preserving inference can use genuine polynomial activations instead of approximating ReLU under encryption.","feed_headline":"A two-part fix gets polynomial networks to ReLU-level accuracy","feed_subtitle":"Stable high-degree polynomial activations let encrypted inference skip ReLU approximations.","key_machinery":"The central object is the composite training loss, $L_{\\text{total}} = L_{\\text{CE}} + \\lambda \\sum_i L_i$, where $L_i$ is the Boundary Loss on layer $i$'s activation inputs: $L_i = \\frac{1}{|X_i|} \\sum_{j=1}^{|X_i|} (e^{\\max(|X_{i,j}| - \\alpha B, 0)} - 1)$, with $B$ the fitting interval, $\\alpha \\in (0,1]$ a slack factor, and $\\lambda$ the penalty weight. The companion mechanism is Selective Gradient Clipping: gradients of non-BatchNorm parameters are divided by $\\max(1, \\|\\nabla\\theta_{\\text{non-BN}}\\|_2 / c)$ while BatchNorm gradients pass unchanged. The argument is carried by the interaction between these two: the exponential term keeps activation inputs inside the polynomial's reliable domain, clipping absorbs the resulting large gradients, and exempting BatchNorm preserves the running statistics the network depends on.","core_discovery":"The central claim is that PNN training instability has two manageable causes: inputs to the polynomial activations drift outside the interval where the fitted polynomial approximates the target activation, and boundary-penalty gradients damage BatchNorm's running statistics. The paper's Boundary Loss, $L_i = \\frac{1}{|X_i|} \\sum_j (e^{\\max(|X_{i,j}| - \\alpha B, 0)} - 1)$, added to the classification loss with weight $\\lambda = 1000$, pushes activation inputs back inside $[-\\alpha B, \\alpha B]$, while Selective Gradient Clipping rescales by norm only the non-BatchNorm parameters, leaving BatchNorm's scale and shift parameters untouched. The paper argues these mechanisms are synergistic: the loss prevents polynomial outputs from exploding, and the clipping prevents the loss's own steep gradients from destabilizing optimization. On its own terms, the discovery is that high-degree polynomial networks, not just low-degree approximations, are trainable and can approach ReLU accuracy, easing the path to encrypted inference.","pith_inferences":["The boundary-loss idea is a general cure for training any activation with unbounded output, such as Taylor or spline activations: penalize out-of-fit-range inputs during early training rather than redesigning the activation; the paper does not test this transfer.","Because the degree-22 result appears on only one dataset and the paper gives no convergence proof, the practical ceiling is still unknown; a natural next test is reproducing the method on deeper convolutional architectures and larger benchmarks.","The method's sensitivity to $B$ suggests a testable refinement: start with a wide boundary for stability and shrink it as activations settle, which could remove the manual degree-dependent tuning the paper currently recommends.","If BatchNorm-free architectures are used, the selective-clipping component may become unnecessary; the paper's ablations imply the benefit of clipping comes mostly from protecting BatchNorm statistics, but this simplification is not evaluated."],"forward_implications":["High-degree polynomial activations can be used in deep, encryption-compatible architectures without sacrificing accuracy, so encrypted inference no longer depends on low-degree approximations that limit expressivity.","A single trained PNN can serve both plaintext and encrypted inference, since its layers are expressible by additions, multiplications, and affine transforms that fold batch normalization into adjacent linear layers.","BatchNorm parameters should be excluded from any gradient-clipping scheme when training PNNs; clipping them is reported to cause training collapse even at degree 2.","Choosing $B$ near the lower end of the stable training range is a practical recipe that balances accuracy and stability, with the safe range growing as polynomial degree increases.","On the seven datasets tested, degree-8 PNNs match or nearly match ReLU baselines, indicating polynomial activations are not inherently a capacity bottleneck."],"supporting_citations":[{"why":"Provides BatchNorm, the layer whose parameters Selective Gradient Clipping deliberately leaves unclipped so that its running statistics keep adapting.","marker":"Ioffe and Szegedy [2015]"},{"why":"Specifies the AdamW optimizer and learning-rate schedule used for every reported training run.","marker":"Loshchilov and Hutter [2017]"},{"why":"Provides the software implementation of CKKS used to show a degree-2 PNN reaching 94.7 percent accuracy in encrypted inference.","marker":"Badawi et al. [2022]"},{"why":"Defines the approximate-number encryption scheme whose multiplicative-depth limits make low-degree polynomial activations desirable.","marker":"Cheon et al. [2017]"},{"why":"Supplies the Speech Commands dataset, the benchmark where degree-22 activation training is reported.","marker":"Warden [2018]"},{"why":"Supplies CIFAR-10, the main benchmark for the ablation studies and the boundary-sensitivity analysis.","marker":"Krizhevsky [2009]"},{"why":"Documents the exploding-gradient problem in PNN training that the proposed framework is designed to solve.","marker":"Goyal et al. [2020]"},{"why":"Supplies MNIST, one of the seven datasets used to support the generalizability claim.","marker":"Deng [2012]"},{"why":"Supplies the UCI-HAR dataset, used both for accuracy comparison and for the encrypted-inference demonstration.","marker":"Reyes-Ortiz et al. [2013]"}],"fun_headline_variants":["Stable high-degree polynomial nets near ReLU accuracy","Boundary loss and clipping tame polynomial networks","Training framework stabilizes polynomial nets to degree 22","Two tweaks make high-degree polynomial nets near ReLU accuracy","Keep polynomial nets in bounds to approach ReLU accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole scheme rests on the assumption that an exponential penalty on large activation inputs will keep those inputs bounded from the first training steps, before the highest-degree term of the polynomial can diverge; if the penalty is too weak relative to the classification loss, no amount of later clipping can recover the run.","fun_headline_variants_meta":{"raw":{"variants":["Stable high-degree polynomial nets near ReLU accuracy","Boundary loss and clipping tame polynomial networks","Training framework stabilizes polynomial nets to degree 22","Two tweaks make high-degree polynomial nets near ReLU accuracy","Keep polynomial nets in bounds to approach ReLU accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000841,"raw_usage":{"total_tokens":3698,"prompt_tokens":1014,"completion_tokens":2684,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":630,"completion_tokens_details":{"reasoning_tokens":2608}},"tokens_in":630,"tokens_out":2684,"duration_ms":17771,"temperature":1.0,"reasoning_tokens":2608,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:51:27.451455+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On CIFAR-10 with degree-8 activations, set $B=35$, $\\alpha=0.5$, and $\\lambda=1000$, and record the maximum $|X_{i,j}|$ for every layer at every step of the first epoch. If that maximum grows toward or beyond $B$ despite the boundary loss, the presumed balance fails; likewise, if the full method and the no-clipping baseline attain equal accuracy, the claimed synergy between boundary loss and selective clipping is not load-bearing.","supporting_citations":[{"cited_title":"Decoupled weight decay regularization","cited_arxiv_id":null,"evidence_quote":"Specifies the AdamW optimizer and learning-rate schedule used for every reported training run."},{"cited_title":"OpenFHE : Open-source fully homomorphic encryption library","cited_arxiv_id":null,"evidence_quote":"Provides the software implementation of CKKS used to show a degree-2 PNN reaching 94.7 percent accuracy in encrypted inference."},{"cited_title":"Homomorphic encryption for arithmetic of approximate numbers","cited_arxiv_id":null,"evidence_quote":"Defines the approximate-number encryption scheme whose multiplicative-depth limits make low-degree polynomial activations desirable."}],"review_version":1}