Pith. sign in

REVIEW 5 minor 27 references

This paper shows that over a finite field, the expressivity of a shallow polynomial neural network is exactly the number of distinct output tuples its weights can produce—and for several architectures that number is given by explicit formul

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 19:04 UTC pith:SGBA2OQ7

load-bearing objection A solid, honest counting paper that opens a genuinely new finite-field direction in neuroalgebraic geometry; the main formulas are correct and the flaws are typographical or presentational.

arxiv 2607.17090 v1 pith:SGBA2OQ7 submitted 2026-07-19 math.AG cs.NE

Expressivity of Shallow Neural Networks Over Finite Fields

Maksym Zubkov , Carol Wu , Shiwei Yang , Param Mody , Yifei Chen This is my paper
classification math.AG cs.NE MSC 14G1514G0505A30
keywords polynomial neural networksfinite fieldsneuromanifoldexpressivitysymmetric tensorsmatrix rank over finite fieldspoint countingcharacteristic p
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper establishes that the expressivity of a shallow polynomial neural network over a finite field is precisely the number of distinct output tuples its parameter map can produce—the cardinality of its neuromanifold—and it computes this number exactly for several architectures. For a single hidden unit, for two hidden units, and for activation degrees divisible by the field's characteristic, the counts are given by closed formulas built from matrices of bounded rank and from projective-line counts. The authors also prove a general upper bound in terms of the base-p expansion of the activation degree. The most striking consequence is a demonstrated split: over the complex numbers a certain shallow quadratic network generates a Zariski-dense set of functions, while over finite fields it occupies only about half of the ambient space. If the paper is right, expressivity is not a purely architectural property; the characteristic of the coefficient field can halve it.

Core claim

The central claim is that, for architectures (n,1,k), (n,2,k), and for activation degrees divisible by p, the neuromanifold's cardinality is given by exact formulas. In particular, for d=(2,2,2) with r=2, the finite-field neuromanifold has cardinality asymptotically half of the ambient space as p grows, whereas the same architecture over the complex numbers has a Zariski-dense image. The proof rests on a partition of the neuromanifold by the dimension of the span of the output tuple and on the fact that, when r>1, a 2-dimensional subspace of symmetric tensors contains at most two projectivized rank-1 tensors, so each representable tuple has a unique generating pair of rank-1 forms.

What carries the argument

The central object is the parameter map sending a weight matrix pair to the tuple of homogeneous polynomials (sum_s b_{is}(a_{s1}x_1+...+a_{sn}x_n)^r)_{i=1..k}; its image is the neuromanifold. The counting machinery combines four ingredients: a multinomial congruence theorem showing which monomial coefficients can be nonzero modulo p, giving the upper bound q^{γ_{n,p}(r)k}; the classical formula for the number of symmetric matrices of given rank over a finite field, used for the square-activation single-output case; projective-space counting, which turns the neuromanifold into a product P^{n-1} × P^{k-1} for m=1 and into a union of fibers over rank-1 tensor pairs for m=2; and the perfection

Load-bearing premise

The counting formulas for two hidden units assume that, when the activation degree exceeds 1, a two-dimensional subspace of symmetric tensors contains at most two projectivized rank-1 tensors, so that every representable tuple has a unique generating pair; if a subspace ever contained three such rank-1 directions, the partition would overcount and the closed-form counts would be wrong.

What would settle it

For a small case such as d=(2,2,2), r=2 over F_5, enumerate all 5^8 weight assignments, compute the resulting pair of quadratic forms, and count distinct pairs; the count should equal 7945 as reported in the paper's Table 2. A mismatch would refute Proposition 4.3's partition argument. As a sharper test, compute the exact cardinality over F_13 and check whether it is near half of 13^6, which the conjectured limit would predict.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the formulas hold, expressivity over a finite field becomes an exact, computable number: for a given architecture one can enumerate every representable k-tuple of homogeneous polynomials without training.
  • Over the complex numbers, the (n,n,2), r=2 network fills the ambient space in the Zariski sense, but over F_p its arithmetic capacity tends to 1/2 for n=2; the same architecture is thus expressive or half-expressive depending on the coefficient field.
  • Activation degrees that differ by a factor of p are equally expressive over F_q: the neuromanifold cardinality is unchanged when r is replaced by r/p, because every field element is a p-th power.
  • When the activation degree is a power of p, the neuromanifold coincides exactly with the set of k×n matrices of rank at most m, reducing the network to a linear map with a rank constraint.
  • The general upper bound q^{γ_{n,p}(r)k} means that the base-p digits of r, not just its size, control what a shallow network can represent.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The 1/2 limit for (2,2,2), r=2 suggests a pattern: events that are 'probability zero' over the complex numbers, such as repeated eigenvalues of a pencil of symmetric matrices, become events of positive density over finite fields; quantifying this density for other architectures could predict when low-precision networks lose expressivity.
  • The exact enumeration makes a concrete test possible for quantized networks: for small q one can brute-force all weight assignments and compare the representable function count against the formulas, giving a training-free measure of capacity.
  • The equality for p-multiple degrees implies that in characteristic p, monomial activations of degree r and r/p behave identically; this could guide hardware designers choosing between activation degrees in low-precision settings.
  • A natural extension left open by the paper is the m≥3 case, where the unique-generating-pair argument fails; incidence-counting of decompositions into three or more rank-1 tensors would be needed to extend the exact formulas.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

0 major / 5 minor

Summary. The paper introduces an algebraic notion of expressivity for shallow polynomial neural networks over finite fields, measured by the cardinality of the neuromanifold (the image of the parameter map in a product of polynomial rings). It proves a general upper bound via the multinomial congruence in characteristic p, computes exact counts for single-output square activation (d=(n,m,1), r=2) using MacWilliams' enumeration of symmetric matrices, and obtains projective counting formulas for d=(n,1,k) and d=(n,2,k) for arbitrary monomial degrees. The key step for m=2 is a disjoint-partition argument based on the uniqueness of generating pairs of rank-1 tensors when r>1 and p∤r. For degrees divisible by p, a Frobenius reduction equates the count with the p-free part. The paper also highlights a contrast with the complex case: for d=(2,2,2), r=2, the finite-field neuromanifold occupies roughly half of the ambient space, whereas over C it is Zariski dense.

Significance. The results provide the first systematic point counts for neuromanifolds of shallow polynomial neural networks over finite fields, linking network expressivity to classical finite-field enumeration. The counting arguments are rigorous; the critical uniqueness step in Proposition 4.3 is valid, and the Frobenius reduction for p|r is correct. The paper is transparent about limitations: Conjecture 3.6 is stated as a conjecture supported only by four data points, and Remark 4.5 explicitly notes that the m=2 partition does not extend to m≥3. The contrast between finite-field and complex behavior is interesting and may motivate further arithmetic study of neural network expressivity.

minor comments (5)
  1. [Section 4, Lemma 4.2] The statement writes |\mathfrak{M}_{d,r}|, but the proof establishes the projective cardinality |\bar{\mathfrak{M}}_{d,r}| = |\mathbb{P}^{n-1}| |\mathbb{P}^{k-1}|. The affine cardinality is (q-1) times this plus 1 by Lemma 4.1. The same projective/affine confusion appears in Table 1, row (n,1,k). Please correct the statement and table.
  2. [Table 2] The row labelled p=11 has ambient size 4,826,809 = 13^6, and the listed |M| matches Proposition 4.3 at q=13, not q=11. The label should be q=13 (or the row recomputed).
  3. [Section 3.2, proof of Theorem 3.5] V2 is defined as the intersection over all (i,j,k) of the vanishing of the (j,k)-th minor determinants of A_i, but the set {det A_1 = det A_2 = 0} is not equal to that intersection. Also, V1 uses A_1^{-1}A_2, which is not a polynomial condition when A_1 is singular. The proof should be repaired by defining the bad locus as {det A_1 det A_2 = 0} ∪ {discriminant of charpoly(A_1^{-1}A_2) = 0} after clearing denominators. This is a local issue and does not affect the finite-field counting results.
  4. [Corollary 4.5.1] The dichotomy 'r ≠ p^i' vs 'r = p^i' is not correct when r has a p-free factor i>1 and p|r (e.g., r=2p with p odd). The intended reduction via Corollary 5.1.1 is: if the p-free part of r is 1, use Equation (14); otherwise use Equation (13) with the p-free part.
  5. [General presentation] There are minor typesetting issues, e.g., the running header 'M. Zubkov et al.:Preprint submitted to ElsevierPage 1 of 16' and inconsistent use of double/triple bars for projective cardinalities. Please harmonize notation. Also, Corollary 5.3.1's 'min' is redundant when m≥min(n,k); consider simplifying.

Circularity Check

0 steps flagged

No significant circularity: central counts are derived from external classical theorems and self-contained combinatorial arguments.

full rationale

The paper's central claims are exact cardinalities of neuromanifolds. These are obtained by bounding or partitioning the image of the parameter map, not by fitting a parameter to the quantity being 'predicted.' The upper bound (Prop. 2.5) uses the external multinomial congruence theorem; the r=2 symmetric-matrix counts use Axler's diagonalization theorem and MacWilliams' rank counts; the (n,1,k) count is a projective product decomposition proved in Lemma 4.2; the (n,2,k) counts (Props. 4.3, 4.4) rest on an internal uniqueness argument for generating pairs of rank-1 symmetric tensors, with the critical step proved in the text rather than imported; and the p|r reductions (Props. 5.1, 5.2) use only Frobenius, perfectness of finite fields, and the standard rank-factorization lemma. No parameter is fitted to data, and no 'prediction' is a renamed input. The only apparent self-citation, Finkel et al. (2025), includes author Wu but appears solely in Related Work and is never used in any proof, so it is not load-bearing. The noted typographical issues (Lemma 4.2 display mixing projective/non-projective cardinalities and Table 2's p=11 row) are presentation errors and do not affect the derivation chain.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

The paper rests on standard theorems: multinomial congruence (Conrad), MacWilliams symmetric-matrix counts, Axler's diagonalization, and the perfectness of finite fields. No free parameters are fitted, and no new entities are postulated.

axioms (5)
  • standard math Multinomial congruence (Conrad 1995): (r choose α) ≡ ∏_t (r_t choose α_t) mod p for base-p digits.
    Used to compute the number γ_{n,p}(r) of monomials that survive in the upper bound (Prop 2.5).
  • standard math MacWilliams (1969) formula for number of n×n symmetric matrices of rank m over F_q (char≠2).
    Used to get exact |M_{(n,m,1),2}| in Cor 3.3.1.
  • standard math Axler (2015) Theorem 9.13: every symmetric matrix over char≠2 is congruent to a diagonal matrix.
    Used to identify the r=2 neuromanifold with symmetric matrices of rank ≤ m (§3.1).
  • standard math Finite fields are perfect, so Frobenius x→x^p is surjective.
    Needed in Prop 5.1 to equate cardinalities for r=ip and r=i.
  • standard math Morrison (2006) count of k×n matrices of given rank.
    Used in Propositions 4.3/4.4 for rank-2 coefficient matrices.

pith-pipeline@v1.3.0-alltime-deepseek · 19804 in / 28523 out tokens · 266107 ms · 2026-08-01T19:04:28.870473+00:00 · methodology

0 comments
read the original abstract

We study the expressivity of shallow polynomial neural networks (PNNs) with monomial activation functions over finite fields. For a given architecture, we define a neuromanifold as the image of the map from all possible network weights into the product of polynomial rings. We quantify the expressivity by the cardinality of the neuromanifold, and derive a natural lower and upper bound. This leads to counting rational points over finite fields, a problem closely linked to the Weil conjectures. Finally, we present an architecture that exhibits a striking difference in the neuromanifolds when considered over a characteristic zero versus a finite-characteristic field, illustrating the critical role of field characteristic in the notion of expressivity.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

27 extracted references · 9 canonical work pages · 2 internal anchors

  1. [8]

    URL:https://publications.polymtl.ca/39043/

    Training deep neural networks with low precision multiplications, in: International Conference on Learning Representations (ICLR 2015). URL:https://publications.polymtl.ca/39043/. Deligne, P.,

    2015

  2. [9]

    Integer sequences and matrices over finite fields

    URL:https://doi.org/10.48550/ arXiv.math/0606056, doi:10.48550/arXiv.math/0606056. Shahverdi, V ., Marchetti, G.L., Kohn, K.,

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.math/0606056

  3. [11]

    Algebraic Statistics 16, 113–130

    Activation degree thresholds and expressiveness of polynomial neural networks. Algebraic Statistics 16, 113–130. doi:10.2140/astat.2025.16.113. Friedland, S., Stawiska, M.,

    doi:10.2140/astat.2025.16.113 2025

  4. [13]

    Chapman & Hall/CRC, p

    A survey of quantization methods for efficient neural network inference, in: Low-Power Computer Vision. Chapman & Hall/CRC, p. 291–326. doi:10.1201/9781003162810-13. Görtz, U., Wedhorn, T.,

    doi:10.1201/9781003162810-13

  5. [14]

    2 ed., Springer Spektrum Wiesbaden

    Algebraic Geometry I: Schemes: With Examples and Exercises. 2 ed., Springer Spektrum Wiesbaden. doi:10.1007/978-3-658-30733-2. Gühring, I., Raslan, M., Kutyniok, G.,

    doi:10.1007/978-3-658-30733-2

  6. [16]

    Algebraic Statistics 15, 295–328

    Geometry of polynomial neural networks. Algebraic Statistics 15, 295–328. doi:10.2140/astat. 2024.15.295. Kusejko, K.,

    work page doi:10.2140/astat 2024

  7. [17]

    Designs, Codes and Cryptography 79, 565–581

    Simultaneous diagonalization of conics in𝑃𝐺(2,𝑞). Designs, Codes and Cryptography 79, 565–581. URL:http: //dx.doi.org/10.1007/s10623-015-0097-z, doi:10.1007/s10623-015-0097-z. Landsberg, J.M., Teitler, Z.,

    work page doi:10.1007/s10623-015-0097-z

  8. [20]

    CoRR abs/2402.17764

    The era of 1-bit llms: All large language models are in 1.58 bits. CoRR abs/2402.17764. URL:https://doi.org/10.48550/arXiv.2402.17764, doi:10.48550/ARXIV. 2402.17764,arXiv:2402.17764. MacWilliams, J.,

    Pith/arXiv arXiv doi:10.48550/arxiv.2402.17764

  9. [23]

    Expressivity of Shallow and Deep Neural Networks for Polynomial Approximation

    Expressivity of Shallow and Deep Neural Networks for Polynomial Approximation. URL:https://arxiv.org/abs/ 2303.03544, doi:10.48550/arXiv.2303.03544,arXiv:2303.03544. Trager, M., Kohn, K., Bruna, J.,

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2303.03544

  10. [24]

    (Eds.), Advances in Neural Information Processing Systems, Cur- ran Associates, Inc

    Identifiability of deep polynomial neural networks, in: Belgrave, D., Zhang, C., Lin, H., Pascanu, R., Koniusz, P., Ghassemi, M., Chen, N. (Eds.), Advances in Neural Information Processing Systems, Cur- ran Associates, Inc.. pp. 81809–81858. URL:https://proceedings.neurips.cc/paper_files/paper/2025/file/ 7575b05ab751712f950029fb96f2f999-Paper-Conference.p...

    2025

  11. [25]

    IEEE Transactions on Parallel and Distributed Systems 33, 3249–3261

    Niti: Training integer neural networks using integer-only arithmetic. IEEE Transactions on Parallel and Distributed Systems 33, 3249–3261. doi:10.1109/TPDS.2022.3149787. Wu, S., Li, G., Chen, F., Shi, L.,

    arXiv 2022

  12. [27]

    Artificial Intelligence Review 56, 12949–13013

    A comprehensive review of binary neural network. Artificial Intelligence Review 56, 12949–13013. URL: https://doi.org/10.1007/s10462-023-10464-w, doi:10.1007/s10462-023-10464-w. A. Appendix A.1. Projective space definition An affine space𝔽 𝑛 is the𝑛-fold Cartesian product of𝔽. A projective spaceℙ 𝑛−1(𝔽)is the set ℙ𝑛−1(𝔽)∶=(𝔽 𝑛−{(0,…,0)})∕∼ where(𝑥 1,…,𝑥 𝑛...

    doi:10.1007/s10462-023-10464-w 1995

  13. [414]

    Finkel, B., Rodriguez, J.I., Wu, C., Yahl, T.,

    URL:https://www.sciencedirect.com/science/article/pii/S0893608002000321, doi:https://doi.org/ 10.1016/S0893-6080(02)00032-1. Finkel, B., Rodriguez, J.I., Wu, C., Yahl, T.,

    work page doi:10.1016/s0893-6080(02)00032-1

  14. [1969]

    The American Mathematical Monthly 76, 152–164

    Orthogonal matrices over finite fields. The American Mathematical Monthly 76, 152–164. URL:https://doi.org/10.1080/00029890.1969.12000160, doi:10.1080/00029890.1969.12000160, arXiv:https://doi.org/10.1080/00029890.1969.12000160. Marchetti, G.L., Shahverdi, V ., Mereta, S., Trager, M., Kohn, K.,

    arXiv 1969

  15. [1980]

    Publications Mathématiques de l’IHÉS 52, 137–252

    La conjecture de Weil : II. Publications Mathématiques de l’IHÉS 52, 137–252. doi:10.1007/BF02684780. Draghici, S.,

    doi:10.1007/bf02684780

  16. [1994]

    Springer Netherlands, Dordrecht

    Information Geometry and Manifolds of Neural Networks. Springer Netherlands, Dordrecht. pp. 113–138. URL:https: //doi.org/10.1007/978-94-011-1068-6_7, doi:10.1007/978-94-011-1068-6_7. Averkov, G., Hojny, C., Merkert, M.,

    work page doi:10.1007/978-94-011-1068-6_7

  17. [2003]

    Commentarii Mathematici Helvetici 78, 494–517

    On Waring’s problem for several algebraic forms. Commentarii Mathematici Helvetici 78, 494–517. doi:10.1007/s00014-003-0769-6. Chrysos, G.G., Moschoglou, S., Bouritsas, G., Deng, J., Panagakis, Y ., Zafeiriou, S.,

    work page doi:10.1007/s00014-003-0769-6

  18. [2010]

    Foundations of Computational Mathematics 10, 339–366

    On the Ranks and Border Ranks of Symmetric Tensors. Foundations of Computational Mathematics 10, 339–366. URL:https://link.springer.com/article/10.1007/s10208-009-9055-3, doi:10.1007/s10208-009-9055-3. Liu, B., Li, F., Wang, X., Zhang, B., Yan, J.,

    doi:10.1007/s10208-009-9055-3

  19. [2013]

    URL: https://arxiv.org/abs/1311.1561,arXiv:1311.1561

    Best approximation on semi-algebraic sets and k-border rank approximation of symmetric tensors. URL: https://arxiv.org/abs/1311.1561,arXiv:1311.1561. Gholami, A., Kim, S., Dong, Z., Yao, Z., Mahoney, M.W., Keutzer, K.,

    Pith/arXiv arXiv

  20. [2015]

    3 ed., Springer, Cham

    Linear Algebra Done Right. 3 ed., Springer, Cham. URL:https://link.springer.com/book/10.1007/ 978-3-319-11080-6, doi:10.1007/978-3-319-11080-6. M. Zubkov et al.:Preprint submitted to ElsevierPage 14 of 16 Shallow NN expressivity over finite fields Baez, J.C.,

    doi:10.1007/978-3-319-11080-6

  21. [2020]

    SIAM Journal on Matrix Analysis and Applications 41, 1616–1629

    Solving the problem of simultaneous diagonalization of complex symmetric matrices via congruence. SIAM Journal on Matrix Analysis and Applications 41, 1616–1629. URL:https://doi.org/10.1137/19M1280430, doi:10.1137/19M1280430,arXiv:https://doi.org/10.1137/19M1280430. Carlini, E., Chipalkatti, J.V .,

    work page doi:10.1137/19m1280430

  22. [2021]

    1014–1023

    Polynet: Polynomial neural network for 3d shape recognition with polyshape representation, in: 2021 International Conference on 3D Vision (3DV), pp. 1014–1023. doi:10.1109/3DV53792.2021.00109. Yuan, C., Agaian, S.S.,

    arXiv 2021

  23. [2022]

    URL: https://doi.org/10.1109/TPAMI.2021.3058891, doi:10.1109/TPAMI.2021.3058891

    Deep polynomial neural networks 44, 4021–4034. URL: https://doi.org/10.1109/TPAMI.2021.3058891, doi:10.1109/TPAMI.2021.3058891. Conrad, K.,

    arXiv 2021

  24. [2023]

    Ternary weight networks, in: IEEE International Conference on Acoustics, Speech and Signal Processing ICASSP 2023, Rhodes Island, Greece, June 4-10, 2023, IEEE. pp. 1–5. URL:https://doi.org/10.1109/ICASSP49357. 2023.10094626, doi:10.1109/ICASSP49357.2023.10094626. Ma, S., Wang, H., Ma, L., Wang, L., Wang, W., Huang, S., Dong, L., Wang, R., Xue, J., Wei, F.,

    arXiv 2023

  25. [2024]

    (Eds.), Findings of the Association for Computational Linguistics: ACL 2024, Association for Computational Linguistics, Bangkok, Thailand

    A comprehensive evaluation of quantization strategies for large language models, in: Ku, L.W., Martins, A., Srikumar, V . (Eds.), Findings of the Association for Computational Linguistics: ACL 2024, Association for Computational Linguistics, Bangkok, Thailand. pp. 12186–12215. URL:https://aclanthology.org/2024.findings-acl. 726/, doi:10.18653/v1/2024.find...

    doi:10.18653/v1/2024.findings-acl.726 2024

  26. [2025]

    Springer Nature Switzerland, Cham

    Motivating Motives. Springer Nature Switzerland, Cham. pp. 309–323. URL:https://doi.org/10.1007/ 978-3-031-68934-5_11, doi:10.1007/978-3-031-68934-5_11. Bryan, J., Elek, B., Manners, F., Salafatinos, G., Vakil, R.,

    work page doi:10.1007/978-3-031-68934-5_11

  27. [2026]

    URL: https://arxiv.org/abs/2601.07222, doi:10.48550/arXiv.2601.07222,arXiv:2601.07222

    The motivic class of the space of genus0maps to the flag variety. URL: https://arxiv.org/abs/2601.07222, doi:10.48550/arXiv.2601.07222,arXiv:2601.07222. Bustamante, M.D., Mellon, P., Velasco, M.V .,

    doi:10.48550/arxiv.2601.07222