REVIEW 1 major objections 4 minor 13 references
Toric geometry of ReLU neural networks
T0 review · 1 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read A toric intersection equality classifies shallow ReLU realizability
desk verdict A genuinely new toric dictionary for ReLU networks, but the headline realizability criterion is not necessary as stated and the sufficiency proof has a gap. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the toric encoding map, which sends a parameter $\theta$ of an unbiased rational ReLU network to its ReLU fan $\Sigma_{f_\theta}$, defined as the canonical polyhedral complex on which the output and all hidden-layer functions are linear. On this fan one places the ReLU Cartier divisor $D_f$, a $\mathbb{Q}$-divisor whose support function is the network output up to a linear term, and one computes its intersection numbers $D_f \cdot V(\tau)$ with the complete torus-invariant curves $V(\tau)$ indexed by walls $\tau$. These numbers are the amount by which the output function bends across a wall, and the classification is precisely that these bend numbers are constant along the walls of each hyperplane; the proof of sufficiency builds the hidden layer from the hyperplane normals and the last layer from the bend numbers.
What would settle it
Take $f = \max\{0, x\}$ on $\mathbb{R}^2$, which is trivially in $\operatorname{ReLU}^{\mathbb{Q}}_0(2,1)$; its canonical polyhedral complex has two halfspace cells along the line $x=0$, neither strongly convex. Writing down the ReLU fan, the polytope $P_D$, or the intersection numbers $D_f \cdot V(\tau)$ for this $f$ is impossible under the paper's definitions, so the claimed complete classification does not apply to it; any repair of this gap, by subdivision or by allowing generalized fans, can be tested by checking whether the equal-bend criterion still holds after the repair.
Extended reading notes
Core claim
On its own terms, the paper's central claim is that a continuous finitely piecewise linear function $f: \mathbb{R}^{n_0} \to \mathbb{R}$ is exactly realizable by an unbiased shallow ReLU network with rational weights if and only if, after extending the codimension-one part of its non-linear locus to full hyperplanes, those hyperplanes form a fan $\Sigma$ in $\mathbb{R}^{n_0}$ and the ReLU Cartier divisor $D_f$ supported on $\Sigma$ satisfies $D_f \cdot V(\tau_1) = D_f \cdot V(\tau_2)$ for any two walls $\tau_1, \tau_2$ lying in the same hyperplane (Theorems 3 and 4). Here $D_f \cdot V(\tau)$ measures how much the output function bends along the wall $\tau$. The paper further claims that when $f$ is a tropical polynomial, the polytope $P_{-D}$ equals $-\operatorname{Newt}(f)$ and $\operatorname{Vol}(\operatorname{Newt}(f))$ equals the volume of the line bundle $\mathcal{O}_{X_{\Sigma_{f_\theta}}}(-D)$, establishing a bridge between tropical and toric geometry of ReLU networks.
Load-bearing premise
The construction assumes the canonical polyhedral complex is a fan of strongly convex cones, justified only by taking more hidden neurons than input dimensions; when the hidden-layer normals do not span the input space, cells can contain lines and the ReLU toric variety, the ReLU Cartier divisor, and the intersection numbers are not defined.
Editorial extensions
If this is right
- Exact realizability by a one-hidden-layer unbiased rational ReLU network can be checked by finitely many intersection-number computations, with no search over widths or weights.
- Any function violating the equal-bend condition on some hyperplane is provably not realizable by any shallow unbiased rational ReLU network.
- Adding a linear term to a realizable function preserves realizability, so the classification is really a classification of realizable $\mathbb{Q}$-Cartier divisors on the ReLU fan.
- For tropical-polynomial outputs, mixed volume of the Newton polytope equals the volume of the associated line bundle on the ReLU toric variety, giving a toric interpretation of tropical expressivity.
- When the fan condition holds, the hidden width needed is at most the number of distinct hyperplanes of the fan, not the number of linear regions.
Reading between the lines
- Editorial extension: if the equal-intersection condition is rephrased in terms of divisor classes modulo linear functions, realizability testing for fixed $n_0$ and $n_1$ would reduce to a linear-algebra membership problem, which could be implemented by evaluating $f$ on a few generic points.
- Editorial extension: the strong-convexity assumption could likely be repaired by subdividing non-pointed cells; functions like $\max\{0,x\}$ on $\mathbb{R}^2$ would then enter the classification, and the same intersection-number criterion should survive after subdivision.
- Editorial extension: the divisor polytope $P_D$ proposed as a Newton-polytope analogue may give an expressivity measure for non-convex tropical rational functions, testable by comparing volumes across architectures.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper develops a dictionary between unbiased rational-weight ReLU neural networks and toric geometry. For a network with parameters θ, it defines the "ReLU fan" Σ_{f_θ} as the canonical polyhedral complex of the network, the associated toric variety X_{Σ_{f_θ}}, and a Q-Cartier "ReLU divisor" D_f whose support function is the network output. The paper then connects this toric data to tropical geometry, proving that for tropical-polynomial outputs the divisor polytope equals the negative Newton polytope and that volumes agree. The main application is a claimed complete classification of functions realized by unbiased shallow ReLU networks with rational weights: Theorem 3 gives a necessary condition involving equality of intersection numbers D_f·V(τ) for walls from the same hyperplane, Theorem 4 gives a converse, and Example 7.6 presents the resulting "equivalent condition."
Significance. If the classification were established, the paper would contribute a genuinely new algebraic-geometric invariant to the study of exact ReLU realizability, and the toric/tropical bridge (Theorems 6.9 and 6.14) is an elegant and potentially useful observation. The paper is self-contained, with many worked examples and a careful exposition of toric background, and it gives an explicit reduction algorithm for shallow ReLU representations. However, the central classification claim is not supported as stated: the standing fan assumption excludes realizable functions, and the converse proof contains a load-bearing unsupported assertion. The framework is promising and likely repairable, but the advertised necessary-and-sufficient characterization is currently false in its stated form.
major comments (1)
- [§5, Remark 5.2; §7, Theorems 3 and 4 and Example 7.6] In the proof of Theorem 4 (§7), the assertion that if the constructed network output f' is not exactly f then f − f' is linear is made without proof. The construction chooses the rows of L1 from the hyperplanes of Σ and the entries of L2 from the common intersection numbers t_i, but it is not shown that the piecewise linear function determined by these data agrees with f up to a linear function. Equality of wall intersection numbers alone does not obviously determine the Cartier data (and hence the support function) up to a global linear term; an argument is needed, for example by comparing the Cartier data cone-by-cone or by showing the difference vanishes on each maximal cone. This is load-bearing for the sufficiency direction, and without it Theorem 4 is not established.
minor comments (4)
- [§7, Example 7.6] Example 7.6 is too terse to be useful: the function is not actually defined in the text, and the displayed expressions "y 0 x+2y 3y −4y 2x−2y" do not specify the piecewise linear function, the architecture, or the intended bent hyperplane arrangement. The example should be rewritten with an explicit formula or diagram.
- [§6, Remark 6.2] There are several typos throughout, e.g., "Negavie infinity" in Remark 6.2, "the funcion" in Remark 4.48, "finitely-piecewise" in Theorem 4, and "polytop" in Lemma 6.13. These should be corrected.
- [§5, Definition 5.1 and Remark 5.2] The terminology is potentially confusing: Definition 5.1 calls Σ_{f_θ} a "ReLU fan" even though it is defined as the canonical polyhedral complex, which the authors themselves note is only a generalized fan in general. It would be clearer to use a distinct name (e.g., "ReLU complex") until the fan condition is actually established or imposed.
- [§7, Theorem 3] The statement of Theorem 3 refers to "the n1 hyperplanes H_1^{(1)},...,H_{n1}^{(1)}", but if two rows of L1 are opposite normals they define the same hyperplane; the indexing should be clarified to avoid double-counting, especially since Lemma 7.4 explicitly handles this case.
Circularity Check
No significant circularity: the realizability criterion is derived by explicit construction, not by definitional reduction.
full rationale
The paper's central classification (Theorems 3 and 4) is not circular. Theorem 3 derives the equality of intersection numbers from an explicit reduced representation f_red = sum b_i max{0, a_i dot x}; the equality is a consequence of the representation, not an input to it. Theorem 4 goes in the opposite direction: given a piecewise linear function whose kink hyperplanes form a fan and whose divisor has constant intersection numbers, the proof explicitly sets L1 rows to the primitive normals a_i and L2 entries to t_i = -D_f dot V(tau), and then uses the separately proven Theorem 1 to absorb a linear remainder; realizability is never assumed. The toric objects (ReLU fan, ReLU toric variety, ReLU Cartier divisor) are defined from the network and function before the realizability statement is proved, so the iff criterion is not built into the definitions. The cited works [3], [5], and [13] supply standard toric background and the canonical polyhedral complex / tropical rational function facts; they are not self-citations of the author and none of them carries the classification. No parameter is fitted, no subset of data is 'predicted', and no uniqueness theorem is imported from the author's prior work. The main weakness of the paper is a correctness gap, not a circularity: Remark 5.2 assumes n1 > n0 makes the canonical polyhedral complex a fan of strongly convex cones, which is false for f(x,y)=max{0,x}; but this makes Theorems 3 and 4 incomplete or ill-defined for some realizable functions rather than making their conclusions equivalent to their hypotheses.
Assumptions & free parameters
assumptions (5)
- domain assumption The canonical polyhedral complex of an unbiased ReLU network with rational weights is a fan of strongly convex rational polyhedral cones.
- domain assumption The realized function f is positively homogeneous because biases are excluded, so it is a support function on the ReLU fan.
- standard math Standard toric geometry facts: Cartier divisors, intersection numbers with invariant curves, line bundle volumes, and Ehrhart theory.
- domain assumption For Theorem 2 (Theorem 6.9), f is a tropical polynomial, which is a convex (Definition 3.8) homogeneous CPWL function, and its coefficients do not affect Newt(f).
- ad hoc to paper The reduced ReLU representation normalization (Definition 7.1) preserves the realized function and the reduction algorithm is correct.
Cite this review
Pith. "Pith review of Toric geometry of ReLU neural networks." pith.science (2026). https://pith.science/paper/RSJG2TCY
@misc{pith2026250905894,
author = {Pith},
title = {Pith review of: Toric geometry of ReLU neural networks},
year = {2026},
howpublished = {\url{https://pith.science/paper/RSJG2TCY}},
note = {Machine review of arXiv:2509.05894}
}
abstract
Given a continuous finitely piecewise linear function $f:\mathbb{R}^{n_0} \to \mathbb{R}$ and a fixed architecture $(n_0,\ldots,n_k;1)$ of feedforward ReLU neural networks, the exact function realization problem is to determine when some network with the given architecture realizes $f$. To develop a systematic way to answer these questions, we establish a connection between toric geometry and ReLU neural networks. This approach enables us to utilize numerous structures and tools from algebraic geometry to study ReLU neural networks. Starting with an unbiased ReLU neural network with rational weights, we define the ReLU fan, the ReLU toric variety, and the ReLU Cartier divisor associated with the network. This work also reveals the connection between the tropical geometry and the toric geometry of ReLU neural networks. As an application of the toric geometry framework, we prove a necessary and sufficient criterion of functions realizable by unbiased shallow ReLU neural networks by computing intersection numbers of the ReLU Cartier divisor and torus-invariant curves.
Reference graph
Works this paper leans on
-
[1]
Understanding deep neural networks with rectified linear units
Raman Arora, Amitabh Basu, Poorya Mianjy, and Anirbit Mukherjee. Understanding deep neural networks with rectified linear units. In6th International Conference on Learning Representations (ICLR), 2018
work page 2018
-
[2]
Better neural network expressivity: subdividing the simplex.arXiv preprint arXiv:2505.14338, 2025
Egor Bakaev, Florestan Brunck, Christoph Hertrich, Jack Stade, and Amir Yehu- dayoff. Better neural network expressivity: subdividing the simplex.arXiv preprint arXiv:2505.14338, 2025
arXiv 2025
-
[3]
American Mathematical Soc., 2011
David A Cox, John B Little, and Henry K Schenck.Toric varieties, volume 124. American Mathematical Soc., 2011
work page 2011
-
[4]
Activation degree thresholds and expressiveness of polynomial neural networks
Bella Finkel, Jose Israel Rodriguez, Chenxi Wu, and Thomas Yahl. Activation degree thresholds and expressiveness of polynomial neural networks. 2025
work page 2025
-
[5]
J Elisenda Grigsby and Kathryn Lindsey. On transversality of bent hyperplane ar- rangements and the topological expressiveness of relu neural networks.SIAM Journal on Applied Algebra and Geometry, 6(2):216–242, 2022
work page 2022
-
[6]
Christian Haase, Christoph Hertrich, and Georg Loho. Lower bounds on the depth of integral relu neural networks via lattice polytopes.arXiv preprint arXiv:2302.12553, 2023
arXiv 2023
-
[7]
Christoph Hertrich, Amitabh Basu, Marco Di Summa, and Martin Skutella. Towards lower bounds on the depth of relu neural networks.SIAM Journal on Discrete Math- ematics, 37(2):997–1029, 2023
work page 2023
-
[8]
Geometry of polynomial neural networks.Algebraic Statistics, 15(2):295–328, 2024
Kaie Kubjas, Jiayi Li, and Maximilian Wiesmann. Geometry of polynomial neural networks.Algebraic Statistics, 15(2):295–328, 2024
work page 2024
Show all 13 references
-
[9]
American Mathematical Society, 2021
Diane Maclagan and Bernd Sturmfels.Introduction to tropical geometry, volume 161. American Mathematical Society, 2021
2021
-
[10]
Algebra unveils deep learning–an invitation to neuroalgebraic geom- etry.arXiv preprint arXiv:2501.18915, 2025
Giovanni Luca Marchetti, Vahid Shahverdi, Stefano Mereta, Matthew Trager, and Kathl´ en Kohn. Algebra unveils deep learning–an invitation to neuroalgebraic geom- etry.arXiv preprint arXiv:2501.18915, 2025
2025 arXiv
-
[11]
Approximation theory of the mlp model in neural networks.Acta Nu- merica, 8:143–195, 1999
Allan Pinkus. Approximation theory of the mlp model in neural networks.Acta Nu- merica, 8:143–195, 1999
1999
-
[12]
Generalization of hinging hyperplanes.IEEE Trans- actions on Information Theory, 51(12):4425–4431, 2005
Shuning Wang and Xusheng Sun. Generalization of hinging hyperplanes.IEEE Trans- actions on Information Theory, 51(12):4425–4431, 2005. TORIC GEOMETRY OF RELU NEURAL NETWORKS 27
2005
-
[13]
Tropical geometry of deep neu- ral networks
Liwen Zhang, Gregory Naitzat, and Lek-Heng Lim. Tropical geometry of deep neu- ral networks. InInternational Conference on Machine Learning, pages 5824–5832. PMLR, 2018. Boston College; Department of Mathematics; 521 Maloney Hall; Chestnut Hill, MA 02467 Email address:fugh@bc.edu
2018
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.