REVIEW 3 cited by
On the Depth of Monotone ReLU Neural Networks and ICNNs
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
We study two models of ReLU neural networks: monotone networks (ReLU$^+$) and input convex neural networks (ICNN). Our focus is on expressivity, mostly in terms of depth, and we prove the following lower bounds. For the maximum function MAX$_n$ computing the maximum of $n$ real numbers, we show that ReLU$^+$ networks cannot compute MAX$_n$, or even approximate it. We prove a sharp $n$ lower bound on the ICNN depth complexity of MAX$_n$. We also prove depth separations between ReLU networks and ICNNs; for every $k$, there is a depth-2 ReLU network of size $O(k^2)$ that cannot be simulated by a depth-$k$ ICNN. The proofs are based on deep connections between neural networks and polyhedral geometry, and also use isoperimetric properties of triangulations.
Forward citations
Cited by 3 Pith papers
-
Approximation Depth of Convex Polytopes
For every depth d below the exact depth of the n-simplex, every depth-d polytope misses the simplex by an empty-corner distance of exactly n+1-2^d.
-
Tropical Circuits with Scalar Multiplication Gates
Scalar tropical circuits require 2^Ω(n) plus gates for the Birkhoff and directed spanning-tree polytopes, implying exponential mnnc vs polynomial nnc for directed spanning trees.
-
A simplex-based measure of symmetry
A simplex-based symmetry ratio recovers Minkowski measure after affine invariance, improves its stability, characterizes simplices by outer additivity, and sharply bounds the ratio for low-depth polytopes.
Discussion (0). Continue with ORCID to comment.