REVIEW 7 cited by
The Evolution of the Interplay Between Input Distributions and Linear Regions in Networks
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
The Evolution of the Interplay Between Input Distributions and Linear Regions in Networks
read the original abstract
It is commonly recognized that the expressiveness of deep neural networks is contingent upon a range of factors, encompassing their depth, width, and other relevant considerations. Currently, the practical performance of the majority of deep neural networks remains uncertain. For ReLU (Rectified Linear Unit) networks with piecewise linear activations, the number of linear convex regions serves as a natural metric to gauge the network's expressivity. In this paper, we count the number of linear convex regions in deep neural networks based on ReLU. In particular, we prove that for any one-dimensional input, there exists a minimum threshold for the number of neurons required to express it. We also empirically observe that for the same network, intricate inputs hinder its capacity to express linear regions. Furthermore, we unveil the iterative refinement process of decision boundaries in ReLU networks during training. We aspire for our research to serve as an inspiration for network optimization endeavors and aids in the exploration and analysis of the behaviors exhibited by deep networks.
Forward citations
Cited by 7 Pith papers
-
AffineLens: Capturing the Continuous Piecewise Affine Functions of Neural Networks
AffineLens computes and visualizes the maximal continuous piecewise affine regions induced by neural networks, supporting batch-norm, pooling, residuals, MLPs and convolutions, and enables empirical comparison of arch...
-
AffineLens: Capturing the Continuous Piecewise Affine Functions of Neural Networks
AffineLens enumerates the continuous piecewise affine regions induced by neural networks within a given bounded input polytope and provides visualizations and region-complexity metrics.
-
Training-Time Batch Normalization Reshapes Local Partition Geometry in Piecewise-Affine Networks
Training-time BN in CPA networks recenters switching hyperplanes at batch centroids, increasing expected local partition refinement under explicit sufficient conditions.
-
Training-Time Batch Normalization Reshapes Local Partition Geometry in Piecewise-Affine Networks
Training-time batch normalization increases expected local affine-region density in ReLU and piecewise-affine networks by acting as a batch-conditional recentering mechanism on switching hyperplanes.
-
Region Seeding via Pre-Activation Regularization: A Geometric View of Piecewise Affine Neural Networks
A geometric theory yields a region-seeding regularizer that increases realized affine regions and improves early accuracy in piecewise affine neural networks.
-
AffineLens: Capturing the Continuous Piecewise Affine Functions of Neural Networks
AffineLens enumerates the maximal continuous piecewise-affine regions induced by neural networks with batch-norm, pooling, residuals and convolutions inside a bounded input polytope and supplies visualizations and reg...
-
Region Seeding via Pre-Activation Regularization: A Geometric View of Piecewise Affine Neural Networks
A pre-activation regularizer seeds more affine regions near data in piecewise affine networks, increasing local region count and improving early training performance.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.