pith. sign in

arxiv: 2606.21687 · v1 · pith:X3OBP5IInew · submitted 2026-06-19 · 💻 cs.LG

Expressivity Saturation: Reduced Affine Region Usage Under Increasing Task Complexity

Xuan Qi , Yi Wei , Fanqi Yu , Manuel Lecha This is my paper

Pith reviewed 2026-06-26 14:29 UTC · model grok-4.3

classification 💻 cs.LG
keywords expressivity saturationaffine regionspiecewise-affine networksReLU networksneural network expressivityregion enumerationtask complexity
0
0 comments X

The pith

Trained piecewise-affine networks realize fewer affine regions as input-label complexity rises, even with fixed architectural capacity.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proves that the number of affine pieces realized along any affine line-segment probe through a multilayer perceptron is bounded above by an explicit product involving each layer's width and the number of breakpoints per activation. It then enumerates the actual regions realized inside bounded domains after training and finds that higher input-label complexity produces solutions that use markedly fewer of those regions. The authors term this reduction expressivity saturation and note that it occurs while worst-case capacity stays unchanged. In difficult regimes the collapse often aligns with visibly poorer decision boundaries, and training trajectories show progressive refinement of the partition.

Core claim

For multilayer perceptrons with piecewise-affine activations, the number of affine pieces realized along an affine line-segment probe is upper bounded by an explicit product of layer-wise width terms and activation breakpoint factors. Under fixed architectures and training protocols, increasing input-label complexity yields trained solutions with markedly fewer realized regions in the evaluation domain, even though worst-case architectural capacity is unchanged.

What carries the argument

Architecture-dependent upper bound on affine pieces along a line-segment probe, given by the product of per-layer widths and per-neuron breakpoint counts.

If this is right

  • A neuron-threshold lower bound exists for representing target functions that require a prescribed one-dimensional piece count.
  • Region-usage collapse in the hardest regimes coincides with visibly degraded decision boundaries.
  • Training dynamics exhibit a consistent refinement process that partitions the input domain into successively finer affine regions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Optimization may be implicitly favoring low-region solutions on complex tasks, suggesting a bias toward simpler partitions.
  • Capacity measures based on worst-case region counts may overestimate what networks actually deploy after training.
  • The saturation effect could be tested by swapping activations or adding explicit region-count penalties to the loss.

Load-bearing premise

Under fixed architectures and training protocols, increasing input-label complexity yields trained solutions with markedly fewer realized regions.

What would settle it

Enumerating regions after training the same network on a high-complexity task and finding more realized regions than on the corresponding low-complexity task would falsify the saturation claim.

Figures

Figures reproduced from arXiv: 2606.21687 by Fanqi Yu, Manuel Lecha, Xuan Qi, Yi Wei.

Figure 1
Figure 1. Figure 1: Initialization-only stress test (exact 1D counting). Left: maxγ α(f; γ) versus N (mean±std across seeds). Middle: per-seed ρmax scatter with mean±std error bars (log-scale y-axis). Right: non-violation rate of the theoretical upper bound. Controlled synthetic threshold suite. To instantiate Corollary 7 under known target complexity, we train depth-L = 3 ReLU MLPs with equal-width hidden layers [w, w, w] (w… view at source ↗
Figure 2
Figure 2. Figure 2: Exact 1D probe statistics (trained networks). Top: distribution/CDF of α(f; γ) computed by exact enumeration. Bottom: layer-wise split statistics along probes, illustrating how affine-piece refinements accumulate through depth [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Architecture-level 1D trends and threshold proxy. Top: R¯, α, and ηnorm versus N. Bottom: theoretical/practical curves together with an empirical proxy P(α ≥ Qtarget) from probe statistics. compares the theoretical necessary-condition curve from Corollary 7 with the heuristic Q/α and an empirical proxy derived from probe statistics. Summary (1D). Exact enumeration provides (i) a broad consistency check of … view at source ↗
Figure 4
Figure 4. Figure 4: Training summaries for the 2D random-label experiment with architecture [32, 32, 32], aggregated over 5 seeds. Panels a–(D) report epoch vs. training accuracy, epoch vs. training loss, epoch vs. exact affine-region count in Ω = [−1, 1]2 , and exact affine-region count vs. training accuracy, respectively, across different sample sizes. These plots complement [PITH_FULL_IMAGE:figures/full_fig_p013_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Representative exact region and decision-boundary visualizations for random labels with increasing sample size (network [32, 32, 32]). Top: training data. Middle: exactly enumerated affine regions in Ω. Bottom: decision regions. These visualizations are intended to illustrate typical geometric patterns corresponding to the quantitative summaries in [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Training summaries for the 3D random-label experiments. Left and right subpanels within each block correspond to architectures [16, 16, 16] and [32, 32, 32], respectively. Panels a–(D) report epoch vs. training accuracy, epoch vs. training loss, epoch vs. exact affine-region count in Ω = [−1, 1]3 , and exact affine-region count vs. training accuracy. These plots complement [PITH_FULL_IMAGE:figures/full_fi… view at source ↗
Figure 7
Figure 7. Figure 7: Decision-boundary evolution on make_moons. Network [16, 16, 16]. Top: exactly enumerated affine regions in Ω. Bottom: decision regions. Epochs: [0, 2, 5, 8, 10, 20, 50, 100]. domain. A practical implication is therefore interpretive rather than prescriptive: nominal expressive capacity alone may be an incomplete proxy for the nonlinear structure that is actually realized after training. From this perspecti… view at source ↗
Figure 8
Figure 8. Figure 8: Training summaries for the residualized 2D random-label experiments, aggregated over 5 seeds. The figure reports epoch vs. training accuracy and epoch vs. training loss across sample sizes for the residualized architectures considered in this appendix. These summaries complement [PITH_FULL_IMAGE:figures/full_fig_p025_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Two synthetic datasets used for visualizing training dynamics. [PITH_FULL_IMAGE:figures/full_fig_p026_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Training dynamics on the make_moons dataset in Figure 9a. Each panel shows the relationship among affine-region count, accuracy, and training epoch for a different network architecture [PITH_FULL_IMAGE:figures/full_fig_p027_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Training dynamics on the make_gaussian_quantiles dataset in Figure 9b. Each panel shows the relationship among affine-region count, accuracy, and training epoch for a different network architecture [PITH_FULL_IMAGE:figures/full_fig_p028_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Evolution of affine-region partitions and decision regions on the [PITH_FULL_IMAGE:figures/full_fig_p029_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Evolution of affine-region partitions and decision regions on the [PITH_FULL_IMAGE:figures/full_fig_p030_13.png] view at source ↗
read the original abstract

Piecewise-affine neural networks (e.g., with ReLU or LeakyReLU activations) implement continuous piecewise-affine maps, and the number of affine regions provides a natural proxy for expressive capacity. However, the gap between theoretical region capacity and the affine regions realized after training remains insufficiently understood. We study this gap from two complementary perspectives. First, we give a rigorous, architecture-dependent theorem for affine line-segment probes: for multilayer perceptrons with piecewise-affine activations, the number of affine pieces realized along an affine line-segment probe is upper bounded by an explicit product of layer-wise width terms (and activation breakpoint factors). This yields a neuron-threshold lower bound for representing target functions with prescribed one-dimensional piece complexity, formalizing the minimal region budget required for complex signals. Second, we exactly enumerate affine regions realized within bounded 2D and higher-dimensional domains under controlled task complexity. Under fixed architectures and training protocols, increasing input--label complexity yields trained solutions with markedly fewer realized regions in the evaluation domain, even though worst-case architectural capacity is unchanged; we call this reduced region usage expressivity saturation. Moreover, in the most challenging regimes, 2D visualizations show that region-usage collapse often coincides with degraded decision boundaries. Finally, we visualize the training dynamics of affine-region partitions and decision boundaries, revealing a consistent refinement process during optimization.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The paper claims two main contributions for piecewise-affine networks (e.g., ReLU MLPs): (1) a rigorous architecture-dependent upper bound on the number of affine pieces realized along any affine line-segment probe, expressed as an explicit product of layer widths and breakpoint factors; (2) an empirical finding, obtained by exact enumeration of regions in bounded domains, that trained networks realize markedly fewer affine regions as input-label complexity increases under fixed architecture and training protocol, a phenomenon termed 'expressivity saturation' that often coincides with degraded decision boundaries.

Significance. If the empirical saturation result survives controls for optimization quality, the work would usefully quantify the gap between worst-case architectural capacity and post-training realized expressivity, complementing existing region-counting theory with a concrete training-dependent reduction. The line-segment bound itself supplies a clean neuron-threshold lower bound for one-dimensional target complexity.

major comments (3)
  1. [enumeration experiments] Enumeration experiments (abstract): the central claim that 'increasing input-label complexity yields trained solutions with markedly fewer realized regions' lacks any reported controls that hold final training loss or test accuracy fixed across complexity levels; without such controls the reduction could simply track underfitting rather than an intrinsic saturation effect.
  2. [theorem] Line-segment theorem (abstract): while the upper bound is stated to be architecture-dependent, the manuscript provides no post-training comparison of realized region counts against this explicit bound, leaving open whether the observed saturation is operating near the architectural limit or well below it.
  3. [2D visualizations] 2D visualizations (abstract): the observation that region collapse 'often coincides with degraded decision boundaries' is presented without quantitative metrics (e.g., boundary error or loss values) that would allow readers to assess the strength of the coincidence.
minor comments (2)
  1. [abstract] The abstract refers to 'exact enumeration' but supplies no description of the enumeration algorithm, its computational complexity, or verification that all regions inside the bounded domain are counted without omission.
  2. [enumeration experiments] No error bars, statistical tests, or baseline comparisons (random initialization, untrained networks) are mentioned for the reported region counts.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive comments. We address each major point below and indicate the revisions we will make to strengthen the empirical support.

read point-by-point responses

  1. Referee: Enumeration experiments (abstract): the central claim that 'increasing input-label complexity yields trained solutions with markedly fewer realized regions' lacks any reported controls that hold final training loss or test accuracy fixed across complexity levels; without such controls the reduction could simply track underfitting rather than an intrinsic saturation effect.

    Authors: We agree that the absence of controls for final loss or accuracy is a valid concern and could indicate underfitting rather than saturation. The experiments fix the training protocol but do not enforce matched performance. In revision we will report training and test losses/accuracies for every complexity level and, where feasible, add matched-performance controls by varying epoch count or using early stopping before re-enumerating regions. revision: yes

  2. Referee: Line-segment theorem (abstract): while the upper bound is stated to be architecture-dependent, the manuscript provides no post-training comparison of realized region counts against this explicit bound, leaving open whether the observed saturation is operating near the architectural limit or well below it.

    Authors: The theorem supplies an explicit upper bound only for one-dimensional line-segment probes. Our enumeration results concern full two-dimensional domains. We will add, in the revised manuscript, a direct comparison of realized affine pieces along random line segments extracted from the trained networks against the theorem bound, thereby clarifying the distance between observed saturation and architectural capacity. revision: yes

  3. Referee: 2D visualizations (abstract): the observation that region collapse 'often coincides with degraded decision boundaries' is presented without quantitative metrics (e.g., boundary error or loss values) that would allow readers to assess the strength of the coincidence.

    Authors: We acknowledge that the visual observation would be strengthened by quantitative metrics. The revision will include supporting numbers such as test accuracy, a boundary-error measure (fraction of points within a small distance of the decision boundary that are misclassified), and a simple correlation between region count and these quantities. revision: yes

Circularity Check

0 steps flagged

No circularity: theorem is architecture-derived bound; saturation is empirical observation

full rationale

The paper presents two distinct contributions. The first is an architecture-dependent upper bound theorem on affine pieces along line-segment probes, expressed as an explicit product of layer widths and breakpoint factors; this is a standard counting argument over piecewise-affine maps and does not reduce any reported quantity to a fitted parameter or self-citation by construction. The second is an empirical enumeration showing fewer realized regions under higher input-label complexity under fixed training; this is a direct measurement, not a prediction derived from the same data or from any fitted input. No self-citation load-bearing steps, uniqueness theorems, or ansatz smuggling appear in the abstract or described claims. The derivation chain remains self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The central claims rest on the domain assumption that affine-region counts are a faithful proxy for expressivity and that task complexity can be varied independently while keeping architecture and optimizer fixed. No free parameters or invented entities are stated in the abstract.

axioms (1)
  • domain assumption Piecewise-affine neural networks implement continuous piecewise-affine maps whose affine regions serve as a natural proxy for expressive capacity.
    Opening sentence of abstract; used to motivate both the theorem and the saturation experiments.
invented entities (1)
  • expressivity saturation no independent evidence
    purpose: Name for the observed reduction in realized affine regions as task complexity increases.
    Introduced in abstract to label the empirical finding; no independent evidence supplied.

pith-pipeline@v0.9.1-grok · 5775 in / 1372 out tokens · 25164 ms · 2026-06-26T14:29:18.594423+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

21 extracted references · 13 canonical work pages · 2 internal anchors

  1. [1]

    Fast and exact enumeration of deep networks partitions regions.arXiv preprint arXiv:2401.11188,

    Randall Balestriero and Yann LeCun. Fast and exact enumeration of deep networks partitions regions.arXiv preprint arXiv:2401.11188,

    work page arXiv

  2. [2]

    Fast and exact enumeration of deep networks partitions regions.arXiv preprint arXiv:2401.11188,

    doi: 10.48550/arXiv.2401.11188. Arturs Berzins. Polyhedral complex extraction from ReLU networks using edge subdivision. InProc. Int. Conf. Mach. Learn. (ICML), volume 202, pp. 2234–2244. PMLR,

    work page doi:10.48550/arxiv.2401.11188

  3. [3]

    Kuan-Lin Chen, Harinath Garudadri, and Bhaskar D

    doi: 10.1016/j.neunet.2023.09.025. Kuan-Lin Chen, Harinath Garudadri, and Bhaskar D. Rao. Improved bounds on neural complexity for representing piecewise linear functions. InAdvances in Neural Information Processing Systems, volume 35,

    work page doi:10.1016/j.neunet.2023.09.025 2023

  4. [4]

    doi: 10.1016/j.cam.2023.115667. J. Elisenda Grigsby and Kathryn Lindsey. On transversality of bent hyperplane arrangements and the topological expressiveness of ReLU neural networks.SIAM J. Appl. Algebra Geom., 6(2):216–242,

    work page doi:10.1016/j.cam.2023.115667 2023

  5. [5]

    Christian Haase, Christoph Hertrich, and Georg Loho

    doi: 10.1137/20M1368902. Christian Haase, Christoph Hertrich, and Georg Loho. Lower bounds on the depth of integral ReLU neural networks via lattice polytopes. InProc. Int. Conf. Learn. Represent. (ICLR). OpenReview.net,

    work page doi:10.1137/20m1368902

  6. [6]

    17 Published in Transactions on Machine Learning Research (06/2026) Christoph Hertrich, Amitabh Basu, Marco Di Summa, and Martin Skutella

    doi: 10.4208/jcm.1901-m2018-0160. 17 Published in Transactions on Machine Learning Research (06/2026) Christoph Hertrich, Amitabh Basu, Marco Di Summa, and Martin Skutella. Towards lower bounds on the depth of ReLU neural networks.SIAM J. Discret. Math., 37(2):997–1029,

    work page doi:10.4208/jcm.1901-m2018-0160 1901

  7. [7]

    Qiang Hu, Hao Zhang, Feifei Gao, Chengwen Xing, and Jianping An

    doi: 10.1137/22M1489332. Qiang Hu, Hao Zhang, Feifei Gao, Chengwen Xing, and Jianping An. Analysis on the number of linear regions of piecewise linear neural networks.IEEE Trans. Neural Netw. Learn. Syst., 33(2):644–653,

    work page doi:10.1137/22m1489332

  8. [8]

    Ahmed Imtiaz Humayun, Randall Balestriero, Guha Balakrishnan, and Richard G

    doi: 10.1109/TNNLS.2020.3028431. Ahmed Imtiaz Humayun, Randall Balestriero, Guha Balakrishnan, and Richard G. Baraniuk. Splinecam: Exact visualization and characterization of deep network geometry and decision boundaries. InProc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pp. 3789–3798. IEEE,

    work page doi:10.1109/tnnls.2020.3028431 2020

  9. [9]

    Hu Wang, Yuanhong Chen, Congbo Ma, et al

    doi: 10.1109/ CVPR52729.2023.00369. Zhiwei Li and Cheng Wang. Achieving sharp upper bounds on the expressive power of neural networks via tropical polynomials.IEEE Trans. Neural Netw. Learn. Syst., 36(2):2931–2945,

    work page arXiv 2023

  10. [10]

    Adversarial examples: Attacks and defenses for deep learning

    doi: 10.1109/TNNLS. 2024.3350786. Andrew L. Maas, Awni Y. Hannun, and Andrew Y. Ng. Rectifier nonlinearities improve neural network acoustic models. InProc. ICML Workshop on Deep Learning for Audio, Speech and Language Processing, Atlanta, GA, USA,

    work page doi:10.1109/tnnls 2024

  11. [11]

    Training-Time Batch Normalization Reshapes Local Partition Geometry in Piecewise-Affine Networks

    Xuan Qi, Yi Wei, Xue Mei, Ryad Chellali, and Shipin Yang. Comparative analysis of the linear regions in relu and leakyrelu networks. InInternational Conference on Neural Information Processing, pp. 528–539. Springer, 2023a. Xuan Qi, Yi Wei, Xue Mei, Ryad Chellali, and Shipin Yang. Empirical study on the effect of residual networks on the expressiveness of...

    work page internal anchor Pith review Pith/arXiv arXiv

  12. [12]

    The computational complexity of counting linear regions in relu neural networks.Advances in Neural Information Processing Systems,

    18 Published in Transactions on Machine Learning Research (06/2026) Moritz Stargalla, Christoph Hertrich, and Daniel Reichman. The computational complexity of counting linear regions in relu neural networks.Advances in Neural Information Processing Systems,

    2026

  13. [13]

    Region Seeding via Pre-Activation Regularization: A Geometric View of Piecewise Affine Neural Networks

    doi: 10.24963/ijcai.2022/492. Yi Wei, Xuan Qi, and Furao Shen. Region seeding via pre-activation regularization: A geometric view from piecewise affine nerual networks.arXiv preprint arXiv:2605.06300, 2026a. Yi Wei, Xuan Qi, Furao Shen, Jian Zhao, Vittorio Murino, and Cigdem Beyan. Affinelens: Capturing the continuous piecewise affine functions of neural ...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.24963/ijcai.2022/492 2022

  14. [14]

    Xiao Zhang and Dongrui Wu

    doi: 10.1109/TPAMI.2024.3361155. Xiao Zhang and Dongrui Wu. Empirical studies on the properties of linear regions in deep neural networks. InProc. Int. Conf. Learn. Represent. (ICLR). OpenReview.net,

    work page doi:10.1109/tpami.2024.3361155 2024

  15. [15]

    This is the standard setting for line-restricted region counting

    19 Published in Transactions on Machine Learning Research (06/2026) A Proofs A.1 Proof of Theorem 5 Throughout this subsection, we consider aline-segment probe γ(t) =x0 +tv, t∈[0,1],(14) for somex 0,v∈Rd. This is the standard setting for line-restricted region counting. CPWA activation.Letσ:R→Rbe CPWA with (finite) breakpoint set −∞=t0 <t 1 <···<tK <t K+1...

    2026

  16. [16]

    Let[0, 1]be partitioned into Rℓ−1intervals as in equation 20 such thathℓ−1◦γis affine on each open interval

    21 Published in Transactions on Machine Learning Research (06/2026) Inductive step.Fix ℓ∈{1,...,L}and assume equation 32 holds forℓ−1. Let[0, 1]be partitioned into Rℓ−1intervals as in equation 20 such thathℓ−1◦γis affine on each open interval. On each such open interval, Lemma 11 gives a refinement into at mostKnℓ+ 1open subintervals on whichhℓ◦γis affine...

    2026

  17. [17]

    = 1 L ( K L∑ ℓ=1 nℓ+ L∑ ℓ=1 1 ) ,(41) = 1 L(KN+L),(42) = 1 +KN L .(43) Combining equation 39, equation 40, and equation 43 gives Q(g)≤ ( 1 +KN L )L .(44) 22 Published in Transactions on Machine Learning Research (06/2026) TakingL-th roots and rearranging yields N≥L K ( Q(g)1/L−1 ) .(45) Ifn ℓ=wfor allℓ, then equation 39 becomes (Kw+

    2026

  18. [18]

    L ≥Q(g),(46) equivalently w≥Q(g)1/L−1 K .(47) Sincewis an integer, we obtain w≥ ⌈Q(g)1/L−1 K ⌉ ,(48) as claimed. 23 Published in Transactions on Machine Learning Research (06/2026) B Residual-Connection Ablation under Complex 2D Inputs This appendix reports an additional ablation for the 2D random-label setting in which the baseline MLP is augmented with ...

    2026

  19. [19]

    These plots provide optimization context for the final exact region counts in Table 5 and show that the optimization trajectories vary across both sample size and architecture

    Architecture (with one residual) 200 500 1000 2000 3000 5000 7000 10000 [16,16,16]+ residual 2049 ±17 2176 ±43 2189 ±8 1874 ±31 853 ±24 512 ±46 347 ±13 48±7 [32,32,32]+ residual 3748 ±29 5597 ±6 3793 ±41 2147 ±18 2908 ±35 741 ±9 911 ±48 115 ±12 [64,64,64]+ residual 7613 ±12 9050 ±44 7421 ±27 6879 ±5 2683 ±39 1287 ±16 594 ±33 98±15 Training accuracy and lo...

    2000

  20. [20]

    The larger architectures start with and maintain substantially more regions than the smaller architecture, reflecting their larger architectural capacity

    This trend is consistent with the visual intuition that the networks progressively refine their piecewise-affine partitions in order to represent the curved binary decision boundary. The larger architectures start with and maintain substantially more regions than the smaller architecture, reflecting their larger architectural capacity. At the same time, t...

    2026

  21. [21]

    29 Published in Transactions on Machine Learning Research (06/2026) (a)[16,16,16]

    Together with the make_moons results, these visualizations support the view that, in structured and learnable settings, optimization can refine both the affine-region partition and the corresponding decision boundary over time. 29 Published in Transactions on Machine Learning Research (06/2026) (a)[16,16,16]. (b)[32,32,32]. (c)[64,64,64]. Figure 13: Evolu...

    2026