pith. sign in

arxiv: 2510.12700 · v2 · submitted 2025-10-14 · 💻 cs.LG · cs.AI· cs.CG· math.AT· stat.ML

Topological Signatures of ReLU Neural Network Activation Patterns

Vicente Bosca , Tatum Rask , Sunia Tanweer , Andrew R. Tawfeek , Branden Stone This is my paper

Pith reviewed 2026-05-18 07:11 UTC · model grok-4.3

classification 💻 cs.LG cs.AIcs.CGmath.ATstat.ML
keywords ReLU networkstopological signaturesFiedler partitionpolytope decompositiondecision boundaryhomologyactivation patternsbinary classification
0
0 comments X

The pith

The Fiedler partition of the dual graph correlates with the decision boundary in binary classification for ReLU networks.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper examines the polytope decomposition of the input space created by ReLU activation patterns in feedforward networks. It shows that the Fiedler partition of the resulting dual graph tends to align with where the network separates classes in binary classification. In regression experiments the homology of the cellular decomposition reveals that the count of polyhedral cells changes in step with the training loss curve. A reader might care because these topological measures give a concrete way to inspect how the network carves up feature space without relying solely on input-output tests.

Core claim

The authors establish that the Fiedler partition of the dual graph of the polytope decomposition induced by ReLU activations appears to correlate with the decision boundary in the case of binary classification. They further compute the homology of the cellular decomposition in a regression task and observe similar behavioral patterns between the training loss and the polyhedral cell-count as the model is trained.

What carries the argument

The Fiedler partition of the dual graph of the polytope decomposition, which identifies a low-connectivity cut across adjacent activation regions and is checked against the network's separating hyperplanes.

If this is right

  • Topological inspection of the dual graph could locate decision boundaries without exhaustive sampling of the input space.
  • Homology-based cell counts could serve as an auxiliary signal for tracking how model complexity grows during training.
  • These signatures might distinguish between networks that achieve similar accuracy through qualitatively different partitions of the feature space.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same partition technique could be applied to multi-class problems by examining the collection of cuts rather than a single binary split.
  • If the correlation is robust, adversarial inputs might preferentially appear near the topological boundaries identified by the Fiedler vector.
  • Extending the analysis to networks with skip connections or attention layers could test whether the polytope-dual-graph relationship persists beyond plain feedforward ReLU stacks.

Load-bearing premise

The observed alignment between the Fiedler partition and the decision boundary is a genuine property of ReLU networks rather than an artifact of the particular architectures, datasets, or training procedures used.

What would settle it

A trained ReLU network on a simple binary classification problem in which the Fiedler partition of the activation dual graph fails to overlap with the locations of the learned decision boundary.

Figures

Figures reproduced from arXiv: 2510.12700 by Andrew R. Tawfeek, Branden Stone, Sunia Tanweer, Tatum Rask, Vicente Bosca.

Figure 1
Figure 1. Figure 1: Example of polyhedral decomposition of the domain of a FFRNN. Grayed out activation function outputs correspond to ReLU value of zero at that neuron, indicating a 0 entry in the binary state vector. R n of the neural network, as defined in [20] (see [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The two circles and two moons datasets (left column), their respective polytope decompositions (middle left), and the partitions of the dual graph by the unweighted (middle right) and weighted Fiedler vectors (right), indicated by the blue and purple points. Minimal loss and concentration along decision boundary demonstrates grokking behavior. Because of the success of these small examples, we believe that… view at source ↗
Figure 3
Figure 3. Figure 3: (Left) Prediction of a FFRNN trained to learn a paraboloid surface. (Middle) Snapshots of a random filtration on the primal complex of the polyhedral decomposition of such a network with their corresponding Betti numbers. First, randomly add all the vertices of the decomposition (blue), then add the edges (red, cycles are born), and finally add the faces (green, cycles are closed). (Right) Dual graph of th… view at source ↗
Figure 4
Figure 4. Figure 4: (Left) Snapshots (zoomed in to the square [0.3, 0.3]2 ) of the polyhedral decomposition of a PINN at the beginning (epoch 0) and end of training (epoch 10000). In red are the vertices (0-cells), in solid blue the edges (1-cells), and in a lighter blue the faces (2-cells). More cells are observed at the beginning of training. This number seems to decrease as we train, with the emergence of some structure. (… view at source ↗
Figure 5
Figure 5. Figure 5: Heat maps of the Betti curves corresponding to a trained PINN, with the training loss overlaid. (Left) For β0 we observe smoother changes. (Right) For β1, the changes are sharper and correlate with substantial changes in the training loss function. additional cycles. The peak occurs when all edges are added, with the peak value equaling the number of polytopes that remain to be filled. As 2-cells (faces) a… view at source ↗
Figure 6
Figure 6. Figure 6: (Left) The three components of the f-vector across training epochs for a trained PINN with the training loss overlaid. Spikes in the training loss lead to changes in those quantities. (Right) The average f0 (# of 0-cells) for the 25 trials of training a neural network using the periodic dynamical system. Overall, the number of cells decreases along training. to reveal its most significant persistent homolo… view at source ↗
Figure 7
Figure 7. Figure 7: The polyhedral decomposition of the input space R 2 of a small FFNN having architecture (2, 3, 3, 1). Some of the binary vectors for various points in R 2 are highlighted and their neural activation pattern within the network is illustrated. C Cell Complex Structure The polyhedral decomposition of a FFRNN naturally induces a cell complex structure K where: • d-dimensional cells are the polytopes themselves… view at source ↗
Figure 8
Figure 8. Figure 8: Heat maps of the betti curves for Trial 4 corresponding to a trained PINN, with the training loss overlaid. (Left) For β0 we observe smoother changes. (Right) For β1, the changes are sharper and correlate with substantial changes in the training loss function [PITH_FULL_IMAGE:figures/full_fig_p012_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: (top left) The f-vector of another trial of the periodic dynamical system. (top right) The averaged f1 (instead of f0) however these values are related through the Euler Characteristic. (bottom left) is the f-vector of a chaotic dynamical system. (bottom right) The averaged f0 for the 25 trials of the chaotic system. 12 [PITH_FULL_IMAGE:figures/full_fig_p012_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Phase space of a periodic duffing oscillator. D.2 Physics-Informed Loss Function The training of the model relies on a composite loss function that combines both data-driven and physics-informed components. The total loss, Ltotal, is defined as the sum of the Mean Squared Error (MSE) of the data predictions (Ldata) and the MSE of the residual of the governing differential equation (Lphysics): Ltotal = Lda… view at source ↗
read the original abstract

This paper explores the topological signatures of ReLU neural network activation patterns. We consider feedforward neural networks with ReLU activation functions and analyze the polytope decomposition of the feature space induced by the network. Mainly, we investigate how the Fiedler partition of the dual graph and show that it appears to correlate with the decision boundary -- in the case of binary classification. Additionally, we compute the homology of the cellular decomposition -- in a regression task -- to draw similar patterns in behavior between the training loss and polyhedral cell-count, as the model is trained.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper analyzes the polyhedral decomposition of input space induced by ReLU activation patterns in feedforward networks. Its central claim is that the Fiedler partition of the dual graph of this decomposition appears to correlate with the binary decision boundary, supported by visual agreement on trained networks; a secondary claim is that homology of the cellular decomposition exhibits patterns with training loss and cell count during regression training.

Significance. If the observed alignment between Fiedler partitions and decision boundaries proves robust under quantitative controls, the work could provide a novel topological lens on neural network geometry and training dynamics, potentially informing interpretability and generalization studies by linking graph Laplacians to classification boundaries.

major comments (2)
  1. [Binary classification experiments] The correlation between the Fiedler partition and the decision boundary is asserted on the basis of qualitative visual inspection across a limited set of networks and datasets. No quantitative partition-similarity statistics (e.g., adjusted Rand index, normalized mutual information) are reported, nor are error bars or statistical significance tests provided.
  2. [Experimental setup and controls] The manuscript presents no control experiments that would rule out artifacts, such as comparisons against random labelings, untrained networks, or alternative dual-graph constructions (e.g., different edge-weighting schemes). Without these, it remains unclear whether the alignment is induced by the specific activation-pattern graph or is a generic property of the Laplacian.
minor comments (2)
  1. [Abstract] The abstract uses imprecise phrasing ('appears to correlate', 'draw similar patterns') that should be replaced by more concrete statements once quantitative metrics are added.
  2. [Preliminaries] Notation for the dual graph (vertices as polyhedral cells, edges weighted by shared hyperplanes) is introduced without an explicit definition or pseudocode; a small diagram or formal definition would improve clarity.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive and detailed feedback. The comments highlight important aspects of experimental validation that can strengthen the presentation of our results. We address each major comment below and describe the revisions we will incorporate.

read point-by-point responses

  1. Referee: [Binary classification experiments] The correlation between the Fiedler partition and the decision boundary is asserted on the basis of qualitative visual inspection across a limited set of networks and datasets. No quantitative partition-similarity statistics (e.g., adjusted Rand index, normalized mutual information) are reported, nor are error bars or statistical significance tests provided.

    Authors: We agree that quantitative metrics would provide a more rigorous assessment of the observed alignment. In the revised manuscript we will report adjusted Rand index and normalized mutual information between the Fiedler partitions and the ground-truth decision boundaries, computed over multiple independently trained networks and datasets. Error bars from repeated runs with different random seeds will be included, together with statistical significance tests against a null model of random partitions. revision: yes

  2. Referee: [Experimental setup and controls] The manuscript presents no control experiments that would rule out artifacts, such as comparisons against random labelings, untrained networks, or alternative dual-graph constructions (e.g., different edge-weighting schemes). Without these, it remains unclear whether the alignment is induced by the specific activation-pattern graph or is a generic property of the Laplacian.

    Authors: We acknowledge that control experiments are necessary to establish specificity. The revised version will include comparisons of Fiedler partitions obtained from trained networks against those arising from (i) networks trained on random labels, (ii) untrained networks with the same architecture, and (iii) dual graphs constructed with alternative edge-weighting schemes. These controls will demonstrate that the reported alignment is tied to the activation patterns produced by gradient-based training rather than being a generic property of the graph Laplacian. revision: yes

Circularity Check

0 steps flagged

No significant circularity in observational topological analysis of ReLU activations

full rationale

The paper is an empirical computational study that constructs the polyhedral decomposition from ReLU activation patterns, builds the dual graph, computes its Fiedler partition, and reports an observed correlation with the binary decision boundary (plus homology trends versus loss in regression). These quantities are calculated directly from the trained network's own activations and outputs; no first-principles derivation, fitted-parameter prediction, or self-citation chain is claimed or required. The central statements are observational (“appears to correlate”) rather than deductive, so no step reduces to its inputs by construction. The work is self-contained against external benchmarks and receives a low circularity score.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The paper builds on standard assumptions about ReLU networks creating linear regions without introducing new parameters or entities.

axioms (1)
  • domain assumption ReLU networks induce a polytope decomposition of the feature space
    This is a standard property of piecewise linear activations in neural networks.

pith-pipeline@v0.9.0 · 5636 in / 1003 out tokens · 33735 ms · 2026-05-18T07:11:34.230547+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Copositive Matrices with Ordered Off-Diagonal Entries

    math.OC 2026-05 unverdicted novelty 7.0

    Copositive matrices with nondecreasing off-diagonal entries admit a PSD plus nonnegative decomposition, which implies exactness of a natural relaxation for separable quadratic optimization over the simplex.

Reference graph

Works this paper leans on

27 extracted references · 27 canonical work pages · cited by 1 Pith paper

  1. [1]

    Phat – persistent homology algorithms toolbox.Journal of Symbolic Computation, 78:76–90, 2017

    Ulrich Bauer, Michael Kerber, Jan Reininghaus, and Hubert Wagner. Phat – persistent homology algorithms toolbox.Journal of Symbolic Computation, 78:76–90, 2017. Algorithms and Software for Computational Topology

    work page 2017

  2. [2]

    Polyhedral complex extraction from relu networks using edge subdivision, 2023

    Arturs Berzins. Polyhedral complex extraction from relu networks using edge subdivision, 2023

    work page 2023

  3. [3]

    Gol toolbox

    Turgay Caglar. Gol toolbox. Accessed: 31 July 2025

    work page 2025

  4. [4]

    London Mathematical Society Student Texts

    Dragoš Cvetkovi´c, Peter Rowlinson, and Slobodan Simi´c.Laplacians, page 184–227. London Mathematical Society Student Texts. Cambridge University Press, 2009

    work page 2009

  5. [5]

    Understanding physics-informed neural networks: Techniques, applications, trends, and challenges.AI, 5(3):1534–1557, August 2024

    Amer Farea, Olli Yli-Harja, and Frank Emmert-Streib. Understanding physics-informed neural networks: Techniques, applications, trends, and challenges.AI, 5(3):1534–1557, August 2024

    work page 2024

  6. [6]

    Algebraic connectivity of graphs.Czechoslovak Mathematical Journal, 23(2):298–305, 1973

    Miroslav Fiedler. Algebraic connectivity of graphs.Czechoslovak Mathematical Journal, 23(2):298–305, 1973

    work page 1973

  7. [7]

    A property of eigenvectors of nonnegative symmetric matrices and its application to graph theory.Czechoslovak Mathematical Journal, 25(4):619–633, 1975

    Miroslav Fiedler. A property of eigenvectors of nonnegative symmetric matrices and its application to graph theory.Czechoslovak Mathematical Journal, 25(4):619–633, 1975

    work page 1975

  8. [8]

    Nelson, Anjan Dwaraknath, and Primoz Skraba

    Rickard Brüel Gabrielsson, Bradley J. Nelson, Anjan Dwaraknath, and Primoz Skraba. A topol- ogy layer for machine learning. In Silvia Chiappa and Roberto Calandra, editors,Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 ofProceedings of Machine Learning Research, pages 1553–1563. PMLR, 26–28 Aug 2020

    work page 2020

  9. [9]

    Exploring physics-informed neural networks: From fundamentals to applications in complex systems

    Sai Ganga and Ziya Uddin. Exploring physics-informed neural networks: From fundamentals to applications in complex systems. 2024

    work page 2024

  10. [10]

    Complexity of linear regions in deep networks.International Conference on Machine Learning, pages 2596–2604, 2019

    Boris Hanin and David Rolnick. Complexity of linear regions in deep networks.International Conference on Machine Learning, pages 2596–2604, 2019. 9

    work page 2019

  11. [11]

    Cambridge Univ

    Allen Hatcher.Algebraic topology. Cambridge Univ. Press, Cambridge, 2001

    work page 2001

  12. [12]

    Deep networks always grok and here is why

    Ahmed Imtiaz Humayun, Randall Balestriero, and Richard Baraniuk. Deep networks always grok and here is why. InHigh-dimensional Learning Dynamics 2024: The Emergence of Structure and Reasoning, 2024

    work page 2024

  13. [13]

    Hamming similarity and graph laplacians for class partitioning and adversarial image detection

    Huma Jamil, Yajing Liu, Turgay Caglar, Christina Cole, Nathaniel Blanchard, Christopher Peterson, and Michael Kirby. Hamming similarity and graph laplacians for class partitioning and adversarial image detection. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 590–599, June 2023

    work page 2023

  14. [14]

    Anderson Jr

    William N. Anderson Jr. and Thomas D. Morley. Eigenvalues of the Laplacian of a graph. Linear and Multilinear Algebra, 18(2):141–145, 1985

    work page 1985

  15. [15]

    Kirchhoff

    Gustav R. Kirchhoff. Ueber die auflösung der gleichungen, auf welche man bei der untersuchung der linearen vertheilung galvanischer ströme geführt wird.Annalen der Physik, 148:497–508, 1847

    work page

  16. [16]

    Deep learning.nature, 521(7553):436–444, 2015

    Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning.nature, 521(7553):436–444, 2015

    work page 2015

  17. [17]

    Loss spike in training neural networks

    Xiaolong Li, Zhi-Qin John Xu, and Zhongwang Zhang. Loss spike in training neural networks. 2023

    work page 2023

  18. [18]

    A survey of convolutional neural networks: analysis, applications, and prospects.IEEE transactions on neural networks and learning systems, 33(12):6999–7019, 2021

    Zewen Li, Fan Liu, Wenjie Yang, Shouheng Peng, and Jun Zhou. A survey of convolutional neural networks: analysis, applications, and prospects.IEEE transactions on neural networks and learning systems, 33(12):6999–7019, 2021

    work page 2021

  19. [19]

    Integrating geometries of relu feedforward neural networks.Frontiers in Big Data, V olume 6, 2023

    Yajing Liu, Turgay Caglar, Christopher Peterson, and Michael Kirby. Integrating geometries of relu feedforward neural networks.Frontiers in Big Data, V olume 6, 2023

    work page 2023

  20. [20]

    Relu neural networks, polyhedral decompositions, and persistent homology

    Yajing Liu, Christina M Cole, Chris Peterson, and Michael Kirby. Relu neural networks, polyhedral decompositions, and persistent homology. InTopological, Algebraic and Geometric Learning Workshops 2023, pages 455–468. PMLR, 2023

    work page 2023

  21. [21]

    Tracemin-fiedler: A parallel algorithm for computing the fiedler vector

    Murat Manguoglu, Eric Cox, Faisal Saied, and Ahmed Sameh. Tracemin-fiedler: A parallel algorithm for computing the fiedler vector. In José M. Laginha M. Palma, Michel Daydé, Osni Marques, and João Correia Lopes, editors,High Performance Computing for Computational Science – VECPAR 2010, pages 449–455, Berlin, Heidelberg, 2011. Springer Berlin Heidelberg

    work page 2010

  22. [22]

    Algorithmic determination of the combinatorial structure of the linear regions of relu neural networks.SIAM Journal on Applied Algebra and Geometry, 9(2):374–404, 2025

    Marissa Masden. Algorithmic determination of the combinatorial structure of the linear regions of relu neural networks.SIAM Journal on Applied Algebra and Geometry, 9(2):374–404, 2025

    work page 2025

  23. [23]

    Topology of deep neural networks

    Gregory Naitzat, Andrey Zhitnikov, and Lek-Heng Lim. Topology of deep neural networks. Journal of Machine Learning Research, 21(184):1–40, 2020

    work page 2020

  24. [24]

    A roadmap for the computation of persistent homology.EPJ Data Science, 6(1):17, 2017

    Nina Otter, Mason A Porter, Ulrike Tillmann, Peter Grindrod, and Heather A Harrington. A roadmap for the computation of persistent homology.EPJ Data Science, 6(1):17, 2017

    work page 2017

  25. [25]

    Grokking: Generalization beyond overfitting on small algorithmic datasets, 2022

    Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets, 2022

    work page 2022

  26. [26]

    On the expressive power of deep neural networks.International conference on machine learning, pages 2847–2854, 2017

    Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl-Dickstein. On the expressive power of deep neural networks.International conference on machine learning, pages 2847–2854, 2017

    work page 2017

  27. [27]

    Weighted laplacian method and its theoretical applications.IOP Conference Series: Materials Science and Engineering, 768(7):072032, March 2020

    Shijie Xu, Jiayan Fang, and Xiangyang Li. Weighted laplacian method and its theoretical applications.IOP Conference Series: Materials Science and Engineering, 768(7):072032, March 2020. 10 A Feed Forward Networks Consider an(L+ 1)−layer feedforward ReLU neural network (FFRNN): Rm (W1,b1) − − − − − → ReLU Rh1 (W2,b2) − − − − − → ReLU Rh2 →. . .→R hL−1 (WL,...

    work page 2020