{"id":"8d3eabe2-9ecf-4621-a049-088409dcf4f5","arxiv_id":"2502.09954","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A new Hamming-space ratio measures how far a straight input path is from staying convex in a ReLU network's activation space.","lead":"This paper defines a number, called the space folding measure, that tracks how much a ReLU neural network bends a straight line in its input when seen through the network's internal activation patterns. The authors prove a convexity equivalence for the first layer and test the measure on a synthetic fractal dataset and MNIST.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Lemma 1's proof is invalid: it exhibits one shortest path in A while non-convexity requires a shortest path leaving A; deep-layer chi interpretation relies on an assumption the paper itself shows fails.","rationale":"After reading the paper in good faith, the measure chi is clearly defined, the algorithm is concrete, and Lemma 2's bounds are elementary and correct. The load-bearing weakness is Lemma 1, which is used to justify the whole convexity/folding framework and to claim that folding only appears in deep networks. The proof's contradiction step only establishes existence of a shortest path inside A, not the universal property required by the definition. The paper itself flags the additional limitation that the straight-line-to-shortest-path assumption fails for deep layers, yet the experiments and interpretation of chi rely on that assumption. The MNIST correlation on a single post hoc chosen seed is a secondary concern, but the theoretical gap alone warrants rejection or major revision. I therefore agree with the reader's weakest-assumption identification and see no reason to change the verdict.","tokens_in":15610,"tokens_out":11454,"duration_ms":104828,"concrete_test":"Write a small script that enumerates all subsets of cells of a fixed affine hyperplane arrangement in R^2 (e.g., 3 lines) and R^3 (e.g., 4 planes), checks which subsets form a convex union, and tests the corresponding set of activation patterns against Definition 1 with observable points defined by whether some input maps to that pattern. Include line segments that pass through vertices of the arrangement to test simultaneous bit flips. If any convex union yields a pattern set that is not Hamming-convex under the observable definition, Lemma 1 is false; if none is found, the lemma may be salvageable but the proof still needs repair.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central theoretical claim (Lemma 1) is not established. In the direction 'R convex => A convex', the proof assumes A is not Hamming-convex, so some shortest path gamma between two patterns leaves A, but then constructs the activation path along the straight line [P,Q] and notes it is a shortest path inside A. This only shows the existence of one shortest path within A; it does not contradict the existence of another shortest path that exits A. Hamming convexity (Definition 1) requires all observable points on every shortest path to lie in A, so the argument fails at its key step. Additionally, the straight-line path is a shortest path only if each hyperplane is crossed at most once and crossings occur one bit at a time. The paper explicitly acknowledges this holds only for first-hidden-layer hyperplanes that intersect the entire input space (Section 4, limitation note after Lemma 1) and that deeper layers can have adjacent regions with Hamming distance >1 (Example 2). Nevertheless, Section 5 invokes Lemma 1 to interpret chi as a deviation-from-convexity measure, and the deep-network experiments (Section 6) use chi as an indicator of folding. Thus the theoretical foundation for the paper's headline interpretation is currently unsupported, even if the measure itself is well-defined and the bounds in Lemma 2 are correct.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a quantitative measure of 'space folding' in ReLU networks by studying how straight line segments in the Euclidean input space map to paths in the Hamming space of activation patterns. The main theoretical claim (Lemma 1) is an equivalence between convexity of a union of activation regions and convexity of the corresponding set of activation patterns in Hamming space, for tessellations formed by hyperplanes that intersect the entire input space. The paper then introduces a space folding measure chi(Γ) = 1 - max_i d_H(π_1,π_i) / Σ_i d_H(π_i,π_{i+1}), proves bounds on it (Lemma 2), gives an algorithm for its computation, and reports experiments on CantorNet and MNIST suggesting that the measure increases with network depth.","tokens_in":15879,"tokens_out":23146,"duration_ms":226848,"significance":"If Lemma 1 were established, the convexity equivalence would be a clean structural statement about single-hidden-layer ReLU networks, and chi would be a novel, parameter-free probe of how deep layers deviate from that behavior. The measure itself is well-defined, Lemma 2's bounds are correct, the algorithm with its complexity analysis is useful, and the CantorNet example provides a helpful sanity check. However, the proof of Lemma 1 is not valid as written, and the deep-layer experiments rely on an assumption that the paper itself shows fails. The measure remains a promising descriptive statistic, but the paper's main theoretical and interpretational claims require substantial additional work.","major_comments":[{"comment":"The contradiction argument is logically invalid. Non-convexity of A in the Hamming cube means that there exists at least one shortest path between two patterns in A that leaves A; it does not mean that every shortest path leaves A. The proof exhibits the path induced by the straight line segment [P,Q] and notes that it is a shortest path contained in A, which does not contradict the existence of a different shortest path that exits A. Definition 1 explicitly requires all shortest paths to remain in A. Additionally, the proof does not account for the 'observable points' qualification in Definition 1: a shortest path that leaves A only through non-observable binary vectors is not a violation of convexity, but the proof treats any exit from A as a contradiction. The lemma may be true, but this proof does not establish it; a correct proof would need to show that every observable pattern on every shortest path between two patterns in A occurs in some region contained in the convex union R.","section":"Section 4, Lemma 1 (proof, first direction)"},{"comment":"The interpretation of chi as a deviation from convexity is invoked for deep networks despite the paper's own limitation note after Lemma 1. That note states that the single-bit-flip property holds only for first-hidden-layer hyperplanes that intersect the entire input space, and Example 2 shows that for deeper layers adjacent regions can have Hamming distance greater than one. In that setting, the denominator r2(Γ) in Eq. (3) accumulates multiple bit flips per step even for a monotone path, so chi(Γ) can be positive without any backtracking or non-convexity. Consequently, the CantorNet and MNIST experiments in Section 6, which use all hidden layers and interpret chi as space folding, rest on an assumption the paper itself shows is violated. Please either compute chi per layer with the theoretical guarantee applied only to the first layer, or provide a separate justification for the deep-layer interpretation.","section":"Section 5 (Eq. 4 discussion) and Section 6"},{"comment":"The reported Pearson correlation of 0.987 between network depth and aggregated median space folding is based on a single random seed (seed 4) and at most six architectures, and the text notes that only seeds 3 and 4 trained to satisfactory accuracy. This is too thin to support the general hypothesis in Section 7 that the maximal value of chi is associated with generalization capacity and increases with depth. Please report confidence intervals across seeds and architectures, and avoid selective reporting based on the best-performing seed.","section":"Section 6.1 and Section 7"}],"minor_comments":[{"comment":"The telescoping sum in Eq. (2) equals d_H(π_{i+1},π_1), so the maximum over i in {1,...,n-1} is max_{i in {2,...,n}} d_H(π_i,π_1), not max_{i in {1,...,n-1}} d_H(π_i,π_1); Eq. (4) uses the range {1,...,n}, so the two displays are inconsistent.","section":"Section 5, Eq. (2)"},{"comment":"Definition 1 relies on the informal notion of 'observable' points; please define the set of observable binary vectors for a tessellation and state Definition 1 in terms of the induced subgraph of the Hamming cube.","section":"Section 3, Definition 1"},{"comment":"The example says 'we conclude that they are non-observable on the considered domain' after listing intermediate patterns; for clarity, specify the domain [0,1]^2 explicitly in the example statement, since the argument depends on the bounded domain.","section":"Section 4, Example 2"},{"comment":"The phrase '100 pairs, ~1M paths Γ for each pair' is hard to parse; clarify whether this is the number of test-image pairs per digit-pair and how the n intermediate points are chosen.","section":"Section 6.1"}],"recommendation":"major_revision","confidential_remarks":"The reader's report and the stress-test note correctly identify the gap in Lemma 1's proof and the problem with interpreting chi on deep layers. I do not see a circularity issue: chi is defined directly from Hamming distances with no fitted constants, and the proof attempts a first-principles argument. The flaws are fixable in principle (a subcube argument would establish the 'convex union => subcube' direction for full-dimensional convex unions), but the revision would need to be substantial: rewriting Lemma 1, formalizing 'observable', and either restricting or re-grounding the deep-layer interpretation. Hence I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper introduces a simple, clean measure of space folding for ReLU networks: sample a straight line in input space, map it to activation patterns, and compute chi = 1 - (max distance from start)/(total path length). This is genuinely new as a quantitative handle on a phenomenon that has only been discussed qualitatively. The definition is straightforward, Lemma 2's bound (0 <= chi <= 1) is correct, and the CantorNet demo is a nice illustration. The related-work survey looks fair, including the self-citations for the range-measure inspiration and CantorNet; those are not load-bearing for the main result.\n\nThe central theoretical claim is Lemma 1, an equivalence between Euclidean convexity of a union of activation regions and Hamming convexity of the corresponding activation patterns. The proof is not valid as written. The contradiction step assumes A is not Hamming-convex, so some shortest path leaves A, but then notes that the straight line [P,Q] gives a shortest path inside A. That only shows one shortest path stays in A; non-convexity requires that some shortest path leaves A, and the existence of one shortest path inside A does not contradict that. The lemma may well be true — the \"observable points\" caveat in Definition 1 hints the authors saw the issue of non-realized sign patterns — but the proof needs real repair. The paper itself acknowledges the limitation to first-layer hyperplanes, and deep layers can have adjacent regions at Hamming distance >1. Yet the deep-network experiments and the interpretation of chi as \"deviation from convexity\" lean on this lemma. So the theoretical foundation for the most interesting application is currently unsupported.\n\nThe empirical section is thin: six MNIST architectures with fixed total hidden neurons, a post-hoc chosen seed (4 among the few that trained well), no confidence intervals on the reported Pearson correlation of 0.987, and a claim based on medians over digit pairs. It is suggestive but not convincing.\n\nFor a revision, I would ask for a corrected proof of Lemma 1, an honest statement of what chi means for deep layers without that lemma, all seeds reported, and some measure of uncertainty on the depth correlation. The measure itself is well defined and could become a useful descriptive tool. It does not resolve a big open problem and will not change practice on its own, but as a first quantitative fold measure it deserves referee time rather than a desk reject. I just would not accept it in its current form.","headline":"A clean new measure of ReLU space folding with a genuinely flawed proof of its main lemma and thin empirics; worth a real referee, not desk rejection.","tokens_in":16404,"tokens_out":5988,"would_cite":false,"duration_ms":63654,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","52A20"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proves that convexity of a union of activation regions is equivalent to convexity of its activation patterns in the Hamming cube only for first-hidden-layer hyperplanes, and introduces a range-ratio measure $\\chi$ that…","keywords":["space folding","ReLU networks","Hamming activation space","convexity","range measures","linear regions","activation patterns","CantorNet"],"falsifier":"Train a single-hidden-layer ReLU network to high accuracy on any classification task and measure $\\chi$ on straight paths between test points; Lemma 1 predicts $\\chi = 0$ for all such paths, so any nonzero value would refute the convexity equivalence.","tokens_in":15410,"feed_emoji":"📐","tokens_out":7358,"duration_ms":63403,"temperature":0.7,"pith_summary":"The paper's goal is to turn the informal idea that deep ReLU networks fold the input space into a measurable quantity. Its central theoretical result, Lemma 1, states that for hyperplanes of the first hidden layer, a union of activation regions is convex in Euclidean space exactly when the corresponding set of activation patterns is convex in the Hamming cube. The authors then define a space folding measure $\\chi(\\Gamma)$ as one minus the ratio of the maximum Hamming distance reached along a straight input path to the total Hamming distance traveled; $\\chi$ lies in $[0,1]$ and equals zero for a perfectly flat (unfolded) path. Experiments on the synthetic CantorNet benchmark and on MNIST show that $\\chi$ increases with network depth in well-trained networks, and that wider networks show a larger fraction of paths exhibiting folding. The paper argues that $\\chi$ can serve as a network feature describing how twisted the learned input representation is.","feed_headline":"New measure quantifies space folding in deep ReLU networks","feed_subtitle":"chi compares the farthest Hamming distance reached along a straight path to its total length; deeper trained nets fold more.","key_machinery":"The load-bearing object is the space folding measure $\\chi(\\Gamma)$, a ratio of two range metrics on the Hamming cube: $r_1(\\Gamma) = \\max_i d_H(\\pi_i, \\pi_1)$ is the maximum distance from the starting activation pattern, and $r_2(\\Gamma) = \\sum_i d_H(\\pi_i, \\pi_{i+1})$ is the total Hamming length of the path. The ratio $\\phi = r_1/r_2$ equals $1$ for a convex (flat) path, so $\\chi = 1 - \\phi$ measures deviation from convexity. Lemma 1 supplies the convexity equivalence that makes $\\chi$ meaningful: it guarantees that for first-layer hyperplanes, straight input segments map to shortest Hamming paths, so any $\\chi > 0$ signals genuine folding; the paper notes this guarantee lapses for deeper layers, where the measure still quantifies path inefficiency but not necessarily convexity loss.","core_discovery":"The central claim is that ReLU networks fold the input space in a way that can be read off from activation patterns along straight lines. Lemma 1 establishes an equivalence: for a first hidden layer whose hyperplanes intersect the whole input space, a union of linear regions is convex in $\\mathbb{R}^n$ if and only if the corresponding set of activation patterns is convex in the Hamming space $H^m$. Because this equivalence uses the fact that crossing a hyperplane flips exactly one activation bit, it fails for deeper layers, where neighboring regions can have Hamming distance larger than one. To quantify the resulting folding, the paper defines $\\chi(\\Gamma) = 1 - \\max_i d_H(\\pi_i, \\pi_1) \\big/ \\sum_i d_H(\\pi_i, \\pi_{i+1})$ for a straight path $\\Gamma$, interprets the numerator as a range measure and the denominator as total path length, and proves $0 \\le \\chi \\le 1$. Empirically, the median of non-zero $\\chi$ maxima across MNIST digit pairs correlates with depth (Pearson $r = 0.987$ for the best-performing seed), and the fraction of folded paths rises sharply in wider networks.","pith_inferences":["If Lemma 1's assumption fails for deep layers, then $\\chi$ on full-network activations may count non-adjacent Hamming jumps as folding even when the underlying geometry is locally flat; a natural test is to compute $\\chi$ using only first-layer activations and compare with the full-network value on the same paths.","The reported correlation between $\\chi$ and generalization is suggestive but not causal; training the same architectures with shuffled labels would test whether high folding is a learned property or an architecture artifact.","Because $\\chi$ is a per-path quantity, aggregating it as a median of maxima loses information about where along the path folding occurs; a finer analysis could localize folds to specific layer transitions and reveal which layers contribute most."],"forward_implications":["Single-hidden-layer ReLU networks cannot exhibit space folding along straight input paths; any nonzero $\\chi$ indicates depth is at play.","$\\chi$ provides a bounded, algorithmically computable network feature with complexity $O(n(N+C))$ per sample pair that can be compared across architectures.","In well-trained MNIST models, aggregated $\\chi$ grows with depth, while poorly trained models show lower folding, suggesting a link between folding and learned structure.","Wider networks do not necessarily fold more per path, but a much larger share of digit-pair paths exhibit folding, e.g., the ratio of folded paths grows from about $0.35$ for a $2\\times 30$ network to about $0.97$ for a $2\\times 300$ network.","The measure extends in principle to residual, normalization, and ReLU-based attention layers, since it operates on binarized activations."],"supporting_citations":[{"why":"Defines activation regions and bounds their number, giving the paper its notion of linear regions formed by ReLU layers.","marker":"(Montúfar et al., 2014)"},{"why":"Shows that first-hidden-layer hyperplanes intersect the entire input space and that deeper layers' regions can have Hamming distance greater than one, the basis for Lemma 1's limitation.","marker":"(Raghu et al., 2017)"},{"why":"Provides the adapted definition of convexity in the Hamming cube used in Definition 1.","marker":"(Moser et al., 2022)"},{"why":"Source of range measures and discrepancy norms that inspire the $\\chi$ construction.","marker":"(Weyl, 1916)"},{"why":"Establishes the geometric characterization of the range metrics $r_A$ and $r_D$ used to build the folding measure.","marker":"(Moser, 2012)"},{"why":"Supplies the CantorNet benchmark used for the synthetic folding experiments.","marker":"(Lewandowski et al., 2024)"},{"why":"Supplies the MNIST dataset used in the empirical depth and width experiments.","marker":"(LeCun et al., 1998)"}],"fun_headline_variants":["A new metric measures how ReLU nets fold space along lines","Quantifying folding in ReLU networks: a range-based measure","Measuring input space folding in ReLU networks with χ","χ reveals deeper ReLU networks fold input space more","Counting folds: a measure for ReLU network spatial distortion"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof that straight paths correspond to shortest paths in the Hamming cube assumes each hyperplane crossing flips exactly one activation bit, which holds only for first-hidden-layer hyperplanes that cut across the entire input space; for deeper layers the paper itself shows neighboring regions can differ by more than one bit.","fun_headline_variants_meta":{"raw":{"variants":["A new metric measures how ReLU nets fold space along lines","Quantifying folding in ReLU networks: a range-based measure","Measuring input space folding in ReLU networks with χ","χ reveals deeper ReLU networks fold input space more","Counting folds: a measure for ReLU network spatial distortion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000295,"raw_usage":{"total_tokens":1726,"prompt_tokens":970,"completion_tokens":756,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":586,"completion_tokens_details":{"reasoning_tokens":675}},"tokens_in":586,"tokens_out":756,"duration_ms":8030,"temperature":1.0,"reasoning_tokens":675,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T19:59:01.121519+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a single-hidden-layer ReLU network to high accuracy on any classification task and measure $\\chi$ on straight paths between test points; Lemma 1 predicts $\\chi = 0$ for all such paths, so any nonzero value would refute the convexity equivalence.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows that first-hidden-layer hyperplanes intersect the entire input space and that deeper layers' regions can have Hamming distance greater than one, the basis for Lemma 1's limitation."},{"cited_title":"A., Lewandowski, M., Kargaran, S., Zellinger, W., Biggio, B., and Koutschan, C","cited_arxiv_id":null,"evidence_quote":"Provides the adapted definition of convexity in the Hamming cube used in Definition 1."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of range measures and discrepancy norms that inspire the $\\chi$ construction."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the geometric characterization of the range metrics $r_A$ and $r_D$ used to build the folding measure."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the CantorNet benchmark used for the synthetic folding experiments."}],"review_version":1}