{"id":"b8ffe1ae-3f4c-43c8-89de-86c8ca355c03","arxiv_id":"2507.14747","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Magnitude-based pruning (Top-K) increases a measure of topological orderedness in fully connected, weight-tied recurrent layers, without a clear performance drop on two toy tasks.","lead":"This paper asks whether a fully connected neural layer, where every neuron can send signals to every other neuron, can learn to send information mostly in one direction. On two tiny tasks (XOR and a sine function), pruning away small connections increased a measure of one-way information flow without a clear loss in training performance, suggesting directionality can emerge rather than being hard-wired.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (3)'s orderedness metric is not validated against the paper's own W_ij convention; a synthetic acyclicity probe is needed before Table 1 can support 'induced directionality'.","rationale":"The reader's weakest-assumption analysis already identified the validity of the orderedness metric, and specifically the ambiguous direction convention in Eq. (3), as the load-bearing premise. My stress-test agrees: this is the single point on which the central claim rests, and it is under-specified in the manuscript. The paper's own W_ij definition and Algorithm 1 make the triangle convention non-obvious; the min-over-permutations may rescue it, but only if the implementation and the text are aligned. The proposed synthetic probe would settle the issue directly. I do not see a need to move the verdict: the reader's CONDITIONAL judgement is the appropriate one, because the concern is substantive but testable and the paper otherwise provides reproducible code and a clear small-scale empirical setup. The effect sizes are modest and the authors themselves note high variance in Section 5, but those limitations are secondary to the metric-validity question. No ad hominem is intended; the critique is purely about the argument's load-bearing assumption.","tokens_in":7601,"tokens_out":12555,"duration_ms":152442,"concrete_test":"Run the released code's permutation-minimization on a directed 3-cycle (with the paper's convention, W_21=W_32=W_13=1) and on a 3-neuron feedforward chain (W_21=1, W_32=1), using the smallest complete-perceptron configuration the code supports. The cycle must give O<1 and the chain must give O=1 under some allowed permutation for Eq. (3) to measure topological ordering. If the cycle yields O=1, or the chain yields O=0, then the reported Delta O increases cannot be interpreted as induced directionality without redefining the metric.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that magnitude-based Top-K pruning, despite being agnostic to orderedness, increases O(W) as defined in Eq. (3), and that this movement represents induction of feedforward directionality. This interpretation requires Eq. (3) to actually measure topological ordering of information flow under the convention defined in Section 3.1 and Algorithm 1. That requirement is not checked, and the text is ambiguous at best. Section 3.1 defines W_ij as the strength of the connection from neuron j to neuron i, and Algorithm 1 computes s <- sigma(h W^T + b), so neuron i receives sum_j h_j W_ij. Under this convention, a standard topological order (source j before target i) places feedforward edges in the lower triangle, whereas Section 3.2 states that perfectly feedforward simulation corresponds to blocks above the main diagonal. The min-over-permutations in Eq. (3) can reconcile these by using the reverse topological order, but that reconciliation is unstated; it is exactly the assumption that must hold for Table 1 to mean what the abstract claims. If the code's permutation set is restricted or implemented heuristically, the reported Delta O values in Table 1 and Figures 2-3 may not reflect topological ordering at all. This is a concrete, testable validity gap, not a dispute about consensus.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether directionality—defined as topologically ordered information flow—can emerge, rather than be hard-wired, in an all-to-all connected perceptron layer that is mathematically equivalent to a weight-tied recurrent network. It introduces the ``complete perceptron layer'' (Sec. 3.1), an orderedness metric O(W) defined as 1 minus the minimal lower-triangular weight fraction under neuron permutations (Eq. 3), and evaluates several initialisation and pruning schemes on XOR and Sine tasks. The central empirical claim is that magnitude-based Top-K pruning, which is agnostic to the orderedness metric, increases orderedness relative to untrained controls, and that this happens ``without compromising performance.'' The paper also reports exploratory results on the number of hidden units, evolution iterations, and sparsity level.","tokens_in":7824,"tokens_out":8560,"duration_ms":110313,"significance":"If the central claim is established, the paper would contribute a concrete and potentially useful phenomenon: gradient descent combined with simple magnitude pruning can organize a recurrent weight matrix into a nearly topologically ordered structure, suggesting that directionality is an emerging rather than a hard-wired inductive bias. Strengths of the paper include public code, multi-seed experiments, untrained controls, and the explicit acknowledgment that the Tril-Damp methods inject an expected inductive bias. However, the evidence as presented is incomplete: the orderedness metric is not validated against known feedforward/feedback structures, the direction convention is internally inconsistent, performance preservation is not quantified, and the central effect sizes are small relative to the reported variance. The significance of the result therefore depends on additional validation rather than on the current tables alone.","major_comments":[{"comment":"The direction convention is inconsistent. Section 3.1 defines W_ij as the strength of the connection from neuron j to neuron i, and Algorithm 1 computes s <- sigma(h W^T + b), so neuron i receives sum_j h_j W_ij. Under this convention a feedforward edge from an earlier-indexed neuron to a later-indexed neuron lies in the lower triangle (row i > column j), not ``above the main diagonal'' as stated in Section 3.2. Because Eq. (3) minimizes the lower-triangular fraction, the text's claim that a feedforward network corresponds to an upper-triangular W is wrong under the stated convention, or the convention in Eq. (1) is misstated. A reverse ordering of neuron indices can reconcile the two descriptions, but that reconciliation is never stated. The central conclusion that Top-K pruning induces directionality depends on O(W) actually measuring topological information flow under the implemented convention. The authors must fix this convention, specify exactly which rows/columns are permuted (output units, hidden units, inputs) in the min over pi, and validate Eq. (3) on synthetic matrices with known feedforward and feedback structure.","section":"Secs. 3.1, 3.2 and Eq. (3)"},{"comment":"The claim that pruning increases orderedness ``without compromising performance'' is not supported by any quantitative performance result. Table 1 reports only changes in orderedness; Appendix D shows training-loss curves only for Dynamic Top-K (k=0.5) and does not report final losses or accuracies for the other methods or for the unpruned baseline. The paper should include a table of final training (and, where applicable, test) loss or accuracy for every configuration in Table 1, with standard deviations, and should demonstrate that the orderedness increases are not achieved at the cost of task performance.","section":"Abstract, Sec. 5, Table 1"},{"comment":"The statistical evidence for the key Top-K claim is weak as presented. On XOR, trained Top-K gives Delta O = 0.116 +/- 0.060 versus 0.039 +/- 0.032 for untrained Top-K; on Sine the values are 0.086 +/- 0.037 versus 0.039 +/- 0.032. The standard deviations overlap substantially and no significance test or confidence interval is reported. The authors should report paired or independent t-tests/bootstrap intervals across their 10 seeds and effect sizes. In addition, Random pruning produces a larger Delta O (0.243 +/- 0.082 on XOR) than Top-K, so the specific claim that Top-K pruning induces orderedness should be supported by a matched-sparsity comparison showing that Top-K's gain over its untrained control is not simply a sparsity artifact.","section":"Table 1(b), Sec. 5"},{"comment":"The orderedness metric is never validated as a measure of information-flow directionality. O(W) is purely algebraic and does not account for activation dynamics, input clamping, or the fact that a small lower-triangular weight can dominate the recurrent trajectory. The paper should include synthetic probes: (i) matrices with known feedforward block structure, after random neuron relabeling, should yield O close to 1; (ii) random Gaussian matrices should yield a baseline O; (iii) matrices with known feedback cycles should yield lower O; and (iv) ablating the lower-triangular entries of a trained network and measuring the resulting change in network output would directly test whether O tracks functional directionality. Without such validation, the interpretation of Table 1 as evidence of ``induced directionality'' remains an unverified semantic claim.","section":"Sec. 3.3, Eq. (3)"},{"comment":"Several implementation details that affect reproducibility are missing or contradictory. The paper does not specify how often the pruning/damping operations in Appendix B are applied during training, whether the pruning mask is updated every step, or how the min over permutations in Eq. (3) is computed (exact enumeration, heuristic, number of samples). In addition, Table 1 states that the default initialisation is random normal, while Appendix C states that ``random uniform distribution was applied on the weights matrix and values vector.'' These inconsistencies must be resolved before the reported Delta O values can be reproduced.","section":"Appendix B, Appendix C, Sec. 3.4"}],"minor_comments":[{"comment":"The legend labels for the Top-K methods use ``Top-K (1-k)'' and ``Dyn. Top-K (1-k)'', while the text and Table 1 use k as the retained fraction; please clarify whether the horizontal axis is sparsity or retention for each method.","section":"Fig. 3"},{"comment":"The phrase ``W would consist of rectangular blocks above the main diagonal'' should be reconciled with the W_ij convention defined in Section 3.1; see Major Comment 1.","section":"Sec. 3.2"},{"comment":"The sentence explaining the ``spike'' at two iterations is speculative and not tested; consider adding a control experiment that removes hidden-to-hidden connections and shows that performance does not degrade.","section":"Sec. 5"},{"comment":"Eq. (3) should handle the degenerate case S(pi(W_abs)) = 0, and the statement that O is ``1 in the limit'' should be clarified because the min over permutations may not attain exactly 1 even for a feedforward matrix if diagonal entries are allowed and the permutation set is restricted.","section":"Sec. 3.3"},{"comment":"The paper says ``random uniform distribution was applied on the weights matrix and values vector'' while Table 1 says random normal; harmonize these statements.","section":"App. C"}],"recommendation":"major_revision","confidential_remarks":"The paper appears to originate from a workshop submission (MOSS at ICML 2025) and may need repositioning for a full journal venue. The most serious issues are the unresolved direction-convention inconsistency in the orderedness metric and the absence of any performance table; both are fixable within the manuscript's scope, so I do not recommend rejection at this stage. The code availability and multi-seed reporting are positive features that should be preserved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a small, honest empirical paper with one genuinely new observation—magnitude-based Top-K pruning, which never looks at the orderedness metric, nudges a weight-tied RNN toward a more feedforward-ordered weight matrix on two toy tasks. The effect is real but modest, and the paper's interpretation runs ahead of its evidence. It deserves a serious referee, but likely with major revisions.\n\nThe formal core is clean: the complete perceptron layer is a weight-tied RNN with clamped inputs, and the orderedness metric O(W) is the weighted minimum feedback arc set fraction, though the paper doesn't cite that prior art. The best methodological choice is including untrained-pruning controls; the authors also openly say the sparsity results are too noisy for conclusions, which I respect.\n\nThe soft spots are three, in increasing order of importance. First, the metric is never validated. The paper defines W_ij as the strength from neuron j to neuron i, which puts feedforward edges from lower to higher indices in the lower triangle, yet Section 3.2 claims feedforward corresponds to upper triangular. The min over permutations can absorb this reversal, but the paper should state that and verify O on synthetic acyclic matrices. Second, the 'without compromising performance' claim is unsupported: no final loss or accuracy numbers, only training curves, and the delta-O differences (e.g., 0.116 vs 0.039 on XOR) are small with heavily overlapping standard deviations. Third, the paper doesn't say how the permutation minimum is computed at larger hidden counts. All are addressable.\n\nThis is a workshop-scale contribution, not a breakthrough. I'd send it to review, and I'd want the authors to add metric validation, a performance table, and a clearer account of the orientation convention.","headline":"A small, genuinely new observation about pruning inducing orderedness in weight-tied RNNs, undercut by weak quantitative support and an unvalidated metric.","tokens_in":8364,"tokens_out":3732,"would_cite":false,"duration_ms":43908,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pruning by weight magnitude alone pushes a fully connected recurrent layer toward one-way information flow, with no explicit directional bias in the pruning rule, and training performance is preserved on the paper's tasks.","keywords":["recurrent neural networks","pruning","directionality","topological ordering","orderedness","weight-tied networks","sparsity","inductive bias"],"falsifier":"Take the trained complete perceptron layers from the XOR/Sine runs and compare the effect on the output of zeroing an upper-triangular edge versus a lower-triangular edge of comparable magnitude. If removing lower-triangular (feed-back) edges changes the output as much as removing upper-triangular (feed-forward) edges, then the increase in $O(W)$ under Top-K pruning does not reflect one-way information flow.","tokens_in":7353,"feed_emoji":"🧠","tokens_out":9409,"duration_ms":106183,"temperature":0.7,"pith_summary":"The paper asks whether the one-way, feedforward organization of modern neural networks is a prerequisite for learning or a property that training can discover. It builds a fully connected 'complete perceptron' layer, mathematically equivalent to a weight-tied recurrent network, and measures orderedness as the proportion of connection strength that can be aligned into a single direction after relabeling the hidden units. On XOR and Sine tasks, pruning weights by largest absolute value, a rule that knows nothing about direction, reliably increases orderedness after training while keeping training losses close to those of standard MLPs. The paper concludes that directionality is an advantageous inductive bias that can be induced by sparsification plus gradient descent rather than hard-wired into the architecture.","feed_headline":"Pruning alone steers recurrent nets toward one-way flow","feed_subtitle":"Top-K pruning, blind to direction, pushes fully connected layers into a nearly triangular layout.","key_machinery":"The complete perceptron layer is a single all-to-all weight matrix $W$ reused over $T$ iterations — a weight-tied recurrent network with clamped inputs — so temporal depth substitutes for spatial depth. The orderedness metric $O(W) = 1 - \\min_{\\pi\\in\\Pi} L(\\pi(|W|))/S(\\pi(|W|))$ drives the analysis: it finds the relabeling of hidden units that minimizes the lower-triangular absolute weight fraction, with $O=1$ for perfect topological ordering. The pruning mechanism is Top-K masking by absolute magnitude, applied statically or with a dynamic schedule; because the mask never looks at a weight's position relative to the diagonal, the observed increase in $O$ after training is the paper's evidence that gradient descent itself prefers acyclic routes once competing low-magnitude edges are removed.","core_discovery":"On the paper's own terms, the discovery is that directionality in recurrent computation can be induced, not imposed: starting from dense all-to-all weights, Top-K pruning (static, and especially dynamic) selects edges by magnitude alone, and after gradient descent the surviving high-magnitude connections concentrate in one triangle of the weight matrix. The orderedness metric $O(W)=1-\\min_{\\pi\\in\\Pi}L(\\pi(|W|))/S(\\pi(|W|))$ quantifies this as the largest possible fraction of forward-flowing absolute weight under permutation of hidden units. Reported increases are $+0.116$ (XOR) and $+0.086$ (Sine) for Top-K, with dynamic Top-K reaching $+0.149$ on XOR; these exceed the untrained control, indicating that the ordering develops during learning. The paper also finds that random pruning raises $O$ even untrained, which it attributes to the metric's definition, and that two-iteration layers are structurally directional because hidden-to-hidden feedback cannot reach the output.","pith_inferences":["An implication the paper leaves implicit: if Top-K pruning reliably induces triangular structure, the same recipe could be used to compress an already-trained weight-tied RNN into an approximately feedforward network, reducing inference-time recurrence depth — but the paper's experiments train from scratch on toy tasks rather than converting pretrained models.","The authors' two-iteration argument suggests a diagnostic that they do not run: comparing orderedness across iteration counts separates structurally forced directionality (where recurrence depth is too small for feedback to matter) from directionality that learning chooses when feedback is actually usable.","A testable extension would apply the same pruning schedule to a real sequence task and check whether increases in $O(W)$ track test performance; the synthetic XOR/Sine setup leaves the relation between orderedness and generalization open."],"forward_implications":["If the claim holds, a dense weight-tied recurrent layer can be trained to a state where relabeling its hidden units exposes a nearly triangular weight matrix: most connection strength feeds forward, with no layer-wise architecture imposed in advance.","The observed $\\Delta O$ for dynamic Top-K ($+0.149$ on XOR, $+0.086$ on Sine) exceeds the untrained control, so the orderedness is a product of learning under pruning, not an artifact of sparsification alone.","Because Top-K pruning selects by absolute magnitude and never looks at the diagonal position, the triangular organization must arise from gradient descent choosing acyclic routes, which suggests sparsification as a route to more interpretable recurrent computation.","The two-iteration spike in orderedness follows directly from the authors' structural argument: with only two updates, hidden-to-hidden feedback cannot influence the output, so the learned network is directionally constrained regardless of pruning."],"supporting_citations":[{"why":"Establishes the lottery ticket hypothesis, motivating pruning as a route to trainable subnetworks and as the starting point for asking what pruning does to network structure.","marker":"Frankle & Carbin (2018)"},{"why":"Proves that pruning can uncover high-performing subnetworks, providing the theoretical backdrop for treating magnitude pruning as a structural force.","marker":"Malach et al. (2020)"},{"why":"Shows that connection-cost pressure leads networks to evolve hierarchy, the direct inspiration for testing sparsity-induced orderedness.","marker":"Mengistu et al. (2016)"},{"why":"Demonstrates that connection-cost regularization increases modularity and interpretability, extending the same intuition from modularity to directionality.","marker":"Liu et al. (2023)"},{"why":"Supplies the precedent of all-to-all connected, undirected computation in Boltzmann machines.","marker":"Ackley et al. (1985)"},{"why":"Provides biological evidence that recurrent, non-feedforward dynamics support computation, motivating the question of whether directionality is needed at all.","marker":"Mante et al. (2013)"}],"fun_headline_variants":["Pruning induces directional flow in recurrent nets","Sparsity alone can order recurrent computation","Directionality emerges from pruning, not architecture","Top-K pruning discovers triangular weight flow","Pruned recurrent nets learn one-way information flow"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole argument rests on the assumption that the orderedness metric — the smallest lower-triangular weight fraction reachable by relabeling hidden units — really measures whether information flows one way, rather than merely describing the layout of large weights; if the metric tracks the wrong triangle or the permutation search is unreliable, the observed orderedness gains would not show induced directionality.","fun_headline_variants_meta":{"raw":{"variants":["Pruning induces directional flow in recurrent nets","Sparsity alone can order recurrent computation","Directionality emerges from pruning, not architecture","Top-K pruning discovers triangular weight flow","Pruned recurrent nets learn one-way information flow"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000175,"raw_usage":{"total_tokens":1247,"prompt_tokens":865,"completion_tokens":382,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":481,"completion_tokens_details":{"reasoning_tokens":315}},"tokens_in":481,"tokens_out":382,"duration_ms":4444,"temperature":1.0,"reasoning_tokens":315,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:49:44.692463+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the trained complete perceptron layers from the XOR/Sine runs and compare the effect on the output of zeroing an upper-triangular edge versus a lower-triangular edge of comparable magnitude. If removing lower-triangular (feed-back) edges changes the output as much as removing upper-triangular (feed-forward) edges, then the increase in $O(W)$ under Top-K pruning does not reflect one-way information flow.","supporting_citations":[{"cited_title":"Proving the lottery ticket hypothesis: Pruning is all you need","cited_arxiv_id":null,"evidence_quote":"Proves that pruning can uncover high-performing subnetworks, providing the theoretical backdrop for treating magnitude pruning as a structural force."},{"cited_title":"The evolutionary origins of hierarchy","cited_arxiv_id":null,"evidence_quote":"Shows that connection-cost pressure leads networks to evolve hierarchy, the direct inspiration for testing sparsity-induced orderedness."},{"cited_title":"Seeing is believing: Brain-inspired modular training for mechanistic interpretability","cited_arxiv_id":null,"evidence_quote":"Demonstrates that connection-cost regularization increases modularity and interpretability, extending the same intuition from modularity to directionality."},{"cited_title":"V., and Newsome, W","cited_arxiv_id":null,"evidence_quote":"Provides biological evidence that recurrent, non-feedforward dynamics support computation, motivating the question of whether directionality is needed at all."}],"review_version":1}