pith. sign in

arxiv: 2605.15435 · v2 · pith:N7VNTZ53new · submitted 2026-05-14 · 💻 cs.LG · cs.NE

On the Stability of Growth in Structural Plasticity

Lute Lillo , Nick Cheney This is my paper

Pith reviewed 2026-05-19 15:34 UTC · model grok-4.3

classification 💻 cs.LG cs.NE
keywords structural plasticitynetwork growthpruninggradient starvationcontinual learningadaptive architectureinsertion problem
0
0 comments X

The pith

Newborn network units participate in the forward pass but receive weaker gradients than units present from the start.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows that adding units to a neural network during training is not the simple opposite of pruning. Pruning chooses among units that have been trained from the beginning, while growth places new units into an optimization trajectory that has already specialized. This leaves the new units active in computing outputs yet starved of the gradient signals needed to learn quickly. A reader should care because the effect is small in basic multilayer perceptrons but becomes noticeable in convolutional networks on image tasks and in continual-learning settings where plasticity matters. The authors demonstrate that interventions can help new units integrate, yet this does not always translate into stronger final sparse networks.

Core claim

Newborn units are often forward-active but backward-starved: they participate in the forward computation, yet receive much weaker gradient signal than incumbent units. This disadvantage is minor in small MLP benchmarks but becomes clear in harder image-classification settings with a convolutional trunk. Grow can achieve high final accuracy during the structural-editing procedure, while Prune is stronger when performance is averaged over the training trajectory or when the final sparse network is retrained from scratch.

What carries the argument

The insertion problem: new units are added into an already specialized optimization trajectory, producing systematically weaker back-propagated gradients than those received by units that have been present from initialization.

Load-bearing premise

By the time new units are inserted the optimization trajectory has already specialized, so their gradient signals are weaker than those of the original units.

What would settle it

Direct measurement of per-unit gradient magnitudes right after insertion across multiple runs; if new units consistently receive gradient norms comparable to incumbent units, the central claim would not hold.

Figures

Figures reproduced from arXiv: 2605.15435 by Lute Lillo, Nick Cheney.

Figure 1
Figure 1. Figure 1: Cycle vs. Winning-Ticket performance on CIFAR-100. Panels (a)–(d) show mean ± 95% CI with individual seed points. (a) Grow achieves higher final cycle accuracy than Prune, but (b) this advantage vanishes when retraining the final mask from scratch. (c) Viewing the overall trajectory, Prune maintains a stronger or comparable TAA over the cycle, while (d) winning-ticket TAA remains similar across all sparse … view at source ↗
Figure 2
Figure 2. Figure 2: Growth inserts units that participate in the forward pass but receive weak backward signal. Event-aligned cohort diagnostics on CIFAR-100, where log-parity 0 denotes equality between the compared cohorts. (a) At birth, newborn Grow units have positive activation parity, showing that they are not inactive or dead on arrival; however, this forward participation weakens across successive grow cycles. (b) The … view at source ↗
Figure 3
Figure 3. Figure 3: Newborn units approach forward-activity parity, but not backward-signal parity. Post￾birth dynamics on CIFAR-100 measure newly grown units relative to already-active units over the remaining training segment after each growth event; parity is marked by the red dashed line at 1. Across compactness levels, activation ratio (blue) stays near parity and sometimes exceeds it, indicating that newborn units parti… view at source ↗
Figure 4
Figure 4. Figure 4: Early newborn integration predicts adaptive-cycle quality, more than final ticket qual￾ity. Panels (a)–(b) relate early post-birth parity to Cycle-TAA. Early parity is the average log-ratio between newborn and previously active units over the early post-birth window; values closer to 0 indicate closer parity. Large labeled markers denote method means across compactness; lighter points show individual compa… view at source ↗
Figure 5
Figure 5. Figure 5: Repeated-shift benchmarks favor pruning, while integration-friendly growth is the most reliable growth variant. Across six CL benchmarks, Prune is the strongest or near-strongest structural baseline in most rapid-shift settings, consistent with the advantage of preserving mature capacity. Among growth-family methods, Grow + Rand. Smooth-Leaky is the most robust: it consistently improves over Grow, narrows … view at source ↗
Figure 6
Figure 6. Figure 6: Gradient parity as the primary mechanistic signal (CIFAR-100). Cycle-TAA vs. event￾local gradient parity across compactness. The vertical dashed line marks parity (0). Grow occupies the negative-parity regime, indicating newborn gradient disadvantage, whereas Prune occupies the positive-parity regime, indicating that kept units receive stronger learning signal than pruned ones [PITH_FULL_IMAGE:figures/ful… view at source ↗
Figure 7
Figure 7. Figure 7: Activation parity as a sanity check (CIFAR-100). Cycle-TAA vs. event-local activation parity across compactness. The vertical dashed line marks parity (0). Activation parity shows that newborn units are not trivially inactive, but it does not account for the main performance separation as directly as gradient parity. B.3.5 Parity geometry of structural edits [PITH_FULL_IMAGE:figures/full_fig_p026_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Parity geometry of structural edits. Each panel corresponds to a compactness target 𝑐. Points plot log activation parity (x-axis) against log gradient parity (y-axis), color-coded by cycle. Markers distinguish event type: Grow-birth (new vs. old) and Prune-exit (kept vs. pruned). Positive x-values indicate greater activation in the focal cohort, while negative y-values indicate reduced per-unit learning si… view at source ↗
Figure 9
Figure 9. Figure 9: Birth-time parity under activation- and gradient-based Grow. We compare the default activation-based top-𝑘 Grow heuristic with a gradient-based top-𝑘 variant under neutral allocation bias. Top row reports birth activation parity, log(act𝑛𝑒𝑤/act𝑜𝑙𝑑 ), and bottom row reports birth gradient parity, log(grad𝑛𝑒𝑤/grad𝑜𝑙𝑑 ). The dotted line denotes parity. Both heuristics produce forward-active newborn units, but… view at source ↗
Figure 10
Figure 10. Figure 10: Post-insertion ratio under activation- and gradient-based Grow. We report cycle-level ratios from the vitality logs. Top row shows activation ratio act𝑛𝑒𝑤/act𝑜𝑙𝑑 , and bottom row shows gradient ratio, grad𝑛𝑒𝑤/grad𝑜𝑙𝑑 . The dotted line denotes parity. Although newborn activation rates remain close to parity, newborn gradient magnitudes remain below parity for both heuristics. The bottleneck is not simply a… view at source ↗
Figure 11
Figure 11. Figure 11: Cycle vs. Winning-Ticket performance on CIFAR-10 (SGD, 𝜂=0.1). Panels (a)–(d) show mean ± 95% CI with per-seed scatter across compactness for Cycle and Winning-Ticket ACC (a,b) and TAA (c,d). Panel (e) reports the per-seed gap Δ = ticket − cycle in final accuracy. scratch, however, the two methods become nearly indistinguishable: Winning-Ticket ACC and TAA are almost tied across all compactness levels. Th… view at source ↗
Figure 12
Figure 12. Figure 12: Grow cycle stress test on CIFAR-100. Left: Cycle-TAA degrades monotonically with 𝐾 at all compactness levels, indicating worse time-averaged learning when growth events become more frequent. Right: Cycle-ACC is less affected than TAA, with only mild sensitivity at higher compactness [PITH_FULL_IMAGE:figures/full_fig_p032_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Post-birth dynamics under time scarcity. Gradient ratio as a function of newborn age for 𝐾 ∈ {5, 10, 20} across compactness levels. In all cases, parity improves gradually with age, showing that newborn integration is slow and continues over many epochs. Shorter cycles truncate this recovery by reducing the time available before the next growth event. C Two-Speed and Moment Transplant Explanation This app… view at source ↗
Figure 14
Figure 14. Figure 14: Activation-control analysis across eight benchmarks. Each bar reports the signed delta Δ = RSL−ReLU, positive values indicate that replacing ReLU with Rand. Smooth-Leaky improves performance for that method and compactness. The central pattern is not a uniform lift across all methods, but a redistribution of benefit across structural regimes: Rand. Smooth￾Leaky most strongly improves Grow in several of th… view at source ↗
Figure 15
Figure 15. Figure 15: Rand. Smooth-Leaky increases newborn forward participation at birth but does not eliminate the immediate gradient disadvantage. We compare ReLU and Rand. Smooth￾Leaky in the CIFAR-100 Grow setting using event-aligned newborn–old log-parity at the birth snapshot. Positive activation parity indicates that newborn units are forward-active relative to previously active units, while negative gradient parity in… view at source ↗
Figure 16
Figure 16. Figure 16: Rand. Smooth-Leaky improves post-birth dynamics. We compare activation and gradient ratios for newborn units after growth events in CIFAR-100 Grow. Parity corresponds to a ratio of 1. Under ReLU, newborn units often remain below activation parity and receive substantially weaker gradient signal than incumbents. Rand. Smooth-Leaky shifts activation ratios above or closer to parity and consistently increase… view at source ↗
Figure 17
Figure 17. Figure 17: Early-task plasticity across sequential-accumulation and repeated-shift benchmarks. We report Early Task TAA, defined as the average accuracy over only the initial portion of each task, emphasizing immediate post-shift adaptation rather than late within-task convergence. Dense or Prune-based methods retain the strongest early-window performance when adaptation time is severely limited, whereas growth-base… view at source ↗
read the original abstract

Standard deep-learning pipelines usually choose the network architecture before training and keep it fixed throughout optimization. In contrast, a model can also be adapted by editing its structure during training, for example by pruning existing hidden-neuron units or growing new ones. Although growth is appealing for adaptive and continual systems, we show that it is not simply the inverse of pruning. Pruning selects among units that have participated in training from the start, whereas growth inserts new units into an already specialized optimization trajectory. We isolate this insertion problem and show that newborn units are often forward-active but backward-starved: they participate in the forward computation, yet receive much weaker gradient signal than incumbent units. This disadvantage is minor in small MLP benchmarks, but becomes clear in harder image-classification settings with a convolutional trunk. In these settings, \textsc{Grow} can achieve high final accuracy during the structural-editing procedure, while \textsc{Prune} is stronger when performance is averaged over the training trajectory or when the final sparse network is retrained from scratch. Interventions targeting optimizer state, insertion, selection, and trainability show that improving the integration of newborn units can improve adaptive performance, but does not automatically produce better final subnetworks. In continual-learning benchmarks stressing plasticity loss, \textsc{Grow} becomes competitive mainly when new units have enough time to integrate. Together, these results suggest that \textsc{Grow} should be evaluated not only as an architecture-search operator, but as a time-sensitive optimization process whose success depends on insertion stability.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 3 minor

Summary. The paper examines structural plasticity in deep networks by comparing growth (adding new units mid-training) to pruning (removing units). It argues that growth is not the inverse of pruning because new units are inserted into an already-specialized optimization trajectory, resulting in newborn units that are forward-active but backward-starved (participating in the forward pass yet receiving systematically weaker gradient signals than incumbent units). This is demonstrated through controlled experiments on MLPs and convolutional image-classification tasks, interventions targeting optimizer state/insertion/selection/trainability, distinctions between final accuracy and trajectory-averaged/retrained performance, and continual-learning settings where integration time affects competitiveness.

Significance. If the central empirical observations hold, the work usefully highlights an asymmetry in dynamic architecture methods and frames growth as a time-sensitive optimization process rather than a pure architecture-search operator. The targeted interventions and separation of final vs. averaged performance provide concrete guidance for adaptive and continual-learning systems. Credit is due for the reproducible-style interventions and the focus on plasticity loss in continual benchmarks.

major comments (2)
  1. [§4.2] §4.2 (gradient-signal experiments): the claim that newborn units receive 'much weaker gradient signal' is central to the insertion-problem argument, yet the manuscript does not specify the precise metric (e.g., per-unit L2 norm, cosine similarity to incumbent gradients, or layer-wise averages) nor whether any scaling or normalization is applied before comparison; without this, it is difficult to assess whether the observed difference is an artifact of initialization scale or a genuine optimization-trajectory effect.
  2. [Table 3] Table 3 and associated text (image-classification results): the reported advantage of Grow on final accuracy versus Prune on trajectory-averaged or retrained performance lacks error bars, number of random seeds, or statistical tests; given that the central claim rests on these performance distinctions, the absence of variability measures makes it impossible to judge whether the differences are robust or could be explained by post-hoc run selection.
minor comments (3)
  1. [Abstract] Abstract and §2: the phrase 'backward-starved' is used repeatedly before any quantitative definition or figure reference is given; a short parenthetical or footnote linking to the measurement protocol would improve readability.
  2. [§5] §5 (continual-learning experiments): the statement that 'Grow becomes competitive mainly when new units have enough time to integrate' would benefit from an explicit plot or table showing accuracy as a function of insertion timing or number of subsequent epochs.
  3. [§3] Notation: the distinction between 'final accuracy during the structural-editing procedure' and 'performance averaged over the training trajectory' is important but introduced without a compact symbol or equation; adding a brief definition in §3 would reduce ambiguity.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive and detailed feedback on our manuscript examining the asymmetry between growth and pruning in structural plasticity. We address each major comment below and outline the revisions we will make to improve clarity and reporting standards.

read point-by-point responses

  1. Referee: [§4.2] §4.2 (gradient-signal experiments): the claim that newborn units receive 'much weaker gradient signal' is central to the insertion-problem argument, yet the manuscript does not specify the precise metric (e.g., per-unit L2 norm, cosine similarity to incumbent gradients, or layer-wise averages) nor whether any scaling or normalization is applied before comparison; without this, it is difficult to assess whether the observed difference is an artifact of initialization scale or a genuine optimization-trajectory effect.

    Authors: We agree that the gradient metric requires explicit definition to rule out artifacts. In the experiments, gradient signal strength was quantified as the L2 norm of the gradient with respect to each unit's incoming parameters, averaged across the batch and over post-insertion training steps; no additional per-unit scaling or normalization was applied beyond the standard back-propagation and Adam optimizer state. We have revised §4.2 to state this definition verbatim, added a short paragraph explaining why the L2 norm is appropriate for comparing backward starvation, and included a supplementary check confirming that the observed gap remains after matching initialization variance. These changes directly address the concern. revision: yes

  2. Referee: Table 3 and associated text (image-classification results): the reported advantage of Grow on final accuracy versus Prune on trajectory-averaged or retrained performance lacks error bars, number of random seeds, or statistical tests; given that the central claim rests on these performance distinctions, the absence of variability measures makes it impossible to judge whether the differences are robust or could be explained by post-hoc run selection.

    Authors: The referee correctly identifies a reporting gap. The Table 3 results were obtained from five independent random seeds, with the reported trends consistent across all runs. In the revised manuscript we will add standard-deviation error bars to the table, explicitly state the number of seeds, and include a brief statistical note (paired t-tests on the key metrics) either in the main text or as a short supplement paragraph. This will allow readers to assess robustness directly. revision: yes

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The paper advances empirical claims about the stability of structural growth versus pruning in neural networks, supported by benchmark comparisons, interventions on optimizer state and insertion timing, and observations of gradient signals in convolutional and continual-learning settings. No derivation chain, first-principles equations, or fitted parameters are presented that reduce by construction to inputs defined within the paper itself; the central distinction between Grow and Prune rests on externally measurable performance differences rather than self-referential definitions or self-citation load-bearing arguments.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The work relies on standard deep-learning assumptions about gradient flow and trajectory specialization. No new entities are introduced and no free parameters are explicitly fitted to produce the central observation.

axioms (1)
  • domain assumption Once training has progressed, the optimization trajectory has specialized such that newly inserted units receive systematically weaker gradients.
    This premise is invoked to explain why growth is not the inverse of pruning.

pith-pipeline@v0.9.0 · 5797 in / 1288 out tokens · 60106 ms · 2026-05-19T15:34:07.989745+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.