Pith. sign in

REVIEW 4 major objections 4 minor 17 references

Decoupling Search and Learning in Neural Net Training

T0 review · 4 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A two-phase training scheme—evolutionary search over activations, then gradient regression—approaches SGD on MNIST, CIFAR-10, and CIFAR-100.

desk verdict A genuinely new proof-of-concept training scheme that evolves activation targets and regresses to them, landing within a few points of SGD on three benchmarks, but the operator recipe is never ablated and the diversity-scaling claim is weaker than advertised. read the letter →

arxiv 2509.10973 v1 pith:RKJFT6RU submitted 2025-09-13 cs.LG cs.AI

Akshay Vegesna , Samip Dahal This is my paper · ORCID
classification cs.LGcs.AI
keywords evolutionarysearchrepresentationspacedecoupledtrainingbackpropagationalternativesgradientdescentneuralnetworkCIFARMNIST
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that neural network training can be split into two decoupled phases: an evolutionary search that finds good intermediate activations (representations), and a gradient-based learning phase that trains the network to reproduce those representations. The authors show this split is viable: on MNIST, CIFAR-10, and CIFAR-100, networks trained by regressing to searched representations approach standard SGD accuracy without ever backpropagating classification gradients through the network body. They also show that the quality and diversity of searched representations improve with compute, and that the resulting models are qualitatively different from SGD-trained models. The significance is that it offers a concrete way to combine the exploratory power of search with the efficiency of gradient descent, potentially addressing gradient descent's tendency to settle for a single minimum.

What carries the argument

The key mechanism is the decoupling itself: evolutionary search over representation space (the layerwise activation tensors) is separated from gradient-based learning in parameter space. The search space is the set of activations at the outputs of three convolutional blocks plus the final logits; evolution proceeds layer by layer, fixing earlier optimized representations, and uses per-image top-k selection, crossover, channel-wise Gaussian mutation, repeated 3x3 average pooling (spatial smoothing) and per-sample normalization. The learning phase minimizes an MSE regression loss to those cached targets on each block and a KL divergence from the searched class distribution on the logits, with

What would settle it

Run the same search-then-regress pipeline on a fourth dataset (e.g., SVHN or a downsampled ImageNet) with operators unchanged; if accuracy drops far below SGD (well beyond the observed 1–3% gap) or if removing spatial smoothing or normalization makes searched representations unlearnable (training loss does not decrease), the claim that the mechanism generalizes is falsified.

Watch

Extended reading notes

Core claim

The central discovery is that the intermediate activations of a convolutional network can serve as a tractable search space. By evolving these activations layer-by-layer with crossover, channel-wise Gaussian mutation, spatial smoothing, and normalization—using classification cross-entropy as the fitness signal—the authors obtain fixed 'target' representations for every training example. A separate gradient-descent phase then trains network parameters to match these targets via MSE on each convolutional block plus a stop-gradient KL loss on the logits, so no classification gradient reaches the body. This pipeline achieves test accuracy within roughly one percentage point of SGD on all three b

Load-bearing premise

The load-bearing premise, stated in Section 3.1, is that the specific evolutionary operators—particularly repeated spatial smoothing, which the paper calls 'crucial for learnability,' and normalization, which is needed 'for convergence of evolution'—produce representations that are both learnable by gradient descent and generalizing; no ablation is provided, so if these operators are brittle or dataset-specific, the central claim does not transfer beyond the three benchmarks

Editorial extensions

If this is right

  • If search over representations is a viable replacement for backprop through the body, then the network's convolutional layers can be trained without explicit classification gradients, only regression targets.
  • Performance improves with more search compute (larger populations, more generations), saturating, so search budget can be traded for accuracy.
  • The method produces models whose representations are distant from SGD's (cosine distance ~1), demonstrating multiple qualitatively different solutions exist.
  • Diversity of searched solutions grows with compute (effective number of solutions Neff), suggesting search could provide a source of diverse models.
  • The one-shot cached search shows even a single search pass suffices to approach SGD; iterative search-learning cycles are a clear next step.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the operator set generalizes, the same decoupling could be applied to other architectures (e.g., transformers) where a tractable representation space exists, potentially enabling search-based exploration of attention maps or hidden states.
  • The authors' finding that skipping supervision on the first block helps with augmentation suggests that the searched representations for early layers may be harder to fit; one testable extension is to adaptively decide which layers to supervise based on learnability.
  • The reliance on cached representations means the method's effectiveness depends on search quality; an inference is that closing the loop (search, learn, re-search with the trained network) could close the remaining accuracy gap, as the paper's future-work section suggests.
  • Because search is performed independently per image, the method currently ignores inter-image dependencies; a possible extension is to search over batch-level representations to capture class structure.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a two-phase training framework: an evolutionary search over intermediate activations (representation space) produces per-example target representations, and a standard gradient-descent training phase regresses the network body to those targets, with a stop-gradient KL term training only the classification head. Experiments on MNIST, CIFAR-10, and CIFAR-100 report test accuracies within about 1–2.6 percentage points of SGD, and the authors show that increasing search compute (generations or population size) improves the quality of the searched representations and the final validation accuracy. The paper also presents evidence that the resulting models occupy different representational trajectories than SGD models.

Significance. If the empirical claims are robust, the paper makes a useful proof-of-concept contribution: a tractable representation-level search can be decoupled from gradient-based parameter learning and still produce reasonably generalizing models, with the potential to explore more diverse solutions than ordinary SGD. I credit the authors for a clear description of the method, explicit hyperparameters in the appendix, and an honest discussion of limitations. The central claim is empirical rather than theoretical, and there is no in-paper circularity because search is performed on training data and evaluation is on held-out test sets. However, the strength of the claim depends strongly on the target-generation process being learnable and on the evolutionary search actually being the operative ingredient; the paper currently lacks the ablations and controls needed to establish this.

major comments (4)
  1. [§3.1, §A.4] The paper states that spatial smoothing is 'crucial for learnability' and normalization is needed 'for convergence of evolution,' but no ablation isolates these operators. Because the search is performed independently per image (§3.1) and the cached targets are fixed one-per-example (§4.2), the only mechanisms enforcing that the map x -> \hat H(x) is smooth/learnable are the crossover, Gaussian mutation, repeated blur, and normalization. Without a zero-generation/random-target control and ablations that remove or weaken these operators, the reported near-SGD test accuracies could be a property of the hand-tuned smoothing/normalization recipe rather than of evolutionary search. A concrete fix: report training and test accuracy for (i) random targets with the same smoothing/normalization, (ii) targets from generation 0 only, and (iii) targets generated without spatial smoothing or without
  2. [§3.2, Figure 3] The claimed 'diversity scales with compute' is not supported by the experimental setup. Figure 3 plots Neff as a function of the number of independent evolutionary runs, not as a function of population size or generations. Aggregating more independent runs trivially increases the effective number of distinct solutions, even if per-run diversity is constant. The abstract and conclusion claim 'larger populations and more generations produce better and more varied solutions,' but only the fitness part is measured along those axes. The diversity claim should be re-tested by computing Neff within a single search as population size or generations increase, or at least across runs with matched compute.
  3. [§4.3, Tables 1–2, §5] The paper does not compare against target propagation or other layerwise-target baselines, despite citing target propagation as 'the most similar to our method.' More importantly, there is no control showing that the evolutionary search itself, rather than the regression to any sufficiently smooth per-image target, is responsible for the performance. A minimal baseline is to train the same network by MSE regression to (a) randomly sampled targets that undergo the same blur/normalization, and (b) targets obtained from a forward pass of an untrained network with the same noise, i.e., zero generations. Without such controls, the central claim that 'search-based regression' approaches SGD is not isolated from the generic effect of layerwise target fitting.
  4. [§4.2, §A.2–A.3] The method has a large number of free hyperparameters (population size, top-k, c_exp, c_ref, per-layer generations, mutation strength, exploration boost, blur passes, KL weight lambda, learning rate, batch size, epochs, block-depth expansion, skip-block-0). All main results use a single configuration per dataset, and the skip-block-0 variant is introduced ad hoc after the all-layers variant underperforms (Table 2). Since the reported gaps to SGD are small (1.0–2.6 points), a sensitivity analysis over at least the search-critical parameters (mutation strength, blur passes, population/generations, lambda) is necessary to establish that the method is not brittle. Without it, the reported accuracies are anecdotal evidence for a single point in a large hyperparameter space.
minor comments (4)
  1. [Figure 4] The scaling curves for validation accuracy versus population/generations have no error bars or multiple-seed indication. Given that Table 1 reports mean±std over 3 runs, the same should be provided here.
  2. [§A.5] The MNIST configuration is described as 'the same search configuration as CIFAR-10' with a uniform alpha=0.1, but training uses four convolutional layers instead of six. It would help to state explicitly whether the MNIST results use the 'All layers' or 'Skip block 0' supervision scheme, and whether the comparison is to 2-, 4-, or 6-layer SGD baselines.
  3. [§4.2, Eq. (2)] The notation around the stop-gradient in the KL term is slightly ambiguous: it would be clearer to write KL(\hat p(·|x) || p_φ(·| sg(H^{(L-2)}_θ(x)))) to make explicit that φ receives gradients through p_φ while θ does not.
  4. [§A.2] The hyperparameters are given as text; a table would improve reproducibility. Also, the paper could mention whether code will be released.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central claims are empirical, held-out evaluations of a search-then-regress pipeline, and no prediction reduces to a fitted input or self-citation.

full rationale

The paper's central claim is that networks trained by regressing to evolutionarily searched intermediate representations approach SGD's test accuracy on MNIST, CIFAR-10, and CIFAR-100. The searched representations are cached by optimizing fitness (cross-entropy with training labels) on an untrained network, and then a separate, deeper network is trained to match those targets. Test accuracy is measured on held-out data, so the reported numbers are not statistically forced by the training-label fit. No parameter is fitted to a subset of data and then reported as a prediction of closely related data; the searched targets are not test labels or test predictions. There are no self-citations, no imported uniqueness theorems, and no ansatz smuggled in via citation: the cited prior work (target propagation, diversity metrics, neuroevolution) is contextual and not load-bearing for the generalization claim. The closest thing to a construction is that search-based models are explicitly trained to be close to the searched targets, so cosine-distance plots in Section 4.5 show they are closer to those targets than SGD is—but this is a sanity check, not a predictive claim. The paper's limitations (one-shot search, trailing SGD, no iterative feedback) are honest scope statements and not admissions of circularity. Concerns about missing ablations or the brittleness of the smoothing/normalization operators are correctness or generalization risks, not circularity under the definitions used here.

Assumptions & free parameters 12 free parameters · 3 assumptions · 0 invented entities

The method introduces no new physical or mathematical entities. The central claim rests on the practical effectiveness of a large set of hand-tuned search and training hyperparameters, plus two unproven domain assumptions about the tractability of search in activation space and the learnability of the resulting targets.

free parameters (12)
  • Population size (n_pop) = 240
    Number of candidate activations per layer; hand-tuned in Appendix A.2.
  • Top-k selected per image = 20
    Number of parents retained per image per generation; hand-tuned.
  • Exploratory children (c_exp) = 100
    Number of high-mutation samples per generation.
  • Refinement children (c_ref) = 120
    Number of standard-mutation samples per generation.
  • Generations per layer = 300, 100, 100, 10 for layers 0-3
    Evolution schedule chosen by hand.
  • Mutation strength per layer (alpha) = 0.3, 0.2, 0.1, 0.1
    Per-layer mutation scales, hand-tuned.
  • Exploration boost per layer = 5.0x, 3.0x, 3.0x, 1.0x
    Multiplier on mutation strength for exploratory samples.
  • Blur passes per layer = 5, 1, 1, 0
    Number of spatial smoothing passes; chosen by hand.
  • KL divergence weight lambda = 0.03
    Loss weight for logit distribution matching; hand-tuned.
  • Learning rate and momentum = 4.0, 0.85 Nesterov
    SGD hyperparameters from Appendix A.3.
  • Batch size and epochs = 2000, 90 (1000 for MNIST)
    Training schedule.
  • Skip block 0 supervision (structural choice) = True for reported main results
    Design variant selected based on validation performance; the alternative 'All layers' performs worse.
assumptions (3)
  • domain assumption Evolutionary search in activation space is tractable: random perturbations can make progress toward high-fitness solutions.
    Stated as a design principle in Section 2.1, not proven; the empirical results on three benchmarks are the only support.
  • ad hoc to paper Regressing to searched representations yields models that generalize.
    Core hypothesis of Section 4; supported only by the three benchmark experiments and no theory.
  • domain assumption The searched targets lie within the set of activations reachable by gradient descent from the network initialization.
    The paper relies on spatial smoothing and normalization to enforce this (Section 3.1, A.4), but does not measure reachability or provide ablations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Decoupling Search and Learning in Neural Net Training." pith.science (2026). https://pith.science/paper/RKJFT6RU

@misc{pith2026250910973,
  author       = {Pith},
  title        = {Pith review of: Decoupling Search and Learning in Neural Net Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RKJFT6RU}},
  note         = {Machine review of arXiv:2509.10973}
}
read the original abstract

Gradient descent typically converges to a single minimum of the training loss without mechanisms to explore alternative minima that may generalize better. Searching for diverse minima directly in high-dimensional parameter space is generally intractable. To address this, we propose a framework that performs training in two distinct phases: search in a tractable representation space (the space of intermediate activations) to find diverse representational solutions, and gradient-based learning in parameter space by regressing to those searched representations. Through evolutionary search, we discover representational solutions whose fitness and diversity scale with compute--larger populations and more generations produce better and more varied solutions. These representations prove to be learnable: networks trained by regressing to searched representations approach SGD's performance on MNIST, CIFAR-10, and CIFAR-100. Performance improves with search compute up to saturation. The resulting models differ qualitatively from networks trained with gradient descent, following different representational trajectories during training. This work demonstrates how future training algorithms could overcome gradient descent's exploratory limitations by decoupling search in representation space from efficient gradient-based learning in parameter space.

Figures

Figures reproduced from arXiv: 2509.10973 by the authors.

Figure 1
Figure 1. Layerwise forward evolution for H(0) , H(1), and H(2). At each layer ℓ, we evolve a popu￾lation Pℓ (shown within dashed boxes), keep the best as H(ℓ) (red), and fix it for subsequent layers. After mutation, fitness is evaluated by completing the remaining forward pass. Representation sizes: H(0)∈R 256×15×15 , H(1)∈R 256×7×7 , H(2)∈R 256×3×3 . 3.2 SCALING WITH COMPUTE Fitness scaling. We study how search quality impr… view at source ↗
Figure 2
Figure 2. Fitness scales with compute on CIFAR-100. Mean cross-entropy vs. population size (left) [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Diversity grows with compute across convolutional block representations [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Accuracy scales with compute usage in evolutionary search on CIFAR-100 (no data aug [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Cosine distance to searched representations vs. epoch at different searched layers with [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Collision entropy within and between classes for SGD and search-based training (Block [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Collision entropy within and between classes for SGD and search-based training at Block [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 5 linked inside Pith

  1. [1]

    How auto-encoders could provide credit assignment in deep networks via target propagation

    Yoshua Bengio. How auto-encoders could provide credit assignment in deep networks via target propagation. arXiv preprint arXiv:1407.7906, 2014. doi:10.48550/arXiv.1407.7906. URL https://arxiv.org/abs/1407.7906

    arXiv doi:10.48550/arxiv.1407.7906 2014

  2. [2]

    Towards scaling difference target propagation by learning backprop targets

    Maxence M Ernoult, Fabrice Normandin, Abhinav Moudgil, Sean Spinney, Eugene Belilovsky, Irina Rish, Blake Richards, and Yoshua Bengio. Towards scaling difference target propagation by learning backprop targets. In International Conference on Machine Learning, pp.\ 5968--5987. PMLR, 2022. URL https://proceedings.mlr.press/v162/ernoult22a.html

    2022

  3. [3]

    The forward-forward algorithm: Some preliminary investigations

    Geoffrey Hinton. The forward-forward algorithm: Some preliminary investigations. arXiv preprint arXiv:2212.13345, 2022. doi:10.48550/arXiv.2212.13345. URL https://arxiv.org/abs/2212.13345

    arXiv doi:10.48550/arxiv.2212.13345 2022

  4. [4]

    Difference target propagation

    Dong-Hyun Lee, Saizheng Zhang, Asja Fischer, and Yoshua Bengio. Difference target propagation. In Joint european conference on machine learning and knowledge discovery in databases, pp.\ 498--515. Springer, 2015

    2015

  5. [5]

    Random synaptic feedback weights support error backpropagation for deep learning

    Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin J Akerman. Random synaptic feedback weights support error backpropagation for deep learning. Nature communications, 7: 0 13276, 2016. doi:10.1038/ncomms13276. URL https://www.nature.com/articles/ncomms13276

    doi:10.1038/ncomms13276 2016

  6. [6]

    Searching latent program spaces

    Matthew V Macfarlane and Cl \'e ment Bonnet. Searching latent program spaces. arXiv preprint arXiv:2411.08706, 2024. URL https://arxiv.org/abs/2411.08706

    arXiv 2024

  7. [8]

    Layer by layer: Uncovering hidden representations in language models

    Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Nikul Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. Layer by layer: Uncovering hidden representations in language models. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=WGXb7UdvTX

    2025

  8. [9]

    A formal theory of inductive inference

    Ray J Solomonoff. A formal theory of inductive inference. part i. Information and control, 7 0 (1): 0 1--22, 1964

    1964

Show all 17 references
  1. [10]

    Stanley and Risto Miikkulainen

    Kenneth O. Stanley and Risto Miikkulainen. Evolving neural networks through augmenting topologies. Evolutionary Computation, 10 0 (2): 0 99--127, 2002. doi:10.1162/106365602320169811. URL https://direct.mit.edu/evco/article/10/2/99/1123

    2002 doi

  2. [11]

    Stanley, and Jeff Clune

    Felipe Petroski Such, Vashisht Madhavan, Edoardo Conti, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning. arXiv preprint arXiv:1712.06567, 2017. U...

    2017 arXiv

  3. [12]

    Dweep Trivedi, Jesse Zhang, Shao‐Hua Sun, and Joseph J. Lim. Learning to synthesize programs as interpretable and generalizable policies. In Advances in Neural Information Processing Systems, 2021. URL https://arxiv.org/abs/2108.13643

    2021 arXiv

  4. [13]

    Neural network diffusion, 2024

    Kai Wang, xu Zhao Pan, Zhuang Liu, Zelin Zang, Trevor Darrell, and Yang You. Neural network diffusion, 2024. URL https://openreview.net/forum?id=8Q6UmFhhQS

    2024

  5. [14]

    Natural evolution strategies

    Daan Wierstra, Tom Schaul, Tobias Glasmachers, Yi Sun, Jan Peters, and J \"u rgen Schmidhuber. Natural evolution strategies. Journal of Machine Learning Research, 15 0 (27): 0 949--980, 2014. URL https://www.jmlr.org/papers/v15/wierstra14a.html

    2014

  6. [15]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  7. [16]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  8. [17]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  9. [18]

    "" _觟 ҕ+WrOW׮]կ_?5koю ;vKV twe˖ sΝ駟yf :tH 9n[X1u] ?Ufȑ#:u6lؠ˗/WZҘ1cԬY3sÇ o-ܢ> q +7rq C˖-ӳ>k <x^ <׫W/ իWrٶ 5

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2025 arXiv

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.