{"id":"44df93e7-c197-4451-a686-aae7b1ef232f","arxiv_id":"2509.10973","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":12,"one_line_summary":"Training can be split into evolutionary search over intermediate activations and gradient-based regression to those activations, achieving near-SGD accuracy on three benchmarks.","lead":"A new training method searches over a neural network's internal activations with an evolutionary algorithm, then trains the network by regression to the best found representations. The approach reaches close to standard SGD accuracy on MNIST, CIFAR-10 and CIFAR-100, without backpropagating classification gradients through the network body.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No ablation or control for search operators and per-image target learnability; near-SGD results may reflect the hand-tuned smoothing/normalization recipe, not the decoupling claim.","rationale":"The reader's weakest assumption — no ablation of the evolutionary operators — is real and important. I extend it: the per-image independence of the search (Section 3.1) means the target map is defined pointwise, and only the smoothing/normalization operators can enforce cross-example consistency. The paper's own admission that smoothing is 'crucial for learnability' makes this a load-bearing dependency, yet no experiments quantify it. The empirical tables and scaling plots provide genuine support for the claim on the tested benchmarks, so I would not reject; but the missing controls keep the central mechanism underdetermined. A conditional accept remains appropriate, and the proposed three-arm ablation would directly settle whether the reported near-SGD accuracy is due to the search or to the target-construction constraints. I agree with the reader's overall verdict but sharpen the specific locus of concern.","tokens_in":9376,"tokens_out":16786,"duration_ms":231108,"concrete_test":"Run the CIFAR-10 no-augmentation pipeline with identical hyperparameters in three arms: (A) full evolutionary targets as in the paper; (B) generation-0 targets only (initial noisy population, same smoothing/normalization, no selection/mutation/generations); (C) full evolutionary targets but with 0 blur passes at all convolutional search layers. Compare final test accuracy to Table 1's 88.3±0.3. If B or C is within ~1 point of A, the specific evolutionary search/smoothing is not load-bearing; if either drops by >2 points, the corresponding component is load-bearing and the paper's central claim requires qualification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim — that regressing to searched intermediate representations approaches SGD — requires the searched targets to form a learnable, generalizing function of the input. Section 3.1 makes the search per-image ('All selection, mutation, and fitness evaluation operations are performed independently for each image'), and Section 4.2 caches one target sequence per training example. Nothing in the algorithm couples targets across examples or guarantees that x -> \\hat H(x) is smooth. The only mechanisms that could enforce such structure are the mutation/noise, repeated 3x3 spatial smoothing, and per-sample normalization, and the paper itself states (Section 3.1) that spatial smoothing is 'crucial for learnability' and normalization is required 'for convergence of evolution,' but no ablation isolates these contributions. If the target map is not learnable, the trained network would memorize rather than generalize; the close-to-SGD test accuracies in Tables 1–2 are the only evidence that the map is learnable, and they are confined to three datasets with one hyperparameter configuration. This is the load-bearing risk: the method's success may be a property of the hand-selected smoothing/normalization recipe rather than of the claimed decoupling of search and learning. The absence of a zero-generation/random-target control and of a pairwise target-consistency analysis makes it impossible to tell whether the evolutionary search itself, or merely the target-generation constraints, is responsible.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-phase training framework: an evolutionary search over intermediate activations (representation space) produces per-example target representations, and a standard gradient-descent training phase regresses the network body to those targets, with a stop-gradient KL term training only the classification head. Experiments on MNIST, CIFAR-10, and CIFAR-100 report test accuracies within about 1–2.6 percentage points of SGD, and the authors show that increasing search compute (generations or population size) improves the quality of the searched representations and the final validation accuracy. The paper also presents evidence that the resulting models occupy different representational trajectories than SGD models.","tokens_in":9793,"tokens_out":5063,"duration_ms":68407,"significance":"If the empirical claims are robust, the paper makes a useful proof-of-concept contribution: a tractable representation-level search can be decoupled from gradient-based parameter learning and still produce reasonably generalizing models, with the potential to explore more diverse solutions than ordinary SGD. I credit the authors for a clear description of the method, explicit hyperparameters in the appendix, and an honest discussion of limitations. The central claim is empirical rather than theoretical, and there is no in-paper circularity because search is performed on training data and evaluation is on held-out test sets. However, the strength of the claim depends strongly on the target-generation process being learnable and on the evolutionary search actually being the operative ingredient; the paper currently lacks the ablations and controls needed to establish this.","major_comments":[{"comment":"The paper states that spatial smoothing is 'crucial for learnability' and normalization is needed 'for convergence of evolution,' but no ablation isolates these operators. Because the search is performed independently per image (§3.1) and the cached targets are fixed one-per-example (§4.2), the only mechanisms enforcing that the map x -> \\hat H(x) is smooth/learnable are the crossover, Gaussian mutation, repeated blur, and normalization. Without a zero-generation/random-target control and ablations that remove or weaken these operators, the reported near-SGD test accuracies could be a property of the hand-tuned smoothing/normalization recipe rather than of evolutionary search. A concrete fix: report training and test accuracy for (i) random targets with the same smoothing/normalization, (ii) targets from generation 0 only, and (iii) targets generated without spatial smoothing or without","section":"§3.1, §A.4"},{"comment":"The claimed 'diversity scales with compute' is not supported by the experimental setup. Figure 3 plots Neff as a function of the number of independent evolutionary runs, not as a function of population size or generations. Aggregating more independent runs trivially increases the effective number of distinct solutions, even if per-run diversity is constant. The abstract and conclusion claim 'larger populations and more generations produce better and more varied solutions,' but only the fitness part is measured along those axes. The diversity claim should be re-tested by computing Neff within a single search as population size or generations increase, or at least across runs with matched compute.","section":"§3.2, Figure 3"},{"comment":"The paper does not compare against target propagation or other layerwise-target baselines, despite citing target propagation as 'the most similar to our method.' More importantly, there is no control showing that the evolutionary search itself, rather than the regression to any sufficiently smooth per-image target, is responsible for the performance. A minimal baseline is to train the same network by MSE regression to (a) randomly sampled targets that undergo the same blur/normalization, and (b) targets obtained from a forward pass of an untrained network with the same noise, i.e., zero generations. Without such controls, the central claim that 'search-based regression' approaches SGD is not isolated from the generic effect of layerwise target fitting.","section":"§4.3, Tables 1–2, §5"},{"comment":"The method has a large number of free hyperparameters (population size, top-k, c_exp, c_ref, per-layer generations, mutation strength, exploration boost, blur passes, KL weight lambda, learning rate, batch size, epochs, block-depth expansion, skip-block-0). All main results use a single configuration per dataset, and the skip-block-0 variant is introduced ad hoc after the all-layers variant underperforms (Table 2). Since the reported gaps to SGD are small (1.0–2.6 points), a sensitivity analysis over at least the search-critical parameters (mutation strength, blur passes, population/generations, lambda) is necessary to establish that the method is not brittle. Without it, the reported accuracies are anecdotal evidence for a single point in a large hyperparameter space.","section":"§4.2, §A.2–A.3"}],"minor_comments":[{"comment":"The scaling curves for validation accuracy versus population/generations have no error bars or multiple-seed indication. Given that Table 1 reports mean±std over 3 runs, the same should be provided here.","section":"Figure 4"},{"comment":"The MNIST configuration is described as 'the same search configuration as CIFAR-10' with a uniform alpha=0.1, but training uses four convolutional layers instead of six. It would help to state explicitly whether the MNIST results use the 'All layers' or 'Skip block 0' supervision scheme, and whether the comparison is to 2-, 4-, or 6-layer SGD baselines.","section":"§A.5"},{"comment":"The notation around the stop-gradient in the KL term is slightly ambiguous: it would be clearer to write KL(\\hat p(·|x) || p_φ(·| sg(H^{(L-2)}_θ(x)))) to make explicit that φ receives gradients through p_φ while θ does not.","section":"§4.2, Eq. (2)"},{"comment":"The hyperparameters are given as text; a table would improve reproducibility. Also, the paper could mention whether code will be released.","section":"§A.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is relevant to the journal's scope and the idea is interesting as a proof of concept. My main concern is evidential: the central claim is only as strong as the target-generation pipeline, and the manuscript does not yet disentangle the contributions of search, smoothing/normalization, and plain layerwise regression. I would ask for the ablations and controls described in the major comments before considering acceptance. The diversity-scaling claim also needs to be re-measured along the compute axes actually claimed in the abstract."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is a genuine proof of concept, not a breakthrough. The idea—evolve intermediate activations as regression targets, then train the network body with MSE to match them while keeping the classification head separate—is new relative to target propagation and latent-space search. The results are respectable: on three benchmarks, networks trained this way land within 1–2.6 points of SGD, and the models clearly follow different representational trajectories. The authors are upfront that they trail SGD and call it a proof of concept. That honesty earns credit.\n\nWhat's good: the scaling experiments in Section 3.2 show fitness improves with more populations/generations, and the representation distance analysis (cosine distance and collision entropy) supports the claim that search-based training produces different internal states than SGD. The method description is clear, and the limitations section names the biggest gaps itself.\n\nThe soft spots are real but mostly addressable rather than fatal. The biggest one: the operator recipe (crossover, per-channel Gaussian mutation, repeated spatial smoothing, per-sample normalization) is never ablated. The paper states smoothing is 'crucial for learnability' and normalization needed 'for convergence of evolution,' but there's no control showing that merely generating smoothed, normalized random tensors would not produce similar results. Without a zero-generation or random-target baseline, you can't tell whether the evolutionary search is responsible for the generalization or just the constraints that the search happens to enforce. That's a load-bearing gap for the 'decoupling' claim, even if the overall method works as a recipe.\n\nSecond, the diversity-scaling result in Figure 3 is weaker than advertised: it plots effective number of solutions against number of independent runs, not against compute (population size or generations). More runs trivially give you more distinct candidates; that doesn't show compute produces diversity. The fitness scaling is fine, but the diversity claim needs a different experiment.\n\nThird, there are no direct comparisons to target propagation or other related baselines, and the hyperparameter count is large. The authors acknowledge no code release, so reproducibility is limited. These are minor-to-moderate issues.\n\nOverall, the central claim—that separately searching representations can guide gradient learning to near-SGD performance—holds as a proof of concept. It deserves a serious referee, but the referee should push for ablations and a corrected diversity experiment. I'd lean conditional accept, and I'd bring it to a reading group focused on alternative training schemes.","headline":"A genuinely new proof-of-concept training scheme that evolves activation targets and regresses to them, landing within a few points of SGD on three benchmarks, but the operator recipe is never ablated and the diversity-scaling claim is weaker than advertised.","tokens_in":10180,"tokens_out":2698,"would_cite":false,"duration_ms":32868,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A two-phase training scheme—evolutionary search over activations, then gradient regression—approaches SGD on MNIST, CIFAR-10, and CIFAR-100.","keywords":["evolutionary search","representation space","decoupled training","backpropagation alternatives","gradient descent","neural network training","CIFAR","MNIST"],"falsifier":"Run the same search-then-regress pipeline on a fourth dataset (e.g., SVHN or a downsampled ImageNet) with operators unchanged; if accuracy drops far below SGD (well beyond the observed 1–3% gap) or if removing spatial smoothing or normalization makes searched representations unlearnable (training loss does not decrease), the claim that the mechanism generalizes is falsified.","tokens_in":9302,"feed_emoji":"🧬","tokens_out":3806,"duration_ms":43893,"temperature":0.7,"pith_summary":"The paper tries to establish that neural network training can be split into two decoupled phases: an evolutionary search that finds good intermediate activations (representations), and a gradient-based learning phase that trains the network to reproduce those representations. The authors show this split is viable: on MNIST, CIFAR-10, and CIFAR-100, networks trained by regressing to searched representations approach standard SGD accuracy without ever backpropagating classification gradients through the network body. They also show that the quality and diversity of searched representations improve with compute, and that the resulting models are qualitatively different from SGD-trained models. The significance is that it offers a concrete way to combine the exploratory power of search with the efficiency of gradient descent, potentially addressing gradient descent's tendency to settle for a single minimum.","feed_headline":"Search-then-regress trains nets to near-SGD accuracy","feed_subtitle":"Evolutionary search over activations, then gradient regression, matches SGD on MNIST, CIFAR-10, CIFAR-100.","key_machinery":"The key mechanism is the decoupling itself: evolutionary search over representation space (the layerwise activation tensors) is separated from gradient-based learning in parameter space. The search space is the set of activations at the outputs of three convolutional blocks plus the final logits; evolution proceeds layer by layer, fixing earlier optimized representations, and uses per-image top-k selection, crossover, channel-wise Gaussian mutation, repeated 3x3 average pooling (spatial smoothing) and per-sample normalization. The learning phase minimizes an MSE regression loss to those cached targets on each block and a KL divergence from the searched class distribution on the logits, with","core_discovery":"The central discovery is that the intermediate activations of a convolutional network can serve as a tractable search space. By evolving these activations layer-by-layer with crossover, channel-wise Gaussian mutation, spatial smoothing, and normalization—using classification cross-entropy as the fitness signal—the authors obtain fixed 'target' representations for every training example. A separate gradient-descent phase then trains network parameters to match these targets via MSE on each convolutional block plus a stop-gradient KL loss on the logits, so no classification gradient reaches the body. This pipeline achieves test accuracy within roughly one percentage point of SGD on all three b","pith_inferences":["If the operator set generalizes, the same decoupling could be applied to other architectures (e.g., transformers) where a tractable representation space exists, potentially enabling search-based exploration of attention maps or hidden states.","The authors' finding that skipping supervision on the first block helps with augmentation suggests that the searched representations for early layers may be harder to fit; one testable extension is to adaptively decide which layers to supervise based on learnability.","The reliance on cached representations means the method's effectiveness depends on search quality; an inference is that closing the loop (search, learn, re-search with the trained network) could close the remaining accuracy gap, as the paper's future-work section suggests.","Because search is performed independently per image, the method currently ignores inter-image dependencies; a possible extension is to search over batch-level representations to capture class structure."],"forward_implications":["If search over representations is a viable replacement for backprop through the body, then the network's convolutional layers can be trained without explicit classification gradients, only regression targets.","Performance improves with more search compute (larger populations, more generations), saturating, so search budget can be traded for accuracy.","The method produces models whose representations are distant from SGD's (cosine distance ~1), demonstrating multiple qualitatively different solutions exist.","Diversity of searched solutions grows with compute (effective number of solutions Neff), suggesting search could provide a source of diverse models.","The one-shot cached search shows even a single search pass suffices to approach SGD; iterative search-learning cycles are a clear next step."],"fun_headline_variants":["Evolve activations, then regress: near-SGD accuracy","Search activations, learn weights: matches SGD","Two-phase training: evolve then regress to match SGD","Activations as search space: evolution matches SGD learning","Decoupled training: evolution finds representations, gradients learn"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise, stated in Section 3.1, is that the specific evolutionary operators—particularly repeated spatial smoothing, which the paper calls 'crucial for learnability,' and normalization, which is needed 'for convergence of evolution'—produce representations that are both learnable by gradient descent and generalizing; no ablation is provided, so if these operators are brittle or dataset-specific, the central claim does not transfer beyond the three benchmarks","fun_headline_variants_meta":{"raw":{"variants":["Evolve activations, then regress: near-SGD accuracy","Search activations, learn weights: matches SGD","Two-phase training: evolve then regress to match SGD","Activations as search space: evolution matches SGD learning","Decoupled training: evolution finds representations, gradients learn"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000266,"raw_usage":{"total_tokens":1427,"prompt_tokens":701,"completion_tokens":726,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":445,"completion_tokens_details":{"reasoning_tokens":646}},"tokens_in":445,"tokens_out":726,"duration_ms":8967,"temperature":1.0,"reasoning_tokens":646,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T17:16:48.439822+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same search-then-regress pipeline on a fourth dataset (e.g., SVHN or a downsampled ImageNet) with operators unchanged; if accuracy drops far below SGD (well beyond the observed 1–3% gap) or if removing spatial smoothing or normalization makes searched representations unlearnable (training loss does not decrease), the claim that the mechanism generalizes is falsified.","supporting_citations":[],"review_version":1}