{"id":"6803c459-270c-419e-b06b-ef47d7481b12","arxiv_id":"2504.17346","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"A two-individual leader-follower genetic algorithm with adaptive layer sizes matches or slightly beats gradient descent on a small cat image benchmark, without code or error bars.","lead":"This paper introduces a genetic algorithm that trains small neural networks using two evolving parameter sets, a 'leader' and a 'follower', and tests it on a cat-versus-non-cat image task. The authors report slightly better test accuracy than a gradient-descent baseline on some architectures, but the evidence rests on a tiny dataset and a single random seed.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported superiority over gradient descent is an artifact of test-set cherry-picking: the 80% and 74% test accuracies are the best among 10 Pareto solutions, not the algorithm's own best-cost output, undermining the central claim.","rationale":"The reader's verdict (CONDITIONAL) correctly notes the fragility of a single seed and a 50-image test set. However, the more load-bearing problem is that the reported accuracies are not obtainable from the algorithm as specified: they are the best test accuracies among the 10 Pareto solutions, while the algorithm's own selection criterion (minimum training cost) yields measurably worse test accuracy (78% vs 80% in the three-layer case; 68% vs 74% in the two-layer case). This is a selection-on-the-test-set artifact. It affects the central claim directly and cannot be fixed by averaging over seeds unless a validation-based selection rule is introduced. Therefore the paper's headline comparison is currently unsupported. The method itself is coherent and the authors acknowledge overfitting, so a revised version with a proper validation split and multiple seeds might still demonstrate a modest advantage; that is why the verdict is 'REJECT' rather than 'UNVERDICTED'—the current evidence is invalid, not merely incomplete.","tokens_in":15439,"tokens_out":14511,"duration_ms":132322,"concrete_test":"Use the paper's own Table 2 and the selection rule stated in Algorithm 1: return the solution with minimum training cost (Leader unless a Follower has strictly lower cost). Recompute the reported train/test accuracies. For max_layer_dims=[12288,20,5,1], the returned model is [12288,20,3,1] with test 78% rather than 80%; for max_layer_dims=[12288,7,1], the returned model is [12288,7,1] with test 68% rather than 74%. If these recomputed numbers are correct, the abstract's claim 'higher performance' is not a property of the algorithm's output but of post-hoc test-set selection. To settle further, hold out 50 of the 209 training images as a validation set, select the best of the 10 solutions by lowest validation cost, and evaluate on the original test set; repeat with 10 seeds and verify whether the mean test accuracy still exceeds GD's.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim in Section 3 is that the proposed GA achieves 80% test accuracy on [12288,17,4,1] versus 74% for GD on the same architecture. But Algorithm 1 (line 17) returns the solution with minimum training cost. For the three-layer search (max_layer_dims=[12288,20,5,1], stop_cost=0.035), the minimum-cost solution in Table 2 is [12288,20,3,1] (cost 0.03391, test 78%), not the reported [12288,17,4,1] (cost 0.05926, test 80%). Similarly, for the two-layer search, the minimum-cost solution is [12288,7,1] (cost 0.01495, test 68%), not the reported 74% (which belongs to a higher-cost Follower solution). The paper never specifies a validation-based rule for choosing among the 10 generated solutions; the reported accuracies are the maximum test accuracies across the Pareto set, selected after observing the 50-image test set. With only 50 test images, choosing the best of 10 architectures inflates expected accuracy by several percentage points. The 6-point gap (80 vs 74) is approximately 3 images and is fully explained by this selection bias plus single-seed variance (random.seed(42)). Therefore the head-to-head comparison is not a fair test of the method's superiority.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a two-individual (Leader-Follower) genetic algorithm for training multilayer perceptrons, with self-adaptive hidden-layer dimensions. The method evolves two parameter sets through row-based crossover, Gaussian mutation, and Pareto-rank-based architecture selection, and is evaluated on the 209-image/50-image cat-vs-non-cat dataset from the Deep Learning Specialization. The paper claims that the proposed method outperforms gradient descent on two-layer and three-layer networks and discusses overfitting in the four-layer case.","tokens_in":15770,"tokens_out":4322,"duration_ms":42047,"significance":"The algorithmic description is unusually detailed, and the authors candidly acknowledge overfitting and run-to-run variability; the proposed mechanism is a reasonable incremental variant of (1+1)-ES combined with harmony-search-like architecture generation. However, the central empirical claim that the evolutionary method beats gradient descent is not established by the evidence in Table 2: the reported 80% test accuracy is the best test accuracy among ten Pareto solutions, obtained from a single seed on a 50-image test set. These issues are load-bearing for the main claim and require new experiments and a pre-specified model-selection rule to resolve.","major_comments":[{"comment":"The reported [12288,17,4,1] result with 80% test accuracy is not the solution selected by Algorithm 1. Algorithm 1 (line 17) returns the minimum-cost leader solution; for max_layer_dims=[12288,20,5,1] that solution is [12288,20,3,1] (cost 0.03391, test 78%), not [12288,17,4,1] (cost 0.05926, test 80%). Because the paper never specifies a validation-based selection rule, the 80% figure appears to be the maximum test accuracy over the ten Pareto solutions, chosen after observing the 50-image test set. This selection bias inflates the reported accuracy and invalidates the head-to-head comparison with gradient descent. Please report the test accuracy of the solution actually returned by the algorithm, or use a pre-registered validation selection rule, and show that the conclusion is unchanged.","section":"Section 3, Table 2"},{"comment":"The comparison rests on a single run with random.seed(42) and a 50-image test set, where 80% versus 74% is a difference of three images. The paper itself states that random mutations cause high run-to-run variability. Provide multiple seeds and, ideally, multiple train/test splits, with means and standard deviations (or confidence intervals) and a paired significance test; otherwise the 6-point test gap is within the noise of the experimental setup.","section":"Section 3, Table 2"},{"comment":"The abstract's headline comparison is apples-to-oranges: it contrasts a three-layer proposed network (80% test) with a four-layer gradient-descent network (80% test), while Section 3 claims superiority over gradient descent on the same architecture [12288,17,4,1] (80% vs 74%). These two claims use different comparison protocols, and the abstract's phrase 'higher performance' is not supported when the test accuracies are equal. Please unify the comparison protocol and ensure that every superiority claim uses the same architecture for both methods.","section":"Abstract and Section 3"},{"comment":"The mutation scale is set to 0.0008 in Algorithm 1 (line 8) but described as 0.008 in the text and in Figure 4. This discrepancy must be resolved, since mutation_scale is a free parameter of the method and directly affects the reported results.","section":"Sections 2.3 and 2.7, Algorithm 1"}],"minor_comments":[{"comment":"The sentence 'the propsed method outperforms gradient' is inaccurate as stated, because for [12288,17,4,1] the proposed method's training accuracy (99.04%) is lower than gradient descent's (99.52%); the claim should be limited to test accuracy.","section":"Section 3, first paragraph"},{"comment":"CR, PAR, and the mutation parameters are fixed ad hoc values, and no sensitivity analysis is reported. At minimum, state how these hyperparameters were chosen and whether the qualitative comparison is robust to reasonable variations.","section":"Section 2.5, Function 1"},{"comment":"Reference [17] is the paper's own arXiv entry, cited as the source for the (1+1)-ES approach it differs from. This is circular; a standard (1+1)-ES reference should be used instead.","section":"Introduction, Reference [17]"},{"comment":"The statement that source code is 'available upon request' limits reproducibility; please deposit the Jupyter Notebook in a permanent public repository with a DOI.","section":"Data Availability Statement"},{"comment":"The table entries mix architecture vectors with cost values inside the same bracket notation, and Figure 4's mutation-rate curve should specify whether the rate starts at 0.9 and decays to 0.1 or vice versa; the current text and Algorithm 1 are ambiguous about the direction of the decay.","section":"Table 2 and Figure 4"}],"recommendation":"major_revision","confidential_remarks":"This is a small empirical study with a single dataset, no archived code, and a self-citation as reference [17]. The referee comments identify load-bearing problems with model selection and statistical evidence; these are fixable with additional experiments, so I recommend major revision rather than rejection. The abstract's mismatch between the three-layer versus four-layer comparison and the same-architecture comparison in Section 3 should also receive editorial attention."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a readable description of a small evolutionary trick, but the central empirical claim doesn't survive contact with the paper's own Table 2.\n\nWhat's new: the dual-individual scheme (leader/follower roles, even-odd layer allocation, roulette-weighted architecture generation) is a modest variant of (1+1)-ES and Harmony Search. The paper gives enough detail to reimplement: Algorithm 1-3, Functions 1-4, and a worked merging example. That's real credit. The authors also openly admit the resemblance to Harmony Search and the overfitting on the four-layer case. The citation pattern is fine, with relevant GA/ES/Harmony Search references.\n\nSoft spots: the comparative claim is load-bearing and unsupported. Algorithm 1's return is the minimum training-cost solution (line 17). In the three-layer run, the min-cost architecture is [12288,20,3,1] with 78% test, not the headline [12288,17,4,1] with 80%. Same for two-layer: min-cost [12288,7,1] gives 68%, while the reported 74% belongs to a higher-cost Follower solution. So the 80% and 74% are the best test accuracies among the ten output architectures, selected after looking at the 50-image test set. That is test-set cherry-picking. On 50 images, 80 vs 74 is three images; the selection effect plus single-seed (random.seed(42)) variance fully explains the gap. The abstract also compares a 3-layer GA to a 4-layer GD baseline, apples-to-oranges. These are not minor nits; they undermine the paper's stated contribution.\n\nAlso minor: no code or data shipped (only \"available upon request\"), and the many free parameters (CR, PAR, mutation scale, stop_cost, max_layer_dims) get no sensitivity analysis.\n\nThe authors are not hiding their limitations: they discuss variance and overfitting and propose regularization as future work. But the abstract and Section 3 frame the result as superior, which the evidence doesn't support.\n\nWho this is for: someone working in neuroevolution might glance at the leader-follower idea, but they'd need to re-run it properly. This is not a serious contribution as it stands.\n\nRecommendation: desk reject; if the editor is forgiving, a major-revision route could work if the authors redo the evaluation with multiple seeds, a validation-based selection rule among the Pareto set, and same-architecture baselines. But the current version does not deserve referee time because the central claim is an artifact.","headline":"The leader-follower GA variant is coherent and honestly described, but the headline accuracy gap is an artifact of picking the best test result from a ten-architecture Pareto set on a 50-image test set.","tokens_in":16280,"tokens_out":3519,"would_cite":false,"duration_ms":31557,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A two-individual genetic algorithm with a Leader and a Follower can train small cat-versus-non-cat networks to match or beat gradient descent on test accuracy for shallow architectures, while overfitting on deeper ones.","keywords":["Genetic Algorithm","dual-individual evolution","Leader-Follower optimization","neural network training","binary image classification","Pareto dominance","self-adaptive layer dimensions","Gaussian mutation"],"falsifier":"Re-run the [12288, 17, 4, 1] comparison many times with fresh random seeds and new 209/50 train/test splits and compare mean test accuracy; since the reported 80% versus 74% is just 40 versus 37 correct images out of 50, the claim stands only if the evolutionary method's mean test accuracy stays above gradient descent's across those runs.","tokens_in":15184,"feed_emoji":"🧬","tokens_out":8310,"duration_ms":68923,"temperature":0.7,"pith_summary":"This paper proposes a genetic algorithm that keeps exactly two evolving candidate solutions, labeled Leader and Follower, and uses their recombination and mutation to train small feed-forward neural networks for binary image classification. The Leader is the current best-cost solution and drives exploitation, while the Follower preserves diversity and can replace the Leader when it improves, so the two roles swap throughout optimization. The method also evolves hidden-layer widths inside fixed bounds, removing manual architecture choice. On the paper's cat-versus-non-cat benchmark, the strongest reported result is a three-layer network reaching 99.04% training and 80% test accuracy with cost 0.06, compared with 99.52% training and 74% test accuracy for gradient descent at cost 0.0378; on a four-layer network the method overfits and falls to 76% test accuracy versus 80% for gradient descent. A sympathetic reading is that evolutionary search with two individuals can match or beat gradient descent on shallow networks while carrying an overfitting risk on deeper ones.","feed_headline":"Two-agent genetic algorithm edges out gradient descent on cat test","feed_subtitle":"On a three-layer network it scores 80% test accuracy to gradient descent's 74%, though on just 50 test images.","key_machinery":"The load-bearing object is the Leader-Follower pair: two parameter dictionaries for weights and biases plus two small lists of candidate layer architectures, with the roles assigned by Pareto-dominance rank and cost, and swapped when the Follower's best solution beats the Leader's. Each generation performs row-wise multi-point crossover on weight matrices, arithmetic crossover for biases, Gaussian mutation with a mutation rate that decays from 0.9 to 0.1 over 20,000 iterations, and architecture generation that resembles Harmony Search with a 0.9 consideration rate and a 0.3 pitch-adjustment rate, weighted by cost through roulette-wheel selection. An anabolism step merges leader, follower, and offspring solutions, trims parameter matrices to the chosen layer widths, and fills any gaps with fresh architectures so both agents keep five candidates. This machinery is what lets the method search weights and architectures simultaneously with only two persistent individuals.","core_discovery":"The central discovery the authors claim is that a population of two role-labeled neural-network parameter sets is enough to train a multilayer perceptron: crossover between Leader and Follower, with Gaussian mutation and Pareto-cost ranking of alternative layer architectures, produces weights that classify a small image set as accurately as gradient descent on two- and three-layer networks, and sometimes more accurately on held-out data. The headline numbers are a three-layer network [12288, 17, 4, 1] with 99.04% training and 80% test accuracy versus the same architecture trained by gradient descent at 99.52% training and 74% test, and a two-layer network [12288, 7, 1] at 100% and 74% versus 100% and 72%. The authors also report that on a four-layer network [12288, 20, 7, 5, 1] the genetic method reaches 99.52% training accuracy but only 76% test accuracy, below gradient descent's 98% and 80%, which they attribute to overfitting from aggressive joint optimization of weights and architecture. They interpret the layer-dimension results as evidence that the algorithm can refine hidden-layer sizes on its own, because the evolved configurations settle onto the same values as the optimized final architectures.","pith_inferences":["Editorial inference: a repeated-seed study is the natural next test; with only 50 test images, the 80%-versus-74% gap is three images, and the paper itself notes run-to-run variance from random mutations, so the reported edge could shrink or reverse under repeated runs.","Editorial inference: the role-swapping design suggests a cheap ensemble, namely averaging the Leader and Follower predictions at the end of training, since the two agents are maintained as distinct solutions and may capture different regions of the loss surface; this is not tested in the paper.","Editorial inference: the mutation schedule, Gaussian noise with fixed scale 0.008 and a rate decaying from 0.9 to 0.1, may alone account for much of the exploration; comparing the dual-individual design against a single-individual evolution strategy with the same mutation budget would isolate the contribution of the Leader-Follower roles."],"forward_implications":["If the central claim holds, evolutionary training with two individuals can replace manual architecture search for small shallow classifiers: the same run that optimizes weights also picks hidden-layer sizes within the declared maximum.","On the three-layer benchmark the genetic method's test accuracy is 80% versus 74% for gradient descent on the same architecture, implying a 6-point generalization gain on a 50-image test set.","The paper's own four-layer result is worse on test accuracy (76% versus 80%) despite higher training accuracy, so the claim of advantage is limited to shallow networks unless regularization is added.","The reported cost values, 0.06 for the genetic three-layer solution versus 0.0378 for gradient descent, show the evolutionary solution reaches comparable or better test accuracy at a higher training cost.","The dynamic role-swapping between Leader and Follower, with the Follower allowed to replace the Leader, is presented as the mechanism that balances exploitation and exploration without eliminating either candidate."],"supporting_citations":[{"why":"Supplies the 209-training and 50-test cat-versus-non-cat image benchmark on which every accuracy comparison in the paper is computed.","marker":"[18]"},{"why":"Provides the Harmony Search consideration and pitch-adjustment concepts that the architecture-generation step adapts for proposing new hidden-layer widths.","marker":"[19]"},{"why":"Gives the roulette-wheel weighting formula used to select among candidate layer architectures with probabilities based on cost.","marker":"[22]"},{"why":"Defines Pareto dominance in multi-objective optimization, which the paper uses to rank and filter candidate solutions.","marker":"[24,25]"},{"why":"Supplies the standard NSGA-II Pareto-ranking approach against which the paper's simplified dominance-count rank is compared.","marker":"[26]"},{"why":"Defines the single-parent (1+1)-Evolution Strategy baseline from which the two-individual design departs.","marker":"[14–16]"}],"fun_headline_variants":["Two-individual genetic algorithm rivals gradient descent on small nets","Genetic algorithm with just two agents trains neural nets on par","Leader-follower crossover optimizes MLPs without manual tuning","Dual-individual GA matches gradient descent on cat classification","Pair of neural nets evolve to edge out gradient descent on test"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison rests on a single random seed and a single train/test split of 209 training and 50 test images, so the reported 6-point test-accuracy edge could be noise rather than a systematic property of the method.","fun_headline_variants_meta":{"raw":{"variants":["Two-individual genetic algorithm rivals gradient descent on small nets","Genetic algorithm with just two agents trains neural nets on par","Leader-follower crossover optimizes MLPs without manual tuning","Dual-individual GA matches gradient descent on cat classification","Pair of neural nets evolve to edge out gradient descent on test"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001055,"raw_usage":{"total_tokens":4488,"prompt_tokens":1061,"completion_tokens":3427,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":677,"completion_tokens_details":{"reasoning_tokens":3345}},"tokens_in":677,"tokens_out":3427,"duration_ms":21067,"temperature":1.0,"reasoning_tokens":3345,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:42:06.151492+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the [12288, 17, 4, 1] comparison many times with fresh random seeds and new 209/50 train/test splits and compare mean test accuracy; since the reported 80% versus 74% is just 40 versus 37 correct images out of 50, the claim stands only if the evolutionary method's mean test accuracy stays above gradient descent's across those runs.","supporting_citations":[{"cited_title":"https://www.coursera.org/specializations/deep - learning (accessed April 7, 2025)","cited_arxiv_id":null,"evidence_quote":"Supplies the 209-training and 50-test cat-versus-non-cat image benchmark on which every accuracy comparison in the paper is computed."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Harmony Search consideration and pitch-adjustment concepts that the architecture-generation step adapts for proposing new hidden-layer widths."},{"cited_title":"Zamli, H.S","cited_arxiv_id":null,"evidence_quote":"Gives the roulette-wheel weighting formula used to select among candidate layer architectures with probabilities based on cost."}],"review_version":1}