{"id":"bc7e3ae1-bdef-4f04-b47e-8f47d4571574","arxiv_id":"1909.02425","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"RESN uses a training-free random-sampling error estimate to guide an evolutionary search over LSTM architectures, matching or beating several baselines at about half the optimization time.","lead":"This paper introduces RESN, an evolutionary algorithm that searches for good recurrent neural network architectures without training each candidate, using a cheap random-weight error score instead. The authors report matching or beating standard methods on four time-series problems while cutting optimization time roughly in half.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MRS proxy validity is untested: without an ablation replacing MRS fitness with random noise, the reported gains could stem entirely from final Adam training.","rationale":"The core proposal is an EA whose only fitness signal is MRS pt; if pt does not rank architectures by their post-training error, the evolutionary loop is close to a random walk, and the paper's claimed advantage over training-based search rests on the final training step alone. The paper contains no direct measurement of this proxy: no correlation analysis, no scatter plot, and no ablation replacing MRS with a dummy fitness. The closest evidence is E1.i, but that baseline (GDET) selects candidates using the test set, so the comparison cannot validate MRS and is methodologically invalid as a training-based competitor. E1.ii and E2 compare against results from previous papers [45, 38], leaving uncontrolled differences in protocol, hardware, and budget. I would not reject the paper: the idea is plausible, the algorithm is clearly specified, the code is available, and the coal-flame result is suggestive. However, the central claim is conditional on the MRS proxy doing real work. A random-fitness control is the minimal experiment that would settle this. If random fitness matches RESN, the contribution reduces to 'train a final RNN with Adam,' and the state-of-the-art statements are not supported. The reader identified exactly this assumption as the weakest link, and I agree.","tokens_in":14099,"tokens_out":6330,"duration_ms":64751,"concrete_test":"Run the exact RESN pipeline from §4.2.1 on the sine or waste problem, but replace the MRS Evaluate step (Algorithm 2 line 8) with a fitness drawn uniformly at random, keeping the same population size, mutation operators, self-adaptation, and final Adam training. Repeat 30 independent runs and compare the final error distributions with a Wilcoxon test. If the random-fitness control is statistically indistinguishable from RESN, the MRS score contributes no selection signal and the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central mechanism is that the MRS probability pt (Algorithm 1, used as the fitness in Algorithm 2 line 8 and the objective in Eq. 1) predicts how well an architecture will perform after real training. This is never measured directly. The only same-protocol comparison against a training-based heuristic, E1.i, defines GDET by training each candidate for one epoch and evaluating it on the test set, so it is not a valid training-based baseline. The other main comparisons, E1.ii and E2, use published results from prior papers [45, 38] rather than rerunning competitors under identical conditions. Consequently, the reported gains could in principle arise from the final Adam/SGD training of an almost arbitrary architecture, with the evolutionary search contributing no selection signal. Section 4.2 explicitly defers MRS parameter tuning to future work and relies on [10], but no correlation between pt and post-training MAE/MSE is reported anywhere in the manuscript. Without such evidence, or an ablation that removes the MRS signal from the evolutionary loop, the conclusion in Section 6 that the MRS is a 'very useful heuristic' is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RESN, an evolutionary algorithm for RNN architecture optimization that uses the Mean Absolute Error Random Sampling (MRS) as a training-free fitness function. MRS estimates, for a given architecture, the probability that a randomly weighted instantiation achieves MAE below a threshold, and this probability is used to guide an (µ+λ) evolutionary search. Only the final selected architecture is trained with Adam (or SGD for the coal-flame experiment). The authors evaluate RESN on four prediction tasks: sine wave, waste generation, coal-flame intensity, and load forecasting, comparing against a gradient-based architecture optimizer (GDET), a random search, two published neuroevolutionary methods (EXALT and EXAMM), and expert-designed solutions. They report that RESN matches or improves on these baselines while halving optimization time, and conclude that MRS is a very useful heuristic.","tokens_in":14346,"tokens_out":6657,"duration_ms":60086,"significance":"RESN addresses a real bottleneck in neural architecture search: the cost of training every candidate. If the MRS proxy is valid, the approach offers a significant speedup and could make evolutionary architecture optimization practical for RNNs. The paper includes a public code repository and uses statistical tests on repeated runs. However, the empirical evidence for the central mechanism is incomplete: the MRS proxy's correlation with post-training error is never measured, the main same-protocol baseline leaks test-set information, and the 'state-of-the-art' claim is contradicted by the EXAMM numbers quoted in the same section. These gaps currently prevent the paper from fully supporting its conclusions.","major_comments":[{"comment":"The GDET baseline selects candidate architectures by computing MAE on the test set during optimization, which leaks test information and makes the comparison invalid for RQ1. A legitimate training-based baseline should use a validation split. The Wilcoxon p-value of 0.001 in Section 5 therefore cannot be interpreted as evidence that RESN matches or exceeds training-based optimization.","section":"Section 4.2.1, E1.i"},{"comment":"The text claims 'state-of-the-art performance' and 'ten times more accurate than the state-of-the-art' after comparing RESN to EXALT, but the same paragraph reports EXAMM averages of 0.001690 and 0.001601, which are both better than RESN's 0.005208. Since EXAMM is a neuroevolutionary method on the same dataset, the unqualified state-of-the-art claim is not supported by the paper's own numbers.","section":"Section 5, E2 (Table 8)"},{"comment":"The central premise that the MRS probability pt is a reliable predictor of post-training performance is never tested. The paper reports no correlation between pt and final MAE/MSE and no ablation that removes or randomizes the MRS signal in the evolutionary loop. Consequently, the conclusion in Section 6 that 'the MRS is a very useful heuristic' is unsupported; the reported error performance might be attributable to the evolutionary operators and the final training of the selected architecture.","section":"Section 3, Algorithms 1-2 and Section 6"},{"comment":"The comparison to expert-designed solutions is based on matching RESN's minimum over 30 runs (1.370) against the minima of the expert methods, whose means are not reported (NA). This is not a valid algorithm-level comparison; moreover, RESN's minimum is not better than the WKNNRW minimum (1.323). The claim that RESN is 'comparable to a human expert' needs a fairer comparison (e.g., means or distributions).","section":"Section 5, E4 (Table 9)"}],"minor_comments":[{"comment":"The abstract and full text should be consistent on the number of prediction problems; the first abstract says three, the full text says four.","section":"Abstract"},{"comment":"The GDET baseline's use of the test set for selection should be explicitly flagged as a methodological limitation, as the current text does not acknowledge the leakage.","section":"Section 4.2.1"},{"comment":"The table would be clearer if the source studies' full statistics were reported; the NA entries make it hard to judge the comparison.","section":"Table 9"},{"comment":"The code repository link should include a version/commit identifier and a description of the computational environment to support reproducibility.","section":"Section 5"},{"comment":"The MRS threshold (0.01) and weight distribution are fixed across datasets of different scales; the paper should discuss whether this is appropriate or whether the threshold should be normalized per problem.","section":"Section 2.2"}],"recommendation":"major_revision","confidential_remarks":"The paper builds on the authors' own prior MRS work and several baselines are their own previous methods; independent replication of the baselines would strengthen the claims. The state-of-the-art claim should be resolved with respect to EXAMM, whose results the authors themselves report as better than RESN's on the coal-flame problem."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: RESN is a genuine new combination—MRS as a training-free fitness inside an evolutionary search, with only the winner trained. The paper is clearly written, the code is public, and the experiments use 30 runs and Wilcoxon tests. The soft spots are real but not fatal: one baseline selects on test error, the MRS proxy is inherited rather than measured, and “state-of-the-art” appears once too often. The random-search baseline actually gives the central claim indirect support.\n\nThe main thing you should know: this is not just another neuroevolution paper. The authors promote their earlier MRS idea from an architecture-comparison tool to the fitness function in a (mu+lambda) EA. That is a legitimate step. Four problems, repeated runs, statistical tests, and a reimplementable algorithm description. Credit where due.\n\nWhere it gets softer. E1.i defines GDET by training each candidate one epoch and evaluating it on the test set. That is not a valid training-based baseline; it has seen test labels. It doesn’t sink the paper—if anything, it gives GDET an unfair advantage and RESN still wins—but it muddies the claim that RESN beats a purely training-based optimizer on equal terms. E2 compares to published EXALT and EXAMM numbers rather than re-running them; acceptable but weaker than a head-to-head. The authors note EXAMM’s MSE is 0.0016 while RESN is 0.0052, then still say “state-of-the-art” in the conclusions. The body is more honest; the conclusion overreaches.\n\nThe bigger question is whether MRS actually predicts trained performance. The paper does not report a correlation between pt and post-training MAE/MSE, and the MRS parameters come from the earlier paper without tuning. Strictly speaking, that is the load-bearing assumption. But the paper includes E1.iii, a random search baseline that trains candidates. On the waste problem, RESN (mean MAE 0.079) beats that random search (0.091), p=0.017. If the MRS signal were pure noise, RESN would be picking architectures at random and should not beat a selection-from-trained-candidates baseline. So the worry that gains stem entirely from final Adam training does not hold up for that dataset. Still, an ablation replacing MRS with random fitness, plus a simple correlation plot, would settle the proxy question directly.\n\nWho is this for? Anyone working on cheap architecture search or neuroevolution for RNNs. It deserves serious peer review and a request for revision, not a desk reject. Send it out; the authors need to fix the baseline, tone down the SOTA claim, and measure the proxy, but the core idea is sound and worth engaging.","headline":"A new and credible training-free fitness for RNN architecture search, with the central proxy assumption left only indirectly tested and one unfair baseline muddying the main comparison.","tokens_in":14846,"tokens_out":5407,"would_cite":true,"duration_ms":57414,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A training-free score can replace error-based fitness in RNN architecture search, matching trained search at roughly half the optimization time.","keywords":["neuroevolution","recurrent neural networks","architecture optimization","mean absolute error random sampling","training-free evaluation","evolutionary algorithm","LSTM","time series prediction"],"falsifier":"Rank a set of candidate architectures by the MRS probability $p_t$ and also by their test error after full training; if the rank correlation is close to zero on a fresh dataset, the search signal cannot be doing the claimed work. The paper reports no such direct check.","tokens_in":1490,"feed_emoji":"🧠","tokens_out":1902,"duration_ms":57734,"temperature":0.7,"pith_summary":"The paper claims that a recurrent neural network architecture can be chosen without training any candidate: sample random weight sets, record their mean absolute errors, fit a truncated normal distribution, and use the probability that the error falls below a threshold as the fitness in an evolutionary search. This method, called RESN, is reported to match training-based architecture optimization on sine-wave and waste-generation prediction, to beat a published neuroevolutionary baseline on coal-flame prediction, and to cut optimization time roughly in half. The reason to care is that architecture search is normally dominated by training cost; if a cheap untrained proxy can rank architectures correctly, much larger search spaces become feasible within the same compute budget.","feed_headline":"No-training shortcut halves RNN architecture search time","feed_subtitle":"An evolutionary search ranks LSTM stacks by random-weight error probability, training only the winner.","key_machinery":"The central object is the Mean Absolute Error Random Sampling (MRS) score $p_t$. For a candidate architecture, the algorithm generates several sets of random normally distributed weights, evaluates the network's MAE on the data for each set, fits a truncated normal distribution to those MAE values, and computes the probability that a sample lies below a threshold (0.01 in this paper). That probability serves as the fitness guiding an evolutionary search with binary tournament selection, cell-count and layer-count mutations, and self-adapting mutation parameters; because no gradient training happens during search, candidate evaluation is cheap. Once the search ends, the single best architecture is trained with Adam.","core_discovery":"RESN's central claim is that the MRS score $p_t$ — the estimated probability that a randomly weighted instance of an architecture has mean absolute error below a threshold — can stand in for actual training error during architecture search. The evolutionary algorithm maximizes $p_t$ over LSTM stack configurations, and only the single final winner is trained, using Adam. The reported consequence is that this training-free search matches or improves on training-based and neuroevolutionary baselines: mean MAE 0.105 versus 0.142 on sine prediction, median MAE 0.073 versus 0.073 on waste prediction, and average MSE 0.005208 versus 0.024432 on coal-flame prediction, while reducing optimization time by about half and producing results comparable to a human-expert-designed load-forecasting solution.","pith_inferences":["If the MRS proxy holds across problem types, the same trick could be applied to other training-heavy searches, such as convolutional network or transformer architecture search, by replacing MAE with a task-appropriate error metric.","The threshold and number of random samples are taken from the original MRS paper without tuning; varying them could change ranking quality, and finding a principled way to set them might make the proxy more reliable.","Since only the final solution is trained, the time savings of RESN should grow with dataset size — training-based search becomes prohibitive on large datasets while the MRS forward-pass cost stays comparatively small.","A direct testable extension is to combine the training-free proxy with learning-curve extrapolation: use MRS for early pruning of bad architectures and reserve expensive training for the survivors."],"forward_implications":["Architecture search can be run without training candidates, so the same time budget can explore far more architectures than training-based search.","The search phase and the training phase are cleanly separated: the search uses a training-free proxy, and only the final architecture incurs training cost.","On the waste-generation problem, RESN matches the short-training baseline in median MAE while roughly halving the total optimization time.","On coal-flame prediction, the paper reports an error about one order of magnitude lower than the EXALT neuroevolutionary baseline it compares against.","The evolutionary framework is not tied to the MRS fitness: the paper notes that the evaluation function can be swapped for any other fitness, so the search structure is reusable."],"supporting_citations":[{"why":"Introduces the MRS score $p_t$ and its parameters, which RESN adopts as its fitness function.","marker":"[10]"},{"why":"Provides the short-training baseline and run times that RESN matches on waste prediction.","marker":"[45]"},{"why":"Supplies the EXALT results and experimental design that RESN is compared against on coal-flame prediction.","marker":"[38]"},{"why":"Introduces the coal-flame dataset and the EXAMM results used for the qualitative comparison.","marker":"[37]"},{"why":"Defines the load-forecasting competition and the SVM baseline that RESN is compared against in the expert-design experiment.","marker":"[42]"},{"why":"Provides the Adam optimizer used to train the final selected architecture.","marker":"[40]"},{"why":"Introduces the waste-generation prediction problem and the earlier per-container baselines that RESN improves upon.","marker":"[44]"}],"fun_headline_variants":["Training-free search halves RNN architecture tuning time","Random-error sampling speeds neural architecture search","No-training RNN architecture search matches training methods","Evolutionary RNN search uses error probability not training","Random sampling predicts RNN performance, cuts search time"],"cache_read_input_tokens":17024,"weakest_assumption_plain":"The whole method rests on assuming that the probability of a random-weight version of a network scoring low error tells you how good that network will be after it is actually trained — a link the paper relies on but never directly measures.","fun_headline_variants_meta":{"raw":{"variants":["Training-free search halves RNN architecture tuning time","Random-error sampling speeds neural architecture search","No-training RNN architecture search matches training methods","Evolutionary RNN search uses error probability not training","Random sampling predicts RNN performance, cuts search time"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000677,"raw_usage":{"total_tokens":3022,"prompt_tokens":835,"completion_tokens":2187,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":451,"completion_tokens_details":{"reasoning_tokens":2116}},"tokens_in":451,"tokens_out":2187,"duration_ms":15127,"temperature":1.0,"reasoning_tokens":2116,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:10:25.942523+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rank a set of candidate architectures by the MRS probability $p_t$ and also by their test error after full training; if the rank correlation is close to zero on a fresh dataset, the search signal cannot be doing the claimed work. The paper reports no such direct check.","supporting_citations":[{"cited_title":"Camero, J","cited_arxiv_id":null,"evidence_quote":"Provides the short-training baseline and run times that RESN matches on waste prediction."},{"cited_title":"ElSaid, S","cited_arxiv_id":null,"evidence_quote":"Supplies the EXALT results and experimental design that RESN is compared against on coal-flame prediction."},{"cited_title":"Ororbia, A","cited_arxiv_id":null,"evidence_quote":"Introduces the coal-flame dataset and the EXAMM results used for the qualitative comparison."},{"cited_title":"Chen, M.-W","cited_arxiv_id":null,"evidence_quote":"Defines the load-forecasting competition and the SVM baseline that RESN is compared against in the expert-design experiment."},{"cited_title":"Ferrer, E","cited_arxiv_id":null,"evidence_quote":"Introduces the waste-generation prediction problem and the earlier per-container baselines that RESN improves upon."}],"review_version":1}