{"id":"977bda40-e7e4-4405-bfb3-7eb30cfa1ee8","arxiv_id":"2411.15290","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"GreenMachine evolves new zero-cost proxy formulas that rank untrained neural networks by expected accuracy, reaching a best Kendall correlation of 0.89 on NATS-Bench size search space with CIFAR-10.","lead":"GreenMachine uses an evolutionary algorithm to invent formulas that rank neural network designs before any training happens. On the NATS-Bench benchmark, some invented formulas match true accuracy better than existing zero-cost proxies, which could make architecture search faster and more energy-efficient.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GreenMachine's superiority over baselines appears only on stratified validation sets; on naturally sampled sets it does not beat EZNAS or Synflow, so the central claim overreaches.","rationale":"The reader identified a plausible concern about the small 120-network fitness sample and whether it generalizes. However, Section 3.2.1 does specify the sampling strategy: stratified sampling with 20 networks per (search space, dataset) combination, totaling 120. So the reader's weakest assumption is partly addressed. The more load-bearing issue is the distribution mismatch between the stratified evaluation protocol (Table 4) and the natural accuracy distribution of the search spaces (Figure 4). The paper's own non-stratified results (Table 3) show that GreenMachine proxies do not outperform existing baselines on natural samples: on TSS, EZNAS wins all three datasets; on SSS, GreenMachine wins only one of three. This means the central claim, as stated in the abstract and conclusion, is only true under a contrived evaluation setting that requires access to ground-truth accuracies to construct the validation sample. In practice, a NAS algorithm samples architectures from the natural distribution, where the proxies show no advantage. The paper could be made publishable by re-framing the claim to the stratified setting, adding natural-distribution results, comparing with recent baselines ParZC and AZ-NAS, and providing a single proxy or selection rule. Therefore, I recommend keeping the conditional verdict, but the conditions should explicitly include demonstrating transfer to natural-distribution ranking or clearly limiting the claim to the stratified scenario.","tokens_in":83,"tokens_out":6964,"duration_ms":129867,"concrete_test":"Re-run the validation of GM-1, GM-2, GM-3, EZNAS, Synflow, and ZiCo on 30 sets of 150 networks sampled proportionally to the true accuracy histograms in Figure 4 for each (search space, dataset) pair, and also on the set of all remaining NATS-Bench networks not used in fitness. If GreenMachine's Kendall correlations do not exceed EZNAS/Synflow on these natural-distribution sets, the central claim is not supported outside the stratified protocol. Additionally, report which single GreenMachine proxy (or selection rule) a user should apply per dataset, since no single proxy matches the abstract's headline numbers.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that GreenMachine's proxies outperform existing zero-cost proxies at distinguishing low- from high-performing networks is supported only by Table 4, where validation sets are stratified on ground-truth accuracy. Both the fitness samples (Section 3.2.1: 120 networks, 20 per search space/dataset, stratified into five accuracy bins) and the validation samples are deliberately balanced across accuracy bins. This is not the distribution encountered when ranking a NATS-Bench search space in practice: Figure 4 shows the actual accuracy distribution is heavily skewed toward high-performing networks. On the non-stratified validation sets (Table 3), which approximate the natural distribution, GreenMachine does not consistently beat baselines: on TSS, EZNAS is best on all three datasets; on SSS, GreenMachine wins only CIFAR-100 (GM-3: 66.8 vs Synflow 59.0) and loses CIFAR-10 (Synflow 76.8 vs GM-3 75.6) and ImageNet16-120 (Synflow 79.5 vs GM-2 78.0). Thus the reported superiority is an artifact of evaluating on a distribution that oversamples low-performing networks, which a NAS practitioner does not have access to because ground-truth labels are unknown before evaluation. The claim would require evidence that the stratified-sample correlation transfers to natural samples or to downstream NAS ranking quality. Without that, the observed gains are consistent with overfitting to the stratified evaluation protocol.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"GreenMachine proposes an evolutionary (structured grammatical evolution) approach to automatically synthesize zero-cost proxies for neural architecture performance prediction. Proxies are scored by Kendall's tau between proxy output and test accuracy on samples from NATS-Bench (TSS and SSS; CIFAR-10, CIFAR-100, ImageNet16-120). The fitness function is the sum of absolute Kendall correlations over six search-space/dataset combinations. The paper reports that on stratified validation sets, the discovered proxies outperform existing zero-cost proxies, with headline Kendall correlations of 0.89 (SSS CIFAR-10), 0.77 (SSS CIFAR-100), 0.78 (TSS CIFAR-10), and 0.71 (TSS CIFAR-100). It also reports results on non-stratified validation sets, where the advantage is much weaker or reversed. The manuscript concludes that the discovered proxies perform better than existing methods when distinguishing between low- and high-performing networks.","tokens_in":16074,"tokens_out":6378,"duration_ms":63594,"significance":"If the central claim survives scrutiny, the paper offers a practical way to automate the design of training-free NAS proxies and would strengthen the evidence that evolved, parameter-free formulas can rival hand-designed zero-cost proxies. Positive features of the manuscript are that the code is made publicly available, the evolved formulas contain no fitted numeric constants, and the validation sets appear to be disjoint from the networks used during evolution. The significance is limited, however, by the fact that all experiments are confined to NATS-Bench and by the evaluation-protocol issues discussed below, which currently prevent the abstract's headline claim from being attributed to a single, well-specified GreenMachine solution.","major_comments":[{"comment":"The headline correlations are not produced by a single GreenMachine proxy. In Table 4, the 0.89 on SSS CIFAR-10 belongs to GM-3, the 0.77 on SSS CIFAR-100 belongs to GM-2, and the 0.78 and 0.71 on TSS CIFAR-10 and CIFAR-100 belong to GM-1. The manuscript does not explain how GM-1, GM-2, and GM-3 were selected after the five evolutionary runs (best per run, best per benchmark, or best per dataset). As written, the abstract and conclusion attribute these numbers to 'our approach' or 'the zero-cost proxies discovered by our approach', which invites post-hoc selection. Please report the full 3x6 correlation matrix for each of GM-1, GM-2, and GM-3, and either specify an a-priori selection rule or explicitly reframe the claim as 'for each benchmark/dataset a different evolved proxy can be chosen'.","section":"Abstract and Section 4, Tables 3 and 4"},{"comment":"The claimed superiority is established only under stratified validation sampling, which is not the distribution a NAS practitioner encounters. On the non-stratified sets (Table 3), EZNAS is better than all GreenMachine proxies on TSS for all three datasets (e.g., 61.1 vs 50.2 on CIFAR-10), and Synflow beats GM-3 on SSS CIFAR-10 (76.8 vs 75.6) and beats GM-2 on SSS ImageNet16-120 (79.5 vs 78.0). The abstract qualifies the results as being on the stratified strategy, but the conclusion generalizes to 'distinguishing between low- and high-performing networks' without noting that this capability is demonstrated only when the validation set is deliberately balanced across accuracy bins. Since test-accuracy stratification uses ground-truth labels that are unknown at proxy-deployment time, the paper should either restrict the central claim to the stratified diagnostic protocol or provide transfer evidence, such as top-k rank agreement, search-space-level ranking performance, or downstream NAS search quality on naturally sampled networks.","section":"Section 4, Tables 3 and 4; Section 5"},{"comment":"The fitness-evaluation sampling is not fully specified, and the description of the validation sets is ambiguous. Section 3.2.1 describes stratifying the search space into five accuracy bins and sampling 20 networks per dataset per search space, which yields 120 networks (consistent with Table 1), but it does not state how those 120 networks are split among the five bins, whether the same 120 networks are used for every generation/run, or what random seeds are used. Section 4 says validation is performed on '4,500 new networks' and also says '30 sets of 150 networks from each search space and dataset'; these two statements are arithmetically inconsistent unless '4,500' refers to total unique networks and the per-condition sets are overlapping. The paper should state explicitly how the 120 fitness networks are sampled, confirm that the validation sets exclude all 120 fitness networks, and report the exact number of unique validation networks per search-space/dataset condition.","section":"Section 3.2.1 and Section 3.2.5"}],"minor_comments":[{"comment":"The text says 'see Tab. 3' when discussing the results obtained with stratified sampling, but those results are in Table 4; please correct the cross-reference.","section":"Section 4, last paragraph before Section 5"},{"comment":"The terminal symbols used in the evolved formulas, such as pass_noise_wt, pass_perturbation_fwd_output, and random_grad, are not defined in the main text or in the supplementary material. Please provide a complete list of terminals and their definitions alongside the feature-extraction description.","section":"Section 3.2.3 and Figure 7 / Table 6"},{"comment":"The feature-extraction description says 20 features are extracted per layer but does not enumerate them. Please list the 20 features explicitly, or state that the full list is in the supplementary material if it is.","section":"Section 3.2.3"},{"comment":"The phrase '4,500 new networks' is ambiguous, as noted above; please clarify whether this is the total number of unique validation networks or the number of sets times set size, and give per-condition counts.","section":"Section 4"},{"comment":"The paper reports means and standard deviations over 30 sets but no significance tests; for key comparisons, such as GM-3 (88.8 ± 0.9) versus EZNAS (82.8 ± 0.9) on SSS CIFAR-10 in Table 4, please report a test or a confidence interval to support the claim of superiority.","section":"Tables 3 and 4"}],"recommendation":"major_revision","confidential_remarks":"The empirical protocol is mostly sound, but the central claim is currently overreaching because the headline results come from different evolved proxies and only on stratified validation sets. The authors should be asked to clarify the selection of GM-1/GM-2/GM-3, to justify or qualify the stratified-only claim, and to add a natural-distribution or downstream-NAS evaluation. The scope is appropriate for a cs.LG venue, but the presentation needs to match the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a real, if incremental, advance over EZNAS. The authors evolve zero-cost proxies with a structured grammatical evolution representation, richer feature extraction, and a stratified fitness sample, then validate the formulas on 30 held-out sets of 150 new networks. The SSS results are genuinely strong (GM-2 reaches 0.856 on ImageNet16-120 stratified, beating ZiCo's 0.809) and the standard deviations across repeated samples are small. The formulas contain no fitted numeric constants, so the correlations are not circular in the obvious sense. That part of the paper is solid.\n\nThe soft spots are proportional. First, the abstract and conclusion gather the best numbers from three different evolved formulas: GM-3 for SSS CIFAR-10, GM-2 for SSS CIFAR-100 and IN16-120, GM-1 for TSS CIFAR-10/100. No single proxy delivers the touted performance, and the paper gives no principled rule for picking among them. If a user wants one ranking function, this matters. Second, on naturally sampled (non-stratified) sets the GreenMachine proxies do not consistently beat Synflow or EZNAS; the conclusion's sentence claiming they 'perform better than existing zero-cost proxy methods' is too broad. The abstract is more careful, saying outperformance on the stratified strategy, but the framing in the conclusion overreaches. Third, the paper cites ParZC and AZ-NAS in the introduction but omits them from the experiments, so the comparison is incomplete. Fourth, for a paper about energy efficiency, there is no runtime or energy measurement of the proxies themselves; some evolved formulas use backward passes, gradients, and many operations and may be more expensive than Synflow. Finally, the fitness is computed on 120 stratified networks; validation is on held-out networks, so this isn't circular, but the selection of the three best solutions across runs and datasets does introduce a selection bias that the paper doesn't address.\n\nOn the stress-test note: I think the complaint that 'superiority appears only on stratified sets' is partly an overstatement, because the paper explicitly states its main result is on the stratified sampling strategy. The real problem is the practical relevance of that strategy: a NAS practitioner ranking a natural sample won't see a distribution that is balanced across accuracy bins, so the strong stratified correlation may not transfer. The authors need to either show transfer to natural samples, or reposition the claim as a diagnostic evaluation of discrimination ability across the performance range.\n\nWho this is for: people working on zero-cost proxies, evolutionary NAS, and reproducible proxy evaluation. It should definitely get peer review, not a desk reject. I'd send it back for major revision with the requests above. I wouldn't cite it as a SOTA result until a single proxy or selection rule is provided.","headline":"Evolved zero-cost proxies with strong stratified correlations, but the headline numbers come from three different formulas and don't hold on natural samples; worth reviewing but needs a major revision.","tokens_in":16615,"tokens_out":4675,"would_cite":false,"duration_ms":46088,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GreenMachine evolves zero-cost proxies that rank untrained neural networks more accurately than existing training-free methods on NATS-Bench, reaching a Kendall correlation of 0.89 on CIFAR-10.","keywords":["neural architecture search","zero-cost proxies","genetic programming","structured grammatical evolution","training-free NAS","NATS-Bench","Kendall correlation","stratified sampling"],"falsifier":"Score a new search space such as NAS-Bench-201 with the best GreenMachine proxy and compute the Kendall correlation against true test accuracy; if it drops to the level of a random score or below existing zero-cost proxies, the generalization claim fails. A second test is to re-run evolution with 120 randomly sampled architectures instead of stratified ones and check whether the stratified validation correlations collapse.","tokens_in":15567,"feed_emoji":"🧬","tokens_out":7748,"duration_ms":68940,"temperature":0.7,"pith_summary":"Neural architecture search normally requires training candidate networks to know which ones are good, making it slow and energy-hungry. Zero-cost proxies try to rank networks before training by scoring their weights and activations, but hand-designed proxies often fail when the candidate set includes poor networks as well as good ones. GreenMachine evolves proxy formulas with grammar-based genetic programming, using stratified samples of the NATS-Bench search space so that a proxy only scores well if it separates low-performing from high-performing architectures. On stratified validation sets, its best evolved proxies reach Kendall correlations of 0.89 on CIFAR-10 and 0.77 on CIFAR-100 for the size search space, and 0.78 and 0.71 for the topology search space, beating every compared zero-cost proxy. This points toward training-free architecture ranking that is cheaper and works across the full performance range.","feed_headline":"Evolved proxies rank untrained networks at 0.89 Kendall tau","feed_subtitle":"Automated search discovers zero-cost formulas that separate good from bad architectures on NATS-Bench.","key_machinery":"The search engine is grammar-based genetic programming using Structured Grammatical Evolution, which represents each proxy as a syntax tree over per-layer statistics collected from a randomly initialized network: weights and gradients, plus forward and backward activations after a batch of random data, a real data batch, or a noisy batch. Each tree maps those tensors through arithmetic, norm, activation, comparison, and similarity operations into a single score, and the network score is the mean over layers. Fitness is the sum of absolute Kendall $\\tau$ correlations across search spaces and datasets. The other load-bearing component is stratified sampling: the search space is divided into five accuracy bins and the evaluation set is built to represent all bins, so proxies are rewarded for ordering poor networks below good ones instead of exploiting the natural skew toward high accuracy.","core_discovery":"The paper's central claim is that zero-cost proxies discovered by evolution outperform existing zero-cost proxy methods specifically when the task is to distinguish low- and high-performing networks. The authors demonstrate this by evaluating the evolved formulas on NATS-Bench using stratified sampling that covers the full accuracy range. Their best solutions achieve mean absolute Kendall $\\tau$ correlations of 0.89 on CIFAR-10 and 0.77 on CIFAR-100 in the size search space (NATS-Bench-SSS), and 0.78 on CIFAR-10 and 0.71 on CIFAR-100 in the topology search space (NATS-Bench-TSS), exceeding all compared baselines on stratified validation sets. On randomly sampled validation sets, where most networks are high-performing and similar to one another, the advantage largely disappears; the authors interpret this as showing that stratified evaluation is what makes the ranking capability visible and useful.","pith_inferences":["Beyond the reported results, the same evolutionary pipeline should be tested on other search spaces and datasets, since the paper validates only on NATS-Bench.","An implicit methodological point is that the 120-network fitness sample may itself need to be stratified; the paper describes stratification for evaluation but does not specify how the 120 training architectures are chosen, so the contribution of sampling versus the evolved formulas is not cleanly separated.","A stronger downstream test would run an actual architecture search with the evolved proxy and compare the final networks' accuracy with searches guided by existing proxies, rather than only comparing ranking correlations."],"forward_implications":["Training-free architecture search can become substantially more reliable on broad search spaces, since the evolved proxies rank unseen networks across the full accuracy range.","Automatic proxy discovery is a viable alternative to hand-designed proxies: the best GreenMachine solutions beat all compared baselines on stratified NATS-Bench sets.","Because scoring does not require training networks, architecture search with such proxies consumes much less energy, which is the sustainability motivation of the paper.","Evaluation protocol matters: comparisons on randomly sampled networks can mask a proxy's ability to separate good and bad architectures, so stratified correlation should be reported alongside random-sampling results."],"supporting_citations":[{"why":"Supplies the NATS-Bench benchmark with precomputed accuracies for the SSS and TSS search spaces used to evolve and validate proxies.","marker":"[12]"},{"why":"Provides the Structured Grammatical Evolution representation that maps integer genotypes to proxy expression trees.","marker":"[25]"},{"why":"EZNAS, the closest prior work, is the evolutionary zero-cost-proxy baseline whose approach GreenMachine extends and compares against.","marker":"[2]"},{"why":"Synflow, a data-independent pruning-based proxy, is one of the state-of-the-art baselines on the comparison tables.","marker":"[35]"},{"why":"NASWOT, the training-free activation-pattern proxy, is a baseline on all search spaces.","marker":"[27]"},{"why":"Gradnorm is a gradient-norm proxy baseline included in the empirical comparison.","marker":"[1]"},{"why":"ZiCo is a zero-shot proxy baseline based on gradient coefficient of variation, compared on stratified and non-stratified sets.","marker":"[22]"},{"why":"TE-NAS is a theoretical-inspired zero-shot ranking baseline used in the comparison.","marker":"[6]"},{"why":"Zen-NAS is a zero-shot expressivity-score baseline from the comparison.","marker":"[23]"}],"fun_headline_variants":["Evolved zero-cost proxies hit 0.89 Kendall tau on CIFAR-10","GreenMachine evolves cheap rankers for untrained networks","Automatic zero-cost proxies beat handcrafted NAS baselines","Rank DNNs without training: evolved formulas score 0.89 tau","Evolved proxies separate good and bad architectures at zero cost"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that fitness computed on only 120 sampled architectures is a reliable guide to performance across the full search space, and the paper does not specify how those 120 networks are sampled; if that sample is unrepresentative, the evolved proxies may not generalize beyond the benchmark.","fun_headline_variants_meta":{"raw":{"variants":["Evolved zero-cost proxies hit 0.89 Kendall tau on CIFAR-10","GreenMachine evolves cheap rankers for untrained networks","Automatic zero-cost proxies beat handcrafted NAS baselines","Rank DNNs without training: evolved formulas score 0.89 tau","Evolved proxies separate good and bad architectures at zero cost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000211,"raw_usage":{"total_tokens":1449,"prompt_tokens":1016,"completion_tokens":433,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":632,"completion_tokens_details":{"reasoning_tokens":341}},"tokens_in":632,"tokens_out":433,"duration_ms":4888,"temperature":1.0,"reasoning_tokens":341,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:31:21.378958+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Score a new search space such as NAS-Bench-201 with the best GreenMachine proxy and compute the Kendall correlation against true test accuracy; if it drops to the level of a random score or below existing zero-cost proxies, the generalization claim fails. A second test is to re-run evolution with 120 randomly sampled architectures instead of stratified ones and check whether the stratified validation correlations collapse.","supporting_citations":[{"cited_title":"NATS-Bench: Benchmarking NAS algorithms for architecture topology and size","cited_arxiv_id":null,"evidence_quote":"Supplies the NATS-Bench benchmark with precomputed accuracies for the SSS and TSS search spaces used to evolve and validate proxies."},{"cited_title":"Pereira, Ernesto Costa, and Penousal Machado","cited_arxiv_id":null,"evidence_quote":"Provides the Structured Grammatical Evolution representation that maps integer genotypes to proxy expression trees."},{"cited_title":"EZNAS: evolving zero-cost proxies for neural ar- chitecture scoring","cited_arxiv_id":null,"evidence_quote":"EZNAS, the closest prior work, is the evolutionary zero-cost-proxy baseline whose approach GreenMachine extends and compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Synflow, a data-independent pruning-based proxy, is one of the state-of-the-art baselines on the comparison tables."},{"cited_title":"Neural architecture search without training","cited_arxiv_id":null,"evidence_quote":"NASWOT, the training-free activation-pattern proxy, is a baseline on all search spaces."},{"cited_title":"Abdelfattah, Abhinav Mehrotra, Lukasz Dudziak, and Nicholas Donald Lane","cited_arxiv_id":null,"evidence_quote":"Gradnorm is a gradient-norm proxy baseline included in the empirical comparison."},{"cited_title":"Zico: Zero-shot NAS via inverse co- efficient of variation on gradients","cited_arxiv_id":null,"evidence_quote":"ZiCo is a zero-shot proxy baseline based on gradient coefficient of variation, compared on stratified and non-stratified sets."},{"cited_title":"Neural architecture search on imagenet in four GPU hours: A theoretically inspired perspective","cited_arxiv_id":null,"evidence_quote":"TE-NAS is a theoretical-inspired zero-shot ranking baseline used in the comparison."},{"cited_title":"Zen-nas: A zero-shot NAS for high-performance image recogni- tion","cited_arxiv_id":null,"evidence_quote":"Zen-NAS is a zero-shot expressivity-score baseline from the comparison."}],"review_version":1}