{"id":"1d30cdb0-26f8-48bd-ade1-4097ecd488b8","arxiv_id":"2505.09344","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A random forest trained on 21 zero-cost proxies and network metadata predicts NATS-Bench test accuracy with Kendall correlations above 0.9 on held-out splits.","lead":"GreenFactory combines many zero-cost neural network scoring rules with a random forest to predict final test accuracy on NATS-Bench without training the networks. A smart generalist might read it because cheaper architecture search could reduce the energy cost of building and tuning AI models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark-specific feature selection and dataset-indicator features leave the reported correlations untested on unseen datasets or search spaces.","rationale":"The paper is internally careful: it uses explicit 70-15-15 splits, reports a large held-out test set, and evaluates both random and stratified sampling, so the within-NATS-Bench result is likely reproducible. The central weakness is external validity. GreenFactory's accuracy as a general zero-cost estimator depends on proxy features that were selected and, in the GreenMachine case, evolved on the same benchmark. The dataset indicator, while legitimate for ranking within a known dataset, provides no mechanism for an unseen dataset and cannot support the claim of robustness beyond NATS-Bench. The reader's conditional verdict is appropriate: the paper does what it claims on NATS-Bench, but the generalization it motivates is untested. A leave-one-dataset-out or leave-one-search-space-out experiment would settle whether the reported correlations reflect a reusable estimator or benchmark-specific interpolation. I therefore see no reason to change the reader's CONDITIONAL verdict, and I agree that the weakest assumption is the transferability of the feature set.","tokens_in":16403,"tokens_out":8598,"duration_ms":98841,"concrete_test":"Run the full GreenFactory pipeline with leave-one-dataset-out on NATS-Bench: train on two of the three datasets and evaluate Kendall correlation on the held-out dataset, repeating for all three, and similarly train on one search space and test on the other. If correlations on the held-out dataset or search space drop substantially below the reported 0.90 to 0.95 range, the central generalization claim fails. This test directly addresses the benchmark-specific feature selection and the one-hot dataset indicator limitation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reported Kendall correlations are computed on held-out architectures from the same benchmark that was used to select the proxy features, so they do not establish a general zero-cost accuracy predictor. The dominant features in GreenFactory are GreenMachine proxies E, F, and J, which were discovered by an evolutionary process whose fitness was evaluated on NATS-Bench. Thus the test architectures, while disjoint from the random-forest training set, were not independent of the choice of feature functional forms. Additionally, the final feature set includes one-hot dataset indicators, and the six-feature GreenFactory-Fast model explicitly retains the CIFAR-10 indicator; a new dataset is an unseen category with no defined feature value. The paper itself acknowledges this in Section 5.1, stating that expanding to additional benchmarks could enhance generalizability and proposing a leave-one-out test. Without such an evaluation, the high correlations in Tables 5 and 6 are consistent with overfitting to NATS-Bench-specific feature-accuracy relationships rather than with robust transfer to new search spaces or datasets.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes GreenFactory, a random forest regressor that combines multiple zero-cost proxies (NASWOT, Synflow, GradNorm, TE-NAS, Zen-NAS, ZiCo, EZNAS, AZ-NAS, expressivity/progressivity/trainability, ten GreenMachine proxies, network parameters, and FLOPs) to directly predict test accuracy. The method is evaluated on NATS-Bench-SSS and NATS-Bench-TSS across CIFAR-10, CIFAR-100, and ImageNet-16-120. Recursive feature elimination selects either a 25-feature GreenFactory model or a 6-feature GreenFactory-Fast model, and hyperparameters are tuned with Optuna. The paper reports Kendall correlations between 0.907 and 0.945 on the size search space and between 0.908 and 0.929 on the topology search space, which improve on individual proxies, and shows that the fast model still outperforms most single proxies at lower computational cost.","tokens_in":16525,"tokens_out":6117,"duration_ms":55709,"significance":"If the reported correlations hold outside NATS-Bench, GreenFactory would be a valuable contribution to training-free performance prediction. The direct prediction of accuracy, rather than a rank-only score, is a step beyond standard zero-cost proxies, and the systematic comparison on thousands of architectures is a strength. The supplementary material with proxy formulas and the detailed recursive feature elimination trace support reproducibility. However, the generalization claim is currently supported only by held-out architectures within the same benchmark used for feature design and selection, so the significance of the work as a general-purpose predictor is not yet established.","major_comments":[{"comment":"The reported Kendall and Spearman correlations are computed on held-out architectures from NATS-Bench, but the features fed to the random forest are not independent of the test benchmark. The GreenMachine proxies were evolved using fitness evaluated on NATS-Bench accuracy (Section 2.1 and reference [9]), and the feature subset was chosen by recursive feature elimination on the same NATS-Bench data (Section 3, Feature Selection). The paper itself acknowledges in Section 5.1 that expanding to additional benchmarks is needed to enhance generalizability and proposes a leave-one-out test as future work. Without an out-of-benchmark evaluation, for example on NAS-Bench-201 or a held-out dataset, the central claim that GreenFactory is a robust zero-cost accuracy predictor is not supported by the current evidence. I request either such an experiment or a revised abstract that qualifies the claim as valid for NATS-Bench only.","section":"Section 4 (Tables 5-6) and Section 5.1"},{"comment":"The dataset includes one-hot encoded indicators for the search space and dataset, and the final feature sets retain these indicators: the six-feature GreenFactory-Fast model explicitly includes the CIFAR-10 indicator, and the 25-feature GreenFactory model includes CIFAR-10 and CIFAR-100 indicators (Figure 4). For any new dataset, these one-hot features would have an unseen-category value with no training support, so a deployment on a new dataset cannot use them as constructed. This introduces a form of target-environment leakage: the model can rely on dataset identity to adjust its predictions, which is not a zero-cost proxy signal. The paper's future-work suggestion to replace one-hot encoding with abstract dataset features (Section 5.1) acknowledges the issue, but the current evaluation does not address it.","section":"Section 3 (Data Collection) and Figure 4"},{"comment":"No trivial-feature baseline is reported. A random forest trained on only the cheap features (number of parameters, FLOPs, and the three dataset indicators) would control for the information already available without computing any zero-cost proxy. Since the number of parameters alone reaches a Kendall tau of 0.665 on SSS CIFAR-10 (Table 5), and the reported TSS rows show FLOPs identical to #Params, such a baseline could already be competitive. Without this control, the incremental contribution of the 20+ proxies to the ensemble is not demonstrated. Adding this baseline is needed to support the claim that the zero-cost proxies are what make GreenFactory effective.","section":"Section 3 (Feature Selection) and Section 4 (Evaluation)"}],"minor_comments":[{"comment":"The RMSE values are presumably in percentage points because test accuracy ranges from 0 to 100, but the units are not stated; please clarify.","section":"Section 4, Table 4"},{"comment":"The text says 'the 10 most promising proxies obtained by the GreenMachine algorithm,' but later tables refer to GreenMachine-1/2/3 and GreenMachine-A through J; please clarify the mapping and the total count to aid reproducibility.","section":"Section 3, Data Collection"},{"comment":"The statement that EZNAS 'lacks clarity regarding other proxies, relies on partial layer statistics, and uses a low recombination rate' is presented without a citation or supporting comparison; please provide evidence or rephrase as a general observation.","section":"Section 2.1"},{"comment":"The abstract's phrase 'robust results across multiple datasets' overstates the evidence because all datasets belong to NATS-Bench; consider qualifying this as 'within the NATS-Bench benchmark.'","section":"Section 5.1"},{"comment":"The Acknowledgements section contains placeholder text ('By using the ack environment to insert your (optional) acknowledgements...') that should be removed.","section":"Acknowledgements"},{"comment":"The paper does not state whether the Green-NATS-Bench dataset and code will be made available; please add a reproducibility or data-availability statement.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is methodologically sound for the narrow claim of high rank correlation on held-out NATS-Bench architectures. The main risk is overclaiming generalization, because the feature representation and feature subset are derived from the same benchmark that is used for evaluation. I recommend requiring an out-of-benchmark experiment (or a clear scope restriction) and a trivial-feature baseline before publication. No issues of citation integrity or misconduct are apparent; the placeholder acknowledgements text should simply be removed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the within-benchmark result is strong and probably reproducible: a random forest over 21 zero-cost proxies predicts NATS-Bench test accuracy with Kendall correlations above 0.9, and the paper does a careful job with stratification, feature selection, and hyperparameter tuning. Second, the paper's general claim—that this is a robust, generalizable accuracy predictor—is not actually tested. Every experiment is on NATS-Bench, and the feature set itself was selected and, for the GreenMachine proxies, evolved on NATS-Bench. The test architectures are disjoint from the training set, but they are not independent of the feature-construction process.\n\nWhat is genuinely new: Green-NATS-Bench, a dataset of proxy scores for NATS-Bench, and the finding that a simple random forest ensemble with recursive feature elimination beats each individual proxy on rank correlation. GreenFactory-Fast, using six features (three GreenMachine proxies plus GradNorm, EZ-NAS, and the CIFAR-10 indicator), is a practical cheap predictor. The comparison tables are thorough, and the paper is honest enough to include the baseline proxies and the stratified split. Credit where due: the experiments are well-designed for what they cover.\n\nThe soft spots are the usual ones for benchmark-bound NAS work. The dataset indicators are one-hot, so a new dataset is an unseen category—the model has no defined feature value for it. The dominant features, GreenMachine E, F, and J, were discovered by evolutionary search whose fitness was computed on NATS-Bench accuracy, so the ensemble is partly refitting features that were already shaped by the target. There are no error bars on the final Kendall correlations, and no trivial-feature baseline (say, number of parameters plus dataset indicator) to show how much the proxy scores actually contribute. The paper's own future-work section flags the leave-one-out test; that is the experiment that should have been in the main body.\n\nNone of this sinks the paper. The within-benchmark finding is useful, and the dataset alone is a contribution. But the abstract's \"reliability\" language goes beyond the evidence. A serious referee should ask for: (1) a leave-one-benchmark-out or cross-dataset evaluation, (2) a trivial-feature control, and (3) code and data release.\n\nWho this is for: anyone working on zero-cost NAS who wants a better ranking method on NATS-Bench or a ready-made proxy-score dataset. It deserves a real peer-review round, not a desk reject, but the revision needs to earn the generalization claim.","headline":"A solid, well-measured ensemble zero-cost proxy for NATS-Bench that overclaims generalization: the correlations are real on-benchmark, but nothing yet tests transfer to unseen datasets or search spaces.","tokens_in":17111,"tokens_out":1455,"would_cite":true,"duration_ms":16337,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An ensemble of zero-cost proxies can directly predict the test accuracy of an untrained neural network, reaching Kendall correlations above 0.9 on NATS-Bench.","keywords":["zero-cost proxies","neural architecture search","random forest regression","performance prediction","NATS-Bench","training-free NAS","ensemble regression","GreenFactory"],"falsifier":"Run a leave-one-dataset-out experiment inside NATS-Bench, training on two datasets and testing on the third with the dataset one-hot indicators removed; if Kendall correlation on the held-out dataset drops far below the reported 0.9 while each individual proxy keeps its usual correlation, the ensemble is exploiting dataset-specific calibration rather than a transferable proxy signal.","tokens_in":16150,"feed_emoji":"⚡","tokens_out":7624,"duration_ms":67459,"temperature":0.7,"pith_summary":"Zero-cost proxies are quick calculations on an untrained network that estimate how well it would perform after training, but most only rank architectures and fail to generalize across datasets. This paper claims that a random forest trained on the scores of 21 such proxies, plus parameter count and FLOPs, can directly output a network's final test accuracy. On the NATS-Bench benchmark, the method reaches Kendall correlations of 0.907 to 0.945 on the size search space and 0.908 to 0.929 on the topology search space, with Spearman correlations above 0.98 in several settings. If the claim holds, architecture search could screen candidates without training them, replacing hundreds of GPU-hours with minutes of proxy computation.","feed_headline":"Zero-cost proxy ensemble predicts neural net accuracy","feed_subtitle":"Random-forest blend of training-free scores reaches 0.9+ rank correlation on NATS-Bench.","key_machinery":"The central object is Green-NATS-Bench, a table in which each row is an architecture, the columns are scores from 21 zero-cost proxies, parameter count, FLOPs, one-hot indicators of dataset and search space, and the target is the fully trained test accuracy. A random forest regressor learns the mapping from proxy scores to accuracy, chosen because it gave the lowest root mean squared error among the regressors tested. Recursive feature elimination prunes the feature set, and stratified sampling by accuracy bin counteracts the benchmark's heavy skew toward high-performing networks.","core_discovery":"A random forest regressor over an ensemble of zero-cost proxy scores, measured on untrained networks, can predict the trained test accuracy of a deep neural network directly rather than producing only a relative ranking. The paper builds a dataset, Green-NATS-Bench, from 21 proxies plus parameter count and FLOPs for every architecture in NATS-Bench's size and topology search spaces, then selects features and tunes hyperparameters. The resulting GreenFactory model attains Kendall correlations of 0.907, 0.945, and 0.920 on NATS-Bench-SSS for CIFAR-10, CIFAR-100, and ImageNet-16-120, and 0.921, 0.929, and 0.908 on NATS-Bench-TSS. A stripped-down six-feature variant, GreenFactory-Fast, uses roughly 11 percent of the computation time and still outperforms every individual proxy on most settings.","pith_inferences":["Inference: because the model includes one-hot dataset indicators, a leave-one-dataset-out test is needed to show the accuracy values are not partly dataset-specific calibration; the paper names this test as future work.","Inference: the GreenMachine proxies that dominate feature importance were discovered by an evolutionary process, and the paper does not test whether their signal transfers outside NATS-Bench.","Inference: a concrete extension would be to use GreenFactory-Fast inside a NAS loop to discard the bottom half of a candidate pool before training, then fully train only the survivors."],"forward_implications":["NAS methods could replace full training evaluation with GreenFactory's accuracy estimate for ranking candidate architectures, cutting the compute of a search from hundreds of GPU-hours to minutes.","With six features and only 11 percent of the computation time, GreenFactory-Fast retains most of the rank correlation, making it usable as a cheap pre-filter before any training.","Spearman correlations above 0.98 in several configurations mean the predicted ordering of thousands of networks is nearly identical to the true test-accuracy ordering.","The stratified evaluation shows the model tracks low-performing networks as well as high-performing ones, so it can also be used to discard bad candidates with confidence."],"supporting_citations":[{"why":"Supplies the NATS-Bench search spaces, datasets, and trained test accuracies that form the prediction target.","marker":"[13]"},{"why":"Defines zero-cost proxies and the Kendall and Spearman correlation metrics used for evaluation.","marker":"[29]"},{"why":"Provides the ten GreenMachine proxies that dominate feature importance in both optimized models.","marker":"[9]"},{"why":"Contributes NASWOT, one of the input zero-cost proxies based on untrained activation patterns.","marker":"[21]"},{"why":"Contributes Synflow, the data-independent pruning-based proxy used as an input feature.","marker":"[26]"},{"why":"Contributes GradNorm, the gradient-norm proxy included in the ensemble and in the fast model.","marker":"[1]"},{"why":"Supplies recursive feature elimination, the method used to reduce the proxy set to 25 and then to six features.","marker":"[17]"},{"why":"Provides the Optuna hyperparameter optimization framework used to tune the random forest.","marker":"[2]"}],"fun_headline_variants":["Random forest of zero-cost proxies forecasts DNN accuracy","GreenFactory: predicting test accuracy without training","Ensemble of zero-cost proxies hits 0.9+ correlation","Training-free proxy ensemble estimates net accuracy","Zero-cost proxy blend predicts neural network performance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the proxy features selected and tuned on NATS-Bench will keep their predictive value on architectures and datasets outside the benchmark; if they do not, the reported correlations are benchmark-specific rather than a general zero-cost accuracy estimate.","fun_headline_variants_meta":{"raw":{"variants":["Random forest of zero-cost proxies forecasts DNN accuracy","GreenFactory: predicting test accuracy without training","Ensemble of zero-cost proxies hits 0.9+ correlation","Training-free proxy ensemble estimates net accuracy","Zero-cost proxy blend predicts neural network performance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000205,"raw_usage":{"total_tokens":1402,"prompt_tokens":961,"completion_tokens":441,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":369}},"tokens_in":577,"tokens_out":441,"duration_ms":4512,"temperature":1.0,"reasoning_tokens":369,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:34:25.077543+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a leave-one-dataset-out experiment inside NATS-Bench, training on two datasets and testing on the third with the dataset one-hot indicators removed; if Kendall correlation on the held-out dataset drops far below the reported 0.9 while each individual proxy keeps its usual correlation, the ensemble is exploiting dataset-specific calibration rather than a transferable proxy signal.","supporting_citations":[{"cited_title":"Wu and C.-W","cited_arxiv_id":null,"evidence_quote":"Defines zero-cost proxies and the Kendall and Spearman correlation metrics used for evaluation."},{"cited_title":"GreenMachine: Automatic Design of Zero-Cost Proxies for Energy-Efficient NAS","cited_arxiv_id":"2411.15290","evidence_quote":"Provides the ten GreenMachine proxies that dominate feature importance in both optimized models."},{"cited_title":"Mellor, J","cited_arxiv_id":null,"evidence_quote":"Contributes NASWOT, one of the input zero-cost proxies based on untrained activation patterns."},{"cited_title":"Tanaka, D","cited_arxiv_id":null,"evidence_quote":"Contributes Synflow, the data-independent pruning-based proxy used as an input feature."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes GradNorm, the gradient-norm proxy included in the ensemble and in the fast model."},{"cited_title":"Akiba, S","cited_arxiv_id":null,"evidence_quote":"Provides the Optuna hyperparameter optimization framework used to tune the random forest."}],"review_version":1}