REVIEW 3 major objections 6 minor 35 references
GreenFactory: Ensembling Zero-Cost Proxies to Estimate Performance of Neural Networks
T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read An ensemble of zero-cost proxies can directly predict the test accuracy of an untrained neural network, reaching Kendall correlations above 0.9 on NATS-Bench.
desk verdict A solid, well-measured ensemble zero-cost proxy for NATS-Bench that overclaims generalization: the correlations are real on-benchmark, but nothing yet tests transfer to unseen datasets or search spaces. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is Green-NATS-Bench, a table in which each row is an architecture, the columns are scores from 21 zero-cost proxies, parameter count, FLOPs, one-hot indicators of dataset and search space, and the target is the fully trained test accuracy. A random forest regressor learns the mapping from proxy scores to accuracy, chosen because it gave the lowest root mean squared error among the regressors tested. Recursive feature elimination prunes the feature set, and stratified sampling by accuracy bin counteracts the benchmark's heavy skew toward high-performing networks.
What would settle it
Run a leave-one-dataset-out experiment inside NATS-Bench, training on two datasets and testing on the third with the dataset one-hot indicators removed; if Kendall correlation on the held-out dataset drops far below the reported 0.9 while each individual proxy keeps its usual correlation, the ensemble is exploiting dataset-specific calibration rather than a transferable proxy signal.
Extended reading notes
Core claim
A random forest regressor over an ensemble of zero-cost proxy scores, measured on untrained networks, can predict the trained test accuracy of a deep neural network directly rather than producing only a relative ranking. The paper builds a dataset, Green-NATS-Bench, from 21 proxies plus parameter count and FLOPs for every architecture in NATS-Bench's size and topology search spaces, then selects features and tunes hyperparameters. The resulting GreenFactory model attains Kendall correlations of 0.907, 0.945, and 0.920 on NATS-Bench-SSS for CIFAR-10, CIFAR-100, and ImageNet-16-120, and 0.921, 0.929, and 0.908 on NATS-Bench-TSS. A stripped-down six-feature variant, GreenFactory-Fast, uses roughly 11 percent of the computation time and still outperforms every individual proxy on most settings.
Load-bearing premise
The load-bearing premise is that the proxy features selected and tuned on NATS-Bench will keep their predictive value on architectures and datasets outside the benchmark; if they do not, the reported correlations are benchmark-specific rather than a general zero-cost accuracy estimate.
Editorial extensions
If this is right
- NAS methods could replace full training evaluation with GreenFactory's accuracy estimate for ranking candidate architectures, cutting the compute of a search from hundreds of GPU-hours to minutes.
- With six features and only 11 percent of the computation time, GreenFactory-Fast retains most of the rank correlation, making it usable as a cheap pre-filter before any training.
- Spearman correlations above 0.98 in several configurations mean the predicted ordering of thousands of networks is nearly identical to the true test-accuracy ordering.
- The stratified evaluation shows the model tracks low-performing networks as well as high-performing ones, so it can also be used to discard bad candidates with confidence.
Reading between the lines
- Inference: because the model includes one-hot dataset indicators, a leave-one-dataset-out test is needed to show the accuracy values are not partly dataset-specific calibration; the paper names this test as future work.
- Inference: the GreenMachine proxies that dominate feature importance were discovered by an evolutionary process, and the paper does not test whether their signal transfers outside NATS-Bench.
- Inference: a concrete extension would be to use GreenFactory-Fast inside a NAS loop to discard the bottom half of a candidate pool before training, then fully train only the survivors.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes GreenFactory, a random forest regressor that combines multiple zero-cost proxies (NASWOT, Synflow, GradNorm, TE-NAS, Zen-NAS, ZiCo, EZNAS, AZ-NAS, expressivity/progressivity/trainability, ten GreenMachine proxies, network parameters, and FLOPs) to directly predict test accuracy. The method is evaluated on NATS-Bench-SSS and NATS-Bench-TSS across CIFAR-10, CIFAR-100, and ImageNet-16-120. Recursive feature elimination selects either a 25-feature GreenFactory model or a 6-feature GreenFactory-Fast model, and hyperparameters are tuned with Optuna. The paper reports Kendall correlations between 0.907 and 0.945 on the size search space and between 0.908 and 0.929 on the topology search space, which improve on individual proxies, and shows that the fast model still outperforms most single proxies at lower computational cost.
Significance. If the reported correlations hold outside NATS-Bench, GreenFactory would be a valuable contribution to training-free performance prediction. The direct prediction of accuracy, rather than a rank-only score, is a step beyond standard zero-cost proxies, and the systematic comparison on thousands of architectures is a strength. The supplementary material with proxy formulas and the detailed recursive feature elimination trace support reproducibility. However, the generalization claim is currently supported only by held-out architectures within the same benchmark used for feature design and selection, so the significance of the work as a general-purpose predictor is not yet established.
major comments (3)
- [Section 4 (Tables 5-6) and Section 5.1] The reported Kendall and Spearman correlations are computed on held-out architectures from NATS-Bench, but the features fed to the random forest are not independent of the test benchmark. The GreenMachine proxies were evolved using fitness evaluated on NATS-Bench accuracy (Section 2.1 and reference [9]), and the feature subset was chosen by recursive feature elimination on the same NATS-Bench data (Section 3, Feature Selection). The paper itself acknowledges in Section 5.1 that expanding to additional benchmarks is needed to enhance generalizability and proposes a leave-one-out test as future work. Without an out-of-benchmark evaluation, for example on NAS-Bench-201 or a held-out dataset, the central claim that GreenFactory is a robust zero-cost accuracy predictor is not supported by the current evidence. I request either such an experiment or a revised abstract that qualifies the claim as valid for NATS-Bench only.
- [Section 3 (Data Collection) and Figure 4] The dataset includes one-hot encoded indicators for the search space and dataset, and the final feature sets retain these indicators: the six-feature GreenFactory-Fast model explicitly includes the CIFAR-10 indicator, and the 25-feature GreenFactory model includes CIFAR-10 and CIFAR-100 indicators (Figure 4). For any new dataset, these one-hot features would have an unseen-category value with no training support, so a deployment on a new dataset cannot use them as constructed. This introduces a form of target-environment leakage: the model can rely on dataset identity to adjust its predictions, which is not a zero-cost proxy signal. The paper's future-work suggestion to replace one-hot encoding with abstract dataset features (Section 5.1) acknowledges the issue, but the current evaluation does not address it.
- [Section 3 (Feature Selection) and Section 4 (Evaluation)] No trivial-feature baseline is reported. A random forest trained on only the cheap features (number of parameters, FLOPs, and the three dataset indicators) would control for the information already available without computing any zero-cost proxy. Since the number of parameters alone reaches a Kendall tau of 0.665 on SSS CIFAR-10 (Table 5), and the reported TSS rows show FLOPs identical to #Params, such a baseline could already be competitive. Without this control, the incremental contribution of the 20+ proxies to the ensemble is not demonstrated. Adding this baseline is needed to support the claim that the zero-cost proxies are what make GreenFactory effective.
minor comments (6)
- [Section 4, Table 4] The RMSE values are presumably in percentage points because test accuracy ranges from 0 to 100, but the units are not stated; please clarify.
- [Section 3, Data Collection] The text says 'the 10 most promising proxies obtained by the GreenMachine algorithm,' but later tables refer to GreenMachine-1/2/3 and GreenMachine-A through J; please clarify the mapping and the total count to aid reproducibility.
- [Section 2.1] The statement that EZNAS 'lacks clarity regarding other proxies, relies on partial layer statistics, and uses a low recombination rate' is presented without a citation or supporting comparison; please provide evidence or rephrase as a general observation.
- [Section 5.1] The abstract's phrase 'robust results across multiple datasets' overstates the evidence because all datasets belong to NATS-Bench; consider qualifying this as 'within the NATS-Bench benchmark.'
- [Acknowledgements] The Acknowledgements section contains placeholder text ('By using the ack environment to insert your (optional) acknowledgements...') that should be removed.
- [General] The paper does not state whether the Green-NATS-Bench dataset and code will be made available; please add a reproducibility or data-availability statement.
Circularity Check
No circular derivation: GreenFactory is a standard supervised regressor evaluated on a held-out split; the authors' own GreenMachine proxies are prominent inputs but are recomputed and individually evaluated, not used as an unverified self-citation.
full rationale
GreenFactory trains a random forest on a 70% split of NATS-Bench proxy scores to predict test accuracy, tunes hyperparameters on a 15% validation split, and reports correlations on a disjoint 15% test split. This is ordinary supervised prediction, not a re-statement of the input. The proxy features (NASWOT, Synflow, GradNorm, TE-NAS, Zen-NAS, ZiCo, EZ-NAS, AZ-NAS, GreenMachine, #Params, FLOPs, and dataset indicators) are fixed formulas or counts; none is defined in terms of the target test accuracy within this paper. The GreenMachine proxies from the authors' prior work [9] dominate feature importance, but the paper reproduces their formulas in Figure 5 and evaluates them individually in Tables 8 and 9, so the central argument does not reduce to the self-citation. One-hot dataset indicators are legitimate categorical features (they encode which dataset a sample comes from, not its accuracy). Section 5.1 explicitly concedes that cross-benchmark generalization is untested ('Expanding to additional benchmarks could enhance the generalizability of our solutions... A leave-one-out test using NATS-Bench datasets could also assess ensemble generalization to unseen datasets'), which is a limitation for the stated goal of robust generalization, not an instance of circular reasoning. The reported Kendall correlations are thus valid held-out architecture predictions within NATS-Bench, though they should not be read as evidence of transfer to unseen search spaces or datasets.
Assumptions & free parameters
free parameters (7)
- Number of estimators (random forest) =
968 (GreenFactory), 945 (GreenFactory-Fast)
- Max features =
8 (GreenFactory), 10 (GreenFactory-Fast)
- Min samples split =
3
- Min samples leaf =
1
- Max depth =
38 (GreenFactory), 73 (GreenFactory-Fast)
- Number of selected features =
25 (GreenFactory), 6 (GreenFactory-Fast)
- Stratification bins =
5
assumptions (5)
- domain assumption Proxy scores computed at initialization are informative about final test accuracy.
- domain assumption NATS-Bench test accuracies are accurate ground truth.
- domain assumption The random forest generalizes within the benchmark distribution.
- domain assumption One-mini-batch proxy estimates are stable enough for regression.
- domain assumption Dataset indicators fully encode cross-dataset differences.
Cite this review
Pith. "Pith review of GreenFactory: Ensembling Zero-Cost Proxies to Estimate Performance of Neural Networks." pith.science (2026). https://pith.science/paper/YUYTKXEB
@misc{pith2026250509344,
author = {Pith},
title = {Pith review of: GreenFactory: Ensembling Zero-Cost Proxies to Estimate Performance of Neural Networks},
year = {2026},
howpublished = {\url{https://pith.science/paper/YUYTKXEB}},
note = {Machine review of arXiv:2505.09344}
}
read the original abstract
Determining the performance of a Deep Neural Network during Neural Architecture Search processes is essential for identifying optimal architectures and hyperparameters. Traditionally, this process requires training and evaluation of each network, which is time-consuming and resource-intensive. Zero-cost proxies estimate performance without training, serving as an alternative to traditional training. However, recent proxies often lack generalization across diverse scenarios and provide only relative rankings rather than predicted accuracies. To address these limitations, we propose GreenFactory, an ensemble of zero-cost proxies that leverages a random forest regressor to combine multiple predictors' strengths and directly predict model test accuracy. We evaluate GreenFactory on NATS-Bench, achieving robust results across multiple datasets. Specifically, GreenFactory achieves high Kendall correlations on NATS-Bench-SSS, indicating substantial agreement between its predicted scores and actual performance: 0.907 for CIFAR-10, 0.945 for CIFAR-100, and 0.920 for ImageNet-16-120. Similarly, on NATS-Bench-TSS, we achieve correlations of 0.921 for CIFAR-10, 0.929 for CIFAR-100, and 0.908 for ImageNet-16-120, showcasing its reliability in both search spaces.
Figures
Reference graph
Works this paper leans on
-
[9]
GreenMachine: Automatic Design of Zero-Cost Proxies for Energy-Efficient NAS
G. Cortês, N. Lourenço, and P. Machado. GreenMachine: Auto- matic design of zero-cost proxies for energy-efficient NAS. CoRR, abs/2411.15290, 2024. doi: 10.48550/ARXIV .2411.15290. URL https: //doi.org/10.48550/arXiv.2411.15290
work page Pith review arXiv doi:10.48550/arxiv.2411.15290 2024
-
[1]
M. S. Abdelfattah, A. Mehrotra, L. Dudziak, and N. D. Lane. Zero- cost proxies for lightweight NAS. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7,
work page 2021
- [2]
-
[3]
A. A. Alatawi, S. M. Alomani, N. I. Alhawiti, and M. Ayaz. Plant disease detection using AI based VGG-16 model.International Journal of Advanced Computer Science and Applications , 2022. URL https: //api.semanticscholar.org/CorpusID:248654264
work page 2022
-
[4]
S. A. Alowais, S. S. Alghamdi, N. Alsuhebany, T. Alqahtani, A. I. Alshaya, S. N. Almohareb, A. Aldairem, M. Alrashed, K. Bin Saleh, H. A. Badreldin, M. S. Al Yami, S. Al Harbi, and A. M. Albekairy. Revolutionizing healthcare: the role of artificial intelligence in clini- cal practice. BMC Medical Education , 23(1):689, Sep 22 2023. doi: 10.1186/s12909-023-04698-z
-
[5]
J. Bergstra, R. Bardenet, Y . Bengio, and B. Kégl. Algorithms for hyper-parameter optimization. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K. Weinberger, editors, Advances in Neural Information Processing Systems , volume 24. Curran Associates, Inc.,
- [6]
-
[7]
W. Chen, X. Gong, and Z. Wang. Neural architecture search on im- agenet in four GPU hours: A theoretically inspired perspective. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021. URL https://openreview.net/forum?id=Cnon5ezMHtu
work page 2021
Show all 35 references
-
[8]
Cortês, N
G. Cortês, N. Lourenço, and P. Machado. Towards physical plausibil- ity in neuroevolution systems. In S. L. Smith, J. Correia, and C. Cin- trano, editors, Applications of Evolutionary Computation - 27th Eu- ropean Conference, EvoApplications 2024, Held as Part of EvoStar 2024,...
2024
-
[10]
B. Dash, M. M. Ansari, P. Sharma, and A. Ali. Threats and opportuni- ties with AI-based cyber security intrusion detection: A review. Inter- national Journal of Software Engineering & Applications , 2022. URL https://api.semanticscholar.org/CorpusID:252865007
2022
-
[11]
de Vries
A. de Vries. The growing energy footprint of artificial intelligence. Joule, 7(10):2191–2194, 2023. ISSN 2542-4351. doi: https://doi. org/10.1016/j.joule.2023.09.004. URL https://www.sciencedirect.com/ science/article/pii/S2542435123003653
2023 doi
-
[12]
Dong and Y
X. Dong and Y . Yang. NAS-Bench-201: Extending the scope of re- producible neural architecture search. In 8th International Confer- ence on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https://openreview.net/ forum?id...
2020
-
[13]
X. Dong, L. Liu, K. Musial, and B. Gabrys. NATS-Bench: Bench- marking NAS algorithms for architecture topology and size. IEEE Trans. Pattern Anal. Mach. Intell. , 44(7):3634–3646, 2022. doi: 10.1109/TPAMI.2021.3054824. URL https://doi.org/10.1109/TPAMI. 2021.3054824
2022
-
[14]
Furman and R
J. Furman and R. Seamans. AI and the economy. Working Paper 24689, National Bureau of Economic Research, June 2018. URL http://www. nber.org/papers/w24689
2018
-
[15]
O. Gomes. I, robot: the three laws of robotics and the ethics of the peopleless economy. AI Ethics , 4(2):257–272, 2024. doi: 10.1007/S43681-023-00263-Y. URL https://doi.org/10.1007/ s43681-023-00263-y
2024 doi
-
[16]
2024 Environmental Report
Google. 2024 Environmental Report. https://sustainability.google/ reports/google-2024-environmental-report/, 2024. [Accessed 05-10- 2024]
2024
-
[17]
Guyon, J
I. Guyon, J. Weston, S. Barnhill, and V . Vapnik. Gene selection for cancer classification using support vector machines. Mach. Learn., 46 (1-3):389–422, 2002. doi: 10.1023/A:1012487302797. URL https:// doi.org/10.1023/A:1012487302797
2002 doi
-
[18]
Kadiresan, Y
A. Kadiresan, Y . Baweja, and O. Ogbanufe. Bias in AI-Based Decision- Making, pages 275–285. Springer International Publishing, Cham,
-
[19]
G. Li, Y . Yang, K. Bhardwaj, and R. Marculescu. Zico: Zero-shot NAS via inverse coefficient of variation on gradients. In The Eleventh In- ternational Conference on Learning Representations, ICLR 2023, Ki- gali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. URL https: //openr...
2023
-
[20]
M. Lin, P. Wang, Z. Sun, H. Chen, X. Sun, Q. Qian, H. Li, and R. Jin. Zen-nas: A zero-shot NAS for high-performance image recognition. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021 , pages 337–346. IEEE, 2021...
2021
-
[21]
Mellor, J
J. Mellor, J. Turner, A. Storkey, and E. J. Crowley. Neural architecture search without training. In M. Meila and T. Zhang, editors, Proceed- Table 5. Comparison of Zero-Cost proxies on the NATS-Bench benchmark across the CIFAR-10, CIFAR-100, and ImageNet-16-120 datasets on th...
-
[22]
D. A. Patterson, J. Gonzalez, U. Hölzle, Q. V . Le, C. Liang, L. Munguia, D. Rothchild, D. R. So, M. Texier, and J. Dean. The carbon footprint of machine learning training will plateau, then shrink. Computer, 55(7): 18–28, 2022. doi: 10.1109/MC.2022.3148714
2022
-
[23]
M. A. Ramirez, S. Kim, H. M. N. A. Hamadi, E. Damiani, Y . Byon, T. Kim, C. Cho, and C. Y . Yeun. Poisoning attacks and defenses on artificial intelligence: A survey. CoRR, abs/2202.10276, 2022. URL https://arxiv.org/abs/2202.10276
2022 arXiv
-
[24]
Robinson
D. Robinson. Microsoft’s carbon emissions up nearly 30 https://www.msn.com/en-us/money/other/ microsofts-carbon-emissions-up-nearly-30-thanks-to-ai/ ar-BB1mvgao, 2024. [Accessed 05-10-2024]
2024
-
[25]
G. Su. Unemployment in the AI age. AI Matters, 3(4):35–43, feb 2018. doi: 10.1145/3175502.3175511. URL https://doi.org/10.1145/3175502. 3175511
2018
-
[26]
Tanaka, D
H. Tanaka, D. Kunin, D. L. K. Yamins, and S. Ganguli. Pruning neural networks without any data by iteratively conserving synap- tic flow. In H. Larochelle, M. Ranzato, R. Hadsell, M. Bal- can, and H. Lin, editors, Advances in Neural Information Process- ing Systems 33: Annual ...
2020
-
[27]
R. Vaddy. AI and ML for Transportation Route Optimization. Interna- tional Transactions in Machine Learning , 5(5):1–19, Dec. 2023. URL https://isjr.co.in/index.php/ITML/article/view/200
2023
-
[28]
van Wynsberghe
A. van Wynsberghe. Sustainable AI: AI for sustainability and the sustainability of AI. AI Ethics , 1(3):213–218, 2021. doi: 10.1007/S43681-021-00043-6. URL https://doi.org/10.1007/ s43681-021-00043-6
2021 doi
-
[29]
Wu and C.-W
M.-T. Wu and C.-W. Tsai. Training-free neural architecture search: A review. ICT Express, 10(1):213–231, 2024. ISSN 2405-9595. doi: https: //doi.org/10.1016/j.icte.2023.11.001. URL https://www.sciencedirect. com/science/article/pii/S2405959523001443
2024 doi
- [30]
-
[2011]
URL https://proceedings.neurips.cc/paper_files/paper/2011/file/ 86e8f7ab32cfd12577bc2619bc635690-Paper.pdf
2011
-
[2021]
URL https://openreview.net/forum?id= 0cmMMy8J5q
OpenReview.net, 2021. URL https://openreview.net/forum?id= 0cmMMy8J5q
2021
-
[2022]
doi: 10.1007/978-3-030-84729-6_19
ISBN 978-3-030-84729-6. doi: 10.1007/978-3-030-84729-6_19. URL https://doi.org/10.1007/978-3-030-84729-6_19
-
[2024]
URL https://doi.org/10
doi: 10.1007/978-3-031-56855-8\_5. URL https://doi.org/10. 1007/978-3-031-56855-8\_5
-
[7598]
URL https://proceedings.mlr.press/v139/ mellor21a.html
PMLR, 18–24 Jul 2021. URL https://proceedings.mlr.press/v139/ mellor21a.html
2021
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.