confopt: A Library for Implementation and Evaluation of Gradient-based One-Shot NAS Methods

Abhash Kumar Jha; Arjun Krishnakumar; Frank Hutter; Martin Rapp; Shakiba Moradian

REVIEW 3 major objections 4 minor 40 references

confopt: A Library for Implementation and Evaluation of Gradient-based One-Shot NAS Methods

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Method rankings in gradient-based one-shot NAS flip across nine DARTS-derived benchmarks, showing that evaluation on the original DARTS search space alone is brittle.

desk verdict Useful library and benchmark suite, but the 'confounds removed' claim outruns the experimental design. read the letter →

arxiv 2507.16533 v1 pith:65H664XP submitted 2025-07-22 cs.LG cs.AI

Abhash Kumar Jha , Shakiba Moradian , Arjun Krishnakumar , Martin Rapp , Frank Hutter This is my paper

classification cs.LGcs.AI

keywords neuralarchitecturesearchdifferentiableone-shotNASDARTS-Bench-Suiteconfoptbenchmarkingevaluationprotocol

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Gradient-based one-shot neural architecture search is typically judged on a single search space, the DARTS benchmark, and the field's reported improvements often sit inside the noise. This paper argues that such single-space evaluation is unreliable and can mislead the ranking of NAS methods. To test the claim, the authors built confopt, a library that decomposes NAS optimizers into interchangeable components, and used it to construct DARTS-Bench-Suite, nine benchmarks that vary the supernet's depth, width, cell count, and candidate operations. Evaluating seven NAS optimizers on all nine benchmarks, they found the relative rankings of the methods change substantially across the suite—even between variants of the same search space—and that the choice of retraining hyperparameters also shifts rankings. The paper concludes that ranking NAS methods on the DARTS search space alone is brittle and that more comprehensive, multi-space evaluation is needed.

What carries the argument

The load-bearing machinery is DARTS-Bench-Suite, a set of nine benchmarks formed by crossing three supernet variants (DARTS-Wide, DARTS-Deep, DARTS-Single-Cell) with three candidate-operation sets (Regular, No-skip, All-skip), together with a two-part evaluation protocol that trains the supernet on one half of CIFAR-10 and retrains the derived discrete model from scratch on the other half using nine learning-rate and weight-decay configurations. The protocol factors out the proxy-to-target rank-correlation problem by making the target network match the supernet's size, and it reduces hyperparameter bias by reporting both mean and best test accuracy across the nine configurations. The confopt library supplies the machinery by decomposing NAS optimizers into components—architecture samplers, supernet mutations, regularization terms, pruning operations, and early stopping—so the same optimizer can be run on any of the nine search spaces through a minimal API.

What would settle it

Recompute the Kendall rank correlation between method rankings across the nine benchmarks; if, on fresh runs with different seeds, the pairwise correlations are mostly high and positive (say average above 0.7) rather than near zero or negative, the claim that DARTS-only evaluation is brittle would fail. The paper's own figure reports a correlation of -0.43 between Wide+All Skip and Deep+No Skip, so an independent replication could specifically test whether that anti-correlation persists.

Watch

Extended reading notes

Core claim

The paper establishes that the relative performance of seven gradient-based one-shot NAS optimizers—DARTS, DrNAS, FairDARTS, GDAS, OLES, PC-DARTS, and SmoothDARTS—is not stable when the search space changes. By taking the DARTS cell search and varying the macro-architecture (wide, deep, or single-cell) and the operation set (regular, no skip, all skip), the authors create nine benchmarks that keep roughly one million learnable parameters and remain cheap to train. Across these benchmarks, method rankings differ substantially, with the Kendall rank correlation between some pairs of benchmarks, such as Wide+All Skip and Deep+No Skip, being moderately negative. This rank disparity persists even after two proposed confounders are removed: the proxy-to-target gap is eliminated by making the target model match the supernet's size, and hyperparameter sensitivity is accounted for by retraining every derived architecture under nine fixed hyperparameter configurations and reporting both mean and best accuracy. The paper concludes that DARTS-only evaluation is brittle and insufficient to establish the superiority of one NAS method over another.

Load-bearing premise

The reimplementations of the seven NAS methods faithfully represent the published algorithms; the paper itself notes that DrNAS was run without its progressive-pruning strategy and with an added 15-epoch warm-up, so any ranking shifts could partly be artifacts of these modifications.

Editorial extensions

If this is right

Claims of superiority for a new gradient-based one-shot NAS optimizer should be backed by evaluation on more than the original DARTS search space.
The nine benchmarks and the confopt codebase give NAS researchers a cheap, standardized way to run multi-space comparisons instead of relying on the single DARTS setup.
Because retraining hyperparameters change method rankings, reporting only accuracy under a single default hyperparameter set can misrepresent an architecture's intrinsic quality.
Repeating this multi-space protocol on larger, more diverse datasets could reveal which methods genuinely generalize rather than fit the DARTS search space.
The observed ranking instability suggests that statistically significant improvements on one benchmark should not be read as method-level superiority until confirmed across search-space variants.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the brittleness result generalizes, many previously published DARTS-only comparisons in the literature may have ranked methods largely by incidental compatibility with the DARTS hyperparameter recipe rather than by architectural quality.
The confopt component decomposition makes it feasible to search the space of NAS method components (samplers, regularizers, mutations) systematically, which could turn NAS-method design itself into a hyperparameter optimization problem.
A testable extension would be to run the same protocol on non-DARTS search spaces (e.g., NAS-Bench-201 or a transformer cell space) to see whether the instability is specific to DARTS-derived cells or a general property of one-shot NAS evaluation.
Since the wide and deep benchmarks already anti-correlate, designers of future benchmarks might need to control for depth-width interactions explicitly rather than treating a single cell topology as representative.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague—

This paper is worth your time, but treat the headline claim with caution. The confopt library and the nine-benchmark DARTS-Bench-Suite are real artifacts, and the split-half protocol with nine retraining hyperparameter configurations is a sensible attempt to separate architecture quality from evaluation noise. The code is public, the compute reporting is detailed, and the authors acknowledge that the suite is still DARTS-derived and CIFAR-only. That is more than many benchmarking papers do.

The interesting empirical observation is that rankings of seven standard NAS methods shift noticeably across these variants of the same search space. Even without accepting every attribution in the paper, that is useful evidence that DARTS-only evaluation is fragile. The line in Section 6.2 that the rank disparity persists 'even with confounding factors removed and the influence of hyperparameters accounted for' is where the paper overstates. What the protocol actually removes is the proxy-target size mismatch, and it accounts for retraining hyperparameters via a 3x3 grid. It does not control the search-phase choices that are fixed per method: GDAS runs 300 epochs instead of 100 with much larger batch sizes; DrNAS drops progressive pruning, and both DrNAS and PC-DARTS get a 15-epoch warm-up and different learning rates. So a ranking change across Wide/Deep/Single-Cell can just as easily come from an interaction between a method's search configuration and the benchmark variant as from the search space itself.

The statistical analysis is also thinner than the wording implies. One architecture per method per benchmark is selected from three search seeds and retrained with a single seed. Kendall correlations are computed over seven methods with no confidence intervals, so values like -0.43 are well inside sampling noise. The checklist claims significance testing, but I don't see it in the main text.

None of this kills the paper. The library is useful, the benchmark suite is a genuine resource, and the direction of the finding is consistent with earlier work like NAS-Bench-Suite and LHD. But the 'critical flaw' framing should be softened, and a serious revision should add proper significance analysis and ideally varied search-phase settings to separate method from configuration. I'd send this to peer review, not desk reject. The artifacts alone justify referee time, and the empirical claim is worth pressure-testing. I'd bring it to a reading group that cares about NAS benchmarking.

Referee Report

3 major / 4 minor

Summary. The paper presents confopt, an open-source library for implementing and evaluating gradient-based one-shot NAS methods, and introduces DARTS-Bench-Suite, a set of nine benchmarks formed by crossing three supernet variants (Wide, Deep, Single-Cell) with three operation sets (Regular, No-skip, All-skip). Seven NAS optimizers (DARTS, DrNAS, FairDARTS, GDAS, OLES, PC-DARTS, SmoothDARTS) are evaluated on all nine benchmarks using an evaluation protocol that splits CIFAR-10 so that supernet training and discrete-architecture retraining use disjoint halves, and retrains each discovered architecture under nine hyperparameter configurations. The central empirical claim is that method rankings differ substantially across benchmarks, with some benchmark pairs showing negative rank correlation, and that this instability indicates that evaluating on the original DARTS search space alone is brittle.

Significance. If the central claim is established, the paper makes a valuable contribution by providing both a reusable software library and a concrete demonstration that DARTS-only evaluation can mislead. The library design, with modular samplers, supernet mutations, and profile-based configuration, is genuinely useful for the NAS community, and the decision to reduce proxy-target mismatch by matching supernet and target sizes is a sensible protocol improvement. The paper also ships code, reports total compute, and describes the hyperparameter grids, which are strengths for reproducibility. However, the significance of the empirical finding depends on whether the observed rank instability can be attributed to the methods themselves rather than to method-specific search budgets and hyperparameters; that attribution is the load-bearing step and is currently not well supported.

major comments (3)

[Section 6.2, Appendix B.1, Table 14] The assertion that rank disparity persists 'even with confounding factors removed and the influence of hyperparameters accounted for' is not supported by the experimental design. The evaluation controls only the proxy/target mismatch and varies the retraining hyperparameters; the search phase itself uses method-specific budgets and hyperparameters. In particular, GDAS runs for 300 epochs while all other methods run for 100; GDAS uses batch sizes 320/480/480 versus 64/96/96 for DARTS and DrNAS; DrNAS uses a learning rate of 6e-3 instead of 3e-3, omits progressive pruning, and both DrNAS and PC-DARTS receive a 15-epoch warm-up. These search-phase choices are fixed per method, so a ranking change across Wide, Deep, and Single-Cell could reflect an interaction between a method's search hyperparameters and the benchmark variant rather than an intrinsic property of the method or of the search space. Please either control these search-phase factors (e.g., matched epoch budgets, matched optimizer settings, and ablations with and without warm-up/pruning) or substantially weaken the 'confounds removed' claim in Section 6.2 and the abstract.
[Appendix A, Figures 4 and 5] The Kendall rank correlations are computed over only seven methods, with no confidence intervals or significance tests. With n=7, a tau value of 0.43, or even 0.71, is not statistically distinguishable from zero at conventional levels, and the single negative value of -0.43 is likewise within sampling noise. The statement that Wide and Deep benchmarks are 'moderately anti-correlated' is therefore not established. Please report permutation-based p-values or bootstrap confidence intervals for the correlations, and include the underlying per-method raw accuracies with their uncertainties so that readers can assess the stability of the rankings.
[Section 6, Appendix B.2] Each method contributes one architecture per benchmark, selected as the architecture with the lowest validation loss across three search seeds, and that single architecture is then retrained with one random seed (seed 0). Rankings are thus based on a single search-retraining trajectory per method per benchmark, and seed noise in either phase could flip ranks. The paper should report retraining variance over multiple seeds for each selected architecture, or bootstrap over the searched architectures, before drawing conclusions about cross-benchmark rank instability.

minor comments (4)

[Abstract and Section 6.1] The phrase 'best test accuracy' should be defined precisely; it appears to mean the maximum over the nine retraining hyperparameter configurations, but this should be stated explicitly at first use.
[Listing 1, Section 4.2] The minimal GDAS example sets epochs=50, while the experiments in Appendix B.1 run GDAS for 300 epochs. Please add a note clarifying that the short epoch count is for illustration only, or change the example to match the experimental setting.
[Table 16, Appendix B.3] Reporting total GPU hours per benchmark is useful, but a per-method breakdown would make the different search budgets (especially GDAS's 300 epochs) more transparent and would help readers judge the budget confound discussed in the major comments.
[References] Several references contain the typo 'The Eigth International Conference'; it should be 'The Eighth International Conference'.

Circularity Check

0 steps flagged · score 1.0 of 10

No load-bearing circularity: the rank-instability claim is an empirical benchmark finding, not a derived prediction, and the self-citations are not load-bearing.

full rationale

The central claim of Section 6.2 is that the rankings of seven NAS optimizers differ substantially across the nine DARTS-derived benchmarks and that this instability is not explained away by the protocol's controls. This is an experimental observation, not a quantity forced by construction: a stable ranking across the nine benchmarks would have directly contradicted the claim, so the benchmark definitions and evaluation protocol do not encode the conclusion. The suite is specified independently of the outcome, as 'nine benchmarks derived from the DARTS search space, combining three supernet architecture variants with three sets of candidate operations' (Section 5), and the retraining hyperparameter grid is fixed in advance: 'These hyperparameters are not chosen using hyperparameter optimization (HPO)' (Section 5.1). No fitted parameter is later renamed as a prediction; the only selection rule, choosing the architecture with the lowest validation loss among three search seeds, is a standard evaluation step rather than a fitted input to the ranking claim. The paper does cite its own prior work (e.g., Krishnakumar et al. 2024 in Section 4.1 and Mehta et al. 2022 in Section 2.4), but these citations are not load-bearing: no step of the rank-instability argument reduces to those works, and no uniqueness theorem or ansatz is imported from them. The appended limitation, 'Despite introducing more heterogeneous search spaces, we still rely heavily on the DARTS search space,' narrows the scope of the conclusion but does not make it circular. The skeptic's concerns about unequal search epochs, learning rates, batch sizes, and the omitted progressive pruning in DrNAS (Section B.1, Table 14) are validity and confound critiques of the attribution 'even with confounding factors removed,' not circularity: the observed ranking changes are not entailed by those implementation choices and could have failed to appear. The score of 1 reflects the presence of non-load-bearing self-citations and no actual circular step.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper is an empirical benchmark contribution rather than a derivation, so there are no free parameters fitted to a model. The load-bearing assumptions are the validity of the split-half protocol, the proxy-gap removal, and the representativeness of the nine benchmark configurations.

assumptions (3)

domain assumption The two disjoint halves of CIFAR-10 are exchangeable, so training the supernet on the first half and retraining the discrete model on the second half ranks architectures on the original distribution.
Section 5.1 introduces the split; the validity of the ranking depends on this exchangeability.
domain assumption Removing the proxy gap by matching supernet and target model size yields a fair evaluation of the NAS method.
The paper argues in Sections 3.1 and 5 that the target network matching the supernet size factors out rank correlation; this is a design assumption about evaluation validity.
ad hoc to paper The nine benchmark configurations (three supernet shapes x three operation sets) are reasonable instantiations of the DARTS search space, and a stable NAS method should rank similarly across them.
The paper uses this premise to interpret rank correlation differences as evidence of brittleness (Section 6.2). This specific expectation is the hypothesis being tested, not an established fact.

how reviews work

0 comments

Cite this review

Pith. "Pith review of confopt: A Library for Implementation and Evaluation of Gradient-based One-Shot NAS Methods." pith.science (2026). https://pith.science/paper/65H664XP

@misc{pith2026250716533,
  author       = {Pith},
  title        = {Pith review of: confopt: A Library for Implementation and Evaluation of Gradient-based One-Shot NAS Methods},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/65H664XP}},
  note         = {Machine review of arXiv:2507.16533}
}

read the original abstract

Gradient-based one-shot neural architecture search (NAS) has significantly reduced the cost of exploring architectural spaces with discrete design choices, such as selecting operations within a model. However, the field faces two major challenges. First, evaluations of gradient-based NAS methods heavily rely on the DARTS benchmark, despite the existence of other available benchmarks. This overreliance has led to saturation, with reported improvements often falling within the margin of noise. Second, implementations of gradient-based one-shot NAS methods are fragmented across disparate repositories, complicating fair and reproducible comparisons and further development. In this paper, we introduce Configurable Optimizer (confopt), an extensible library designed to streamline the development and evaluation of gradient-based one-shot NAS methods. Confopt provides a minimal API that makes it easy for users to integrate new search spaces, while also supporting the decomposition of NAS optimizers into their core components. We use this framework to create a suite of new DARTS-based benchmarks, and combine them with a novel evaluation protocol to reveal a critical flaw in how gradient-based one-shot NAS methods are currently assessed. The code can be found at https://github.com/automl/ConfigurableOptimizer.

Figures

Figures reproduced from arXiv: 2507.16533 by the authors.

**Figure 1.** An overview of DARTS-Bench-Suite. The benchsuite consists of nine benchmarks, which are a combination of three supernet variants with three sets of candidate operations. The NAS method is run on each benchmark thrice. Then, the architecture from the trial with the lowest validation loss is chosen to be evaluated for each benchmark. A discrete model with this architecture is then trained on nine different hyperparame… view at source ↗

**Figure 2.** Performance of the NAS methods on all nine benchmarks across nine evaluations. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

**Figure 3.** CD Plots of the NAS Methods The results are shown in [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Correlation between benchmarks (based on the best test accuracy obtained per NAS method) 15 [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]

**Figure 5.** Figure 5: Correlation between benchmarks (based on the [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]

**Figure 6.** Figure 6: Win rates of NAS methods (best test accuracy) Method Mean Test Accuracy (%) Maximum Test Accuracy (%) DARTS 81.19 ± 6.03 87.35 GDAS 78.48 ± 6.63 86.54 PC-DARTS 75.37 ± 9.16 86.02 Fair DARTS 79.19 ± 8.80 88.15 SmoothDARTS 77.87 ± 8.74 87.06 DrNAS 83.04 ± 6.36 89.49 OLES…

**Figure 7.** Figure 7: Win rates of NAS methods (mean test accuracy) Method Mean Test Accuracy (%) Maximum Test Accuracy (%) DARTS 87.63 ± 2.31 90.41 GDAS 88.30 ± 2.16 91.30 PC-DARTS 87.57 ± 2.53 90.93 Fair DARTS 87.35 ± 2.53 90.73 SmoothDARTS 86.51 ± 2.48 89.80 DrNAS 87.91 ± 2.50 90.94 OLES…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

40 extracted references · 35 canonical work pages

[1]

Baker, B., Gupta, O., Naik, N., and Raskar, R. (2017). Designing neural network architectures using reinforcement learning. In The Fifth International Conference on Learning Representations ( ICLR '17) . ICLR

work page 2017
[2]

Bi, K., Hu, C., Xie, L., Chen, X., Wei, L., and Tian, Q. (2019). Stabilizing darts with amended gradient estimation on architectural parameters. arXiv:1910.11831 [cs.LG]

work page Pith review arXiv 2019
[3]

Biewald, L. (2020). Experiment tracking with weights and biases. Software available from wandb.com

work page 2020
[4]

Cai, H., Zhu, L., and Han, S. (2019). ProxylessNAS : Direct neural architecture search on target task and hardware. In The Seventh International Conference on Learning Representations ( ICLR '19) . ICLR

work page 2019
[5]

and Hsieh, C.-J

Chen, X. and Hsieh, C.-J. (2020). Stabilizing differentiable architecture search via perturbation-based regularization. In Daume III , H. and Singh, A., editors, Proceedings of the 37th International Conference on Machine Learning ( ICML '20) , volume 98, pages 1554--1565. PMLR, Proceedings of Machine Learning Research

work page 2020
[6]

Chen, X., Wang, R., Cheng, M., Tang, X., and Hsieh, C.-J. (2021). DrNAS : Dirichlet neural architecture search. In The Ninth International Conference on Learning Representations ( ICLR '21) . ICLR

work page 2021
[7]

Chen, X., Xie, L., Wu, J., and Tian, Q. (2019). Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In Proceedings of the 22nd IEEE/CVF International Conference on Computer Vision ( ICCV '19) , pages 1294--1303. Computer Vision Foundation and IEEE Computer Society, IEEE

work page 2019
[8]

Chu, X., Wang, X., Zhang, B., Lu, S., Wei, X., and Yan, J. (2021). DARTS- : Robustly stepping out of performance collapse without indicators. In The Ninth International Conference on Learning Representations ( ICLR '21) . ICLR. arXiv preprint arXiv:2009.01027

arXiv 2021

Show all 40 references

[9]

Chu, X., Zhou, T., Zhang, B., and Li, J. (2020). Fair DARTS : Eliminating unfair advantages in differentiable architecture search. In Vedaldi, A., Bischof, H., Brox, T., and Frahm, J., editors, 16th European Conference on Computer Vision ( ECCV '20) , pages 465--480. Springer,...

2020
[10]

and Yang, Y

Dong, X. and Yang, Y. (2019). Searching for a robust neural architecture in four gpu hours. In Proceedings of the International Conference on Computer Vision and Pattern Recognition ( CVPR '19) . Computer Vision Foundation and IEEE Computer Society, IEEE

2019
[11]

and Yang, Y

Dong, X. and Yang, Y. (2020). NAS-Bench-201 : Extending the scope of reproducible neural architecture search. In The Eigth International Conference on Learning Representations ( ICLR '20) . ICLR

2020
[12]

Jiang, S., Ji, Z., Zhu, G., Yuan, C., and Huang, Y. (2023). Operation-level early stopping for robustifying differentiable NAS . In Proceedings of the 37th International Conference on Neural Information Processing Systems , NIPS '23

2023
[13]

Kingma, D. P. and Ba, J. (2015). Adam: A method for stochastic optimization. In The Third International Conference on Learning Representations ( ICLR '15) . ICLR. Published online: iclr.cc

2015
[14]

K., Moradian, S., Rapp, M., and Hutter, F

Krishnakumar, A., Jha, A. K., Moradian, S., Rapp, M., and Hutter, F. (2024). LoRA-DARTS : Low rank adaptation for differentiable architecture search. In Lindauer, M., Eggensperger, K., Garnett, R., Vanschoren, J., and Gardner, J., editors, AutoML Conference 2024 (Workshop Trac...

2024
[15]

Krizhevsky, A. (2009). Learning multiple layers of features from tiny images. Technical report, University of Toronto

2009
[16]

Li, L., Khodak, M., Balcan, M.-F., and Talwalkar, A. (2021). Geometry-Aware gradient algorithms for neural architecture search. In The Ninth International Conference on Learning Representations ( ICLR '21) . ICLR

2021
[17]

Liang, H., Zhang, S., Sun, J., He, X., Huang, W., Zhuang, K., and Li, Z. (2019). DARTS+ : Improved differentiable architecture search with early stopping. arXiv:1909.06035 [cs.CV]

2019 arXiv
[18]

Liu, C., Zoph, B., Neumann, M., Shlens, J., Hua, W., Li, L.-J., Fei-Fei, L., Yuille, A., Huang, J., and Murphy, K. (2018). Progressive neural architecture search. In Ferrari, V., Herbert, M., Sminchisescu, C., and Weiss, Y., editors, 14th European Conference on Computer Vision...

2018
[19]

Liu, H., Simonyan, K., and Yang, Y. (2019). DARTS : Differentiable architecture search. In The Seventh International Conference on Learning Representations ( ICLR '19) . ICLR

2019
[20]

Mehta, Y., White, C., Zela, A., Krishnakumar, A., Zabergja, G., Moradian, S., Safari, M., Yu, K., and Hutter, F. (2022). NAS-Bench-Suite : NAS evaluation is (now) surprisingly easy. In The Tenth International Conference on Learning Representations ( ICLR '22) . ICLR

2022
[21]

Movahedi, S., Adabinejad, M., Imani, A., Keshavarz, A., Dehghani, M., Shakery, A., and Araabi, B. N. (2023). -DARTS : Mitigating performance collapse by harmonizing operation selection among cells

2023
[22]

and Gordon, G

Negrinho, R. and Gordon, G. (2017). DeepArchitect : Automatically designing and training deep architectures. arXiv preprint arXiv:1704.08792

2017 arXiv
[23]

Negrinho, R., Patil, D., Le, N., Ferreira, D., Gormley, M., and Gordon, G. (2019). Towards modular and programmable architecture search. Neural Information Processing Systems

2019
[24]

Ning, X., Tang, C., Li, W., Zhou, Z., Liang, S., Yang, H., and Wang, Y. (2021). Evaluating efficient performance estimators of neural architectures. arXiv:2008.03064 [cs.CV]

2021 arXiv
[25]

Y., Zoph, B., Le, Q

Pham, H., Guan, M. Y., Zoph, B., Le, Q. V., and Dean, J. (2018). Efficient neural architecture search via parameter sharing. In Dy, J. and Krause, A., editors, Proceedings of the 35th International Conference on Machine Learning ( ICML '18) , volume 80. Proceedings of Machine ...

2018
[26]

Real, E., Aggarwal, A., Huang, Y., and Le, Q. V. (2019). Regularized evolution for image classifier architecture search. In Hentenryck, P. V. and Zhou, Z., editors, Proceedings of the Thirty-Third Conference on Artificial Intelligence ( AAAI '19) , volume 33, pages 4780--4789....

2019
[27]

Real, E., Moore, S., Selle, A., Saxena, S., Suematsu, Y., Le, Q., and Kurakin, A. (2017). Large-scale evolution of image classifiers. In Precup, D. and Teh, Y., editors, Proceedings of the 34th International Conference on Machine Learning ( ICML '17) , volume 70. Proceedings o...

2017
[28]

Shah, S., Dey, D., de Rosa, G., Mendes, C., Kauffmann, P., Lovett, C., Del Giorno, A., Javaheripi, M., Ofer, D., and Research, M. (2020). Archai

2020
[29]

S., Krishnakumar, A., Safari, M., and Hutter, F

Sukthanker, R. S., Krishnakumar, A., Safari, M., and Hutter, F. (2024). Weight-entanglement meets gradient-based neural architecture search. In Lindauer, M., Eggensperger, K., Garnett, R., Vanschoren, J., and Gardner, J., editors, Third International Conference on Automated Ma...

2024
[30]

Tu, R., Roberts, N., Khodak, M., Shen, J., Sala, F., and Talwalkar, A. (2022). NAS-Bench-360 : Benchmarking neural architecture search on diverse tasks. In Proceedings of the 36th International Conference on Neural Information Processing Systems , NIPS '22

2022
[31]

Wang, R., Cheng, M., Chen, X., Tang, X., and Hsieh, C.-J. (2021). Rethinking architecture selection in differentiable NAS . In International Conference on Learning Representation . ICLR

2021
[32]

S., Ru, B., Elsken, T., Zela, A., Dey, D., and Hutter, F

White, C., Safari, M., Sukthanker, R. S., Ru, B., Elsken, T., Zela, A., Dey, D., and Hutter, F. (2023). Neural architecture search: Insights from 1000 papers. arXiv:2301.08727 [cs.LG]

2023 arXiv
[33]

Xu, Y., Xie, L., Zhang, X., Chen, X., Qi, G.-J., Tian, Q., and Xiong, H. (2019). PC-DARTS : Partial channel connections for memory-efficient architecture search. arXiv:1907.05737 [cs.CV]

2019 arXiv
[34]

M., and Carlucci, F

Yang, A., Esperan c a, P. M., and Carlucci, F. M. (2020). NAS evaluation is frustratingly hard. In The Eigth International Conference on Learning Representations ( ICLR '20) . ICLR

2020
[35]

Ying, C., Klein, A., Real, E., Christiansen, E., Murphy, K., and Hutter, F. (2019). NAS-Bench-101 : Towards reproducible neural architecture search. In Chaudhuri, K. and Salakhutdinov, R., editors, Proceedings of the 36th International Conference on Machine Learning ( ICML '19...

2019
[36]

Zela, A., Elsken, T., Saikia, T., Marrakchi, Y., Brox, T., and Hutter, F. (2020). Understanding and robustifying differentiable architecture search. In The Eigth International Conference on Learning Representations ( ICLR '20) . ICLR

2020
[37]

Zela, A., Siems, J., Zimmer, L., Lukasik, J., Keuper, M., and Hutter, F. (2022). Surrogate NAS benchmarks: Going beyond the limited search spaces of tabular NAS benchmarks. In The Tenth International Conference on Learning Representations ( ICLR '22) . ICLR

2022
[38]

and Ding, Z

Zhang, J. and Ding, Z. (2023). Rethink darts search space and renovate a new benchmark. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J., editors, International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Re...

2023
[39]

and Le, Q

Zoph, B. and Le, Q. V. (2017). Neural architecture search with reinforcement learning. In The Fifth International Conference on Learning Representations ( ICLR '17) . ICLR

2017
[40]

Zoph, B., Vasudevan, V., Shlens, J., and Le, Q. V. (2018). Learning transferable architectures for scalable image recognition. In Proceedings of the International Conference on Computer Vision and Pattern Recognition ( CVPR '18) . Computer Vision Foundation and IEEE Computer S...

2018

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Baker, B., Gupta, O., Naik, N., and Raskar, R. (2017). Designing neural network architectures using reinforcement learning. In The Fifth International Conference on Learning Representations ( ICLR '17) . ICLR

work page 2017

[2] [2]

Bi, K., Hu, C., Xie, L., Chen, X., Wei, L., and Tian, Q. (2019). Stabilizing darts with amended gradient estimation on architectural parameters. arXiv:1910.11831 [cs.LG]

work page Pith review arXiv 2019

[3] [3]

Biewald, L. (2020). Experiment tracking with weights and biases. Software available from wandb.com

work page 2020

[4] [4]

Cai, H., Zhu, L., and Han, S. (2019). ProxylessNAS : Direct neural architecture search on target task and hardware. In The Seventh International Conference on Learning Representations ( ICLR '19) . ICLR

work page 2019

[5] [5]

and Hsieh, C.-J

Chen, X. and Hsieh, C.-J. (2020). Stabilizing differentiable architecture search via perturbation-based regularization. In Daume III , H. and Singh, A., editors, Proceedings of the 37th International Conference on Machine Learning ( ICML '20) , volume 98, pages 1554--1565. PMLR, Proceedings of Machine Learning Research

work page 2020

[6] [6]

Chen, X., Wang, R., Cheng, M., Tang, X., and Hsieh, C.-J. (2021). DrNAS : Dirichlet neural architecture search. In The Ninth International Conference on Learning Representations ( ICLR '21) . ICLR

work page 2021

[7] [7]

Chen, X., Xie, L., Wu, J., and Tian, Q. (2019). Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In Proceedings of the 22nd IEEE/CVF International Conference on Computer Vision ( ICCV '19) , pages 1294--1303. Computer Vision Foundation and IEEE Computer Society, IEEE

work page 2019

[8] [8]

Chu, X., Wang, X., Zhang, B., Lu, S., Wei, X., and Yan, J. (2021). DARTS- : Robustly stepping out of performance collapse without indicators. In The Ninth International Conference on Learning Representations ( ICLR '21) . ICLR. arXiv preprint arXiv:2009.01027

arXiv 2021