OptiProxy-NAS: Optimization Proxy based End-to-End Neural Architecture Search

Bo Lyu; Ke Li; Tuo Shi; Yu Cui

REVIEW 3 major objections 5 minor 58 references

OptiProxy-NAS: Optimization Proxy based End-to-End Neural Architecture Search

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that neural architecture search can be turned into a differentiable end-to-end optimization problem by maximizing a learned smooth predictor over Gumbel-softmax relaxed architecture variables, and that this gradient-based…

desk verdict Strong empirical NAS results undermined by a wrong theoretical claim about unbiased gradients; the method may still work as a heuristic, but the paper needs major revision. read the letter →

arxiv 2509.05656 v1 pith:OJQZN6ON submitted 2025-09-06 cs.LG cs.AI

Bo Lyu , Yu Cui , Tuo Shi , Ke Li This is my paper · ORCID

classification cs.LGcs.AI

keywords neuralarchitecturesearchdifferentiableGumbel-softmaxrelaxationoptimizationproxypredictor-guidedhardware-awareNASlow-fidelityevaluationNAS-Bench-201

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes OptiProxy-NAS, which treats neural architecture search as a differentiable optimization problem rather than a discrete black-box search. Its central claim is that training a smooth predictor over relaxed architecture variables and then ascending its gradient through Gumbel-softmax approximations of categorical operation choices and Bernoulli edge decisions drives sampling toward high-performance architectures far more efficiently than predictor-guided discrete samplers. On NAS-Bench-201 the method reaches the global optimum in 100 queries, and on NAS-Bench-301 and NAS-Bench-NLP it claims a roughly fourfold query-efficiency improvement over the strongest published baselines. The same framework extends to hardware-aware objectives, beating AG-Net on all 12 HW-NAS-Bench latency settings with 50 percent fewer queries, and to low-fidelity evaluation with only about a third of the training budget. The load-bearing assumption is that the softened-sample gradient is a valid proxy for how true accuracy would change, a premise the current convergence proof does not fully establish.

What carries the argument

The central object is the optimization proxy: a differentiable predictor $\hat{f}_\theta$, typically a two-layer GCN, that maps a graph encoding of an architecture, one-hot operation features plus an adjacency matrix, to a target metric. Around it, the method relaxes architecture variables into $\alpha$ and $\beta$, reparameterizes discrete sampling through Gumbel-softmax and sigmoid-Gumbel samples $\hat{s}_o$ and $\hat{s}_t$, and then runs gradient ascent from the predictor output back to $\alpha$ and $\beta$. The Gumbel relaxation is what creates the differentiable forward path; temperature decay balances exploration and exploitation; a sequential model-based optimization loop alternately refits the predictor on queried architectures and uses the proxy to propose the next batch, while multiple $(\alpha,\beta)$ groups initialized by Latin hypercube sampling add parallel exploration.

What would settle it

A control experiment on NAS-Bench-201 that replaces the backpropagated gradients with random directions, keeping the same Gumbel noise, temperature schedule, predictor refits, and sampling, would settle whether gradient direction is doing the work: if the random-gradient control reaches the same optima, the central gradient-validity claim is falsified. A sharper check is to estimate, over many update steps, the correlation between $\nabla_{\alpha,\beta}\hat{f}_\theta(\hat{s})$ and the actual change in true accuracy of discrete architectures sampled after the update, where zero or negative correlation refutes the proxy-gradient claim.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that the NAS objective over the discrete architecture space can be replaced by an end-to-end differentiable objective over relaxed variables $\alpha$ (operation logits) and $\beta$ (edge log-odds), fit through a learned surrogate $\hat{f}_\theta$ and searched by gradient ascent, provided the discrete sampling function $s$ is replaced by its Gumbel-softmax relaxation $\hat{s}$. The paper argues that as the temperature $\tau\to 0$, $\hat{s}$ converges in probability to the categorical and Bernoulli samples $s$ (Eqs. 11 to 12), and therefore the proxy gradients $\partial\hat{f}/\partial\hat{s}\cdot\partial\hat{s}/\partial\alpha$ approximate the original gradients (Eq. 13). This is what the paper takes to make the relaxation unbiased and to let one pipeline optimize accuracy, latency, or low-fidelity metrics. On its benchmarks, the claim is that this single framework reaches global optima faster than predictor-guided RL, EA, and BO samplers, and faster than DARTS-style supernetwork methods, while adding negligible overhead.

Load-bearing premise

The approach depends on the untested premise that the gradient of the learned predictor through the softened random samples points toward architectures that actually improve true accuracy; if that gradient signal is misleading, the search is following noise rather than the real landscape.

Editorial extensions

If this is right

On NAS-Bench-201, reaching the global optimum with roughly 100 queries on CIFAR-10 and CIFAR-100 and 280 on ImageNet16-120 would make search cost nearly negligible for tabular spaces.
On NAS-Bench-301 and NAS-Bench-NLP, a fourfold query-efficiency gain over the compared generative baselines would let surrogate-space search run under much tighter query budgets.
On HW-NAS-Bench, beating all 12 latency-constrained settings with 50 percent fewer queries would make deployment-aware NAS practical on edge devices where each architecture evaluation is expensive.
The reported overhead of 20 seconds and 10MB, or 0.037 percent of runtime and 0.25 percent of memory, would make the optimization proxy a drop-in search module on top of existing predictors and graph encodings.
The demonstrated compatibility with 5-epoch low-fidelity evaluation, cutting total training epochs by factors of roughly 2 to 6 on NAS-Bench-201 tasks, suggests the framework can combine with multi-fidelity search strategies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's own Section 2.2 notes that the discrete sampling function is not differentiable, so the proof gap sits exactly at Eq. (13): the convergence proof in Appendix A.5 establishes convergence of samples, not of reparameterization gradients; a score-function estimator comparison on a small tabular space would reveal whether gradient validity is the active ingredient.
If the central premise generalizes, the predict-then-ascend template should transfer to other expensive discrete design spaces, such as chip floorplans or molecule graphs, where a differentiable surrogate can be trained; this paper does not test that transfer, so it remains an extension.
The reported gains might come substantially from the exploration induced by Gumbel noise, temperature decay, and multi-$\alpha,\beta$ starts rather than from gradient direction; a random-gradient control under the same sequential model-based loop would separate these causes.
On the noisier ImageNet16-120 portion of NAS-Bench-201 the method still needs 280 queries, suggesting the smooth-proxy benefit shrinks as the landscape becomes more rugged; a natural extension is roughness-adaptive temperature and multi-start scheduling.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the empirical work is real and fairly broad; the theoretical core, as written, is not. I would send this to review, but the authors need to fix the gradient story before publication.

What is new: they relax both operation choices (categorical) and topology (Bernoulli) with Gumbel-softmax, then run gradient ascent on the relaxed variables through a learned predictor inside an SMBO loop. NAO optimizes an embedding, DARTS uses supernets, AG-Net/DiNAS use discrete generative sampling. This specific combination is novel, and the experiments are extensive: four search spaces, 12 tasks, hardware latency constraints, low-fidelity settings, ablations, and 10 seeds. The query-efficiency numbers are credible: reaching NB201 optimum in 100 queries, beating AG-Net on HW-NAS with half the queries. If those numbers reproduce, the method is useful.

The soft spot is load-bearing. Eq. (13) claims that as tau goes to 0 the relaxed gradient dfhat/dshat * dshat/dalpha converges to dfhat/ds * ds/dalpha, but ds/dalpha does not exist for discrete sampling, as they themselves say in Section 2.2. Appendix A.5 proves only convergence in probability of the soft samples to the hard samples. That says nothing about gradients. Pathwise, the derivative of the one-hot limit is zero almost everywhere, and their own implementation uses min temperature 0.2, far from zero. So the central "unbiased relaxation" claim is unsupported, and the search is not justified by the given theory.

There is also a confusing contradiction about the proxy model: Section 3 and Appendix A.4 describe a GCN, but Appendix A.7.1 says they use XGBoost as the surrogate, matching AG-Net and DiNAS. If the fitted model is XGBoost, there is no gradient path through it at all, which would make the whole mechanism incoherent. Either the text is wrong, or the method is different from what the main body describes. They need to state exactly which model produced each table.

The empirical results could still be real, because the relaxed objective is a smooth surrogate that may guide search even without unbiasedness. The paper should be reframed that way: a continuous relaxation with annealed temperature, not an unbiased gradient estimator. That is a defensible heuristic, and the experiments would support it.

Who this is for: people doing predictor-based NAS or Gumbel-softmax optimization. It deserves a serious referee, with major revision requested. The experiments are worth engaging, but the theory section and the model description must be corrected or removed before I would trust the claims.

Referee Report

3 major / 5 minor

Summary. The paper proposes OptiProxy-NAS, a predictor-guided NAS framework in which the discrete architecture distribution parameters (alpha, beta) are optimized by gradient ascent through a differentiable relaxation (Gumbel-softmax / binary concrete) and a fitted differentiable proxy model. The proxy model is retrained iteratively on queried architectures. The authors claim that this relaxation makes the search unbiased and end-to-end differentiable, and they report strong query-efficiency results on NAS-Bench-101/201/301, NAS-Bench-NLP, and HW-NAS-Bench, with 10-seed statistics, ablations, low-fidelity experiments, and a plan for code release.

Significance. If the central claims held, this would be a meaningful step for NAS: reaching the global optimum on NAS-Bench-201 with 100 queries, achieving roughly 4x query efficiency on NAS-Bench-301 and NAS-Bench-NLP, and consistently beating published results on 12 HW-NAS-Bench latency settings with half the query budget are impressive empirical results. The paper also provides 10-seed runs, ablation studies, low-fidelity comparisons, and a concrete resource-cost analysis. However, the theoretical foundation of the gradient proxy is not established, and there is a major internal inconsistency about which proxy model actually produced the reported numbers. The empirical evidence is therefore not currently supported by a sound explanation of why the method works.

major comments (3)

[§2.3, Eq. (13); Appendix A.5] The central claim that the Gumbel-softmax relaxation gives unbiased proxy gradients is unsupported. Appendix A.5 proves only that the relaxed samples converge in probability to the discrete samples as τ→0 (Eqs. A.7 and A.11), which is a statement about random variables, not about derivatives. The right-hand side of Eq. (13), ∂f̂/∂s · ∂s/∂α, is not even defined because s is discrete, as the paper itself states in §2.2. Moreover, for the Gumbel-softmax, ∂ŝ_j/∂α_i = (1/τ)(ŝ_i δ_ij − ŝ_i ŝ_j), so the pathwise derivative does not converge to any gradient of the discrete sampler; at the implemented minimum temperature τ=0.2 (Table 6 in Appendix A.9.1) the τ→0 regime is never approached. The authors should either supply a correct proof of a gradient relationship, replace the claim with a weaker statement, or use an estimator with known gradient properties.
[§3, §A.4, §A.7.1] The manuscript is internally inconsistent about which proxy model generates the reported results. Main text §3 and Appendix A.4 describe a two-layer GCN with 139.7K parameters and standard backpropagation, while Appendix A.7.1 states that XGBoost is employed as the surrogate model, consistent with AG-Net and DiNAS, and that NAS-Bench-301 uses XGBoost and NAS-Bench-NLP uses 'svd lgb'. XGBoost and LightGBM are not differentiable with respect to their inputs in the sense required by Eq. (8), so the end-to-end gradient path in Figure 3 cannot exist for those experiments. The authors must clarify which surrogate actually produced each table and, if tree-based models were used, explain how gradients through the surrogate were obtained; this is load-bearing for the method's central claim.
[Appendix A.8.1 and §2.3] The treatment of the NAS-Bench-201 adjacency is a simplifying assumption that is acknowledged but potentially consequential. Figure 6 shows that choosing 'none' or 'skip_connect' changes the computational graph, yet the encoding used for the proxy (Appendix A.8.1) fixes the adjacency matrix to the canonical form. Because the gradient signal in Eq. (13) is computed with respect to this fixed-topology encoding, the search objective does not represent the true architecture space for a large fraction of candidates. The authors should justify that this simplification does not materially bias the reported query-efficiency numbers, or evaluate the method with an encoding that captures the coupling.

minor comments (5)

[Abstract and §3.1] The phrase 'nearly 100% less query cost' is mathematically odd; 100% less would be zero queries. Please clarify whether the intended meaning is 'roughly half the queries' or 'about 50% fewer queries' as stated elsewhere.
[§2.2, Eqs. (9) and (10)] The index bounds in the relaxed sampling functions appear to be off by one: Eq. (9) writes '0≤i<N−1, 0≤j<M−1' and Eq. (10) writes '0≤h,k<N−1', but the softmax and sigmoid are defined over all M operations and N nodes. Please correct the bounds.
[Appendix A.8.3] The sentence 'The candidate operation set of NAS-Bench-101 space is as follows' appears at the start of the NAS-Bench-301 subsection; it should refer to NAS-Bench-301.
[Table 1] The ImageNet16-120 entry for OptiProxy-NAS at 280 queries, marked '46.77*', is hard to parse together with the footnote 'Reach * even with 70% queries'. Please clarify which starred value is actually matched and at what query budget.
[Appendix A.9.1, Table 6] The description of 'mintemp' says it 'determines the unbiased nature of the proxy gradient and the original gradients, commonly set close to 0', but the reported value is 0.2. Please reconcile this with the claimed τ→0 unbiasedness.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular reduction found: the reported architectures are independently evaluated on public benchmarks after surrogate-guided search, and the only questionable step (Eq. 13 gradient relaxation) is a technical gap rather than a definitional or self-citation circularity.

full rationale

OptiProxy-NAS's derivation chain is a standard predictor-guided loop with a differentiable relaxation. The surrogate f_hat is fit to queried architectures (Eq. 3 and Algorithm 2), and the final reported architecture is the historical best in the evaluated set D, so the headline results are measured against public benchmark ground-truth labels rather than read back from the fitted model. The Gumbel-softmax relaxation (Eqs. 9-10) and the 'optimization proxy' terminology are attributed to external works (Jang et al., 2017; Maddison et al., 2017; Chen et al., 2021), not to the present authors' prior results, so no self-citation chain carries the argument. The one substantive flaw is the jump in Section 2.3/Eq. (13): the appendix proves only that the Gumbel-softmax random variables converge in probability to the categorical/Bernoulli variables (Eqs. A.7 and A.11), which does not imply convergence of the derivatives; indeed the paper itself states that the discrete sampling function makes Eq. (7) not calculable, so the 'original gradient' in Eq. (13) is not a well-defined target. This is an omitted justificatory step and a correctness concern, but it is not circular: the claim is not obtained by defining the output in terms of the fitted input, by renaming a fitted parameter as a prediction, or by relying on the authors' own prior theorems. The empirical query-efficiency comparisons are independently grounded, so no circularity score is warranted.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim depends on the surrogate's predictive quality, the smoothness of the relaxed landscape, and a gradient-approximation step that is not rigorously established. The only mathematically standard axiom is the Concrete distribution consistency. No new physical entities are introduced.

free parameters (4)

Gumbel temperature schedule (base_temp, min_temp) = 0.7 -> 0.2
Annealing temperature from 0.7 to 0.2 is hand-chosen to balance exploration and exploitation; the paper advises it is commonly set in [0.5, 1].
Learning rates for alpha and beta = 0.02 / 0.001
Learning rates for the relaxed architecture variables are empirical choices reported in Table 6 and must be co-adjusted with temperature.
GCN hidden size and layers = 144 hidden, 2 layers
Proxy model capacity chosen by hand; the paper says two layers are sufficient across all tasks.
Search-loop hyperparameters (search_epochs, o_epochs, t_epochs, parallel_batch, num_sample, num_sample_init… = 300, 15, 15, 5, 10, 10, 9, 0.9-0.1
Empirically adjusted per search space; the paper states search and sampling parameters need to be adjusted according to space scale and query budget.

assumptions (4)

domain assumption A DAG architecture can be faithfully represented by node operation one-hots and an adjacency matrix, and a GCN can predict its accuracy well enough to guide search.
Used throughout; encoding scheme in Section 2.5 and Appendix A.8; the paper's own Fig. 2 shows imperfect prediction scatter.
ad hoc to paper The relaxed proxy space X spanned by (α, β) is smooth enough that gradient ascent on the surrogate landscape moves toward high-performance regions.
This is the core hypothesis stated in the Introduction; no smoothness measure or proof is provided, only empirical evidence.
ad hoc to paper The Bernoulli relaxation treats each edge independently, and coupling between operation choice and adjacency can be neglected for NB201.
Acknowledged in Appendix A.5 as simplifying; Appendix A.8.1 states coupling factors 'can be neglected' for the encoding.
standard math Gumbel-softmax samples converge to the discrete categorical/Bernoulli distribution as temperature goes to 0 (standard Concrete distribution results).
Invoked in Eq. (11)-(12) and Appendix A.5, citing Maddison et al. 2017; the paper then over-extends this to gradient convergence.

how reviews work

0 comments

Cite this review

Pith. "Pith review of OptiProxy-NAS: Optimization Proxy based End-to-End Neural Architecture Search." pith.science (2026). https://pith.science/paper/OJQZN6ON

@misc{pith2026250905656,
  author       = {Pith},
  title        = {Pith review of: OptiProxy-NAS: Optimization Proxy based End-to-End Neural Architecture Search},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OJQZN6ON}},
  note         = {Machine review of arXiv:2509.05656}
}

abstract

Neural architecture search (NAS) is a hard computationally expensive optimization problem with a discrete, vast, and spiky search space. One of the key research efforts dedicated to this space focuses on accelerating NAS via certain proxy evaluations of neural architectures. Different from the prevalent predictor-based methods using surrogate models and differentiable architecture search via supernetworks, we propose an optimization proxy to streamline the NAS as an end-to-end optimization framework, named OptiProxy-NAS. In particular, using a proxy representation, the NAS space is reformulated to be continuous, differentiable, and smooth. Thereby, any differentiable optimization method can be applied to the gradient-based search of the relaxed architecture parameters. Our comprehensive experiments on $12$ NAS tasks of $4$ search spaces across three different domains including computer vision, natural language processing, and resource-constrained NAS fully demonstrate the superior search results and efficiency. Further experiments on low-fidelity scenarios verify the flexibility.

Figures

Figures reproduced from arXiv: 2509.05656 by the authors.

**Figure 2.** Illustration of the differentiable search strategy. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Gradient calculation in the computation graph. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (11 more)

**Figure 4.** Figure 4: Comparison results of low-fidelity (5 epochs) search and full training (200 epochs) search ( [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

**Figure 1.** Figure 1: Framework for Optimization Proxy based end-to-end Neural Architecture Search (OptiProxy-NAS). without requiring exhaustive training. The fundamental principle is to leverage saliency criteria, e.g., Fisher, Grad Norm, Grasp, Jac. Cov., SynFlow, SNIP (Mellor et al., 20…

**Figure 2.** Figure 2: The scatter of the evaluated architectures. We distinguish the sampling order of the architecture points by coloring [PITH_FULL_IMAGE:figures/full_fig_p015_2.png]

**Figure 3.** Figure 3: The trajectories in each subplot illustrate the statistics of the best-so-far values versus the query number across [PITH_FULL_IMAGE:figures/full_fig_p017_3.png]

**Figure 5.** Figure 5: (a) Illustration of search space of NAS-Bench-201. [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]

**Figure 4.** Figure 4: Visualization of the search space architecture dis [PITH_FULL_IMAGE:figures/full_fig_p019_4.png]

**Figure 7.** Figure 7: Illustration of the search space of NAS-Bench [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]

**Figure 9.** Figure 9: The illustration of the adjacency matrix of the sam [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]

**Figure 8.** Figure 8: Illustration of the search space of NAS-Bench [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]

**Figure 11.** Figure 11: The discrete sampling of topological structure [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]

**Figure 12.** Figure 12: The combination of adjacency matrices of “Nor [PITH_FULL_IMAGE:figures/full_fig_p023_12.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 52 canonical work pages

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
[2]

S., Mehrotra, A., Dudziak, L., and Lane, N

Abdelfattah, M. S., Mehrotra, A., Dudziak, L., and Lane, N. D. Zero-cost proxies for lightweight NAS . In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021

work page 2021
[3]

An, S., Lee, H., Jo, J., Lee, S., and Hwang, S. J. Diffusionnag: Task-guided neural architecture generation with diffusion models. CoRR, abs/2305.16943, 2023

work page Pith review arXiv 2023
[4]

Multi-conditioned Graph Diffusion for Neural Architecture Search

Asthana, R., Conrad, J., Dawoud, Y., Ortmanns, M., and Belagiannis, V. Multi-conditioned graph diffusion for neural architecture search. CoRR, abs/2403.06020, 2024

work page Pith review arXiv 2024
[5]

Accelerating neural architecture search using performance prediction

Baker, B., Gupta, O., Raskar, R., and Naik, N. Accelerating neural architecture search using performance prediction. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Workshop Track Proceedings . OpenReview.net, 2018

work page 2018
[6]

M., and Weston, N

Brock, A., Lim, T., Ritchie, J. M., and Weston, N. Smash: one-shot model architecture search through hypernetworks. In International Conference on Learning Representations, Vancouver, BC, Canada, pp.\ 1--21, 2018 a

work page 2018
[7]

M., and Weston, N

Brock, A., Lim, T., Ritchie, J. M., and Weston, N. SMASH: one-shot model architecture search through hypernetworks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net, 2018 b

work page 2018
[8]

Proxylessnas: Direct neural architecture search on target task and hardware

Cai, H., Zhu, L., and Han, S. Proxylessnas: Direct neural architecture search on target task and hardware. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019

work page 2019

Show all 58 references

[9]

Chen, W., Park, S., Tanneau, M., and Hentenryck, P. V. Learning optimization proxies for large-scale security-constrained economic dispatch. CoRR, abs/2112.13469, 2021

2021 arXiv
[10]

Chen, W., Tanneau, M., and Hentenryck, P. V. End-to-end feasible optimization proxies for large-scale economic dispatch. CoRR, abs/2304.11726, 2023

2023 arXiv
[11]

Progressive differentiable architecture search: bridging the depth gap between search and evaluation

Chen, X., Xie, L., Wu, J., and Tian, Q. Progressive differentiable architecture search: bridging the depth gap between search and evaluation. In IEEE/CVF International Conference on Computer Vision, Seoul, Korea (South) , pp.\ 1294--1303. IEEE , 2019

2019
[12]

and Yang, Y

Dong, X. and Yang, Y. Nas-bench-201: Extending the scope of reproducible neural architecture search. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020

2020
[13]

S., Lee, R., Kim, H., and Lane, N

Dudziak, L., Chau, T., Abdelfattah, M. S., Lee, R., Kim, H., and Lane, N. D. BRP-NAS: prediction-based NAS using gcns. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural...

2020
[14]

predict, then optimize

Elmachtoub, A. N. and Grigas, P. Smart "predict, then optimize". Manag. Sci., 68 0 (1): 0 9--26, 2022

2022
[15]

Single path one-shot neural architecture search with uniform sampling

Guo, Z., Zhang, X., Mu, H., Heng, W., Liu, Z., Wei, Y., and Sun, J. Single path one-shot neural architecture search with uniform sampling. In Vedaldi, A., Bischof, H., Brox, T., and Frahm, J. (eds.), European Conference on Computer Vision, Glasgow, UK, volume 12361 of Lecture ...

2020
[16]

and Chu, W

Huang, S. and Chu, W. Searching by generating: Flexible and efficient one-shot NAS with architecture generator. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021 , pp.\ 983--992. Computer Vision Foundation / IEEE , 2021

2021
[17]

Categorical reparameterization with gumbel-softmax

Jang, E., Gu, S., and Poole, B. Categorical reparameterization with gumbel-softmax. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017

2017
[18]

Nas-bench-nlp: neural architecture search benchmark for natural language processing

Klyuchnikov, N., Trofimov, I., Artemova, E., Salnikov, M., Fedorov, M., Filippov, A., and Burnaev, E. Nas-bench-nlp: neural architecture search benchmark for natural language processing. IEEE Access, 10: 0 45736--45747, 2022

2022
[19]

D., Christopher, J., Hentenryck, P

Kotary, J., Vito, V. D., Christopher, J., Hentenryck, P. V., and Fioretto, F. Predict-then-optimize by proxy: Learning joint models of prediction and optimization. CoRR, abs/2311.13087, 2023

2023 arXiv
[20]

Lee, H., Hyung, E., and Hwang, S. J. Rapid neural architecture search by learning to generate graphs from datasets. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021

2021
[21]

Hw-nas-bench: Hardware-aware neural architecture search benchmark

Li, C., Yu, Z., Fu, Y., Zhang, Y., Zhao, Y., You, H., Yu, Q., Wang, Y., Hao, C., and Lin, Y. Hw-nas-bench: Hardware-aware neural architecture search benchmark. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenR...

2021
[22]

C., M \" u ller, M., Thabet, A

Li, G., Qian, G., Delgadillo, I. C., M \" u ller, M., Thabet, A. K., and Ghanem, B. SGAS: sequential greedy architecture search. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020 , pp.\ 1617--1627. Computer Vi...

2020
[23]

and Talwalkar, A

Li, L. and Talwalkar, A. Random search and reproducibility for neural architecture search. In Globerson, A. and Silva, R. (eds.), Proceedings of the Thirty-Fifth Conference on Uncertainty in Artificial Intelligence, UAI 2019, Tel Aviv, Israel, July 22-25, 2019 , volume 115 of ...

2019
[24]

Darts: differentiable architecture search

Liu, H., Simonyan, K., and Yang, Y. Darts: differentiable architecture search. In International Conference on Learning Representations, New Orleans, LA, USA, pp.\ 1--11, 2019

2019
[25]

Learning where to look - generative NAS is surprisingly efficient

Lukasik, J., Jung, S., and Keuper, M. Learning where to look - generative NAS is surprisingly efficient. CoRR, abs/2203.08734, 2022

2022 arXiv
[26]

Neural architecture optimization

Luo, R., Tian, F., Qin, T., Chen, E., and Liu, T. Neural architecture optimization. In Bengio, S., Wallach, H. M., Larochelle, H., Grauman, K., Cesa - Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Informa...

2018
[27]

J., Mnih, A., and Teh, Y

Maddison, C. J., Mnih, A., and Teh, Y. W. The concrete distribution: A continuous relaxation of discrete random variables. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017

2017
[28]

J., and Crowley, E

Mellor, J., Turner, J., Storkey, A. J., and Crowley, E. J. Neural architecture search without training. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceeding...

2021
[29]

Y., Zoph, B., Le, Q

Pham, H., Guan, M. Y., Zoph, B., Le, Q. V., and Dean, J. Efficient neural architecture search via parameter sharing. In Dy, J. G. and Krause, A. (eds.), International Conference on Machine Learning, Stockholm, Sweden, volume 80, pp.\ 4092--4101, 2018

2018
[30]

Real, E., Aggarwal, A., Huang, Y., and Le, Q. V. Regularized evolution for image classifier architecture search. In The Thirty-Third AAAI Conference on Artificial Intelligence, Honolulu, Hawaii, USA , pp.\ 4780--4789, 2019

2019
[31]

Rezaei, S. S. C., Han, F. X., Niu, D., Salameh, M., Mills, K. G., Lian, S., Lu, W., and Jui, S. Generative adversarial neural architecture search. In Zhou, Z. (ed.), Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Eve...

2021
[32]

High-resolution image synthesis with latent diffusion models

Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pp.\ 10674--10685. IEEE , 2022

2022
[33]

Transfer NAS with meta-learned bayesian surrogates

Shala, G., Elsken, T., Hutter, F., and Grabocka, J. Transfer NAS with meta-learned bayesian surrogates. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023

2023
[34]

Nas-bench-301 and the case for surrogate benchmarks for neural architecture search

Siems, J., Zimmer, L., Zela, A., Lukasik, J., Keuper, M., and Hutter, F. Nas-bench-301 and the case for surrogate benchmarks for neural architecture search. CoRR, abs/2008.09777, 2020

2008 arXiv
[35]

Snoek, J., Rippel, O., Swersky, K., Kiros, R., Satish, N., Sundaram, N., Patwary, M. M. A., Prabhat, and Adams, R. P. Scalable bayesian optimization using deep neural networks. In Bach, F. R. and Blei, D. M. (eds.), Proceedings of the 32nd International Conference on Machine L...

2015
[36]

G., and Lv, J

Sun, Y., Xue, B., Zhang, M., Yen, G. G., and Lv, J. Automatically designing cnn architectures using the genetic algorithm for image classification. IEEE transactions on cybernetics, 50 0 (9): 0 3840--3854, 2020

2020
[37]

NPENAS: neural predictor guided evolution for neural architecture search

Wei, C., Niu, C., Tang, Y., Wang, Y., Hu, H., and Liang, J. NPENAS: neural predictor guided evolution for neural architecture search. IEEE Trans. Neural Networks Learn. Syst. , 34 0 (11): 0 8441--8455, 2023. doi:10.1109/TNNLS.2022.3151160

2023
[38]

H., Bender, G., and Kindermans, P

Wen, W., Liu, H., Chen, Y., Li, H. H., Bender, G., and Kindermans, P. Neural predictor for neural architecture search. In Vedaldi, A., Bischof, H., Brox, T., and Frahm, J. (eds.), European Conference on Computer Vision, Glasgow, UK, volume 12374, pp.\ 660--676, 2020

2020
[39]

A study on encodings for neural architecture search

White, C., Neiswanger, W., Nolen, S., and Savani, Y. A study on encodings for neural architecture search. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information ...

2020
[40]

BANANAS: bayesian optimization with neural architectures for neural architecture search

White, C., Neiswanger, W., and Savani, Y. BANANAS: bayesian optimization with neural architectures for neural architecture search. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021 , pp.\ 10293--10301. AAAI Press, 2021 a

2021
[41]

Exploring the loss landscape in neural architecture search

White, C., Nolen, S., and Savani, Y. Exploring the loss landscape in neural architecture search. In de Campos, C. P., Maathuis, M. H., and Quaeghebeur, E. (eds.), Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence, UAI 2021, Virtual Event, 2...

2021
[42]

How powerful are performance predictors in neural architecture search? In Ranzato, M., Beygelzimer, A., Dauphin, Y

White, C., Zela, A., Ru, R., Liu, Y., and Hutter, F. How powerful are performance predictors in neural architecture search? In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems 34: Annual Confe...

2021
[43]

A deeper look at zero-cost proxies for lightweight nas

White, C., Khodak, M., Tu, R., Shah, S., Bubeck, S., and Dey, D. A deeper look at zero-cost proxies for lightweight nas. In ICLR Blog Track, 2022. https://iclr-blog-track.github.io/2022/03/25/zero-cost-proxies/

2022
[44]

Neural architecture search: Insights from 1000 papers

White, C., Safari, M., Sukthanker, R., Ru, B., Elsken, T., Zela, A., Dey, D., and Hutter, F. Neural architecture search: Insights from 1000 papers. CoRR, abs/2301.08727, 2023

2023 arXiv
[45]

and Pedapati, T

Wistuba, M. and Pedapati, T. Learning to rank learning curves. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research, pp.\ 10303--10312. PMLR , 2020

2020
[46]

Stronger NAS with weaker predictors

Wu, J., Dai, X., Chen, D., Chen, Y., Liu, M., Yu, Y., Wang, Z., Liu, Z., Chen, M., and Yuan, L. Stronger NAS with weaker predictors. In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems 34: Ann...

2021
[47]

Shapley-nas: Discovering operation contribution for neural architecture search

Xiao, H., Wang, Z., Zhu, Z., Zhou, J., and Lu, J. Shapley-nas: Discovering operation contribution for neural architecture search. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pp.\ 11882--11891. IEEE , 2022

2022
[48]

Weight-sharing neural architecture search: a battle to shrink the optimization gap

Xie, L., Chen, X., Bi, K., Wei, L., Xu, Y., Wang, L., Chen, Z., Xiao, A., Chang, J., Zhang, X., et al. Weight-sharing neural architecture search: a battle to shrink the optimization gap. ACM Computing Surveys, 54 0 (9): 0 1--37, 2021

2021
[49]

SNAS: stochastic neural architecture search

Xie, S., Zheng, H., Liu, C., and Lin, L. SNAS: stochastic neural architecture search. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019

2019
[50]

Does unsupervised architecture representation learning help neural architecture search? In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H

Yan, S., Zheng, Y., Ao, W., Zeng, X., and Zhang, M. Does unsupervised architecture representation learning help neural architecture search? In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual...

2020
[51]

Nas-bench-x11 and the power of learning curves

Yan, S., White, C., Savani, Y., and Hutter, F. Nas-bench-x11 and the power of learning curves. In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information P...

2021
[52]

 -darts: Beta-decay regularization for differentiable architecture search

Ye, P., Li, B., Li, Y., Chen, T., Fan, J., and Ouyang, W.  -darts: Beta-decay regularization for differentiable architecture search. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pp.\ 10864--10873. I...

2022
[53]

Nas-bench-101: towards reproducible neural architecture search

Ying, C., Klein, A., Christiansen, E., Real, E., Murphy, K., and Hutter, F. Nas-bench-101: towards reproducible neural architecture search. In Chaudhuri, K. and Salakhutdinov, R. (eds.), International Conference on Machine Learning, California, USA , volume 97, pp.\ 7105--7114, 2019

2019
[54]

Yuan, E., Chen, W., and Hentenryck, P. V. Reinforcement learning from optimization proxy for ride-hailing vehicle relocation. J. Artif. Intell. Res., 75: 0 985--1002, 2022

2022
[55]

Understanding and robustifying differentiable architecture search

Zela, A., Elsken, T., Saikia, T., Marrakchi, Y., Brox, T., and Hutter, F. Understanding and robustifying differentiable architecture search. In International Conference on Learning Representations, Addis, Ethiopia, pp.\ 1--11. OpenReview.net, 2020

2020
[56]

C., Yi, S., Zhang, X., and Ouyang, W

Zhou, D., Zhou, X., Zhang, W., Loy, C. C., Yi, S., Zhang, X., and Ouyang, W. Econas: Finding proxies for economical neural architecture search. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020 , pp.\ 11393--1...

2020
[57]

and Le, Q

Zoph, B. and Le, Q. V. Neural architecture search with reinforcement learning. In International Conference on Learning Representations, Toulon, France, pp.\ 1--16, 2017

2017
[58]

Zoph, B., Vasudevan, V., Shlens, J., and Le, Q. V. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 8697--8710, 2018

2018

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

[2] [2]

S., Mehrotra, A., Dudziak, L., and Lane, N

Abdelfattah, M. S., Mehrotra, A., Dudziak, L., and Lane, N. D. Zero-cost proxies for lightweight NAS . In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021

work page 2021

[3] [3]

An, S., Lee, H., Jo, J., Lee, S., and Hwang, S. J. Diffusionnag: Task-guided neural architecture generation with diffusion models. CoRR, abs/2305.16943, 2023

work page Pith review arXiv 2023

[4] [4]

Multi-conditioned Graph Diffusion for Neural Architecture Search

Asthana, R., Conrad, J., Dawoud, Y., Ortmanns, M., and Belagiannis, V. Multi-conditioned graph diffusion for neural architecture search. CoRR, abs/2403.06020, 2024

work page Pith review arXiv 2024

[5] [5]

Accelerating neural architecture search using performance prediction

Baker, B., Gupta, O., Raskar, R., and Naik, N. Accelerating neural architecture search using performance prediction. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Workshop Track Proceedings . OpenReview.net, 2018

work page 2018

[6] [6]

M., and Weston, N

Brock, A., Lim, T., Ritchie, J. M., and Weston, N. Smash: one-shot model architecture search through hypernetworks. In International Conference on Learning Representations, Vancouver, BC, Canada, pp.\ 1--21, 2018 a

work page 2018

[7] [7]

M., and Weston, N

Brock, A., Lim, T., Ritchie, J. M., and Weston, N. SMASH: one-shot model architecture search through hypernetworks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net, 2018 b

work page 2018

[8] [8]

Proxylessnas: Direct neural architecture search on target task and hardware

Cai, H., Zhu, L., and Han, S. Proxylessnas: Direct neural architecture search on target task and hardware. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019

work page 2019