{"id":"ff45dc32-d22e-4c23-823f-baa0d245e7ba","arxiv_id":"2509.05656","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"OptiProxy-NAS is a neural architecture search method that relaxes network graphs into differentiable parameters and runs gradient ascent on a learned accuracy predictor to propose architectures, beating several baselines with fewer queries.","lead":"OptiProxy-NAS turns the discrete architecture search problem into a continuous, differentiable one by letting gradient ascent on a learned predictor propose new neural networks. It reports state-of-the-art results on several NAS benchmarks at equal or lower query cost, plus experiments on hardware-aware and low-fidelity settings.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (13)'s claim of unbiased gradient flow is unsupported: τ→0 convergence in distribution of soft samples does not imply convergence of gradients, and at the implemented τ=0.2 the Gumbel-softmax gradient is a biased, partially vanishing signal for the discrete objective.","rationale":"Reader's weakest assumption and my concern coincide. The paper's own text (§2.2) concedes Eq. (7) is not calculable, and the Appendix A.5 proof establishes only distributional convergence of samples, not of derivatives; standard Gumbel-softmax theory does not provide unbiased gradient estimates of a discrete objective as τ→0 (the pathwise derivative vanishes off ties). Since the entire novelty claim is 'end-to-end gradient information flow' (Fig. 1c, Algorithm 1), an unsupported gradient relationship is not a cosmetic gap: it is the bridge between the relaxed optimization and the discrete NAS problem. The empirical tables are extensive and appear to follow NAS best practices, and reaching the NB201 optimum in 100 queries is a strong falsifiable result, so I do not call for rejection. However, reproducibility is currently hindered by the code-release promise and by the GCN/XGBoost inconsistency in A.7.1 vs §§2.5,3 and A.4; if the deployed predictor is non-differentiable XGBoost, the end-to-end story cannot be literally true. My recommended disposition is unchanged: CONDITIONAL, conditioned on (a) a correct derivation or explicit statement that the relaxation is a heuristic with no unbiasedness guarantee, (b) clarification of the proxy model, and (c) code release. The concrete finite-difference test above would settle whether the relaxed gradient actually points toward high-accuracy discrete architectures; if it does not, the central claim should be downgraded.","tokens_in":30487,"tokens_out":6031,"duration_ms":54305,"concrete_test":"Primary check: on NAS-Bench-201 CIFAR-10, train the exact GCN predictor (Appendix A.4) on 100 random architectures, then for 200 random α values compute three quantities at the implemented temperature τ=0.2 and at τ∈{1.0,0.5,0.1}: (i) Monte-Carlo Gumbel-softmax gradient g_GS=(1/S)Σ∂fhat/∂shat·∂shat/∂α; (ii) score-function estimator g_SF=(1/S)Σ fhat(A_s)∇_α log p(A_s;α) over discrete samples A_s; and (iii) central finite differences in α of a large-MC estimate of E_{A~p(α)}[fhat(A)]. If the cosine similarity between g_GS and g_SF or between g_GS and the finite-difference direction is not significantly positive, Eq. (13) is false. An even more decisive end-to-end variant: run Algorithm 1 with g_GS and compare against the same loop with random gradients of equal L2 norm; if 100-query best accuracy is statistically indistinguishable, gradient direction is not the driver.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.2 defines the objective as ψ=fhat(s(α,β),θ*) and writes ∇αψ=(∂fhat/∂s)(∂s/∂α) in Eq. (7), while admitting s is discrete and Eq. (7) is not calculable. The method then replaces ∂s/∂α by ∂shat/∂α (Eq. 8) and claims in Eq. (13) that as τ→0 this proxy gradient converges to the original one. Appendix A.5 proves only convergence in probability of the Gumbel-softmax samples to the categorical/Bernoulli samples (Eqs. A.7 and A.11). That is a statement about random variables, not about derivatives. Pathwise, the derivative of a one-hot limit is zero almost everywhere: ∂shat_i/∂α_j=(1/τ)(shat_iδ_ij−shat_i shat_j) tends to 0 for the winning component and has spikes only on measure-zero ties, so the relaxed gradient does not converge to any gradient of discrete sampling. Moreover, the search uses mintemp=0.2 (Table 6), so the τ→0 regime is never reached. Thus the central mechanism—gradient ascent on (α,β) moving probability mass toward high-accuracy discrete architectures—has no supporting derivation. The empirical query-efficiency results could in principle still be real (e.g., if the continuous relaxation is itself a useful search heuristic), but the paper's theoretical justification is the load-bearing part that fails. The implementation confusion between GCN (§3, A.4) and XGBoost (A.7.1) exacerbates this: if the fitted proxy is really XGBoost, no gradient path through it exists.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes OptiProxy-NAS, a predictor-guided NAS framework in which the discrete architecture distribution parameters (alpha, beta) are optimized by gradient ascent through a differentiable relaxation (Gumbel-softmax / binary concrete) and a fitted differentiable proxy model. The proxy model is retrained iteratively on queried architectures. The authors claim that this relaxation makes the search unbiased and end-to-end differentiable, and they report strong query-efficiency results on NAS-Bench-101/201/301, NAS-Bench-NLP, and HW-NAS-Bench, with 10-seed statistics, ablations, low-fidelity experiments, and a plan for code release.","tokens_in":30878,"tokens_out":5240,"duration_ms":45772,"significance":"If the central claims held, this would be a meaningful step for NAS: reaching the global optimum on NAS-Bench-201 with 100 queries, achieving roughly 4x query efficiency on NAS-Bench-301 and NAS-Bench-NLP, and consistently beating published results on 12 HW-NAS-Bench latency settings with half the query budget are impressive empirical results. The paper also provides 10-seed runs, ablation studies, low-fidelity comparisons, and a concrete resource-cost analysis. However, the theoretical foundation of the gradient proxy is not established, and there is a major internal inconsistency about which proxy model actually produced the reported numbers. The empirical evidence is therefore not currently supported by a sound explanation of why the method works.","major_comments":[{"comment":"The central claim that the Gumbel-softmax relaxation gives unbiased proxy gradients is unsupported. Appendix A.5 proves only that the relaxed samples converge in probability to the discrete samples as τ→0 (Eqs. A.7 and A.11), which is a statement about random variables, not about derivatives. The right-hand side of Eq. (13), ∂f̂/∂s · ∂s/∂α, is not even defined because s is discrete, as the paper itself states in §2.2. Moreover, for the Gumbel-softmax, ∂ŝ_j/∂α_i = (1/τ)(ŝ_i δ_ij − ŝ_i ŝ_j), so the pathwise derivative does not converge to any gradient of the discrete sampler; at the implemented minimum temperature τ=0.2 (Table 6 in Appendix A.9.1) the τ→0 regime is never approached. The authors should either supply a correct proof of a gradient relationship, replace the claim with a weaker statement, or use an estimator with known gradient properties.","section":"§2.3, Eq. (13); Appendix A.5"},{"comment":"The manuscript is internally inconsistent about which proxy model generates the reported results. Main text §3 and Appendix A.4 describe a two-layer GCN with 139.7K parameters and standard backpropagation, while Appendix A.7.1 states that XGBoost is employed as the surrogate model, consistent with AG-Net and DiNAS, and that NAS-Bench-301 uses XGBoost and NAS-Bench-NLP uses 'svd lgb'. XGBoost and LightGBM are not differentiable with respect to their inputs in the sense required by Eq. (8), so the end-to-end gradient path in Figure 3 cannot exist for those experiments. The authors must clarify which surrogate actually produced each table and, if tree-based models were used, explain how gradients through the surrogate were obtained; this is load-bearing for the method's central claim.","section":"§3, §A.4, §A.7.1"},{"comment":"The treatment of the NAS-Bench-201 adjacency is a simplifying assumption that is acknowledged but potentially consequential. Figure 6 shows that choosing 'none' or 'skip_connect' changes the computational graph, yet the encoding used for the proxy (Appendix A.8.1) fixes the adjacency matrix to the canonical form. Because the gradient signal in Eq. (13) is computed with respect to this fixed-topology encoding, the search objective does not represent the true architecture space for a large fraction of candidates. The authors should justify that this simplification does not materially bias the reported query-efficiency numbers, or evaluate the method with an encoding that captures the coupling.","section":"Appendix A.8.1 and §2.3"}],"minor_comments":[{"comment":"The phrase 'nearly 100% less query cost' is mathematically odd; 100% less would be zero queries. Please clarify whether the intended meaning is 'roughly half the queries' or 'about 50% fewer queries' as stated elsewhere.","section":"Abstract and §3.1"},{"comment":"The index bounds in the relaxed sampling functions appear to be off by one: Eq. (9) writes '0≤i<N−1, 0≤j<M−1' and Eq. (10) writes '0≤h,k<N−1', but the softmax and sigmoid are defined over all M operations and N nodes. Please correct the bounds.","section":"§2.2, Eqs. (9) and (10)"},{"comment":"The sentence 'The candidate operation set of NAS-Bench-101 space is as follows' appears at the start of the NAS-Bench-301 subsection; it should refer to NAS-Bench-301.","section":"Appendix A.8.3"},{"comment":"The ImageNet16-120 entry for OptiProxy-NAS at 280 queries, marked '46.77*', is hard to parse together with the footnote 'Reach * even with 70% queries'. Please clarify which starred value is actually matched and at what query budget.","section":"Table 1"},{"comment":"The description of 'mintemp' says it 'determines the unbiased nature of the proxy gradient and the original gradients, commonly set close to 0', but the reported value is 0.2. Please reconcile this with the claimed τ→0 unbiasedness.","section":"Appendix A.9.1, Table 6"}],"recommendation":"major_revision","confidential_remarks":"The empirical results are strong and the paper is carefully benchmarked with 10 seeds, but the theoretical claim in Eq. (13) is not supported by the provided proof, and the proxy-model inconsistency between GCN and XGBoost is serious. I believe the paper can be made publishable if the authors either prove a correct gradient statement or explicitly reframe the relaxation as a heuristic, and if they clearly identify and justify the actual differentiable surrogate used in each experiment. This is a substantial revision rather than a rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the empirical work is real and fairly broad; the theoretical core, as written, is not. I would send this to review, but the authors need to fix the gradient story before publication.\n\nWhat is new: they relax both operation choices (categorical) and topology (Bernoulli) with Gumbel-softmax, then run gradient ascent on the relaxed variables through a learned predictor inside an SMBO loop. NAO optimizes an embedding, DARTS uses supernets, AG-Net/DiNAS use discrete generative sampling. This specific combination is novel, and the experiments are extensive: four search spaces, 12 tasks, hardware latency constraints, low-fidelity settings, ablations, and 10 seeds. The query-efficiency numbers are credible: reaching NB201 optimum in 100 queries, beating AG-Net on HW-NAS with half the queries. If those numbers reproduce, the method is useful.\n\nThe soft spot is load-bearing. Eq. (13) claims that as tau goes to 0 the relaxed gradient dfhat/dshat * dshat/dalpha converges to dfhat/ds * ds/dalpha, but ds/dalpha does not exist for discrete sampling, as they themselves say in Section 2.2. Appendix A.5 proves only convergence in probability of the soft samples to the hard samples. That says nothing about gradients. Pathwise, the derivative of the one-hot limit is zero almost everywhere, and their own implementation uses min temperature 0.2, far from zero. So the central \"unbiased relaxation\" claim is unsupported, and the search is not justified by the given theory.\n\nThere is also a confusing contradiction about the proxy model: Section 3 and Appendix A.4 describe a GCN, but Appendix A.7.1 says they use XGBoost as the surrogate, matching AG-Net and DiNAS. If the fitted model is XGBoost, there is no gradient path through it at all, which would make the whole mechanism incoherent. Either the text is wrong, or the method is different from what the main body describes. They need to state exactly which model produced each table.\n\nThe empirical results could still be real, because the relaxed objective is a smooth surrogate that may guide search even without unbiasedness. The paper should be reframed that way: a continuous relaxation with annealed temperature, not an unbiased gradient estimator. That is a defensible heuristic, and the experiments would support it.\n\nWho this is for: people doing predictor-based NAS or Gumbel-softmax optimization. It deserves a serious referee, with major revision requested. The experiments are worth engaging, but the theory section and the model description must be corrected or removed before I would trust the claims.","headline":"Strong empirical NAS results undermined by a wrong theoretical claim about unbiased gradients; the method may still work as a heuristic, but the paper needs major revision.","tokens_in":31418,"tokens_out":3061,"would_cite":false,"duration_ms":28000,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that neural architecture search can be turned into a differentiable end-to-end optimization problem by maximizing a learned smooth predictor over Gumbel-softmax relaxed architecture variables, and that this gradient-based…","keywords":["neural architecture search","differentiable architecture search","Gumbel-softmax relaxation","optimization proxy","predictor-guided search","hardware-aware NAS","low-fidelity evaluation","NAS-Bench-201"],"falsifier":"A control experiment on NAS-Bench-201 that replaces the backpropagated gradients with random directions, keeping the same Gumbel noise, temperature schedule, predictor refits, and sampling, would settle whether gradient direction is doing the work: if the random-gradient control reaches the same optima, the central gradient-validity claim is falsified. A sharper check is to estimate, over many update steps, the correlation between $\\nabla_{\\alpha,\\beta}\\hat{f}_\\theta(\\hat{s})$ and the actual change in true accuracy of discrete architectures sampled after the update, where zero or negative correlation refutes the proxy-gradient claim.","tokens_in":2090,"feed_emoji":"🧠","tokens_out":2444,"duration_ms":96781,"temperature":0.7,"pith_summary":"The paper proposes OptiProxy-NAS, which treats neural architecture search as a differentiable optimization problem rather than a discrete black-box search. Its central claim is that training a smooth predictor over relaxed architecture variables and then ascending its gradient through Gumbel-softmax approximations of categorical operation choices and Bernoulli edge decisions drives sampling toward high-performance architectures far more efficiently than predictor-guided discrete samplers. On NAS-Bench-201 the method reaches the global optimum in 100 queries, and on NAS-Bench-301 and NAS-Bench-NLP it claims a roughly fourfold query-efficiency improvement over the strongest published baselines. The same framework extends to hardware-aware objectives, beating AG-Net on all 12 HW-NAS-Bench latency settings with 50 percent fewer queries, and to low-fidelity evaluation with only about a third of the training budget. The load-bearing assumption is that the softened-sample gradient is a valid proxy for how true accuracy would change, a premise the current convergence proof does not fully establish.","feed_headline":"Gumbel-softmax proxy finds NAS optima in 100 queries","feed_subtitle":"A learned smooth proxy plus gradient ascent beats discrete samplers, cutting queries up to 4x on standard benchmarks.","key_machinery":"The central object is the optimization proxy: a differentiable predictor $\\hat{f}_\\theta$, typically a two-layer GCN, that maps a graph encoding of an architecture, one-hot operation features plus an adjacency matrix, to a target metric. Around it, the method relaxes architecture variables into $\\alpha$ and $\\beta$, reparameterizes discrete sampling through Gumbel-softmax and sigmoid-Gumbel samples $\\hat{s}_o$ and $\\hat{s}_t$, and then runs gradient ascent from the predictor output back to $\\alpha$ and $\\beta$. The Gumbel relaxation is what creates the differentiable forward path; temperature decay balances exploration and exploitation; a sequential model-based optimization loop alternately refits the predictor on queried architectures and uses the proxy to propose the next batch, while multiple $(\\alpha,\\beta)$ groups initialized by Latin hypercube sampling add parallel exploration.","core_discovery":"On the paper's own terms, the central claim is that the NAS objective over the discrete architecture space can be replaced by an end-to-end differentiable objective over relaxed variables $\\alpha$ (operation logits) and $\\beta$ (edge log-odds), fit through a learned surrogate $\\hat{f}_\\theta$ and searched by gradient ascent, provided the discrete sampling function $s$ is replaced by its Gumbel-softmax relaxation $\\hat{s}$. The paper argues that as the temperature $\\tau\\to 0$, $\\hat{s}$ converges in probability to the categorical and Bernoulli samples $s$ (Eqs. 11 to 12), and therefore the proxy gradients $\\partial\\hat{f}/\\partial\\hat{s}\\cdot\\partial\\hat{s}/\\partial\\alpha$ approximate the original gradients (Eq. 13). This is what the paper takes to make the relaxation unbiased and to let one pipeline optimize accuracy, latency, or low-fidelity metrics. On its benchmarks, the claim is that this single framework reaches global optima faster than predictor-guided RL, EA, and BO samplers, and faster than DARTS-style supernetwork methods, while adding negligible overhead.","pith_inferences":["The paper's own Section 2.2 notes that the discrete sampling function is not differentiable, so the proof gap sits exactly at Eq. (13): the convergence proof in Appendix A.5 establishes convergence of samples, not of reparameterization gradients; a score-function estimator comparison on a small tabular space would reveal whether gradient validity is the active ingredient.","If the central premise generalizes, the predict-then-ascend template should transfer to other expensive discrete design spaces, such as chip floorplans or molecule graphs, where a differentiable surrogate can be trained; this paper does not test that transfer, so it remains an extension.","The reported gains might come substantially from the exploration induced by Gumbel noise, temperature decay, and multi-$\\alpha,\\beta$ starts rather than from gradient direction; a random-gradient control under the same sequential model-based loop would separate these causes.","On the noisier ImageNet16-120 portion of NAS-Bench-201 the method still needs 280 queries, suggesting the smooth-proxy benefit shrinks as the landscape becomes more rugged; a natural extension is roughness-adaptive temperature and multi-start scheduling."],"forward_implications":["On NAS-Bench-201, reaching the global optimum with roughly 100 queries on CIFAR-10 and CIFAR-100 and 280 on ImageNet16-120 would make search cost nearly negligible for tabular spaces.","On NAS-Bench-301 and NAS-Bench-NLP, a fourfold query-efficiency gain over the compared generative baselines would let surrogate-space search run under much tighter query budgets.","On HW-NAS-Bench, beating all 12 latency-constrained settings with 50 percent fewer queries would make deployment-aware NAS practical on edge devices where each architecture evaluation is expensive.","The reported overhead of 20 seconds and 10MB, or 0.037 percent of runtime and 0.25 percent of memory, would make the optimization proxy a drop-in search module on top of existing predictors and graph encodings.","The demonstrated compatibility with 5-epoch low-fidelity evaluation, cutting total training epochs by factors of roughly 2 to 6 on NAS-Bench-201 tasks, suggests the framework can combine with multi-fidelity search strategies."],"supporting_citations":[{"why":"Supplies the Gumbel-softmax reparameterization used to make operation sampling differentiable.","marker":"[Jang et al., 2017]"},{"why":"Provides the Concrete-distribution convergence-in-probability result that the paper's unbiased-relaxation argument depends on.","marker":"[Maddison et al., 2017]"},{"why":"Supplies the GCN predictor architecture and graph encoding scheme that the optimization proxy is built from.","marker":"[Wen et al., 2020]"},{"why":"BANANAS is the predictor-guided baseline whose query costs and results OptiProxy-NAS is compared against on tabular benchmarks.","marker":"[White et al., 2021a]"},{"why":"Creates NAS-Bench-201, the tabular benchmark where the 100-query global optimum is claimed.","marker":"[Dong & Yang, 2020]"},{"why":"AG-Net is the generative baseline used for the headline comparison across NAS-Bench-201 and HW-NAS-Bench.","marker":"[Lukasik et al., 2022]"},{"why":"DiNAS is the current generative baseline whose query efficiency OptiProxy-NAS aims to beat across benchmarks.","marker":"[Asthana et al., 2024]"},{"why":"Creates HW-NAS-Bench, the 12-setting latency-constrained benchmark used for hardware-aware search.","marker":"[Li et al., 2021]"},{"why":"Creates NAS-Bench-301, the smooth surrogate benchmark where the largest efficiency gains are reported.","marker":"[Siems et al., 2020]"},{"why":"Creates NAS-Bench-NLP, the RNN-space surrogate benchmark used for cross-domain validation.","marker":"[Klyuchnikov et al., 2022]"}],"fun_headline_variants":["Smooth proxy makes NAS differentiable end-to-end","Gumbel-softmax proxy accelerates NAS search","Optimization proxy cuts NAS queries up to 4x","End-to-end NAS via continuous proxy"],"cache_read_input_tokens":33408,"weakest_assumption_plain":"The approach depends on the untested premise that the gradient of the learned predictor through the softened random samples points toward architectures that actually improve true accuracy; if that gradient signal is misleading, the search is following noise rather than the real landscape.","fun_headline_variants_meta":{"raw":{"variants":["Smooth proxy makes NAS differentiable end-to-end","Gumbel-softmax proxy accelerates NAS search","Optimization proxy cuts NAS queries up to 4x","End-to-end NAS via continuous proxy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000658,"raw_usage":{"total_tokens":3010,"prompt_tokens":941,"completion_tokens":2069,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":2009}},"tokens_in":557,"tokens_out":2069,"duration_ms":14480,"temperature":1.0,"reasoning_tokens":2009,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:21:46.766093+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A control experiment on NAS-Bench-201 that replaces the backpropagated gradients with random directions, keeping the same Gumbel noise, temperature schedule, predictor refits, and sampling, would settle whether gradient direction is doing the work: if the random-gradient control reaches the same optima, the central gradient-validity claim is falsified. A sharper check is to estimate, over many update steps, the correlation between $\\nabla_{\\alpha,\\beta}\\hat{f}_\\theta(\\hat{s})$ and the actual change in true accuracy of discrete architectures sampled after the update, where zero or negative correlation refutes the proxy-gradient claim.","supporting_citations":[{"cited_title":"Categorical reparameterization with gumbel-softmax","cited_arxiv_id":null,"evidence_quote":"Supplies the Gumbel-softmax reparameterization used to make operation sampling differentiable."},{"cited_title":"J., Mnih, A., and Teh, Y","cited_arxiv_id":null,"evidence_quote":"Provides the Concrete-distribution convergence-in-probability result that the paper's unbiased-relaxation argument depends on."},{"cited_title":"H., Bender, G., and Kindermans, P","cited_arxiv_id":null,"evidence_quote":"Supplies the GCN predictor architecture and graph encoding scheme that the optimization proxy is built from."},{"cited_title":"and Yang, Y","cited_arxiv_id":null,"evidence_quote":"Creates NAS-Bench-201, the tabular benchmark where the 100-query global optimum is claimed."},{"cited_title":"Learning Where To Look -- Generative NAS is Surprisingly Efficient","cited_arxiv_id":"2203.08734","evidence_quote":"AG-Net is the generative baseline used for the headline comparison across NAS-Bench-201 and HW-NAS-Bench."},{"cited_title":"Multi-conditioned Graph Diffusion for Neural Architecture Search","cited_arxiv_id":"2403.06020","evidence_quote":"DiNAS is the current generative baseline whose query efficiency OptiProxy-NAS aims to beat across benchmarks."},{"cited_title":"Nas-bench-nlp: neural architecture search benchmark for natural language processing","cited_arxiv_id":null,"evidence_quote":"Creates NAS-Bench-NLP, the RNN-space surrogate benchmark used for cross-domain validation."}],"review_version":2}