{"id":"10c81a58-3472-47c1-b32f-d53cf2daa4ad","arxiv_id":"2607.07984","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"An LLM seeds and slots a task-specific architecture search space that conventional NAS then explores, reaching SOTA on 11 of 17 diverse NAS benchmarks.","lead":"AgentNAS lets an LLM invent a seed neural network, then carve it into named module slots so ordinary NAS can recombine those slots without hand-built search spaces. On 17 diverse tasks it beats published baselines on 11, and ablations show LLM design and combinatorial NAS are complementary.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified beyond the reader's already-flagged search-space fidelity premise; that premise is real but already stress-tested and does not overturn the complementarity claim.","rationale":"The central claim is empirical and multi-task: LLM seed + auto-slotted NAS reaches SOTA on 11/17 and the two mechanisms are complementary (seed already strong; NAS adds combinatorial gains that matched-budget LLM sampling does not). Supporting evidence is concrete (Tables 1–3, Figure 3, A.3–A.5, A.7.3) and the paper is unusually honest about failures (6 underperforming tasks, hardware bias, imperfect budgets, weaker-LLM space collapse, DeepSea val/test misalignment). The single softest load-bearing condition is exactly the one the reader named—fidelity/density of the LLM-constructed slotted neighborhood under frozen macro structure. That condition is most stressed on Darcy Flow and with weaker models, but the paper already documents those cases and still shows NAS value after LLM saturation on the majority of analysis tasks, plus fixed-seed evidence that Phase 2 itself is stronger than einspace's grammar. No internal inconsistency, circular selection, or claim-without-evidence failure appears. Remaining issues (commercial LLM dependence, unmatched absolute budgets, unreproduced fixed-seed numbers) are real but already priced into CONDITIONAL. My read therefore does not move the verdict: keep CONDITIONAL with HIGH confidence. The proposed concrete_test is a targeted macro-slot relaxation that would settle whether the frozen-macro premise is quietly capping the reported NAS deltas.","tokens_in":34648,"tokens_out":794,"duration_ms":8432,"concrete_test":"On Spherical, CIFARTile, and Chesseract (the three largest NAS deltas), re-run Phase 3 after forcing one additional macro-level slot family the current planner freezes—e.g., stage-count {N−1,N,N+1} or width-multiplier {0.75×,1×,1.25×}—with the same 160-eval RE budget. If the best non-seed test metric improves by more than the multi-seed std in Table 10 (≈1.7–1.9 pp), the frozen-macro premise is excluding better designs and the complementarity claim needs a caveat; if not, the neighborhood assumption holds for the reported gains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reader's weakest assumption is correct and is the right load-bearing point: Phase 2's slotted space (≤20 slots × ≤8 alternatives, macro structure frozen after optional modularization) is assumed to be a high-density neighborhood of good architectures rather than a local cage around a lucky seed. If that fails systematically, NAS gains would be recovery or noise, not genuine complementarity. The paper itself supplies the sharpest counter-example (Section 5.3 Darcy Flow: Sonnet's spectral slots retain the same einsum channel-mixing form, so NAS has no useful variation; Opus's element-wise spectral alternative succeeds). Weaker models can also emit degenerate spaces (Limitations; Haiku on Darcy). That said, this is not an unexamined hole: fixed-seed + Phase 2–3 still beats einspace-500 on Spherical/CIFARTile (Table 3); modularization ablation (A.7.3) shows the alternative (over-fragmentation) is worse; RE vs RS vs GDAS (A.3) and multi-seed (A.4) show the space is dense enough for even random search; and Figure 3 shows NAS gains after LLM saturation across three model tiers on most of the six analysis tasks. The premise is therefore stressed but not falsified for the headline SOTA-on-11/17 and complementarity claims.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes AgentNAS, a three-phase pipeline that uses an LLM to generate a seed architecture and training recipe (Phase 1), decompose that seed into a slotted architecture with named module-level and additive glue slots plus a learning-rate multiplier slot (Phase 2), and then run conventional discrete NAS (default: regularized evolution) inside that automatically constructed space (Phase 3). On 17 tasks from NAS-Bench-360 and Unseen NAS, the full pipeline reports new best results on 11 tasks; ablations argue that LLM seed design and NAS combinatorial recombination are complementary, that matched-budget further LLM sampling saturates while NAS still improves, and that the pattern holds across three Claude models of different capability.","tokens_in":35034,"tokens_out":1136,"duration_ms":12483,"significance":"If the results hold under the stated protocols, the work is a clear contribution to automated architecture design: it cleanly separates LLM-driven space construction from NAS-driven combinatorial search, evaluates on two diverse public suites (including a blind Unseen-NAS setting), and supplies multi-LLM, fixed-seed, search-algorithm, multi-seed, and modularization ablations plus released code. The slotted-architecture mechanism is a practical answer to the long-standing manual search-space bottleneck in NAS, and the complementarity evidence (Tables 1–3, Figure 3, Appendices A.3–A.4, A.7.3) is more carefully instrumented than most LLM-for-NAS papers.","major_comments":[{"comment":"The central complementarity claim rests on the premise that the Phase-2 slotted space (≤20 slots × ≤8 alternatives, macro structure frozen after optional modularization) is a high-density neighborhood of good architectures rather than a local cage around a lucky seed. Section 5.3 already shows this can fail: Sonnet’s Darcy Flow spectral slots retain the same einsum channel-mixing form, so NAS has no useful variation, while Opus’s element-wise spectral alternative succeeds. Limitations also note that weaker models can emit degenerate spaces. The paper should elevate this from a case study into an explicit discussion of when Phase 2 is expected to succeed or fail, and ideally add a simple quantitative diagnostic of slot diversity / structural variation (e.g., fraction of alternatives that change operator family or connectivity) correlated with NAS gain magnitude across the six analysis tas","section":null},{"comment":"Budget and selection protocol are carefully described but still leave residual ambiguity for the headline SOTA-on-11 claim. Phase 1 and Phase 3 each use a 160-evaluation cap with patience 64, proxy training at 0.3 of proposed epochs, and a non-seed best-validation selection rule that reports second-best when NAS fails to beat the seed (Section 4.3). Actual budgets therefore vary by early stopping and by LLM early termination (Limitations). Table 3 also notes unreproduced fixed-seed numbers and OOM on Darcy under the 11 GB constraint. A short consolidated budget table (evaluations actually run, wall-clock/GPU hours already partly in A.1, and whether the reported architecture came from Phase 1 or Phase 3) for all 17 tasks would make the comparison to einspace-500 and conventional NAS fully auditable.","section":null}],"minor_comments":[{"comment":"Tables 1–2 mix metrics (1−acc, Rel. L2, MAE8, 1−AUROC, 1−F1, 1−mAP). A one-line note in each caption that lower is always better (already stated in the table headers) and that ranks are computed on the reported metric would reduce reader friction.","section":null},{"comment":"Figure 3 is central to the complementarity argument; ensure axis scales and the Haiku Darcy special case (NAS from 160-budget seed while LLM continues to 480) are fully legible in the caption without requiring the main text.","section":null},{"comment":"Appendix A.2 training-recipe table is valuable; a brief main-text pointer that NAS reuses the LLM recipe except for the LR multiplier slot would help readers who skip the appendix.","section":null},{"comment":"Related Work §2.2 correctly notes that prior LLM-NAS work uses LLMs as operators inside the loop rather than as search-space constructors; a short explicit contrast table (role of LLM / role of NAS / space source) would make the novelty claim sharper.","section":null},{"comment":"Minor presentation: arXiv id and Claude model dates in the manuscript should be checked for consistency with the public release; code URL is appreciated and should remain stable.","section":null}],"recommendation":"minor_revision","confidential_remarks":"The manuscript is already stronger than typical LLM-for-NAS submissions on experimental hygiene (blind protocol, multi-model ablations, honest non-seed reporting). The two major points are fixable without new large-scale experiments if the authors expand the Section 5.3 discussion and add a budget/selection audit table. I would not require new tasks or new model families for acceptance. Fit for a serious ML venue is good; the main risk is over-claiming universality of the slotted space rather than presenting it as a strong, sometimes-failing, empirically useful division of labor."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful idea here is simple and new: the LLM does not stay in the search loop as a mutator. It builds a seed, then carves that seed into a slotted scaffold (module slots, identity-default glue slots, LR multipliers) so ordinary NAS can recombine modules without anyone hand-designing a space. That division of labor is what prior LLM-NAS papers did not study cleanly.\n\nThey run it on 17 tasks from NAS-Bench-360 and Unseen NAS, including a real blind protocol on the latter. Full pipeline hits SOTA on 11 tasks; Phase-1 seeds alone already beat most published baselines; Phase-3 still adds gains after LLM sampling saturates (Figure 3), and that pattern holds across Haiku/Sonnet/Opus. Fixed-seed + their Phase 2–3 beats einspace-500 on some tasks, which is evidence the slotted space itself is not just a lucky seed. Code is released. Selection is validation-only; they even report second-best when NAS does not beat the seed. That is honest engineering.\n\nSoft spots, in proportion: the load-bearing premise is that freezing macro structure and exposing ≤20×8 module/glue alternatives is a high-density neighborhood rather than a cage. They stress this themselves on Darcy Flow (Sonnet’s spectral slots leave NAS nothing useful; Opus’s do). Weaker models can emit degenerate spaces. Budgets are not perfectly matched, hardware biases toward smaller nets, and they lose on 6 tasks to hand-designed spaces. None of that overturns the complementarity claim on the bulk of the suite, but it means the method is not a free lunch and still depends on LLM quality for space design.\n\nThis is for AutoML and scientific-ML people who care about search-space construction more than another controller trick. Math is light (pipeline + evolution/GDAS/RS), data and citations look solid, no circular scoring. I would send it to referees and I would bring it to reading group. Worth citing if you work on LLM-assisted architecture search or open-ended AutoML.","headline":"Clean LLM-to-NAS handoff via slotted architectures; SOTA on 11/17 tasks with ablations that actually separate the two mechanisms.","tokens_in":35650,"tokens_out":524,"would_cite":true,"duration_ms":6963,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"An LLM builds a task-specific NAS search space by turning a seed network into named module slots, and the two methods together set new records on 11 of 17 diverse tasks.","keywords":["neural architecture search","large language models","slotted architecture","search space construction","AgentNAS","combinatorial recombination","NAS-Bench-360","Unseen NAS"],"falsifier":"On a held-out suite of the same diversity, run matched-budget independent LLM sampling past the saturation point and show that it matches or exceeds the LLM-seed-plus-NAS trajectory on most tasks; or show that a stronger, manually engineered or open-ended space around the same seeds systematically beats the LLM-constructed slotted space.","tokens_in":35523,"feed_emoji":"🧩","tokens_out":1001,"duration_ms":10354,"temperature":0.7,"pith_summary":"Neural architecture search usually needs a hand-built search space for every new task, while large language models can invent whole networks but only sample them one at a time. This paper claims the right division of labor is: let the language model first design a strong seed network and its training recipe, then carve that seed into a “slotted architecture”—a scaffold of named, interchangeable modules—so ordinary NAS can combinatorially recombine the modules without any manual space engineering. Instantiated as AgentNAS, the pipeline is tested on 17 classification, regression, segmentation, and tagging tasks spanning many data modalities. The language-model seed alone already beats published baselines on most tasks; NAS then adds further gains on most of them by recombining slots in ways independent language-model samples cannot. The same pattern holds across three models of different strength, and the full system sets a new best result on 11 of the 17 tasks, including under a blind protocol that hides domain labels.","feed_headline":"LLM slots a seed network so NAS can recombine it","feed_subtitle":"On 17 diverse tasks the hybrid pipeline sets new bests on 11; pure LLM sampling saturates first","key_machinery":"The slotted architecture: after the language model produces a seed network, a planner decomposes it into a scaffold with named module-level slots (and optional additive “glue” slots plus a learning-rate multiplier slot) whose alternatives automatically define a bounded discrete search space that standard NAS algorithms can explore while preserving the seed’s macro structure.","core_discovery":"LLM-driven seed design and conventional NAS are complementary rather than substitutable: the language model supplies both a high-quality seed and an automatically constructed, task-specific search space (the slotted architecture), while NAS finds additional performance by combinatorial recombination across slots—a mode of search that matched-budget independent LLM sampling does not replicate—yielding new state-of-the-art results on 11 of 17 diverse tasks.","pith_inferences":["Search-space quality, not only seed quality, is a first-class object of study for agentic design; the Darcy Flow contrast shows that structurally diverse slot alternatives matter more than raw seed score.","If modularization is required to keep block-stack seeds searchable, automatic granularity control becomes a necessary sub-problem for any LLM-to-NAS handoff.","The method suggests a practical template for other design domains where an open-ended generator can propose a scaffold and a combinatorial optimizer can recombine parts.","Blind-setting results imply that data-driven exploration can sometimes outperform domain-prior prescriptions when the language model otherwise short-circuits exploration."],"forward_implications":["Task-specific NAS search spaces no longer need to be hand-designed; a language model can emit both the seed and the space.","Architecture search can be budget-allocated as “LLM until saturation, then NAS recombination,” rather than pure LLM sampling or pure fixed-space NAS.","The same three-phase pipeline can be dropped onto new modalities and blind settings without rewriting search-space grammars.","Gains attributed to “LLM NAS” should be decomposed into seed quality versus combinatorial search over the induced slots.","Weaker language models may still benefit if a stronger model only designs the slots, because the complementarity pattern holds across capability levels."],"fun_headline_variants":["LLM seeds slotted scaffold for NAS module recombination","AgentNAS: LLM builds task slots then NAS recombines them","Seed-and-slot hybrid lets NAS recombine beyond pure LLM sampling","LLM-defined slots turn open design into bounded NAS search","Complementary LLM seed plus slot recombination hits 11 of 17 SOTAs"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"That the language model’s slotted neighborhood around the seed—keeping the seed’s overall depth, width, and backbone while only swapping modules and glue—contains enough of the good architectures for the task that NAS gains reflect real complementarity rather than recovery from a lucky seed or a poorly varied space.","fun_headline_variants_meta":{"raw":{"variants":["LLM seeds slotted scaffold for NAS module recombination","AgentNAS: LLM builds task slots then NAS recombines them","Seed-and-slot hybrid lets NAS recombine beyond pure LLM sampling","LLM-defined slots turn open design into bounded NAS search","Complementary LLM seed plus slot recombination hits 11 of 17 SOTAs"]},"model":"grok-4.5","effort":"low","cost_usd":0.00373,"raw_usage":{"total_tokens":1177,"prompt_tokens":833,"num_sources_used":0,"completion_tokens":90,"cost_in_usd_ticks":37300000,"prompt_tokens_details":{"text_tokens":833,"audio_tokens":0,"image_tokens":0,"cached_tokens":0},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":254,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":833,"tokens_out":90,"duration_ms":3433,"temperature":1.0,"reasoning_tokens":254,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-10T14:12:55.510022+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"On a held-out suite of the same diversity, run matched-budget independent LLM sampling past the saturation point and show that it matches or exceeds the LLM-seed-plus-NAS trajectory on most tasks; or show that a stronger, manually engineered or open-ended space around the same seeds systematically beats the LLM-constructed slotted space.","supporting_citations":[],"review_version":1}