{"id":"0c85c8a5-5191-4447-94fc-ddd7cf0a5d50","arxiv_id":"2412.19206","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"NADER uses a multi-agent LLM team with a graph-based block representation and a reflection memory to iteratively propose and test modified neural architectures, claiming gains beyond NAS-Bench-201's optimum on CIFAR and ImageNet16-120.","lead":"NADER is a framework where several large-language-model agents collaborate to redesign neural network blocks, using a graph description instead of code. The paper reports that it finds architectures beating the best in a standard search space on small image benchmarks, but the evaluation uses the test set during the search, which undermines the result.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The search is guided by test-set accuracy (Algorithm 2 line 16), so the claimed superiority over the NAS-Bench-201 optimum is a test-selection artifact; a validation-only rerun is needed.","rationale":"The reader's weakest assumption is the most load-bearing concern, and my independent reading of the manuscript confirms it. Algorithm 2 line 16 explicitly obtains test accuracy during the search, and Section 3.3.2 explains that the Proposer uses the performance recorded in the modification tree to choose the next candidate and modification. This makes the test set part of the optimization loop. The central empirical claim, that NADER surpasses the NAS-Bench-201 optimal architecture, is a comparison of test accuracies; if one side is selected using test labels and the other is not, the comparison is not meaningful. This is not a question of novelty or writing quality; the multi-agent design is coherent and the paper is clearly written. The issue is that the evidence does not support the headline result as reported. A corrected version that selects on validation and evaluates test only at the end, with released code, could restore the comparison; until then the reader's REJECT verdict stands. No change to the verdict is needed, and I do not see a separate concern that is more load-bearing than the test-set leakage.","tokens_in":19949,"tokens_out":3265,"duration_ms":33344,"concrete_test":"Rerun the NADER pipeline exactly as described, but replace Algorithm 2 line 16 with validation accuracy, querying the test set only once after the final architecture is selected, using the same random seeds and budgets. Compare the resulting test accuracy to the NAS-Bench-201 optimal values: if the selected architectures no longer exceed 47.31% on ImageNet16-120 or 73.51% on CIFAR-100, or if test accuracy now tracks validation accuracy instead of beating it, the reported advantage is explained by test-set selection. Releasing the code would make this rerun independently checkable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, stated in Section 4.2, is that NADER designs architectures surpassing the optimal accuracy achievable within the predetermined NAS-Bench-201 search space. That claim rests on the reported test accuracies being honest estimates of generalization. Algorithm 2 line 16 trains each candidate and 'get its test accuracy on test dataset'; line 17 stores that number in the modification tree, and Section 3.3.2 states that the Proposer navigates this tree using the recorded performance data and prioritizes nodes with higher performance. Test accuracy is therefore the search objective. Repeatedly querying the test set during search biases the final reported accuracy upward, so comparing the best such architecture to the NAS-Bench-201 'Optimal' value or to baselines that select on validation is invalid. The reported numbers are consistent with this artifact: with random initialization and 10 architectures, validation/test accuracy is 91.16/94.40 on CIFAR-10, 74.41/74.51 on CIFAR-100, and 50.07/49.63 on ImageNet16-120; the 500-architecture CIFAR-100 run reports 75.72 validation versus 76.00 test. Selecting the maximum of a test-evaluated set can push test accuracy above validation accuracy. Appendix A3.2 confirms test accuracy is used throughout, showing the distribution of test accuracy of 500 models and selecting the optimal model on the CIFAR-100 test set. The empirical core of the paper is therefore invalid as reported; a corrected protocol with validation-based selection and a single final test evaluation is required before the 'beyond predetermined search spaces' claim can be assessed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes NADER, a multi-agent LLM framework for neural architecture design (NAD) that iteratively modifies a base architecture using a Reader, Proposer, Modifier, and Reflector, with a graph-based architecture representation and an experience database for reflection. The authors claim that NADER discovers architectures that, in nearly all scenarios, surpass the optimal accuracy achievable within the NAS-Bench-201 search space, using only 5-10 generated architectures, and they report large-scale experiments on CIFAR-100 reaching 76.0% test accuracy. The paper also introduces a benchmark for evaluating LLM-based architecture modification and reports ablation studies on the framework components.","tokens_in":20169,"tokens_out":3697,"duration_ms":34159,"significance":"If the reported results were valid, the framework would be a notable contribution: it combines multi-agent LLM collaboration, graph-based architecture representation, and reflection from both immediate feedback and stored experience to explore architectures outside predefined search spaces, and it explicitly credits the Reader/Proposer/Reflector design and the graph representation as enabling more efficient design than code-level generation. The NAD benchmark and the systematic ablation studies (Tables 2 and 3) are useful steps. However, the empirical core of the paper is undermined by the use of test-set accuracy to guide the search, which makes the claimed superiority over NAS-Bench-201 baselines an artifact of test-set selection rather than an unbiased estimate of generalization. The significance of the framework itself can only be assessed after the evaluation protocol is corrected.","major_comments":[{"comment":"The search objective is the test-set accuracy. Algorithm 2 lines 15-17 reads 'arch_acc_n <- Train architecture arch_n on train dataset and get its test accuracy on test dataset' and then stores that accuracy in the modification tree, while Section 3.3.2 states that the Proposer 'prioritizes nodes with higher performance scores' when navigating the tree. The test set is therefore queried repeatedly during search, and the final architecture is selected on the basis of these queries. Appendix A3.2 confirms this by describing the distribution of test accuracy of 500 models and selecting the optimal model on the CIFAR-100 test set. Consequently, the reported test accuracies are not independent estimates of generalization, and the core claim that NADER 'surpasses the optimal accuracy achievable within the predetermined search space' (Section 4.2) is not supported. A corrected protocol should perform architecture selection on the validation split and evaluate the final architecture on the test set exactly once.","section":"Algorithm 2 / Section 3.3.2 / Appendix A3.2"},{"comment":"The comparisons in Table 1 are unfair because NADER selects on test accuracy while the NAS baselines (RS, REA, BOHB, GENIUS, LLMatic, etc.) and the 'Optimal' row are computed with validation-based selection or are the known test accuracy of the best architecture in the search space. The reported numbers show the expected signature of test-set selection: validation accuracy is sometimes lower than test accuracy for the same method (e.g., Random initialization, 10 architectures on CIFAR-10: 91.16 validation vs 94.40 test; ResNet initialization, 500 architectures on CIFAR-100: 75.72 validation vs 76.00 test). Selecting the maximum of a set of noisy test evaluations can push the best test accuracy above both the validation accuracy and the NAS-Bench-201 'Optimal' test accuracy, so the claimed superiority over the search-space optimum is not evidence of out-of-search-space performance.","section":"Table 1"},{"comment":"The large-scale NAD experiment is also affected by the same protocol problem. Section 4.4 reports that the final architecture achieves 76.0% on the test set, and Figure 4 colors nodes by test accuracy; Appendix A3.2 explicitly states that the distribution of test accuracy of 500 models is used and that the optimal model is selected on the CIFAR-100 test set. The consequent cross-dataset generalization evaluation in A3.3 (NADER-500 evaluated on GTSRB and STL-10) is not a clean test of generalization, because the architecture itself was selected using the CIFAR-100 test set. A validation-based rerun of the large-scale experiment is needed before any conclusions about the method's ability to discover architectures beyond the search space can be drawn.","section":"Section 4.4 / Figure 4 / Appendix A3.3"}],"minor_comments":[{"comment":"The column headers in Table 2 run together ('validation test validation test'), making the table hard to read; please reformat with clear separators.","section":"Table 2"},{"comment":"There are several typos: 'paramerts' for 'parameters' in Section 4.2, 'Futhermore, we evalute' in Section 4.4, 'Trail' for 'Trial' in Appendix A3.1, and 'Algorithm 2 shows the design process of neural architecture' in Section 3.1 appears to be missing the word 'design'. Please proofread.","section":"Various places"},{"comment":"The caption says 'We report the mean validation and test accuracies for 5 three runs'; the phrase '5 three runs' is unclear and should be rewritten, e.g., 'for three runs per setting'.","section":"Table 1 caption"}],"recommendation":"reject","confidential_remarks":"The paper has a promising framework and the authors have put effort into the multi-agent design and the graph representation, but the central empirical claim is invalid because the search is guided by test-set accuracy. This is not a minor fix: the headline numbers, the comparison table, and the large-scale experiment all need to be redone with validation-based selection. If the authors are willing to rerun the experiments under a correct protocol and the results still support their claims, a future revision could be considered, but the current manuscript cannot be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the test-set leakage in Algorithm 2 is real, and it undermines the headline claim. The paper transparently reports using test accuracy to guide the search (Algorithm 2 line 16, Section 3.3.2, Appendix A3.2), so the reported 'beyond predetermined search space' accuracies are not honest estimates of generalization. Comparing those numbers to the NAS-Bench-201 optimum—a fixed value, not a test-selected maximum—is invalid, and the comparison against NAS baselines that select on validation is unfair for the same reason.\n\nThat said, the system itself is worth taking seriously. The multi-agent design (Reader/Proposer/Modifier/Reflector), the DAG representation, and the experience memory form a genuine combination not found in the cited GENIUS/EvoPrompting/LLMatic/LeMo-NADe work. The writing is clear, and the ablations give some insight into which components matter. The graph representation is a sensible way to keep LLMs focused on structure rather than code syntax. If the protocol were fixed to select on validation and to touch the test set only once at the end, the framework could plausibly become a useful tool for open-ended architecture design.\n\nThe soft spots are mostly downstream of that one flaw. Three-seed runs are noisy, no code is released, and the self-created NAD benchmark is hard to interpret without code. But none of those matter as much as the central issue. The authors are transparent about their protocol, which suggests the flaw is a protocol mistake rather than an attempt to hide anything.\n\nAs submitted, I would reject. The central empirical claim is not supported. But this is a paper that deserves a serious referee if the evaluation is redone: validation-based selection, a single final test evaluation, released code, and more seeds. I would be happy to see that version.","headline":"The framework is novel and clearly described, but the headline result is a test-selection artifact because Algorithm 2 uses test-set accuracy to drive the search.","tokens_in":20808,"tokens_out":1922,"would_cite":false,"duration_ms":19650,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"NADER shows that multi-agent LLM collaboration can design neural networks that outperform the best architecture inside the predetermined NAS-Bench-201 search space.","keywords":["neural architecture design","multi-agent LLM collaboration","LLM agents","NAS-Bench-201","graph-based architecture representation","experience reflection","open-ended architecture search","test-set guided selection"],"falsifier":"Retrain the best architectures reported for ImageNet16-120 and CIFAR-100 from scratch while selecting modifications using validation accuracy only, never test accuracy, then evaluate once on the test split; if the resulting accuracy falls to or below the NAS-Bench-201 optimum, test-set selection drove the claimed improvement.","tokens_in":19690,"feed_emoji":"🤖","tokens_out":6237,"duration_ms":60519,"temperature":0.7,"pith_summary":"NADER is a claim about open-ended architecture design: a team of large-language-model agents can improve a base network by proposing structural modifications one step at a time, without being confined to a search space that an expert predefines. The paper reports that this loop finds networks exceeding the best accuracy achievable inside NAS-Bench-201, a standard benchmark search space, on nearly every dataset it tries. For example, ten designed architectures reach 49.63% test accuracy on ImageNet16-120, above the space's 47.31% optimum, and 500 designs reach 76.00% on CIFAR-100, 5.14 points above its ResNet starting point. If correct, this would mean the bottleneck in architecture discovery is not the search algorithm alone but the ceiling imposed by the search space itself.","feed_headline":"Multi-agent LLMs beat NAS-Bench-201's optimum in 10 designs","feed_subtitle":"An open-ended LLM design loop with learned experience reaches 49.63% on ImageNet16-120, past the 47.31% search-space ceiling.","key_machinery":"The mechanism is a four-agent loop plus a graph-based network representation. Architectures are written as single directed acyclic graphs, with nodes as operations and edges as information flow, so the LLM works on structure rather than code syntax. The Reader distills design ideas from recent papers, the Proposer picks a candidate and a modification suggestion, the Modifier rewrites the graph, and the Reflector checks executability, explains failures, and stores reusable design experience. The modification tree records every attempted architecture and its measured accuracy, and the Proposer navigates that tree to choose which network to develop next.","core_discovery":"On the paper's own terms, the central discovery is that an LLM-based multi-agent system can treat architecture design as iterative, open-ended modification and, in nearly all benchmark scenarios, produce networks that beat the optimal accuracy reachable inside the NAS-Bench-201 search space. Starting from either a ResNet or a randomly sampled network, NADER runs a loop in which the Proposer selects a promising node in a modification tree, the Modifier edits the network's graph representation, and the Reflector verifies executability and accumulates design experience. With ten architectures designed, test accuracy reaches 49.63% on ImageNet16-120, above the 47.31% optimum; with 500 designs on CIFAR-100 it reaches 76.00%, 5.14 points above its ResNet initialization. The paper also reports that this large run cost only about $0.046 per architecture in LLM tokens.","pith_inferences":["A validation-only replay of the search would show how much of the reported lead over the NAS-Bench-201 optimum comes from test-set selection rather than from architectural quality.","The same loop could be run with accuracy replaced by latency or memory, turning the design objective into any measurable property; the architecture graph is a convenient substrate for such multi-objective design.","The paper's own observation that 19.2% of 500 designs fail to converge suggests a cheap early-stopping signal based on loss behavior in the first epochs could make the search considerably cheaper, an optimization the authors mention but do not formalize.","Because the Reader ingests recent papers, the framework's knowledge base will age; tracking which retrieved inspirations actually cause accuracy gains would let the system prune stale knowledge automatically."],"forward_implications":["If the framework works as claimed, a handful of design cycles (5 to 10) can match or exceed what thousands of evaluations find inside a fixed search space.","The graph representation and experience-retrieval design suggest the same agents can be pointed at other building blocks, such as GoogLeNet, ConvNeXt, or SE-ResNet blocks, without redefining a search space.","The 500-iteration result implies the method keeps improving with budget, and the resulting architecture transfers to other datasets, including GTSRB and STL-10.","The reported success rates and token counts imply the cost of open-ended design is small enough to be a practical alternative to conventional NAS.","The paper's graph representation also supports isomorphism checking, so repeated or duplicate architectures are not trained twice."],"supporting_citations":[{"why":"Supplies the LLM backbone (GPT-4o) that every agent in the system runs on.","marker":"[1]"},{"why":"Provides the code-LLM evolutionary baseline for architecture generation that NADER compares against.","marker":"[4]"},{"why":"Defines the NAS-Bench-201 search space, the three datasets, and the optimal-accuracy reference that the central claim is measured against.","marker":"[11]"},{"why":"Supplies the ResNet block used as the initial architecture for the design loop.","marker":"[14]"},{"why":"Supplies the multi-agent collaborative assembly-line pattern that the Research and Development team structure extends.","marker":"[15]"},{"why":"Provides the quality-diversity LLM search baseline that NADER is compared with.","marker":"[26]"},{"why":"Supplies the closest earlier NAD method, which NADER outperforms with fewer trials.","marker":"[31]"},{"why":"Supplies the earlier LLM-based NAS method whose prompt-based search NADER contrasts with multi-agent design.","marker":"[50]"}],"fun_headline_variants":["LLM agent team designs nets beyond NAS-Bench-201 ceiling","Multi-agent LLMs crack NAS-Bench-201 in 10 tries","NADER: AI agents design better nets than search spaces allow","Open-ended LLM design loop beats NAS-Bench-201 optimum","LLM multi-agent design hits 49.63% on ImageNet16-120"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The search loop reads the test accuracy of each candidate architecture and uses that accuracy to choose the next modification; if using the test set to guide design is not a valid protocol, the reported numbers overstate how well the method generalizes.","fun_headline_variants_meta":{"raw":{"variants":["LLM agent team designs nets beyond NAS-Bench-201 ceiling","Multi-agent LLMs crack NAS-Bench-201 in 10 tries","NADER: AI agents design better nets than search spaces allow","Open-ended LLM design loop beats NAS-Bench-201 optimum","LLM multi-agent design hits 49.63% on ImageNet16-120"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000205,"raw_usage":{"total_tokens":1396,"prompt_tokens":948,"completion_tokens":448,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":353}},"tokens_in":564,"tokens_out":448,"duration_ms":5208,"temperature":1.0,"reasoning_tokens":353,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:49:17.511852+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the best architectures reported for ImageNet16-120 and CIFAR-100 from scratch while selecting modifications using validation accuracy only, never test accuracy, then evaluate once on the test split; if the resulting accuracy falls to or below the NAS-Bench-201 optimum, test-set selection drove the claimed improvement.","supporting_citations":[{"cited_title":"Evoprompting: language models for code-level neural architecture search","cited_arxiv_id":null,"evidence_quote":"Provides the code-LLM evolutionary baseline for architecture generation that NADER compares against."},{"cited_title":"Deep residual learning for image recognition","cited_arxiv_id":null,"evidence_quote":"Supplies the ResNet block used as the initial architecture for the design loop."}],"review_version":1}