{"id":"3a9d3e59-38b2-4ba5-ab72-a434ea4534f0","arxiv_id":"2504.12971","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A fine-tuned language model on grammar derivation strings predicts architecture accuracy across datasets and improves evolutionary NAS search in the expressive einspace space.","lead":"Surrogate models trained on zero-cost proxies, graph features, or fine-tuned language models can predict the accuracy of neural architectures in the grammar-based einspace search space, and can guide evolutionary search to better results on most of nine image classification datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline claim of 'consistently stronger final performances' is not supported by Table 4, and the transfer-RF row rests on a disclosed bug and imputed truncated runs.","rationale":"The reader's stated weakest assumption is the sufficiency of the derivation-tree-plus-shape encoding. I do not think that is the most load-bearing point: Appendix C.1 directly tests the encoding and shows that adding shape metadata raises Kendall correlation from 0.448 to 0.612, so the paper has explicit evidence on that question and the encoding is at least empirically adequate. The more serious, unresolved issue is that the payoff results of the paper, namely the search improvements and transfer claims, are partly built on a disclosed implementation bug and truncated, imputed runs, while the text overstates consistency using its own Table 4. The reader's rationale already flags the synflow bug and the 'consistently' contradiction, so I partially agree; I differ on which assumption is weakest. The disclosed Appendix B.4 footnote is an in-scope admission that the reported transfer-RF results are not final. A conditional verdict remains appropriate: accept only after the corrected runs are reported and the abstract is aligned with the actual per-task results.","tokens_in":31033,"tokens_out":12633,"duration_ms":130508,"concrete_test":"Re-run the seven truncated transfer-RF experiments to 300 iterations with the corrected synflow scaling (two seeds on AddNIST and MultNIST; one seed each on Chesseract, Gutenberg, GeoClassing and Language) and recompute the Evolution(RF Transfer) row of Table 4 and Figure 2. Independently, for each dataset compute a paired bootstrap over the three seeds for Evolution(BERT) minus baseline Evolution and report 95% confidence intervals. If the corrected transfer row changes materially, or if the BERT advantage is not clearly significant on more than a couple of tasks, the 'consistently stronger' claim in the abstract should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that surrogate-guided evolution beats baseline 'consistently' and that surrogates transfer across datasets. Two pieces of evidence undermine this as currently reported. First, the transfer version of the RF surrogate in Table 4 and Figure 2 was produced with an inconsistent synflow scaling that the authors found and fixed only shortly before the deadline; 7 of the 27 transfer experiments were not re-run to 300 iterations, and their remaining iterations were imputed with the best value found so far (Appendix B.4 footnote). Any conclusion about transfer-RF search, specifically the claim that transfer surrogates 'filter out bad architectures when searching on novel datasets', is therefore unverified. Second, the 'consistently' language in the abstract and Section 1 is contradicted by Table 4 itself: Evolution(BERT) is below baseline on MultNIST, CIFARTile and GeoClassing; Evolution(BERT as obj) is below baseline on 7 of 9 tasks; and with only 3 seeds, most reported differences are within the stated standard errors. The central idea may survive a corrected rerun, but the strength of the current claims outruns the evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies surrogate models for neural architecture search in the grammar-based einspace search space. It proposes two families of surrogates: a random forest trained on zero-cost proxy and GRAF features, and a fine-tuned ModernBERT language model trained on string representations of architecture derivation trees augmented with output-shape metadata. The authors report rank correlations within and across datasets, evaluate surrogate-guided regularized evolution, and also test using the surrogate directly as the search objective. Their central claims are that the surrogates have high predictive power, that they transfer across datasets, that they can filter out bad architectures when searching on novel datasets, and that they can be used as a standalone search objective for large speed-ups.","tokens_in":31265,"tokens_out":7879,"duration_ms":73164,"significance":"If the claims hold, this would be a valuable step toward efficient NAS in large, expressive, grammar-based search spaces where fixed cell encodings do not apply. The paper is the first to evaluate a broad set of performance predictors in einspace, and the LM-based surrogate on derivation-tree strings is a novel and interesting idea. The authors are transparent about standard errors and explicitly list limitations, and the appendix discloses a bug that affects the transfer-RF experiments. However, the strength of the headline claims currently exceeds what the evidence supports, and the main transfer-related search result is explicitly labeled as not final. The central idea is plausible and worth publishing after a substantive revision that brings the claims in line with the data.","major_comments":[{"comment":"The claim that surrogate-guided evolution 'consistently achieves stronger final architecture performances' is contradicted by Table 4. Evolution(BERT) is numerically worse than baseline on MultNIST (0.765 vs 0.841) and CIFARTile (0.341 vs 0.358), and the apparent gains on GeoClassing (0.735 vs 0.725) and Chesseract (0.606 vs 0.595) are within one standard error; several other differences (e.g., AddNIST, Language) are also within one standard error. With only three seeds per condition, the evidence supports 'competitive or better on some tasks', not 'consistently stronger final architecture performances'.","section":"Abstract; Section 4.1, Table 4"},{"comment":"The transfer-RF search results are presented as final despite the footnote in Appendix B.4 stating that an inconsistency in synflow scaling was found and fixed only shortly before the deadline, that 7 of 27 transfer experiments were not re-run to 300 iterations, and that the remaining iterations were imputed using the best value found so far. The authors write that they 'will update the results for the final version of the paper.' Since claim (ii) about filtering out bad architectures on novel datasets rests on this row, the evidence is unverified as submitted. The affected experiments should be re-run at full length, or the corresponding claims should be narrowed or marked preliminary.","section":"Appendix B.4 footnote; Table 4 'Evolution(RF Transfer)'"},{"comment":"The text states that 'ModernBERT-large consistently outperforms the random forest model', but Table 3 shows the random forest with higher Spearman/Kendall correlations on Isabella (0.308/0.211 vs 0.273/0.186), GeoClassing (0.693/0.504 vs 0.661/0.475), and Chesseract (0.599/0.423 vs 0.545/0.384). The word 'consistently' should be removed, and the description should acknowledge that the LM advantage is an average trend rather than a per-dataset guarantee.","section":"Section 4.1, Tables 2 and 3"},{"comment":"The claim that surrogates 'can be used to filter out bad architectures when searching on novel datasets' is tested in search only with the random-forest transfer surrogate (Evolution(RF Transfer)). No search experiment uses a transfer-trained ModernBERT-large surrogate; the LM transfer evidence is limited to static correlation tables (Tables 2 and 3). As a result, the transfer capability of the paper's proposed best method is untested in the search setting. Either add a transfer-LM search experiment or limit the transfer-search claim to the RF variant.","section":"Section 3.4; Section 4.1, Table 4"},{"comment":"The paper states that the best LM checkpoint is saved based on Kendall Tau correlation on the evaluation set, but Table 1 reports correlations on an evaluation set for the same CIFAR10 data. Please clarify whether the reported evaluation set is disjoint from the checkpoint-selection set; if the same data are used for both checkpoint selection and final reporting, the absolute correlations are optimistically biased and model comparisons may be unfair.","section":"Appendix B.2; Table 1"}],"minor_comments":[{"comment":"The header contains the typo 'Langauge' instead of 'Language'.","section":"Table 4"},{"comment":"The sentence 'This augmentation changes the the architecture functionally' contains a duplicated article; it should read 'changes the architecture functionally'.","section":"Appendix A.1, item 4"},{"comment":"The caption ends with a stray '>' character after 'baseline'; it should read 'outperforms the baseline.'","section":"Figure 2 caption"},{"comment":"The table caption does not explain the slash-separated format (e.g., '0.612/0.648'); please add a note stating which number belongs to ModernBERT-large and which to the random forest.","section":"Table 2"},{"comment":"The phrase 'we restrict our analysis to a single operation type' is vague; please specify which operation type was used and how it was selected for the node-degree features.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The Appendix B.4 footnote indicates that the transfer experiments were still being corrected and re-run at the time of submission. Given that the paper's title and abstract emphasize transferability, I strongly recommend that the final corrected transfer results be required as part of the revision rather than allowing the current preliminary numbers to stand."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a legitimately useful empirical paper on surrogate predictors in the einspace grammar-based NAS space, with a genuinely new LM-based surrogate that encodes derivation strings plus shape metadata. It is the first broad evaluation of predictors in that space, and the correlation results are solid. But the search claims are louder than the evidence: Table 4 does not show consistent improvements, and the transfer-RF numbers are compromised by a disclosed bug and imputed runs. The core idea is probably right, but the paper needs a corrected rerun and a toned-down abstract before I'd trust the headline.\n\nWhat's new and good: the encoding study is careful. Adding shape metadata to derivation strings lifts Kendall from 0.448 to 0.612, and the leave-one-out transfer correlations are decent, especially on MultNIST and Gutenberg. The GRAF adaptation is sensible, and the authors are transparent about standard errors and limitations. They also disclose the synflow scaling bug in Appendix B.4 and say they will update results, which is honest.\n\nSoft spots, in proportion: the abstract and intro claim the surrogates 'consistently achieve stronger final architecture performances than baseline search.' Table 4 contradicts that. Evolution(BERT) is below baseline on MultNIST (0.765 vs 0.841) and CIFARTile (0.341 vs 0.358), and Evolution(BERT as obj) is below baseline on 7 of 9 tasks. With three seeds, most differences are within the reported standard errors. So 'consistently' is not supported. The transfer-RF results are weaker still: 7 of 27 experiments were not re-run to 300 iterations, and the remaining iterations were imputed with the best value found so far. Any conclusion about transfer surrogates filtering out bad architectures on novel datasets is unverified as reported. The authors flag this, but they still present the results as evidence.\n\nThe weakest assumption, the encoding being sufficient, is actually tested in the appendix and looks reasonable. The shape ablation suggests the encoding captures useful signal. I don't see a circularity problem: the surrogates are fit to held-out runs, and GRAF is externally published.\n\nBottom line: the correlation results and the LM surrogate are worth having. The search-results section and the abstract need revision, and the transfer-RF experiments should be re-run before publication. This deserves a serious referee: send it out, but a good reviewer will push for the rerun and a claim-level rewrite. I'd bring it to a reading group with the caveat, and I might cite the correlation study once the search claims are fixed.","headline":"Useful empirical study of LM surrogates in einspace, but the search claims outrun the evidence: Table 4 contradicts 'consistently' and the transfer-RF results rest on imputed runs.","tokens_in":31813,"tokens_out":2476,"would_cite":false,"duration_ms":24953,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a fine-tuned language model can act as a surrogate for architecture accuracy in a flexible grammar-based search space, transferring across datasets and steering evolutionary search to stronger final models in far…","keywords":["neural architecture search","surrogate models","language models","grammar-based search space","evolutionary search","zero-cost proxies","performance prediction","transfer learning"],"falsifier":"A decisive test is to search for string collisions: two architectures whose derivation-tree strings and shape annotations are identical but whose compiled networks differ, for example through stochastic training or hidden implementation choices in the grammar's operators. If the true validation accuracies of such pairs differ substantially, the encoding is under-informative and the surrogate cannot be a reliable predictor, since it has no way to distinguish them. A second check is leave-one-out transfer to a task whose architectures use operation statistics far outside the training distributions, such as a very different image resolution or sequence length; a collapse in rank correlation would indicate that the transfer claim is limited to aligned tasks.","tokens_in":30844,"feed_emoji":"🧠","tokens_out":7463,"duration_ms":73938,"temperature":0.7,"pith_summary":"This paper is trying to establish that cheap surrogate models can make neural architecture search (NAS) practical in large, expressive search spaces defined by context-free grammars, where previous predictors designed for small cell-based spaces do not transfer. The central result is that a fine-tuned ModernBERT-large language model, fed the string form of an architecture's derivation tree enriched with output tensor shapes, predicts validation accuracy with Kendall rank correlation around 0.61 on CIFAR-10 and transfers to the unseen NAS benchmark tasks, reaching 0.658 on Gutenberg in leave-one-out experiments. These predictions are strong enough to guide regularised evolution: surrogate-guided search reaches the baseline's best accuracy in far fewer iterations and, on most of the nine datasets, ends with higher final accuracy than the unguided baseline. The paper further claims the surrogate can be used directly as the search objective, with a large speed-up and competitive results on some tasks. If right, this would remove the need to fully train most candidate architectures during search in grammar-based spaces.","feed_headline":"Language model predicts neural architecture accuracy, speeding search","feed_subtitle":"Fine-tuned on derivation-tree strings with shape info, it beats unguided evolution on most tasks in fewer evaluations.","key_machinery":"The load-bearing object is the derivation-tree string encoding of an architecture: a bracket-structured text that spells out the branching, routing, aggregation and computation operations generated by the context-free grammar, optionally annotated with the output tensor shape of each operation. This string is what a language model is fine-tuned on, using a mean-squared-error loss against true validation accuracy; the shape annotations matter, since they lift Kendall correlation from 0.448 to 0.612 in the paper's ablation. For the non-LM baseline, the machinery is a descriptor concatenating zero-cost proxy scores (grad_norm, snip, grasp, fisher, jacob_cov, plain, synflow) with GRAF topological features, fed to a random forest or gradient-boosted tree regressor. The surrogate is inserted into regularised evolution at two points: selecting the top-k offspring to fully evaluate, and optionally replacing the evaluation objective entirely.","core_discovery":"On its own terms, the paper's discovery is that the derivation-tree string representation of architectures in einspace is a usable substrate for performance prediction. ModernBERT-large fine-tuned on these strings with output-shape metadata achieves the highest rank correlations among all tested predictors (Spearman 0.745, Kendall 0.612 on held-out CIFAR-10 architectures; 0.769/0.628 with data augmentation), clearly above a random forest trained on zero-cost proxy and GRAF graph features (0.663/0.514). The same model generalises across datasets, with leave-one-out Kendall correlations up to 0.658 on Gutenberg and 0.625 on MultNIST, and it outperforms plain few-shot prompting with open LLMs. In search, using the surrogate to select which mutated offspring to train (sampling k=5 from n=20 candidates) improves final accuracy over baseline evolution on most of the Unseen NAS datasets, and the surrogate-as-objective variant is competitive on some datasets while skipping most full evaluations.","pith_inferences":["We infer that the output-shape annotations act as a disambiguating signal for tensor flow, telling the language model how information enters and leaves each module; without it, the string under-determines the computation.","We infer that the same string-surrogate approach should extend to other grammar-defined search spaces, since only the grammar and the string encoder need to change, though the paper does not test this.","We infer that the Isabella and Chesseract results reveal a domain-alignment bottleneck: surrogates transfer well within a modality but struggle when architecture statistics differ sharply, so a practical system would likely need per-modality surrogates.","We infer that the augmentation scheme (reordering branches, identity insertions, shape perturbation) acts as an implicit regulariser and could be pushed further with contrastive or variational string objectives."],"forward_implications":["Surrogate-guided search can reach the accuracy of the unguided baseline in many fewer iterations, so the wall-clock cost of exploring a grammar-defined space drops sharply.","A surrogate trained once on several datasets can be applied zero-shot to a new task, meaning a new NAS run may not need an initial fully-evaluated population of architectures, instead using transferred predictions to filter candidates.","Using the surrogate directly as the objective removes almost all full training from the search loop, trading final accuracy for an order-of-magnitude speed-up and offering a cheap way to initialise or prune search populations.","The gap between the best surrogate (fine-tuned ModernBERT-large) and the random forest baseline shows that learned encodings capture more of what determines accuracy than hand-engineered graph and cost features in expressive spaces."],"supporting_citations":[{"why":"Defines the einspace grammar-based search space and the baseline regularised-evolution search that the surrogates are built on and compared against.","marker":"Ericsson et al. (2024)"},{"why":"Supplies the GRAF topological features that, combined with zero-cost proxies, form the descriptor for the random-forest surrogate.","marker":"Kadlecová et al. (2024)"},{"why":"Provides the zero-cost proxy metrics (grad_norm, snip, grasp, fisher, jacob_cov, plain, synflow) used as part of the feature-based surrogate.","marker":"Abdelfattah et al. (2021)"},{"why":"Introduces the LLM performance-predictor setup and PP prompts that the paper adapts for few-shot and fine-tuned language-model surrogates.","marker":"Jawahar et al. (2024)"},{"why":"Introduces the ModernBERT model family from which the best-performing surrogate is fine-tuned.","marker":"Warner et al. (2024)"},{"why":"Defines the regularised evolution algorithm that the search procedure extends with surrogate-based selection.","marker":"Real et al. (2019)"},{"why":"Provides the Unseen NAS benchmark datasets used to test within- and cross-dataset transfer of the surrogates.","marker":"Geada et al. (2024)"}],"fun_headline_variants":["LM on architecture strings predicts accuracy, speeds NAS","Cross-dataset surrogate: fine-tuned LM outperforms proxies","Surrogate as search objective: huge NAS speed-up","Read architecture strings like text to predict model performance","Transferable LM surrogates accelerate neural architecture search"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's strongest results rest on the assumption that the derivation-tree string plus output tensor shapes is a faithful encoding of everything that determines an architecture's accuracy; if two functionally different architectures can share the same string, or if the string omits a decisive detail, the language-model surrogate cannot be right, no matter how well it is trained.","fun_headline_variants_meta":{"raw":{"variants":["LM on architecture strings predicts accuracy, speeds NAS","Cross-dataset surrogate: fine-tuned LM outperforms proxies","Surrogate as search objective: huge NAS speed-up","Read architecture strings like text to predict model performance","Transferable LM surrogates accelerate neural architecture search"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000847,"raw_usage":{"total_tokens":3655,"prompt_tokens":882,"completion_tokens":2773,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":2697}},"tokens_in":498,"tokens_out":2773,"duration_ms":21995,"temperature":1.0,"reasoning_tokens":2697,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:17:45.547199+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive test is to search for string collisions: two architectures whose derivation-tree strings and shape annotations are identical but whose compiled networks differ, for example through stochastic training or hidden implementation choices in the grammar's operators. If the true validation accuracies of such pairs differ substantially, the encoding is under-informative and the surrogate cannot be a reliable predictor, since it has no way to distinguish them. A second check is leave-one-out transfer to a task whose architectures use operation statistics far outside the training distributions, such as a very different image resolution or sequence length; a collapse in rank correlation would indicate that the transfer claim is limited to aligned tasks.","supporting_citations":[{"cited_title":"J., Cohen, S","cited_arxiv_id":null,"evidence_quote":"Defines the einspace grammar-based search space and the baseline regularised-evolution search that the surrogates are built on and compared against."},{"cited_title":"S., Mehrotra, A., Dudziak, ., and Lane, N","cited_arxiv_id":null,"evidence_quote":"Provides the zero-cost proxy metrics (grad_norm, snip, grasp, fisher, jacob_cov, plain, synflow) used as part of the feature-based surrogate."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the LLM performance-predictor setup and PP prompts that the paper adapts for few-shot and fine-tuned language-model surrogates."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the regularised evolution algorithm that the search procedure extends with surrogate-based selection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Unseen NAS benchmark datasets used to test within- and cross-dataset transfer of the surrogates."}],"review_version":1}