Pith. sign in

REVIEW 2 major objections 26 references

LLM-Driven Neural Network Generation with Same-Family Architecture Guidance: Disentangling Transfer and Adaptation

T0 review · 2 major / 0 minor · reviewed 2026-07-11 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A stronger same-family network helps an LLM improve a weak target more than target-only search, mainly by adapting training recipes rather than copying them.

desk verdict Clean four-arm protocol plus hp_copy ablation that actually separates recipe transfer from LLM adaptation; real gains on CIFAR-10 and SVHN AlexNet, but family-narrow and one-epoch ranking only partially checked. read the letter →

arxiv 2607.05704 v1 pith:2DK3TVCT submitted 2026-07-06 cs.LG cs.CV

Kabir Dev Paul Baghel , Radu Timofte , Dmitry Ignatov This is my paper
classification cs.LGcs.CV
keywords LLMcodegenerationneuralarchitecturesearchsame-familytransferhyperparameterrecipeadaptationcandidatevalidityvsaccuracy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Unrestricted LLM generation of neural networks often yields invalid or weak code. This paper asks a narrower question: when you already have a weak target model, does giving an LLM a stronger same-family source model produce better valid candidates than letting it edit the target alone under the same evaluation budget? The authors define a four-arm protocol that compares non-source controls against source-conditioned hyperparameter/transform transfer and architecture edits, and they add a no-LLM direct-copy baseline that pastes the source recipe into the target. On CIFAR-10 a weak target rises from 0.1254 to 0.5049 with source guidance versus 0.2398 without it; on SVHN AlexNet the gap reaches +0.56 accuracy. The direct-copy ablation shows two regimes: sometimes the source recipe itself is already useful, and sometimes the LLM must rewrite it because the source learning rate is too small for a weak model. The practical takeaway is to run both arms and keep the best valid candidate only when it beats the original target.

What carries the argument

The frozen four-arm source-guided candidate-generation protocol (hp_default, baseline_edit, hp_transfer, analogical_edit) together with the hp_copy ablation: non-source versus source-conditioned generation under equal budgets, with validity reported separately from accuracy and acceptance only when the best valid candidate beats the original target.

What would settle it

Re-train the full candidate pools (or a large random sample) from the main CIFAR-10 and SVHN AlexNet runs for five or more epochs and test whether the source-guided best-valid advantage disappears or reverses relative to the best non-source candidates.

Watch

Extended reading notes

Core claim

Under a fixed candidate budget, exposing a stronger same-family source model to an LLM improves the best valid accuracy of a weak target over non-source controls for certain architecture families, and a no-LLM recipe-copy ablation shows the gain comes from LLM-mediated adaptation of the source training recipe rather than simple pass-through copying.

Load-bearing premise

One-epoch training accuracy is assumed to rank candidates well enough that the source-guided advantage will still hold after longer training, yet that check is shown for only four selected winners.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 0 minor

Summary. The paper studies whether a stronger same-family source model from a neural-network database helps an LLM generate better valid candidates for a weak target than non-source controls under equal evaluation budgets. It defines a four-arm protocol (hp_default, baseline_edit, hp_transfer, analogical_edit), reports validity separately from accuracy, and adds a no-LLM hp_copy ablation that decomposes recipe transfer from LLM adaptation. On CIFAR-10 the strongest source-guided candidate reaches 0.5049 vs 0.2398 non-source (+0.2651); on SVHN AlexNet with DeepSeek-Coder-6.7B it reaches 0.7880 vs 0.2254 (+0.5626), with an independent repeat and five-epoch checks. Family-level win rates are highest for AlexNet (6/8) and alt_nn1 (8/10). The authors conclude that source guidance is useful as an additional search arm, especially via hyperparameter/transform transfer, and that the effect is regime-dependent (recipe-transfer vs recipe-adaptation).

Significance. If the result holds, the paper supplies a clean, budget-controlled protocol for measuring source-conditioned LLM candidate generation rather than unrestricted code synthesis. Strengths include explicit non-source controls, validity-versus-accuracy separation, bootstrap CIs over dataset–LLM–target pairs, an independent SVHN repeat (Table 6), and the hp_copy ablation (Table 14) that falsifies pure recipe copying on SVHN AlexNet. These are concrete methodological contributions for LLM-based AutoML and retrieval-conditioned generation. The practical claim is modest and scoped: evaluate source-guided and target-only arms together and keep the best valid improver. The main limitation is that positive signals concentrate on two families and two datasets under one-epoch ranking, so the result is best read as a controlled existence proof rather than a general AutoML method.

major comments (2)
  1. Sections 4.3 and 5.1–5.2, Tables 3–4 and 7: the central claim is measured almost entirely by one-epoch best-valid accuracy. The only multi-epoch evidence is a five-epoch re-evaluation of four pre-selected strongest winners. On CIFAR-10 the advantage is preserved, but on SVHN AlexNet it collapses from +0.56 to ~+0.01–0.02 because non-source candidates catch up. The paper never re-ranks the full candidate pool (or a random subsample of non-winners) after multi-epoch training, so it is unknown whether the one-epoch argmax remains the multi-epoch argmax. This is load-bearing for the claim that source guidance improves candidate quality under equal budgets; the Limitations section already flags the gap, but the main tables should either restrict the claim to one-epoch search or add a multi-epoch re-ranking check on at least the main CIFAR-10 and SVHN settings.
  2. Section 5.3 and Tables 8–11: positive best-valid advantages concentrate on AlexNet and alt_nn1; SVHN robustness pairs (BagNet, AirNext, DPN68) mostly favor non-source, and non-AlexNet CIFAR-100/Imagenette probes are small or unstable. Family-level win rates pool runs with N < 32 and mixed LLMs. The abstract and conclusion should state more sharply that the reliable positive signal is family- and dataset-dependent rather than a general same-family transfer effect, and the main claim should be scoped to the regimes where the four-arm N=32 protocol was frozen.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claim is an empirical head-to-head of source-guided vs non-source arms under fixed budgets, not a result forced by definition or self-citation.

full rationale

This is an empirical candidate-generation study. The load-bearing claim—that same-family source guidance improves best-valid accuracy over non-source controls under equal evaluation budgets—is measured by training and evaluating independently generated candidates (Tables 3–4, 6). Source-guided advantage is defined as max a(T'_source) − max a(T'_non) over valid candidates (Eq. 2), which is not identical to any fitted input. The hp_copy ablation (Section 5.4, Table 14) further separates direct recipe copy from LLM-mediated adaptation and shows the two regimes are not the same by construction (e.g., SVHN AlexNet: hp_copy = 0.1959 matches the weak target while hp_transfer reaches 0.7880). Self-citations to LEMUR/NNGPT/ABrain work supply the model database and generation pipeline; they do not supply a uniqueness theorem, ansatz, or fitted parameter that forces the reported deltas. Validity is reported separately from accuracy, and the selection rule (keep best only if it beats the original target) is a practical policy, not a definitional identity. Partial multi-epoch validation (Table 7) is a scope/correctness limitation, not circularity. No step reduces a claimed prediction to its inputs by construction.

Assumptions & free parameters 4 free parameters · 3 assumptions · 3 invented entities

The work is empirical. Load-bearing elements are experimental design choices (candidate budget, one-epoch proxy, same-family retrieval) and domain assumptions about LLM code generation and recipe transferability; no new physical entities are postulated. Free parameters are the protocol hyperparameters that define the comparison.

free parameters (4)
  • candidates per arm N = 32 (main), 8/4 (probes)
    Main tables use N=32; robustness probes use N=8 or 4. The reported win rates and bootstrap intervals depend on this budget choice.
  • LLM sampling temperature / top-k / top-p = 0.75 / 50 / 0.95
    Fixed at 0.75 / 50 / 0.95 for all reported runs; generation diversity and validity rates are conditioned on these values.
  • training epochs per candidate = 1 (search), 5 (selected checks)
    One-epoch evaluation is the primary ranking signal; five-epoch checks are applied only post-hoc to selected winners.
  • source selection rule (highsource vs archbest)
    Two deterministic retrieval rules are used; reported advantages differ by rule, so the claim is conditioned on the chosen retrieval policy.
assumptions (3)
  • domain assumption Stronger same-family models contain transferable recipe or architectural evidence useful for improving a weak target of the same family.
    Stated as the scientific question in Section 1 and enforced by the retrieval rule in Section 3.2; without it source guidance has no expected value.
  • domain assumption Constrained LLM output contracts (hyperparameters, transforms, structured JSON edits) plus minimal deterministic validation produce comparable valid candidates across arms.
    Sections 3.3–3.4; the entire equal-budget comparison rests on this generation-and-repair contract.
  • ad hoc to paper One-epoch accuracy ranks candidate quality sufficiently for measuring source-guided advantage.
    Adopted as the primary evaluator (Section 4.3); only partially stress-tested by five-epoch re-evaluations of selected winners.
invented entities (3)
  • four-arm source-guided candidate-generation protocol (hp_default, baseline_edit, hp_transfer, analogical_edit)
    purpose: Isolate source-conditioned generation from non-source LLM baselines under equal evaluation budgets.
    Defined in Table 1 and Section 3.3; the central empirical comparison is defined by these arms.
  • hp_copy no-LLM ablation
    purpose: Separate the value of the retrieved source recipe itself from any LLM-mediated adaptation of that recipe.
    Introduced in Section 5.4; used to define the recipe-transfer versus recipe-adaptation regimes.
  • recipe-transfer vs recipe-adaptation regimes
    purpose: Interpret when source guidance succeeds by direct copy versus by LLM rewriting of the source recipe.
    Characterized via the hp_copy decomposition in Section 5.4 and Discussion; interpretive labels rather than independently measured constructs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLM-Driven Neural Network Generation with Same-Family Architecture Guidance: Disentangling Transfer and Adaptation." pith.science (2026). https://pith.science/paper/2DK3TVCT

@misc{pith2026260705704,
  author       = {Pith},
  title        = {Pith review of: LLM-Driven Neural Network Generation with Same-Family Architecture Guidance: Disentangling Transfer and Adaptation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2DK3TVCT}},
  note         = {Machine review of arXiv:2607.05704}
}
read the original abstract

Large language models (LLMs) can generate neural-network modifications, but unrestricted generation is often invalid or harmful. This paper studies a narrower setting: improving a weak target model using a stronger same-family source model from a neural-network database. We propose a source-guided candidate-generation protocol with non-source controls, source-conditioned candidates, and a no-LLM hp_copy ablation under equal evaluation budgets. The protocol reports validity separately from accuracy and selects the best valid candidate only when it improves the target. On CIFAR-10, the strongest source-guided candidate reaches 0.5049 accuracy versus 0.2398 for the best non-source candidate, a +0.2651 advantage, while improving a weak target originally at 0.1254; a five-epoch check preserves the gain at 0.7686 versus 0.4839. On SVHN AlexNet with DeepSeek-Coder-6.7B, source-guided transfer reaches 0.7880 versus 0.2254, a +0.5626 advantage; a fresh repeat reaches 0.8069 versus 0.2509, a +0.5560 advantage. Direct source-recipe copy produces 0.1959 on SVHN AlexNet, matching the original target, while hp_transfer reaches 0.7880, showing that the LLM adapts rather than copies the source recipe. Family-level analysis shows the clearest positive signals for AlexNet, with 6/8 wins across SVHN, Imagenette, and CelebA-Gender, and alt_nn1, with 8/10 wins on CIFAR-10.

Figures

Figures reproduced from arXiv: 2607.05704 by the authors.

Figure 1
Figure 1. Valid evaluation rates for the main N = 32 results. Bars show observed valid-evaluation rates; whiskers show Wilson 95% confidence intervals. Source-guided candidates have higher validity in the main grouped results, especially on CIFAR-10. Run Non valid Non best Src valid Src best Best adv. Original 64/64 0.2254 63/64 0.7880 +0.5626 Fresh repeat 64/64 0.2509 64/64 0.8069 +0.5559 [PITH_FULL_IMAGE:figures/full_fig_p… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

26 extracted references · 26 canonical work pages

  1. [1]

    Optuna: A next-generation hyperparameter optimization framework

    Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. InProceedings of the 25th ACM SIGKDD International Conference on Knowl- edge Discovery and Data Mining, pages 2623–2631, 2019. 2

    work page 2019

  2. [2]

    Bergstra, Remi Bardenet, Yoshua Bengio, and Bal- azs Kegl

    James S. Bergstra, Remi Bardenet, Yoshua Bengio, and Bal- azs Kegl. Algorithms for hyper-parameter optimization. In Advances in Neural Information Processing Systems, pages 2546–2554, 2011. 2

    work page 2011

  3. [3]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, Sand- hini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, 8 Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateu...

    work page 1901

  4. [4]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Hen- rique Ponde de Oliveira Pinto, Jared Kaplan, Harri Ed- wards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bava...

    work page Pith review arXiv 2021

  5. [5]

    Enhancing LLM-based neural network generation: Few-shot prompting and efficient vali- dation for automated architecture design

    Raghuvir Duvvuri, Chandini Vysyaraju, Avi Goyal, Dmitry Ignatov, and Radu Timofte. Enhancing LLM-based neural network generation: Few-shot prompting and efficient vali- dation for automated architecture design. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2026. to appear. 1, 2

    work page 2026

  6. [6]

    Neural architecture search: A survey.Journal of Machine Learning Research, 20(55):1–21, 2019

    Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey.Journal of Machine Learning Research, 20(55):1–21, 2019. 1

    work page 2019

  7. [7]

    LEMUR Neural Network Dataset: Towards Seamless AutoML

    Arash Torabi Goodarzi, Roman Kochnev, Waleed Khalid, Furui Qin, Tolgay Atinc Uzun, Yashkumar Sanjaybhai Dhameliya, Yash Kanubhai Kathiriya, Zofia Antonina Ben- tyn, Dmitry Ignatov, and Radu Timofte. LEMUR neural net- work dataset: Towards seamless AutoML.arXiv preprint, arXiv:2504.10552, 2025. 2

    work page Pith review arXiv 2025

  8. [8]

    Resource- efficient iterative LLM-based NAS with feedback memory

    Xiaojie Gu, Dmitry Ignatov, and Radu Timofte. Resource- efficient iterative LLM-based NAS with feedback memory. arXiv preprint, arXiv:2603.12091, 2026. 1, 2

    work page arXiv 2026

Show all 26 references
  1. [9]

    Connecting large language models with evolutionary algo- rithms yields powerful prompt optimizers.arXiv preprint, arXiv:2309.08532, 2023

    Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. Connecting large language models with evolutionary algo- rithms yields powerful prompt optimizers.arXiv preprint, arXiv:2309.08532, 2023. 2

    2023 arXiv

  2. [10]

    Roman Kochnev, Arash Torabi Goodarzi, Zofia Antonina Bentyn, Dmitry Ignatov, and Radu Timofte. Optuna vs code llama: Are LLMs a new paradigm for hyperparameter tun- ing? InProceedings of the IEEE/CVF International Confer- ence on Computer Vision Workshops (ICCVW), pages 5664– ...

    2025

  3. [11]

    NNGPT: Rethinking AutoML with large language models

    Roman Kochnev, Waleed Khalid, Tolgay Atinc Uzun, Xi Zhang, Yashkumar Sanjaybhai Dhameliya, Furui Qin, Chan- dini Vysyaraju, Raghuvir Duvvuri, Avi Goyal, Dmitry Igna- tov, and Radu Timofte. NNGPT: Rethinking AutoML with large language models. InProceedings of the IEEE/CVF Confe...

    2026

  4. [12]

    Retrieval-augmented generation for knowledge-intensive NLP tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kut- tler, Mike Lewis, Wen-tau Yih, Tim Rocktaschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. InAdvances in Neura...

    2020

  5. [13]

    Hyperband: A novel bandit-based approach to hyperparameter optimization

    Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Ros- tamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. Journal of Machine Learning Research, 18(185):1–52, 2018. 2

    2018

  6. [14]

    Competition- level code generation with AlphaCode.Science, 378(6624): 1092–1097, 2022

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Ju- lian Schrittwieser, Remi Leblond, Tom Eccles, James Keel- ing, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Pe- ter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowa...

    2022

  7. [15]

    Best practices for scien- tific research on neural architecture search.Journal of Ma- chine Learning Research, 21(243):1–18, 2020

    Marius Lindauer and Frank Hutter. Best practices for scien- tific research on neural architecture search.Journal of Ma- chine Learning Research, 21(243):1–18, 2020. 2

    2020

  8. [16]

    DARTS: Differentiable architecture search

    Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. InInternational Confer- ence on Learning Representations, 2019. 2

    2019

  9. [17]

    Self-refine: Iterative refinement with self-feedback

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hal- linan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterat...

    2023

  10. [18]

    Nasir, Sam Earle, Christopher Cleghorn, Steven James, and Julian Togelius

    Muhammad U. Nasir, Sam Earle, Christopher Cleghorn, Steven James, and Julian Togelius. LLMatic: Neural archi- tecture search via large language models and quality diver- sity optimization.arXiv preprint, arXiv:2306.01102, 2023. 2

    2023 arXiv

  11. [19]

    Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V . Le. Regularized evolution for image classifier architecture search. InProceedings of the AAAI Conference on Artificial Intelligence, pages 4780–4789, 2019. 2

    2019

  12. [20]

    So, and Quoc V

    Esteban Real, Chen Liang, David R. So, and Quoc V . Le. AutoML-Zero: Evolving machine learning algorithms from scratch.arXiv preprint, arXiv:2003.03384, 2020. 2

    2003 arXiv

  13. [21]

    Reflex- ion: Language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflex- ion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, 2023. 2

    2023

  14. [22]

    Mingxing Tan and Quoc V . Le. EfficientNet: Rethinking model scaling for convolutional neural networks. InPro- ceedings of the 36th International Conference on Machine Learning, pages 6105–6114, 2019. 2 9

    2019

  15. [23]

    LEMUR 2: Unlocking neural net- work diversity for AI

    Tolgay Atinc Uzun, Waleed Khalid, Saif U Din, Sai Re- vanth Mulukuledu, Akashdeep Singh, Chandini Vysyaraju, Raghuvir Duvvuri, Avi Goyal, Yashkumar Rajeshbhai Lukhi, Ahsan Hussain, Krunal Jesani, Usha Shrestha, Yash Mittal, Roman Kochnev, Pritam Kadam, Mohsin Ikram, Harsh Rame...

    2026

  16. [24]

    Le, Denny Zhou, and Xinyun Chen

    Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V . Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers.arXiv preprint, arXiv:2309.03409,

    arXiv

  17. [25]

    ReAct: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations, 2023. 2

    2023

  18. [26]

    Barret Zoph and Quoc V . Le. Neural architecture search with reinforcement learning.arXiv preprint, arXiv:1611.01578,

    arXiv

Pith tools

Reviewed July 11, 2026 · model on record in the stance chip above.