Pith. sign in

REVIEW 3 major objections 21 references

Ordering neural-architecture code by MinHash similarity during LLM fine-tuning raises generation reliability and structural novelty without sacrificing accuracy.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 04:32 UTC pith:IGTGEZNW

load-bearing objection Useful MinHash curriculum for LLM-NAS with honest ablations, but the progressive schedule is net harmful for interface reliability at high diversity and the evidence is single-seed N=15. the 3 major comments →

arxiv 2607.11591 v1 pith:IGTGEZNW submitted 2026-07-13 cs.CV

Similarity-Guided Curriculum Fine-Tuning of LLMs for Neural Architecture Synthesis

Anujaya Vijayakumar , Radu Timofte , Dmitry Ignatov This is my paper
classification cs.CV
keywords neural architecture searchcurriculum learningMinHashLLM fine-tuningLoRA mergingcode synthesisCIFAR-10
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Large language models can invent neural-network code for image classifiers, but throwing every architectural style at them at once tends to produce code that the automatic evaluator rejects. This paper shows that a progressive curriculum ordered by structural similarity of the source code itself lets the model first lock in the required interface conventions, then gradually absorb more heterogeneous designs. Similarity is measured with MinHash signatures over 7-gram token shingles; the best low-rank adapter from each stage is merged into the backbone before the next, more diverse stage begins. On CIFAR-10 the high-similarity stage reaches a 60 percent peak success rate with no post-processing repair, while later stages generate architectures that are measurably less similar to the original anchor yet keep high classification accuracy. A controlled ablation further shows that interface repair and curriculum ordering fix different failure modes: merge steps can erase interface knowledge that the base model still retains, so both tools are needed together.

Core claim

A MinHash-based curriculum that presents neural-architecture code in bands of increasing structural heterogeneity, with cumulative LoRA merging, produces evaluator-compatible generations (peak 60 percent success rate at high similarity without repair) and drives genuine architectural divergence from the fixed anchor while preserving or improving accuracy; interface repair and curriculum scheduling address distinct failure modes.

What carries the argument

128-permutation MinHash signatures over normalised 7-gram source-code shingles that partition the reference pool into named similarity bands (high J in [0.95,0.98), medium [0.85,0.95), low/very-low-near [0.30,0.85)), presented in order with sequential adapter merge after each stage.

Load-bearing premise

That single-seed trajectories of only fifteen generations per epoch, one base model, one fixed anchor, and one random-order comparison are enough to credit the observed reliability gains to similarity ordering rather than noise or merge artefacts.

What would settle it

Run the same curriculum-versus-random-order comparison for at least three additional random seeds with N greater than or equal to 30 generations per epoch; if the curriculum no longer shows lower epoch-to-epoch volatility or fewer total-failure epochs, the ordering claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 0 minor

Summary. The paper proposes a MinHash-based curriculum for fine-tuning a code LLM (OlympicCoder-7B) on neural-architecture source code for LLM-based NAS. Using 128-permutation MinHash over normalised 7-gram shingles, the LEMUR CIFAR-10 pool is partitioned into similarity bands of increasing heterogeneity; the best LoRA adapter from each stage is merged into the backbone before the next stage. On CIFAR-10 with N=15 candidates per epoch, the high-similarity band reaches 60% peak success rate without repair. A 2×2 ablation at the most diverse level (curriculum vs base, with/without partial interface repair) shows that without repair the base model (47% SR) outperforms the curriculum model (7% SR), while partial repair equalises both at 53% SR; the authors attribute this to merge-level drift erasing interface priors. Additional analyses cover structural divergence of generations from a fixed anchor, a single-seed random-order comparison, and an observational SVHN transfer run.

Significance. If the progressive similarity schedule were shown to improve reliability or architectural novelty under controlled multi-seed conditions, the work would supply a practical, single-GPU curriculum recipe for LLM-based NAS and a reusable MinHash retrieval layer over large architecture corpora. Strengths already present include a concrete, reproducible MinHash pipeline integrated into LEMUR (sub-second SQL UDF retrieval, pre-computed 512-byte signatures), an explicit 2×2 ablation that honestly surfaces the merge-drift failure mode, and a structural-similarity analysis showing that generations diverge from the anchor while accuracy remains high. These elements are useful even if the curriculum benefit itself remains provisional.

major comments (3)
  1. Table 3 (§5.2) and the abstract claim that curriculum and interface repair target distinct failure modes. Without repair the curriculum arm collapses to 7% SR while the base model reaches 47% SR; with partial repair both reach identical 53% SR / 0.516 score. The numbers are consistent with merge-level drift erasing L1 interface priors, but they show that the progressive schedule is net harmful for interface reliability at the hard band once diversity rises. The claimed interaction is therefore not demonstrated as a benefit of curriculum; the high-similarity 60% SR (Table 2 L1) is the easy band. Either the abstract/conclusion must be re-scoped to the observed drift phenomenon, or multi-seed evidence that curriculum improves reliability after controlling for repair must be supplied.
  2. §4.2 Statistical scope and §5.7 Limitations state that all results are single-seed, single-model, single-anchor, N=15, with no significance tests and with score differences <0.05 treated as noise. Table 6 (random-order baseline) likewise rests on one seed (42). Attribution of SR differences or reduced volatility to similarity ordering rather than seed noise, quantisation drift, or prompt changes is therefore under-supported. At minimum, 2–3 additional random-order seeds and multi-seed curriculum trajectories (or a clear re-framing as preliminary single-run observations) are required before the central ordering claim can be treated as established.
  3. Table 2 L3 k=2 is marked with partial repair while L3 k=3 and k=4 are run without repair; the 2×2 ablation is performed only at k=2. Direct comparison of curriculum progression across k values at Level 3 is therefore confounded. A uniform repair policy (or an explicit repair-only baseline) across all L3 configurations is needed to isolate the effect of reference diversity.

Circularity Check

0 steps flagged

No significant circularity: success rates and composite scores are external evaluator metrics; MinHash bands are independent of generation outcomes; self-citations supply infrastructure only.

full rationale

The paper is an empirical systems study, not a first-principles derivation. Its load-bearing quantities—success rate SR = |V|/N, composite score S = SR × max accuracy (Eq. 3), and MinHash Jaccard estimates over 7-gram code shingles (Eq. 1)—are either measured by an external LEMUR evaluator or precomputed from source-code structure before any generation. Curriculum bands are defined by fixed Jaccard ranges on stored signatures, not by generation outcomes, so band membership does not tautologically determine SR. Self-citations to NNGPT/LEMUR provide the database and evaluator infrastructure; they do not import a uniqueness theorem or force the reported SR numbers. Best-epoch selection on the evaluation set (acknowledged in §5.7) is optimistic reporting bias, not a reduction of a claimed prediction to its fitted inputs. The 2×2 ablation and random-order comparison further treat ordering as a testable variable rather than defining success by construction. No step reduces a claimed result to its inputs by definition, fit, or load-bearing self-citation chain.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 1 invented entities

The work is empirical systems research. Load-bearing choices are the MinHash configuration, band cut-points, LoRA/merge procedure, single-seed protocol, and the assumption that Jaccard over 7-gram code shingles is a meaningful structural difficulty measure. No new physical entities or free parameters fitted to a scientific law appear; free parameters are engineering hyperparameters chosen by the authors.

free parameters (4)
  • MinHash permutations / shingle length = 128 perm, n=7
    128 permutations and 7-gram shingles are fixed design choices that define all similarity bands; no sensitivity analysis is reported.
  • Similarity band boundaries = 0.95/0.85/0.30
    L1 [0.95,0.98), L2 [0.85,0.95), L3 [0.30,0.85) are hand-chosen cut-points that determine curriculum stages.
  • LoRA rank, alpha, learning rate = r=32, α=32, 1e-6
    r=32, α=32, lr=1e-6 control adapter capacity and update magnitude; chosen without ablation.
  • N candidates per epoch = 15
    N=15 sets the statistical resolution of every SR and composite-score claim.
axioms (4)
  • domain assumption Jaccard similarity over normalised 7-gram source-code shingles is a valid proxy for architectural heterogeneity / curriculum difficulty.
    Stated in §3.2 and used to construct all bands; alternative measures (AST, CodeBERT) are discussed but not used for the reported runs.
  • ad hoc to paper Sequential dequantize–merge–requantize of LoRA adapters preserves earlier coding conventions sufficiently for later stages.
    §3.5; the 2×2 ablation later shows this assumption fails for interface priors, which is itself a result.
  • ad hoc to paper Single-seed, N=15 trajectories are informative enough to report peak/mean SR and to compare curriculum vs base vs random order.
    Explicitly acknowledged in §4.2 and §5.7 as a limitation, yet all tables rest on it.
  • domain assumption LEMUR evaluator acceptance (structural validity + one-epoch CIFAR-10 training) is the correct success criterion for architecture synthesis.
    Inherited from the LEMUR benchmark; used throughout.
invented entities (1)
  • MinHash-based similarity curriculum bands for neural-architecture code no independent evidence
    purpose: Partition the reference pool into progressive structural-difficulty stages for LoRA fine-tuning.
    The band schedule and cumulative merge protocol are the paper’s central methodological construct; independent evidence is limited to the single-seed CIFAR-10 runs reported here.

pith-pipeline@v1.1.0-grok45 · 20506 in / 3279 out tokens · 27282 ms · 2026-07-14T04:32:43.351541+00:00 · methodology

0 comments
read the original abstract

Introduce a MinHash-based similarity scheduling framework that constructs a progressive curriculum over neural architecture code for LLM-based neural architecture search (NAS). Using 128-permutation MinHash signatures over normalised 7-gram source code shingles, we partition the reference pool into similarity bands and present them in increasing architectural heterogeneity, with the best LoRA adapter from each stage merged cumulatively into the backbone. We evaluate the framework on OlympicCoder-7B within the LEMUR benchmark on CIFAR-10 image classification, generating N =15 candidate architectures per epoch across six progressive fine-tuning steps. The curriculum achieves 60% peak success rate at the high-similarity level without post-processing repair. A 2*2 ablation at the most diverse level curriculum versus base model, with versus without partial interface repair reveals that without repair the base model (47% peak SR) substantially outperforms the curriculum model (7% SR), while adding partial repair brings both to 53% SR. This pattern is consistent with merge-level weight drift progressively erasing evaluator-interface priors, and suggests that interface repair and curriculum scheduling target distinct failure modes. We further report a cross-dataset transfer observation on SVHN, where direct base-model generation without curriculum warmup yields 27% peak SR at substantially lower accuracy (60.5%) than the CIFAR-10 equivalent, consistent with the increased synthesis difficulty of the unq-family anchor architecture.

Figures

Figures reproduced from arXiv: 2607.11591 by Anujaya Vijayakumar, Dmitry Ignatov, Radu Timofte.

Figure 1
Figure 1. Figure 1: Level 1 k=2 (left) and Level 2 k=2 (right) per-epoch trajectories. Top: accuracy with 95% CI. Middle: SR with Wilson 95% CI. Bottom: composite score with sliding-window CI. Dashed vertical line = epoch selected for adapter merge. a collapse at epoch A3 (accuracy 77.6%, SR 7%) at￾tributable to a pathologically misconfigured hyper￾parameter generation, before recovering to 33% SR at A4. The scores at k=3 and… view at source ↗
Figure 2
Figure 2. Figure 2: Level 2 k=3 (left) and Level 3 k=2 (right). L3 k=2 uses partial repair; dashed vertical line = merge epoch. Model Repair Best SR Mean SR Score Base ✗ 47% 26% 0.450 Curriculum ✗ 7% 7% 0.064 Base ✓ 53% – 0.516 Curriculum ✓ 53% 37% 0.516 [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Level 3 k=3 (left) and Level 3 k=4 (right). Both without repair; dashed vertical line = merge epoch. periments would need to address. 5.3 Structural Similarity Between Generated and Reference Archi￾tectures To examine whether the curriculum produces genuine architectural synthesis rather than direct reference reproduction, the structural similarity between each generated model and the fixed anchor architec… view at source ↗
Figure 4
Figure 4. Figure 4: Architecture comparison between the an￾chor reference model (left) and the best generated model at Level 3 k=4 (right). Both share the Frac￾talNet motif and pretrained backbone integration, but differ in fusion topology (3-way parallel vs. 2-way chained), conv block design (single vs. triple convolu￾tion), backbone pair selection, and spatial resolution adaptation. MinHash Jaccard: ¯j = 0.545 (very-low￾nea… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

21 extracted references · 7 linked inside Pith

  1. [1]

    NNGPT: Rethinking AutoML with large language models,

    R. Kochnev, W. Khalid, T. A. Uzun, X. Zhang, Y. S. Dhameliya, F. Qin, C. Vysyaraju, R. Duvvuri, A. Goyal, D. Ignatov, and R. Timofte, “NNGPT: Rethinking AutoML with large language models,” in Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition Workshops (CVPR W), 2026, to appear. 1

    2026

  2. [2]

    LEMUR neu- ral network dataset: Towards seamless AutoML,

    A. T. Goodarzi, R. Kochnev, W. Khalid, F. Qin, T. A. Uzun, Y. S. Dhameliya, Y. K. Kathiriya, Z. A. Bentyn, D. Ignatov, and R. Timofte, “LEMUR neu- ral network dataset: Towards seamless AutoML,” arXiv preprint, vol. arXiv:2504.10552, 2025. 1

    Pith/arXiv arXiv 2025

  3. [3]

    LEMUR 2: Unlocking neural network diversity for AI,

    T. A. Uzun, W. Khalid, S. U. Din, S. R. Mulukuledu, A. Singh, C. Vysyaraju, R. Duvvuri, A. Goyal, Y. R. Lukhi, A. Hussain, K. Jesani, U. Shrestha, Y. Mittal, R. Kochnev, P. Kadam, M. Ikram, H. R. Moradiya, A. Arslanian, D. Ignatov, and R. Timofte, “LEMUR 2: Unlocking neural network diversity for AI,” in Proceedings of the IEEE/CVF Conference on Com- puter...

    2026

  4. [4]

    Large language models as optimiz- ers,

    C. Yang, X. Wang, Y. Lu, H. Liu, Q. V. Le, D. Zhou, and X. Chen, “Large language models as optimiz- ers,”arXiv preprint arXiv:2309.03409, 2023. 2, 13 14

    Pith/arXiv arXiv 2023

  5. [5]

    Mathematical discoveries from program search with large language models,

    B. Romera-Paredes, M. Barekatain, A. Novikov, M. Balog, M. P. Kumar, E. Dupont, F. J. R. Ruiz, J. S. Ellenberg, P. Wang, O. Fawzi, P. Kohli, and A. Fawzi, “Mathematical discoveries from program search with large language models,”Nature, vol. 625, pp. 468–475, 2024. 2, 13

    2024

  6. [6]

    EvoPrompting: Language models for code-level neural architecture search,

    A. Chen, D. Dohan, and D. So, “EvoPrompting: Language models for code-level neural architecture search,” inAdvances in Neural Information Process- ing Systems (NeurIPS), vol. 36, 2023. 2, 13

    2023

  7. [7]

    ReEvo: Large language models as hyper-heuristics with reflective evolution,

    H. Ye, J. Wang, Z. Cao, F. Berto, C. Hua, H. Kim, J. Park, and G. Song, “ReEvo: Large language models as hyper-heuristics with reflective evolution,” arXiv preprint arXiv:2402.01145, 2024. 2, 13

    Pith/arXiv arXiv 2024

  8. [8]

    LLMatic: Neural architecture search via large language models and quality-diversity optimization,

    M. Zheng, X. Peng, and M. Alber, “LLMatic: Neural architecture search via large language models and quality-diversity optimization,” inProceedings of the Genetic and Evolutionary Computation Conference (GECCO), 2024. 2

    2024

  9. [9]

    GPTNAS: Large language models as neural architecture searchers,

    M. U. Nasir, J. Wan, M. Rashid, S. Khan, and F. Chollet, “GPTNAS: Large language models as neural architecture searchers,”arXiv preprint arXiv:2405.18143, 2024. 2

    Pith/arXiv arXiv 2024

  10. [10]

    Curriculum learning,

    Y. Bengio, J. Louradour, R. Collobert, and J. We- ston, “Curriculum learning,” inProceedings of the 26th International Conference on Machine Learning (ICML), 2009, pp. 41–48. 2

    2009

  11. [11]

    Self-paced learning for latent variable models,

    M. P. Kumar, B. Packer, and D. Koller, “Self-paced learning for latent variable models,” inAdvances in Neural Information Processing Systems (NeurIPS), vol. 23, 2010. 2

    2010

  12. [12]

    Automated curriculum learn- ing for neural networks,

    A. Graves, M. G. Bellemare, J. Menick, R. Munos, and K. Kavukcuoglu, “Automated curriculum learn- ing for neural networks,” inProceedings of the 34th International Conference on Machine Learning (ICML), 2017, pp. 1311–1320. 2

    2017

  13. [13]

    On the power of curriculum learning in training deep networks,

    G. Hacohen and D. Weinshall, “On the power of curriculum learning in training deep networks,” in Proceedings of the 36th International Conference on Machine Learning (ICML), 2019, pp. 2535–2544. 2

    2019

  14. [14]

    On the resemblance and containment of documents,

    A. Z. Broder, “On the resemblance and containment of documents,” inProceedings of the Compression and Complexity of Sequences, 1997, pp. 21–29. 2

    1997

  15. [15]

    A novel neural source code representation based on abstract syntax tree,

    J. Zhang, X. Wang, H. Zhang, H. Sun, K. Wang, and X. Liu, “A novel neural source code representation based on abstract syntax tree,” inProceedings of the 41st International Conference on Software Engineer- ing (ICSE), 2019, pp. 783–794. 2

    2019

  16. [16]

    CodeBERT: A pre-trained model for programming and natural languages,

    Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou, “CodeBERT: A pre-trained model for programming and natural languages,” inFind- ings of the Association for Computational Linguis- tics (EMNLP), 2020, pp. 1536–1547. 2

    2020

  17. [17]

    LoRA: Low-rank adaptation of large language models,

    E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,”arXiv preprint arXiv:2106.09685, 2021. 2

    Pith/arXiv arXiv 2021

  18. [18]

    Editing models with task arithmetic,

    G. Ilharco, M. T. Ribeiro, M. Wortsman, L. Schmidt, H. Hajishirzi, and A. Farhadi, “Editing models with task arithmetic,” inInternational Conference on Learning Representations (ICLR), 2023. 2

    2023

  19. [19]

    TIES-merging: Resolving interference when merging models,

    P. Yadav, D. Tam, L. Choshen, C. Raffel, and M. Bansal, “TIES-merging: Resolving interference when merging models,” inAdvances in Neural Infor- mation Processing Systems (NeurIPS), vol. 36, 2023. 2

    2023

  20. [20]

    Language models are super Mario: Absorbing abilities from homologous models as a free lunch,

    L. Yu, B. Yu, H. Yu, F. Huang, and Y. Li, “Language models are super Mario: Absorbing abilities from homologous models as a free lunch,”arXiv preprint arXiv:2311.03099, 2023. 2

    Pith/arXiv arXiv 2023

  21. [21]

    OlympicCoder: A chal- lenging benchmark for code generation with com- petitive programming problems,

    Open-Reasoner Team, “OlympicCoder: A chal- lenging benchmark for code generation with com- petitive programming problems,”arXiv preprint arXiv:2503.14941, 2025. [Online]. Available: https: //huggingface.co/open-r1/OlympicCoder-7B 5 15

    Pith/arXiv arXiv 2025