Pith. sign in

REVIEW 3 major objections 5 minor 13 references

In scientific equation discovery, LLM evolution does not beat fresh sampling; discovery is carried by set-level selection over a term dictionary.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 21:37 UTC pith:5JNSTMTP

load-bearing objection The matched-budget audit is the real contribution: parent-conditioned LLM evolution is just resampling, and set-level selection over term dictionaries does the work. the 3 major comments →

arxiv 2607.04108 v1 pith:5JNSTMTP submitted 2026-07-05 cs.LG

Dictionaries, Not Darwin: Set-Level Selection Beats LLM Evolution in Scientific Equation Discovery

Pan Li This is my paper
classification cs.LG
keywords scientific equation discoveryLLM evolutionsymbolic regressionset-level selectionsparse dictionary regressionproposal banksidentifiabilityPTB-Search
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper audits a common practice: using large language models as evolutionary engines that generate candidate equations, keep winners, and mutate them across generations. Under matched call budgets on scientific equation discovery, that parent-conditioned loop is no better than drawing the same number of independent proposals, multi-parent crossover is worse, and success is already fixed by the quality of the first proposals. What the loop actually produces is a dictionary of reusable terms. The authors turn that diagnosis into PTB-Search: sample proposals once, extract additive terms, and run one train-only sparse selection that scores candidate sets jointly with least-squares coefficients. On identical dictionaries, set-level selectors roughly double the solves of single-term credit rules. On the official 239-problem benchmark they report 73.2% numeric accuracy with a small open model and 77.0% with a stronger single-seed anchor, versus 49.2% for the best reported baseline, at one tenth of the standardized LLM budget. A program-domain stress test keeps generation count from becoming the active ingredient and points instead to retained external state. The upshot is a division of labor: LLMs supply material; discovery is done by external set-level selection over reusable components.

Core claim

Under matched LLM-call budgets in continuous equation discovery, parent-conditioned evolution is indistinguishable from fresh independent sampling, so the loop does not compound scientific structure. Operationally it reduces to a dictionary of candidate terms; discovery is then carried by train-only, set-level sparse recombination, because underdetermined data identifies the joint behavior of term sets rather than reliable per-term credit. PTB-Search implements that reduction and substantially outperforms the best reported baseline at one tenth of the standardized call budget.

What carries the argument

PTB-Search (proposal term bank search): one-shot independent LLM proposals, extraction of reusable additive terms into a per-problem dictionary, then a single global train-only best-subset sparse selection with least-squares coefficients. Its load-bearing principle is set-level identifiability—score candidate supports jointly rather than reduce to per-term statistics.

Load-bearing premise

The method assumes that useful scientific expressions can be broken into flat, reusable additive terms that can be extracted into a dictionary and scored jointly under a small support-size budget; if the true structure is non-additive or terms cannot be cleanly extracted, the reduction and the gains do not transfer.

What would settle it

On a budget-matched equation-discovery grid with the same proposer and train-only selection, show that parent-conditioned multi-generation proposals systematically beat the same number of independent samples on held-out OOD accuracy, or that single-term credit selectors match set-level joint selectors on identical dictionaries.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper audits parent-conditioned LLM evolution loops for scientific equation discovery under matched call budgets and finds them indistinguishable from fresh independent sampling (median OOD NMSE 0.045 vs 0.049), with instructed multi-parent crossover worse, final success predicted by initial proposals (AUC 0.959), and three iteration schemes flat or destructive. It reduces the loop operationally to a dictionary of candidate terms and proposes PTB-Search: one-shot independent LLM proposals, term extraction into a per-problem dictionary (pooled with generic primitives), and a single train-only set-level sparse subset search with least-squares coefficients (|S|≤6). On identical dictionaries, set-level selectors solve 165–169 of 717 cells versus 74–78 for single-term reductions. On the official 239-problem LLM-SRBench split, PTB-Search reaches 73.2% Acc0.1 (Llama-3.1-8B) and 77.0% (single-seed DeepSeek-V4 anchor) versus 49.2% for the best reported baseline at one-tenth the standardized LLM-call budget. A scoped program-domain stress test finds generation count still unreliable while retained external state can help on harder instances.

Significance. If the matched-budget audit and selector-family separation hold, the paper supplies a load-bearing correction to a widely used narrative in LLM scientific discovery: parent-conditioned “evolution” is often resampling, and discovery is carried by external set-level selection over reusable components. The work is unusually careful for this literature—budget-matched paired arms, train-only selection with leakage audits, pre-registered interpretation branches, same-dictionary selector contrasts with extreme paired significance, collapse ablations for every architectural piece, and an explicit scope boundary via the program stress test and complexity guard. The official Acc0.1 gains at 1/10 budget, together with the two-family selector result (165–169 vs 74–78), are of clear interest to symbolic regression, LLM-for-science, and sparse dictionary methods. Strengths that should be credited include the leakage contract, frozen blind splits, and the falsifiable Principles 1–3.

major comments (3)
  1. [Table 2; Appendix D / Table 10] Appendix D / Table 10 (Llama 717-cell full benchmark): GenericPrimitiveSparse records more near and strict-solve cells (244 / 160) than Hybrid PTB-Search (239 / 135), while Hybrid wins only on median OOD NMSE (2.75e-2 vs 5.15e-2). The official headline in Table 2 reports Hybrid Acc0.1 = 73.2% but does not report the same Acc0.1 aggregation for a generic-primitives-only control under the identical frozen selector and protocol. Because the paper’s thesis is that “LLMs supply the material” while selection carries discovery, the official contribution of LLM-derived terms versus the selector-plus-generics stack is incompletely quantified on the metric used for the main claim. Please add generic-only (and, if feasible, LLM-only) Acc0.1 on the official 239-problem aggregation, or revise the material-supplier wording to match the Llama evidence that generics can dominate strict solves when the p
  2. [§5; Table 2; Abstract / Figure 1] §5 / Table 2 and Appendix C: The DeepSeek-V4 official result is a single-seed (seed-0) anchor of 23,900 proposals, while the Llama result is a three-seed 717-cell grid. The manuscript labels this carefully, but the abstract and teaser still juxtapose 77.0% and 73.2% as parallel headline numbers against the multi-seed baseline composite. For the central cross-backbone stability claim (“within four points”), either restrict the abstract/teaser to the multi-seed Llama figure and treat DeepSeek strictly as a corroborating anchor in the body, or add at least a second DeepSeek seed on a declared subset so the gap is not seed-0-dependent.
  3. [§4; §7; Figure 6; Principle 3] §4 hypothesis class and §7 complexity guard: The method is defined as sparse linear combinations with |S|≤6, and the nonlinear wrap/product-of-sums probe is excluded because it multiplies formula length (~8×) for only marginal near gains. That scope is honest, but the remaining-error autopsy (Fig. 6) attributes a non-zero composition gap on the Llama benchmark (5%) and a large availability gap (41%). The paper should state more explicitly in the main claims (not only Principle 3 / §8) that PTB-Search’s Acc0.1 gains are conditional on ground-truth structure being well approximated by a short additive support over extractable terms, and that LSR-Transform weakness is the designated failure regime under that assumption rather than a residual selector defect.
minor comments (5)
  1. [Table 2] Symbolic accuracy is correctly caveated, but Table 2 still leaves SA as ‘– †’ for PTB-Search while listing baseline SA percentages. A short footnote in the table itself (not only the caption) would prevent readers from treating the blank as missing data.
  2. [Figure 5; Appendix E / Table 13] Figure 5 (BPG10) is excellent for Principle 2; the main text could point more explicitly to Table 13’s train-tied / OOD-separated supports so readers can verify the one-term swap without hunting the appendix.
  3. [§8; Table 4; Appendix F] Program-domain hard grid excludes seeds 3–4 after API/billing errors (Appendix F). The main text already scopes the result; one sentence noting that the sign-tests are on a 3×3 clean grid would reduce the chance of over-reading Table 4.
  4. [Title; throughout] Typos / formatting: title line breaks (“NotDarwin”, “SET-LEVELSELECTION”) appear to be PDF extraction artifacts in places; ensure the camera-ready title spacing is clean. Also standardize “Acc 0.1” vs “Acc0.1” notation.
  5. [§9; Table 3] Related work cites concurrent IGSR (Saveliev et al., 2026) appropriately for the per-term vs set-level distinction; a one-sentence clarification that the single-term reductions in Table 3 are re-implementations on identical dictionaries (not a re-run of their full system) would avoid a fairness quibble.

Circularity Check

0 steps flagged

No significant circularity: claims are empirical comparisons against matched-sampling nulls and external LLM-SRBench baselines, not quantities defined by the method's own fit.

full rationale

The paper's load-bearing claims are experimental, not first-principles derivations. Parent-conditioned evolution is compared to fresh independent sampling under matched LLM-call budgets (Table 1; median OOD NMSE 0.045 vs 0.049); set-level vs single-term selectors are compared on identical frozen dictionaries with zero extra LLM calls (Table 3; 165–169 vs 74–78 solves of 717); PTB-Search is scored on the official 239-problem split against published baselines at 1/10 call budget (Table 2). Selection, dictionary construction, and the train-side guard use training data only; OOD and ground-truth formulas are never used for fit or ranking. The failure-mode autopsy oracles are explicitly post-hoc and not fed back into any method component. Sparse least-squares coefficients given a support are classical estimation, not a self-definitional prediction of the same quantity. Hyperparameters and the |S|≤6 cap were fixed on a development split before blind and official evaluation—standard protocol, not circular reduction. Citations (Brunton/SINDy, LLM-SR, FunSearch, etc.) are external prior art, not self-citation uniqueness theorems. Within the paper's own flat-component scope (Principle 3 / §8), the derivation chain does not reduce by construction to its inputs.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

The paper is empirical/methodological rather than axiomatic theory. Load-bearing structure comes from (i) experimental protocol choices treated as fixed (budget, support cap, proposal count), (ii) domain assumptions that expressions are additive and underdetermined training data identifies sets not terms, and (iii) the PTB-Search architecture as an invented pipeline. No new physical entities. Free parameters are hyperparameters fixed on the development split, not constants fitted to the official headline metric after the fact.

free parameters (4)
  • support_size_cap_|S|≤6
    Hard cap on number of selected terms in the sparse hypothesis class; justified by complexity-guard experiment but still a design choice that bounds scope and formula length.
  • n_proposals_per_problem=100
    One-shot LLM proposal budget used to build each dictionary; capacity sweep shows sensitivity but 100 is the main operating point and official budget.
  • beam_search_and_selector_hyperparameters
    Beam search, pairwise-synergy scoring, stability-generated supports, and train-side guard-fold tiebreak settings fixed on the DeepSeek development split before blind/official evaluation.
  • internal_70/30_train_split_and_guard_fold
    Train-only selection protocol parameters that define how supports are scored and ties broken without touching OOD.
axioms (5)
  • domain assumption Finite training samples make continuous equation structure underdetermined: many term sets can fit train NMSE while OOD separates them.
    Core identifiability premise of Principles 1–2 and the BPG10 figure; standard in symbolic regression but essential to why set-level selection is required.
  • domain assumption Useful scientific expressions in this benchmark can be represented as sparse linear combinations of extracted additive terms plus generic primitives.
    Defines the PTB-Search hypothesis class in §4; complexity guard excludes nonlinear composition that buys length without OOD gain.
  • standard math Best-subset / sparse dictionary regression with least-squares coefficients is a valid train-only estimator once a term dictionary is given (Brunton et al. 2016 lineage).
    Classical estimator family reused rather than re-derived; coefficients given support are ordinary least squares.
  • domain assumption LLM proposals supply problem-compatible reusable terms that can be parsed and canonicalized into a shared dictionary.
    Operational reduction of the evolution loop to a dictionary generator; extraction quality is assumed sufficient for the method to work.
  • ad hoc to paper Parent-conditioned prompt lineage is the right null target for ‘LLM evolution’ claims in this literature.
    Audit design choice: compares fresh sampling, parent-conditioned mutation, and instructed crossover under matched calls; other evolutionary operators (external archives) are treated separately in the program stress test.
invented entities (2)
  • PTB-Search pipeline independent evidence
    purpose: One-generation architecture: independent LLM proposals → term dictionary → global set-level sparse recombination with least squares.
    Named method that packages classical sparse selection with LLM proposal banks; not a physical entity but the paper’s primary constructed object.
  • set-level identifiability principle (Principle 2) independent evidence
    purpose: States that underdetermined data identifies joint term-set behavior, not per-term credit, justifying joint selectors over marginal/stability reductions.
    Conceptual principle supported by credit-AUC near chance and the two-family solve gap; falsifiable by showing a per-term statistic that matches set-level solves on identical dictionaries.

pith-pipeline@v1.1.0-grok45 · 27030 in / 3989 out tokens · 37385 ms · 2026-07-11T21:37:37.211830+00:00 · methodology

0 comments
read the original abstract

Large language models are increasingly used as evolutionary engines for scientific discovery: generate candidates, select winners, feed them back as parents, and repeat. We audit whether this loop actually compounds discovery in scientific equation discovery, a setting where finite samples make structure underdetermined and interpolation easy. Under matched LLM-call budgets, parent-conditioned evolution is indistinguishable from fresh independent sampling: median OOD NMSE is 0.045 vs. 0.049, instructed multi-parent crossover is worse, final success is predicted by initial proposal quality, and multiple iteration schemes fail to add solved problems. Operationally, the loop reduces to what it produces: a dictionary of candidate terms. We turn that diagnosis into PTB-Search, a one-generation method for componentized scientific discovery. PTB-Search samples independent LLM proposals once, extracts reusable terms into a per-problem dictionary, and performs train-only set-level sparse selection with least-squares coefficients. Its central principle is that underdetermined data identifies the joint behavior of term sets, not reliable per-term credit. On identical dictionaries and zero additional LLM calls, set-level selectors solve 165--169 of 717 cells, while single-term reductions solve only 74--78. On the official 239-problem LLM-SRBench split, PTB-Search reaches 73.2% Acc0.1 with Llama-3.1-8B and 77.0% with a single-seed DeepSeek-V4 anchor, versus 49.2% for the best reported baseline, using one tenth of the standardized call budget. A program-domain stress test gives a scoped boundary: generation count remains unreliable, while retained external state can help in harder non-linear spaces. Across these results, LLMs are best understood as material suppliers; discovery is carried by external set-level selection over reusable components.

Figures

Figures reproduced from arXiv: 2607.04108 by Pan Li.

Figure 1
Figure 1. Figure 1: Teaser. A matched-budget audit shows that parent-conditioned LLM “evolution” behaves like re￾sampling; PTB-Search keeps the useful part—proposal material—and replaces generations with one train-only, set-level recombination over a term dictionary, reaching 73.2% official Acc0.1 at one tenth of the standardized call budget. ABSTRACT Large language models (LLMs) are widely deployed as evolutionary engines fo… view at source ↗
Figure 2
Figure 2. Figure 2: Generations do not add solves. Guarded population-level recombination is flat across generations, while naive re-extraction destroys solves; the loop has no residual cumulative work after the dictionary has been built. problems. When reporting on the official split we follow Table-1 style exactly: per-problem mean over our three seeds, then median across problems. For method-development audits we additiona… view at source ↗
Figure 3
Figure 3. Figure 3: Selector family, not clever scoring, decides the result. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: More material does not rescue the wrong selector. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: One solved problem makes set-level identifiability visible. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: The remaining failures are mostly proposer-limited or selector-limited, not composition [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Program search reveals the boundary of the account. [PITH_FULL_IMAGE:figures/full_fig_p011_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Accuracy is not bought by unconstrained formula length. [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

13 extracted references · 7 linked inside Pith

  1. [1]

    Miles Cranmer

    doi: 10.1073/pnas.1517384113. Miles Cranmer. Interpretable machine learning for science with PySR and SymbolicRegression.jl. arXiv preprint arXiv:2305.01582,

    Pith/arXiv arXiv doi:10.1073/pnas.1517384113

  2. [2]

    Juan Cruz-Benito, Andrew W

    URLhttps://arxiv.org/abs/2305.01582. Juan Cruz-Benito, Andrew W. Cross, David Kremer, and Ismael Faro. Evolutionary discovery of bivariate bicycle codes with LLM-guided search.arXiv preprint arXiv:2606.02418,

    Pith/arXiv arXiv

  3. [3]

    Jianqing Fan and Jinchi Lv

    URL https://arxiv.org/abs/2606.02418. Jianqing Fan and Jinchi Lv. Sure independence screening for ultrahigh dimensional feature space. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 70(5):849–911,

    Pith/arXiv arXiv

  4. [4]

    Arya Grayeli, Atharva Sehgal, Omar Costilla-Reyes, Miles Cranmer, and Swarat Chaudhuri

    doi: 10.1111/j.1467-9868.2008.00674.x. Arya Grayeli, Atharva Sehgal, Omar Costilla-Reyes, Miles Cranmer, and Swarat Chaudhuri. Symbolic regression with a learned concept library. InAd- vances in Neural Information Processing Systems, volume 37,

    doi:10.1111/j.1467-9868.2008.00674.x 2008

  5. [5]

    Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger

    URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/ 4ec3ddc465c6d650c9c419fb91f1c00a-Abstract-Conference.html. Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters.Proceedings of the AAAI Conference on Artificial Intelligence, 32(1),

    2024

  6. [6]

    Mikel Landajuela, Chak Shing Lee, Jiachen Yang, Ruben Glatt, Claudio P

    doi: 10.1609/aaai.v32i1.11694. Mikel Landajuela, Chak Shing Lee, Jiachen Yang, Ruben Glatt, Claudio P. Santi- ago, Ignacio Aravena, Terrell N. Mundhenk, Garrett Mulcahy, and Brenden K. Pe- tersen. A unified framework for deep symbolic regression. InAdvances in Neu- ral Information Processing Systems, volume 35, pages 33985–33998,

    doi:10.1609/aaai.v32i1.11694

  7. [8]

    Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell

    URLhttps: //arxiv.org/abs/2509.19349. Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. InInternational Conference on Learning Representations,

    Pith/arXiv arXiv

  8. [9]

    To appear

    URLhttps://arxiv.org/abs/2605.29184. To appear. Michael Schmidt and Hod Lipson. Distilling free-form natural laws from experimental data.Science, 324(5923):81–85,

    Pith/arXiv arXiv

  9. [10]

    Parshin Shojaee, Kazem Meidani, Shashank Gupta, Amir Barati Farimani, and Chandan K

    doi: 10.1126/science.1165893. Parshin Shojaee, Kazem Meidani, Shashank Gupta, Amir Barati Farimani, and Chandan K. Reddy. LLM-SR: Scientific equation discovery via programming with large language mod- els. InThe Thirteenth International Conference on Learning Representations, 2025a. URL https://openreview.net/forum?id=m2nmp8P5in. Parshin Shojaee, Ngoc-Hie...

    doi:10.1126/science.1165893

  10. [11]

    Haorui Wang, Parshin Shojaee, Kazem Meidani, Kunyang Sun, Jos ´e Miguel Hern ´andez-Lobato, Teresa Head-Gordon, Jiajun He, Chandan K

    doi: 10.1126/sciadv.aay2631. Haorui Wang, Parshin Shojaee, Kazem Meidani, Kunyang Sun, Jos ´e Miguel Hern ´andez-Lobato, Teresa Head-Gordon, Jiajun He, Chandan K. Reddy, Chao Zhang, and Yuanqi Du. Towards diverse scientific hypothesis search with large language models. InInternational Conference on Machine Learning,

    doi:10.1126/sciadv.aay2631

  11. [12]

    To appear

    URLhttps://arxiv.org/abs/2606.10587. To appear. Masakazu Yoshimura, Zitang Sun, Yuiko Sakuma, Junji Otsuka, Atsushi Irie, and Takeshi Ohashi. LLM as a tool, not an agent: Code-mined tree transformations for neural architecture search. arXiv preprint arXiv:2604.16555,

    Pith/arXiv arXiv

  12. [13]

    Peng Zhao and Bin Yu

    URLhttps://arxiv.org/abs/2604.16555. Peng Zhao and Bin Yu. On model selection consistency of Lasso.Journal of Machine Learning Re- search, 7(90):2541–2563,

    Pith/arXiv arXiv

  13. [14]

    PySR rows are local CPU references under our protocol, not official LLM- SRBench rows

    B ADDITIONAL ABLATIONS AND LOCAL REFERENCES Table 5 records the rows most relevant for interpreting PTB-Search against non-LLM or weakly- LLM references. PySR rows are local CPU references under our protocol, not official LLM- SRBench rows. Table 5:Additional local references. PySR and dense-dictionary rows are protocol context for failure-mode interpreta...

    arXiv