Pith. sign in

REVIEW 3 major objections 6 minor 10 references

Fast Symbolic Regression Benchmarking

T0 review · 3 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Pith's one-line read A benchmark that accepts functionally equivalent expressions and stops on discovery raises SymbolicRegression.jl's measured rediscovery rate from 26.7% to 44.7% and cuts compute by 41.2%.

desk verdict The curated-acceptance-list and early-termination idea is real and useful, but the headline 26.7%→44.7% gain is confounded by per-problem complexity caps and retroactive list growth; worth revising, not rejecting. read the letter →

arxiv 2508.14481 v1 pith:LEZ2WNYQ submitted 2025-08-20 cs.LG

classification cs.LG
keywords symbolicregressionbenchmarkingrediscoveryrateequivalentexpressionsearlyterminationSRSDbenchmarkhyperparametertuninggeneticprogramming
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Ground-truth benchmarks for symbolic regression typically demand that a search recover one canonical form of a known formula, and they keep spending computation after the formula has been found. This paper claims both habits distort results and waste resources, and proposes two fixes: judge success against a curated list of functionally equivalent, concise forms, and terminate the run roughly 15 seconds after an acceptable form appears. On the 120 SRSD benchmark problems, the method raises SymbolicRegression.jl's measured rediscovery rate from the 26.7% reported for the original benchmark to 44.7%, at 41.2% lower computation, and TiSR reaches 69.4% while saving 63% of time. If the method is right, symbolic-regression tools can be compared on whether they find a usable, concise answer, and the search can stop the moment one is found.

What carries the argument

Two mechanisms carry the argument. First, the curated acceptable-expression list: for each benchmark problem, a growing set of functionally equivalent forms, produced by simplifying candidates with the SymPy computer algebra system, rounding constants to five significant digits, and requiring no more than 20% more operators and operands than the reference expression. Second, the early-termination callback: roughly every 15 seconds, hall-of-fame candidates (the best non-dominated expressions found so far) that clear a test-data error threshold (relative error below 0.000001%) are simplified, rounded, and compared with the list; a match records the rediscovery and stops the run. Recorded near-

What would settle it

Re-run the 120-problem benchmark with a single fixed complexity cap for all problems and no answer-derived configuration, and check whether SymbolicRegression.jl's rediscovery rate stays near 44.7% or falls back toward the 26.7% baseline. Separately, take a sample of early-terminated runs, continue them to the full 30-minute budget, and verify that an acceptable expression still sits in the hall of fame at the end.

Watch

Extended reading notes

Core claim

Rediscovery-based symbolic-regression benchmarks are too strict and too wasteful: they accept one expression form, or rely on a computer algebra system that fails on many equivalent forms, and they keep searching after discovery. The remedy has two parts: curated lists of acceptable expressions — equivalent forms rounded to five significant digits with at most 20% more operators than the reference — and a callback that checks the hall of fame every 15 seconds and halts the run on a match. On the SRSD problems, this lifts SymbolicRegression.jl's rediscovery from 26.7% to 44.7% at 41.2% lower cost; TiSR scores 69.4% at 63% time savings. This mirrors real use: a researcher can rearrange any con

Load-bearing premise

The load-bearing premise is that capping each problem's maximum expression complexity at 50% above the ground-truth answer's complexity — information a real benchmark user would not have — does not inflate the reported rediscovery gains; the paper itself flags this as a known bias in Section 4.1.

Editorial extensions

If this is right

  • Symbolic-regression tools can be compared on genuine rediscovery — any concise, functionally equivalent form counts — instead of on spelling one canonical expression, so measured capability gaps become clearer.
  • Benchmarking compute drops by 41.2% for SymbolicRegression.jl and 63% for TiSR on the reported runs, making extensive hyperparameter tuning of genetic-programming-based tools affordable.
  • The curated lists grow with each benchmark use, so repeated runs get cheaper and cover more equivalent forms without changing the acceptance rules.
  • On these problems the two tools separate clearly (TiSR 69.4%, SymbolicRegression.jl 44.7%), giving benchmark users a sharper signal for tool selection.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The self-acknowledged bias in Section 4.1 is the main confounder: maximum expression complexity is capped at 50% above each ground-truth answer's complexity, information a real benchmark user would not have. A control run with one fixed cap for all problems would show how much of the 26.7% to 44.7% gain survives without answer-derived configuration.
  • Because a match terminates the run, early stopping interacts with the randomness of genetic search: a lucky early find truncates the run, so per-run variance may shift. Repeating runs with and without termination, or continuing terminated runs to the full budget, would quantify any distortion.
  • The equivalence check inherits SymPy's simplification limits; expression families that SymPy cannot canonicalize will remain under-counted. Pushing the protocol onto such families, or adding stronger equivalence checking, is a natural stress test.
  • The protocol generalizes beyond the 120 SRSD problems: any benchmark with known ground-truth expressions could carry curated lists and a live callback, turning manual post-hoc inspection of discovered forms into a reusable, growing asset.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes a new ground-truth symbolic-regression benchmarking protocol: curated lists of acceptable equivalent expressions replace a single ground-truth form, and a live callback terminates each search as soon as a candidate matches an acceptable form. The method is evaluated on the 120-problem SRSD benchmark with SymbolicRegression.jl and TiSR. The paper reports that SymbolicRegression.jl's rediscovery rate rises from 26.7% (as reported by Yoshitomo et al.) to 44.7% under the new protocol, with 41.2% computational savings, while TiSR achieves 69.4% rediscovery with 63% savings. The headline claim is that curated acceptable-expression lists plus early termination both measure rediscovery more realistically and reduce benchmark cost.

Significance. If the reported gains are real, the method is a useful practical contribution to SR benchmarking: early termination via a callback is sensible, the curated-list idea addresses a known limitation of single-form or SymPy-only equivalence checks, and the choice of pinned commits, public code/data links, and a documented source diff supports reproducibility. The strict test-fit threshold (relative error below 1e-8 on 200 held-out points) makes false positive rediscoveries unlikely, and the time-savings figures are plausible. However, the significance of the headline comparison is not currently established: the protocol changes the success criterion and the search configuration simultaneously, and the acceptance list itself is extended using the outputs of the runs being scored. These are load-bearing confounds that require additional experiments.

major comments (3)
  1. [§4.1, Table 4.1; abstract] The headline improvement (26.7% vs. 44.7%) is confounded. The new runs use a per-problem maximum complexity cap set to '50% higher than the reference' (§4.1), whereas the compared baseline [5] used a fixed maximal complexity. This cap reveals the ground-truth expression's complexity to the search configuration, prunes the search space, and can by itself raise rediscovery rates, independent of the proposed callback/list method. The paper's caveat that this 'introduces some bias' is not quantified. An ablation using the baseline's fixed complexity cap, or a report of rediscovery under both settings, is required before the gain can be attributed to the proposed benchmarking method.
  2. [§3 (retroactive acceptance)] The acceptance lists are extended from the outputs of the runs that the benchmark scores: 'Those are retroactively accepted, and added to the list of acceptable expressions for future runs.' If retroactively accepted forms contribute to the reported rediscovery rates, the success criterion is partly learned from the measured runs, making the comparison circular. If they do not contribute to the reported rates, that should be stated explicitly. Moreover, because different tools produce different near-miss expressions, per-tool acceptance lists can diverge over time, undermining cross-tool comparability. Please report the number of accepted forms added retroactively and repeat scoring with the initial curated lists only as a robustness check.
  3. [§4.1, §2 (baseline comparability)] The comparison with Yoshitomo et al. assumes that the 120-problem set, the data sampling, the function set, and the number of runs are identical to those used in [5]. Three problems (B4, B11, III.9.52) are omitted from the current statistics due to 'repeated SymPy issues.' If the baseline's 26.7% was computed on a different problem set, the comparison may be affected. Please state whether the omitted problems are excluded from the baseline statistics as well, and clarify whether any of the other setup choices (e.g., 200 training/test points per run, no multiprocessing, 15-second callback interval) differ from [5].
minor comments (6)
  1. [Abstract; §4.1] Typographical and wording issues: 'Yoshitomo et at.' in the abstract should be 'et al.'; 'This is in contrast allowing' should read 'in contrast to allowing'; 'TiSRsupports' should have a space. Also, 'below 0.000001%' is clearer as 'relative error below 1e-8' to match the formula.
  2. [§4.1] The function set and 'some operator nesting is prohibited' are described only verbally. For reproducibility, the exact allowed operator set and nesting restrictions should be listed explicitly, and the TiSR commit should be accompanied by a note on whether the used version includes any local changes.
  3. [Table 4.1] The easy/medium/hard categories are referenced as in [5] but their defining criteria are not given. Define or cite the category thresholds so the binning can be reproduced.
  4. [Figure 4.1] The figure is described in the text but does not appear in the manuscript version I reviewed. Ensure the final version includes the figure with legible per-problem markers and clearly labeled axes.
  5. [Algorithm 1 and §4.1] The threshold in Algorithm 1 is tested as δ(e) ≥ δmax, so 'below a set threshold' is consistent with δ(e) < δmax. The same threshold is called δmax everywhere; this is fine, but please keep the notation consistent between the algorithm and the text.
  6. [§4.2] The text says 'no fallback method is employed, i.e., only rediscovery of the expressions.' Since the acceptable-expression list includes functional equivalents, this sentence is slightly misleading; 'only rediscovery, as defined by the acceptable-expression lists' would be more precise.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: measured claims rest on external ground truth; noted confounds are validity risks, not self-referential derivations.

full rationale

The paper's central claims are empirical benchmark measurements rather than derivations from a model, so most circularity patterns do not apply. The ground-truth expressions and the baseline 26.7% rediscovery rate come from the external SRSD benchmark of Yoshitomo et al. [5], and the proposed method's success is assessed by algebraic equivalence (via SymPy) to those external ground-truth expressions. Two methodological concerns appear in the text. First, Section 3 states that expressions recorded during a run but not matching the current acceptable list are 'retroactively accepted' and added to the list for future runs; this makes the acceptance criterion partially learned from the runs it scores, which could inflate measured rediscovery rates relative to a fixed, pre-registered oracle. Second, Section 4.1 states that 'For each of the benchmark problems, we adapt the maximum allowed expression complexity to 50% higher than the reference,' and the author concedes 'this greatly increases efficiency at the cost of introducing some bias.' That per-problem complexity cap leaks information from the known reference expression into the search configuration, confounding the headline comparison of 26.7% vs. 44.7% because the compared baseline [5] used a fixed maximal complexity. These are genuine threats to the validity of the comparison, but they are not circularity in the required sense: no parameter is fitted and then renamed as a prediction, no equation reduces to its own inputs by construction, and no load-bearing claim depends on a self-citation. The TiSR results are the author's own tool but are normal empirical measurements rather than imported uniqueness or ansatz claims. Therefore no circular step can be exhibited, and the appropriate score is low.

Assumptions & free parameters 7 free parameters · 4 assumptions · 1 invented entities

The central claim rests on hand-chosen thresholds (error, rounding, complexity bounds, callback interval), on a per-problem complexity cap that uses ground-truth knowledge of the target, and on the assumption that SymPy normalization plus a very tight error threshold faithfully recognizes functional equivalence. No physically invented entities; the methodological artifact that carries the most weight is the acceptance list itself, which is not an independent oracle because it is extended from the scored runs.

free parameters (7)
  • per-problem complexity cap for the search = 1.5x the ground-truth reference complexity (50% higher)
    Section 4.1: the cap is derived from each target expression's known complexity; the author states it 'greatly increases efficiency at the cost of introducing some bias.'
  • acceptable-form complexity bound = 1.2x the reference complexity (20% higher)
    Section 3: operator/operand budget for accepting an equivalent form; hand-chosen.
  • fit-quality threshold for callback processing = relative error below 0.000001% (1e-8), with offset 1e-100
    Section 3 and Algorithm 1: gates which candidates are simplified and matched; hand-chosen.
  • constant rounding precision = 5 significant digits
    Section 3: rounding before list matching; tuned to the motivating examples such as I.34.1 that SymPy cannot reconcile.
  • callback polling interval = 15 seconds
    Section 3 and Appendix A (globally gated by time()).
  • train/test sample counts per run = 200 points each
    Section 4.1: per-run resampling; affects the reliability of the 1e-8 error verdict.
  • runs per problem and time budget = 5 runs x 30 min
    Section 4.1: fixed budget; rates reported without variance across runs.
assumptions (4)
  • domain assumption The SRSD ground-truth expressions and sampling ranges are correct (inherited from [5])
    Section 4.1 takes 'the 120 benchmark problems defined in the SRSD benchmark' and samples 'according to the specifications of [5]'; all reported rates inherit SRSD's correctness.
  • domain assumption SymPy simplification plus rounding to five significant digits preserves the intended equivalence classes
    Section 3: candidates are simplified and rounded before list matching; the paper both criticizes SymPy equivalence for rounded forms and relies on SymPy normalization to render candidates comparable, without validating against an independent equivalence oracle.
  • domain assumption A relative error below 1e-8 on 200 test points certifies functional equivalence
    Section 3 and Algorithm 1: this threshold converts a numerical fit into a rediscovery verdict; no false-acceptance probability is analyzed, though the threshold is very strict for noise-free data.
  • domain assumption The hall-of-fame dominance premise: any superior solution that replaces a discovered acceptable form is also acceptable
    Section 3: 'In the absence of noise, and if the hall of fame's selection criteria are perfectly aligned with a solution's utility, any such superior solution should also be deemed acceptable.' This justifies cutting the search short.
invented entities (1)
  • curated acceptable-expression lists (FastSRB)
    purpose: The scoring oracle for each SRSD problem: a candidate expression counts as a rediscovery only if it matches the list after simplification and rounding.
    The lists are the acceptance criterion for the measured rediscovery rates, yet they are authored by the study's author and extended retroactively from expressions found during the scored runs (Section 3), so they are not an independent, fixed ground truth despite being externally inspectable in the repository.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Fast Symbolic Regression Benchmarking." pith.science (2026). https://pith.science/paper/LEZ2WNYQ

@misc{pith2026250814481,
  author       = {Pith},
  title        = {Pith review of: Fast Symbolic Regression Benchmarking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LEZ2WNYQ}},
  note         = {Machine review of arXiv:2508.14481}
}
read the original abstract

Symbolic regression (SR) uncovers mathematical models from data. Several benchmarks have been proposed to compare the performance of SR algorithms. However, existing ground-truth rediscovery benchmarks overemphasize the recovery of "the one" expression form or rely solely on computer algebra systems (such as SymPy) to assess success. Furthermore, existing benchmarks continue the expression search even after its discovery. We improve upon these issues by introducing curated lists of acceptable expressions, and a callback mechanism for early termination. As a starting point, we use the symbolic regression for scientific discovery (SRSD) benchmark problems proposed by Yoshitomo et al., and benchmark the two SR packages SymbolicRegression.jl and TiSR. The new benchmarking method increases the rediscovery rate of SymbolicRegression.jl from 26.7%, as reported by Yoshitomo et at., to 44.7%. Performing the benchmark takes 41.2% less computational expense. TiSR's rediscovery rate is 69.4%, while performing the benchmark saves 63% time.

Figures

Figures reproduced from arXiv: 2508.14481 by the authors.

Figure 4
Figure 4. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 4.1
Figure 4.1. The rediscovery rates of SymbolicRegression.jl (PySR in the figure legend) and TiSR using the proposed bench￾marking method (FastSRB) for the benchmark problems defined by [5]. The results are shown in three figures, gathering problems according to the “easy”, “medium”, and “hard” categories proposed by [5]. Benchmark problems B4, B11, and III.9.52 are omitted from all statistics, due to repeated SymPy issues during… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

10 extracted references · 6 canonical work pages

  1. [5]

    Rethinking Symbolic Re- gression Datasets and Benchmarks for Scientific Discovery.Journal of Data- centric Machine Learning Research.https :/ /openreview

    Matsubara, Y.; Chiba, N.; Igarashi, R.; Ushiku, Y. Rethinking Symbolic Re- gression Datasets and Benchmarks for Scientific Discovery.Journal of Data- centric Machine Learning Research.https :/ /openreview . net/forum ? id = qrUdrXsiXX (2024)

  2. [1]

    Interpretable machine learning for science with PySR and Sym- bolicRegression.jlarXiv: 2305.01582

    Cranmer, M. Interpretable machine learning for science with PySR and Sym- bolicRegression.jlarXiv: 2305.01582

  3. [2]

    H.Contemporary symbolic regression methods and their relative performancearXiv: 2107.14351

    La Cava, W.; Orzechowski, P.; Burlacu, B.; Olivetti de França, F.; Virgolin, M.; Jin, Y.; Kommenda, M.; Moore, J. H.Contemporary symbolic regression methods and their relative performancearXiv: 2107.14351

  4. [3]

    Introducing Thermodynamics-Informed Symbolic Regression -- A Tool for Thermodynamic Equations of State Development

    Martinek,V.;Frotscher,O.;Richter,M.;Herzog,R. Introducing thermodynamics- informed symbolic regression – a tool for thermodynamic equations of state development arXiv: 2309.02805

  5. [4]

    Thermodynamics-informed Symbolic Regression - A Tool for the Thermodynamic Equation of State De- velopment 2025

    Martinek,V.;Frotscher,O.;Richter,M.;Herzog,R. Thermodynamics-informed Symbolic Regression - A Tool for the Thermodynamic Equation of State De- velopment 2025. doi:10.5281/zenodo.8317546

  6. [6]

    P.; Paprocki, M.; Čertík, O.; Kirpichev, S

    Meurer, A.; Smith, C. P.; Paprocki, M.; Čertík, O.; Kirpichev, S. B.; Rocklin, M.; Kumar, A.; Ivanov, S.; Moore, J. K.; Singh, S.; Rathnayake, T.; Vig, S.; Granger, B. E.; Muller, R. P.; Bonazzi, F.; Gupta, H.; Vats, S.; Johansson, F.; Pedregosa, F.; Curry, M. J.; Terrel, A. R.; Roučka, Š.; Saboo, A.; Fernando, I.; Kulal, S.; Cimrman, R.; Scopatz, A. SymP...

  7. [7]

    GNU Parallel 2018 doi:10

    Tange, O. GNU Parallel 2018 doi:10 . 5281/zenodo . 1146014 (Ole Tange, 2018)

  8. [8]

    AI Feynman: A physics-inspired method for symbolic regression

    Udrescu, S.-M.; Tegmark, M. AI Feynman: A physics-inspired method for symbolic regression. Science Advances 6, eaay2631. doi:10 . 1126/sciadv . aay2631 (2020)

Show all 10 references
  1. [9]

    Van Rossum, G.; Drake Jr, F. L. Python reference manual (Centrum voor Wiskunde en Informatica Amsterdam, 1995)

  2. [10]

    P.Symbolic regression is NP-hardarXiv: 2207.01018

    Virgolin, M.; Pissis, S. P.Symbolic regression is NP-hardarXiv: 2207.01018. (V. Martinek) Interdisciplinary Center for Scientific Computing, Heidelberg Uni- versity, 69120 Heidelberg, Germany Email address: viktor.martinek@iwr.uni-heidelberg.de

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.