REVIEW 3 major objections 6 minor 10 references
Fast Symbolic Regression Benchmarking
T0 review · 3 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read A benchmark that accepts functionally equivalent expressions and stops on discovery raises SymbolicRegression.jl's measured rediscovery rate from 26.7% to 44.7% and cuts compute by 41.2%.
desk verdict The curated-acceptance-list and early-termination idea is real and useful, but the headline 26.7%→44.7% gain is confounded by per-problem complexity caps and retroactive list growth; worth revising, not rejecting. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Two mechanisms carry the argument. First, the curated acceptable-expression list: for each benchmark problem, a growing set of functionally equivalent forms, produced by simplifying candidates with the SymPy computer algebra system, rounding constants to five significant digits, and requiring no more than 20% more operators and operands than the reference expression. Second, the early-termination callback: roughly every 15 seconds, hall-of-fame candidates (the best non-dominated expressions found so far) that clear a test-data error threshold (relative error below 0.000001%) are simplified, rounded, and compared with the list; a match records the rediscovery and stops the run. Recorded near-
What would settle it
Re-run the 120-problem benchmark with a single fixed complexity cap for all problems and no answer-derived configuration, and check whether SymbolicRegression.jl's rediscovery rate stays near 44.7% or falls back toward the 26.7% baseline. Separately, take a sample of early-terminated runs, continue them to the full 30-minute budget, and verify that an acceptable expression still sits in the hall of fame at the end.
Extended reading notes
Core claim
Rediscovery-based symbolic-regression benchmarks are too strict and too wasteful: they accept one expression form, or rely on a computer algebra system that fails on many equivalent forms, and they keep searching after discovery. The remedy has two parts: curated lists of acceptable expressions — equivalent forms rounded to five significant digits with at most 20% more operators than the reference — and a callback that checks the hall of fame every 15 seconds and halts the run on a match. On the SRSD problems, this lifts SymbolicRegression.jl's rediscovery from 26.7% to 44.7% at 41.2% lower cost; TiSR scores 69.4% at 63% time savings. This mirrors real use: a researcher can rearrange any con
Load-bearing premise
The load-bearing premise is that capping each problem's maximum expression complexity at 50% above the ground-truth answer's complexity — information a real benchmark user would not have — does not inflate the reported rediscovery gains; the paper itself flags this as a known bias in Section 4.1.
Editorial extensions
If this is right
- Symbolic-regression tools can be compared on genuine rediscovery — any concise, functionally equivalent form counts — instead of on spelling one canonical expression, so measured capability gaps become clearer.
- Benchmarking compute drops by 41.2% for SymbolicRegression.jl and 63% for TiSR on the reported runs, making extensive hyperparameter tuning of genetic-programming-based tools affordable.
- The curated lists grow with each benchmark use, so repeated runs get cheaper and cover more equivalent forms without changing the acceptance rules.
- On these problems the two tools separate clearly (TiSR 69.4%, SymbolicRegression.jl 44.7%), giving benchmark users a sharper signal for tool selection.
Reading between the lines
- The self-acknowledged bias in Section 4.1 is the main confounder: maximum expression complexity is capped at 50% above each ground-truth answer's complexity, information a real benchmark user would not have. A control run with one fixed cap for all problems would show how much of the 26.7% to 44.7% gain survives without answer-derived configuration.
- Because a match terminates the run, early stopping interacts with the randomness of genetic search: a lucky early find truncates the run, so per-run variance may shift. Repeating runs with and without termination, or continuing terminated runs to the full budget, would quantify any distortion.
- The equivalence check inherits SymPy's simplification limits; expression families that SymPy cannot canonicalize will remain under-counted. Pushing the protocol onto such families, or adding stronger equivalence checking, is a natural stress test.
- The protocol generalizes beyond the 120 SRSD problems: any benchmark with known ground-truth expressions could carry curated lists and a live callback, turning manual post-hoc inspection of discovered forms into a reusable, growing asset.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a new ground-truth symbolic-regression benchmarking protocol: curated lists of acceptable equivalent expressions replace a single ground-truth form, and a live callback terminates each search as soon as a candidate matches an acceptable form. The method is evaluated on the 120-problem SRSD benchmark with SymbolicRegression.jl and TiSR. The paper reports that SymbolicRegression.jl's rediscovery rate rises from 26.7% (as reported by Yoshitomo et al.) to 44.7% under the new protocol, with 41.2% computational savings, while TiSR achieves 69.4% rediscovery with 63% savings. The headline claim is that curated acceptable-expression lists plus early termination both measure rediscovery more realistically and reduce benchmark cost.
Significance. If the reported gains are real, the method is a useful practical contribution to SR benchmarking: early termination via a callback is sensible, the curated-list idea addresses a known limitation of single-form or SymPy-only equivalence checks, and the choice of pinned commits, public code/data links, and a documented source diff supports reproducibility. The strict test-fit threshold (relative error below 1e-8 on 200 held-out points) makes false positive rediscoveries unlikely, and the time-savings figures are plausible. However, the significance of the headline comparison is not currently established: the protocol changes the success criterion and the search configuration simultaneously, and the acceptance list itself is extended using the outputs of the runs being scored. These are load-bearing confounds that require additional experiments.
major comments (3)
- [§4.1, Table 4.1; abstract] The headline improvement (26.7% vs. 44.7%) is confounded. The new runs use a per-problem maximum complexity cap set to '50% higher than the reference' (§4.1), whereas the compared baseline [5] used a fixed maximal complexity. This cap reveals the ground-truth expression's complexity to the search configuration, prunes the search space, and can by itself raise rediscovery rates, independent of the proposed callback/list method. The paper's caveat that this 'introduces some bias' is not quantified. An ablation using the baseline's fixed complexity cap, or a report of rediscovery under both settings, is required before the gain can be attributed to the proposed benchmarking method.
- [§3 (retroactive acceptance)] The acceptance lists are extended from the outputs of the runs that the benchmark scores: 'Those are retroactively accepted, and added to the list of acceptable expressions for future runs.' If retroactively accepted forms contribute to the reported rediscovery rates, the success criterion is partly learned from the measured runs, making the comparison circular. If they do not contribute to the reported rates, that should be stated explicitly. Moreover, because different tools produce different near-miss expressions, per-tool acceptance lists can diverge over time, undermining cross-tool comparability. Please report the number of accepted forms added retroactively and repeat scoring with the initial curated lists only as a robustness check.
- [§4.1, §2 (baseline comparability)] The comparison with Yoshitomo et al. assumes that the 120-problem set, the data sampling, the function set, and the number of runs are identical to those used in [5]. Three problems (B4, B11, III.9.52) are omitted from the current statistics due to 'repeated SymPy issues.' If the baseline's 26.7% was computed on a different problem set, the comparison may be affected. Please state whether the omitted problems are excluded from the baseline statistics as well, and clarify whether any of the other setup choices (e.g., 200 training/test points per run, no multiprocessing, 15-second callback interval) differ from [5].
minor comments (6)
- [Abstract; §4.1] Typographical and wording issues: 'Yoshitomo et at.' in the abstract should be 'et al.'; 'This is in contrast allowing' should read 'in contrast to allowing'; 'TiSRsupports' should have a space. Also, 'below 0.000001%' is clearer as 'relative error below 1e-8' to match the formula.
- [§4.1] The function set and 'some operator nesting is prohibited' are described only verbally. For reproducibility, the exact allowed operator set and nesting restrictions should be listed explicitly, and the TiSR commit should be accompanied by a note on whether the used version includes any local changes.
- [Table 4.1] The easy/medium/hard categories are referenced as in [5] but their defining criteria are not given. Define or cite the category thresholds so the binning can be reproduced.
- [Figure 4.1] The figure is described in the text but does not appear in the manuscript version I reviewed. Ensure the final version includes the figure with legible per-problem markers and clearly labeled axes.
- [Algorithm 1 and §4.1] The threshold in Algorithm 1 is tested as δ(e) ≥ δmax, so 'below a set threshold' is consistent with δ(e) < δmax. The same threshold is called δmax everywhere; this is fine, but please keep the notation consistent between the algorithm and the text.
- [§4.2] The text says 'no fallback method is employed, i.e., only rediscovery of the expressions.' Since the acceptable-expression list includes functional equivalents, this sentence is slightly misleading; 'only rediscovery, as defined by the acceptable-expression lists' would be more precise.
Circularity Check
No significant circularity: measured claims rest on external ground truth; noted confounds are validity risks, not self-referential derivations.
full rationale
The paper's central claims are empirical benchmark measurements rather than derivations from a model, so most circularity patterns do not apply. The ground-truth expressions and the baseline 26.7% rediscovery rate come from the external SRSD benchmark of Yoshitomo et al. [5], and the proposed method's success is assessed by algebraic equivalence (via SymPy) to those external ground-truth expressions. Two methodological concerns appear in the text. First, Section 3 states that expressions recorded during a run but not matching the current acceptable list are 'retroactively accepted' and added to the list for future runs; this makes the acceptance criterion partially learned from the runs it scores, which could inflate measured rediscovery rates relative to a fixed, pre-registered oracle. Second, Section 4.1 states that 'For each of the benchmark problems, we adapt the maximum allowed expression complexity to 50% higher than the reference,' and the author concedes 'this greatly increases efficiency at the cost of introducing some bias.' That per-problem complexity cap leaks information from the known reference expression into the search configuration, confounding the headline comparison of 26.7% vs. 44.7% because the compared baseline [5] used a fixed maximal complexity. These are genuine threats to the validity of the comparison, but they are not circularity in the required sense: no parameter is fitted and then renamed as a prediction, no equation reduces to its own inputs by construction, and no load-bearing claim depends on a self-citation. The TiSR results are the author's own tool but are normal empirical measurements rather than imported uniqueness or ansatz claims. Therefore no circular step can be exhibited, and the appropriate score is low.
Assumptions & free parameters
free parameters (7)
- per-problem complexity cap for the search =
1.5x the ground-truth reference complexity (50% higher)
- acceptable-form complexity bound =
1.2x the reference complexity (20% higher)
- fit-quality threshold for callback processing =
relative error below 0.000001% (1e-8), with offset 1e-100
- constant rounding precision =
5 significant digits
- callback polling interval =
15 seconds
- train/test sample counts per run =
200 points each
- runs per problem and time budget =
5 runs x 30 min
assumptions (4)
- domain assumption The SRSD ground-truth expressions and sampling ranges are correct (inherited from [5])
- domain assumption SymPy simplification plus rounding to five significant digits preserves the intended equivalence classes
- domain assumption A relative error below 1e-8 on 200 test points certifies functional equivalence
- domain assumption The hall-of-fame dominance premise: any superior solution that replaces a discovered acceptable form is also acceptable
invented entities (1)
-
curated acceptable-expression lists (FastSRB)
Cite this review
Pith. "Pith review of Fast Symbolic Regression Benchmarking." pith.science (2026). https://pith.science/paper/LEZ2WNYQ
@misc{pith2026250814481,
author = {Pith},
title = {Pith review of: Fast Symbolic Regression Benchmarking},
year = {2026},
howpublished = {\url{https://pith.science/paper/LEZ2WNYQ}},
note = {Machine review of arXiv:2508.14481}
}
read the original abstract
Symbolic regression (SR) uncovers mathematical models from data. Several benchmarks have been proposed to compare the performance of SR algorithms. However, existing ground-truth rediscovery benchmarks overemphasize the recovery of "the one" expression form or rely solely on computer algebra systems (such as SymPy) to assess success. Furthermore, existing benchmarks continue the expression search even after its discovery. We improve upon these issues by introducing curated lists of acceptable expressions, and a callback mechanism for early termination. As a starting point, we use the symbolic regression for scientific discovery (SRSD) benchmark problems proposed by Yoshitomo et al., and benchmark the two SR packages SymbolicRegression.jl and TiSR. The new benchmarking method increases the rediscovery rate of SymbolicRegression.jl from 26.7%, as reported by Yoshitomo et at., to 44.7%. Performing the benchmark takes 41.2% less computational expense. TiSR's rediscovery rate is 69.4%, while performing the benchmark saves 63% time.
Figures
Reference graph
Works this paper leans on
-
[5]
Matsubara, Y.; Chiba, N.; Igarashi, R.; Ushiku, Y. Rethinking Symbolic Re- gression Datasets and Benchmarks for Scientific Discovery.Journal of Data- centric Machine Learning Research.https :/ /openreview . net/forum ? id = qrUdrXsiXX (2024)
work page 2024
-
[1]
Interpretable machine learning for science with PySR and Sym- bolicRegression.jlarXiv: 2305.01582
Cranmer, M. Interpretable machine learning for science with PySR and Sym- bolicRegression.jlarXiv: 2305.01582
-
[2]
H.Contemporary symbolic regression methods and their relative performancearXiv: 2107.14351
La Cava, W.; Orzechowski, P.; Burlacu, B.; Olivetti de França, F.; Virgolin, M.; Jin, Y.; Kommenda, M.; Moore, J. H.Contemporary symbolic regression methods and their relative performancearXiv: 2107.14351
-
[3]
Martinek,V.;Frotscher,O.;Richter,M.;Herzog,R. Introducing thermodynamics- informed symbolic regression – a tool for thermodynamic equations of state development arXiv: 2309.02805
-
[4]
Martinek,V.;Frotscher,O.;Richter,M.;Herzog,R. Thermodynamics-informed Symbolic Regression - A Tool for the Thermodynamic Equation of State De- velopment 2025. doi:10.5281/zenodo.8317546
-
[6]
P.; Paprocki, M.; Čertík, O.; Kirpichev, S
Meurer, A.; Smith, C. P.; Paprocki, M.; Čertík, O.; Kirpichev, S. B.; Rocklin, M.; Kumar, A.; Ivanov, S.; Moore, J. K.; Singh, S.; Rathnayake, T.; Vig, S.; Granger, B. E.; Muller, R. P.; Bonazzi, F.; Gupta, H.; Vats, S.; Johansson, F.; Pedregosa, F.; Curry, M. J.; Terrel, A. R.; Roučka, Š.; Saboo, A.; Fernando, I.; Kulal, S.; Cimrman, R.; Scopatz, A. SymP...
-
[7]
Tange, O. GNU Parallel 2018 doi:10 . 5281/zenodo . 1146014 (Ole Tange, 2018)
work page 2018
-
[8]
AI Feynman: A physics-inspired method for symbolic regression
Udrescu, S.-M.; Tegmark, M. AI Feynman: A physics-inspired method for symbolic regression. Science Advances 6, eaay2631. doi:10 . 1126/sciadv . aay2631 (2020)
work page 2020
Show all 10 references
-
[9]
Van Rossum, G.; Drake Jr, F. L. Python reference manual (Centrum voor Wiskunde en Informatica Amsterdam, 1995)
1995
-
[10]
P.Symbolic regression is NP-hardarXiv: 2207.01018
Virgolin, M.; Pissis, S. P.Symbolic regression is NP-hardarXiv: 2207.01018. (V. Martinek) Interdisciplinary Center for Scientific Computing, Heidelberg Uni- versity, 69120 Heidelberg, Germany Email address: viktor.martinek@iwr.uni-heidelberg.de
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.