{"id":"26e8070a-a1c2-4f42-a11d-c664a580b33c","arxiv_id":"2505.03977","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"An expanded SRBench comparison of 25 symbolic regression methods across 24 datasets shows no algorithm dominates all tasks and advocates for a living community benchmark.","lead":"The authors present SRBench 2.0, an expanded benchmark that compares 25 symbolic regression algorithms on 24 datasets with 30 repeated runs, energy tracking, and new accuracy-complexity visualizations. The results show no single algorithm wins across all problems, and the paper proposes community standards for keeping the benchmark alive.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Dataset selection in Section 3.1 uses SRBench 1.0 algorithm performance as clustering features, so the 'no single algorithm dominates' result may be an artifact of how the 12 black-box datasets were chosen.","rationale":"The paper's contribution is primarily an updated benchmark and a practical recommendation: because no single algorithm dominates, users should match algorithms to datasets. That recommendation rests entirely on the diversity of the selected datasets. Section 3.1's use of SRBench 1.0 algorithm performance as clustering features is a direct form of selection on the outcome variable. It is not merely a representativeness concern; it is a potential circularity: the benchmark is assembled to produce varied algorithm rankings, and then the paper reports that rankings are varied. The internal uDSR inconsistency (listed in Table 2 and Figure 2/3 but reported as incompatible in Section 5.1) is a separate data-quality issue, but it does not threaten the central claim as directly. The reader's weakest_assumption identified the same selection issue; I agree. The condition already placed by the reader—statistical reporting and clarification—should be extended to include a re-selection control. Since the reader already assigned CONDITIONAL and this concern reinforces that verdict rather than moving it, I leave the verdict unchanged.","tokens_in":17154,"tokens_out":4705,"duration_ms":45673,"concrete_test":"Rerun the Section 3.1 dataset selection on the same PMLB source pool, replacing the algorithm-performance features with intrinsic dataset statistics only (e.g., number of samples, number of features, feature-target correlations, target skewness/kurtosis, and baseline linear-regression R2). Keep the same R2 > 0.99 exclusion, the same 12-cluster t-SNE/k-means procedure, and the same 25 algorithms with the same 30-run protocol and grid search. If a single algorithm attains the best AUC on all 12 newly selected datasets (or on a large majority), the original no-free-lunch finding is an artifact of selection on prior performance. Complement this with a permutation test: draw 100 random 12-dataset subsets from PMLB after the exclusion and count how often no single algorithm dominates; a rare occurrence would show the chosen datasets are atypical and the general claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and Section 5.2 claim that 'no single algorithm dominates across all datasets,' and the paper uses this to argue that benchmarks should report per-dataset results and that users should select algorithms by problem characteristics. The load-bearing assumption is that the 12 black-box datasets in the benchmark are representative of SR problems. Section 3.1, however, constructs the black-box track by a selection procedure that directly uses algorithm performance: after filtering out datasets where R2 > 0.99 for all algorithms or linear regression, the authors form a feature vector per dataset containing 'the performance of each algorithm tested in the SRBench 1.0,' apply t-SNE and k-means, and pick the dataset closest to each of 12 centroids. This maximizes diversity in historical algorithm performance profiles. Because the algorithms evaluated here substantially overlap with those in SRBench 1.0, and because relative performance is at least moderately stable, the selected datasets are likely chosen to ensure that no single method wins everywhere. The 'no free lunch' result is therefore partly an artifact of the selection rule, not an independent property of the algorithm family. The paper also caps Friedman-type datasets at 25%, another indication that the selection is deliberately steered toward diversity. No control selection (random, or based only on intrinsic dataset statistics) is reported, so the generality of the central claim is untested.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents SRBench 2.0, an updated benchmark for symbolic regression that nearly doubles the number of evaluated methods to 25, increases independent runs to 30, and introduces 24 datasets split into a black-box track and a phenomenological/first-principles track. The benchmark uses performance-profile plots and cluster maps for visualization, measures energy consumption, and proposes community guidelines for a living benchmark, including hyperparameter-tuning standards and deprecation criteria. The central empirical claim, stated in the abstract and in Section 5.2, is that no single algorithm dominates across all datasets, which the authors use to argue for per-dataset performance reporting and problem-dependent method selection.","tokens_in":17459,"tokens_out":3615,"duration_ms":38111,"significance":"If the central claim holds, the paper makes a useful contribution: it provides the largest SR comparison under a unified experimental setup, ships open code and data, and offers concrete methodological innovations (performance profiles, Pareto-front visualizations, energy profiling, and a deprecation proposal). The paper is also honest about several limitations, such as the optimistic max-R2 aggregation and the incompatibility of uDSR. However, the significance is substantially tempered by the dataset-selection procedure, which uses previous algorithm performance to choose the black-box datasets, meaning that the 'no free lunch' result may be partly built into the benchmark rather than discovered from an independent sample of problems.","major_comments":[{"comment":"The selection of the 12 black-box datasets uses the performance of each algorithm from SRBench 1.0 as clustering features, followed by t-SNE and k-means, with the dataset closest to each centroid chosen. Since the algorithms evaluated in this paper substantially overlap with those in SRBench 1.0, this selection directly injects historical algorithm performance into the benchmark construction. The central claim that 'no single algorithm dominates across all datasets' (abstract and Section 5.2) may therefore be an artifact of choosing datasets that maximize diversity in previous performance profiles, rather than a property of the current algorithm set on representative problems. The additional cap on Friedman-derived datasets at 25% reinforces that selection is deliberately steered toward diversity. I request a control analysis: either random dataset selection, selection based only on intrinsic dataset statistics (samples, features, distribution), or a demonstration that the main conclusion is robust to alternative selection rules.","section":"Section 3.1"},{"comment":"The performance profiles and the cluster map are built from the maximum R2 across 30 runs, which the authors call an 'optimistic perspective.' No confidence intervals or variance measures are reported for the AUC values. Because the central comparison relies on these best-case values, a single lucky run can dominate the reported performance, and the conclusion that no method dominates may not hold under mean or median aggregation. I ask the authors to report mean/median profiles alongside the max, or to provide error bars or interval estimates for the AUC values, at least for the headline comparisons.","section":"Section 3.3 / Figure 2"},{"comment":"The text states that 'uDSR was not compatible with the library during the experiments,' yet Figure 3 includes uDSR as a column and Table 2 lists it as one of the evaluated algorithms, apparently with numerical results. This is internally inconsistent. If uDSR results are missing or invalid, they should be removed from all figures and tables or explicitly marked as unavailable; otherwise readers will incorrectly count uDSR among the successfully benchmarked methods.","section":"Section 5.1 / Figure 3 / Table 2"},{"comment":"Energy consumption is measured only after hyperparameter tuning, while the tuning budget is 6 hours per configuration compared with 1 hour for the final training run. The energy and runtime comparisons in Figure 1 and the discussion in Section 5.1 therefore exclude a large and potentially dominant part of the total computational cost. The authors should either report total energy including tuning or clearly state that the reported figures cover only the final training phase, and discuss how this affects the energy-related conclusions.","section":"Section 3.4 / Section 5.1"}],"minor_comments":[{"comment":"Several spacing errors occur in phrases such as 'acall for actionfrom' and 'Thiscall to action'; please correct these typographical issues.","section":"Abstract and Section 1"},{"comment":"The y-axis label 'P[R2 ≥ x]' should specify that the probability is computed from the maximum R2 over 30 runs, matching the 'max aggregation' description in the caption.","section":"Figure 2"},{"comment":"The k-means step uses 12 clusters and t-SNE with two dimensions, but no details are given about the t-SNE hyperparameters, the number of k-means restarts, or the stability of the resulting selection; adding these details would improve reproducibility.","section":"Section 3.1"},{"comment":"The text alternates between 'first-principles' and 'first-principle' (e.g., 'some first-principle equations'); please standardize the terminology.","section":"Section 5.4"},{"comment":"The equations in Table 3 would be easier to compare to the ground-truth formulas if the ground-truth expressions were shown alongside the recovered expressions, rather than only in separate subfigures of Figure 4.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a transparent, useful update to SRBench: 25 methods, 30 runs per dataset, performance profiles, energy accounting, containerized infrastructure, and a second track of first-principles problems. The code and data are released, and the authors are upfront about early termination and the missing uDSR. That alone makes it worth a serious referee.\n\nThe main result – no single algorithm dominates – is plausible but overstated. The black-box dataset selection in Section 3.1 uses SRBench 1.0 algorithm performance as clustering features and picks datasets closest to k-means centroids. That procedure is designed to maximize diversity in historical performance profiles, so the finding is at least partly an artifact of the selection rule rather than an independent discovery about symbolic regression algorithms. A random or intrinsic-statistics-based control would have tested the robustness. The paper would be more honest if it said “across this benchmark, constructed to cover diverse performance profiles, no algorithm wins everywhere” instead of presenting it as a general property.\n\nOther soft spots are minor. The AUC aggregation uses the max R2 over 30 runs, which is optimistic; the paper acknowledges this. There are no confidence intervals or significance tests, which matters more in a paper whose central claim is a negative one. The uDSR incompatibility is mentioned but not explained.\n\nWhat the paper does well is its infrastructural contribution: standardizing hyperparameter search, time budgets, model-size counting, and energy measurement across a wide range of implementations. The call for action is reasonable, and the deprecation criterion is a practical suggestion.\n\nWho is this for? Anyone benchmarking symbolic regression or developing SR methods. It does not resolve a foundational scientific question, and it doesn't claim to. The writing is clear and the analysis is generally careful.\n\nMy recommendation: send it to peer review. Require the authors to temper or empirically support the no-free-lunch claim, and to report per-dataset uncertainty. I would not desk-reject it.","headline":"Solid SRBench update, but the headline no-free-lunch result is partly an artifact of the dataset selection.","tokens_in":18026,"tokens_out":2023,"would_cite":true,"duration_ms":20468,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper's central empirical claim is that no single symbolic regression algorithm dominates across all datasets, so the field should replace single state-of-the-art rankings with per-problem, multi-objective benchmarking.","keywords":["symbolic regression","benchmarking","SRBench","genetic programming","performance profiles","no free lunch","model complexity","energy consumption"],"falsifier":"Re-run the same 25 methods, with the same tuning and time budget, on a much larger and independently chosen set of regression problems and check whether one method attains the best AUC on nearly every dataset; if it does, the no-domination claim collapses. The first-principles claim would be falsified by any SR method that, on several physics datasets, returns an expression at least as accurate and smaller than the published governing equation under the same protocol.","tokens_in":16985,"feed_emoji":"📊","tokens_out":10134,"duration_ms":98350,"temperature":0.7,"pith_summary":"This paper sets out to modernize the community benchmark for symbolic regression (SR)—searching data for interpretable mathematical expressions—by comparing 25 methods on 24 curated datasets with 30 independent runs each. Its central empirical claim is that no single algorithm dominates across all datasets: each method wins on some problems and struggles on others, so the field should stop reporting a single state-of-the-art method. The paper also reports that, on small and noisy physics datasets with known governing equations, no SR method found an expression that strictly beat the original equation in the trade-off between accuracy and model size, and that most top-performing methods share a constant-optimization step. This matters because those findings argue for a living, standardized benchmark that maps problems to algorithms and reports accuracy, model size, and energy cost together.","feed_headline":"No symbolic regression algorithm dominates every dataset","feed_subtitle":"A 25-method, 24-dataset benchmark with 30 runs each maps winners per problem and tracks size and energy.","key_machinery":"The argument is carried by the benchmark design plus its visualization machinery. The black-box track is built by first discarding datasets that all methods or plain linear regression already solve (training $R^2 > 0.99$), encoding each remaining dataset by row count, feature count, and the previous benchmark's algorithm-performance scores, reducing those encodings with t-SNE, and choosing the dataset nearest each of 12 k-means cluster centroids; a cap of 25% limits any one problem family. Results are read through performance profile plots—each algorithm's empirical probability of reaching any $R^2$ threshold, summarized as an area under the curve (AUC)—and a cluster heatmap that pairs per-dataset AUC with the size of the best expression. The multi-objective picture is completed by Pareto-front plots of $R^2$ against model size on the physics track and by energy-consumption estimates measured after hyperparameter tuning.","core_discovery":"Under one unified protocol—25 methods, 24 datasets, 30 seeded runs, grid-search hyperparameter tuning, and a one-hour training budget after tuning—the benchmark finds that no single algorithm dominates across all datasets: the best method differs by dataset, no method fails every problem, and methods cluster into stronger and weaker groups. Among the strongest performers, almost all include a constant-optimization step (linear, gradient-based, or non-linear), which the paper identifies as a key driver of success. On the phenomenological and first-principles track, no SR method produced an expression that strictly dominated the known governing equation in the accuracy-versus-model-size trade-off, and only one method recovered one of the equations; on noisy real data, higher $R^2$ often came with larger expressions that appear to fit noise rather than the underlying law. The paper additionally reports that the most energy-hungry methods tend to be Python-based or to include inner hyperparameter tuning, while GPU-based methods did not show clearly different energy use from CPU-based ones.","pith_inferences":["Because the black-box selection clusters on prior algorithm-performance scores, a neutral re-selection based only on dataset statistics (rows, features, codomain, noise level) would test whether the no-domination result is robust or an artifact of the curation procedure.","The per-dataset winners imply a practical meta-learning route: predict the best algorithm from dataset metadata, turning the benchmark into a recommender rather than a leaderboard.","The difficulty of beating known physics equations suggests adding distance-to-ground-truth as an explicit metric, so recovering the exact formula is rewarded even when label noise keeps raw $R^2$ below 1.","Energy and runtime could be folded into a single multi-objective score; that would penalize high-accuracy methods that consume orders of magnitude more compute and reward efficiency."],"forward_implications":["Practitioners should match algorithm to problem rather than trust a single state-of-the-art ranking, since the best method here varies by dataset.","Benchmark reports should present per-dataset outcomes and at least two objectives—accuracy and model size, plus energy—because aggregate scores hide which method wins where.","New SR methods should build in constant or parameter optimization, the component shared by the top-scoring algorithms in this comparison.","A living benchmark needs standardized hyperparameter tuning, fixed time budgets, and clear deprecation rules for unmaintained methods that do not appear on any Pareto front.","Energy use and early termination behavior should be reported alongside accuracy, since algorithms differ widely in runtime and some stop long before their time limit."],"supporting_citations":[{"why":"Defines the previous benchmark this work extends; its algorithm-performance metadata is used as clustering features for the black-box dataset selection.","marker":"[38]"},{"why":"Supplies the 12 black-box regression datasets and their sample and feature metadata.","marker":"[52]"},{"why":"Introduces the performance profile plot used to summarize each algorithm's $R^2$ distribution and compute AUC.","marker":"[19]"},{"why":"Provides one of the symbolic regression implementations evaluated and several of the first-principles datasets and governing equations used as ground truth.","marker":"[10]"},{"why":"Supplies real-world measurement datasets for the phenomenological track.","marker":"[53]"},{"why":"A prior principled benchmark whose track design and runtime-limit practices inform the updated protocol.","marker":"[14]"},{"why":"Documents the Friedman-family dataset bias in the previous benchmark that motivates the 25% cap on that problem family.","marker":"[16]"},{"why":"Provides the energy-consumption estimation method used in the energy analysis.","marker":"[7]"}],"fun_headline_variants":["No symbolic regression method wins every dataset","Symbolic regression: no one-size-fits-all algorithm","SRBench: constant optimization separates top solvers","Energy-aware symbolic regression benchmark exposes trade-offs","Call for living benchmarks in symbolic regression"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 24 selected datasets represent the range of problems symbolic regression will meet; because the black-box selection uses the previous benchmark's algorithm-performance scores as clustering features, the diversity behind the 'no single winner' conclusion could inherit the old benchmark's biases rather than being a general property of the algorithms.","fun_headline_variants_meta":{"raw":{"variants":["No symbolic regression method wins every dataset","Symbolic regression: no one-size-fits-all algorithm","SRBench: constant optimization separates top solvers","Energy-aware symbolic regression benchmark exposes trade-offs","Call for living benchmarks in symbolic regression"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000323,"raw_usage":{"total_tokens":1794,"prompt_tokens":908,"completion_tokens":886,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":524,"completion_tokens_details":{"reasoning_tokens":818}},"tokens_in":524,"tokens_out":886,"duration_ms":8215,"temperature":1.0,"reasoning_tokens":818,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:40:29.870887+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same 25 methods, with the same tuning and time budget, on a much larger and independently chosen set of regression problems and check whether one method attains the best AUC on nearly every dataset; if it does, the no-domination claim collapses. The first-principles claim would be falsified by any SR method that, on several physics datasets, returns an expression at least as accurate and smaller than the published governing equation under the same protocol.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the performance profile plot used to summarize each algorithm's $R^2$ distribution and compute AUC."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the Friedman-family dataset bias in the previous benchmark that motivates the 25% cap on that problem family."}],"review_version":1}