Breaking the Simplification Bottleneck in Amortized Neural Symbolic Regression

Paul Saegert; Ullrich K\"othe

REVIEW 4 major objections 4 minor 10 references

Amortized neural symbolic regression can match evolutionary-search accuracy on realistic scientific equations once expression simplification becomes fast pattern matching — and finds more concise laws as the sampling budget grows.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 03:05 UTC pith:FUZF4YIK

load-bearing objection SimpliPy is a real engineering contribution, but the paper's own Figure 4 undercuts the 'comparable quality' claim at the L_max=4 setting used everywhere else. the 4 major comments →

arxiv 2602.08885 v5 pith:FUZF4YIK submitted 2026-02-09 cs.LG cs.AIcs.SC

Breaking the Simplification Bottleneck in Amortized Neural Symbolic Regression

Paul Saegert , Ullrich K\"othe This is my paper

classification cs.LG cs.AIcs.SC

keywords symbolic regressionamortized inferencetransformerexpression simplificationpattern matchingparsimonymachine-precision recoverydecontamination

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Symbolic regression is usually done by evolutionary search, but a learned, amortized model could in principle transfer knowledge across datasets. The paper argues that the reason amortized models have lagged behind is a data-generation bottleneck: simplifying millions of randomly generated expressions with a general-purpose computer algebra system is too slow, so training corpora are small or unsimplified. It introduces a rule-based simplification engine that precomputes normalization patterns offline and applies them by table lookup, achieving roughly a hundredfold speed-up at comparable simplification quality. With that engine in the training loop, a transformer is trained on 512 million on-the-fly generated and simplified expressions and, on a benchmark of physically motivated test equations, recovers machine-precision solutions at a rate comparable to a leading genetic-programming method while producing more concise expressions as the inference budget increases. If the central claim holds, the main obstacle to scaling amortized symbolic regression is not model capacity but cheap normalization of training targets.

Core claim

The central claim is that the simplification step, not model architecture, is the bottleneck blocking amortized symbolic regression from scaling, and that simplification can itself be amortized: a rule set over expressions of bounded size is discovered once offline; runtime simplification then reduces to hash lookups and local cancellations on token sequences. With this engine feeding a transformer trained on 512M on-the-fly expressions, the largest model recovers 55.8% of benchmark equations to 32-bit machine precision at a thousand-second budget, versus 50.0% for the genetic-programming baseline, while earlier amortized models top out near 10%. The same evaluation shows a parsimony inversi

What carries the argument

The carrying mechanism is SimpliPy, a pattern-matching simplification engine. It operates directly on tokenized prefix expressions and uses two interleaved passes: a rule-application pass that matches precomputed rewrite patterns, and a cancellation pass that merges associative and commutative clusters and cancels inverse pairs. The rules are discovered offline by building a directed minimum spanning forest over a graph of expressions up to seven symbols and four variables, so runtime normalization reduces to bounded lookups. Around this engine, Flash-ANSR trains an encoder-decoder transformer on a synchronous on-the-fly data stream, and at inference samples many candidate skeletons, simplif

Load-bearing premise

The whole claimed advantage rests on the bounded simplification rule set (patterns no longer than four symbols) being complete enough to normalize the expressions the model will meet; the paper trains with that cap but shows simplification quality only reaches parity with the general-purpose simplifier when longer patterns are allowed.

What would settle it

Train the same model family with the simplification rule cap raised to five or six symbols and measure recovery on the same benchmark; if the recovery rate does not improve, the bounded rule set is not the binding constraint. Independently, count what fraction of benchmark expressions (with constants pruned) the four-symbol rule set fails to reduce to the same normal form as the full rule set — a non-negligible fraction would mean the model was trained on redundant targets despite the speedup.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

Training data for amortized symbolic regression can be generated at a scale (512M expressions) that was previously impractical, because simplifying each expression costs milliseconds rather than roughly a tenth of a second.
At machine-precision criteria, a learned posterior can match or slightly exceed a strong evolutionary search at medium-to-high inference budgets, making amortized models a viable alternative rather than a toy.
With more inference compute, the recovered expressions become shorter, so spending additional time does not trade interpretability for accuracy the way evolutionary search does.
The paper's decontamination and time-normalized evaluation protocol gives a reproducible standard for separating genuine generalization from leakage.
Scaling model size and sampling budget follows a power law in recovery, so the gains observed at the smaller model sizes are expected to continue.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The paper trains and evaluates with the rule cap at four symbols, yet its own quality comparison shows the engine matches the general-purpose simplifier only when the cap is five or higher. A natural next experiment is training with the larger rule set; the paper leaves this untested.
The parsimony inversion suggests a broader principle: a learner that assigns probability mass near concise forms spends additional budget sampling rare, simple candidates, whereas a local mutator spends it refining complex approximations. Other sampling-based generative models should show similar 'concise-first' scaling if their training targets are normalized.
The decontamination trick — pruning constants and comparing structural images on random inputs — could be lifted out of symbolic regression and applied to any benchmark where equivalent syntactic variants of a target are plausible.
If noise is added during training, the observed drop in noise robustness may disappear, because the encoder would no longer treat every fluctuation as signal; this is a direct, testable fix suggested by the paper's own diagnosis.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

SimpliPy is a real engineering contribution, but the paper's own Figure 4 undercuts the 'comparable quality' claim at the L_max=4 setting used everywhere else.

read the letter

The short version: SimpliPy is a genuine engineering contribution, and the Flash-ANSR training pipeline is executed with more care than most work in this area. But the paper's own Figure 4 shows that L_max=4 — the setting used for all training and inference — does not achieve SymPy-comparable simplification quality. The '100-fold speedup at comparable quality' claim is therefore not supported as stated.

What's actually new: the offline rule discovery (Kruskal-style spanning forest) plus online hash and subtree matching is a clever way to amortize simplification. Training on 512M on-the-fly simplified expressions is a real step up in scale, and the evaluation protocol (machine-precision recovery, time-normalized Pareto frontier, decontamination, confidence intervals) is more rigorous than anything in the amortized SR baselines. The parsimony inversion — where Flash-ANSR tends to find shorter expressions as compute increases while PySR adds complexity — is an interesting result worth taking seriously.

The L_max issue is the main soft spot. Figure 4 (right) explicitly says Lmax ≥5 achieves comparable simplification ratios to SymPy; the inset shows Lmax=4 leaves a heavier tail of under-simplified expressions. Since all model training and inference uses Lmax=4, the paper's premise that SimpliPy matches SymPy quality at the deployed setting is not demonstrated. This is not fatal to the benchmark comparison against PySR — the model still trains on simplified targets, just not as thoroughly — but it directly undercuts the speedup-at-comparable-quality headline. The SymPy baseline in that comparison also substitutes random constants to get around SymPy's handling of free constants, which is a reasonable workaround but could skew the time measurement. And five FastSRB equations are excluded; no sensitivity analysis on that.

The noise limitation is acknowledged honestly, and the paper doesn't oversell its robustness there. All in all, the core engineering is sound, the evaluation is thoughtful, and the claims are mostly clear. The main fix is simple: run the model with an L_max≥5 rule set and release that rule set along with checkpoints. If that still shows near-SymPy quality at high speed, the paper is solid.

Who this is for: anyone working on amortized symbolic regression or on fast symbolic simplification for training data generation. I'd send it to peer review with the expectation of a minor revision. The L_max issue should be caught by an attentive referee.

Referee Report

4 major / 4 minor

Summary. The paper identifies the computational cost of CAS-based simplification (e.g., SymPy) as a bottleneck for amortized neural symbolic regression and proposes SimpliPy, a rule-based/hash-based simplification engine that operates directly on tokenized prefix expressions. SimpliPy precomputes a bounded set of rewriting rules offline and applies them via pattern matching, claiming up to 100× speedup over SymPy at comparable quality. The authors integrate SimpliPy into Flash-ANSR, an encoder-decoder Transformer trained on 512M on-the-fly generated and simplified expression–dataset pairs. On the FastSRB benchmark they report that Flash-ANSR outperforms static/unsimplified amortized baselines (NeSymReS, E2E) and matches or partially surpasses PySR at larger inference budgets, while exhibiting a decreasing expression-length ratio with more compute. The paper also emphasizes rigorous evaluation protocols, including machine-precision recovery thresholds, bootstrapped confidence intervals, and symbolic/numeric decontamination of training data.

Significance. If the central claims hold, the paper makes a strong practical contribution: it provides a fast simplification engine that could enable large-scale dynamic generation of high-quality symbolic regression training data, and it demonstrates that amortized methods can be competitive with genetic programming on realistic benchmarks. The manuscript also contains several commendable practices: detailed hyperparameter tables, release of code (though not all artifacts are verifiable here), bootstrapped confidence intervals, and a thoughtful test-time-compute evaluation protocol. However, the headline claim of '100× speed-up at comparable quality' is not actually demonstrated for the configuration used in training and inference (L_max=4), because the paper's own Figure 4 shows quality parity with SymPy only at L_max≥5. This gap is load-bearing for the method's motivation and for the downstream performance claims. Additional fairness concerns about the expression-length comparison and the SymPy baseline further weaken the current evidence. With targeted additional experiments and clarifications, the paper could be a significant contribution to amortized symbolic regression.

major comments (4)

[§5.2, Fig. 4; §3.3; App. B] The paper's central premise is that SimpliPy provides 'comparable quality' to SymPy, but the quality comparison in Fig. 4 (right) explicitly shows that this holds only for L_max ≥ 5. All training (§3.3), inference (§3.5), and the reported speedup use L_max = 4. The under-simplified tail at L_max=4 means training targets may retain the syntactic redundancy that the method is designed to eliminate, and the downstream results do not isolate the benefit of the simplifier at the deployed configuration. Please either retrain/evaluate with L_max ≥ 5 or quantify how the simplification-quality gap affects the Flash-ANSR results.
[§5.1, Fig. 2 (right); §4.4; §3.5] The parsimony comparison ('parsimony inversion') may be unfair: Flash-ANSR candidates are explicitly simplified with SimpliPy before evaluation (§3.5), but the paper does not state whether PySR and NeSymReS outputs are also normalized before computing expression length ratios. If PySR expressions are not simplified, their increasing length ratio over time could largely reflect accumulated algebraic redundancy rather than genuine complexity growth. Please clarify the protocol or apply the same simplifier to all methods before measuring |τ̂|/|τ|.
[§4.5, Fig. 4 (left)] The SymPy baseline is measured through a compatibility layer that substitutes random constants into expressions before parsing and then masks numerical values after simplification. This changes the simplification problem (SymPy is no longer performing symbolic simplification of constants) and conflates conversion/parsing overhead with simplify-routine cost, whereas SimpliPy operates directly on tokenized input. The claimed 100× speedup at 'comparable quality' is therefore not an apples-to-apples comparison. A fairer benchmark would use SymPy's native symbolic simplification (or state clearly why the compatibility layer is necessary) and report both simplify-only and end-to-end times.
[§5.1, Fig. 2 (left)] The text states that the 120M model 'partially surpasses' PySR, reaching 55.8% vNRR versus 50.0%, but does not report whether the difference is statistically significant given the bootstrapped 95% confidence intervals shown in the figure. If the intervals overlap at the relevant budgets, the claim of surpassing PySR should be softened to 'matching within uncertainty.' Please report explicit confidence intervals and the inference-time budgets at which the comparison is made.

minor comments (4)

[Throughout] The name is written inconsistently as 'SimpliPy' and 'SIMPLIPY'; please standardize. Also, 'SimpliPy' vs 'SIMPLIPY' in the abstract and body should be unified.
[Appendix B, Table 3] The 'Logical CPUs' row shows 4, 32, 8, 8, which seems counterintuitive for the larger models; please clarify whether this is the number of CPU workers for data generation or for evaluation.
[§3.5] The text says 'SciPy.optimize.curve fit' — should be 'scipy.optimize.curve_fit'.
[Algorithm 2] The condition 'if no such simplification exists' in Step 2 of Phase 1 is clear, but in Algorithm 2 τ* is always computed using SIMPLIPY_R; if τ* exists, the later loop over j starts from 1 to min(|τ*|−1, Ltgt). Please clarify whether the 'Selection' step in the main text corresponds to the case where SIMPLIPY_R returns τ unchanged or to something else.

Circularity Check

0 steps flagged

No significant circularity: central claims are tested against external benchmarks and an independent symbolic engine.

full rationale

The core derivations are not circular. SimpliPy's rule set is mined offline by numerical equivalence checks (Algorithm 1) using random constants and Levenberg-Marquardt fitting; the online simplifier then applies those mined rules. The subsequent speed/quality claim is measured against SymPy, an external system, on 64k expressions from the training distribution (Section 4.5), and quality is reported separately by Lmax (Figure 4). Flash-ANSR's recovery-rate claims are evaluated on the held-out FastSRB benchmark under a strict FVU <= epsilon_32 criterion, with an explicit decontamination procedure that removes training expressions numerically equivalent to test skeletons; comparisons to NeSymReS, E2E, and PySR are external and not fitted to the reported outcome. The main weakness is a validity/evidence gap, not circularity: the deployed simplification setting is Lmax=4 (Appendix B), while Figure 4 (right) shows quality parity with SymPy only at Lmax>=5. Thus the 'comparable quality' premise is not demonstrated at the exact configuration used to train and evaluate Flash-ANSR. This is a correctness concern, not a circular derivation: no predicted quantity is made equal to an input by construction. The Figure 3 scaling-law extrapolation is explicitly a fit to the same observed curves and is presented as extrapolation, not as an independently tested prediction; it is also not load-bearing for the paper's main results. No self-citation chain or imported uniqueness theorem forces the conclusions.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 3 invented entities

The paper's contribution is heavily empirical: one engineered artifact (rule set) and one fitted model family. The load-bearing assumptions are the completeness of the offline rule set at Lmax=4, the validity of Algorithm 1's numeric equivalence checks, and the alignment of the hand-crafted training prior with the FastSRB benchmark. The most significant free parameters (rule set, operator weights, prior, γ, decontamination thresholds) are chosen by hand or fitted to the benchmark, not derived.

free parameters (6)

Simplification rule set R (rules of length ≤ L_max=4) = not enumerated in manuscript; derived offline via Algorithm 2 on up to 7-symbol expressions
The rule set is the entire simplifying power of SimpliPy; it is fitted/selected against the equivalence checker, and the paper does not provide the full rule table (only a random subset in Appendix M). Every simplification performed during training and inference is mediated by this fitted artifact.
Operator weights: +,-,*,/ = 10, others = 1 = 10 vs 1
Section 3.3: the training prior is manually chosen; it shifts the distribution of training expressions.
Parsimony regularization coefficient γ = 0.05 (0.15 for Figure 3)
Equation (3): calibrates how much complexity is penalized; affects the expression-length comparisons and final candidate selection.
Training prior: nops ~ exp[nops^0.7], constants ~ N(0,σ=5), support M~U(1,1024), variable count U(1,min(nleaves,D)) = various
Section 3.3: multiple hand-set parameters shaping the training posterior; the ablation in Section 5.5 shows such choices strongly affect recovery.
Decontamination threshold = 10^-4 absolute difference on X_check
Section 3.3: the numerical match threshold is set by hand; changing it changes how many equivalent families leak into training.
Lmax pattern cap = 4 during training/inference = 4
Section 3.1 and Appendix B: a central speed/quality trade-off; Figure 4 shows quality parity with SymPy only at Lmax≥5, so the chosen value may under-simplify the training distribution.

axioms (5)

ad hoc to paper Algorithm 1's numeric equivalence check with random constants (K=16 challenges, retries R=16, LM fitting) correctly decides functional equivalence of generated expressions.
Section E: if the check misses equivalences (wrong local minima, sampling coverage), SimpliPy's rule mining and the training-set decontamination silently fail.
domain assumption The Lample & Charton (2019) skeleton-generation procedure covers the distribution of physical laws in FastSRB.
Section 3.3: the training prior is sampled from skeletons generated by this procedure; FastSRB contains physically motivated laws, and no guarantee is given that the two distributions align.
domain assumption FVU ≤ ε_32 on a finite validation split implies the correct law was recovered.
Section 4.4: for sufficiently expressive functions, finite-point machine-precision fits can be achieved by wrong expressions; the paper relies on the validation split to make this unlikely.
domain assumption The autoregressive softmax-sampling procedure eventually assigns non-negligible probability to the correct skeleton for each FastSRB expression.
Section 3.5 and Fig. 2: without this, the memoryless sampling strategy could not find the 'needle' regardless of budget.
ad hoc to paper Pre-computation of rules for expressions with up to 4 variables and 7 symbols (with literals ±1, π, e, ±∞, NaN) is sufficient to capture a large fraction of simplifications encountered in the training distribution.
Section 3.1: the offline amortization is bounded to these sizes; the paper provides no analysis of what fraction of the training distribution this covers beyond anecdotal examples.

invented entities (3)

SimpliPy rule-based simplification engine independent evidence
purpose: Fast symbolic simplification of training and inference expressions.
The engine is published as a GitHub repository; its speed/effectiveness on expressions (Fig. 4) is a falsifiable claim independent of the neural model. However, the exact rule table is not in the paper.
Flash-ANSR models (v23.0-3M/20M/120M/1B) no independent evidence
purpose: Trained transformer systems for amortized symbolic regression.
The model checkpoints are not released, so the specific trained systems cannot be independently evaluated.
Masked RMSSetNorm independent evidence
purpose: Normalization layer for the set encoder.
Defined mathematically (Appendix C); it is a standard architectural modification and could be implemented independently.

pith-pipeline@v1.3.0-alltime-deepseek · 19005 in / 9539 out tokens · 76870 ms · 2026-08-03T03:05:53.644640+00:00 · methodology

0 comments

Cite this review

Pith. "Pith review of Breaking the Simplification Bottleneck in Amortized Neural Symbolic Regression." pith.science (2026). https://pith.science/paper/FUZF4YIK

@misc{pith2026260208885,
  author       = {Pith},
  title        = {Pith review of: Breaking the Simplification Bottleneck in Amortized Neural Symbolic Regression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FUZF4YIK}},
  note         = {Machine review of arXiv:2602.08885}
}

Share X Bluesky LinkedIn Reddit HN

read the original abstract

Symbolic regression (SR) aims to discover interpretable analytical expressions that accurately describe observed data. Amortized SR promises to be much more efficient than the predominant genetic programming SR methods, but currently struggles to scale to realistic scientific complexity. We find that a key obstacle is the lack of a fast reduction of equivalent expressions to a concise normalized form. Amortized SR has addressed this with general-purpose Computer Algebra Systems (CAS) like SymPy, but the high computational cost severely limits training and inference speed. We propose SimpliPy, a rule-based simplification engine achieving a 100-fold speed-up over SymPy at comparable quality. This enables substantial improvements in amortized SR, including scalability to much larger training sets, more efficient use of the per-expression token budget, and systematic training set decontamination with respect to equivalent test expressions. We demonstrate these advantages in our Flash-ANSR framework, which achieves much better accuracy than amortized baselines (NeSymReS, E2E) on the FastSRB benchmark. Moreover, it performs on par with state-of-the-art direct optimization (PySR) while recovering more concise rather than more complex expressions with increasing inference budget.

Figures

Figures reproduced from arXiv: 2602.08885 by Paul Saegert, Ullrich K\"othe.

**Figure 1.** Figure 1: The FLASH-ANSR training pipeline. Following the established standard encoder-decoder paradigm, our framework integrates SimpliPy (top center) into the loop for synchronous simplification of on-the-fly generated training expressions. tion engine to resolve it. 2. We present FLASH-ANSR, which leverages SIMPLIPY to train on 512M on-the-fly generated and simplified data-expression pairs, scaling to higher-dime… view at source ↗

**Figure 2.** Figure 2: Left: Validation Numeric Recovery Rate (vNRR) as a function of inference time (log scale). FLASH-ANSR models (shades of blue) scale monotonically with compute, with the 120M model partially surpassing the PySR baseline (red). Baselines NeSymReS and E2E fail to generalize to the benchmark. Right: Expression Length Ratio |τˆ|/|τ | versus compute. We observe a parsimony inversion: while PySR increases comp… view at source ↗

**Figure 3.** Figure 3: FLASH-ANSR fits to its own and PySR’s scaling curves log10(T) vs vNRR from [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

**Figure 5.** Figure 5: Left: Validation Numeric Recovery Rate (vNRR) vs. the number of support points M. FLASH-ANSR (120M) outperforms PySR in the dense data regime (M > 64). Right: Expression Length Ratio |τˆ|/|τ |. We observe a distinct “Complexity Peak” at M ≈ 8, where the model generates expressions significantly longer than the ground truth. This peak coincides with a regime of high uncertainty (low log-probability) and exc… view at source ↗

**Figure 4.** Figure 4: characterizes the performance of our SIMPLIPY engine and the SymPy simplify routine [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

**Figure 7.** Figure 7: highlights the importance of a well-designed training prior. Restricting constant values to a narrow positive range U(1, 5) as in prior work (Biggio et al., 2021) severely limits the model’s ability to generalize to test expressions with constants near zero or negative values. By expanding the training prior to include negative values and values near zero, the model learns to associate subtle variations i… view at source ↗

**Figure 6.** Figure 6: Left: Validation Numeric Recovery Rate (vNRR) vs. the noise level η (log scale). While FLASH-ANSR (shades of blue) achieves competitive performance in the noiseless regime (circles), PySR (red) retains higher recovery rates at larger noise levels. Right: Expression Length Ratio |τˆ|/|τ |. On noisy data, PySR (red) and NeSymReS (orange) produce expressions of similar complexity to the ground truth whereas … view at source ↗

**Figure 8.** Figure 8: FLASH-ANSR fits (v23.0-120M, γ = 0.05, 32k samples ≈ 100s) (red curves) on uniformly sampled data points (black dots) from several 1D expressions (M = 64, η = 0). Many ground truth expressions (bold) are recovered exactly, while others are recovered in alternative forms (e.g. yˆ = (x + 1)2 for y = x 2 + 2x + 1) [PITH_FULL_IMAGE:figures/full_fig_p012_8.png] view at source ↗

**Figure 9.** Figure 9: Diverse FLASH-ANSR solutions (v23.0-120M, γ = 0.05, 32k samples ≈ 100s) (red curves) on uniformly sampled data points (black dots) depicting the trajectory of a damped harmonic oscillator (M = 150, η ≈ 0.04). Multiple distinct expressions fit the data equally well given the noisy data. The ground truth expression (top left, bold) is exactly recovered with high probability (top right). Other solutions (bott… view at source ↗

**Figure 10.** Figure 10: FLASH-ANSR fits (v23.0-120M, γ = 0.05, 32k samples ≈ 100s) (middle columns) on uniformly sampled data points (black dots) from several 2D expressions from Biggio et al. (2021) (M = 100, η = 0). Many ground truth expressions (left columns, bold) are recovered exactly, while few others have additional terms that do not affect the relative fit error (right columns) B. Hyperparameters We provide a detailed sp… view at source ↗

**Figure 11.** Figure 11: Visual depiction of the Flash-ANSR model architecture. The Set Transformer encoder ingests a variable-sized set of input-output pairs and produces a fixed-size latent representation via Induced Set Attention Blocks (ISAB) and Set Attention Blocks (SAB). The Transformer decoder autoregressively generates a symbolic expression token-by-token, attending to the encoded dataset at each step. E. Expression Equi… view at source ↗

**Figure 12.** Figure 12: Training data statistics. Left: Empirical Cumulative Distribution Functions (ECDFs) of the number of constants, unique variables, variable nodes and the expression length in the training data. Half of the simplified training expressions contain at least 5 unique variables and 24 symbols. Right: 2D histogram showing the number of unique variables and the number of variable nodes. The training expressions c… view at source ↗

**Figure 13.** Figure 13: Training dynamics and scaling behavior. Left: Validation Cross-Entropy loss (log-scale) as a function of estimated training FLOPs (log-scale) We use the approximation FLOPS = 6ND following Kaplan et al. (2020), and count FLOPS separately for the encoder with its dataset input and the decoder with its tokenized expression input before adding both for the total FLOPs. The performance envelope generally foll… view at source ↗

**Figure 14.** Figure 14: Detailed comparison various time-normalized numerical and symbolic metrics. With increasing time budget, FLASH-ANSR consistently generates more concise (lower expression length ratio), specific (fewer excess constants), and natural (lower total nestedness) expressions than PySR. L.2. Detailed Data Sparsity Results [PITH_FULL_IMAGE:figures/full_fig_p020_14.png] view at source ↗

**Figure 15.** Figure 15: Detailed results for varying number of data points M on the FASTSRB benchmark. In the Interpolation Regime (N ≈ 16), FLASH-ANSR maintains a lower mean total nestedness than PySR. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_15.png] view at source ↗

**Figure 16.** Figure 16: Detailed results for varying noise levels η on the FASTSRB benchmark. Across all noise levels, expressions by FLASH-ANSR maintain similar qualities. With higher noise levels, PySR still achieves shorter and more specific expressions. M. SimpliPy Rules [PITH_FULL_IMAGE:figures/full_fig_p021_16.png] view at source ↗

**Figure 17.** Figure 17: Median and 95% confidence intervals of the FVU with v23.0-120M, γ = 0.05, 8 optimizer restarts, 265k samples ≈ 1000s across individual expressions in the FASTSRB benchmark stratified by difficulty level following (Matsubara et al., 2024). All but three “easy” expressions are solved perfectly (left, green dots). In both “medium” and “hard” categories, we observe a large number of approximate fits (orange) … view at source ↗

**Figure 18.** Figure 18: Token embeddings for the Flash-ANSR model family (v23.0) across model sizes. Across all scales, we observe high similarity of semantically related operators (e.g., binary operators, trigonometric functions, exponentials and powers) and variable tokens, and low similarity between different types of tokens (e.g. operators vs variables). The overall strength of these relationships varies across model sizes w… view at source ↗

discussion (0)

Reference graph

Works this paper leans on

10 extracted references · 2 linked inside Pith

[1]

Skeleton draw.We samplen ops = 4and produce the prefix skeleton with a free constant placeholder ⋄ : ¯τ (1) = [ + , mult2 , sin , x1 , pow2 , ⋄ ]
[2]

Simplification.SimpliPy (L max = 4) absorbs pow2 into the constant placeholder and simplifies the expression to ¯τ (1)∗ = [ + , mult2 , sin , x1 , ⋄ ]
[3]

Assume the held-out pool contains the expression as itsi-th entry τ (i) − = [ mult2 , sin , / , x1 , ⋄ ]

Test-set guard.We drop constants from both train candidates and held-out expressions, reducing the candidate to ˜τ (1) = [ mult2 , sin , x1 ]. Assume the held-out pool contains the expression as itsi-th entry τ (i) − = [ mult2 , sin , / , x1 , ⋄ ]. which reduces to ˜τ (i) − = [ mult2 , sin , x1 ]. Since ˜τ (1) = ˜τ (i) − , we reject this instance and proc...
[4]

URL https: //doi.org/10.1007/978-3-031-57065-0

doi: 10.1007/978-3-031-57065-0. URL https: //doi.org/10.1007/978-3-031-57065-0. 10 Breaking the Simplification Bottleneck in Amortized Neural Symbolic Regression Schmidt, M. D. and Lipson, H. Distilling free-form nat- ural laws from experimental data.Science, 324:81 – 85, 2009. URL https://api.semanticscholar. org/CorpusID:7366016. Su, J., Lu, Y ., Pan, S...

Pith/arXiv arXiv 2009
[5]

easy” expressions are solved perfectly (left, green dots). In both “medium

Constant and data draw.The constant is sampled from N(0, σ= 5) as c1 = 1.15. We sample M= 137 data points with Xj ∼ U(aj, bj) where aj, bj ∼ N(0, σ= 10). Evaluating y=f ¯τ (2) (X;c 1) yields finite, real values, so the instance is accepted without further resampling. J. Data Statistics Figure 12.Training data statistics. Left: Empirical Cumulative Distrib...

2020
[9]

17 Breaking the Simplification Bottleneck in Amortized Neural Symbolic Regression Simplification yields ¯τ (2)∗ = [ * , / , x1 , pow2 , x2 , ⋄ ]

Resampling.A new skeleton is drawn withn ops = 6, ¯τ (2) = [ * , / , x1 , pow2 , x2 , * , ⋄ , + , x1 , neg , x1 ]. 17 Breaking the Simplification Bottleneck in Amortized Neural Symbolic Regression Simplification yields ¯τ (2)∗ = [ * , / , x1 , pow2 , x2 , ⋄ ]. which, after removing constants, becomes ˜τ (2) = [ / , x1 , pow2 , x2 ]. No held-out expression...
[945]

Burlacu, B., Kronberger, G., and Kommenda, M

Pmlr, 2021. Burlacu, B., Kronberger, G., and Kommenda, M. Operon c++: An efficient genetic programming framework for symbolic regression. InProceedings of the 2020 Genetic and Evolutionary Computation Conference Companion, GECCO ’20, pp. 1562–1570, New York, NY , USA, 2020. Association for Computing Machin- ery. ISBN 9781450371278. doi: 10.1145/3377929. 3...

arXiv 2021
[2022]

org/CorpusID:250048548

URL https://api.semanticscholar. org/CorpusID:250048548. 11 Breaking the Simplification Bottleneck in Amortized Neural Symbolic Regression A. Examples Figure 8.FLASH-ANSR fits (v23.0-120M, γ= 0.05, 32k samples ≈100 s) (red curves) on uniformly sampled data points (black dots) from several 1D expressions (M= 64, η= 0). Many ground truth expressions (bold) ...

2021
[2024]

doi: 10.1145/3620665.3640366. Ba, J. L., Kiros, J. R., and Hinton, G. E. Layer normaliza- tion, 2016. URL https://arxiv.org/abs/1607. 06450. Belkin, M., Hsu, D. J., Ma, S., and Mandal, S. Reconciling modern machine learning and the bias-variance trade- off.ArXiv, abs/1812.11118, 2018. URL https://api. semanticscholar.org/CorpusID:57189215. Bendinelli, T.,...

arXiv 2016
[5992]

URL https://doi

doi: 10.7717/peerj-cs.103. URL https://doi. org/10.7717/peerj-cs.103. Nakkiran, P., Kaplun, G., Bansal, Y ., Yang, T., Barak, B., and Sutskever, I. Deep double descent: Where big- ger models and more data hurt, 2019. URL https: //arxiv.org/abs/1912.02292. Petersen, B. K., Landajuela, M., Mundhenk, T. N., Santi- ago, C. P., Kim, S. K., and Kim, J. T. Deep ...

Pith/arXiv arXiv 2019

[1] [1]

Skeleton draw.We samplen ops = 4and produce the prefix skeleton with a free constant placeholder ⋄ : ¯τ (1) = [ + , mult2 , sin , x1 , pow2 , ⋄ ]

[2] [2]

Simplification.SimpliPy (L max = 4) absorbs pow2 into the constant placeholder and simplifies the expression to ¯τ (1)∗ = [ + , mult2 , sin , x1 , ⋄ ]

[3] [3]

Assume the held-out pool contains the expression as itsi-th entry τ (i) − = [ mult2 , sin , / , x1 , ⋄ ]

Test-set guard.We drop constants from both train candidates and held-out expressions, reducing the candidate to ˜τ (1) = [ mult2 , sin , x1 ]. Assume the held-out pool contains the expression as itsi-th entry τ (i) − = [ mult2 , sin , / , x1 , ⋄ ]. which reduces to ˜τ (i) − = [ mult2 , sin , x1 ]. Since ˜τ (1) = ˜τ (i) − , we reject this instance and proc...

[4] [4]

URL https: //doi.org/10.1007/978-3-031-57065-0

doi: 10.1007/978-3-031-57065-0. URL https: //doi.org/10.1007/978-3-031-57065-0. 10 Breaking the Simplification Bottleneck in Amortized Neural Symbolic Regression Schmidt, M. D. and Lipson, H. Distilling free-form nat- ural laws from experimental data.Science, 324:81 – 85, 2009. URL https://api.semanticscholar. org/CorpusID:7366016. Su, J., Lu, Y ., Pan, S...

Pith/arXiv arXiv 2009

[5] [5]

easy” expressions are solved perfectly (left, green dots). In both “medium

Constant and data draw.The constant is sampled from N(0, σ= 5) as c1 = 1.15. We sample M= 137 data points with Xj ∼ U(aj, bj) where aj, bj ∼ N(0, σ= 10). Evaluating y=f ¯τ (2) (X;c 1) yields finite, real values, so the instance is accepted without further resampling. J. Data Statistics Figure 12.Training data statistics. Left: Empirical Cumulative Distrib...

2020

[6] [9]

17 Breaking the Simplification Bottleneck in Amortized Neural Symbolic Regression Simplification yields ¯τ (2)∗ = [ * , / , x1 , pow2 , x2 , ⋄ ]

Resampling.A new skeleton is drawn withn ops = 6, ¯τ (2) = [ * , / , x1 , pow2 , x2 , * , ⋄ , + , x1 , neg , x1 ]. 17 Breaking the Simplification Bottleneck in Amortized Neural Symbolic Regression Simplification yields ¯τ (2)∗ = [ * , / , x1 , pow2 , x2 , ⋄ ]. which, after removing constants, becomes ˜τ (2) = [ / , x1 , pow2 , x2 ]. No held-out expression...

[7] [945]

Burlacu, B., Kronberger, G., and Kommenda, M

Pmlr, 2021. Burlacu, B., Kronberger, G., and Kommenda, M. Operon c++: An efficient genetic programming framework for symbolic regression. InProceedings of the 2020 Genetic and Evolutionary Computation Conference Companion, GECCO ’20, pp. 1562–1570, New York, NY , USA, 2020. Association for Computing Machin- ery. ISBN 9781450371278. doi: 10.1145/3377929. 3...

arXiv 2021

[8] [2022]

org/CorpusID:250048548

URL https://api.semanticscholar. org/CorpusID:250048548. 11 Breaking the Simplification Bottleneck in Amortized Neural Symbolic Regression A. Examples Figure 8.FLASH-ANSR fits (v23.0-120M, γ= 0.05, 32k samples ≈100 s) (red curves) on uniformly sampled data points (black dots) from several 1D expressions (M= 64, η= 0). Many ground truth expressions (bold) ...

2021

[9] [2024]

doi: 10.1145/3620665.3640366. Ba, J. L., Kiros, J. R., and Hinton, G. E. Layer normaliza- tion, 2016. URL https://arxiv.org/abs/1607. 06450. Belkin, M., Hsu, D. J., Ma, S., and Mandal, S. Reconciling modern machine learning and the bias-variance trade- off.ArXiv, abs/1812.11118, 2018. URL https://api. semanticscholar.org/CorpusID:57189215. Bendinelli, T.,...

arXiv 2016

[10] [5992]

URL https://doi

doi: 10.7717/peerj-cs.103. URL https://doi. org/10.7717/peerj-cs.103. Nakkiran, P., Kaplun, G., Bansal, Y ., Yang, T., Barak, B., and Sutskever, I. Deep double descent: Where big- ger models and more data hurt, 2019. URL https: //arxiv.org/abs/1912.02292. Petersen, B. K., Landajuela, M., Mundhenk, T. N., Santi- ago, C. P., Kim, S. K., and Kim, J. T. Deep ...

Pith/arXiv arXiv 2019

Breaking the Simplification Bottleneck in Amortized Neural Symbolic Regression

Core claim §

What carries the argument

Load-bearing premise §

What would settle it §

If this is right

Where Pith is reading between the lines

discussion (0)

Core claim

Load-bearing premise

What would settle it