REVIEW 4 major objections 5 minor 32 references
Sparse Interpretable Deep Learning with LIES Networks for Symbolic Regression
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A fixed network of four interpretable activations, trained and pruned, recovers exact physics formulas on the AI Feynman benchmark more often than all SRBench baselines.
desk verdict LIES is a worthwhile new neural SR pipeline with strong ablations, but the 'outperforming all baselines' claim needs re-scoping and a like-for-like evaluation protocol. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the LIES layer: four neurons computing a bounded logarithm $L(x)=\ln(x)$ for $x>x_l$, identity $I(x)=x$, bounded exponential $E(x)=\exp(x)$ for $x<x_e$, and sine $S(x)=\sin(x)$, with dense residual connections, no bias terms, and a bias implemented by an extra constant input unit. The argument is carried by the multi-stage sparsification pipeline: weak ADMM training, oversampling in high-error input bins, strong ADMM training, node pruning through auxiliary identity edges, gradient-based pruning based on first-order output variation, gradient rounding to zero, least-squares coefficient optimization, and a final rounding pass. Theorem 1 is the universality claim; Theorem 2 supplies the bound that makes pruning and rounding safe.
What would settle it
Run LIES and the SRBench baselines on all 100 Feynman equations using a held-out train/test split, including the cases with more than four inputs and negative-input trigonometric compositions; if LIES no longer has the highest symbolic solution rate, the paper's central comparison collapses.
Extended reading notes
Core claim
LIES treats expression discovery as network sparsification: a fixed feed-forward network with four neurons per layer, one per primitive activation $L$, $I$, $E$, and $S$, is trained with a loss combining exponential error, ADMM sparsity, $\ell^1$ regularization, and masked activation-domain penalties; then weight pruning, node pruning, gradient-based pruning, rounding to zero, coefficient optimization, and final rounding distill a formula. The paper argues that this fixed architecture is a universal approximator (Theorem 1) and that first-order sensitivity bounds justify the pruning and rounding decisions (Theorem 2). On a subset of 61 Feynman equations (at most four inputs and no negative-input trigonometric compositions), LIES attains a higher symbolic solution rate than all compared SRBench baselines, with moderately lower accuracy in some cases attributed to small constant offsets in otherwise correct structures.
Load-bearing premise
The load-bearing premise is that the 61-formula subset and the evaluation protocol are representative of the benchmark: formulas with more than four inputs and trigonometric functions with negative inputs are excluded because LIES handles them poorly, and $R^2$ is computed on the training inputs rather than a held-out test set.
Editorial extensions
If this is right
- Symbolic regression becomes a differentiable optimization problem: a fixed architecture is trained once, then simplified, instead of searching a combinatorial space of expression trees.
- Because the activations are unary, multiplication and division fall out as $\exp(\ln a \pm \ln b)$, so laws built from products and ratios are directly representable.
- The ablation indicates every pipeline stage is load-bearing: removing gradient pruning leads to timeout on all formulas, and removing oversampling, rounding, or coefficient optimization sharply lowers symbolic recovery.
- Higher symbolic solution rate means the recovered expressions are often mathematically identical to the ground-truth formula, up to additive or multiplicative constants.
- LIES currently applies to formulas with at most four inputs and no compositions that take logarithms of negative trigonometric outputs; scaling to deeper and wider laws is left open.
Reading between the lines
- Beyond the paper, a natural extension is to test the same pruning pipeline with additional primitive activations, such as $\cos$ or $\tanh$, to cover a wider function vocabulary without changing the overall architecture.
- Beyond the paper, if the comparison were expanded to the full 100-equation Feynman set, the reported SSR advantage could shrink because the excluded cases are exactly those the authors identify as hard for LIES.
- Beyond the paper, the $R^2$ values are computed on the input data rather than a held-out test split, so generalization to unseen regions is not yet established by the paper.
- Beyond the paper, the complex-domain training mentioned in the discussion could remove the log-of-negative barrier and would be a concrete test of whether trigonometric multiplications become learnable.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper introduces LIES, a fixed feedforward network whose activations are logarithm, identity, exponential, and sine, together with a training and pruning pipeline (ADMM weight pruning, node pruning, gradient-based pruning and rounding, oversampling, and coefficient optimization) intended to extract compact symbolic formulas. The authors report experiments on 61 of the 100 AI Feynman equations, comparing accuracy (R2 > 0.99) and a symbolic solution rate (SSR) against SRBench baselines, and they include an ablation study of the pipeline components. The central claim is that LIES outperforms all baselines in symbolic recovery rate while remaining competitive in accuracy.
Significance. If the empirical claim were established under a like-for-like protocol, LIES would be a meaningful addition to deep-learning-based symbolic regression: it avoids population-based search, has a public implementation, and the paper includes a complexity analysis and ablations. The universal-approximation claim and the pruning rationale are also interesting. However, the current evaluation protocol—favorable subset selection, R2 computed on training data, and an unstated relationship between LIES's SSR and SRBench's solution rate—limits the significance until the comparison is made fair and transparent.
major comments (4)
- [Section IV-B and IV-C (Figs. 5-6)] The manuscript defines SSR for LIES as the frequency of exact symbolic equivalence, but it does not state whether the SRBench baseline SSR values in Fig. 6 were recomputed under this same definition or taken from SRBench's 'solution rate,' which is an R2-based metric. If the baseline values are the latter, Fig. 6 compares LIES's exact-symbolic-recovery rate against an R2-based solution rate, so the claim that LIES 'outperforms all baselines' is not supported. Please specify the baseline SSR computation and, if necessary, recompute baseline SSR by applying the same symbolic-equivalence check to recovered baseline expressions.
- [Section IV-A and V] The evaluation set is 61 of 100 AI Feynman equations, with the exclusions (more than four inputs and trigonometric functions with negative inputs) being exactly the failure modes identified in Section V. Because these exclusions are tied to known weaknesses of LIES, the subset is not clearly a neutral benchmark for comparing against methods without those limitations. The 'outperforming all baselines' claim should be scoped to this subset, and the paper should report results on the excluded equations or at least discuss the direction of the bias. The present text gives no evidence about how LIES would compare on the full 100-equation benchmark.
- [Section IV-C and IV-D] Section IV-C states that R2 is computed by evaluating the final equation on the input data, whereas SRBench baselines are evaluated under SRBench's standardized procedure, which uses held-out test data. This is a protocol mismatch, and the 'test set R2' phrase in Section IV-D is never defined, since no held-out split is described anywhere. Please evaluate LIES on a held-out split and report the same train/test protocol for all methods; otherwise the accuracy comparison in Fig. 5 cannot be interpreted.
- [Section IV-B] The definition of symbolic equivalence is underspecified: 'if we add or multiply a constant to it' does not state whether the constant is arbitrary per formula or must be the same throughout, nor does it specify the automated procedure for checking equivalence. Since SSR is the primary metric behind the headline claim, this needs to be made precise and reproducible.
minor comments (5)
- [Section III-B, Theorem 1] The proof of Theorem 1 is a reference to Figures 2a and 2b and a sentence about replicating polynomials; it does not give the explicit weight assignments needed to establish universal approximation. The claim is plausible, but the proof should be made self-contained or replaced with a citation to a formal result.
- [Section III-G] The abstract describes L as a 'bounded logarithm,' but the activation L(x) defined in Section III-G is constant only for x <= x_l and grows without bound for large x. Please correct the description or the activation.
- [Section IV-C, Figs. 5-6] The text reports means and standard deviations in the figures but gives no numerical values in the body. Add a table with the exact mean and standard deviation for each method and metric to make the comparison auditable.
- [Section IV-D, Table I] The ablation study is run only on the 30 formulas for which the full pipeline succeeded in at least one trial; this selection makes the reported 'out-of-time' and 'symbolic solution' percentages optimistic relative to the full benchmark. Please state this selection explicitly as a limitation.
- [Section III-C] The terms 'weak ADMM' and 'strong ADMM' are used without formal definitions; the hyperparameter values in Section IV-A are introduced ad hoc. Please define the two variants precisely, for example by the rho and lambda schedules used.
Circularity Check
No structural circularity: SSR is checked against external AI-Feynman ground-truth expressions; only the R² accuracy is in-sample by construction (constants fitted and evaluated on the same input data).
-
fitted input called prediction
[Section III-F ('Coefficient Optimization'); Section IV-C ('Performance Comparison'); Section IV-D ('Ablation Study')]
"We frame this as a regression problem and utilize least squares fitting to minimize the discrepancy between the obtained formula and the ground truth by adjusting only the constants within the expression. ... The R² score is computed by evaluating the final recovered equation on the input data and comparing its predictions to the ground-truth outputs. ... For cases where the solution is either correct or incorrect, we additionally report the frequency of achieving a test set R² >0.99."
By construction, the constants of the extracted formula are least-squares fitted to the input data (III-F), and the reported accuracy metric is then computed on that same input data (IV-C). The R² therefore measures the in-sample fit of the very optimization that set the constants — the least-squares residual it minimizes is the same sum of squared errors that R² evaluates — so the reported 'accuracy (R² > 0.99)' is statistically forced rather than a held-out prediction. Section IV-D even labels this quantity 'test set R²' although no held-out split is described anywhere. The reduction is direct and localized: the headline SSR claim in Fig.
full rationale
This paper's central quantitative claim — that LIES achieves a higher Symbolic Solution Rate (SSR) than SRBench baselines (Fig. 6) — is grounded in external ground truth: each AI-Feynman task provides the true symbolic expression, and the paper's SSR counts trials where the recovered formula is symbolically equivalent to that external target. This metric cannot be defined into existence by the training or fitting procedure. The universality argument (Theorem 1, Section III-B) reduces to the standard MLP universal-approximation theorem by constructing a sigmoid gadget from L, I, E units; the proof is informal and figure-based (an omitted-formal-proof concern, not a circular one) and does not presuppose its conclusion. The primitive-set ansatz (L, I, E, S) and the identities exp(ln(a)+ln(b)) = ab, exp(ln(a)-ln(b)) = a/b, cos(a) = sin(pi/2 - a) are explicitly hypotheses, not results imported from the authors' own prior work; the reference list contains no self-citations, so no self-citation or imported-uniqueness chain supports the claims. The one reduction-by-construction element found is on the accuracy side: Section III-F fits all numerical constants by least squares on the input data, and Section IV-C reports R² by evaluating the recovered equation on the same input data, with Section IV-D once labeling this 'test set R²'. That makes the reported R²/accuracy figures in-sample fits rather than predictions — a mild instance of fitted input called prediction that affects only the accuracy portion of the evaluation; the paper itself concedes lower accuracy on some formulae and rests 'outperforming all baselines' on SSR. Other flagged concerns are evaluation-fairness issues, not circularity: the restriction to 61 of 100 equations that excludes the cases Section V identifies as weaknesses (>4 inputs; trig with negative inputs); baselines trained on the full dataset while LIES uses 10%, with no demonstration that baseline 'SSR' values were recomputed under the same symbolic-equivalence rule; and the absence of a held-out test split for the reported R². These are like-for-like and rigor risks for the 'outperforming all baselines' headline, but they do not make any result equivalent to its inputs by definition. Score 2: one mild by-construction step, central claim independently grounded.
Assumptions & free parameters
free parameters (7)
- Logarithm activation cutoff x_l =
5e-3
- Exponential activation cutoff x_e =
4
- Gradient pruning sensitivity threshold =
0.01
- Gradient rounding threshold =
0.1
- Strong ADMM lambda =
5 * 10^{-(n-1)}
- Weak ADMM rho and lambda =
rho=0.5, lambda=5e-4
- Oversampling parameters P, k, max_iter =
P=30, k=8, max_iter=2
assumptions (5)
- standard math Mean value theorem and continuity of derivatives
- standard math Universal approximation of standard multilayer perceptrons
- domain assumption The 61-equation AI Feynman subset is a valid benchmark for the general claim
- domain assumption Log-space transformation and max normalization are appropriate for all selected equations
- ad hoc to paper The sigmoid emulation diagram in Fig. 2a correctly shows how LIES can approximate sigmoid
Cite this review
Pith. "Pith review of Sparse Interpretable Deep Learning with LIES Networks for Symbolic Regression." pith.science (2026). https://pith.science/paper/G5LUTNDP
@misc{pith2026250608267,
author = {Pith},
title = {Pith review of: Sparse Interpretable Deep Learning with LIES Networks for Symbolic Regression},
year = {2026},
howpublished = {\url{https://pith.science/paper/G5LUTNDP}},
note = {Machine review of arXiv:2506.08267}
}
read the original abstract
Symbolic regression (SR) aims to discover closed-form mathematical expressions that accurately describe data, offering interpretability and analytical insight beyond standard black-box models. Existing SR methods often rely on population-based search or autoregressive modeling, which struggle with scalability and symbolic consistency. We introduce LIES (Logarithm, Identity, Exponential, Sine), a fixed neural network architecture with interpretable primitive activations that are optimized to model symbolic expressions. We develop a framework to extract compact formulae from LIES networks by training with an appropriate oversampling strategy and a tailored loss function to promote sparsity and to prevent gradient instability. After training, it applies additional pruning strategies to further simplify the learned expressions into compact formulae. Our experiments on SR benchmarks show that the LIES framework consistently produces sparse and accurate symbolic formulae outperforming all baselines. We also demonstrate the importance of each design component through ablation studies.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
L. S. Keren, A. Liberzon, and T. Lazebnik, “A computational framework for physics-informed symbolic regression with straightforward integra- tion of domain knowledge,”Scientific Reports, vol. 13, no. 1, p. 1249, 2023
work page 2023
-
[2]
Deep learning and symbolic regression for discovering parametric equations,
M. Zhang, S. Kim, P. Y . Lu, and M. Solja ˇci´c, “Deep learning and symbolic regression for discovering parametric equations,”IEEE Trans- actions on Neural Networks and Learning Systems, 2023
work page 2023
-
[3]
W. Tenachi, R. Ibata, and F. I. Diakogiannis, “Deep symbolic regression for physics guided by units constraints: toward the automated discovery of physical laws,”The Astrophysical Journal, vol. 959, no. 2, p. 99, 2023
work page 2023
-
[4]
Data-driven discovery of partial differential equations,
S. H. Rudy, S. L. Brunton, J. L. Proctor, and J. N. Kutz, “Data-driven discovery of partial differential equations,”Science advances, vol. 3, no. 4, p. e1602614, 2017
work page 2017
-
[6]
Symbolic regression in materials science,
Y . Wang, N. Wagner, and J. M. Rondinelli, “Symbolic regression in materials science,”MRS communications, vol. 9, no. 3, pp. 793–805, 2019
work page 2019
-
[7]
Discovering symbolic models from deep learn- ing with inductive biases,
M. Cranmer, A. Sanchez Gonzalez, P. Battaglia, R. Xu, K. Cranmer, D. Spergel, and S. Ho, “Discovering symbolic models from deep learn- ing with inductive biases,”Advances in neural information processing systems, vol. 33, pp. 17 429–17 442, 2020
work page 2020
-
[8]
Ai feynman: A physics-inspired method for symbolic regression,
S.-M. Udrescu and M. Tegmark, “Ai feynman: A physics-inspired method for symbolic regression,”Science advances, vol. 6, no. 16, p. eaay2631, 2020
work page 2020
-
[9]
Contemporary symbolic regression methods and their relative performance,
W. La Cava, B. Burlacu, M. Virgolin, M. Kommenda, P. Orzechowski, F. O. de Franc ¸a, Y . Jin, and J. H. Moore, “Contemporary symbolic regression methods and their relative performance,”Advances in neural information processing systems, vol. 2021, no. DB1, p. 1, 2021
work page 2021
Show all 32 references
-
[10]
Distilling free-form natural laws from experimental data,
M. Schmidt and H. Lipson, “Distilling free-form natural laws from experimental data,”science, vol. 324, no. 5923, pp. 81–85, 2009
2009
-
[11]
Taylor genetic pro- gramming for symbolic regression,
B. He, Q. Lu, Q. Yang, J. Luo, and Z. Wang, “Taylor genetic pro- gramming for symbolic regression,” inProceedings of the genetic and evolutionary computation conference, 2022, pp. 946–954
2022
-
[12]
Interpretable machine learning for science with pysr and symbolicregression. jl,
M. Cranmer, “Interpretable machine learning for science with pysr and symbolicregression. jl,”arXiv preprint arXiv:2305.01582, 2023
2023 arXiv
-
[13]
Transformer- based planning for symbolic regression,
P. Shojaee, K. Meidani, A. Barati Farimani, and C. Reddy, “Transformer- based planning for symbolic regression,”Advances in Neural Informa- tion Processing Systems, vol. 36, pp. 45 907–45 919, 2023
2023
-
[14]
Sym- former: End-to-end symbolic regression using transformer-based archi- tecture,
M. Vastl, J. Kulh ´anek, J. Kubal ´ık, E. Derner, and R. Babu ˇska, “Sym- former: End-to-end symbolic regression using transformer-based archi- tecture,”IEEE Access, 2024
2024
-
[15]
Deep gener- ative symbolic regression with monte-carlo-tree-search,
P.-A. Kamienny, G. Lample, S. Lamprier, and M. Virgolin, “Deep gener- ative symbolic regression with monte-carlo-tree-search,” inInternational Conference on Machine Learning. PMLR, 2023, pp. 15 655–15 668
2023
-
[16]
Genetic programming as a means for programming comput- ers by natural selection,
J. R. Koza, “Genetic programming as a means for programming comput- ers by natural selection,”Statistics and computing, vol. 4, pp. 87–112, 1994
1994
-
[17]
On improving genetic programming for symbolic regression,
S. Gustafson, E. K. Burke, and N. Krasnogor, “On improving genetic programming for symbolic regression,” in2005 IEEE Congress on Evolutionary Computation, vol. 1. IEEE, 2005, pp. 912–919
2005
-
[18]
Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients,
B. K. Petersen, M. Landajuela, T. N. Mundhenk, C. P. Santiago, S. K. Kim, and J. T. Kim, “Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients,” arXiv preprint arXiv:1912.04871, 2019
1912 arXiv
-
[19]
A seq2seq approach to symbolic regression,
L. Biggio, T. Bendinelli, A. Lucchi, and G. Parascandolo, “A seq2seq approach to symbolic regression,”Learning Meets Combinatorial Algo- rithms at NeurIPS2020, 2020
2020
-
[20]
Gaussian processes for machine learning.,(mit press: Cambridge, ma),
C. Rasmussen and C. Williams, “Gaussian processes for machine learning.,(mit press: Cambridge, ma),” 2006
2006
-
[21]
Scientific machine learning through physics–informed neural networks: Where we are and what’s next,
S. Cuomo, V . S. Di Cola, F. Giampaolo, G. Rozza, M. Raissi, and F. Piccialli, “Scientific machine learning through physics–informed neural networks: Where we are and what’s next,”Journal of Scientific Computing, vol. 92, no. 3, p. 88, 2022
2022
-
[22]
Learning equations for extrap- olation and control,
S. Sahoo, C. Lampert, and G. Martius, “Learning equations for extrap- olation and control,” inInternational Conference on Machine Learning. Pmlr, 2018, pp. 4442–4450
2018
-
[23]
Integration of neural network-based symbolic regression in deep learning for scientific discovery,
S. Kim, P. Y . Lu, S. Mukherjee, M. Gilbert, L. Jing, V . ˇCeperi´c, and M. Solja ˇci´c, “Integration of neural network-based symbolic regression in deep learning for scientific discovery,”IEEE transactions on neural networks and learning systems, vol. 32, no. 9, pp. 4166–4177, 2020
2020
-
[24]
Discovering governing equations from data by sparse identification of nonlinear dynamical systems,
S. L. Brunton, J. L. Proctor, and J. N. Kutz, “Discovering governing equations from data by sparse identification of nonlinear dynamical systems,”Proceedings of the national academy of sciences, vol. 113, no. 15, pp. 3932–3937, 2016
2016
-
[25]
Efficient symbolic policy learning with differentiable symbolic expression,
J. Guo, R. Zhang, S. Peng, Q. Yi, X. Hu, R. Chen, Z. Du, L. Li, Q. Guo, Y . Chenet al., “Efficient symbolic policy learning with differentiable symbolic expression,”Advances in Neural Information Processing Systems, vol. 36, pp. 36 278–36 304, 2023
2023
-
[26]
Noise-resilient symbolic regression with dynamic gating reinforcement learning,
C. Sun, S. Shen, W. Tao, D. Xue, and Z. Zhou, “Noise-resilient symbolic regression with dynamic gating reinforcement learning,”Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 19, pp. 20 690–20 698, 2025
2025
-
[27]
A systematic dnn weight pruning framework using alternating direction method of multipliers,
T. Zhang, S. Ye, K. Zhang, J. Tang, W. Wen, M. Fardad, and Y . Wang, “A systematic dnn weight pruning framework using alternating direction method of multipliers,” inProceedings of the European conference on computer vision (ECCV), 2018, pp. 184–199
2018
-
[28]
Distributed optimization and statistical learning via the alternating direction method of multipliers,
S. Boyd, N. Parikh, E. Chu, B. Peleato, J. Ecksteinet al., “Distributed optimization and statistical learning via the alternating direction method of multipliers,”Foundations and Trends® in Machine learning, vol. 3, no. 1, pp. 1–122, 2011
2011
-
[29]
Pruning convolutional neural networks for resource efficient inference,
P. Molchanov, S. Tyree, T. Karras, T. Aila, and J. Kautz, “Pruning convolutional neural networks for resource efficient inference,”arXiv preprint arXiv:1611.06440, 2016
2016 arXiv
-
[30]
Optimal brain damage,
Y . LeCun, J. Denker, and S. Solla, “Optimal brain damage,”Advances in neural information processing systems, vol. 2, 1989
1989
-
[31]
Interactive symbolic regression with co-design mechanism through offline reinforcement learning,
Y . Tian, W. Zhou, M. Viscione, H. Dong, D. S. Kammer, and O. Fink, “Interactive symbolic regression with co-design mechanism through offline reinforcement learning,”Nature Communications, vol. 16, no. 1, p. 3930, 2025
2025
-
[32]
Enhancing sparsity by reweightedℓ 1 minimization,
E. J. Candes, M. B. Wakin, and S. P. Boyd, “Enhancing sparsity by reweightedℓ 1 minimization,”Journal of Fourier analysis and applica- tions, vol. 14, pp. 877–905, 2008
2008
-
[33]
Darts: Differentiable architecture search,
H. Liu, K. Simonyan, and Y . Yang, “Darts: Differentiable architecture search,”arXiv preprint arXiv:1806.09055, 2018
2018 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.