REVIEW 4 major objections 6 minor 1 cited by
Learning from Less: SINDy Surrogates in RL
T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read With as few as 75 environment transitions, the SINDy algorithm fits a surrogate RL environment whose state predictions correlate with the true physics engine at 0.997 or better, and agents trained inside that surrogate match…
desk verdict SINDy surrogates are a plausible idea, but this paper never tests whether policies trained in them actually perform in the original environments, so its headline efficiency claims are unsupported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The engine of the paper is the SINDy surrogate: a sparse regression model that selects a few terms from a library of candidate functions—polynomials, trigonometric terms, and rational functions—via Sequential Thresholded Least Squares, yielding an explicit transition rule $s_{t+1}=f_{SINDy}(s_t,a_t)$. The authors wrap that learned rule in the same RL environment interface as the original benchmarks so any policy algorithm can train against it in place of the physics engine. The fitting procedure that carries the argument is a four-step loop: initial STLSQ fit, residual-driven addition of nonlinear terms, grid search over sparsity threshold and regularization strength, and cross-validated selection by mean squared error. The paper's key modeling lesson is that the candidate library must be matched to the dynamics: trigonometric functions are needed for Mountain Car's oscillatory behavior, while polynomials suffice for Lunar Lander.
What would settle it
Take the same SINDy pipeline but collect the 75 Mountain Car transitions from a trajectory that never enters, or only barely enters, the goal region; if the fitted surrogate's held-out predictions over the full state space drop below 0.99 correlation, the single-episode coverage assumption is broken. Separately, take the surrogate-trained Lunar Lander policy and run it in the original environment: if its success rate is materially below a natively trained policy, the policy-transfer claim is falsified.
Extended reading notes
Core claim
The discovery the authors argue for is that sparse dynamics identification generalizes from a single trajectory to the full state-transition function well enough to support policy learning. Fitting SINDy to 75 Mountain Car transitions, collected under an $\epsilon$-greedy version of a pretrained Soft Actor-Critic policy, and to 1,000 Lunar Lander transitions, collected under a pretrained Proximal Policy Optimization agent, produced explicit maps $s_{t+1}=f_{SINDy}(s_t,a_t)$ whose held-out state predictions correlate with the true simulator at 0.997–1.000 for most variables; the reported worst case is 0.989 for Lunar Lander angular velocity, and the mean squared errors reach $3.11\times10^{-6}$ for Mountain Car velocity and $1.42\times10^{-6}$ for Lunar Lander position. Agents trained in the surrogates converged much like agents trained in the original environments, and the surrogate-trained runs needed 65,075 steps versus 100,000 for Mountain Car and 801,000 versus 1,000,000 for Lunar Lander.
Load-bearing premise
The whole argument rests on the assumption that the 75 Mountain Car or 1,000 Lunar Lander transitions, collected from a single episode under an $\epsilon$-greedy pretrained policy, are representative enough of the full state-action distribution that SINDy's sparse equation generalizes to every region a trained policy will visit.
Editorial extensions
If this is right
- Because surrogate rollouts come from an explicit formula rather than a physics engine, model-based RL becomes practical when interaction budgets are tiny, such as physical robots or safety-critical control tasks.
- Agents can be pre-trained in the surrogate and then fine-tuned in the original environment, since simulating the sparse equations is nearly free once the model is learned.
- The explicit governing equations let engineers audit what the agent is modeling, debug failure modes, and check safety properties in ways a neural-network surrogate does not easily allow.
- The library-selection result gives practitioners a usable heuristic: include trigonometric basis functions when the environment has oscillatory dynamics, and otherwise prefer low-order polynomial terms.
- The measured 20% (Lunar Lander) and 35% (Mountain Car) reductions in total training steps give a concrete expectation for the computational savings a sparse surrogate can deliver.
Reading between the lines
- The validity of the 75-transition fit is tied to the single episode used for collection; in a new task, one would need to verify that the sampled trajectory actually covers the state-action region the final policy visits, since SINDy only fits within the library functions it selects.
- A natural test the paper leaves implicit is to deploy the surrogate-trained policies directly in the original environments and measure return; the 20–35% step savings only translate to real systems if the surrogate-to-true gap stays small enough not to break the transferred policy.
- The data requirement could probably be pushed further by active collection: use the SINDy model's current prediction error to ask for the next transition where the model is most uncertain, rather than relying on one fixed exploratory episode.
- The reported worst-case correlation of 0.989 for Lunar Lander angular velocity suggests that aggregate correlation can hide a weak state; a stricter per-component floor, with library or threshold selection targeted at the weakest variable, would be a cheap robustness improvement.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes using Sparse Identification of Nonlinear Dynamics (SINDy) to learn surrogate models of two OpenAI Gym environments, Mountain Car and Lunar Lander, from very small datasets (75 and 1,000 transitions, respectively), and then training RL agents within these surrogates. It reports high state-prediction accuracy (correlations above 0.99) and claims that agents trained in the surrogates achieve comparable performance to agents trained in the original environments while using 20-35% fewer training steps. The manuscript also discusses the importance of the feature library (trigonometric vs. polynomial terms) and the interpretability benefits of SINDy.
Significance. If the claims were substantiated, the work would offer a data-efficient and interpretable alternative to neural-network surrogate models in model-based RL, with potential practical value. The paper's pipeline is clearly described and uses standard benchmarks, and the reported dynamics-model accuracy is promising. However, the central claim of 'comparable agent performance' is never quantitatively tested: the manuscript's main evidence consists of visual policy heatmaps and training-step counts. The paper also credits itself with a 'systematic evaluation framework' that is not actually present in the reported experiments. As a result, the significance of the contribution cannot be assessed from the current text, and the empirical core needs substantial strengthening.
major comments (4)
- [Section 3.1 and Abstract] The central claim that RL agents trained in the surrogate environments achieve comparable performance to agents trained in the original environments is not supported by any quantitative evaluation. The paper reports only per-variable MSE and correlation for the SINDy model (Table 1) and visual policy heatmaps (Figures 1 and 2); it does not report cumulative rewards, success rates, or final performance metrics for surrogate-trained policies evaluated in the original Mountain Car or Lunar Lander environments. Consequently, the abstract's assertions of 'comparable performance' and 'similar ... final performance metrics' are unverified. To support the central claim, the authors must add quantitative results, including return curves or success rates, for policies trained in the surrogate and evaluated in the original environment, with multiple seeds and error bars.
- [Section 2.1 and Section 4] The data collection strategy uses a single episode from a pre-trained epsilon-greedy policy (75 transitions for Mountain Car, 1,000 for Lunar Lander). This does not establish that the learned SINDy model is accurate on the state-action distribution visited by an RL agent trained in the surrogate, which will explore regions beyond the single training trajectory and may exploit model errors. The paper's own limitation statement in Section 4 ('generalization capability to significantly different initial conditions remains to be fully explored') acknowledges this gap. The authors should provide evidence of state-space coverage (e.g., visitation histograms or rollout error on states reached during surrogate training) or collect data from a more exploratory policy.
- [Section 3.2] The 'fewer total steps' comparison (65,075 vs. 100,000 for Mountain Car; 801,000 vs. 1,000,000 for Lunar Lander) is not a valid measure of computational cost reduction as claimed in the abstract ('reducing computational costs by 20-35%'). These numbers compare training steps in the surrogate versus the original environment, but they ignore the 75/1,000 real-environment interactions needed for data collection and the cost of training the SINDy model. Moreover, the paper does not report wall-clock time or any hardware-independent compute metric. The claim of reduced computational cost must be either redefined as 'fewer surrogate training steps' or supported by actual computational measurements.
- [Section 3.2] The comparison to a neural-network surrogate (MSE 3.11e-6 vs. 4.45e-6; approximately 95% less computational resources) is not reproducible because the neural network architecture, training procedure, and evaluation protocol are not described anywhere in the paper. Either provide complete experimental details for this baseline or remove the comparison and its associated efficiency claim.
minor comments (6)
- [Table 1] In the lower part of Table 1, the columns 'MC MSE' and 'LL MSE' do not specify which state variable (e.g., position or velocity) the mean squared error is computed over, which makes the library-function comparison difficult to interpret. Please clarify.
- [Figure 2] The subplot labels in Figure 2 are duplicated: both the top and bottom rows use labels (a), (b), (c). Rename the six panels as (a)-(f) to avoid confusion.
- [Algorithm 1] The pseudocode in Algorithm 1 loops over N episodes for data collection, but the experiments in Section 2.1 use a single episode (N=1). Please make the pseudocode consistent with the reported experiments or clarify that N=1 in the current study.
- [Introduction] The claim that 'This work represents the first application of SINDy to RL environment modeling' is contradicted by the cited references, e.g., Zolman et al. (2024) 'Sindy-rl: Interpretable and efficient model-based reinforcement learning' and Arora et al. (2022). Please either revise the novelty statement to accurately reflect prior work or clarify the specific distinction from these existing methods.
- [Abstract] The abstract states 'reducing computational costs by 20-35%,' but the only quantitative evidence in Section 3.2 is a reduction in the number of training steps, not a measure of computational cost. Please use terminology that matches the actual measurements (e.g., 'training steps').
- [Section 3.1] The term 'state-wise correlations exceeding 0.997' is not a standard accuracy metric for dynamics models; correlation can be high even with systematic bias. Please report residual statistics or ensure MSE values are provided with uncertainty estimates as well.
Circularity Check
Reported surrogate accuracy is the model-selection criterion rather than an independent prediction; policy-transfer claims are untested but not circular.
-
fitted input called prediction
[Section 2.2 'Cross-Validation' and Table 1 / Abstract]
"Cross-Validation: Models were validated using held-out data to ensure generalization performance, with final selection based on minimal MSE. ... we achieve state-wise correlations exceeding 0.997, with mean squared errors as low as 3.11e-06"
The paper reports held-out MSE and correlation values as evidence of SINDy fidelity, but these are exactly the validation metrics used for model selection ('final selection based on minimal MSE'). No separate test set is described, so the reported accuracy is the optimization objective itself: the chosen model minimizes the same loss that is then presented as an independent predictive result. With only 75 or 1000 transitions, this is a fitted input reported as a prediction, not an independent test of the surrogate's generalization.
full rationale
The SINDy modeling itself is not circular: the surrogate is fitted to genuine OpenAI Gym transition data and could, in principle, be validated against independent rollouts or by transferring trained policies back to the original environment. However, the only quantitative evidence of fidelity in the paper is the held-out MSE/correlation used during model selection, with no indication of a separate test set; thus the accuracy claim reduces to reporting the model-selection loss. The downstream RL comparison consists of policy visualizations and convergence descriptions, and no final-return table or original-environment evaluation of surrogate-trained policies is provided, so the 'comparable performance' and 'fewer total steps' claims are unsupported rather than circular. There is no load-bearing self-citation: the cited SINDy-in-RL works are by different authors and are used only for the surrogate-environment integration. Overall, the central accuracy metric is partially circular because it is the fitted selection objective, yielding a score of 6.
Assumptions & free parameters
free parameters (5)
- STLSQ threshold =
not reported
- STLSQ regularization coefficient =
not reported
- Feature library composition =
Mountain Car: polynomial + trigonometric; Lunar Lander: polynomial
- Data collection epsilon =
0.2
- Number of collected transitions =
75 (Mountain Car), 1,000 (Lunar Lander)
assumptions (4)
- domain assumption The SINDy feature library (chosen via residual analysis) contains the true terms of the environment dynamics.
- domain assumption The reward function from the original environment remains valid when the dynamics are replaced by the SINDy model.
- ad hoc to paper A single episode from a pre-trained agent with epsilon-greedy explores enough of the state space to identify dynamics.
- standard math STLSQ sparse regression converges to the true dynamics when data is noiseless and the library is complete.
Cite this review
Pith. "Pith review of Learning from Less: SINDy Surrogates in RL." pith.science (2026). https://pith.science/paper/A44E3PEK
@misc{pith2026250418113,
author = {Pith},
title = {Pith review of: Learning from Less: SINDy Surrogates in RL},
year = {2026},
howpublished = {\url{https://pith.science/paper/A44E3PEK}},
note = {Machine review of arXiv:2504.18113}
}
read the original abstract
This paper introduces an approach for developing surrogate environments in reinforcement learning (RL) using the Sparse Identification of Nonlinear Dynamics (SINDy) algorithm. We demonstrate the effectiveness of our approach through extensive experiments in OpenAI Gym environments, particularly Mountain Car and Lunar Lander. Our results show that SINDy-based surrogate models can accurately capture the underlying dynamics of these environments while reducing computational costs by 20-35%. With only 75 interactions for Mountain Car and 1000 for Lunar Lander, we achieve state-wise correlations exceeding 0.997, with mean squared errors as low as 3.11e-06 for Mountain Car velocity and 1.42e-06 for LunarLander position. RL agents trained in these surrogate environments require fewer total steps (65,075 vs. 100,000 for Mountain Car and 801,000 vs. 1,000,000 for Lunar Lander) while achieving comparable performance to those trained in the original environments, exhibiting similar convergence patterns and final performance metrics. This work contributes to the field of model-based RL by providing an efficient method for generating accurate, interpretable surrogate environments.
Figures
Forward citations
Cited by 1 Pith paper
-
Learning to Modulate, Not to Cycle: Soft Actor---Critic Recovers Inverter-Style Heat-Pump Control
With a per-start wear penalty in the reward, SAC learns a policy that never switches the compressor off (zero starts/day), while PPO cycles more than a baseline thermostat.
Reference graph
Works this paper leans on
-
[1]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
-
[2]
Model-Based Reinforcement Learning with SINDy
Rishabh Arora, Bruno Castro da Silva, and Eben Moss. Model-based reinforcement learning with sindy. arXiv preprint arXiv:2208.14501, 2022
work page Pith review arXiv 2022
-
[3]
Discovering governing equations from data by sparse identification of nonlinear dynamical systems
Steven L Brunton, Joshua L Proctor, and J Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the National Academy of Sciences, 113 0 (15): 0 3932--3937, 2016
work page 2016
-
[4]
Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. International Conference on Machine Learning, pp.\ 1861--1870, 2018
work page 2018
-
[6]
Reinforcement learning: An introduction
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018
2018
-
[7]
Nicholas Zolman, Urs Fasel, J. Nathan Kutz, and Steven L. Brunton. Sindy-rl: Interpretable and efficient model-based reinforcement learning. arXiv preprint arXiv:2403.09110, 2024
arXiv 2024
-
[8]
@esa (Ref
\@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...
-
[9]
\@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...
Show all 9 references
-
[10]
Nathan and Brunton, Steven L
@open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...
2018 arXiv
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.