{"id":"08c7b086-6f68-4a76-94a8-eda8de71f31a","arxiv_id":"2504.18113","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"SINDy-fitted surrogate environments for Mountain Car and Lunar Lander reproduce state dynamics from 75 to 1,000 samples and train RL agents with fewer steps, though policy transfer to the real environments is unquantified.","lead":"This paper replaces the physics of two OpenAI Gym tasks with sparse equations fitted from very few samples, then trains reinforcement learning agents inside those cheap surrogates. It claims comparable policies with 20-35% fewer training steps, but the transfer results are not directly measured.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Surrogate-trained policies are never evaluated in the original environments, so the headline claims of comparable performance and fewer total steps are unsupported; the single-episode data coverage assumption remains untested.","rationale":"The reader's weakest assumption is the representativeness of 75/1000 single-episode transitions; this is exactly the coverage assumption that a transfer evaluation would test. I agree with the reader's REJECT verdict because the paper's own reported evidence stops at model fidelity metrics on the training distribution and policy visualizations, not at measured performance of surrogate-trained policies in the real environments. The absence of quantitative transfer results is the single most load-bearing gap: even a perfect SINDy fit to one trajectory would not guarantee that an RL agent optimizing against the surrogate learns a policy that works in the original dynamics, because RL exploration pushes into states not represented in the data. The manuscript also states 'first application of SINDy to RL environment modeling' while citing Arora et al. (2022) and Zolman et al. (2024) on SINDy-based model-based RL; this weakens the novelty claim but is secondary to the missing transfer evidence. The proposed concrete test would settle the concern: if surrogate-trained policies achieve comparable returns when rolled out in the original environments, the central claim would be substantially supported; if not, the efficiency and fidelity claims reduce to curve-fitting on a single trajectory. I keep the reader's verdict unchanged rather than moving it because the reader already reached REJECT for essentially this reason.","tokens_in":4934,"tokens_out":3779,"duration_ms":40855,"concrete_test":"Retrain 10 SAC agents in the SINDy surrogate for Mountain Car with the same hyperparameters, then evaluate each in the original Gym MountainCar-v0 for 100 episodes; report mean and 95% CI of episode return and success rate against the original-environment-trained baseline. If the surrogate-trained agents do not reach comparable return and success, the headline efficiency claim fails. For Lunar Lander, repeat with PPO and 50 evaluation episodes per trained policy in the original LunarLander-v2.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Conclusion: 'SINDy can create highly effective surrogate environments for reinforcement learning while requiring remarkably little training data') requires two linked facts: (i) the SINDy model is accurate on the state-action distribution visited during RL training in the surrogate, and (ii) a policy trained in the surrogate transfers to the original environment with comparable performance. The paper establishes neither quantitatively. Section 3.1 reports only per-variable MSE/correlation of the SINDy model on held-out transitions and visual policy comparisons inside each environment; there is no rollout of surrogate-trained policies in the original Mountain Car or Lunar Lander environments, no final-return table, no success-rate comparison, and no error bars. Because data collection (Section 2.1) uses one epsilon-greedy episode (75/1000 transitions) from a pretrained agent, the fitted dynamics can be accurate along that trajectory while being arbitrary elsewhere; RL training in the surrogate will explore exactly those off-trajectory regions and may exploit model errors. Section 4 concedes that 'generalization capability to significantly different initial conditions remains to be fully explored,' which is an admission that this coverage requirement is untested. Until surrogate-trained policies are evaluated in the real environment, the 'comparable performance' and 'fewer total steps' claims in the abstract are not supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes using Sparse Identification of Nonlinear Dynamics (SINDy) to learn surrogate models of two OpenAI Gym environments, Mountain Car and Lunar Lander, from very small datasets (75 and 1,000 transitions, respectively), and then training RL agents within these surrogates. It reports high state-prediction accuracy (correlations above 0.99) and claims that agents trained in the surrogates achieve comparable performance to agents trained in the original environments while using 20-35% fewer training steps. The manuscript also discusses the importance of the feature library (trigonometric vs. polynomial terms) and the interpretability benefits of SINDy.","tokens_in":5217,"tokens_out":9699,"duration_ms":89011,"significance":"If the claims were substantiated, the work would offer a data-efficient and interpretable alternative to neural-network surrogate models in model-based RL, with potential practical value. The paper's pipeline is clearly described and uses standard benchmarks, and the reported dynamics-model accuracy is promising. However, the central claim of 'comparable agent performance' is never quantitatively tested: the manuscript's main evidence consists of visual policy heatmaps and training-step counts. The paper also credits itself with a 'systematic evaluation framework' that is not actually present in the reported experiments. As a result, the significance of the contribution cannot be assessed from the current text, and the empirical core needs substantial strengthening.","major_comments":[{"comment":"The central claim that RL agents trained in the surrogate environments achieve comparable performance to agents trained in the original environments is not supported by any quantitative evaluation. The paper reports only per-variable MSE and correlation for the SINDy model (Table 1) and visual policy heatmaps (Figures 1 and 2); it does not report cumulative rewards, success rates, or final performance metrics for surrogate-trained policies evaluated in the original Mountain Car or Lunar Lander environments. Consequently, the abstract's assertions of 'comparable performance' and 'similar ... final performance metrics' are unverified. To support the central claim, the authors must add quantitative results, including return curves or success rates, for policies trained in the surrogate and evaluated in the original environment, with multiple seeds and error bars.","section":"Section 3.1 and Abstract"},{"comment":"The data collection strategy uses a single episode from a pre-trained epsilon-greedy policy (75 transitions for Mountain Car, 1,000 for Lunar Lander). This does not establish that the learned SINDy model is accurate on the state-action distribution visited by an RL agent trained in the surrogate, which will explore regions beyond the single training trajectory and may exploit model errors. The paper's own limitation statement in Section 4 ('generalization capability to significantly different initial conditions remains to be fully explored') acknowledges this gap. The authors should provide evidence of state-space coverage (e.g., visitation histograms or rollout error on states reached during surrogate training) or collect data from a more exploratory policy.","section":"Section 2.1 and Section 4"},{"comment":"The 'fewer total steps' comparison (65,075 vs. 100,000 for Mountain Car; 801,000 vs. 1,000,000 for Lunar Lander) is not a valid measure of computational cost reduction as claimed in the abstract ('reducing computational costs by 20-35%'). These numbers compare training steps in the surrogate versus the original environment, but they ignore the 75/1,000 real-environment interactions needed for data collection and the cost of training the SINDy model. Moreover, the paper does not report wall-clock time or any hardware-independent compute metric. The claim of reduced computational cost must be either redefined as 'fewer surrogate training steps' or supported by actual computational measurements.","section":"Section 3.2"},{"comment":"The comparison to a neural-network surrogate (MSE 3.11e-6 vs. 4.45e-6; approximately 95% less computational resources) is not reproducible because the neural network architecture, training procedure, and evaluation protocol are not described anywhere in the paper. Either provide complete experimental details for this baseline or remove the comparison and its associated efficiency claim.","section":"Section 3.2"}],"minor_comments":[{"comment":"In the lower part of Table 1, the columns 'MC MSE' and 'LL MSE' do not specify which state variable (e.g., position or velocity) the mean squared error is computed over, which makes the library-function comparison difficult to interpret. Please clarify.","section":"Table 1"},{"comment":"The subplot labels in Figure 2 are duplicated: both the top and bottom rows use labels (a), (b), (c). Rename the six panels as (a)-(f) to avoid confusion.","section":"Figure 2"},{"comment":"The pseudocode in Algorithm 1 loops over N episodes for data collection, but the experiments in Section 2.1 use a single episode (N=1). Please make the pseudocode consistent with the reported experiments or clarify that N=1 in the current study.","section":"Algorithm 1"},{"comment":"The claim that 'This work represents the first application of SINDy to RL environment modeling' is contradicted by the cited references, e.g., Zolman et al. (2024) 'Sindy-rl: Interpretable and efficient model-based reinforcement learning' and Arora et al. (2022). Please either revise the novelty statement to accurately reflect prior work or clarify the specific distinction from these existing methods.","section":"Introduction"},{"comment":"The abstract states 'reducing computational costs by 20-35%,' but the only quantitative evidence in Section 3.2 is a reduction in the number of training steps, not a measure of computational cost. Please use terminology that matches the actual measurements (e.g., 'training steps').","section":"Abstract"},{"comment":"The term 'state-wise correlations exceeding 0.997' is not a standard accuracy metric for dynamics models; correlation can be high even with systematic bias. Please report residual statistics or ensure MSE values are provided with uncertainty estimates as well.","section":"Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a workshop paper and the central claim is currently unsupported. I considered reject, but the idea is plausible and the missing quantitative evaluation is a clear, addressable gap. I recommend major revision rather than reject, with the expectation that the authors provide empirical evidence of transfer performance and address the data coverage issue. If they cannot provide such evidence, the paper should be rejected."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short take: the paper's headline claim — that SINDy surrogates give comparable RL policies from 75–1000 interactions — is not backed by the evidence it presents. There is no transfer evaluation of surrogate-trained policies back into the original environments. All we get is fit quality on held-out transitions and visual policy plots.\n\nWhat's genuinely useful: the paper shows that with a well-chosen feature library (trigonometric for Mountain Car, polynomial for Lunar Lander), SINDy can fit the dynamics of these two gym environments from a single episode of a pretrained policy. The MSE values (10^-6) are believable, and the comparison of library functions is a nice empirical detail. The interpretable equations are a real plus over neural surrogates.\n\nSoft spots: first, the central claim of comparable performance is unsupported. Section 3.1 has no return table, no success-rate comparison, no error bars for policies trained in the surrogate and evaluated in the original environment. The paper only compares policy maps inside each environment. The abstract promises 'comparable performance' but the evidence is visual. That's a load-bearing gap. Second, the novelty claim is wrong: the paper cites Arora et al. (2022) and Zolman et al. (2024), both doing SINDy-based RL surrogates, then claims 'first application' in the introduction. That contradiction suggests carelessness at best. Third, the data collection is one epsilon-greedy episode from a pretrained agent; the fitted dynamics are only validated on held-out transitions from that same trajectory. RL training in the surrogate will explore off-trajectory states, and the paper does not show the model is accurate there. Section 4 concedes exactly this: 'generalization capability to significantly different initial conditions remains to be fully explored.' So the efficiency claim is real only if the surrogate dynamics are reliable for exploration, which is untested.\n\nWho is this for? Researchers working on model-based RL who want a concrete example of SINDy applied to simple benchmarks. The paper's value is as a proof-of-concept, not a validated method. It deserves a serious referee: the question is interesting and the missing transfer evaluation is fixable with a straightforward experiment — return and success-rate tables for surrogate-trained policies in the original environments, with error bars. But in current form it should not be accepted.\n\nI'd send it to review, but I would not accept it without those numbers.","headline":"SINDy surrogates are a plausible idea, but this paper never tests whether policies trained in them actually perform in the original environments, so its headline efficiency claims are unsupported.","tokens_in":5748,"tokens_out":4208,"would_cite":false,"duration_ms":38268,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"With as few as 75 environment transitions, the SINDy algorithm fits a surrogate RL environment whose state predictions correlate with the true physics engine at 0.997 or better, and agents trained inside that surrogate match…","keywords":["model-based reinforcement learning","SINDy","sparse identification of nonlinear dynamics","surrogate environment","sample-efficient RL","Mountain Car","Lunar Lander","interpretable dynamics"],"falsifier":"Take the same SINDy pipeline but collect the 75 Mountain Car transitions from a trajectory that never enters, or only barely enters, the goal region; if the fitted surrogate's held-out predictions over the full state space drop below 0.99 correlation, the single-episode coverage assumption is broken. Separately, take the surrogate-trained Lunar Lander policy and run it in the original environment: if its success rate is materially below a natively trained policy, the policy-transfer claim is falsified.","tokens_in":4756,"feed_emoji":"🎮","tokens_out":12915,"duration_ms":117119,"temperature":0.7,"pith_summary":"The paper's central claim is that SINDy, the sparse-identification method for nonlinear dynamics, can build a surrogate reinforcement-learning environment from a tiny number of recorded transitions and still reproduce the original dynamics faithfully. Specifically, it reports that 75 transitions from one episode suffice for Mountain Car and 1,000 for Lunar Lander, giving state-wise next-state correlations between 0.99 and 1.00 and mean squared errors down to $3.11\\times10^{-6}$. If that holds, agents can be trained mostly inside a cheap analytical model rather than a physics engine, cutting total training steps by 20–35% while yielding comparable final policies and leaving the engineer explicit equations to inspect.","feed_headline":"75 interactions produce a 0.997-faithful RL surrogate","feed_subtitle":"SINDy turns one Mountain Car episode into a fast surrogate that cuts training steps by a third.","key_machinery":"The engine of the paper is the SINDy surrogate: a sparse regression model that selects a few terms from a library of candidate functions—polynomials, trigonometric terms, and rational functions—via Sequential Thresholded Least Squares, yielding an explicit transition rule $s_{t+1}=f_{SINDy}(s_t,a_t)$. The authors wrap that learned rule in the same RL environment interface as the original benchmarks so any policy algorithm can train against it in place of the physics engine. The fitting procedure that carries the argument is a four-step loop: initial STLSQ fit, residual-driven addition of nonlinear terms, grid search over sparsity threshold and regularization strength, and cross-validated selection by mean squared error. The paper's key modeling lesson is that the candidate library must be matched to the dynamics: trigonometric functions are needed for Mountain Car's oscillatory behavior, while polynomials suffice for Lunar Lander.","core_discovery":"The discovery the authors argue for is that sparse dynamics identification generalizes from a single trajectory to the full state-transition function well enough to support policy learning. Fitting SINDy to 75 Mountain Car transitions, collected under an $\\epsilon$-greedy version of a pretrained Soft Actor-Critic policy, and to 1,000 Lunar Lander transitions, collected under a pretrained Proximal Policy Optimization agent, produced explicit maps $s_{t+1}=f_{SINDy}(s_t,a_t)$ whose held-out state predictions correlate with the true simulator at 0.997–1.000 for most variables; the reported worst case is 0.989 for Lunar Lander angular velocity, and the mean squared errors reach $3.11\\times10^{-6}$ for Mountain Car velocity and $1.42\\times10^{-6}$ for Lunar Lander position. Agents trained in the surrogates converged much like agents trained in the original environments, and the surrogate-trained runs needed 65,075 steps versus 100,000 for Mountain Car and 801,000 versus 1,000,000 for Lunar Lander.","pith_inferences":["The validity of the 75-transition fit is tied to the single episode used for collection; in a new task, one would need to verify that the sampled trajectory actually covers the state-action region the final policy visits, since SINDy only fits within the library functions it selects.","A natural test the paper leaves implicit is to deploy the surrogate-trained policies directly in the original environments and measure return; the 20–35% step savings only translate to real systems if the surrogate-to-true gap stays small enough not to break the transferred policy.","The data requirement could probably be pushed further by active collection: use the SINDy model's current prediction error to ask for the next transition where the model is most uncertain, rather than relying on one fixed exploratory episode.","The reported worst-case correlation of 0.989 for Lunar Lander angular velocity suggests that aggregate correlation can hide a weak state; a stricter per-component floor, with library or threshold selection targeted at the weakest variable, would be a cheap robustness improvement."],"forward_implications":["Because surrogate rollouts come from an explicit formula rather than a physics engine, model-based RL becomes practical when interaction budgets are tiny, such as physical robots or safety-critical control tasks.","Agents can be pre-trained in the surrogate and then fine-tuned in the original environment, since simulating the sparse equations is nearly free once the model is learned.","The explicit governing equations let engineers audit what the agent is modeling, debug failure modes, and check safety properties in ways a neural-network surrogate does not easily allow.","The library-selection result gives practitioners a usable heuristic: include trigonometric basis functions when the environment has oscillatory dynamics, and otherwise prefer low-order polynomial terms.","The measured 20% (Lunar Lander) and 35% (Mountain Car) reductions in total training steps give a concrete expectation for the computational savings a sparse surrogate can deliver."],"supporting_citations":[{"why":"introduces SINDy, the sparse-identification algorithm whose library regression and thresholding the surrogate pipeline is built on.","marker":"Brunton et al., 2016"},{"why":"provides Soft Actor-Critic, the pretrained agent used to collect the 75 Mountain Car transitions.","marker":"Haarnoja et al., 2018"},{"why":"provides Proximal Policy Optimization, the pretrained agent used to collect the 1,000 Lunar Lander transitions.","marker":"Schulman et al., 2017"},{"why":"prior work that combines model-based RL with SINDy and supplies the surrogate-environment integration pattern this paper adapts.","marker":"Arora et al., 2022"},{"why":"SINDy-RL, the closest existing framework for SINDy-driven reinforcement learning, which the paper extends to very sparse data regimes.","marker":"Zolman et al., 2024"}],"fun_headline_variants":["75 steps to a 0.997-faithful RL surrogate","SINDy builds RL worlds from one roll-out","Surrogates from sparse data cut RL steps 35%","One episode, high-fidelity RL environment model","Sparse identification yields cheap RL training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole argument rests on the assumption that the 75 Mountain Car or 1,000 Lunar Lander transitions, collected from a single episode under an $\\epsilon$-greedy pretrained policy, are representative enough of the full state-action distribution that SINDy's sparse equation generalizes to every region a trained policy will visit.","fun_headline_variants_meta":{"raw":{"variants":["75 steps to a 0.997-faithful RL surrogate","SINDy builds RL worlds from one roll-out","Surrogates from sparse data cut RL steps 35%","One episode, high-fidelity RL environment model","Sparse identification yields cheap RL training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000215,"raw_usage":{"total_tokens":1442,"prompt_tokens":973,"completion_tokens":469,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":392}},"tokens_in":589,"tokens_out":469,"duration_ms":5522,"temperature":1.0,"reasoning_tokens":392,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:23:58.807425+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same SINDy pipeline but collect the 75 Mountain Car transitions from a trajectory that never enters, or only barely enters, the goal region; if the fitted surrogate's held-out predictions over the full state space drop below 0.99 correlation, the single-episode coverage assumption is broken. Separately, take the surrogate-trained Lunar Lander policy and run it in the original environment: if its success rate is materially below a natively trained policy, the policy-transfer claim is falsified.","supporting_citations":[{"cited_title":"Discovering governing equations from data by sparse identification of nonlinear dynamical systems","cited_arxiv_id":null,"evidence_quote":"introduces SINDy, the sparse-identification algorithm whose library regression and thresholding the surrogate pipeline is built on."},{"cited_title":"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor","cited_arxiv_id":null,"evidence_quote":"provides Soft Actor-Critic, the pretrained agent used to collect the 75 Mountain Car transitions."},{"cited_title":"Model-Based Reinforcement Learning with SINDy","cited_arxiv_id":"2208.14501","evidence_quote":"prior work that combines model-based RL with SINDy and supplies the surrogate-environment integration pattern this paper adapts."}],"review_version":1}