{"id":"42553e2c-67b6-4211-bd42-fc2fd957d058","arxiv_id":"2509.03036","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Adding an LLM-based physical plausibility score to the loss function of symbolic regression improves reconstruction of three textbook physics equations from noisy synthetic data.","lead":"This paper tests using a large language model as an automatic physics referee during symbolic regression, scoring each candidate equation for physical sense and feeding that score into the search. The authors report that the LLM-guided search finds cleaner and more accurate equations for three simple physics problems than ordinary symbolic regression.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The LLM scorer is never validated and the loss weights are unreported, so the reported gains may be a complexity-regularization artifact rather than evidence of physics knowledge integration.","rationale":"The paper proposes a plausible and cheap mechanism: add a pre-trained LLM's zero-shot judgment as a third term in the symbolic regression loss. The central claim is that this consistently improves reconstruction of physical dynamics and robustness to noise. For that claim to hold, the LLM score must carry information about physical validity that is not already captured by data-fit and complexity. The weakest point is precisely the attribution of the improvement to that information. The reader identified the same load-bearing assumption, and the paper's own limitation statement concedes the scores were never evaluated by human experts. My stress-test sharpens this into a concrete, testable attribution problem: because the LLM score includes a simplicity component and because the loss weights are unreported, the method is not currently distinguishable from a standard complexity-regularized SR run with a larger penalty on expression-tree size. The prompt-sensitivity experiment adds a separate concern: it treats perfect recovery in settings where the ground-truth equation is given to the LLM as success, which is not diagnostic of physics reasoning. I credit the paper for honest limitations, a released code repository, and a systematic 3x3x3 sweep; those are real strengths. However, no machine-checked proof, no repeated-run statistics, and no baseline-comparison rows in the noise table accompany the headline claim. The correct verdict remains CONDITIONAL, matching the reader's assessment: the idea is worth pursuing, but acceptance should require the ablation and weight-reporting checks described above. I therefore recommend no change to the reader's verdict.","tokens_in":22676,"tokens_out":3093,"duration_ms":30732,"concrete_test":"Run the full 3x3x3 benchmarking with the LLM scorer replaced by a deterministic null scorer c' = 1 - normalized_node_count (or a fixed-seed random score in [0,1]), holding all other settings and the unreported weights identical. If the null scorer reproduces the Table 2 improvements over no-LLM baselines, then the gains are attributable to complexity regularization rather than LLM physics knowledge. Additionally, report the values of w1, w2, w3 used in every run and repeat each condition over at least 10 independent seeds with error bars to establish whether the effect is robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the LLM score c in Eq. (1) measures physical validity, so the added term channels genuine physics knowledge into the symbolic regression search. That requirement is not established. The loss weights w1, w2, w3 are never reported, and c itself contains a simplicity component: c := 1 - (c1 + c2 + c3)/3, where c2 is explicitly 'simp, 0 (complex) -> 1 (simple)'. Consequently L = w1*e + (w2 + w3/3)*s + w3/3*(1 - c1 - c3). If the LLM's c1 and c3 scores are weakly informative or collinear with tree size, the observed improvements in Table 2 are indistinguishable from merely increasing the complexity penalty. Section 6 concedes exactly this: 'it is challenging to gauge whether performance gains were due to genuinely helpful scores in terms of physical realism or the score captures more technical properties, such as the equation's length.' No human validation of the scores, no correlation analysis with physical correctness, and no ablation separating c from s appear anywhere in the paper. The prompt-sensitivity results do not rescue the claim: prompts D, F, G, and H insert the ground-truth equation into the prompt, so the perfect expression-tree scores there are expected and cannot serve as evidence of physics-aware reasoning. Table 4's robustness claim is also unsubstantiated because no LLM-free baseline rows are shown, even though the text states the comparison is to 'the same model without the LLM'. The central claim may be true, but the current experiments do not rule out the simpler explanation that the LLM term acts only as a complexity regularizer.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes integrating pre-trained large language models into symbolic regression by adding an LLM-based score term to the SR loss function. The method is evaluated on three physical dynamics (free fall, simple harmonic motion, damped wave), three SR algorithms (DEAP, gplearn, PySR), and three LLMs (Mistral, Llama 2, Falcon). The authors report consistent improvements in MAE, MSE, R2, and expression-tree score, investigate eight prompt variants, and test robustness to 1%–5% Gaussian noise. The central claim is that a generic LLM prompt can serve as a reusable physics prior that improves reconstruction and robustness of SR models.","tokens_in":23051,"tokens_out":2921,"duration_ms":27439,"significance":"If the central claim is validated, the proposed approach would be a practically useful and relatively accessible way to inject domain knowledge into symbolic regression, since it avoids handcrafted constraints and specialized formulations. The paper has several strengths: the code and data are publicly released, the LLM sampling temperature is set to zero for reproducibility, the evaluation covers a 3x3x3 grid of components, and the expression-tree score provides a structural similarity metric in addition to numerical fit. However, the current evidence does not yet establish that the reported gains come from physics knowledge encoded in the LLM scores rather than from an implicit complexity penalty or from other confounds. The authors themselves acknowledge in Section 6 that the LLM score was not validated by human experts and that it is unclear whether gains reflect physical realism or technical properties such as equation length. Because this point is load-bearing for the paper's main claim, the manuscript needs additional experiments and reporting before it can be accepted.","major_comments":[{"comment":"The central claim requires that the LLM score c in Eq. (1) measures physical validity, but the reported definition c := 1 - (c1 + c2 + c3)/3 includes c2, which is explicitly a simplicity score. Expanding Eq. (1) gives L = w1*e + (w2 + w3/3)*s + w3/3*(1 - c1 - c3), so increasing w3 is partly just increasing the complexity penalty. Since the loss weights w1, w2, w3 are never reported, a reader cannot distinguish the proposed knowledge-integration mechanism from a conventional complexity regularizer. The manuscript's own Section 6 limitation concedes this exact concern. The authors should report the weights, provide an ablation that removes the simplicity component (or uses only c1 and c3), and compare against a non-LLM complexity-only baseline with matched penalty strength.","section":"Eq. (1), Section 3, Section 6"},{"comment":"The noise robustness analysis does not support the stated claim. Section 6 says that 'the LLM-integrated SR model improves the robustness to high levels of all three noise types compared to the same model without the LLM (baseline),' but Table 4 contains only DEAP, gplearn, and PySR rows and no LLM-free baseline rows. Without those baseline rows, the table shows only that PySR degrades more gracefully than DEAP and gplearn under noise, which is a property of the SR algorithms, not of LLM integration. The authors should add the corresponding no-LLM baseline results for all three SR models under the same noise conditions.","section":"Table 4, Section 4.2.3, Section 6"},{"comment":"Every entry in Table 2 appears to be a single run of a stochastic symbolic regression algorithm. Since DEAP, gplearn, and PySR all have randomized search components, the reported uniform improvements could be within run-to-run variability. The authors should report multiple independent seeds with means and standard deviations, or otherwise provide a statistical comparison (e.g., paired tests) between the LLM-integrated and baseline configurations.","section":"Table 2, Section 4.2.1"},{"comment":"The prompt-sensitivity experiment contains a circular evaluation. Prompts D, F, G, and H append the ground-truth equation to the LLM prompt, so the resulting expression-tree scores of 1.00 and 'perfect reconstruction' are expected consequences of providing the answer, not evidence that the LLM contributes physics-aware reasoning. These rows cannot be used to support the claim that richer prompts improve knowledge integration. The paper should either exclude the GT-containing prompts from the main evidence or reframe them explicitly as an upper-bound check rather than as a demonstration of the method.","section":"Table 3, Section 4.2.2"},{"comment":"In Table 3, the rows for prompts E, F, G, and H are identical within each LLM-SR pair (same MAE, MSE, R2, and tree score). This is surprising because these prompts differ in content, and it is unclear whether this reflects a copy/paste error, a reporting artifact, or a genuine insensitivity of the pipeline to all prompt differences once variable descriptions and experiment descriptions are present. The authors should clarify this in the text and verify the reported values against the actual experimental logs.","section":"Table 3 (rows E, F, G, H), Section 4.2.2"}],"minor_comments":[{"comment":"The sentence 'the three physical cases are: free fall of a ball, simple harmonic motion, and damped wave—were' contains a grammatical error and should be rewritten.","section":"Section 4.1.1"},{"comment":"There are typographical errors: 'locally-depolyable' in Section 4.1.2, 'comparision' in Section 4, and 'DEAP, Falcom' in Section 6 should be corrected.","section":"Section 4.1.2, Section 6"},{"comment":"The hyperparameter table appears inconsistent with the text: Section 4.1.3 states DEAP uses crossover probability 0.6 and mutation probability 0.05, but Table 1 lists Crossover probability '—' for DEAP and Mutation probability 0.01. Please align the table with the text or explain the discrepancy.","section":"Table 1"},{"comment":"The expression-tree distance definition includes a coefficient alpha in [0,1] for differing numeric leaf values, but the value of alpha used in the experiments is never reported. Please state the value or explain how it was chosen.","section":"Section 4.3"},{"comment":"References [113] and [114] appear to refer to the same work (Sharlin and Josephson, 'In Context Learning and Reasoning for Symbolic Regression with Large Language Models') and should be merged or distinguished.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and interesting problem, and the open code/data release is commendable. However, the main evidential gap—lack of validation that the LLM score measures physical validity rather than complexity—is acknowledged in the manuscript itself, and the noise-robustness table is missing the baseline that the text claims to compare against. These issues are fixable with additional experiments and reporting, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the one-line take: the core idea—using an LLM as a scoring term inside the SR loss—is new and worth a serious look, but the experiments as reported do not yet separate physics knowledge from added complexity regularization.\n\nWhat's actually new: prior LLM-SR systems use the LLM to propose or refine equations outside the search loop. This paper moves the LLM inside the loss as an evaluator, which is a real architectural change. That is a genuine contribution. The authors also run a fairly broad grid: three SR algorithms, three 7B LLMs, three dynamics, with code and data released. The prompt design is transparent and deterministic sampling is a good call. Credit where due: the main benchmark table shows uniform improvement, which is encouraging.\n\nWhere it's soft: the load-bearing claim is that the LLM scores physical validity. That is never validated. As the authors concede in Section 6, the gains could come from the simplicity component in c. Because c includes c2 (simplicity) and the loss weights w1-w3 are never reported, the observed gains are indistinguishable from a heavier complexity penalty. That is the central soft spot, and it is not a small one. The noise-robustness table is also missing the no-LLM baseline that the text claims to compare against, and every number in the main table appears to be a single run of stochastic SR with no variance. Those are fixable but they are exactly the details a referee would need.\n\nThe prompt-sensitivity analysis has a circular element: prompts D, F, G, H include the ground-truth equation, so a tree score of 1.00 there is expected. The main benchmark uses Prompt A without context, so the circularity does not kill the main claim—just don't count those rows as evidence.\n\nMinor stuff: a few placeholder arXiv IDs, at least one typo in the discussion where 'Falcom' appears, and the expression-tree section says non-commutative operators 'e.g., \"*\"' but multiplication is commutative.\n\nBottom line: the paper is a legitimate new mechanism with an honest limitations section, but the empirical support is not yet airtight. Who is this for? Anyone working on LLM-guided symbolic regression or physics-informed discovery. It deserves a serious referee; the central question is whether the LLM term genuinely adds physics knowledge or just extra regularization, and that can be resolved with validation of the scores and an ablation separating c from s.","headline":"LLM-as-scorer inside the SR loss is a genuinely new mechanism, but the experiments don't yet rule out the simpler story that it is just another complexity penalty.","tokens_in":23552,"tokens_out":1934,"would_cite":false,"duration_ms":17820,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A pre-trained LLM's equation scores, added to the symbolic-regression loss, consistently improve recovery of physical dynamics from noisy data, with richer prompts helping more.","keywords":["symbolic regression","physics-informed symbolic regression","large language models","knowledge integration","loss function","expression tree score","noise robustness","prompt engineering"],"falsifier":"Give the same three tasks to the method with the LLM term replaced by a pure equation-length penalty, or compare the LLM's scores to expert physicists' ratings of the same candidate equations; if either control reproduces or fails to correlate with the reported gains, the physical-knowledge claim is undermined.","tokens_in":22400,"feed_emoji":"📐","tokens_out":9022,"duration_ms":75467,"temperature":0.7,"pith_summary":"Symbolic regression finds equations from data, but data alone often leads to fits that are numerically good and physically wrong. This paper tries to make the search physically aware without hand-crafted constraints by asking a pre-trained large language model to grade each candidate equation on dimensional correctness, simplicity, and realism, and adding that grade to the regression loss. Across three genetic-programming symbolic-regression algorithms, three language models, and three physical dynamics (free fall, harmonic motion, damped wave), the LLM term consistently lowered error and improved structural recovery of the true equations relative to the same algorithms without the LLM. The paper also reports that the method degrades more gracefully than baselines as Gaussian noise grows, and that prompts carrying more experimental context produce better and more exact reconstructions. If this result holds, a generic language model can act as a reusable physics prior for equation discovery, replacing hand-built constraints.","feed_headline":"LLM scores in the loss recover physical laws from noisy data","feed_subtitle":"Adding a language-model judgment to equation search improves recovery across three dynamics and three regression engines.","key_machinery":"The load-bearing object is the LLM score term $c$ and the prompt that produces it. The prompt fixes a role for the model as a scientific-reasoning assistant, defines the three metrics with numerical scales, supplies three few-shot examples, and demands a Python-style list output; the sampling temperature is set to zero so the score is deterministic. This converts an open-ended language model into a repeatable evaluator whose score is combined with data fit and equation size in the symbolic-regression loss, steering the genetic search toward equations that are simultaneously accurate, compact, and physically plausible.","core_discovery":"On the paper's own terms, the central discovery is that a pre-trained LLM can serve as a physics-informed evaluator inside the symbolic-regression loop. The proposed loss is $L=w_1 e + w_2 s + w_3 c$, where $e$ is the mean squared error between predicted and observed values, $s$ is the candidate equation's size in expression-tree nodes, and $c$ is the LLM's penalty score. The LLM is prompted to return three sub-scores, dimensional correctness, simplicity, and realism, on $[0,1]$ scales, and $c$ is defined as $1-(c_1+c_2+c_3)/3$. The paper argues that this single term injects domain knowledge that is normally encoded by specialized formulations or manual feature engineering, and presents experiments in which every LLM-integrated configuration outperforms its no-LLM baseline on fitting error and expression-tree similarity, with the best configurations recovering the ground-truth equation exactly.","pith_inferences":["A decisive control experiment would replace the LLM with an automatic penalty that only rewards shorter equations; if the gains persist, the reported improvements come from complexity regularization rather than physical knowledge.","The perfect structural scores under prompts that reveal the ground-truth formula are expected if the LLM simply echoes the formula back; the more interesting test is whether the gain survives when the LLM is not shown the answer.","If the mechanism is genuine physics knowledge, the benefit should transfer to dynamics underrepresented in the LLM's training corpus; a hold-out test on an unfamiliar physics domain would separate memorization from reasoning.","Because the LLM score is sampled at zero temperature, score stability across stochastic samples is untested; sampling at higher temperatures could reveal how reliably the LLM ranks competing equations."],"forward_implications":["Every tested LLM-integrated configuration outperformed the same symbolic-regression algorithm without the LLM, so the reported improvement does not depend on a single regression engine.","Prompts that supplied variable descriptions, experiment context, or the ground-truth formula improved performance monotonically, and combining all three cues made the recovered expression tree structurally identical to the ground truth in every engine-model pair.","Adding the LLM term raises the expression-tree score at every tested noise level from 1% to 5%, with the largest gains visible when both input features and target values are corrupted.","The method's sensitivity to noise still grows with the complexity of the true equation, meaning the LLM prior reduces but does not eliminate the difficulty of hard reconstruction problems.","Because the LLM term is prompt-driven, it can be applied to new physical domains by changing the context string, with no change to the symbolic-regression engine."],"supporting_citations":[{"why":"Supplies the physics-informed symbolic-regression framework and the domain-knowledge integration approach that the paper's loss-term method builds on.","marker":"[43]"},{"why":"Provides the noise-robustness experimental design: Gaussian noise added separately to input features, target values, or both at 1% to 5% levels.","marker":"[136]"},{"why":"Establishes an in-context symbolic-regression baseline in which an LLM proposes candidate equations from previously tested formulas and fitness scores; the paper contrasts this with putting the LLM inside the loss.","marker":"[112]"},{"why":"Prior work using an LLM to propose expressions that are optimized and re-fed to the LLM, representing the pipeline approach the paper positions itself against.","marker":"[114]"},{"why":"Another LLM-driven generate-and-search pipeline that treats the LLM as a controller before or after the symbolic-regression search.","marker":"[115]"},{"why":"Reference for one of the three genetic-programming symbolic-regression engines used as a baseline in the benchmarking experiments.","marker":"[88]"},{"why":"Cited as the reference for the DEAP evolutionary computation library used as one of the symbolic-regression baselines.","marker":"[84]"},{"why":"Reference for the gplearn genetic-programming library used as another symbolic-regression baseline.","marker":"[80]"}],"fun_headline_variants":["LLM judgment in loss improves physics equation recovery","Language model scoring sharpens symbolic regression","LLM penalty term guides equation search to true laws","Pre-trained LLMs boost symbolic regression on noisy data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the LLM's zero-shot scores for dimensional consistency, simplicity, and realism measure physical validity and not surface features like equation length or phrasing familiarity, because the loss term can only help if the score rewards physics rather than style.","fun_headline_variants_meta":{"raw":{"variants":["LLM judgment in loss improves physics equation recovery","Language model scoring sharpens symbolic regression","LLM penalty term guides equation search to true laws","Pre-trained LLMs boost symbolic regression on noisy data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00021,"raw_usage":{"total_tokens":1439,"prompt_tokens":1004,"completion_tokens":435,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":376}},"tokens_in":620,"tokens_out":435,"duration_ms":4415,"temperature":1.0,"reasoning_tokens":376,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:34:02.523630+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give the same three tasks to the method with the LLM term replaced by a pure equation-length penalty, or compare the LLM's scores to expert physicists' ratings of the same candidate equations; if either control reproduces or fails to correlate with the reported gains, the physical-knowledge claim is undermined.","supporting_citations":[],"review_version":1}