{"id":"a03083ae-2909-4d65-a706-dffec8feb19a","arxiv_id":"2508.17376","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"VIPER-R1 fine-tunes a vision-language model to read kinematic plots and propose symbolic equations, then refines them with symbolic regression, but its final metric is computed on the same data used for the refinement fit.","lead":"This preprint's metadata and full text are two different papers. The full text describes VIPER-R1, a vision-language model that guesses physics equations from kinematic plots, then refines them with symbolic regression. Its headline accuracy is inflated because the refinement step fits the answer to the same test data used for scoring.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Post-SR² MSE is fit on the same trajectory used for evaluation, so the headline discovery result is an in-sample fit rather than predictive evidence.","rationale":"The reader's REJECT verdict is well supported, and the weakest assumption identified is exactly the load-bearing flaw in the central claim. The evaluation protocol in Algorithm 2 and Eq. 13 fits the residual to the same trajectory on which the final MSE is computed, so the headline 'more precise discovery' result is an in-sample fit. The structural and accuracy scores (0.812/0.487) may still show that fine-tuning a VLM helps within the narrow synthetic PhysSymbol family, but those scores are secondary to the discovery claim. The absence of a standalone symbolic-regression baseline and the lack of released code or data further prevent assessing whether the VLM contributes beyond ordinary SR. I also note the submission-level metadata mismatch: the listed title/abstract describe 'ShaLa', while the full text is a different paper on VIPER-R1; that mismatch independently makes the submission unverifiable as titled, but the scientific concern above is the decisive one for the full text's central claim. No change to the reader's verdict is needed.","tokens_in":17657,"tokens_out":2562,"duration_ms":29561,"concrete_test":"Run a temporal hold-out: for each PhysSymbol test instance, fit the SR² residual on the first half of the trajectory (inputs t, targets a_GT - a_VLM), then evaluate the final law from Eq. 8 on the held-out second half. If the held-out MSE is substantially worse than the reported 0.032, the headline metric is in-sample. As a second check, run PySR directly on the full test trajectories and compare its MSE; if it matches or beats 0.032 without any VLM prior, the claimed discovery advantage is not demonstrated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline Post-SR² MSE (Table 1; Eq. 13) is computed on the same trajectory that SR² uses to fit the residual. Algorithm 2 Step 2 sets f_residual = SR(inputs=(t), target=(a_GT - a_VLM)) using the test instance's own trajectory data, and Eq. 8 composes that fitted residual with the VLM ansatz. Eq. 13 then evaluates the resulting f_final on those same N trajectory points. Thus the reported MSE of 0.032 is a training error on the evaluation instance, not a measure of whether the discovered law generalizes to new observations. The structural and accuracy scores are not affected by this circularity, but the paper's central claim of 'enabling more precise discovery of physical laws' rests on the Post-SR² MSE. No held-out split, no standalone symbolic-regression baseline, and no code or data are provided to rule out that plain SR on the trajectory achieves the same or better fit.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript, whose arXiv metadata and abstract describe a different paper ('ShaLa'), is in fact a full text titled 'VIPER-R1: Mimicking the Physicist's Eye.' It proposes a vision-language model for physics formula discovery: given phase-space and time-series plots plus trajectory data, the model generates a causal chain of thought and a symbolic ansatz, is trained with supervised fine-tuning and reinforcement learning, and at inference time invokes a symbolic regression tool to fit a residual to its ansatz. The authors introduce a synthetic PhysSymbol corpus of 5,000 instances and report structural, accuracy, and post-regression MSE improvements over several commercial VLMs, with VIPER-R1-7B achieving structural score 0.812, accuracy 0.487, and Post-SR² MSE 0.032.","tokens_in":17835,"tokens_out":7673,"duration_ms":83147,"significance":"If the empirical claims held, the idea of using a VLM's visual reasoning to seed a symbolic regression search would be a genuinely interesting contribution to AI-for-science, and the PhysSymbol corpus could be a useful resource. The two-stage training curriculum and the ablation study are clearly presented, and the structural-score gains over the base Qwen-VL-2.5 model are large. However, the headline end-to-end metric is computed in-sample on the same trajectory used for residual fitting, which undermines the central claim of improved physical-law discovery. The paper also lacks a documented train/test split, a standalone symbolic-regression baseline, and released code or data, so the external validity of the results is currently not established.","major_comments":[{"comment":"The reported Post-SR² MSE is computed on the same trajectory that SR² uses to fit the residual. In Algorithm 2, f_residual is obtained by symbolic regression on target (a_GT - a_VLM) over the evidence trajectory's time points, f_final is the sum f_VLM + f_residual, and Eq. (13) evaluates f_final on those same N points. This makes Table 1's MSE a training error on the evaluation instance, not a measure of whether the discovered law generalizes to new observations. The conclusion that the method enables 'more precise discovery of physical laws' is not supported by this metric. The authors should evaluate f_final on held-out trajectories not used by either the VLM or the SR² step and report both the in-sample and the generalization MSE.","section":"§3.4, §4.1, Appendix B.4 (Algorithm 2, Eq. 6-8, Eq. 13)"},{"comment":"The evaluation section refers to a 'PhysSymbol test set' (Figure 4, Table 1), but nowhere in Section 4.1 or Appendix C is the train/test split of the 5,000 instances defined. Because the same dataset is used for MSI and RGSC training, and because the RGSC reward (Eq. 4 and Eq. 11) directly uses the ground-truth equation, the absence of a split description raises a contamination risk. The authors must state exactly how many instances are held out, confirm that no training instance is used in evaluation, and ideally release the split indices.","section":"§4.1, Appendix C"},{"comment":"Table 1 compares VIPER-R1 only against VLM baselines, while the conclusion claims the method 'outperforms ... traditional methods.' No standalone symbolic-regression baseline (e.g., PySR, AI Feynman, or an LLM-SR variant) is reported. Without such a baseline, the Post-SR² MSE numbers do not establish that the VLM's ansatz improves over running symbolic regression directly on the trajectory data. Add this comparison, including the same residual-fitting protocol applied to a trivial or empty ansatz.","section":"§4.1, Table 1, §5"},{"comment":"The manuscript's title, arXiv metadata, and abstract describe 'ShaLa: Multimodal Shared Latent Space Modelling,' but the full text is a different paper, 'VIPER-R1: Mimicking the Physicist's Eye,' about physics formula discovery. This is not a superficial typo: the abstract's claims about multimodal VAEs and shared latent spaces are absent from the body, and the body's claims about formula discovery are absent from the abstract. The title, metadata, and abstract must be aligned with the actual content before the manuscript can be evaluated as a scientific submission.","section":"Title, abstract, and full text"}],"minor_comments":[{"comment":"Equations (1) and (2) contain garbled notation, including raw '...' in the summands, and should be typeset properly with well-defined indices.","section":"§3.2, Eq. 1-2"},{"comment":"Algorithm 1, line 8 says the model is updated by 'descending the gradient' of the log-likelihood; the update should ascend the log-likelihood or minimize its negative.","section":"Appendix B.4, Algorithm 1"},{"comment":"Table 1 reports only point estimates; confidence intervals or standard deviations across repeated evaluations (or bootstrap intervals) should be provided, since some baseline differences are small.","section":"Table 1"},{"comment":"Appendix C.6 refers to a 'three-stage training pipeline,' but the paper describes two training stages plus an inference stage; the terminology should be aligned.","section":"Appendix C.6"},{"comment":"The dataset includes stochastic perturbations, and Eq. (13) computes MSE on a single stochastic trajectory; this further confounds the in-sample metric, because the residual regressor can fit the realized noise rather than the underlying law.","section":"§C.2, §4.1"},{"comment":"The paper provides a project page but no code, data, or reproduction script; please release the PhysSymbol generation and evaluation code to support reproducibility.","section":"Appendix B and C"}],"recommendation":"major_revision","confidential_remarks":"I seriously considered rejection because the headline Post-SR² MSE is circular and the train/test split is undocumented. I am recommending major revision because the structural and accuracy results, together with the two-stage training idea, may be salvageable if the authors provide a held-out generalization evaluation, a standalone symbolic-regression baseline, and a clear data split. The title/abstract mismatch is highly unusual and should be checked by the editor before further processing; if the mismatch is not a housekeeping error, it alone would justify returning the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper you sent is really two papers in one envelope. The arXiv metadata says “ShaLa: Multimodal Shared Latent Space Modelling,” but the full text is “Mimicking the Physicist’s Eye: A VLM-Centric Approach for Physics Formula Discovery.” That alone is enough for a desk reject, but I read the full text and there is substance underneath.\n\nWhat is actually new: fine-tuning a VLM to look at phase-space and trajectory plots and output a symbolic ansatz, trained with a two-step SFT (joint C-CoT + equation, then equation-only) and an RL stage with a structural reward. The structural reward is parameter-agnostic Jaccard similarity on skeletonized terms, which is sensible. The inference-time residual regression (SR²) is a nice touch—it reframes symbolic regression as a perturbation correction rather than a full search. The PhysSymbol dataset, if released, could be a useful benchmark for vision-grounded equation discovery.\n\nThe structural and accuracy scores in Table 1 are the legitimate part of the evaluation. A 0.812 structural score on a synthetic test set, versus 0.518 for the best baseline, suggests the fine-tuning genuinely transfers to reading plots. That is a real result, if the test set is not contaminated. The paper says the dataset is synthetic and new, so contamination is unlikely.\n\nThe soft spot is the headline Post-SR² MSE. The stress test is right: Algorithm 2 and Equations 6–8 fit the residual to the test instance’s own trajectory, then evaluate on that same trajectory. The MSE of 0.032 is therefore an in-sample fit, not a predictive discovery metric. It does not tell you whether the final equation generalizes to new initial conditions or new data. The paper also does not compare against plain symbolic regression on the trajectory alone; that baseline could well match or beat 0.032 without any VLM. No code or data are shipped, so I cannot verify the pipeline. These flaws are load-bearing for the “more precise discovery of physical laws” claim.\n\nWho this is for: researchers working on LLM/VLM-driven scientific discovery, especially equation discovery. It deserves a serious referee, but only if the authors fix the metadata and redo the evaluation with a proper held-out protocol—e.g., train the residual on a subset of trajectories and test on held-out initial conditions, or at least report the VLM-only accuracy without the in-sample residual. The structural result may survive that. As submitted, I would not cite the MSE numbers.\n\nMy recommendation: send it back with a request for major revision, and make the authors address the circular evaluation head-on. If they can produce a clean held-out result, this becomes a useful paper.","headline":"The submitted manuscript has a critical metadata mismatch and the headline MSE is an in-sample fit, but the core idea—fine-tuning a VLM to read kinematic plots and propose symbolic structure—is worth a serious look if the evaluation is fixed.","tokens_in":18374,"tokens_out":1047,"would_cite":false,"duration_ms":12450,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Trained on phase plots, a vision-language model beats top VLMs at physics formula discovery.","keywords":["physics formula discovery","vision-language model","symbolic regression","phase portrait","causal chain-of-thought","reinforcement learning","PhysSymbol"],"falsifier":"Compute the Post-SR² MSE using coefficients fit on one trajectory from an equation and evaluate the formula on a second trajectory from the same equation with a different initial condition. If the error jumps to the baseline level, the reported 0.032 is an in-sample fit rather than evidence of law discovery.","tokens_in":17410,"feed_emoji":"🔬","tokens_out":5082,"duration_ms":49948,"temperature":0.7,"pith_summary":"The paper claims that a vision-language model can discover governing equations by looking at the same kinematic plots physicists use. It introduces a two-stage curriculum that first teaches the model to reason about phase-space and trajectory images, then sharpens the symbolic structure of its hypotheses with reinforcement learning. At inference, the model invokes an external symbolic regression tool to fit the residual between its guess and the data, an agentic step the authors call Symbolic Residual Realignment. On the new 5,000-instance PhysSymbol benchmark, the 7B model reaches a structural score of 0.812 and a final mean-squared error of 0.032, roughly three times lower than the best zero-shot VLM baseline. The claim matters because it suggests visual pattern recognition can supply the strong priors that make symbolic regression tractable.","feed_headline":"VLMs that read phase plots beat top models at physics discovery","feed_subtitle":"A vision-language model trained on plots and trajectories finds physical laws with a final error of 0.032, far below all baselines.","key_machinery":"The engine of the method is a two-part training curriculum followed by an agentic inference step. Motion Structure Induction (MSI) is a supervised fine-tuning stage that teaches the model to read plots and produce reasoning chains plus hypotheses; Reward-Guided Symbolic Calibration (RGSC) uses GRPO reinforcement learning with a parameter-agnostic structural reward, a Jaccard similarity between skeletonized term sets, to favor topologically correct equations; and Symbolic Residual Realignment (SR²) composes the model's ansatz with a residual expression found by an external symbolic regression tool. The structural reward is what lets the model improve even when coefficients are wrong, and the residual step is what converts a structurally reasonable guess into a numerically accurate law.","core_discovery":"The paper's central claim is that visual induction plus symbolic residual refinement can replace blind search over equations. Given only a phase-space portrait, a time-series plot, and trajectory data, VIPER-R1 first produces a causal chain of thought and a symbolic ansatz; a structural reward then pushes the model toward topologically correct formulas; and finally the model composes its ansatz with a symbolic regression of the remaining residual to obtain the final law. The authors report that VIPER-R1-7B achieves a structural score of 0.812 and an accuracy score of 0.487 on PhysSymbol, outperforming all tested general VLMs, with the final Post-SR² MSE of 0.032.","pith_inferences":["The residual stage could be bolted onto any vision-language-model-generated ansatz, so the framework likely generalizes beyond the specific model and training recipe used in the paper.","A direct test would remove one modality at a time, such as supplying only trajectory numbers or only plots, to measure how much of the structural score actually comes from visual input.","Because the benchmark equations are built by sampling two to five terms from an eleven-category library, performance on equations outside that known family, such as partial differential equations or chaotic systems, remains open."],"forward_implications":["Symbolic regression no longer has to search cold; a vision-language model that reads plots can give it a structurally correct starting point, shrinking the search space dramatically.","Reinforcement learning with a structure-only reward raises both structural and exact-match scores, showing that coefficient-agnostic training can improve symbolic precision.","The residual-realignment step turns any plausible hypothesis into a low-MSE final law, making the quality of the initial guess the key bottleneck for discovery.","The same two-stage recipe transfers to the smaller 3B model, which already beats all zero-shot baselines, and the paper states the pipeline is intended to scale to chaotic systems, partial differential equations, and real experimental video."],"supporting_citations":[{"why":"PySR is the external symbolic regression engine invoked during the SR² refinement stage.","marker":"(Cranmer, 2023)"},{"why":"LLM-SR is the main LLM-based equation discovery baseline and the design source for generating equation hypotheses.","marker":"(Shojaee et al., 2025a)"},{"why":"GRPO is the reinforcement learning algorithm used in Reward-Guided Symbolic Calibration.","marker":"(Shao et al., 2024)"},{"why":"Qwen2.5-VL provides the backbone architecture that MSI and RGSC fine-tune into VIPER-R1.","marker":"(Bai et al., 2025a)"},{"why":"AI Feynman represents the physics-inspired symbolic regression approach the paper positions its visual-prior strategy against.","marker":"(Udrescu & Tegmark, 2020)"},{"why":"Supplies the claim that physicists read phase portraits to infer conservation laws and damping, motivating the visual input design.","marker":"(Strogatz, 2001)"}],"fun_headline_variants":["ShaLa boosts multimodal VAE synthesis with a diffusion prior","ShaLa: Sharper shared latents, better multimodal synthesis","A diffusion prior that fixes multimodal VAE inference","ShaLa scales shared latent models to many modalities"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline error numbers assume that fitting the final equation to the same trajectory data used to build it tests discovery, when it actually measures how well the formula fits that particular trajectory.","fun_headline_variants_meta":{"raw":{"variants":["ShaLa boosts multimodal VAE synthesis with a diffusion prior","ShaLa: Sharper shared latents, better multimodal synthesis","A diffusion prior that fixes multimodal VAE inference","ShaLa scales shared latent models to many modalities"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000257,"raw_usage":{"total_tokens":1536,"prompt_tokens":857,"completion_tokens":679,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":473,"completion_tokens_details":{"reasoning_tokens":615}},"tokens_in":473,"tokens_out":679,"duration_ms":7292,"temperature":1.0,"reasoning_tokens":615,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:04:22.500798+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the Post-SR² MSE using coefficients fit on one trajectory from an equation and evaluate the formula on a second trajectory from the same equation with a different initial condition. If the error jumps to the baseline level, the reported 0.032 is an in-sample fit rather than evidence of law discovery.","supporting_citations":[],"review_version":1}