{"id":"f4937360-d2b3-4c9a-be19-84bc2d29986c","arxiv_id":"2502.00963","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A framework of fine-tuned LLMs autoformalizes PDE control problems into signal temporal logic, generates Gurobi solver code, and proposes subgoal constraints; it beats generic LLMs in decoupled tests but shows mixed end-to-end utility gains.","lead":"This paper trains language models to turn plain-English descriptions of physics problems into formal math constraints and working solver code, and to plan intermediate goals that improve control of heat and wave systems. It is a useful proof of concept for LLMs in scientific computing, but the headline utility gains mostly disappear when the full pipeline is tested end to end.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 62% utility-gain headline rests on oracle code (Table 9 footnote 5); end-to-end Table 17 shows heat Δr = -0.731, so the central claim of improved PDE control utility fails for the actual pipeline.","rationale":"The reader's verdict was CONDITIONAL, and this stress-test supports keeping that verdict: the paper is a substantial engineering contribution with a large released dataset, fine-grained metrics, and an unusually honest appendix reporting end-to-end results. However, the single most load-bearing part of the strongest claim is the 62% utility-gain figure, and the paper's own Table 17 shows that the end-to-end system does not deliver positive heat utility gain (Δr = -0.731) and does not beat GPT-4o on wave utility gain (0.698 vs 0.778). The gap between the oracle-code Table 9 and the full-pipeline Table 17 is exactly the kind of internal evidence that should control the headline claim. The reader's stated weakest_assumption focused on the synthetic distribution and the manual-data IoU drop; this pass instead centers the oracle-code decoupling, which the reader mentioned in the rationale but not as the weakest assumption. Because the contradiction is already present in the paper and the correct fix is a re-scoped claim or an end-to-end re-benchmark, the verdict remains CONDITIONAL rather than moving to ACCEPT or REJECT: the contributions (dataset, translator, coder, controller training) are real and reproducible, but the central utility-improvement claim must be re-stated or re-validated before it can be accepted as written.","tokens_in":32679,"tokens_out":4084,"duration_ms":40576,"concrete_test":"Recompute Table 9's Δr without oracle code: run the full Translator→Controller→Coder pipeline on the same heat test anchors, execute only the generated Python programs that are valid, and compute Δr on that subset. If the end-to-end heat Δr remains near -0.731 (or negative) while the oracle-code Δr is 1.453, then the utility-gain claim must be restricted to the oracle setting. Additionally, report the same metric for the wave All row, where Ours (0.698) already trails GPT-4o (0.778).","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claim is the 62% utility gain on heat problems (Δr = 1.453 vs 0.897, Table 9). That table is a decoupled evaluation: footnote 5 states that 'we provide the true Python code for any valid generated subgoal STL ϕ′', meaning the Coder's output is replaced by ground-truth code. When the full pipeline is run end-to-end (Table 17), the same Controller achieves heat All Δr = -0.731, so subgoal reasoning degrades utility relative to direct solving on average; on wave All, Ours (0.698) trails GPT-4o (0.778). The abstract's 'improving utility for PDE control' and the Sec. 4.2 claim that our Controller 'most significantly improves the utility' are therefore not supported by the actual system as deployed; they hold only under an oracle-code assumption. This is an internal inconsistency, not a matter of consensus, because the paper itself reports the contradictory end-to-end numbers. The manual-data evaluation (Table 5) and the stated failure of all models to produce meaningful reasoning on manual data (Sec. 4.2) further limit the claim to synthetic problems. The 62% figure should be re-scoped as 'utility gain in a decoupled setting with ground-truth code', or the end-to-end gap must be explained and fixed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents PDE-Controller, a fine-tuned LLM framework for 1D heat- and wave-equation control. The pipeline has three trained components: a Translator (natural language to Signal Temporal Logic), a Controller (proposes intermediate subgoal STLs, trained with DPO), and a Coder (STL to Python/Gurobi code). Training uses a synthetically generated dataset of roughly 2.13 million (NL, STL, Python) triplets, plus 34 manually written problems. The authors report strong synthetic performance for autoformalization (IoU 0.992), program synthesis (executability up to 0.9978), and reasoning, including a headline \"up to 62% improvement in utility gain\" on heat problems. The central utility-gain claim, however, comes from a decoupled evaluation in which generated subgoal STLs are paired with ground-truth Python code, and the paper's own end-to-end evaluation reports a negative average utility gain on heat problems for the full pipeline.","tokens_in":33054,"tokens_out":4622,"duration_ms":47396,"significance":"If the end-to-end utility gains were real, this would be a substantial contribution: it would demonstrate a trained LLM pipeline capable of formalizing informal PDE control tasks and improving open-loop control quality through subgoal reasoning, backed by a large released dataset and reproducible model checkpoints. The Translator and Coder results on synthetic data are strong, and the decoupled reasoning evaluation is a useful methodological contribution. However, the headline claim is not supported by the deployed system: Table 17 shows heat All Δr = -0.731 for the full pipeline, and on wave problems the proposed Controller trails GPT-4o in end-to-end utility gain. The significance of the paper is therefore conditional on either repairing the end-to-end gap or explicitly re-scoping all utility-improvement claims to the oracle-code setting.","major_comments":[{"comment":"The central claim of improved PDE control utility is internally inconsistent with the paper's own end-to-end results. The abstract and Section 4.2 advertise \"up to 62% improvement in utility gain\" based on Table 9, but footnote 5 states that this evaluation provides true Python code for any valid generated subgoal STL. When the full pipeline is run end-to-end (Table 17), the same Controller achieves heat All Δr = -0.731, meaning subgoal reasoning degrades utility relative to direct solving, and on wave All the proposed model's Δr = 0.698 is below GPT-4o's 0.778. The claim that PDE-Controller \"improves the utility of PDE control\" is therefore not supported for the actual deployed system; it holds only in the decoupled oracle-code setting. The authors must either provide an end-to-end evaluation supporting the claim, or re-scope the abstract and Section 4.2 to state explicitly that utility improvements apply only when the Coder's output is replaced by ground-truth code.","section":"Abstract; Section 4.2; Table 9; Table 17"},{"comment":"The baseline comparison in the decoupled reasoning evaluation is difficult to interpret because nearly all baseline subgoal STLs are invalid. Table 9 reports valid STL rates of 82.70% for Ours, 42.45% for MathCoder2, 2.55% for GPT-4o, and 0.04% for GPT-o1-mini, and Table 18 gives per-difficulty proportions of valid ϕ′ as low as 0.008 for GPT-4o on wave problems. Since Success Rate P and Utility Gain Δr are computed only over proposed subgoal STLs that are syntactically valid, the GPT baselines contribute only a handful of samples to their Δr estimates; the claim that GPT-4o is the \"second-best\" with Δr = 0.897 on heat rests on an extremely small valid subset. The authors should report the number of valid proposals per model and difficulty level, and should also present a comparison in which invalid proposals are explicitly counted as failures (e.g., zero utility or a penalty), otherwise the headline improvement over GPT models in reasoning is not robustly established.","section":"Table 9; Table 18; Section 4.2"},{"comment":"The paper's applicability to real-world problems is materially narrower than the abstract suggests. Section 4.2 states that on manual data all models fail to generate meaningful reasoning steps due to invalid subgoal STL proposals, and Table 5 shows Translator IoU dropping from 0.992 on synthetic heat data to 0.711 on manual heat data, while Table 16 shows end-to-end heat executability dropping to 0.451 for Ours. The claims of PDE reasoning and utility gains should be explicitly restricted to the synthetic template distribution defined in Section 3.2 and Tables 12-13, with a clear statement that the manual evaluation covers only autoformalization and program synthesis, not the reasoning/utility contribution. The abstract and contribution list currently overstate the framework's real-world behavior.","section":"Section 4.2; Table 5; Table 16; Appendix D"}],"minor_comments":[{"comment":"The parenthetical deviations in Tables 4 and 5 are unclear: for example, IoU = 0.992 with a reported deviation of 0.07 appears inconsistent with a metric bounded in [0,1], and the same applies to IoU = 0.772 with deviation 0.35. Please clarify whether these are standard deviations, standard errors, or percentages of some quantity.","section":"Table 4; Table 5"},{"comment":"The metric name \"Utility RMSE\" is described as \"Relative mean square error\" in the footnote to Table 3, which mixes RMSE and relative-error terminology. Please use one consistent definition and state the exact formula.","section":"Table 3; Appendix A.4"},{"comment":"The column header \"Math-Coderv2\" appears with a typo in several tables; it should read \"MathCoder2\".","section":"Tables 9, 17, 18, 19"},{"comment":"The claim that this is the \"first comprehensive dataset\" for PDE control should be tempered, since the synthetic portion is generated from 1374 hand-designed STL templates over two PDE types with fixed parametric ranges; the manual portion has only 34 samples. Some statement of coverage limitations would be more appropriate than \"comprehensive\".","section":"Section 3.2; Appendix H.2"}],"recommendation":"major_revision","confidential_remarks":"The main reason for major revision rather than rejection is that the autoformalization and program-synthesis components appear technically solid and the released dataset is a useful contribution. The blocking issue is that the headline utility-gain result is contradicted by the paper's own end-to-end experiment, and the baseline comparisons in the decoupled reasoning table are weakened by extremely low valid-STL rates for GPT models. Both issues are fixable in principle by re-scoping claims, reporting valid-sample counts, and adding an end-to-end analysis, but the current version overstates what the system achieves as deployed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First: the headline 62% utility-gain claim is not supported by the paper's own end-to-end numbers. Table 9 gets heat Δr=1.453, but that table replaces the Coder's output with true Python code for any valid subgoal STL (footnote 5). When the actual Coder is used (Table 17), the same Controller on heat All gives Δr = -0.731—reasoning actively hurts. The paper reports both numbers and doesn't reconcile them. The abstract and Sec 4.2 lines about 'improving utility for PDE control' need to be re-scoped to the decoupled setting or the end-to-end gap explained.\n\nThat said, there is real substance here. The 2.13M-sample NL/STL/Python dataset for heat and wave control is a solid resource, released per the datasheet. The Translator/Coder results on synthetic data are good (IoU ~0.99, executability ~0.99, low utility RMSE), and the four-constraint generalization test is a nice touch. Manual-data collection with actual problem statements is honest, and the paper openly notes that all models fail to produce meaningful reasoning steps there. The subgoal-STL reasoning protocol is new for PDE control, and the DPO training setup with SFT regularization is reasonable.\n\nSoft spots beyond the headline: the GPT baselines in Table 9 produce 0.04-2.55% valid STLs, so 'outperforms GPT' is largely a validity comparison, not a fair reasoning comparison. The manual-data Coder anomaly in Table 5—MathCoder2 with IoU 0.34 but utility RMSE 0.0004 while Ours has IoU 0.71 and RMSE 2.47—needs a check; it suggests the evaluation is sensitive to executable-only samples or code that ignores constraints. Also, the subgoal time placement rule (before anchor constraints) is a heuristic; fine, but it bounds the space of reasoning the Controller can learn.\n\nBottom line: the engineering is worth engaging with, and the dataset deserves a real referee. But the central performance claim has a load-bearing inconsistency and should be fixed before publication. I'd send it to review, with a clear request to reconcile Tables 9 and 17 and re-benchmark baselines with valid-STL conditioning.","headline":"A substantial dataset and trained pipeline for NL-to-STL PDE control, but the 62% utility-gain headline collapses in the paper's own end-to-end table.","tokens_in":33570,"tokens_out":2651,"would_cite":true,"duration_ms":24725,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T50","68T07","93C20","35K05","35L05"],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuned LLMs autoformalize plain-English PDE control problems, write solver code, and propose subgoal constraints, improving utility gain by up to 62% over GPT-4o.","keywords":["PDE control","signal temporal logic","autoformalization","reasoning","program synthesis","heat equation","wave equation","direct preference optimization"],"falsifier":"A concrete experiment: measure the valid-subgoal rate and mean utility gain $\\Delta r$ of the released Controller on the 34 human-written problems with the authors' parsing tool. The paper reports this value is effectively zero for every model; if it stays zero when the base model is scaled up or the prompt format changes, the 62% utility-gain claim is a property of the synthetic distribution, not of PDE control generally.","tokens_in":32503,"feed_emoji":"⚙️","tokens_out":13450,"duration_ms":116397,"temperature":0.7,"pith_summary":"This paper claims that large language models can take plain-English descriptions of PDE control problems and handle the whole pipeline: formalizing constraints, writing solver code, and reasoning about how to control the system better. Its three fine-tuned components — a Translator that converts natural language into Signal Temporal Logic (STL), a Coder that writes Python for the Gurobi MILP solver, and a Controller trained by DPO to propose intermediate subgoal specifications — are trained on a released dataset of over two million synthetic 1D heat- and wave-equation problems. On synthetic benchmarks the Controller raises the utility gain for heat control by 62% over GPT-4o ($\\Delta r = 1.453$ vs $0.897$), with 0.992 IoU autoformalization and 0.998 code executability; on 34 human-written problems the Translator's IoU drops to about 0.71 (heat) and no model produces valid reasoning subgoals. The authors frame the work as a first testbed for LLMs in applied mathematics and release data, code, and checkpoints.","feed_headline":"LLMs improve PDE control utility gain by 62%","feed_subtitle":"Autoformalization plus subgoal planning lifted heat-control utility from 0.897 to 1.453.","key_machinery":"The load-bearing object is the subgoal STL $\\phi'$: a Signal Temporal Logic formula — a spatiotemporal constraint of the form $G_{[t_1,t_2]}(\\forall x \\in [x_1,x_2]: u(x) \\lessgtr ax+b)$ or its $F$ ('eventually') variant — whose continuous semantics assign a real utility $r(\\phi)$ via min/max over the constraint's space-time region. The reasoning mechanism is sequential two-stage optimization: solve $\\phi'$ with Gurobi (capped at 120 seconds), use the resulting field as the new initial condition for the anchor $\\phi$, and record $r(\\phi|\\phi')$; the Controller LLM is trained with DPO on win/lose pairs of subgoals labeled by whether $r(\\phi|\\phi') > r(\\phi)$, regularized by an SFT term so it keeps producing valid STL syntax. This converts the non-convex MILP of direct control into a learned warm-start search over initial conditions.","core_discovery":"The paper's central claim is that a fine-tuned LLM can improve open-loop PDE control by decomposing the target specification into a subgoal: for an anchor STL $\\phi$, the Controller proposes a subgoal $\\phi'$, the system optimizes $\\phi'$ first, adopts the reached state as the new initial condition, and then optimizes $\\phi$, scoring the pair as $r(\\phi|\\phi')$ against the direct-solve utility $r(\\phi)$. The authors report that heat problems reach a success rate of 0.812 and a mean utility gain of $\\Delta r = 1.453$, a 62% improvement over GPT-4o's $0.897$, and that the full pipeline — Translator, Coder, Controller — handles synthetic autoformalization at 0.992 IoU and code executability at 0.998. On the 34 human-written problems, the Translator's IoU drops to about 0.71 (heat) and no model, including PDE-Controller, produces valid reasoning subgoals, which the authors attribute to unstructured phrasing, inconsistent units, and unfamiliar notation in the manual data.","pith_inferences":["The Controller's subgoal proposal is effectively a learned warm-start for a non-convex MILP; replacing the Gurobi solver with a differentiable physics surrogate would allow the subgoal to be trained end-to-end on gradient signals rather than win/lose labels, a path the paper does not take.","Because the preference labels are computed purely from the utility $r(\\phi|\\phi')$, the same DPO recipe transfers to other objectives — such as control-input energy, tracking error, or satisfaction margin — without changing the framework.","The manual-data failure suggests the reported gains are tied to the synthetic templates' parameter ranges and phrasing; a direct check is whether fine-tuning on human-written problems, or augmenting templates with the observed manual defects (mixed units, 'ho' for rho, missing material data), restores valid subgoal proposals.","The framework is limited to open-loop control; extending the Controller's proposals to a receding-horizon, closed-loop scheme would test whether the learned subgoal distribution remains useful when re-planning from feedback states."],"forward_implications":["A fine-tuned 7B open-weights checkpoint can exceed prompted frontier models specifically on PDE-control formalization and reasoning: the Translator reaches 0.992 IoU on synthetic problems and the Controller produces valid subgoal STLs at 82.7%, while GPT o1-mini yields essentially no valid subgoal STLs on the synthetic benchmark.","The components compose end-to-end: feeding the Translator's noisy STL predictions into the Coder costs only 0.57% additional utility RMSE, so autoformalization errors are largely absorbed by the code generator.","The subgoal-reasoning skill transfers to unseen 4-constraint STL formats (IoU 0.934–1.0), suggesting the learned subgoal prior generalizes beyond the 1–3 constraint training distribution.","The gains are strongest for heat problems ($\\Delta r = 1.453$, +62% over GPT-4o's 0.897); wave gains are smaller, and no model improves the hardest wave anchors."],"supporting_citations":[{"why":"Defines the temporal-logic monitoring of continuous signals that the STL formalism in Eq. (1) relies on.","marker":"Maler & Nickovic, 2004"},{"why":"Supplies the STL-based PDE control formalism and the utility semantics $r(\\phi)$ that define the paper's reward and evaluation.","marker":"Alvarez, 2020"},{"why":"Gives the continuous robustness semantics of STL used to compute the utility $r(\\phi)$ in Appendix A.2.","marker":"Donzé & Maler, 2010"},{"why":"Provides the MILP encoding of temporal-logic specifications that turns the discretized PDE control problem into a Gurobi-solvable program.","marker":"Sadraddini & Belta, 2015"},{"why":"The external MILP solver whose solutions produce every reported utility and preference label.","marker":"Gurobi Optimization, LLC, 2024"},{"why":"The DPO objective used to train the Controller on win/lose subgoal pairs.","marker":"Rafailov et al., 2024"},{"why":"MathCoder2, the base 7B checkpoint that Translator, Coder, and Controller are fine-tuned from.","marker":"Lu et al., 2024"},{"why":"Source of the SFT regularization term (with weight $\\lambda$) that keeps DPO training from degrading the Controller's STL generation.","marker":"Pang et al., 2024"}],"fun_headline_variants":["LLM subgoal planning boosts PDE control utility by 62%","Autoformalization plus subgoals: 62% better PDE control","PDE-Controller: LLMs reason in subgoals to improve control","LLM framework achieves 62% gain in PDE control utility"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains assume that the synthetic problem distribution — hand-picked STL templates with sampled parameters, solved through Gurobi's MILP formulation of the discretized heat and wave equations — faithfully represents real PDE control quality; the authors' own manual test set strains this assumption, with Translator IoU dropping from 0.992 to 0.711 (heat) and no model generating valid reasoning subgoals on manual problems.","fun_headline_variants_meta":{"raw":{"variants":["LLM subgoal planning boosts PDE control utility by 62%","Autoformalization plus subgoals: 62% better PDE control","PDE-Controller: LLMs reason in subgoals to improve control","LLM framework achieves 62% gain in PDE control utility"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000307,"raw_usage":{"total_tokens":1769,"prompt_tokens":969,"completion_tokens":800,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":585,"completion_tokens_details":{"reasoning_tokens":723}},"tokens_in":585,"tokens_out":800,"duration_ms":7931,"temperature":1.0,"reasoning_tokens":723,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T17:05:15.673022+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete experiment: measure the valid-subgoal rate and mean utility gain $\\Delta r$ of the released Controller on the 34 human-written problems with the authors' parsing tool. The paper reports this value is effectively zero for every model; if it stays zero when the base model is scaled up or the prompt format changes, the 62% utility-gain claim is a property of the synthetic distribution, not of PDE control generally.","supporting_citations":[],"review_version":1}