{"id":"9837c2b1-3095-44d9-9a28-e8c12d908ef4","arxiv_id":"2602.07083","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A physics-aware two-stage SFT+GRPO training method with period/AST/sandbox rewards raises small open LLMs from ~0-2% to ~68-77% on a strict OpenSeesPy building-modeling benchmark (BMEval).","lead":"An engineering-AI paper trains language models to write OpenSeesPy building-simulation code from plain-language descriptions, reporting that its two-stage training method lifts small open-source models from near-zero to roughly 70-80% on a strict pass metric. The practical target is letting engineers generate simulation-ready structural models in minutes instead of days.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"BMEval's strict-pass metric can be satisfied by printing an empirical-formula period and a 'PASS' keyword; it never checks that the period is the OpenSees eigen result, so RLA-SPC's gains may reflect output imitation rather than physical consistency.","rationale":"The reader's conditional verdict is appropriate. My concern is the most load-bearing because it attacks the construct validity of the dependent variable: if BMEval's strict-pass can be achieved by output-surface imitation, the paper's central claim about physical consistency is unsupported, even though the method may still produce useful code and the internal comparisons are consistent. I considered other issues (128-case benchmark, no error bars, naming inconsistencies), but they are secondary; the output-surface metric is the one that would invalidate the interpretation. Credit: the paper releases code/dataset, specifies prompts, and reports large consistent gains; those are real artifacts. The concrete test is feasible from the released repo: intercept ops.eigen in the sandbox and compare outputs. If the test shows sensitivity, the paper's claim is substantially strengthened; if not, the benchmark needs redesign (e.g., verify T1 via an independent eigen run on the generated model, and check code structure). Thus the verdict stays conditional; no change from the reader's CONDITIONAL is warranted at this stage.","tokens_in":23319,"tokens_out":7471,"duration_ms":76358,"concrete_test":"Instrument the evaluation sandbox used for Table 2: for each of the 128 BMEval prompts, run the final RLA-SPC-generated candidates twice—once normally and once with ops.eigen/ops.getEigenvalues monkey-patched to return deliberately wrong eigenvalues (e.g., 5x the true frequencies). Recompute Pass@5_strict under both conditions and report the fraction of strict-pass samples whose printed T1 changes. If that fraction is low (e.g., <20%) or the strict-pass rate declines by less than 10 points despite 5x-wrong eigenvalues, the period criterion is not measuring the simulation, and the headline gains are best interpreted as output-format imitation rather than physics-consistent generation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Central claim: RLA-SPC raises open LLMs to high strict-pass on BMEval, demonstrating physically consistent, simulation-executable generation. This claim depends on BMEval's strict criterion actually requiring the generated code's simulated dynamics to match structural physics. That requirement is not enforced in the reported protocol. Section 4.2/Appendix A.2 define Pass@5_period by extracting T_pred from program output and comparing it to an empirical-formula ground truth; Pass@5_compliance is keyword matching. Section 5.3.3 rewards the same surface quantities: Eq. (14) computes |T_pred - T_gt|/T_gt and Eq. (15) grades it, while Eq. (8) weights execution/format. Nothing in the paper requires T_pred to be an eigenvalue of the OpenSees model: no AST check that the printed value is derived from ops.eigen/ops.getEigenvalues, no independent re-simulation to obtain the model's actual fundamental period, no substitution/injection test. A candidate program that executes, prints a period from the same empirical formula (or a memorized mapping), and prints 'PASS' satisfies all three strict conditions even if the model itself is physically inconsistent or the code never runs modal analysis. This is compounded by BMEval being generated from the same parameter ranges and prompt templates as CivilInstruct Part 2 (Table 3, Figures 8-9), so the empirical period formula is in-distribution for RLA-SPC. The before/after improvements in Table 2 are real under BMEval's operationalization, but they do not establish physical consistency unless the period check is coupled to the actual simulation. The BMEval ground truth also being an empirical formula (Section 4.2) rather than an FE benchmark further weakens the claim that strict-pass measures physics.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper formalizes Automatic Building Modeling (AutoBM) as a code-generation task in which an LLM must produce OpenSeesPy models that are executable, physically consistent, and compliant with engineering constraints. It introduces CivilInstruct, a 10,912-sample instruction dataset spanning API tutorials, parameterized long-code generation, debugging chain-of-thought data, and physics-informed expert examples; BMEval, a 128-case benchmark that scores generated programs on executability, fundamental-period accuracy, and keyword-based compliance; and RLA-SPC, a two-stage pipeline that applies supervised fine-tuning followed by group-relative policy optimization with a multi-granularity hybrid reward. Experiments on BMEval report large before/after gains for three open models, e.g., CodeLlama-7B-Instruct rising from Pass@5_strict 0.00 to 67.73 and Seed-Coder-8B-R from 0.78 to 77.14, exceeding the strict-pass rates of the commercial models evaluated.","tokens_in":23762,"tokens_out":4531,"duration_ms":46366,"significance":"If the evaluation protocol is valid, the paper makes a useful practical contribution: it demonstrates that relatively small open LLMs can be aligned to generate long, executable OpenSeesPy scripts with high pass rates on a domain-specific benchmark, and it releases the dataset and code. The two-stage training recipe and the detailed failure-mode analysis are valuable for future work on physics-aware code generation. However, the central claim that the models achieve 'physically consistent, simulation-executable' generation is not currently supported by the evidence, because the BMEval strict criterion does not verify that the printed period is the eigen-solution of the generated OpenSees model, and the compliance check is only keyword matching. The reported gains are real under the paper's operationalization, but their engineering significance depends on an independence and validity of the benchmark that is not established.","major_comments":[{"comment":"Pass@5_period extracts T_pred from 'program outputs using a deterministic, rule-based matching method' and compares it to an empirical-formula ground truth. Nothing in the protocol requires T_pred to be computed by the OpenSees model (e.g., from ops.eigen or ops.getEigenvalues). A candidate program that executes, prints a period obtained from the same empirical formula used to generate the ground truth, and prints 'PASS' satisfies all three strict conditions even if the code never runs modal analysis or is physically inconsistent. The paper should either (a) re-execute each generated program in a separate environment and extract the fundamental period from the simulation results, (b) perform an AST/dataflow check that the printed value is derived from the eigen-solver, or (c) add substitution/injection tests where a manually corrupted model with the same printed period fails. As written,","section":"Section 4.2 / Appendix A.2, Eq. (16)"},{"comment":"The training reward R_phy uses the relative error between the extracted period and the empirical-formula ground truth, with tolerance bins (Eq. 15), and the evaluation metric Pass@5_period uses the same relative-error comparison with tolerance 0.30 (Eq. 16). This is not formal circularity, but it means the reward and the headline metric are two views of the same surface quantity, extracted from program output with the same rule-based method. The reported improvements are therefore expected under reward optimization and do not independently validate physical consistency. An independent re-simulation check, or at least a held-out set with different parameter ranges and prompt templates, is needed to separate genuine physics understanding from optimization of the extracted-output statistic.","section":"Section 5.3.3, Eq. (14) vs. Section 4.2, Eq. (16)"},{"comment":"All pass@k numbers are computed on 128 samples with no confidence intervals, no standard deviations across training seeds, and no significance tests. For a 50% observed rate the standard error is roughly 4.4 percentage points, so the small differences among commercial models in Table 1 (e.g., Pass@5_strict 16.62 vs. 18.84) are not interpretable. The large RLA-SPC gains in Table 2 are likely real, but the paper should report bootstrap intervals or multiple-seed variance, and the number n of candidates per problem used in Eq. (3) should be stated explicitly.","section":"Tables 1 and 2, Section 6.1"},{"comment":"Pass@5_compliance is a keyword-matching test: a program that prints an affirmative term such as 'pass' or 'satisfy' counts as compliant, with only simple negative-semantic processing. The paper's own failure-mode analysis (Section 6.3) shows that Compliance.Drift is a major failure mode, meaning generated models often violate drift limits even when execution succeeds. A metric that does not parse the actual computed drift or compare it against the 1/550 limit (Figure 9) cannot distinguish a genuine verification conclusion from a hallucinated one. The strict metric inherits this problem, so the headline Pass@5_strict numbers should be interpreted as measuring whether the model prints the expected keywords, not whether the model is engineering-compliant.","section":"Appendix A.2, Engineering Compliance Assessment"}],"minor_comments":[{"comment":"The benchmark is called MBEval in the abstract and BMEval throughout the main text and tables. Please unify the name. Also clarify the relationship among BMBench, CivilInstruct, and BMEval in Section 4; the introduction says BMBench is a 'dataset' with 10,912 samples and 128 cases, which conflates training data and evaluation data.","section":"Abstract vs. Section 4"},{"comment":"The same overview figure appears twice with different captions ('Task formulation ...' and 'Overview of this study'), and the second occurrence also includes text that seems to be a duplicate of the first. This makes the motivation and framework harder to follow. Please consolidate.","section":"Figures 1 and 2"},{"comment":"The ablation text says CodeLlama-7B-Instruct's Pass@1 rises from 50.82% to 55.73% with Part 3 data, while Table 2 reports a Pass@1 of 7.03% for the untrained baseline. These are different models (SFT-only vs. pretrained), but the presentation is confusing because no SFT-only rows appear in Table 2. Please label the configurations explicitly.","section":"Appendix C.2"},{"comment":"The pass@k estimator is standard, but the value of n and k used in the experiments is never stated in Section 6.1 or Table 1. Please specify how many candidates were sampled per problem and how the unbiased estimator was applied in practice, including any early-stopping when n<=k.","section":"Eq. (3)"},{"comment":"The progress reward R_prog uses l_err/L_tot, but it is not stated how l_err is obtained for errors that occur during import or in the OpenSees interpreter. Minor, but relevant for reproducibility.","section":"Section 5.3.3, Eq. (13)"}],"recommendation":"major_revision","confidential_remarks":"The central evaluation needs a substantive fix before publication: independent verification that the printed period is the eigen-solution of the generated model, and ideally a compliance check that parses actual computed values. If the authors add such checks and report confidence intervals, the paper could become a solid contribution. I am not recommending rejection because the dataset, training pipeline, and failure-mode analysis are useful, and the large internal gains are likely reproducible under the current operationalization, but the current evidence does not support the 'physically consistent' claim as stated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid engineering contribution—a 10.9k-sample OpenSeesPy instruction dataset, a 128-case benchmark, and a two-stage SFT+GRPO pipeline with a multi-part reward—and the reported jumps are real on the paper's own terms (e.g., Qwen2.5-Coder-7B from 0.78 to 71.82 Pass@5_strict). The individual pieces are known, but the domain-specific assembly for structural modeling is new and should be useful to people building LLM-based simulation-code generators.\n\nWhat it does well: the data pipeline is thoughtful (API docs, Bug-CoT, physics-informed expert samples), the reward design is pragmatic, and the ablations (SFT vs. SFT+GRPO, Bug-CoT on/off) are sensible. Releasing the code and data is also a real plus.\n\nThe main soft spot is the evaluation protocol. Pass@5_period extracts T_pred from program output with a deterministic matcher and compares it to an empirical-formula ground truth; nothing checks that T_pred came from ops.eigen or from re-simulating the generated model. So a program that prints a plausible empirical-formula value and says \"PASS\" can satisfy all three strict conditions without being physically consistent. The reward function (Eq. 14) uses the same surface quantity, so training and evaluation share the same proxy. That does not look like fraud, but it is a design gap. It is compounded by BMEval being generated from the same parameter ranges and prompt templates as CivilInstruct Part 2, making the test set effectively in-distribution for RLA-SPC. The internal comparison—SFT+GRPO beats the baselines on this benchmark—still holds, but the headline \"physically consistent\" claim is not established.\n\nMinor issues: 128 test cases with no error bars or significance tests; compliance is keyword matching; artifact naming is inconsistent (CivilInstruct, MBEval, BMBench, BMEval all appear); some appendix numbers partly duplicate Table 2 in ambiguous ways.\n\nBottom line: worth engaging with as a domain benchmark and RL recipe for engineering code generation. The physical-consistency evaluation needs a stronger protocol: re-simulate the generated model, compare the eigen-period from that actual model, and add a substitution/injection check to ensure the printed value is derived from the simulation. Without that, the results are real for the proxy, but the central claim is unproven. Send it to peer review—it deserves a serious referee—and expect heavy revision.","headline":"A useful domain-specific RL recipe with eye-catching gains on its own benchmark, but the benchmark's strict-pass metric can be satisfied without the period actually coming from the OpenSees eigen analysis—so the physical-consistency claim is not yet supported.","tokens_in":24300,"tokens_out":2470,"would_cite":false,"duration_ms":25126,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a two-stage recipe—supervised fine-tuning on engineering data followed by reinforcement learning with a sandbox-executed physics reward—raises open 7-8B LLMs from near-zero to 68–77% strict-pass on generating executabl","keywords":["Automatic Building Modeling","OpenSeesPy","physics-constrained reinforcement learning","GRPO","code generation","structural engineering","verification-driven benchmark","LLM alignment"],"falsifier":"Run the RLA-SPC-trained models on building specifications outside CivilInstruct's parameter envelope—say 30-story, 3D, or steel/braced-frame structures—and validate the generated code by comparing predicted periods against full finite-element eigen-solutions and by numerically parsing drift and capacity results from the OpenSees output. If strict-pass rates collapse or executability drops sharply, the claimed physics alignment is specific to the training distribution rather than general physical consistency.","tokens_in":23237,"feed_emoji":"🏗️","tokens_out":8230,"duration_ms":71103,"temperature":0.7,"pith_summary":"This paper is trying to establish that a general-purpose LLM can be turned into a reliable generator of structural modeling code through a domain-specific data pipeline and a two-stage training recipe: first supervised fine-tuning on engineering instruction–code pairs, then reinforcement learning whose reward is computed by actually executing the code in a structural-analysis sandbox and checking the returned fundamental period against a reference. The authors report that this recipe takes open 7-8B-parameter models from near-zero strict-pass rates to 68–77% on their benchmark, where a strict pass means the code runs, reproduces the building's fundamental period within 30%, and prints a code-compliance conclusion. The motivation is that small, open models—not just frontier systems—could automate the labor-intensive step of turning a natural-language building description into a simulation-ready model. The paper also offers BMEval, a verification-driven benchmark, as a way to measure executability, numerical accuracy, and engineering compliance jointly rather than as isolated code-generation metrics.","feed_headline":"Physics reward lifts small LLMs to 77% strict pass on building code","feed_subtitle":"Domain fine-tuning plus sandbox-executed period checks turn 7-8B models into simulation-ready OpenSeesPy generators.","key_machinery":"The load-bearing mechanism is RLA-SPC, a two-stage alignment strategy. Stage one is supervised fine-tuning on CivilInstruct, which teaches OpenSeesPy syntax, API usage, long-code structure, and debugging patterns. Stage two is SPC-GRPO, a variant of group-relative policy optimization in which, for each prompt, several candidate programs are sampled, executed in a sandbox, and scored by the Multi-Granularity Hybrid Reward (MGHR). MGHR weights three signals: a format reward for strict output structure (weight 0.05), an AST-based logical-completeness reward measuring coverage of topology, load, and analysis APIs and penalizing undefined variables (weight 0.25), and an execution reward (weight 0","core_discovery":"The central discovery claim is that physical consistency in LLM-generated modeling code is trainable, not emergent from scale. Using CivilInstruct—about 3,800 API-learning samples, 3,100 parameterized long-code instances, 3,500 bug-fix chain-of-thought samples, and 512 physics-annotated expert codes—the authors first fine-tune a base model, then apply SPC-GRPO, a group-relative policy optimization that rewards format compliance, logical completeness via tiered API coverage, and, above all, sandbox execution success measured by the relative error of the predicted fundamental period. On BMEval's 128 cases, CodeLlama-7B's strict pass rate at k=5 rises from 0.00% to 67.73%, Qwen2.5-Coder-7B from","pith_inferences":["Because BMEval ground-truth periods come from empirical formulas and the training data are drawn from the same parameter ranges and prompt templates, the strict-pass numbers should be read as in-distribution proficiency; out-of-distribution structural types (taller buildings, 3D or non-frame systems, steel or braced frames) remain untested, so I would not extrapolate to general engineering compete","The MGHR recipe—formatting, static AST completeness, and sandbox execution against a scalar physical quantity—is a transferable template for any simulator-backed code-generation domain (CFD, heat transfer, circuit simulation) where a measurable output can stand in for physical correctness; that is my extension, not the paper's.","If the compliance signal is keyword-based and the period signal is a single scalar, a model could learn to game both without true structural reasoning; parsing drift and capacity results numerically from the OpenSees output would settle whether the gains are physical or textual."],"forward_implications":["If the results hold, open-weights models in the 7-8B range can be turned into usable structural-modeling code generators with two fine-tuning stages, without relying on proprietary frontier models.","The strict-pass criterion bundles executability, numerical accuracy (fundamental period within 30%), and an explicit compliance conclusion, so the metric itself models the full engineering workflow rather than just syntax.","The failure-mode analysis—syntax errors, drift violations, and eigenvector setup—identifies where LLM-generated modeling code breaks, giving concrete targets for data curation and reward design.","The two-stage design implies a division of labor: SFT supplies syntax and API competence, while the physics-constrained RL stage suppresses physically implausible outputs."],"fun_headline_variants":["RL fine-tune lifts small LLMs to 68% physics-valid code","Small LLMs hit 77% simulation-readiness via SPC-GRPO","Verification-driven RL makes LLMs write executable models","From 0% to 68%: LLMs learn physical consistency"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing assumption is that BMEval is an independent, valid measure of physical correctness: its ground-truth periods come from empirical formulas and its compliance judgement from keyword matching, and its sampled parameter ranges and prompt templates overlap with the training data, so if those proxies do not reflect true structural behavior, the strict-pass gains overstate real engineering capability.","fun_headline_variants_meta":{"raw":{"variants":["RL fine-tune lifts small LLMs to 68% physics-valid code","Small LLMs hit 77% simulation-readiness via SPC-GRPO","Verification-driven RL makes LLMs write executable models","From 0% to 68%: LLMs learn physical consistency"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000259,"raw_usage":{"total_tokens":1419,"prompt_tokens":740,"completion_tokens":679,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":484,"completion_tokens_details":{"reasoning_tokens":600}},"tokens_in":484,"tokens_out":679,"duration_ms":6914,"temperature":1.0,"reasoning_tokens":600,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T03:54:28.901371+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the RLA-SPC-trained models on building specifications outside CivilInstruct's parameter envelope—say 30-story, 3D, or steel/braced-frame structures—and validate the generated code by comparing predicted periods against full finite-element eigen-solutions and by numerically parsing drift and capacity results from the OpenSees output. If strict-pass rates collapse or executability drops sharply, the claimed physics alignment is specific to the training distribution rather than general physical consistency.","supporting_citations":[],"review_version":1}