Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

The paper claims that a two-stage recipe—supervised fine-tuning on engineering data followed by reinforcement learning with a sandbox-executed physics reward—raises open 7-8B LLMs from near-zero to 68–77% strict-pass on generating executabl

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 03:54 UTC pith:CC54ZBLF

load-bearing objection A useful domain-specific RL recipe with eye-catching gains on its own benchmark, but the benchmark's strict-pass metric can be satisfied without the period actually coming from the OpenSees eigen analysis—so the physical-consistency claim is not yet supported. the 4 major comments →

arxiv 2602.07083 v2 pith:CC54ZBLF submitted 2026-02-06 cs.SE cs.AI

Rethinking Scientific Modeling: Toward Physically Consistent and Simulation-Executable Programmatic Generation

Yongqing Jiang , Jianze Wang , Zhiqi Shen , Zhenghong Lin , Jiayuan Wang , Yijian Yang , Kaoshan Dai , Haoran Luo This is my paper
classification cs.SE cs.AI
keywords Automatic Building ModelingOpenSeesPyphysics-constrained reinforcement learningGRPOcode generationstructural engineeringverification-driven benchmarkLLM alignment
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper is trying to establish that a general-purpose LLM can be turned into a reliable generator of structural modeling code through a domain-specific data pipeline and a two-stage training recipe: first supervised fine-tuning on engineering instruction–code pairs, then reinforcement learning whose reward is computed by actually executing the code in a structural-analysis sandbox and checking the returned fundamental period against a reference. The authors report that this recipe takes open 7-8B-parameter models from near-zero strict-pass rates to 68–77% on their benchmark, where a strict pass means the code runs, reproduces the building's fundamental period within 30%, and prints a code-compliance conclusion. The motivation is that small, open models—not just frontier systems—could automate the labor-intensive step of turning a natural-language building description into a simulation-ready model. The paper also offers BMEval, a verification-driven benchmark, as a way to measure executability, numerical accuracy, and engineering compliance jointly rather than as isolated code-generation metrics.

Core claim

The central discovery claim is that physical consistency in LLM-generated modeling code is trainable, not emergent from scale. Using CivilInstruct—about 3,800 API-learning samples, 3,100 parameterized long-code instances, 3,500 bug-fix chain-of-thought samples, and 512 physics-annotated expert codes—the authors first fine-tune a base model, then apply SPC-GRPO, a group-relative policy optimization that rewards format compliance, logical completeness via tiered API coverage, and, above all, sandbox execution success measured by the relative error of the predicted fundamental period. On BMEval's 128 cases, CodeLlama-7B's strict pass rate at k=5 rises from 0.00% to 67.73%, Qwen2.5-Coder-7B from

What carries the argument

The load-bearing mechanism is RLA-SPC, a two-stage alignment strategy. Stage one is supervised fine-tuning on CivilInstruct, which teaches OpenSeesPy syntax, API usage, long-code structure, and debugging patterns. Stage two is SPC-GRPO, a variant of group-relative policy optimization in which, for each prompt, several candidate programs are sampled, executed in a sandbox, and scored by the Multi-Granularity Hybrid Reward (MGHR). MGHR weights three signals: a format reward for strict output structure (weight 0.05), an AST-based logical-completeness reward measuring coverage of topology, load, and analysis APIs and penalizing undefined variables (weight 0.25), and an execution reward (weight 0

Load-bearing premise

The load-bearing assumption is that BMEval is an independent, valid measure of physical correctness: its ground-truth periods come from empirical formulas and its compliance judgement from keyword matching, and its sampled parameter ranges and prompt templates overlap with the training data, so if those proxies do not reflect true structural behavior, the strict-pass gains overstate real engineering capability.

What would settle it

Run the RLA-SPC-trained models on building specifications outside CivilInstruct's parameter envelope—say 30-story, 3D, or steel/braced-frame structures—and validate the generated code by comparing predicted periods against full finite-element eigen-solutions and by numerically parsing drift and capacity results from the OpenSees output. If strict-pass rates collapse or executability drops sharply, the claimed physics alignment is specific to the training distribution rather than general physical consistency.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the results hold, open-weights models in the 7-8B range can be turned into usable structural-modeling code generators with two fine-tuning stages, without relying on proprietary frontier models.
  • The strict-pass criterion bundles executability, numerical accuracy (fundamental period within 30%), and an explicit compliance conclusion, so the metric itself models the full engineering workflow rather than just syntax.
  • The failure-mode analysis—syntax errors, drift violations, and eigenvector setup—identifies where LLM-generated modeling code breaks, giving concrete targets for data curation and reward design.
  • The two-stage design implies a division of labor: SFT supplies syntax and API competence, while the physics-constrained RL stage suppresses physically implausible outputs.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because BMEval ground-truth periods come from empirical formulas and the training data are drawn from the same parameter ranges and prompt templates, the strict-pass numbers should be read as in-distribution proficiency; out-of-distribution structural types (taller buildings, 3D or non-frame systems, steel or braced frames) remain untested, so I would not extrapolate to general engineering compete
  • The MGHR recipe—formatting, static AST completeness, and sandbox execution against a scalar physical quantity—is a transferable template for any simulator-backed code-generation domain (CFD, heat transfer, circuit simulation) where a measurable output can stand in for physical correctness; that is my extension, not the paper's.
  • If the compliance signal is keyword-based and the period signal is a single scalar, a model could learn to game both without true structural reasoning; parsing drift and capacity results numerically from the OpenSees output would settle whether the gains are physical or textual.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper formalizes Automatic Building Modeling (AutoBM) as a code-generation task in which an LLM must produce OpenSeesPy models that are executable, physically consistent, and compliant with engineering constraints. It introduces CivilInstruct, a 10,912-sample instruction dataset spanning API tutorials, parameterized long-code generation, debugging chain-of-thought data, and physics-informed expert examples; BMEval, a 128-case benchmark that scores generated programs on executability, fundamental-period accuracy, and keyword-based compliance; and RLA-SPC, a two-stage pipeline that applies supervised fine-tuning followed by group-relative policy optimization with a multi-granularity hybrid reward. Experiments on BMEval report large before/after gains for three open models, e.g., CodeLlama-7B-Instruct rising from Pass@5_strict 0.00 to 67.73 and Seed-Coder-8B-R from 0.78 to 77.14, exceeding the strict-pass rates of the commercial models evaluated.

Significance. If the evaluation protocol is valid, the paper makes a useful practical contribution: it demonstrates that relatively small open LLMs can be aligned to generate long, executable OpenSeesPy scripts with high pass rates on a domain-specific benchmark, and it releases the dataset and code. The two-stage training recipe and the detailed failure-mode analysis are valuable for future work on physics-aware code generation. However, the central claim that the models achieve 'physically consistent, simulation-executable' generation is not currently supported by the evidence, because the BMEval strict criterion does not verify that the printed period is the eigen-solution of the generated OpenSees model, and the compliance check is only keyword matching. The reported gains are real under the paper's operationalization, but their engineering significance depends on an independence and validity of the benchmark that is not established.

major comments (4)
  1. [Section 4.2 / Appendix A.2, Eq. (16)] Pass@5_period extracts T_pred from 'program outputs using a deterministic, rule-based matching method' and compares it to an empirical-formula ground truth. Nothing in the protocol requires T_pred to be computed by the OpenSees model (e.g., from ops.eigen or ops.getEigenvalues). A candidate program that executes, prints a period obtained from the same empirical formula used to generate the ground truth, and prints 'PASS' satisfies all three strict conditions even if the code never runs modal analysis or is physically inconsistent. The paper should either (a) re-execute each generated program in a separate environment and extract the fundamental period from the simulation results, (b) perform an AST/dataflow check that the printed value is derived from the eigen-solver, or (c) add substitution/injection tests where a manually corrupted model with the same printed period fails. As written,
  2. [Section 5.3.3, Eq. (14) vs. Section 4.2, Eq. (16)] The training reward R_phy uses the relative error between the extracted period and the empirical-formula ground truth, with tolerance bins (Eq. 15), and the evaluation metric Pass@5_period uses the same relative-error comparison with tolerance 0.30 (Eq. 16). This is not formal circularity, but it means the reward and the headline metric are two views of the same surface quantity, extracted from program output with the same rule-based method. The reported improvements are therefore expected under reward optimization and do not independently validate physical consistency. An independent re-simulation check, or at least a held-out set with different parameter ranges and prompt templates, is needed to separate genuine physics understanding from optimization of the extracted-output statistic.
  3. [Tables 1 and 2, Section 6.1] All pass@k numbers are computed on 128 samples with no confidence intervals, no standard deviations across training seeds, and no significance tests. For a 50% observed rate the standard error is roughly 4.4 percentage points, so the small differences among commercial models in Table 1 (e.g., Pass@5_strict 16.62 vs. 18.84) are not interpretable. The large RLA-SPC gains in Table 2 are likely real, but the paper should report bootstrap intervals or multiple-seed variance, and the number n of candidates per problem used in Eq. (3) should be stated explicitly.
  4. [Appendix A.2, Engineering Compliance Assessment] Pass@5_compliance is a keyword-matching test: a program that prints an affirmative term such as 'pass' or 'satisfy' counts as compliant, with only simple negative-semantic processing. The paper's own failure-mode analysis (Section 6.3) shows that Compliance.Drift is a major failure mode, meaning generated models often violate drift limits even when execution succeeds. A metric that does not parse the actual computed drift or compare it against the 1/550 limit (Figure 9) cannot distinguish a genuine verification conclusion from a hallucinated one. The strict metric inherits this problem, so the headline Pass@5_strict numbers should be interpreted as measuring whether the model prints the expected keywords, not whether the model is engineering-compliant.
minor comments (5)
  1. [Abstract vs. Section 4] The benchmark is called MBEval in the abstract and BMEval throughout the main text and tables. Please unify the name. Also clarify the relationship among BMBench, CivilInstruct, and BMEval in Section 4; the introduction says BMBench is a 'dataset' with 10,912 samples and 128 cases, which conflates training data and evaluation data.
  2. [Figures 1 and 2] The same overview figure appears twice with different captions ('Task formulation ...' and 'Overview of this study'), and the second occurrence also includes text that seems to be a duplicate of the first. This makes the motivation and framework harder to follow. Please consolidate.
  3. [Appendix C.2] The ablation text says CodeLlama-7B-Instruct's Pass@1 rises from 50.82% to 55.73% with Part 3 data, while Table 2 reports a Pass@1 of 7.03% for the untrained baseline. These are different models (SFT-only vs. pretrained), but the presentation is confusing because no SFT-only rows appear in Table 2. Please label the configurations explicitly.
  4. [Eq. (3)] The pass@k estimator is standard, but the value of n and k used in the experiments is never stated in Section 6.1 or Table 1. Please specify how many candidates were sampled per problem and how the unbiased estimator was applied in practice, including any early-stopping when n<=k.
  5. [Section 5.3.3, Eq. (13)] The progress reward R_prog uses l_err/L_tot, but it is not stated how l_err is obtained for errors that occur during import or in the OpenSees interpreter. Minor, but relevant for reproducibility.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 0 invented entities

No new physical entities, forces, or dimensions are postulated. The new artifacts (CivilInstruct, BMEval, RLA-SPC, MGHR) are datasets/methods, not invented entities in the graviton-problem sense. The load-bearing assumptions are domain-level: OpenSeesPy fidelity, empirical-formula ground truth, and train/test independence.

free parameters (6)
  • MGHR weights (w_fmt, w_ast, w_exec) = 0.05, 0.25, 0.70
    Hand-set in Eq. (8); they determine how format, AST coverage, and execution rewards are blended, and the reported gains depend on this balance.
  • R_phy period tolerance bins = epsilon <= 0.10 -> 1.00; <=0.20 -> 0.90; <=0.40 -> 0.80; else 0.70
    Hand-chosen thresholds in Eq. (15) define what counts as physically consistent during RL; no sensitivity analysis is reported.
  • Progress reward coefficients alpha_base, beta_step = not specified
    Eq. (13) uses these for runtime-failure partial credit; values are never reported.
  • KL weight beta, group size G = not specified
    Algorithm 1 and Eq. (7) depend on these GRPO hyperparameters; no values or sweeps are given.
  • AST tier weights alpha_k and undefined-variable penalty lambda = not specified
    Eq. (11) uses three-tier coverage weights and an undefined-variable penalty; values are absent.
  • BMEval period tolerance epsilon (eval) = 0.30
    Eq. (16) sets the period pass threshold; it is the same physical concept as the reward tolerance but not the identical value, and it directly controls all headline strict-pass numbers.
axioms (4)
  • domain assumption OpenSeesPy is a valid proxy for building structural dynamics, and execution success plus period closeness implies physical meaningfulness.
    The entire reward and evaluation pipeline treats sandbox execution and fundamental-period output as ground truth (Section 5.3.3, Appendix A.2).
  • domain assumption Empirical building-code formulas give correct ground-truth fundamental periods for the 128 BMEval samples.
    Section 4.2: 'empirical formulas were used to compute the fundamental period... as ground-truth reference values'; if these formulas are wrong for the sample space, all period/strict metrics inherit the error.
  • domain assumption BMEval test samples are independent of the CivilInstruct training distribution.
    BMEval samples are randomly generated over similar parameter ranges as CivilInstruct Part 2 (Table 3); independence is assumed but not demonstrated.
  • standard math The unbiased pass@k estimator (Eq. 3) and the GRPO policy-gradient update are correct.
    Standard results reused without proof; acceptable, but the original sources for GRPO and the unbiased pass@k estimator are not cited.

pith-pipeline@v1.3.0-alltime-deepseek · 22984 in / 15287 out tokens · 147910 ms · 2026-08-03T03:54:28.901371+00:00 · methodology

0 comments
read the original abstract

Structural modeling is a fundamental component of computational engineering science, in which even minor physical inconsistencies or specification violations may invalidate downstream simulations. The potential of large language models (LLMs) for automatic generation of modeling code has been demonstrated. However, non-executable or physically inconsistent outputs remain prevalent under stringent engineering constraints. A framework for physics-consistent automatic building modeling is therefore proposed, integrating domain knowledge construction, constraint-oriented model alignment, and verification-driven evaluation. CivilInstruct is introduced as a domain-specific dataset that formalizes structural engineering knowledge and constraint reasoning to enable simulation-ready model generation. A two-stage fine-tuning strategy is further employed to enforce constraint satisfaction and application programming interface compliance, substantially reducing hallucinated and non-conforming outputs. MBEval is presented as a verification-driven benchmark that evaluates executability and structural dynamics consistency through closed-loop validation. Experimental results show consistent improvements over baselines across rigorous verification metrics. Our code is available at https://github.com/Jovanqing/AutoBM.

Figures

Figures reproduced from arXiv: 2602.07083 by Haoran Luo, Jianze Wang, Jiayuan Wang, Kaoshan Dai, Yijian Yang, Yongqing Jiang, Zhenghong Lin, Zhiqi Shen.

Figure 1
Figure 1. Figure 1: Task formulation of llm-driven automatic building [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 1
Figure 1. Figure 1: Overview of this study: Automatic Modeling of Building (AutoMB) task, its motivation, data foundations, and the associated method Physics-Aware LLM Framework Geometry & Topology Modeling [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: The definition and overview of the AutoBM task. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Overview of the CivilInstruct construction procedure. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The results of the report generation tasks. (a) LLM performance on the BMEval benchmark. (b) & (c) Evolution of [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: The Limitations of AI-Generated AutoBM Task. An analysis based on 640 sets of modeling code generated by Gemini [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Building Attributes Correlation Matrix B Prompt used in AutoBM B.1 Prompt Template for Structural Design and Modeling Code Generation To generate analysis-ready reinforced concrete (RC) frame mod￾els in a consistent and code-compliant manner, we adopt a rule￾constrained, prompt-based design workflow (as shown in [PITH_FULL_IMAGE:figures/full_fig_p011_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Prompt template for structural scheme design [PITH_FULL_IMAGE:figures/full_fig_p012_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Prompt template for structural modeling programming [PITH_FULL_IMAGE:figures/full_fig_p013_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Prompt template for dataset construction based on the OpenSeesPy API [PITH_FULL_IMAGE:figures/full_fig_p014_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Prompt template for instruction rewriting [PITH_FULL_IMAGE:figures/full_fig_p014_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. PowerAtlas: Towards Electricity-Computing Co-Scheduling for Power Systems

    cs.LG 2026-07 conditional novelty 6.0

    SFT plus feasibility-aware GRPO lets 3B LLMs produce electricity–computing co-schedules that are far more grid-feasible and cheaper than untrained or frontier zero-shot baselines on ECBench.

Reference graph

Works this paper leans on

55 extracted references · 13 canonical work pages · cited by 1 Pith paper · 2 internal anchors

  1. [1]

    2023.Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Flo- rencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, and Shyamal Anadkat. 2023.Gpt-4 technical report. arXiv:2303.08774 doi:10.48550/ arXiv.2303.08774

    Pith/arXiv arXiv doi:10.48550/arxiv.2303.08774 2023

  2. [2]

    2026.The Llama 4 Herd: Architecture, Training, Evaluation, and Deployment Notes

    Aaron Adcock, Aayushi Srivastava, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pande, Abhinav Pandey, Abhinav Sharma, Abhishek Kadian, Abhishek Kumawat, and Adam Kelsey. 2026.The Llama 4 Herd: Architecture, Training, Evaluation, and Deployment Notes. arXiv:2601.11659

    arXiv 2026

  3. [3]

    Carlos Avila, Daniel Ilbay, Paola Tapia, and David Rivera. 2025. Toward Responsi- ble AI in High-Stakes Domains: A Dataset for Building Static Analysis with LLMs in Structural Engineering.Data10, 11 (2025), 169. doi:10.3390/data10110169

    work page doi:10.3390/data10110169 2025

  4. [4]

    Mehmet Baris Batukan and Oya Mercan. 2024. ModularBuildingPy: A Python- Based Numerical Modeling and Analysis Tool for Volumetric Modular Steel Buildings. InCanadian Society of Civil Engineering Annual Conference. Springer, 309–320. doi:10.1007/978-3-032-01078-0_26

    work page doi:10.1007/978-3-032-01078-0_26 2024

  5. [5]

    Arihant Bedagkar, Sayandeep Mitra, Raveendra Medicherla, Ravindra Naik, and Samiran Pal. 2025. LLM Driven Smart Assistant for Data Mapping. In 2025 IEEE/ACM 47th International Conference on Software Engineering: Soft- ware Engineering in Practice (ICSE-SEIP). IEEE, 181–191. doi:10.1109/ICSE- SEIP66354.2025.00022

    arXiv 2025

  6. [6]

    2025.LLM-Driven Stationarity-A ware Expert Demonstrations for Multi-Agent Reinforcement Learning in Mobile Systems

    Arihant Bedagkar, Sayandeep Mitra, Raveendra Medicherla, Ravindra Naik, and Samiran Pal. 2025.LLM-Driven Stationarity-A ware Expert Demonstrations for Multi-Agent Reinforcement Learning in Mobile Systems. arXiv:2511.19368 doi:10. 48550/arXiv.2511.19368

    doi:10.48550/arxiv.2511.19368 2025

  7. [7]

    Patrick Beutler, Manuel Biedermann, Urs Hofmann, Ralph Rosenbauer, and Mirko Meboldt. 2022. Automated Design Workflow for Structural Nodes of Space Frame Structures.Procedia CIRP109 (2022), 419–424. doi:10.1016/j.procir.2022.05.272

    work page doi:10.1016/j.procir.2022.05.272 2022

  8. [8]

    Nicolas Bougie and Narimawa Watanabe. 2025. Citysim: Modeling urban behav- iors and city dynamics with large-scale llm-driven agent simulation. InProceed- ings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track. 215–229. doi:10.18653/v1/2025.emnlp-industry.15

    doi:10.18653/v1/2025.emnlp-industry.15 2025

  9. [9]

    2023.Multi-agent con- sensus seeking via large language models

    Huaben Chen, Wenkang Ji, Lufeng Xu, and Shiyu Zhao. 2023.Multi-agent con- sensus seeking via large language models. arXiv:2310.20151 doi:10.48550/arXiv. 2310.20151

    Pith/arXiv arXiv doi:10.48550/arxiv.2310.20151 2023

  10. [10]

    2025.Gemini 2.5: Pushing the frontier with advanced reasoning, multi- modality, long context, and next generation agentic capabilities

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, and Evan Rosen. 2025.Gemini 2.5: Pushing the frontier with advanced reasoning, multi- modality, long context, and next generation agentic capabilities. arXiv:2507.06261 doi:10.48550/arXiv.2507.06261

    Pith/arXiv arXiv doi:10.48550/arxiv.2507.06261 2025

  11. [11]

    Ahmet Bahaddin Ersoz, Onur Pekcan, and Emre Akbas. 2024. AIDCON: An Aerial Image Dataset and Benchmark for Construction Machinery.Remote Sensing16, 17 (2024), 3295. doi:10.3390/rs16173295

    work page doi:10.3390/rs16173295 2024

  12. [12]

    2025.A lightweight large language model-based multi-agent system for 2d frame structural analysis

    Ziheng Geng, Jiachen Liu, Ran Cao, Lu Cheng, Haifeng Wang, and Minghui Cheng. 2025.A lightweight large language model-based multi-agent system for 2d frame structural analysis. arXiv:2510.05414 doi:10.48550/arXiv.2510.05414

    doi:10.48550/arxiv.2510.05414 2025

  13. [13]

    Vinícius Pereira Gonçalves, Andre Luiz Marques Serrano, Gabriel Arquelau Pi- menta Rodrigues, Matheus Noschang de Oliveira, Rodolfo Ipolito Meneguette, Guilherme Dantas Bispo, Maria Gabriela Mendonça Peixoto, and Geraldo Pereira Rocha Filho. 2025. An Empirically Validated Framework for Auto- mated and Personalized Residential Energy-Management Integrating...

    work page doi:10.3390/en18143744 2025

  14. [14]

    Xingquan Guan, Henry Burton, and Thomas Sabol. 2020. Python-based compu- tational platform to automate seismic design, nonlinear structural model con- struction and analysis of steel moment resisting frames.Engineering Structures 224 (2020), 111199. doi:10.1016/j.engstruct.2020.111199

    arXiv 2020

  15. [15]

    2024.DeepSeek-Coder: When the Large Lan- guage Model Meets Programming–The Rise of Code Intelligence

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guant- ing Chen, Xiao Bi, Yu Wu, and YK Li. 2024.DeepSeek-Coder: When the Large Lan- guage Model Meets Programming–The Rise of Code Intelligence. arXiv:2401.14196 doi:10.48550/arXiv.2401.14196

    Pith/arXiv arXiv doi:10.48550/arxiv.2401.14196 2024

  16. [16]

    Pengwei Guo, Weina Meng, and Yi Bao. 2024. Knowledge graph-guided data- driven design of ultra-high-performance concrete (UHPC) with interpretability and physicochemical reaction discovery capability.Construction and Building Materials430 (2024), 136502. doi:10.1016/j.conbuildmat.2024.136502

    arXiv 2024

  17. [17]

    Mahdi Heshmati, Alireza Khatami, and Hamzeh Shakib. 2021. FEMA P695 methodology for safety margin evaluation of steel moment resisting frames subjected to near-field and far-field records.SN Applied Sciences3, 2 (2021), 185. doi:10.1007/s42452-021-04230-2

    work page doi:10.1007/s42452-021-04230-2 2021

  18. [18]

    Siming Huang, Tianhao Cheng, Jason Klein Liu, Weidi Xu, Jiaran Hao, Liuyihan Song, Yang Xu, Jian Yang, Jiaheng Liu, Chenchen Zhang, et al. 2025. Opencoder: The open cookbook for top-tier code large language models. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 33167–33193. doi:10.18653/v...

    doi:10.18653/v1/2025.acl-long.1591 2025

  19. [19]

    Jaewon Jeoung, Seunghoon Jung, and Taehoon Hong. 2025. Zero-shot framework for construction equipment task monitoring.Computer-Aided Civil and Infras- tructure Engineering(2025). doi:10.1111/mice.13506DigitalObjectIdentifier(DOI)

    work page doi:10.1111/mice.13506digitalobjectidentifier(doi 2025

  20. [20]

    Yongqing Jiang, Jianze Wang, Xinyi Shen, and Kaoshan Dai. 2025. Large language model for post-earthquake structural damage assessment of buildings.Computer- Aided Civil and Infrastructure Engineering40, 31 (2025), 6324–6342. doi:10.1111/ mice.70010DigitalObjectIdentifier(DOI)

    2025

  21. [21]

    Yongqing Jiang, Jianze Wang, Xinyi Shen, Kaoshan Dai, and Qingzi Ge. 2026. Multitask unified large vision-language model for post-earthquake structural damage assessment of buildings.Automation in Construction182 (2026), 106720. doi:10.1016/j.autcon.2025.106720

    arXiv 2026

  22. [22]

    Taegu Kim, Tae Sup Yun, and Hyoung Suk Suh. 2025. Can ChatGPT implement finite element models for geotechnical engineering applications?International Journal for Numerical and Analytical Methods in Geomechanics49, 6 (2025), 1747–

    2025

  23. [23]

    Maximilian Kretzschmar, Alexander Gundermann, Jan Mehlstäubl, Alejan- dro Pradas Gómez, Bernhard Saske, and Kristin Paetzold-Byhain. 2025. Using Role-Specialized LLM Agents for Workflow Execution in a Product Development Setting. In2025 9th International Conference on Inventive Systems and Control (ICISC). IEEE, 642–650. doi:10.1109/ICISC65841.2025.11188152

    arXiv 2025

  24. [24]

    Hao Li, Rongzheng Yang, Shuangshuang Xu, Yao Xiao, and Hongyu Zhao. 2024. Intelligent checking method for construction schemes via fusion of knowledge graph and large language models.Buildings14, 8 (2024), 2502. doi:10.3390/ buildings14082502

    2024

  25. [25]

    DrafterBench: Benchmarking Large Language Models for Tasks Automation in Civil Engineering

    Yinsheng Li, Zhen Dong, and Yi Shao. 2025.Drafterbench: Benchmarking large language models for tasks automation in civil engineering. arXiv:2507.11527 doi:10.48550/arXiv.2507.11527

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2507.11527 2025

  26. [26]

    2025.Integrating Large Language Models for Automated Structural Analysis

    Haoran Liang, Mohammad Talebi Kalaleh, and Qipei Mei. 2025.Integrating Large Language Models for Automated Structural Analysis. arXiv:2504.09754 doi:10.48550/arXiv.2504.09754

    Pith/arXiv arXiv doi:10.48550/arxiv.2504.09754 2025

  27. [27]

    2025.Au- tomating Structural Engineering Workflows with Large Language Model Agents

    Haoran Liang, Yufa Zhou, Mohammad Talebi Kalaleh, and Qipei Mei. 2025.Au- tomating Structural Engineering Workflows with Large Language Model Agents. arXiv:2510.11004 doi:10.48550/arXiv.2510.11004

    work page doi:10.48550/arxiv.2510.11004 2025

  28. [28]

    DG Lignos, F Zareian, and H Krawinkler. 2010. A steel component database for deterioration modeling of steel beams with RBS under cyclic loading. InStructures Congress 2010. 1241–1252. doi:10.1061/41130(369)113

    work page doi:10.1061/41130(369)113 2010

  29. [29]

    2025.Kbqa-o1: Agentic knowledge base question answering with monte carlo tree search

    Haoran Luo, Yikai Guo, Qika Lin, Xiaobao Wu, Xinyu Mu, Wenhao Liu, Meina Song, Yifan Zhu, Luu Anh Tuan, et al. 2025.Kbqa-o1: Agentic knowledge base question answering with monte carlo tree search. arXiv:2501.18922 doi:10.48550/ arXiv.2501.18922

    Pith/arXiv arXiv doi:10.48550/arxiv.2501.18922 2025

  30. [30]

    Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie-Yan Liu. 2022. BioGPT: generative pre-trained transformer for biomedical text generation and mining.Briefings in bioinformatics23, 6 (2022), bbac409. doi:10.1093/bib/bbac409

    doi:10.1093/bib/bbac409 2022

  31. [31]

    Iolanda Nuzzo, Francesca Ciliento, and Nicola Caterino. 2020. DIBRAST: a computer-aided seismic design procedure for frame structures equipped with hysteretic devices.Frontiers in Built Environment6 (2020), 13. doi:10.3389/fbuil. 2020.00013

    arXiv 2020

  32. [32]

    Sandeep Pandey, Ran Xu, Wenkang Wang, and Xu Chu. 2025. OpenFOAMGPT: A retrieval-augmented large language model (LLM) agent for OpenFOAM-based computational fluid dynamics.Physics of Fluids37, 3 (2025). doi:10.1063/5.0257555

    doi:10.1063/5.0257555 2025

  33. [33]

    Sizhong Qin, Hong Guan, Wenjie Liao, Yi Gu, Zhe Zheng, Hongjing Xue, and Xinzheng Lu. 2024. Intelligent design and optimization system for shear wall structures based on large language models and generative artificial intelligence. Journal of Building Engineering95 (2024), 109996. doi:10.1016/j.jobe.2024.109996

    arXiv 2024

  34. [34]

    Md Nishat Raihan, Antonios Anastasopoulos, and Marcos Zampieri. 2025. mHumanEval-a multilingual benchmark to evaluate large language models for code generation. InProceedings of the 2025 Conference of the Nations of the Amer- icas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 11432–11461. do...

    doi:10.18653/v1/2025.naacl- 2025

  35. [35]

    Shima Sdeghzadeh, Cristoforo Demartino, and Camillo Nuti. 2025. Modeling of the Colle Castino Bridge in OpenSeesPy for Structural and Seismic Analysis. Proceedings of the 2024 Eurasian OpenSees Days638 (2025), 97. doi:10.1007/978-3- 031-90690-9_9

    doi:10.1007/978-3- 2025

  36. [36]

    2025.Seed-coder: Let the code model curate data for itself

    ByteDance Seed, Yuyu Zhang, Jing Su, Yifan Sun, Chenguang Xi, Xia Xiao, Shen Zheng, Anxiang Zhang, Kaibo Liu, and Daoguang Zan. 2025.Seed-coder: Let the code model curate data for itself. arXiv:2506.03524 doi:10.48550/arXiv.2506.03524

    Pith/arXiv arXiv doi:10.48550/arxiv.2506.03524 2025

  37. [37]

    Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, et al . 2025. Toward expert-level medical question answering with large language models. Nature Medicine31, 3 (2025), 943–950. doi:10.1038/s41591-024-03423-7

    doi:10.1038/s41591-024-03423-7 2025

  38. [38]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. InAdvances in Neural Information Processing Systems, I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran Associates, Inc

    2017

  39. [39]

    2025.Som-1k: A thousand-problem benchmark dataset for strength of materials

    Qixin Wan, Zilong Wang, Jingwen Zhou, Wanting Wang, Ziheng Geng, Jiachen Liu, Ran Cao, Minghui Cheng, and Lu Cheng. 2025.Som-1k: A thousand-problem benchmark dataset for strength of materials. arXiv:2509.21079 doi:10.48550/arXiv. 2509.21079 KDD ’26, 2026, Jeju, South Korea Yongqing Jiang, Jianze Wang, Zhiqi Shen, Zhenghong Lin, Jiayuan Wang, Yijian Yang, ...

    doi:10.48550/arxiv.2509.21079 2025

  40. [40]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. InAdvances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35. Curran Associates, Inc., 24824–24837

    2022

  41. [41]

    Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. 2023. Bloomberggpt: A large language model for finance. arXiv:2303.17564 doi:10.48550/ arXiv.2303.17564

    Pith/arXiv arXiv doi:10.48550/arxiv.2303.17564 2023

  42. [42]

    2024.WizardLM: Empowering large pre-trained language models to follow complex instructions

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. 2024.WizardLM: Empowering large pre-trained language models to follow complex instructions. arXiv:2304.12244 doi:10.48550/arXiv.2304.12244

    Pith/arXiv arXiv doi:10.48550/arxiv.2304.12244 2024

  43. [43]

    Xiaorui Xue, Jiansong Zhang, and Yunfeng Chen. 2024. Question-answering framework for building codes using fine-tuned and distilled pre-trained trans- former models.Automation in Construction168 (2024), 105730. doi:10.1016/j. autcon.2024.105730

    arXiv 2024

  44. [44]

    Yexiang Yan and Yazhou Xie. 2025. opstool: A Python library for OpenSeesPy analysis automation, streamlined pre-and post-processing, and enhanced data visualization.SoftwareX30 (2025), 102126. doi:10.1016/j.softx.2025.102126

    arXiv 2025

  45. [45]

    2025.Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, and Chenxu Lv. 2025.Qwen3 technical report. arXiv:2505.09388 doi:10.48550/arXiv.2505.09388

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.09388 2025

  46. [46]

    Fan Yang and Jiansong Zhang. 2024. Prompt-based automation of building code information transformation for compliance checking.Automation in Construction 168 (2024), 105817. doi:10.1016/j.autcon.2024.105817

    arXiv 2024

  47. [47]

    2025.MAS-GPT: Training LLMs to build LLM-based multi-agent systems

    Rui Ye, Shuo Tang, Rui Ge, Yaxin Du, Zhenfei Yin, Siheng Chen, and Jing Shao. 2025.MAS-GPT: Training LLMs to build LLM-based multi-agent systems. arXiv:2503.03686 doi:10.48550/arXiv.2503.03686

    Pith/arXiv arXiv doi:10.48550/arxiv.2503.03686 2025

  48. [48]

    Zhaojian Yu, Yilun Zhao, Arman Cohan, and Xiao-Ping Zhang. 2025. HumanEval pro and MBPP pro: Evaluating large language models on self-invoking code generation task. InFindings of the Association for Computational Linguistics: ACL

    2025

  49. [49]

    F Zareian, DG Lignos, and H Krawinkler. 2010. Evaluation of seismic collapse performance of steel special moment resisting frames using FEMA P695 (ATC-63) methodology. InStructures Congress 2010. 1275–1286. doi:10.1061/41130(369)116

    work page doi:10.1061/41130(369)116 2010

  50. [50]

    Renaissance of Literate Programming in the Era of LLMs: Enhancing LLM-Based Code Generation in Large-Scale Projects

    Wuyang Zhang, Yansong Li, Zeyu Dong, Yu Wu, Yingyao Zhou, Duolei Wang, Songsirou Xing, Chichun Zhou, and Da Shen. 2024.Renaissance of Literate Programming in the Era of LLMs: Enhancing LLM-Based Code Generation in Large- Scale Projects. arXiv:2502.17441 doi:10.48550/arXiv.2502.17441

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2502.17441 2024

  51. [51]

    Zhu Zhongming, Lu Linong, Yao Xiaona, Zhang Wangqiang, and Liu Wei. 2009. Ministry of housing and urban-rural development of the people’s republic of china. (2009)

    2009

  52. [52]

    Ting Zhou, Kezhao Sun, Zhihua Chen, Zhexi Yang, and Hongbo Liu. 2023. Auto- mated optimum design of light steel frame structures in Chinese Rural areas using building information modeling and simulated annealing algorithm.Sustainability 15, 11 (2023), 9000. doi:10.3390/su15119000

    work page doi:10.3390/su15119000 2023

  53. [53]

    pass”, “satisfy

    Minjie Zhu, Frank McKenna, and Michael H Scott. 2018. OpenSeesPy: Python library for the OpenSees finite element framework.SoftwareX7 (2018), 6–11. doi:10.1016/j.softx.2017.10.009 Appendix A BMBench Implementation Details A.1 CivilInstruct Dataset Construction Details Part 1: Fine-Grained Supervised Data.A supervised dataset comprising 3,881 instruction-c...

    doi:10.1016/j.softx.2017.10.009 2018

  54. [1766]

    doi:10.1002/nag.3956

    doi:10.1002/nag.3956

  55. [2025]

    doi:10.18653/v1/2025.findings-acl.686

    13253–13279. doi:10.18653/v1/2025.findings-acl.686

    doi:10.18653/v1/2025.findings-acl.686 2025