Pith. sign in

REVIEW 3 major objections 5 minor 56 references

This paper argues that structural-engineering LLM agents should be judged by the chain of executable artifacts they produce — model, validation records, solver outputs, code-check results, report — not by the final answer alone, and provide

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 00:45 UTC pith:EQRT7ENW

load-bearing objection A useful, unusually transparent systems/benchmark paper whose headline claim about artifact-centered evaluation being more rigorous than final-answer scoring isn't actually tested—the experiments compare two agent workflows under the same artifact assertions. the 3 major comments →

arxiv 2607.14896 v1 pith:EQRT7ENW submitted 2026-07-16 cs.SE cs.AIcs.MA

StructureClaw: Traceable LLM Agents and an Executable Benchmark for Structural Engineering Workflows

Sizhong Qin , Yi Gu , Yao Jiang , Ao Cai , Changjian Zhou , Shaoxuan Shuai , Jiachang Wang , Tianhao Shen
show 8 more authors
Yueqiang Li Xinhao Li Li Zeng Yueshi Chen Dachen Gao Genrong Xu Wenjie Liao Xinzheng Lu
This is my paper
classification cs.SE cs.AIcs.MA
keywords structural engineering agentsartifact-centered evaluationevidence chainLLM agentsexecutable benchmarkworkflow-level assertionssafe non-executionmultimodal reconstruction
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

StructureClaw claims that a structural-engineering request is properly answered only when the agent produces a complete, mutually consistent chain of executable artifacts: interpreted requirements, a computable model, validation records, solver outputs, code-check results, and a report grounded in that trace. To make this standard measurable, the paper introduces a governed workbench and an executable benchmark of 150 scenarios in which a scenario passes only if every typed assertion over routing, artifacts, execution, interaction, and reporting passes in a single run. On 50 well-specified cases, the bundled automatic workflow raises mean Success Rate from 56.8% to 88.6% relative to a generic-skill configuration, and the same assertion-level diagnostics expose invalid-input handling and multimodal model reconstruction as the two weakest points. The paper's central claim is that artifact-centered evaluation reveals workflow-level failures that final-response evaluation hides.

Core claim

The discovery, stated on the paper's own terms, is that an LLM engineering agent can be evaluated reliably only over the complete evidence chain, not over a single output. Success is defined conjunctively: a scenario passes only when all required assertions over routing, artifact existence and reference-fixture matching, backend execution, safe interaction, and reporting hold in one run. The measured consequence is a 31.8-point mean lift from generic-only execution (56.8%) to the full automatic workflow (88.6%) across ten configurations on the same 50 standard scenarios, with every configuration improving. The same diagnostics show invalid-value cases succeeding at 70.9% versus 91.0% overall

What carries the argument

The central mechanism is the evidence chain: a linked sequence of typed artifacts — structural model, validation diagnostics, solver traces, code-check records, report — plus tool-call records, carried in shared artifact state across a reasoning–action execution loop. Domain skills declare triggers, input and output schemas, validation requirements, backend constraints, and artifact contracts; typed tools enforce runtime preconditions; provider bindings connect tool contracts to analysis backends. Evaluation uses typed assertions whose conjunction defines Success Rate, with negative evidence required for safety cases so that abstaining without a recorded reason does not count as success.

Load-bearing premise

The whole evaluation rests on the assumption that the benchmark's acceptance rules recognize engineering validity well enough that a passing scenario is actually a competent workflow — but the matching rule checks only lower-bound counts, coordinate spans within about half a meter, and total load within about twenty percent, and the report judge is an LLM with no reported human agreement.

What would settle it

Take any scenario, alter only model topology or support conditions so that an engineer would reject the result while node and element counts, coordinate spans, and total load still satisfy the fixture; if the agent's run still passes, the benchmark's claim to workflow-level rigor is contradicted. A systematic version is to have qualified engineers review all 150 scenarios' passing traces and measure agreement with the automated pass/fail labels.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Engineering agents should be scored by whether their trace supports the reported result, not by the plausibility of their prose; a fluent answer without a matching solver or code-check record should count as a failure.
  • Structured domain skills as a system can move mean end-to-end success from roughly 57% to 89% on well-specified design-to-analysis requests across the evaluated models.
  • Assertion diagnostics localize improvements: deterministic pre-execution checks for invalid numerical inputs and inspectable intermediate representations for image and DXF reconstruction are the two concrete intervention points the data exposes.
  • Aggregate gains are not uniform; continuous-beam cases regress, so release criteria should include structure-family checks alongside mean Success Rate.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The evidence-chain criterion is portable: any discipline where a conclusion is valid only through an artifact trail — mechanical, civil, or architectural design — could reuse the same conjunctive-assertion setup, with the same failure-localization benefits the paper demonstrates for structures.
  • The coarse matching rules are a lower-bound instrument; adding topology, support-equivalence, and member-by-member checks would make success harder to earn and would likely lower the quoted 88.6% figure, but it would also bring the benchmark closer to what a reviewing engineer accepts.
  • Because every configuration–scenario pair is executed once, the quoted rates are single-attempt estimates; repeated runs would convert Success Rate into a distribution and could expose agents whose reliability is lower than their mean suggests.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces StructureClaw, an artifact-centered workbench for structural-engineering LLM agents, and StructureClaw-Bench, an executable benchmark of 150 scenarios. Success is defined as passing a conjunction of typed assertions over routing, artifacts, execution, interaction, and reporting in a single run (Eq. 1/5). On 50 standard cases, the bundled automatic workflow reaches 88.6% mean Success Rate versus 56.8% for a generic-only configuration, a 31.8-point gain. The paper also presents interactive-robustness and multimodal-reconstruction results and reports a continuous-beam regression in automatic mode. The central claim is that artifact-centered evaluation exposes workflow-level failures that are difficult to identify from final responses alone, thereby providing a more rigorous evaluation basis.

Significance. If the central comparative claim were supported, this would be a useful contribution to engineering-agent evaluation: the evidence chain is well specified, the benchmark is executable and released, unavailable outcomes count as failures under Eq. (4)/(5), and the authors are unusually transparent about single runs, regressions, and small subsets. The continuous-beam counterexample in §6/D.1 and the explicit validity boundaries in Table 9 are genuine strengths. However, the experiments do not actually test the headline comparative claim, and the artifact comparator has construct-validity limits that are acknowledged but not resolved. The work is a solid system-and-benchmark paper, but the abstract and conclusion currently assert more than the evaluation can support.

major comments (3)
  1. [Abstract; §5.1; Appendix C.4; Eq. (1), Eq. (7)] The headline comparison is automatic mode versus generic-only mode, not artifact-centered evaluation versus final-response evaluation. Both modes are scored with the same full assertion set over routing, artifacts, execution, interaction, and reporting; generic-only changes the agent's capability path, not the evaluation instrument. There is no condition in which only the final response is judged. The 88.6% versus 56.8% result therefore supports the claim that the bundled workflow improves behavior under artifact assertions, but not the abstract's claim that artifact-centered evaluation is 'more rigorous' than final-response evaluation. This is a missing control for a load-bearing claim. Please add a final-response-only baseline (e.g., grading only the final report text, or a QA-style judgment of the final answer) or qualify the claims throughout accordingly.
  2. [Appendix B.4; Table 6] The model-matching rule checks lower-bound node/element counts, coordinate spans to a 0.5 m tolerance, and approximate total load to a 20% tolerance. It does not compare topology, support equivalence, member-by-member connectivity, section, or material properties. The report-presence rule requires only more than 100 markdown characters, and the natural-language judge is an LLM with fixed settings and no reported human agreement. These coarse checks can accept artifacts that an engineer would reject, which directly weakens the 'more rigorous' claim. The authors disclose these limits, but they do not provide a human-rated equivalence sample or a stricter comparator to justify the rigor claim. Please either strengthen the comparator or narrow the claim to 'assertion-level consistency' rather than 'engineering rigor.'
  3. [§4.3; §C.6; Table 9] The central 31.8-point difference rests on a single run per configuration–scenario cell. The authors repeatedly call all subset comparisons 'descriptive' and disclose the single-run protocol, but the abstract and conclusion still present the point estimate as a headline finding. Without repeated trials or run variability, there is no stated uncertainty for the primary comparison. A small repeated-run stability analysis for at least the standard-workflow family, or an explicit move to 'illustrative point estimate' language, would be needed to make this quantity load-bearing.
minor comments (5)
  1. [Abstract] The GitHub URL has a missing space: 'athttps' should be 'at https'.
  2. [§2.2] Typo: 'V oyager' should be 'Voyager'.
  3. [Appendix B.1] Typo: 'scenariosis' should be 'scenarios is'.
  4. [Figure 4] The 50–100 radial zoom could mislead readers into over-reading small differences; the caption and text explain this, but a visual note on the figure itself would help.
  5. [Table 7] The 'Cases' column is easy to misread as total cases rather than per configuration; consider renaming to 'Cases per configuration'.

Circularity Check

0 steps flagged

No circular derivation found; the key comparative claim lacks a final-response-only control, but that is a validity gap, not circularity.

full rationale

StructureClaw is an empirical systems-and-benchmark paper. Its only formal derivation is the Success Rate definition in Eqs. (1), (4), and (5), which is computed from observed assertion outcomes; no fitted parameter is later renamed as a prediction, and no asserted result reduces to an input by construction. The headline result (88.6% vs 56.8%) compares two workflow configurations under the same assertion set, and the paper explicitly labels this a paired system-level comparison in Eq. (7) and Sec. C.4, with no component-level attribution. The automatic workflow was designed around the artifact protocol that the benchmark checks, but this is a same-team benchmark-alignment concern rather than a circular derivation: the preferred mode was not forced to win, as shown by the continuous-beam regression in Sec. D.1, and the benchmark and evaluation code are released for independent reruns. The abstract's comparative claim that artifact-centered evaluation is more rigorous than final-response-only evaluation is indeed untested, because no condition scores final responses alone; however, a missing control is a correctness/validity threat, not a circular reduction of the result to its own inputs. The paper's own Table 9 and Appendix C.6 acknowledge attribution and sampling limitations. Related-work self-citations (e.g., refs. [12], [13], [39]–[42]) are not load-bearing: they support the literature review, not the central benchmark claim. Therefore no circular step can be quoted, and the appropriate score is 0.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 2 invented entities

The central contribution is an evaluation harness, not a derivation. Every free parameter is an evaluation threshold or judge setting; the axioms are the assumptions that transform the benchmark into evidence about engineering correctness. The most consequential choices are the coarse model-matching rule and the self-authored fixture set, both disclosed in the paper.

free parameters (6)
  • Model-matching coordinate tolerance = 0.5 m default
    Hand-chosen absolute tolerance for x/y/z coordinate ranges in the model-matching rule (App. B.4); affects pass/fail of the central 'model matching' metric.
  • Model-matching load tolerance = 20% relative
    Hand-chosen relative tolerance for approximate total load comparison; looser tolerance raises Success Rate.
  • Report presence threshold = >100 markdown characters
    Arbitrary threshold used to assert report existence (Table 6); a report with 101 chars passes regardless of content faithfulness.
  • Minimum model completeness defaults = 2 nodes, 1 element
    Default for model-existence assertion (Table 6); below this a model artifact is considered absent.
  • Per-case timeout = 15 minutes
    Operational timeout in child process; a timeout is counted as execution failure (App. C.1), so this choice changes observed Success Rate.
  • Semantic judge settings = temperature 0, 500-token limit, 30s timeout
    Operational settings for the LLM judge that grades natural-language assertions; no human agreement study is presented (App. B.3).
axioms (5)
  • domain assumption The typed assertion suite and reference-fixture comparators are adequate proxies for engineering workflow success.
    Model matching checks counts, spans, and approximate loads only; it does not verify topology, support equivalence, connectivity, or section/material properties (App. B.4). If the comparator accepts incorrect models, Success Rate overstates workflow correctness.
  • domain assumption OpenSeesPy solver results are acceptable as ground-truth analysis evidence.
    The execution backend is OpenSees via OpenSeesPy, but the paper states backend correctness still depends on adapter, solver, units, and input model being valid (App. A.4); solver output is not independently verified.
  • domain assumption The LLM judge produces valid natural-language behavior assessments.
    A separate judge evaluates scenario-specific semantics with fixed temperature/token settings; no inter-annotator/human validation is reported (Table 6, App. B.3).
  • domain assumption The 150 scenarios are a useful, controlled diagnostic sample rather than a random sample of practice.
    Authors explicitly state the suite is not a random sample and subsets are small/descriptive (Sec. 4.3, App. D); conclusions generalize only under this assumption.
  • domain assumption Generic-only mode is an informative control for automatic workflow comparison.
    The mode comparison jointly changes routing, priors, artifact expectations, and validation guidance; it is a system-level comparison, not isolation of one component (Sec. 5.1, Eq. 7).
invented entities (2)
  • StructureClaw shared artifact protocol / evidence chain independent evidence
    purpose: Defines the chain of artifacts (model, validation, solver trace, code-check, report) used to govern and audit agent execution; the central unit of evaluation.
    The protocol is implemented in the released GitHub code and benchmark fixtures, so external groups can run it and check whether the claimed traceability holds.
  • StructureClaw-Bench scenario fixtures and assertions independent evidence
    purpose: Operationalize workflow success as binary pass of typed assertions; provide reference targets for model matching.
    The 150 scenarios, fixtures, and evaluation runners are publicly released, enabling external runs against other systems; however these are author-designed and not externally normalized.

pith-pipeline@v1.3.0-alltime-deepseek · 22752 in / 14094 out tokens · 130042 ms · 2026-08-02T00:45:09.035604+00:00 · methodology

0 comments
read the original abstract

Addressing a structural-engineering request requires more than a single answer; it requires a chain of interdependent artifacts: interpreted requirements, a computable model, validation records, solver outputs, code-check records, and a final report. Evaluations centered on question answering or script generation rarely verify this complete evidence chain and may therefore reward fluent outputs even when the underlying engineering workflow is incomplete, internally inconsistent, or non-executable. To address this limitation, we present StructureClaw, an artifact-centered workbench in which LLM agents operate through governed engineering skills, typed tools, shared artifact state, and local analysis backends. We also introduce StructureClaw-Bench, an executable benchmark of 150 controlled scenarios spanning standard workflow execution, interactive robustness, and multimodal structural-model reconstruction. A scenario succeeds only when all required artifact- and execution-level assertions pass in a single run. Across ten agent-model configurations, each evaluated on the same 50 standard cases, the average Success Rate rises from 56.8% with the generic-skill baseline to 88.6% with the full automatic workflow. The interactive and multimodal evaluations identify two prominent remaining challenges: safe handling of invalid numerical inputs and fixture-consistent reconstruction of structural models. These findings show that artifact-centered evaluation can expose workflow-level failures that are difficult to identify from final responses alone, providing a more rigorous basis for evaluating and improving structural-engineering agents. The code and benchmark are available at https://github.com/structureclaw/structureclaw.

Figures

Figures reproduced from arXiv: 2607.14896 by Ao Cai, Changjian Zhou, Dachen Gao, Genrong Xu, Jiachang Wang, Li Zeng, Shaoxuan Shuai, Sizhong Qin, Tianhao Shen, Wenjie Liao, Xinhao Li, Xinzheng Lu, Yao Jiang, Yi Gu, Yueqiang Li, Yueshi Chen.

Figure 1
Figure 1. Figure 1: From a structural-engineering request to reviewable ev [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overall framework of StructureClaw. Multimodal engineering context enters a StructureClaw agent kernel, where capability [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Composition of StructureClaw-Bench. The heatmap [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Descriptive evaluation profiles. The radial display zooms [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Standard workflow results. Purple points denote generic [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Multimodal structural-model reconstruction results. Bars [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Steel-frame example of the declarative skill and artifact [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗
Figure 10
Figure 10. Figure 10: Assertion-level diagnostics by setting. Gray cells with [PITH_FULL_IMAGE:figures/full_fig_p019_10.png] view at source ↗
Figure 12
Figure 12. Figure 12: Success Rate by recorded locale and multimodal input [PITH_FULL_IMAGE:figures/full_fig_p020_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Success Rate versus recorded mean wall-clock dura [PITH_FULL_IMAGE:figures/full_fig_p020_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

56 extracted references · 5 linked inside Pith

  1. [1]

    Learning to simulate and design for structural engineering

    Kai-Hung Chang and Chin-Yi Cheng. Learning to simulate and design for structural engineering. InProceedings of the 37th International Conference on Machine Learning, pages 1426–1436. PMLR, 2020. 3

    2020

  2. [2]

    Multi-agent large language model framework for code-compliant automated design of rein- forced concrete structures.Automation in Construction, 177: 106331, 2025

    Jinxin Chen and Yi Bao. Multi-agent large language model framework for code-compliant automated design of rein- forced concrete structures.Automation in Construction, 177: 106331, 2025. 2, 3

    2025

  3. [3]

    RAMASC: A retrieval-augmented multi-agent framework for automated structural calculation.Advanced Engineering Informatics, 74:104698, 2026

    Kichang Choi, Minwoo Jeong, Taegeon Kim, Seokhwan Kim, Seungwon Baek, and Hongjo Kim. RAMASC: A retrieval-augmented multi-agent framework for automated structural calculation.Advanced Engineering Informatics, 74:104698, 2026. 2, 3

    2026

  4. [4]

    BIMgent: Towards autonomous building modeling via computer-use agents

    Zihan Deng, Changyu Du, Stavros Nousias, and Andr ´e Bor- rmann. BIMgent: Towards autonomous building modeling via computer-use agents. InICML 2025 Workshop on Com- puter Use Agents, 2025. 3

    2025

  5. [5]

    Text2BIM: Generating building models using a large language model-based multiagent framework.Journal of Computing in Civil Engineering, 40(2):04025142, 2026

    Changyu Du, Sebastian Esser, Stavros Nousias, and Andr ´e Borrmann. Text2BIM: Generating building models using a large language model-based multiagent framework.Journal of Computing in Civil Engineering, 40(2):04025142, 2026. 3

    2026

  6. [6]

    Dual generative adversarial networks for automated component layout design of steel frame-brace structures.Automation in Construction, 146:104661, 2023

    Bochao Fu, Yuqing Gao, and Wei Wang. Dual generative adversarial networks for automated component layout design of steel frame-brace structures.Automation in Construction, 146:104661, 2023. 3

    2023

  7. [7]

    Physical rule- guided generative adversarial network for automated struc- tural layout design of steel frame-brace structures.Journal of Building Engineering, 86:108943, 2024

    Bochao Fu, Wei Wang, and Yuqing Gao. Physical rule- guided generative adversarial network for automated struc- tural layout design of steel frame-brace structures.Journal of Building Engineering, 86:108943, 2024. 3

    2024

  8. [8]

    PAL: Program-aided language models

    Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. PAL: Program-aided language models. InProceedings of the 40th International Conference on Machine Learning, pages 10764–10799. PMLR, 2023. 2, 4

    2023

  9. [9]

    A lightweight large language model-based multi-agent system for 2D frame structural analysis.arXiv preprint arXiv:2510.05414, 2025

    Ziheng Geng, Jiachen Liu, Ran Cao, Lu Cheng, Haifeng Wang, and Minghui Cheng. A lightweight large language model-based multi-agent system for 2D frame structural analysis.arXiv preprint arXiv:2510.05414, 2025. 3

    arXiv 2025

  10. [10]

    Agentic large language models for automated structural analysis of 3D frame sys- tems.arXiv preprint arXiv:2606.06525, 2026

    Ziheng Geng, Ian Franklin, Santiago Martinez, Jiachen Liu, Yunhe Zhao, and Minghui Cheng. Agentic large language models for automated structural analysis of 3D frame sys- tems.arXiv preprint arXiv:2606.06525, 2026. 3

    Pith/arXiv arXiv 2026

  11. [11]

    Frangopol, and Minghui Cheng

    Ziheng Geng, Jiachen Liu, Ian Franklin, Ran Cao, Dan M. Frangopol, and Minghui Cheng. Automating structural anal- ysis across multiple software platforms using large language models.arXiv preprint arXiv:2604.09866, 2026. 3

    Pith/arXiv arXiv 2026

  12. [12]

    Intel- ligent design of shear wall layout based on diffusion mod- els.Computer-Aided Civil and Infrastructure Engineering, 39(23):3610–3625, 2024

    Yi Gu, Yuli Huang, Wenjie Liao, and Xinzheng Lu. Intel- ligent design of shear wall layout based on diffusion mod- els.Computer-Aided Civil and Infrastructure Engineering, 39(23):3610–3625, 2024. 2, 3

    2024

  13. [13]

    In- telligent design of dimensions of reinforced concrete frame structure components using diffusion models.Computers in Industry, 175:104428, 2026

    Yi Gu, Sizhong Qin, Wenjie Liao, and Xinzheng Lu. In- telligent design of dimensions of reinforced concrete frame structure components using diffusion models.Computers in Industry, 175:104428, 2026. 3

    2026

  14. [14]

    Toward engineering AGI: Benchmarking the engineering design capabilities of LLMs

    Xingang Guo, Yaxin Li, Xiangyi Kong, Yilan Jiang, Xiayu Zhao, Zhihua Gong, Yufan Zhang, Daixuan Li, Tianle Sang, Beixiao Zhu, et al. Toward engineering AGI: Benchmarking the engineering design capabilities of LLMs. InAdvances in Neural Information Processing Systems, 2025. Datasets and Benchmarks Track. 2, 4

    2025

  15. [15]

    Enhancing reliability and automation of LLM- based structural analysis using a hybrid multi-agent pipeline

    Seokjae Heo. Enhancing reliability and automation of LLM- based structural analysis using a hybrid multi-agent pipeline. Scientific Reports, 16:19458, 2026. 4

    2026

  16. [16]

    Leveraging large language models for BIM-based automated compliance checking.Au- tomation in Construction, 182:106707, 2026

    Odin Iversen and Lizhen Huang. Leveraging large language models for BIM-based automated compliance checking.Au- tomation in Construction, 182:106707, 2026. 3

    2026

  17. [17]

    Deep reinforcement learn- ing for automated design of reinforced concrete structures

    Jong-Hyun Jeong and Hongki Jo. Deep reinforcement learn- ing for automated design of reinforced concrete structures. Computer-Aided Civil and Infrastructure Engineering, 36 (12):1508–1529, 2021. 3

    2021

  18. [18]

    SoK: Agentic skills – beyond tool use in LLM agents.arXiv preprint arXiv:2602.20867, 2026

    Yanna Jiang, Delong Li, Haiyu Deng, Baihe Ma, Xu Wang, Qin Wang, and Guangsheng Yu. SoK: Agentic skills – beyond tool use in LLM agents.arXiv preprint arXiv:2602.20867, 2026. 4

    Pith/arXiv arXiv 2026

  19. [19]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE- bench: Can language models resolve real-world GitHub is- sues? InInternational Conference on Learning Representa- tions, 2024. 4

    2024

  20. [20]

    A review of LLMs and their applications in the architecture, engineering and construction industry.Artificial Intelligence Review, 58:250,

    Dimitrios Kampelopoulos, Athina Tsanousa, Stefanos Vrochidis, and Ioannis Kompatsiaris. A review of LLMs and their applications in the architecture, engineering and construction industry.Artificial Intelligence Review, 58:250,

  21. [21]

    Hsain, and Guido Maciocci

    Aleksei Kondratenko, Mussie Birhane, Houssame E. Hsain, and Guido Maciocci. AECV-Bench: Benchmarking multi- modal models on architectural and engineering drawings un- derstanding.arXiv preprint arXiv:2601.04819, 2026. 4

    arXiv 2026

  22. [22]

    API-Bank: A comprehensive benchmark for tool-augmented LLMs

    Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. API-Bank: A comprehensive benchmark for tool-augmented LLMs. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3102–3116, Singapore, 2023. Association for Computational Linguistics. 4

    2023

  23. [23]

    DrafterBench: Benchmarking large language models for tasks automation in civil engineering.arXiv preprint arXiv:2507.11527, 2025

    Yinsheng Li, Zhen Dong, and Yi Shao. DrafterBench: Benchmarking large language models for tasks automation in civil engineering.arXiv preprint arXiv:2507.11527, 2025. 4

    Pith/arXiv arXiv 2025

  24. [24]

    AECBench: A hierarchical benchmark for knowledge evaluation of large language models in the AEC field.Advanced Engineering Informatics, 71:104314, 2026

    Chen Liang, Zhaoqi Huang, Haofen Wang, Fu Chai, Chuny- ing Yu, Huanhuan Wei, Zhengjie Liu, Yanpeng Li, Hongjun Wang, Ruifeng Luo, and Xianzhong Zhao. AECBench: A hierarchical benchmark for knowledge evaluation of large language models in the AEC field.Advanced Engineering Informatics, 71:104314, 2026. 4

    2026

  25. [25]

    Integrating large language models for automated structural analysis.arXiv preprint arXiv:2504.09754, 2025

    Haoran Liang, Mohammad Talebi Kalaleh, and Qipei Mei. Integrating large language models for automated structural analysis.arXiv preprint arXiv:2504.09754, 2025. 3

    Pith/arXiv arXiv 2025

  26. [26]

    Automating structural engineering work- flows with large language model agents.arXiv preprint arXiv:2510.11004, 2025

    Haoran Liang, Yufa Zhou, Mohammad Talebi Kalaleh, and Qipei Mei. Automating structural engineering work- flows with large language model agents.arXiv preprint arXiv:2510.11004, 2025. 2, 3

    arXiv 2025

  27. [27]

    Automated structural design of shear wall residential buildings using generative adversarial networks

    Wenjie Liao, Xinzheng Lu, Yuli Huang, Zhe Zheng, and Yuanqing Lin. Automated structural design of shear wall residential buildings using generative adversarial networks. 10 Automation in Construction, 132:103931, 2021. 3

    2021

  28. [28]

    Generative AI design for building structures.Au- tomation in Construction, 157:105187, 2024

    Wenjie Liao, Xinzheng Lu, Yifan Fei, Yi Gu, and Yuli Huang. Generative AI design for building structures.Au- tomation in Construction, 157:105187, 2024. 2

    2024

  29. [29]

    SkillAct: Using skill abstractions im- proves LLM agents

    Anthony Zhe Liu, Jongwook Choi, Sungryull Sohn, Yao Fu, Jaekyeom Kim, Dong-Ki Kim, Xinhe Wang, Jaewon Yoo, and Honglak Lee. SkillAct: Using skill abstractions im- proves LLM agents. InICML 2024 Workshop on LLMs and Cognition, 2024. 4

    2024

  30. [30]

    A large language model- empowered agent for reliable and robust structural analysis

    Jiachen Liu, Ziheng Geng, Ran Cao, Lu Cheng, Paolo Bocchini, and Minghui Cheng. A large language model- empowered agent for reliable and robust structural analysis. Structure and Infrastructure Engineering, 2026. 2, 3

    2026

  31. [31]

    AgentBench: Evaluating LLMs as agents

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengx- iao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. AgentBench: Evaluating LLMs as agents. InInternational Conference on Learning Representat...

    2024

  32. [32]

    Large lan- guage model-driven code compliance checking in building information modeling.Electronics, 14(11):2146, 2025

    Soumya Madireddy, Lu Gao, Zia Ud Din, Kinam Kim, Ahmed Senouci, Zhe Han, and Yunpeng Zhang. Large lan- guage model-driven code compliance checking in building information modeling.Electronics, 14(11):2146, 2025. 3

    2025

  33. [33]

    AEC-Bench: A multimodal benchmark for agentic systems in architecture, engineering, and construc- tion.arXiv preprint arXiv:2603.29199, 2026

    Harsh Mankodiya, Chase Gallik, Theodoros Galanos, and Andriy Mulyar. AEC-Bench: A multimodal benchmark for agentic systems in architecture, engineering, and construc- tion.arXiv preprint arXiv:2603.29199, 2026. 2, 4

    arXiv 2026

  34. [34]

    OpenSees: A framework for earthquake en- gineering simulation.Computing in Science & Engineering, 13(4):58–66, 2011

    Frank McKenna. OpenSees: A framework for earthquake en- gineering simulation.Computing in Science & Engineering, 13(4):58–66, 2011. 2, 5

    2011

  35. [35]

    MCP4IFC: IFC-based build- ing design using large language models.arXiv preprint arXiv:2511.05533, 2025

    Bharathi Kannan Nithyanantham, Tobias Sesterhenn, Ash- win Nedungadi, Sergio Peral Garijo, Janis Zenkner, Chris- tian Bartelt, and Stefan L¨udtke. MCP4IFC: IFC-based build- ing design using large language models.arXiv preprint arXiv:2511.05533, 2025. 3

    arXiv 2025

  36. [36]

    Patil, Tianjun Zhang, Xin Wang, and Joseph E

    Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive APIs. InAdvances in Neural Information Process- ing Systems, pages 126544–126565, 2024. 4

    2024

  37. [37]

    Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng- Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E

    Shishir G. Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng- Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The Berkeley function calling leaderboard (BFCL): From tool use to agentic evaluation of large language models. In Proceedings of the 42nd International Conference on Ma- chine Learning, pages 48371–48392. PMLR, 2025. 4

    2025

  38. [38]

    Pizarro, Leonardo M

    Pablo N. Pizarro, Leonardo M. Massone, Fabi ´an R. Rojas, and Rafael O. Ruiz. Use of convolutional networks in the conceptual structural design of shear wall buildings layout. Engineering Structures, 239:112311, 2021. 3

    2021

  39. [39]

    Sizhong Qin, Hong Guan, Wenjie Liao, Yi Gu, Zhe Zheng, Hongjing Xue, and Xinzheng Lu. Intelligent design and opti- mization system for shear wall structures based on large lan- guage models and generative artificial intelligence.Journal of Building Engineering, 95:109996, 2024. 3

    2024

  40. [40]

    AIstructure- Copilot: Assistant for generative AI-driven intelligent design of building structures.Smart Construction, 1(1):0001, 2024

    Sizhong Qin, Wenjie Liao, Shengnan Huang, Kongguo Hu, Zhuang Tan, Yuan Gao, and Xinzheng Lu. AIstructure- Copilot: Assistant for generative AI-driven intelligent design of building structures.Smart Construction, 1(1):0001, 2024. 3

    2024

  41. [41]

    Leveraging data-driven artificial intelligence in optimization design for building structures: A review.Engineering Struc- tures, 341:120810, 2025

    Sizhong Qin, Yifan Fei, Wenjie Liao, and Xinzheng Lu. Leveraging data-driven artificial intelligence in optimization design for building structures: A review.Engineering Struc- tures, 341:120810, 2025. 2

    2025

  42. [42]

    Intelligent design for compo- nent size generation in reinforced concrete frame structures using heterogeneous graph neural networks.Automation in Construction, 171:105967, 2025

    Sizhong Qin, Wenjie Liao, Yuli Huang, Shulu Zhang, Yi Gu, Jin Han, and Xinzheng Lu. Intelligent design for compo- nent size generation in reinforced concrete frame structures using heterogeneous graph neural networks.Automation in Construction, 171:105967, 2025. 3

    2025

  43. [43]

    ToolLLM: Facilitating large language models to mas- ter 16000+ real-world APIs

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. ToolLLM: Facilitating large language models to mas- ter 16000+ real-world APIs. InInternational Conference on Learning Represent...

    2024

  44. [44]

    GPT models in construction industry: Oppor- tunities, limitations, and a use case validation.Developments in the Built Environment, 17:100300, 2024

    Abdullahi Saka, Ridwan Taiwo, Nurudeen Saka, Ba- batunde Abiodun Salami, Saheed Ajayi, Kabiru Akande, and Hadi Kazemi. GPT models in construction industry: Oppor- tunities, limitations, and a use case validation.Developments in the Built Environment, 17:100300, 2024. 1

    2024

  45. [45]

    Toolformer: Lan- guage models can teach themselves to use tools

    Timo Schick, Jane Dwivedi-Yu, Roberto Dess `ı, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Lan- guage models can teach themselves to use tools. InAdvances in Neural Information Processing Systems, pages 68539– 68551, 2023. 4

    2023

  46. [46]

    HuggingGPT: Solving AI tasks with ChatGPT and its friends in Hugging Face

    Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. HuggingGPT: Solving AI tasks with ChatGPT and its friends in Hugging Face. In Advances in Neural Information Processing Systems, pages 38154–38180, 2023. 4

    2023

  47. [47]

    V oyager: An open-ended embodied agent with large language models.Transactions on Machine Learning Re- search, 2024

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandku- mar. V oyager: An open-ended embodied agent with large language models.Transactions on Machine Learning Re- search, 2024. 4

    2024

  48. [48]

    OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments. InAd- vances in Neural Information...

    2024

  49. [49]

    Narasimhan, and Yuan Cao

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations, 2023. 2, 4, 5

    2023

  50. [50]

    Narasimhan.τ-bench: A benchmark for tool-agent-user in- teraction in real-world domains

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik R. Narasimhan.τ-bench: A benchmark for tool-agent-user in- teraction in real-world domains. InInternational Conference on Learning Representations, 2025. 2, 4

    2025

  51. [51]

    End-to-end generation of structural topology for complex architectural layouts with graph neural networks

    Chong Zhang, Mu-Xuan Tao, Chen Wang, and Jian-Sheng Fan. End-to-end generation of structural topology for complex architectural layouts with graph neural networks. Computer-Aided Civil and Infrastructure Engineering, 39 (5):756–775, 2024. 3 11

    2024

  52. [52]

    Intelligent design of shear wall layout based on graph neural networks.Advanced Engineering Informatics, 55:101886, 2023

    Pengju Zhao, Wenjie Liao, Yuli Huang, and Xinzheng Lu. Intelligent design of shear wall layout based on graph neural networks.Advanced Engineering Informatics, 55:101886, 2023

    2023

  53. [53]

    Beam layout design of shear wall structures based on graph neural networks.Automation in Construction, 158:105223,

    Pengju Zhao, Wenjie Liao, Yuli Huang, and Xinzheng Lu. Beam layout design of shear wall structures based on graph neural networks.Automation in Construction, 158:105223,

  54. [54]

    Dynamic prompt-based virtual assistant framework for BIM information search.Au- tomation in Construction, 155:105067, 2023

    Junwen Zheng and Martin Fischer. Dynamic prompt-based virtual assistant framework for BIM information search.Au- tomation in Construction, 155:105067, 2023. 3

    2023

  55. [55]

    StructDiffusion: End-to-end intelligent shear wall structure layout generation and analysis using diffusion model.Engineering Structures, 309:118068, 2024

    Ying Zhou, Hao Leng, Shiqiao Meng, Hao Wu, and Zheng Zhang. StructDiffusion: End-to-end intelligent shear wall structure layout generation and analysis using diffusion model.Engineering Structures, 309:118068, 2024. 3

    2024

  56. [56]

    Minjie Zhu, Frank McKenna, and Michael H. Scott. OpenSeesPy: Python library for the OpenSees finite element framework.SoftwareX, 7:6–11, 2018. 2, 5 12 A. Implementation and Trace Semantics StructureClaw is implemented around an explicit evidence chain rather than a single conversational output. The chain records what the agent was asked to do, which gover...

    2018