CEQuest: Benchmarking Large Language Models for Construction Estimation

Lufan Wang; Rui Liu; Yanzhao Wu

REVIEW 3 major objections 5 minor 1 cited by

CEQuest: Benchmarking Large Language Models for Construction Estimation

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces CEQuest, a 164-question expert-authored benchmark for construction drawing interpretation and estimation, and reports that all five tested LLMs score below 80%, with GPT-4.1 highest at 75.37%.

desk verdict CEQuest is a small but genuine new benchmark for construction estimation QA; the headline "LLMs below 80%" is plausible but not fully interpretable because the answer keys lack an expert baseline and at least one item depends on an unstated rounding convention. read the letter →

arxiv 2508.16081 v1 pith:KFHYUYZO submitted 2025-08-22 cs.CL cs.LG

Yanzhao Wu , Lufan Wang , Rui Liu This is my paper

classification cs.CLcs.LG

keywords largelanguagemodelsconstructionestimationbenchmarkdatasetdrawinginterpretationquantitytakeoffdomain-specificevaluationmultiple-choicequestionanswering

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CEQuest is an open benchmark of 164 expert-authored questions covering construction drawing interpretation and estimation, with 101 multiple-choice and 63 true/false items across five subject areas. The paper evaluates five language models (Gemma 3, Phi4, LLaVA, Llama 3.3, and GPT-4.1) on accuracy, execution time, and model size. The central result is that no model reaches 80 percent accuracy; GPT-4.1 leads at 75.37 percent, and open-source models range from 61.83 to 65.37 percent. The authors read this as a domain-knowledge gap and offer CEQuest as a shared testbed for developing construction-specific LLMs.

What carries the argument

The CEQuest dataset itself carries the argument. It consists of 164 expert-authored questions, organized into five subject areas that progress from general construction knowledge, drawing element identification, and view types to spatial reasoning and cross-referencing, then to quantity takeoff and estimating. Each question has a single correct answer, and the evaluation harness prompts models for letter-only or boolean answers and scores accuracy, execution time, and model size. This subject-area progression is what lets the paper locate where models succeed and fail.

What would settle it

Have a panel of professional estimators independently answer the 164 CEQuest items in the same format as the models. If experts frequently disagree with the provided key or with one another, or if expert accuracy is close to the models' 61-75 percent range, the benchmark's ground truth and its claimed performance gap are not established.

Watch

Extended reading notes

Core claim

The paper's finding is that current large language models, asked to answer a curated set of construction-estimation questions, uniformly underperform: none reaches 80 percent accuracy despite a simple multiple-choice and true/false format. GPT-4.1 is the strongest at 75.37 percent, while the four open-source models cluster in the low-to-mid 60s. The results also detach model scale from skill: Phi-4 at 14B beats LLaVA at 34B, and models sometimes answer with correct content in the wrong format or miss professional conventions such as rounding material quantities up rather than to the nearest whole number.

Load-bearing premise

The load-bearing premise is that the 164 expert-authored questions, each with one correct answer, form a valid, unambiguous, and representative test of construction estimation ability; without that, the reported scores and the gap they reveal do not follow.

Editorial extensions

If this is right

If CEQuest is a fair test, then general-purpose LLMs are not yet reliable tools for construction estimation without domain-specific adaptation.
The systematic ceiling below 80 percent and the rounding-up error suggest fine-tuning or retrieval of professional estimating rules is the direct next lever.
The size-versus-accuracy decoupling implies architecture and training-data quality deserve as much attention as parameter count when building domain models.
Since the benchmark will be publicly released, future specialized models can be compared on a fixed, repeatable question set.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A text-only CEQuest may understate the gap for real drawings; feeding actual plans, sections, and schedules to vision-language models would test the harder multimodal skill.
The single-correct-answer key treats estimation as binary, but real takeoff permits legitimate method variation; rubric-based or partial-credit scoring could change the ranking.
Formatting failures can be separated from knowledge failures by accepting answers that contain the right option label anywhere in the response; the reported accuracy may shift once that distinction is made.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The one thing you should know: this is a real, new benchmark for a genuinely underserved domain. 164 expert-authored questions on construction drawing interpretation and estimation, five LLMs evaluated, repeated runs, code and data promised. No existing construction QA benchmark of this kind is cited, and the gap is real. That is the actual contribution, and it is honest progress, not a breakthrough.

What the paper does well: the dataset design is straightforward and appropriate for a pilot. The evaluation uses five diverse models, reports mean and std over five runs, and includes a useful case study. The authors are transparent about known failure modes, including format violations and the Q23 rounding issue. They do not oversell: the conclusion is that current LLMs have room to improve, which is supported by the numbers as far as those numbers go.

Now the soft spots, in proportion. The stress-test note is on target. The Q23 grading depends on an unstated convention: 5.33 cy is rounded up to 6 cy in professional practice, so 5 cy is marked wrong. The paper does not show the question stated that convention. With 164 items, each item is about 0.6%, so a handful of such conventions can move a model by several points and shrink or widen the reported gaps. Related: there is no human expert baseline, no inter-rater reliability check, and no ambiguity audit for the answer keys. The paper cannot separate "LLMs lack domain knowledge" from "the answer keys are hard for anyone who does not share the authors' conventions." That is a load-bearing concern for the headline claim.

A second, more minor issue: the output parser is not described in detail. LLaVA's Q10 answer "A. Room finish schedule" is factually correct but format-violating; the paper mentions this but does not state how the parser treated it. That could shift accuracy by a point or two. A random baseline would also help interpret the 61-75% range, since true/false questions alone should give a 50% floor.

None of this kills the paper. The dataset is a useful resource, and the authors seem aware of its limits. But the central claim needs an expert baseline and an ambiguity audit before the accuracy numbers mean what the abstract says they mean.

Who is this for: anyone working on domain-specific LLM benchmarks, especially in construction, AEC, or other professional QA. It deserves a serious referee—ideally for a benchmark track or a domain venue—with the expectation of revisions that add a human baseline and clarify grading conventions.

Referee Report

3 major / 5 minor

Summary. The paper introduces CEQuest, a construction-specific question-answering benchmark of 164 expert-authored items (62% multiple-choice, 38% true/false) spanning five subject areas. It evaluates five LLMs—Gemma 3, Phi4, LLaVA, Llama 3.3, and GPT-4.1—using accuracy, evaluation time, and model size, reporting mean±std over five runs. All models score below 80% (best: GPT-4.1 at 75.37%). The paper concludes that current SOTA LLMs have considerable room for improvement and motivates the need for domain-specific models; the dataset and code are promised open-source.

Significance. If the benchmark's answer keys are accepted as valid and the parsing pipeline is faithful, the paper addresses a real gap: there is no public benchmark specifically designed for LLM construction estimation. The repeated runs, the diversity of models, and the plan to release dataset and code are strengths. However, the central numeric claims depend on two unvalidated choices—the expert-authored answer keys and the output parser—neither of which is characterized in enough detail. With only 164 items, small ambiguities shift percentages materially. Thus the benchmark is potentially valuable, but the reported accuracy numbers are not yet fully interpretable as a measure of LLM construction estimation ability.

major comments (3)

[Sec. 3, Case Study (Q23)] The Q23 discussion shows that grading depends on a convention that may not be stated in the item. The computed volume is 5.33 cy; models selecting 5 cy are marked wrong because in professional practice quantities are 'typically rounded up.' If the question stem asks for the exact volume, then 5.33 is not 5 cy and the models are wrong; if it implicitly asks for the order quantity, the rounding convention should be part of the question. The paper does not provide the Q23 stem or show that the convention was communicated. Because each item contributes 1/164 ≈ 0.61% to accuracy, a small number of such convention-dependent keys can alter reported accuracy by several points and change the 'below 80%' conclusion. Please release item-level wording with the rubric and provide evidence that the keys are unambiguous (e.g., independent agreement among multiple experts).
[Secs. 2.3 and 3] The evaluation section does not describe the parser logic. The paper itself reports that LLaVA's Q10 answer 'A. Room finish schedule' is factually correct but format-violating, and that Llama 3.3 often omits the final letter. Without stating how such outputs are converted to A/B/C/D/E/F or TRUE/FALSE, the accuracy numbers in Table 2 are not reproducible and may systematically underestimate models that give verbose answers. Please specify the parsing rules, report both strict and lenient accuracy, and list the non-conforming outputs per model. This is load-bearing because the 62–75% accuracy spread is the paper's main result.
[Secs. 2.1–2.2] The premise that the 164 expert-authored questions and their single correct answers are valid and representative is asserted, not demonstrated. No human baseline, inter-rater reliability, or ambiguity audit is reported. Without a baseline, 'all models below 80%' cannot be distinguished from 'the answer keys contain ambiguous or overly strict items.' A small human-expert/novice baseline and simple agreement statistics would make the central claim interpretable. This can be added without changing the dataset itself.

minor comments (5)

[Table 1] The average number of choices per question (3.59) should clarify how TRUE/FALSE items are counted; for binary items the number of choices is 2. As written, the reader cannot tell whether the average is over MCQ items only.
[Figure 2] The exact distribution of questions by subject area is not given in the text. Include the counts in the caption or in a small table so that the composition is reproducible.
[References] Several load-bearing claims about gaps in existing benchmarks are supported by informal blog sources (e.g., [4], [7], [35]). Where possible, replace these with peer-reviewed or archival sources.
[Abstract and Sec. 5] The paper says the dataset 'will be open-sourced,' but Sec. 2.1 already provides a GitHub URL. Reconcile this wording to indicate the current release status.
[Sec. 3] The third observation (model size does not always correlate with accuracy) is based on point estimates. Given the reported standard deviations, a simple paired significance test or confidence intervals would strengthen this comparison.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: CEQuest is an empirical benchmark; model accuracies are measured against expert-authored keys and are not derived from the benchmark's own assumptions.

full rationale

This paper is an empirical benchmark study, not a derivation. The central claim—that current LLMs score below 80% and leave room for improvement—comes from running five independent models on 164 expert-authored question-answer items and measuring exact-match accuracy (Section 3, Table 2). There is no fitted parameter that is then renamed as a prediction, no normalization or rescaling derived from the data that later predicts the same data, and no self-citation chain that forces the result. The dataset (Section 2.1) is constructed from textbooks and expert practice, independently of the models being tested. The evaluation (Section 2.3) uses straightforward prompting and exact-answer matching; accuracy is simply the percentage of correct answers. The Q23 discussion about rounding up concrete quantities is a potential issue of answer-key ambiguity or unstated convention, but it is a validity/quality concern, not circularity: the expert key is an input assumption, not a conclusion that is claimed to be derived from first principles. The absence of an inter-rater reliability or human baseline weakens the strength of the 'room for improvement' interpretation, but does not make the reported measurements circular. Since no step in the paper's argument reduces by construction to the benchmark's own inputs, the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new ontologically loaded entities, forces, or conserved quantities. Its contribution is a dataset, which is not an invented entity in the sense of a postulated physical or conceptual construction.

assumptions (3)

domain assumption The 164 expert-authored questions have unambiguous correct answers that serve as ground truth.
Section 2.1 states questions were 'collected and designed by domain experts' with correct answers; no inter-rater reliability or validation is reported.
domain assumption The dataset is representative of construction drawing interpretation and estimation tasks.
Section 2.2 classifies questions into five subject areas, but no evidence is given that these cover the breadth of real-world estimation practice.
domain assumption The evaluation framework accurately parses and scores LLM outputs.
Section 2.3 mentions parsing but does not specify how format-violating responses (e.g., 'A. Room finish schedule') are scored; the case study in Section 3 shows such responses occur.

how reviews work

0 comments

Cite this review

Pith. "Pith review of CEQuest: Benchmarking Large Language Models for Construction Estimation." pith.science (2026). https://pith.science/paper/KFHYUYZO

@misc{pith2026250816081,
  author       = {Pith},
  title        = {Pith review of: CEQuest: Benchmarking Large Language Models for Construction Estimation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KFHYUYZO}},
  note         = {Machine review of arXiv:2508.16081}
}

read the original abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide range of general-domain tasks. However, their effectiveness in specialized fields, such as construction, remains underexplored. In this paper, we introduce CEQuest, a novel benchmark dataset specifically designed to evaluate the performance of LLMs in answering construction-related questions, particularly in the areas of construction drawing interpretation and estimation. We conduct comprehensive experiments using five state-of-the-art LLMs, including Gemma 3, Phi4, LLaVA, Llama 3.3, and GPT-4.1, and evaluate their performance in terms of accuracy, execution time, and model size. Our experimental results demonstrate that current LLMs exhibit considerable room for improvement, highlighting the importance of integrating domain-specific knowledge into these models. To facilitate further research, we will open-source the proposed CEQuest dataset, aiming to foster the development of specialized large language models (LLMs) tailored to the construction domain.

Figures

Figures reproduced from arXiv: 2508.16081 by the authors.

**Figure 1.** Sample Questions and Correct Responses. To address this critical gap, this paper introduces CEQuest, a novel benchmark dataset designed to facilitate the development and evaluation of LLMs for construction drawing interpretation and estimation tasks. We conducted a systematic experimental analysis using CEQuest to benchmark five state-of-the-art (SOTA) LLMs, i.e., Gemma 3 [34], Phi4 [1], LLaVA [24], Llama 3.3 [13], … view at source ↗

**Figure 2.** And the statistics of the CEQuest dataset are summarized [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Comparison of Open-source LLMs on CEQuest. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

DrawingVQA: A Real-World Benchmark for Multi-Depth Visual-Textual Reasoning on Construction Drawings
cs.AI 2026-07 conditional novelty 6.0 of 10

On real construction drawings, the best AI model scores 71.7% versus 94.9% for experienced engineers, with the largest gaps in expert-level reasoning and quantity take-off.

Reference graph

Works this paper leans on

41 extracted references · 26 canonical work pages · cited by 1 Pith paper

[1]

Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauff- mann, et al. 2024. Phi-4 technical report. arXiv preprint arXiv:2412.08905 (2024)

arXiv 2024
[2]

JL Blanco, D Rockhill, A Sanghvi, and A Torres. 2023. From start-up to scale-up: accelerating growth in construction technology. McKinsey & Company (2023)

work page 2023
[3]

Conor Bronsdon. 2024. 6 reasons why the construction industry is slow to em- brace technology. https://underthehardhat.org/6-reasons-why-the-construction- industry-is-slow-to-embrace-technology/. [Online; Accessed on 10-Jun-2025]

work page 2024
[4]

Conor Bronsdon. 2025. 7 Categories of LLM Benchmarks for Evaluating AI Beyond Conventional Metrics. https://galileo.ai/blog/llm-benchmarks-categories. [Online; Accessed on 10-Jun-2025]

work page 2025
[5]

Yu, Qiang Yang, and Xing Xie

Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2024. A Survey on Evaluation of Large Language Models. ACM Trans. Intell. Syst. Technol. 15, 3, Article 39 (March 2024), 45 pages. doi:10.1145/3641289

doi:10.1145/3641289 2024
[6]

Nanjiang Chen, Xuhui Lin, Hai Jiang, and Yi An. 2024. Automated building infor- mation modeling compliance check through a large language model combined with deep learning and ontology. Buildings 14, 7 (2024), 1983

work page 2024
[7]

Adam Chernick, Luke Reeve, and Scott Patten. 2025. AI-Powered Construction Document Analysis by Leveraging Computer Vision and Large Language Mod- els. https://aws.amazon.com/blogs/spatial/ai-powered-construction-document- analysis-by-leveraging-computer-vision-and-large-language-models/. [Online; Accessed on 10-Jun-2025]

work page 2025
[8]

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457 (2018)

arXiv 2018

Show all 41 references

[9]

Zhiwen Fan, Lingjie Zhu, Honghua Li, Xiaohao Chen, Siyu Zhu, and Ping Tan
[10]

Yingqiang Ge, Wenyue Hua, Kai Mei, jianchao ji, Juntao Tan, Shuyuan Xu, Ze- long Li, and Yongfeng Zhang. 2023. OpenAGI: When LLM Meets Domain Ex- perts. In Advances in Neural Information Processing Systems , A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (...

2023
[11]

Prashnna Ghimire. 2025. Framework for Integrating Industry Knowledge into a Large Language Model to Assist Construction Cost Estimation . Ph. D. Dissertation. The University of Nebraska-Lincoln

2025
[12]

Prashnna Ghimire, Kyungki Kim, and Manoj Acharya. 2024. Opportunities and Challenges of Generative AI in Construction Industry: Focusing on Adoption of Text-Based Models. Buildings 14, 1 (2024). doi:10.3390/buildings14010220

2024 doi
[13]

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

2024 arXiv
[14]

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Under- standing. Proceedings of the International Conference on Learning Representations (ICLR) (2021)

2021
[15]

Bon-Gang Hwang, Jasmine Ngo, and Jeremy Zhen Kang Teo. 2022. Challenges and Strategies for the Adoption of Smart Technologies in the Construction Industry: The Case of Singapore. Journal of Management in Engineering 38, 1 (2022), 05021014. doi:10.1061/(ASCE)ME.1943-5479.0000986

2022
[16]

John Joy and Jinane Mounsef. 2021. Automation of Material Takeoff using Computer Vision. In 2021 IEEE International Conference on Industry 4.0, Artifi- cial Intelligence, and Communications Technology (IAICT) . 196–200. doi:10.1109/ IAICT52856.2021.9532514

2021
[17]

Dimitrios Kampelopoulos, Athina Tsanousa, Stefanos Vrochidis, and Ioannis Kompatsiaris. 2025. A review of LLMs and their applications in the architecture, engineering and construction industry. Artificial Intelligence Review 58, 8 (2025),

2025
[18]

Zixuan Ke, Yifei Ming, Xuan-Phi Nguyen, Caiming Xiong, and Shafiq Joty. 2025. Demystifying Domain-adaptive Post-training for Financial LLMs. arXiv preprint arXiv:2501.04961 (2025)

2025
[19]

Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob De- vlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and S...
[20]

Xin Li, Weize Chen, Qizhi Chu, Haopeng Li, Zhaojun Sun, Ran Li, Chen Qian, Yiwei Wei, Zhiyuan Liu, Chuan Shi, Maosong Sun, and Cheng Yang. 2024. Can Large Language Models Analyze Graphs like Professionals? A Benchmark, Datasets and Models. In Advances in Neural Information Pro...

2024
[21]

Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michi- hiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al
[22]

Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring How Models Mimic Human Falsehoods. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Smaranda Muresan, Preslav Nakov, and Aline Villavicen...

2022 doi
[23]

Chen Ling, Xujiang Zhao, Jiaying Lu, Chengyuan Deng, Can Zheng, Junxiang Wang, Tanmoy Chowdhury, Yun Li, Hejie Cui, Xuchao Zhang, et al. 2023. Do- main specialization as the key to make large language models disruptive: A comprehensive survey. arXiv preprint arXiv:2305.18703 (2023)

2023 arXiv
[24]

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual In- struction Tuning. In Advances in Neural Information Processing Systems , A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36. Curran Associates, Inc., 34892–34916. https:/...

2023
[25]

Xiaoyu Liu. 2025. Exploring the power of Large Language Models: Automated compliance checks in architecture engineering and construction industries . Ph. D. Dissertation. Cardiff University

2025
[26]

Ollama. 2025. Ollama: Open Large Language Model API. https://ollama.com/. [Online; accessed 10-Jun-2025]

2025
[27]

OpenAI. 2025. Introducing GPT-4.1 in the API. https://openai.com/index/gpt-4-1/. [Online; accessed 10-Jun-2025]

2025
[28]

Samuel A Prieto, Eyob T Mengiste, and Borja García de Soto. 2023. Investigating the use of ChatGPT for the scheduling of construction projects. Buildings 13, 4 (2023), 857

2023
[29]

Riya Shah. 2024. Construction Industry Statistics (2025). https://upmetrics.co/ blog/construction-industry-statistics. [Online; Accessed on 10-Jun-2025]

2024
[30]

Adnan Shaout and Meaghan Bryant. 2025. Exploring the Utility and Challenges of AI Interpretation of Construction Blueprints.International Journal of Advanced Networking and Applications 16 (01 2025), 6627–6646. doi:10.35444/IJANA.2025. 16603

2025 doi
[31]

Jimeng Shi, Azam Shirali, Bowen Jin, Sizhe Zhou, Wei Hu, Rahuul Rangaraj, Shaowen Wang, Jiawei Han, Zhaonan Wang, Upmanu Lall, et al . 2025. Deep Learning and Foundation Models for Weather Prediction: A Survey.arXiv preprint arXiv:2501.06907 (2025)

2025 arXiv
[32]

Mason Smetana, Lucio Salles de Salles, Igor Sukharev, and Lev Khazanovich. 2024. Highway construction safety analysis using large language models. Applied Sciences 14, 4 (2024), 1352

2024
[33]

Zirui Song, Bin Yan, Yuhan Liu, Miao Fang, Mingzhe Li, Rui Yan, and Xiuying Chen. 2025. Injecting domain-specific knowledge into large language models: a comprehensive survey. arXiv preprint arXiv:2502.10708 (2025)

2025 arXiv
[34]

Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786 (2025)

2025 arXiv
[35]

VIKTOR Team. 2025. Mastering LLMs: Essential Prompting Techniques for AEC Professionals. https://www.viktor.ai/blog/194/mastering-ll-ms-essential- prompting-techniques-for-aec-professionals. [Online; Accessed on 10-Jun-2025]

2025
[36]

Erlan Yu, Xuehong Chu, Wanwan Zhang, Xiangbin Meng, Yaodong Yang, Xun- ming Ji, and Chuanjie Wu. 2025. Large Language Models in Medicine: Appli- cations, Challenges, and Future Directions. Int J Med Sci 22 (2025), 2792–2801. doi:10.7150/ijms.111780

2025 doi
[37]

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a Machine Really Finish Your Sentence?. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , Anna Ko- rhonen, David Traum, and Lluís Màrquez (Ed...

2019 doi
[250]

doi:10.1007/s10462-025-11241-7

doi
[2019]

Transactions of the Association for Computational Linguistics 7 (08 2019), 453–466

Natural Questions: A Benchmark for Question Answering Re- search. Transactions of the Association for Computational Linguistics 7 (08 2019), 453–466. doi:10.1162/tacl_a_00276 arXiv:https://direct.mit.edu/tacl/article- pdf/doi/10.1162/tacl_a_00276/1923288/tacl_a_00276.pdf

2019 doi
[2021]

In 2021 IEEE/CVF International Conference on Computer Vision (ICCV)

FloorPlanCAD: A Large-Scale CAD Drawing Dataset for Panoptic Symbol Spotting. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV) . 10108–10117. doi:10.1109/ICCV48922.2021.00997

2021
[2022]

arXiv preprint arXiv:2211.09110 (2022)

Holistic evaluation of language models. arXiv preprint arXiv:2211.09110 (2022)

2022 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauff- mann, et al. 2024. Phi-4 technical report. arXiv preprint arXiv:2412.08905 (2024)

arXiv 2024

[2] [2]

JL Blanco, D Rockhill, A Sanghvi, and A Torres. 2023. From start-up to scale-up: accelerating growth in construction technology. McKinsey & Company (2023)

work page 2023

[3] [3]

Conor Bronsdon. 2024. 6 reasons why the construction industry is slow to em- brace technology. https://underthehardhat.org/6-reasons-why-the-construction- industry-is-slow-to-embrace-technology/. [Online; Accessed on 10-Jun-2025]

work page 2024

[4] [4]

Conor Bronsdon. 2025. 7 Categories of LLM Benchmarks for Evaluating AI Beyond Conventional Metrics. https://galileo.ai/blog/llm-benchmarks-categories. [Online; Accessed on 10-Jun-2025]

work page 2025

[5] [5]

Yu, Qiang Yang, and Xing Xie

Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2024. A Survey on Evaluation of Large Language Models. ACM Trans. Intell. Syst. Technol. 15, 3, Article 39 (March 2024), 45 pages. doi:10.1145/3641289

doi:10.1145/3641289 2024

[6] [6]

Nanjiang Chen, Xuhui Lin, Hai Jiang, and Yi An. 2024. Automated building infor- mation modeling compliance check through a large language model combined with deep learning and ontology. Buildings 14, 7 (2024), 1983

work page 2024

[7] [7]

Adam Chernick, Luke Reeve, and Scott Patten. 2025. AI-Powered Construction Document Analysis by Leveraging Computer Vision and Large Language Mod- els. https://aws.amazon.com/blogs/spatial/ai-powered-construction-document- analysis-by-leveraging-computer-vision-and-large-language-models/. [Online; Accessed on 10-Jun-2025]

work page 2025

[8] [8]

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457 (2018)

arXiv 2018