Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

DrawingVQA, a benchmark on real construction drawings, shows AI matches interns but not expert engineers.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 23:25 UTC pith:Q5PEABWG

load-bearing objection Solid benchmark resource, but the headline human-vs-model numbers are built on mismatched denominators and need a corrected rewrite. the 3 major comments →

arxiv 2607.15418 v1 pith:Q5PEABWG submitted 2026-07-16 cs.AI cs.CV

DrawingVQA: A Real-World Benchmark for Multi-Depth Visual-Textual Reasoning on Construction Drawings

Yoonhwa Jung , Junryu Fu , Mani Golparvar-Fard This is my paper
classification cs.AI cs.CV
keywords construction drawingsmultimodal large language modelsvisual question answeringbenchmarkexpert reasoningquantity take-offreasoning depthdual categorization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

DrawingVQA is a proposed benchmark for measuring how well multimodal large language models read real-world construction drawings — the dense, legally binding 'Issued for Construction' sheet sets used on building sites. Its central claim is that these documents test a kind of visual–textual reasoning that natural-image and schematic-plan benchmarks do not: combined reading of symbols, tables, cross-references, and annotations at professional depth. On 92 expert-written questions across three reasoning levels, the best model scores 71.7%, above the average human participant (68.4%) but far below seasoned professionals (94.9%). The authors take this as evidence that current AI has entry-level drawing literacy but not expert reasoning, and they propose a dual categorization framework — engineering tasks crossed with AI capabilities — to pinpoint exactly where models fail. The concrete payoff would be a diagnostic tool for deciding when models can be trusted in construction workflows.

Core claim

On its own terms, the paper's discovery is that today's multimodal models read construction drawings competently at the perceptual and contextual levels, then collapse at the expert level: while almost every model's score falls from reasoning level 1 to level 3, experienced professionals score highest exactly at level 3. The largest model shortfall is quantity take-off — counting and extracting quantities from drawings — where the best model reaches 41.7% against 96.6% for professionals. The paper also shows that removing the image drops performance near to random guessing, which it takes as confirmation that the benchmark genuinely tests grounded visual reasoning rather than memorized domai

What carries the argument

The benchmark itself is the central object: 33 'Issued for Construction' structural drawings paired with 92 expert-written QA items, each assigned a reasoning depth (perceptual, contextual, expert). The second load-bearing mechanism is the dual categorization framework, which tags every question along an engineering dimension (administration, element identification, language understanding, dimensional reasoning, design semantics, quantity take-off, compliance) and an AI capability dimension (visual perception, knowledge, reasoning, OCR), so accuracy can be diagnosed by failure type rather than by a single score.

Load-bearing premise

The comparison that produces the paper's headline numbers assumes that the 20-question human sample fairly represents all 92 questions, and that the three experts' authored answers are correct ground truth for every item.

What would settle it

Recruit experienced professionals to answer all 92 DrawingVQA questions under the same timed protocol; if their score drops well below 94.9% or the best model's score rises above 71.7%, the reported model-versus-expert gap is not stable.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Benchmark scores on DrawingVQA would let builders and engineers separate model competence in document retrieval from competence in spatial reasoning about a specific project's sheets.
  • If the paper is right, high pass rates on engineering certification exams tell us little about whether a model can trace a grid line or verify a weld callout — the benchmark re-anchors evaluation onto practical task performance.
  • The consistent quantity take-off failure isolates a single capability — precise object counting in dense line drawings — that future multimodal systems must improve before they can support estimating workflows.
  • The dual categorization framework would let benchmark users attribute an error to perception, OCR, knowledge, or reasoning per engineering task, enabling targeted rather than global model fixes.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the model-vs-human comparison rests on only 20 of the 92 questions, the exact 71.7-vs-94.9 spread is a sample estimate; a full human pass could move the numbers, though the qualitative conclusion that experts outperform models on expert-level items would likely stand.
  • The standardization of US construction drawings (NCS) means the benchmark's difficulty may not transfer to non-US or non-standard drafting conventions; the same benchmark structure applied to those conventions would be a natural test.
  • The authors' PDF-hybrid result suggests resolution is not the bottleneck — even machine-readable vector text fails to ground spatially — which points toward training on technical drawings, not bigger images, as the extension path they leave implicit.
  • An extended inference: a model that cannot handle QTO cannot be trusted for cost estimation; so DrawingVQA-style probes could serve as a practical gate for AI adoption in construction, not just a research score.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. DrawingVQA is a benchmark of 92 expert-authored VQA pairs on 33 real-world 'Issued for Construction' structural drawings, spanning three reasoning depths and a dual categorization (seven construction-engineering dimensions x four MLLM capabilities). The paper evaluates proprietary and open-weight MLLMs and reports that Gemini-2.5-Pro (71.7%) surpasses the average human score (68.4%) but falls far short of experienced professionals (94.9%), with the largest gaps at expert-reasoning depth and quantity take-off. Ablations include text-only collapsing performance, option-order permutation, PDF-hybrid input, and an open-ended format check. The paper argues that current MLLMs have entry-level document literacy but not professional-grade drawing reasoning.

Significance. The dataset's authentic source, expert curation, contamination controls, and several falsifiable ablations are strengths. The qualitative finding that models degrade sharply at expert-reasoning depth and QTO is plausible and internally consistent. However, the headline human/model comparison mixes denominators (92-item model scores vs. 20-item human scores), so the quantitative claim is not yet established. With a corrected statistical footing, the benchmark would be a valuable community resource.

major comments (3)
  1. [Sec. 4.1, Table 2, Sec. 4.2] Model accuracy is computed on all 92 questions, while human scores come from a stratified 20-question subset (4/8/8) with a 20-minute time limit. The reported human rows in Table 2 thus rest on very small per-cell counts, and no analysis shows the subset is representative. The headline 71.7 vs. 94.9 comparison is therefore not quantitatively established. Please report model scores on the same 20 items, collect a full human baseline, or provide confidence intervals and a representativeness check.
  2. [Supplementary F.2, Table 7] Gemini-2.5-Pro is listed with endpoint 'gemini-2.5-flash' and Gemini-2.5-Flash with 'gemini-2.5-pro'. If this is not a typo, the model attributions in Table 2 are unreliable. Please verify endpoints and correct the table.
  3. [Sec. 4.2, Table 2] No error bars or confidence intervals are reported for model or human accuracies. With n=92, Gemini-2.5-Pro (71.7) vs. average human (68.4) is within sampling noise; with n=20 for human subcohorts, per-domain rows are unstable. Report binomial/bootstrap CIs and exact item counts.
minor comments (5)
  1. [Sec. 4.2 vs Supplementary Table 8] Realism ratings differ: main text says 32.7% rated '4' and 38.5% rated '5'; Table 8 reports 32.1% and 39.6%. Please make consistent.
  2. [Sec. 4.1] '52 Responses' is ambiguous: 52 participants or 52 forms? Please report participant counts per cohort and per-question response counts.
  3. [Supplementary D.6, D.7] The MCQ-vs-open and unseen-drawings comparisons report only p-values (0.09, 0.07). Please include sample sizes, effect sizes, and test statistics.
  4. [References] The reference list has duplicate numeric keys and some citations do not match their intended entries. Please clean up to allow verification of related-work claims.
  5. [Table 3] The main-paper text-only ablation shows only two models. Since this is a strong sanity check, consider reporting the full table or explicitly directing readers to the supplementary table.

Circularity Check

0 steps flagged

No significant circularity: benchmark scores are externally measured against real drawings and expert-authored answers, with ablations that could refute the framing.

full rationale

The paper's central contribution is an evaluation benchmark, not a derived prediction. Model accuracies are measured against externally held IFC drawings and expert-validated answer keys; the text-only ablation (Table 3), option-permutation test (Table 14), and PDF-hybrid setting (Table 15) all provide independent checks that the benchmark is vision-grounded and not solvable from linguistic priors. The dual categorization taxonomy is author-defined, but it is used descriptively to organize error patterns rather than to generate the reported scores, so it does not reduce to its own labels. The only self-citation (VisualSiteDialy, ref. [13]) appears in related-work context and is not load-bearing. One genuine limitation exists but is not circularity: the human baseline in §4.1 was collected on a stratified 20-item subset while models are scored on all 92 items, so the quantitative model-vs-expert gap (71.7 vs. 94.9) rests on an assumption of subset representativeness; however, human scores are not fitted parameters or inputs to the model scores, so this is a validity concern rather than a circular derivation. No step in the paper equates a prediction with an input by construction.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 2 invented entities

The benchmark itself is an empirical artifact rather than a derived theory, so there are no fitted model parameters or physical postulates. The main epistemic costs are: (1) 100 PPI was chosen by model performance, (2) expert answers and taxonomy labels are treated as ground truth without inter-rater statistics, (3) representativeness of 33 drawings from 6 campus projects is assumed, and (4) the 20-question human baseline is compared with full-benchmark model scores. These are the load-bearing premises a re-analysis should check first.

free parameters (1)
  • Default image resolution (100 PPI) = 100 PPI
    Chosen from an ablation study (Supp. Table 13) as the resolution with the highest average model accuracy (55.7%), not set a priori from industry practice. Model rankings shift somewhat across resolutions (e.g., Gemini-2.5-flash peaks at 100 PPI, GPT-4o at 125 PPI).
axioms (4)
  • domain assumption Expert-authored QA pairs and their answers are correct, unambiguous ground truth.
    All accuracy numbers and the model-vs-expert gap derive from these labels. Validation is internal multi-pass expert review (§3.3); no inter-annotator agreement statistic is reported.
  • domain assumption The 33 IFC drawings from six campus building projects are representative of professional construction drawings generally.
    Generalization is argued via NCS standardization (§3.3) and tested on only 7 new drawings with 49 questions at p=0.07 (Supp. D.7) — a marginal result with small sample size.
  • ad hoc to paper A 20-question timed human sample is comparable to untimed model evaluation over all 92 questions.
    §4.1: 'we randomly selected a subset of 20 QA pairs... 52 responses were collected from three groups.' §4.2 then compares these human means (68.4 / 94.9) directly with model scores computed over the full benchmark, with no correction or significance test.
  • domain assumption MCQ format with expert distractors does not change task difficulty relative to open-ended questions.
    Tested in Supp. D.6 on 20 items with a paired t-test p=0.09, n=20; the null hypothesis of 'no difference' is not rejected, but the power to detect a difference is low, and the limitation is acknowledged in Supp. C.
invented entities (2)
  • DrawingVQA benchmark dataset independent evidence
    purpose: Evaluates MLLMs on real IFC construction drawings across three reasoning depths and a dual categorization framework.
    The dataset itself is shipped on HuggingFace; independent groups can run it, review images, and check expert answers — a falsifiable handle outside the paper.
  • Dual-categorization taxonomy (7 construction-engineering x 4 MLLM-capability dimensions) no independent evidence
    purpose: Diagnoses model failures by mapping engineering workflows to AI reasoning competencies, e.g., QTO requires Reasoning (Alignment) + Visual Perception + OCR.
    A new author-defined labeling scheme; no inter-annotator reliability or external validation is provided, so its diagnostic claims are self-referential until independently reproduced.

pith-pipeline@v1.3.0-alltime-deepseek · 27029 in / 11193 out tokens · 116843 ms · 2026-08-01T23:25:08.065834+00:00 · methodology

0 comments
read the original abstract

We introduce DrawingVQA, the first benchmark designed to evaluate multimodal large language models (MLLMs) on real-world construction drawings -- a core media in architecture, civil, and many other engineering practices. Unlike natural images or schematic floor plans, construction drawings fuse abstract geometry, symbolic notation, tabular data, annotations, and domain-specific text, forming a uniquely complex visual-textual domain core to engineering workflows. DrawingVQA bridges this gap with 33 "Issued for Construction" drawings and 92 expertly curated question-answer pairs, spanning three reasoning depths: perceptual understanding, contextual interpretation, and domain-expert reasoning. To evaluate model capabilities, we present a dual categorization framework to jointly analyze performance across seven construction-engineering and four MLLM capability dimensions -- the first to explicitly map engineering workflows to AI reasoning competencies. Evaluations of state-of-the-art MLLMs reveal a substantial gap between model and expert performance, particularly at higher reasoning depths. This benchmark lays a foundation for domain-specialized multimodal reasoning to allow for advancement on integration of AI-driven understanding and real-world engineering workflows.

Figures

Figures reproduced from arXiv: 2607.15418 by Junryu Fu, Mani Golparvar-Fard, Yoonhwa Jung.

Figure 1
Figure 1. Figure 1: The Multimodal Challenges of IFC Drawings and DrawingVQA’s Engineering-Level VQA. (A) Deconstructing IFC Drawing Complexity: Real-world “Issued for Construction” (IFC) drawings are information-dense artifacts that present unique multimodal challenges (See Section 3.2). (B) DrawingVQA VQA Examples: We present challenging question-answer pairs from DrawingVQA that demand industry-professional engineering rea… view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of drawing complexity. (a) A low-density, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Visualization of the dual categorization mapping MLLM [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of DrawingVQA with other VQA benchmarks, which contain any of Architecture, Civil, Structural, or Construction Engineering disciplines, or drawing-related questions. Q. means Question, Expln. means Explanations, Profl. means Professional, Eng. means Engineering, MC means multiple-choice, and Open means open-ended. DesignQA is not a pure VQA dataset, with document￾reference dependence. Compared t… view at source ↗
Figure 5
Figure 5. Figure 5: Comparison of multimodal reasoning performance [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Dual-Category Error Breakdown. F.7. More experiments on Image resolution We investigate the impact of image resolution on the performance of MLLMs within the context of engineering drawings. In our dataset construction, we distinguish between Region of Interest (ROI) images and Full Drawing Sheets. While ROI images are typically user-generated crops that do not require standardization, the Full Drawing She… view at source ↗
Figure 7
Figure 7. Figure 7: Visual Perception and Visual Grounding Errors for Structural Beam Identification based on Grid Lines (by Gemini-2.5-pro). [PITH_FULL_IMAGE:figures/full_fig_p024_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: A Mix of Visual and Text Perception Errors for Column Schedule Tabular Identifications (by Gemini-2.5-pro). [PITH_FULL_IMAGE:figures/full_fig_p025_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Text Perception (OCR) Errors on Beam Annotations (by Qwen3-VL-8B-Instruct). [PITH_FULL_IMAGE:figures/full_fig_p026_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Knowledge and perception error on reading section detail tags (by Gemini-2.5-pro). [PITH_FULL_IMAGE:figures/full_fig_p027_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Visual Perception and Reasoning Error on Recognizing Drain Tiles (by Gemini-2.5-pro). [PITH_FULL_IMAGE:figures/full_fig_p028_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Knowledge, Reasoning and Perception Error while calculating length of elements (by Gemini-2.5-pro). [PITH_FULL_IMAGE:figures/full_fig_p029_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Original example on “all-around” welding example (Gemini-2.5-pro). [PITH_FULL_IMAGE:figures/full_fig_p030_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Adversarial example to exclude “all-around” circle symbol to highlight parametric knowledge bias in MLLM (Gemini-2.5-pro). [PITH_FULL_IMAGE:figures/full_fig_p031_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Visual attention maps of Qwen3-VL-8B-Instruct. The maps illustrate where the model “looks” when processing each text token. [PITH_FULL_IMAGE:figures/full_fig_p033_15.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Evidence-Grounded Constraint Checking in Construction Documents

    cs.AI 2026-07 conditional novelty 5.0

    A controlled four-image comparison shows region-focused crops beat page overviews on six projects but lose on 23 projects, indicating a resolution–breadth tradeoff rather than a dominant strategy.

Reference graph

Works this paper leans on

58 extracted references · 11 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Vqa-med: Overview of the medical visual question answering task at imageclef 2019.CLEF (working notes), 2(6):1–11, 2019

    Asma Ben Abacha, Sadid A Hasan, Vivek V Datla, Joey Liu, Dina Demner-Fushman, and Henning M¨ uller. Vqa-med: Overview of the medical visual question answering task at imageclef 2019.CLEF (working notes), 2(6):1–11, 2019

    2019

  2. [2]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  3. [3]

    OCR, Knowledge, Reasoning (Vi- sual, Alignment)

    Discipline Language UnderstandingIdentifying discipline-specific acronyms, abbreviations, and technical nomenclature. OCR, Knowledge, Reasoning (Vi- sual, Alignment)

  4. [4]

    Vqa: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. InProceedings of the IEEE international conference on computer vision, pages 2425– 2433, 2015

    2015

  5. [5]

    Improved baselines with visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023

    2023

  6. [6]

    Are large pre-trained vision language models effective construction safety inspec- tors?, 2025

    Xuezheng Chen and Zhengbo Zou. Are large pre-trained vision language models effective construction safety inspec- tors?, 2025

    2025

  7. [7]

    Doris, Daniele Grandi, Ryan Tomich, Md Ferdous Alam, Mohammadmehdi Ataei, Hyunmin Cheong, and Faez Ahmed

    Anna C. Doris, Daniele Grandi, Ryan Tomich, Md Ferdous Alam, Mohammadmehdi Ataei, Hyunmin Cheong, and Faez Ahmed. Designqa: A multimodal benchmark for evaluating large language models’ understanding of engineering docu- mentation, 2024

    2024

  8. [8]

    Galaz- Montoya, Yuhui Zhang, Yuchang Su, Disha Bhowmik, Zachary Coman, Sarina M Hasan, Alexandra Johannesson, William D

    James Burgess, Jeffrey J Nirschl, Laura Bravo-S ´anchez, Alejandro Lozano, Sanket Rajan Gupte, Jesus G. Galaz- Montoya, Yuhui Zhang, Yuchang Su, Disha Bhowmik, Zachary Coman, Sarina M Hasan, Alexandra Johannesson, William D. Leineweber, Malvika G Nair, Ridhi Yarlagadda, Connor Zuraski, Wah Chiu, Sarah Cohen, Jan N. Hansen, Manuel D Leonetti, Chad Liu, Emm...

    2025

  9. [9]

    Floorplancad: A large-scale cad draw- ing dataset for panoptic symbol spotting

    Zhiwen Fan, Lingjie Zhu, Honghua Li, Xiaohao Chen, Siyu Zhu, and Ping Tan. Floorplancad: A large-scale cad draw- ing dataset for panoptic symbol spotting. InProceedings of the IEEE/CVF international conference on computer vision, pages 10128–10137, 2021

    2021

  10. [10]

    Llava- onevision-1.5: Fully open framework for democratized mul- timodal training, 2025

    Xiang An, Yin Xie, Kaicheng Yang, Wenkang Zhang, Xiuwei Zhao, Zheng Cheng, Yirui Wang, Songcen Xu, Changrui Chen, Chunsheng Wu, Huajie Tan, Chunyuan Li, Jing Yang, Jie Yu, Xiyao Wang, Bin Qin, Yumeng Wang, Zizhen Yan, Ziyong Feng, Ziwei Liu, Bo Li, and Jiankang Deng. Llava- onevision-1.5: Fully open framework for democratized mul- timodal training, 2025

    2025

  11. [11]

    Mme-finance: A multimodal finance benchmark for expert-level understanding and rea- soning

    Ziliang Gan, Dong Zhang, Haohan Li, Yang Wu, Xueyuan Lin, Ji Liu, Haipang Wu, Chaoyou Fu, Zenglin Xu, Rongjunchen Zhang, et al. Mme-finance: A multimodal finance benchmark for expert-level understanding and rea- soning. InProceedings of the 33rd ACM International Con- ference on Multimedia, pages 12867–12874, 2025

    2025

  12. [12]

    RBench: Graduate-level multi-disciplinary benchmarks for LLM &; MLLM complex reasoning evalu- ation

    Meng-Hao Guo, Jiajun Xu, Yi Zhang, Jiaxi Song, Haoyang Peng, Yi-Xuan Deng, Xinzhi Dong, Kiyohiro Nakayama, Zhengyang Geng, Chen Wang, Bolin Ni, Guo-Wei Yang, Yongming Rao, Houwen Peng, Han Hu, Gordon Wetzstein, and Shi-Min Hu. RBench: Graduate-level multi-disciplinary benchmarks for LLM &; MLLM complex reasoning evalu- ation. InProceedings of the 42nd Int...

    2025

  13. [13]

    Omnimedvqa: A new large- scale comprehensive evaluation benchmark for medical lvlm

    Yutao Hu, Tianbin Li, Quanfeng Lu, Wenqi Shao, Junjun He, Yu Qiao, and Ping Luo. Omnimedvqa: A new large- scale comprehensive evaluation benchmark for medical lvlm. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22170–22183, 2024

    2024

  14. [14]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. InProceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 6700–6709, 2019

    2019

  15. [15]

    Yoonhwa Jung, Ikhyun Cho, Shun-Hsiang Hsu, and Mani Golparvar-Fard. Visualsitediary: A detector-free vision- language transformer model for captioning photologs for daily construction reporting and image retrievals.Automation in Construction, 165:105483, 2024

    2024

  16. [16]

    Seed-bench: Benchmark- ing multimodal large language models

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmark- ing multimodal large language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13299–13308, 2024

    2024

  17. [17]

    Drafterbench: Bench- marking large language models for tasks automation in civil engineering, 2025

    Yinsheng Li, Zhen Dong, and Yi Shao. Drafterbench: Bench- marking large language models for tasks automation in civil engineering, 2025

    2025

  18. [18]

    SceMQA: A scientific college entrance level multimodal question answering benchmark

    Zhenwen Liang, Kehan Guo, Gang Liu, Taicheng Guo, Yujun Zhou, Tianyu Yang, Jiajun Jiao, Renjie Pi, Jipeng Zhang, and Xiangliang Zhang. SceMQA: A scientific college entrance level multimodal question answering benchmark. InPro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 109–119, Bang...

    2024

  19. [19]

    Gemex: A large-scale, groundable, and explainable medical vqa benchmark for chest x-ray diagnosis

    Bo Liu, Ke Zou, Li-Ming Zhan, Zexin Lu, Xiaoyu Dong, Yidi Chen, Chengqiang Xie, Jiannong Cao, Xiao-Ming Wu, and Huazhu Fu. Gemex: A large-scale, groundable, and explainable medical vqa benchmark for chest x-ray diagnosis. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 21310–21320, 2025

    2025

  20. [20]

    Mmbench: Is your multi-modal model an all-around player? InComputer Vision – ECCV 2024, pages 216–233, Cham, 2025

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player? InComputer Vision – ECCV 2024, pages 216–233, Cham, 2025. Springer Nature Switzerland

    2024

  21. [21]

    Micro-bench: A microscopy benchmark for vision- language understanding.Advances in Neural Information Processing Systems, 37:30670–30685, 2024

    Alejandro Lozano, Jeffrey Nirschl, James Burgess, San- ket Rajan Gupte, Yuhui Zhang, Alyssa Unell, and Serena Yeung. Micro-bench: A microscopy benchmark for vision- language understanding.Advances in Neural Information Processing Systems, 37:30670–30685, 2024

    2024

  22. [22]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. InAdvances in Neural Information Processing Systems, pages 2507–2521. Curran Associates, Inc., 2022

    2022

  23. [23]

    FinMME: Benchmark dataset for financial multi-modal reasoning evaluation

    Junyu Luo, Zhizhuo Kou, Liming Yang, Xiao Luo, Jinsheng Huang, Zhiping Xiao, Jingshu Peng, Chengzhong Liu, Ji- aming Ji, Xuanzhe Liu, Sirui Han, Ming Zhang, and Yike Guo. FinMME: Benchmark dataset for financial multi-modal reasoning evaluation. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers...

    2025

  24. [24]

    Archcad-400k: A large-scale cad drawings dataset and new baseline for panoptic symbol spotting

    Ruifeng Luo, Zhengjie Liu, Tianxiao Cheng, Jie Wang, Tongjie Wang, Fei Cheng, Fu Chai, Yanpeng Li, Xingguang Wei, Haomin Wang, et al. Archcad-400k: A large-scale cad drawings dataset and new baseline for panoptic symbol spotting. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems

  25. [25]

    Residential floor plan recognition and reconstruction

    Xiaolei Lv, Shengchu Zhao, Xinyang Yu, and Binqiang Zhao. Residential floor plan recognition and reconstruction. InPro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16717–16726, 2021

    2021

  26. [27]

    Phi-4-mini technical re- port: Compact yet powerful multimodal language models via mixture-of-loras, 2025

    Microsoft, :, Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Con- gcong Chen, Dong Chen, Dongdong Chen, Junkun Chen, Weizhu Chen, Yen-Chun Chen, Yi ling Chen, Qi Dai, Xiyang Dai, Ruchao Fan, Mei Gao, Min Gao, Amit Garg, Ab- hishek Goswami, Junheng Hao, Amr Hendy, ...

    2025

  27. [28]

    A-okvqa: A bench- mark for visual question answering using world knowledge

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A bench- mark for visual question answering using world knowledge. InEuropean conference on computer vision, pages 146–162. Springer, 2022

    2022

  28. [29]

    Qwen3 technical report, 2025

    Qwen Team. Qwen3 technical report, 2025

    2025

  29. [31]

    Cequest: Bench- marking large language models for construction estimation

    Yanzhao Wu, Lufan Wang, and Rui Liu. Cequest: Bench- marking large language models for construction estimation. arXiv preprint arXiv:2508.16081, 2025

    Pith/arXiv arXiv 2025

  30. [32]

    Can ai master construction management (cm)? benchmarking state-of-the-art large language models on cm certification exams, 2025

    Ruoxin Xiong, Yanyu Wang, Suat Gunhan, Yimin Zhu, and Charles Berryman. Can ai master construction management (cm)? benchmarking state-of-the-art large language models on cm certification exams, 2025

    2025

  31. [33]

    Mm-vet: evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: evaluating large multimodal models for integrated capabilities. InProceedings of the 41st International Confer- ence on Machine Learning. JMLR.org, 2024

    2024

  32. [34]

    Mmmu: A massive multi-discipline multimodal understand- ing and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Ren- liang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understand- ing and reasoning benchmark for...

    2024

  33. [35]

    Deep floor plan recognition using a multi-task network with room-boundary-guided attention

    Zhiliang Zeng, Xianzhi Li, Ying Kin Yu, and Chi-Wing Fu. Deep floor plan recognition using a multi-task network with room-boundary-guided attention. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 9096–9104, 2019

    2019

  34. [36]

    M3exam: A multilingual, multi- modal, multilevel benchmark for examining large language models.Advances in Neural Information Processing Systems, 36:5484–5505, 2023

    Wenxuan Zhang, Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. M3exam: A multilingual, multi- modal, multilevel benchmark for examining large language models.Advances in Neural Information Processing Systems, 36:5484–5505, 2023

    2023

  35. [37]

    Pmc-vqa: Visual in- struction tuning for medical visual question answering, 2024

    Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc-vqa: Visual in- struction tuning for medical visual question answering, 2024. DrawingVQA: A Real-World Benchmark for Multi-Depth Visual–Textual Reasoning on Construction Drawings Supplementary Material Contents

    2024

  36. [38]

    Overview ofDrawingVQA

    Related Work 2 3.DrawingVQA 3 3.1. Overview ofDrawingVQA. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 3.2. Construction Drawings (Issued for Construction) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 3.3. Data Curation Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ....

  37. [39]

    Baselines

    Experiments 5 4.1. Baselines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 4.2. Main Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

  38. [40]

    expert bottleneck

    Conclusion 8 A . Full Main Results S3 B . Author contributions S3 C . Limitations and future work S4 D . Benchmark details S4 D.1 . AccessingDrawingVQA. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . S4 D.2 . General Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ...

    2025

  39. [41]

    OCR, Visual Perception

    General Admin.Identifying project participants, sheet metadata, dates, and drawing types. OCR, Visual Perception

  40. [42]

    Visual Perception, Reasoning (Alignment, Visual), OCR

    Domain Element IdentificationInterpreting specific architectural, struc- tural element symbols, callout symbols and other administrative components. Visual Perception, Reasoning (Alignment, Visual), OCR

  41. [43]

    Reasoning (Alignment, Spatial), OCR

    Dimensional UnderstandingMapping sections to plan views, locating intersections, and understanding spatial off- sets. Reasoning (Alignment, Spatial), OCR

  42. [44]

    Knowledge, OCR, Reasoning (Vi- sual, Spatial)

    Design Semantic UnderstandingInferring coordination needs, distinguish- ing design alternatives, and interpreting context. Knowledge, OCR, Reasoning (Vi- sual, Spatial)

  43. [45]

    Reasoning (Alignment), Visual Per- ception, OCR

    Quantity Take-Off (QTO)Counting elements, measuring linear/area quantities, and extracting categories from schedules. Reasoning (Alignment), Visual Per- ception, OCR

  44. [46]

    The answer is (X)\

    Compliance & SpecsMatching components to specifications, identifying scope gaps, or verifying code compliance. Knowledge, Reasoning (Alignment, Visual, Spatial), OCR F. Experiments details F.1. Evaluation Prompts and Parsing To evaluate the model’s performance, we structured the input prompts to encourage Chain-of-Thought (CoT) reasoning and specified a s...

    2024

  45. [47]

    Xiong, Y

    R. Xiong, Y. Wang, S. Gunhan, Y. Zhu, and C. Berryman. Can ai master construction management (cm)? benchmarking state-of-the-art large language models on cm certification exams. arXiv:2504.08779, 2025

    Pith/arXiv arXiv 2025

  46. [48]

    A. C. Doris, D. Grandi, R. Tomich, M. F. Alam, M. Ataei, H. Cheong, and F. Ahmed. Designqa: A multimodal benchmark for evaluating large language models’ understanding of engineering documentation. arXiv:2404.07917, 2024

    Pith/arXiv arXiv 2024

  47. [49]

    S. Kang, J. Kim, J. Kim, and S. J. Hwang. See what you are told: Visual attention sink in large multimodal models. arXiv:2503.03321, 2025

    Pith/arXiv arXiv 2025

  48. [50]

    Llama-3.2-11b-vision-instruct

    Meta. Llama-3.2-11b-vision-instruct. Technical report, 2024

    2024

  49. [51]

    H. Liu, C. Li, Y. Li, and Y. J. Lee. Improved baselines with visual instruction tuning. arXiv:2310.03744, 2023

    Pith/arXiv arXiv 2023

  50. [52]

    The llama 4 herd: The beginning of a new era of natively multimodal ai innovation, April 2025

    Meta AI. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation, April 2025

    2025

  51. [53]

    Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras

    Microsoft et al. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras. arXiv:2503.01743, 2025

    Pith/arXiv arXiv 2025

  52. [54]

    Burgess et al

    J. Burgess et al. Microvqa: A multimodal reasoning benchmark for microscopy-based scientific research. InCVPR, pages 19552– 19564, 2025

    2025

  53. [55]

    Yue et al

    X. Yue et al. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. InCVPR, pages 9556–9567, 2024

    2024

  54. [56]

    An et al

    X. An et al. LLaV A-onevision-1.5: Fully open framework for democratized multimodal training. arXiv:2509.23661, 2025

    Pith/arXiv arXiv 2025

  55. [57]

    Qwen3 technical report

    Qwen Team. Qwen3 technical report. arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  56. [58]

    Wang et al

    W. Wang et al. InternVL3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv:2508.18265, 2025

    Pith/arXiv arXiv 2025

  57. [59]

    Z. Yang, P. Jian, and C. Li. Option symbol matters: Investigating and mitigating multiple-choice option symbol bias of large language models. InNAACL, pages 1902–1917, 2025

    1902

  58. [60]

    Zheng, H

    C. Zheng, H. Zhou, F. Meng, J. Zhou, and M. Huang. Large language models are not robust multiple choice selectors. arXiv:2309.03882, 2024

    Pith/arXiv arXiv 2024