Automated Parsing of Engineering Drawings for Structured Information Extraction Using a Fine-tuned Document Understanding Transformer

Jun Ming Tan; Lequn Chen; Muhammad Tayyab Khan; Seung Ki Moon; Wenhe Feng; Zane Yong

REVIEW 4 major objections 5 minor 11 references

Automated Parsing of Engineering Drawings for Structured Information Extraction Using a Fine-tuned Document Understanding Transformer

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A hybrid YOLOv11-obb and fine-tuned Donut pipeline extracts nine types of annotations from 2D engineering drawings as structured JSON, with a single multi-category model beating category-specific ones on precision, recall, and F1.

desk verdict Useful engineering pipeline, but the headline F1 is not verifiable—no train/test split is described, so treat the single-model superiority claim as unconfirmed. read the letter →

arxiv 2505.01530 v3 pith:S3Q4TEO4 submitted 2025-05-02 cs.CV cs.AI

Muhammad Tayyab Khan , Zane Yong , Lequn Chen , Jun Ming Tan , Wenhe Feng , Seung Ki Moon This is my paper · ORCID

classification cs.CVcs.AI

keywords 2DengineeringdrawingsstructuredinformationextractiondocumentunderstandingtransformerDonutYOLOv11orientedboundingboxfine-tuninghallucinationrate

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that pairing oriented object detection with an OCR-free document-parsing transformer can turn cluttered 2D engineering drawings into structured JSON annotations with near-perfect recall and low hallucination. Specifically, a single Donut model trained across all nine annotation categories (GD&T, general tolerances, measures, materials, notes, radii, surface roughness, threads, title blocks) outperforms nine separate category-specific models on precision, F1, and hallucination rate, reaching 97.3% F1 and a 5.23% hallucination rate for GD&T. The design matters because the detection stage first isolates oriented, overlapping symbols that generic OCR handles poorly, and the JSON output is directly usable by downstream manufacturing software. The paper's recommended configuration is the single model for broad, high-recall extraction.

What carries the argument

The load-bearing mechanism is the two-stage pipeline. YOLOv11-obb first predicts oriented bounding boxes for nine annotation categories; each detected box is cropped into a small patch, converting a densely annotated drawing into simple single-annotation images. A fine-tuned Donut model—a transformer that maps document images directly to structured token sequences without OCR—then turns each patch into JSON. Data augmentation (sharpness, contrast, rotation, grayscale, inversion) expands 1,000 expert-labeled patches into 6,000 image-label pairs, and the single-model versus category-specific comparison is the paper's central experimental variable.

What would settle it

Take a held-out set of 2D engineering drawings never used for training or augmentation, run the full YOLOv11-obb plus Donut pipeline, and compute per-category precision, recall, F1, and hallucination rate on that set. If the F1 drops substantially below 97.3% or the single model stops beating the category-specific models, the central generalization claim is refuted; Materials and General Tolerances are the most likely categories to show this.

Watch

Extended reading notes

Core claim

The central claim, stated on the paper's own terms, is that a single fine-tuned Donut model trained across all nine annotation categories outperforms nine separate category-specific models on structured extraction from 2D engineering drawings. Across precision, recall, F1, and hallucination rate, the single model is equal or better on every category, with reported values of 94.77% precision for GD&T, 100% recall on most categories, and 97.3% overall F1, while cutting hallucination rates (5.23% for GD&T). The mechanism offered is exposure to the full annotation space, which gives the model broader contextual patterns and fewer false positives.

Load-bearing premise

The central claim assumes that the reported metrics measure performance on drawings the models have not been trained on or augmented from, but the paper gives no train/validation/test split.

Editorial extensions

If this is right

A single multi-category model is the recommended deployment choice, since it beats category-specific models on every reported metric while keeping recall near 100% on most categories.
The pipeline's JSON output can feed directly into CAD/CAM, quality-assurance, and process-planning tools without additional post-processing.
The semi-automated labeling workflow—detect, crop, label a subset, augment—substantially reduces annotation effort and can be reused to add new drawing categories.
High recall on most categories means few annotations are missed, which is critical in precision manufacturing where a missed tolerance can scrap a part.
Materials and general tolerances remain weak spots, with hallucination rates far above the other categories, so those outputs need separate attention before industrial deployment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the reported metrics come from the same data used for training or augmentation—the paper does not state a train/validation/test split—the 97.3% F1 is likely optimistic, and a held-out evaluation would be the decisive test.
The same 'crop oriented regions, then parse with an OCR-free transformer' template should transfer to other technical documents, such as electrical schematics or maps, where annotations are rotated and overlapping.
As more categories are added, the single-model advantage may erode unless class imbalance is addressed, since the categories that fail here are precisely the low-sample ones.
A human-in-the-loop verification step focused on high-hallucination categories would be a cheap way to make the pipeline production-safe.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Quick take on arXiv:2505.01530. The paper builds a two-stage pipeline: YOLOv11-obb detects nine annotation categories on 2D engineering drawings, then Donut fine-tuned on cropped patches emits JSON. That combination is sensible and practical for manufacturing document understanding. The comparison of one single Donut model versus nine category-specific models is a legitimate experimental question, and the reported results favor the single model on precision, recall, and hallucination rate.

What's genuinely new: the in-house annotated dataset (1,367 drawings, 11,469 patches, 1,000 expert-labeled and augmented to 6,000) and the fine-tuning comparison. The writing is clear and the method is easy to follow. Reporting hallucination rate is a nice touch, rare in this literature. They also honestly note the weak categories (Materials, General Tolerances) where both strategies struggle.

The soft spot is the evaluation protocol, and it is load-bearing. Nowhere does the paper state which images produced the numbers in Section III. Section II describes augmenting 1,000 patches to 6,000 training pairs; Section III reports per-category precision, recall, F1, and hallucination without defining a held-out test set. If those numbers come from training or augmented training data, the 97.3% F1 and the single-model superiority claim measure fit, not generalization. No error bars, no confidence intervals, and no defined matching criterion for JSON predictions. There is also an internal inconsistency: the Title Block recall sentence says recall drops from 72.9% to 56.3%, but 72.9% was earlier reported as the single model's precision for Title Blocks. And Donut is never cited, which is surprising for the model at the core of the paper.

None of this kills the idea, but it means the central empirical claim is not independently checkable. The authors need to specify the split, report test-set metrics with variance, add an end-to-end evaluation that includes YOLO detection errors, and compare against at least their own prior OCR-based method as a baseline. Without those, the paper is a recipe, not a result.

Who is this for? Practitioners working on engineering drawing extraction who want a working recipe. For the broader document-understanding literature it is incremental. Still, the dataset and the comparison are worth referee time if the authors can fix the evaluation. I would send it to review with a clear request for the missing details. I would not cite it in its current form, but I'd keep an eye on a revised version.

Maybe bring it to reading group as a case study in how easy it is to omit a test split.

Referee Report

4 major / 5 minor

Summary. The paper proposes a two-stage pipeline for extracting structured information from 2D engineering drawings: YOLOv11 with oriented bounding boxes detects nine annotation categories (GD&T, General Tolerances, Measures, Materials, Notes, Radii, Surface Roughness, Threads, Title Blocks), and a fine-tuned Donut transformer parses the cropped patches into JSON. The authors compare a single Donut model trained on all categories with nine category-specific models, reporting that the single model gives higher precision, recall, F1, and lower hallucination rates. The claimed best result is 97.3% F1 with a 5.23% hallucination rate for GD&T. The paper is an empirical study on an in-house annotated dataset of 1,367 drawings, with 1,000 manually labeled patches augmented to 6,000 for Donut fine-tuning.

Significance. If the reported results are reliable, the contribution is practically useful: a modular, OCR-free framework that outputs structured JSON directly usable in CAD/CAM and quality-assurance workflows, with a semi-automated labeling procedure that reduces annotation burden. The comparison of single versus category-specific fine-tuning is a relevant design question for document-understanding transformers. The paper also provides concrete training details (30 epochs, Adam, batch size 1, RTX 4090) and a reproducible augmentation list. However, the significance is currently limited by the absence of a clear evaluation protocol: no held-out split, no end-to-end assessment including YOLO detection errors, and no defined matching criterion for JSON predictions. These omissions prevent the reader from verifying whether the reported numbers reflect generalization to unseen drawings.

major comments (4)

[Section II and Section III (Fig. 4)] The manuscript never states which images were used to compute the precision, recall, F1, and hallucination values in Fig. 4. Section II says 1,000 patches are augmented to 6,000 image-label pairs 'used for fine-tuning,' and Section III reports per-category metrics without defining a test set. If these numbers come from the training data or the original 1,000 patches, they measure fit to the training distribution, not generalization to unseen drawings. Please specify the exact train/validation/test split, the source of the reported numbers, and confirm that no augmented copies of test patches were present in training.
[Section III, Title Blocks recall sentence] The sentence 'A slight reduction is observed for Title Blocks in the category-specific model, where recall drops from 72.9% to 56.3%' is internally inconsistent with the previous paragraph, which reports a Title Blocks precision of 72.9% for the single model and 55.1% for the category-specific model. The sentence appears to mix precision and recall values, and the claimed recall drop from 72.9% to 56.3% is not supported by any earlier statement. Please clarify the correct per-category precision and recall values for Title Blocks and audit the rest of the results for similar transcription errors.
[Section II and Section III] The evaluation in Fig. 4 appears to measure only the Donut parsing stage on cropped patches, not the full hybrid pipeline. Since the framework includes YOLOv11-obb detection, detection misses, false positives, and inaccurate bounding boxes will propagate into the final structured output, but no end-to-end precision, recall, or F1 is reported. Please either provide end-to-end metrics on complete drawings or explicitly state and justify that the reported patch-level metrics are the intended evaluation scope.
[Section III, metrics definitions] The definitions of precision, recall, F1, and hallucination rate do not specify the matching criterion between predicted JSON and ground-truth JSON. Without knowing whether a prediction is considered correct only on exact string match, on semantic equivalence of field values, or on some partial-credit scheme, the reported numbers cannot be reproduced or interpreted. Additionally, no error bars, confidence intervals, or repeated-run statistics are given. Please define the matching criterion precisely and report variance or per-run results.

minor comments (5)

[Abstract] The abstract claims 'higher precision (94.77% for GD&T), recall (100% for most categories), and F1 score (97.3%)' but does not state whether these refer to the single model or the category-specific models; the results section should make this explicit.
[Section II, data augmentation] The augmentation list does not mention whether rotations are applied only to axis-aligned patches and how the JSON ground truth is transformed accordingly. Please specify how the ground-truth labels are adjusted under rotation and inversion.
[Section II, hyperparameters] The training setup states batch size 1 and 30 epochs but omits learning rate, warmup schedule, and tokenizer settings for Donut. Please add these details for reproducibility.
[Section III, Materials and General Tolerances] The discussion attributes low F1 for Materials and General Tolerances to 'limited training samples,' but Fig. 2 shows the category distribution before and after augmentation. Please state the exact per-category training and test counts, since the augmentation appears to multiply all categories equally and would not resolve imbalance.
[General] The manuscript contains several grammatical and typographical errors (e.g., 'process es' in Section II, 'requ ire s' in Section II, 'outperform s' in Section IV). A careful proofreading pass is needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper reports empirical supervised-learning results, with no derivation step that reduces to its own inputs or to load-bearing self-citation.

full rationale

This is an empirical systems paper rather than a derivation chain. The proposed pipeline trains YOLOv11-obb on annotated drawings and fine-tunes Donut on cropped patches with JSON ground truth; the reported precision, recall, F1, and hallucination rates are measurements of trained models, not quantities constructed from the models' own outputs. There is no equation in which a prediction variable is defined in terms of the target variable, no fitted parameter that is later relabeled as a prediction, and no uniqueness or equivalence theorem imported from prior work. The citation to the authors' earlier OCR-based work [5] is used only to motivate the new hybrid approach, not as evidence that the current results hold, so it is not load-bearing. The most serious limitation is methodological rather than circular: the paper does not state a train/validation/test split, does not define the matching criterion for JSON predictions, and reports metrics without confidence intervals, so the 97.3% F1 claim may describe training-set performance rather than generalization to unseen drawings. That concern belongs to experimental soundness and reproducibility, not to circular reasoning. Accordingly, no circular steps are identified and the score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central empirical claim rests on a private dataset, hand-chosen hyperparameters, and an unstated evaluation split. No new physical or mathematical entity is introduced.

free parameters (3)

Number of manually annotated patches = 1,000
A representative subset chosen from 11,469 detected patches; no analysis of how this size affects results.
Augmentation configuration = sharpness, contrast p=0.5, rotations 0/90/180/270, grayscale p=0.5, inversion p=0.5
Hand-selected transformations with no ablation study reported.
Training schedule = 30 epochs, batch size 1, Adam, single RTX 4090
Fixed hyperparameters for all models; no sensitivity analysis is provided.

assumptions (4)

domain assumption The 1,367 publicly sourced drawings are representative of industrial 2D engineering drawings.
The dataset is in-house and collected from public sources; no explicit validation against industrial drawing distributions is provided in Section II.
domain assumption The reported precision, recall, F1, and hallucination scores are computed on a held-out set representative of unseen drawings.
Section II does not state a train/validation/test split, so the metrics in Section III are not guaranteed to measure generalization.
domain assumption YOLOv11-obb detection errors do not materially affect the reported parsing performance.
Only Donut patch-level metrics are reported; the paper does not evaluate the full pipeline or the propagation of detection errors.
domain assumption The manually annotated subset is evenly distributed across categories, so fine-tuning is not distorted by category imbalance.
Section II claims balance and diversity, but no quantitative class-balance analysis is reported.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Automated Parsing of Engineering Drawings for Structured Information Extraction Using a Fine-tuned Document Understanding Transformer." pith.science (2026). https://pith.science/paper/S3Q4TEO4

@misc{pith2026250501530,
  author       = {Pith},
  title        = {Pith review of: Automated Parsing of Engineering Drawings for Structured Information Extraction Using a Fine-tuned Document Understanding Transformer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S3Q4TEO4}},
  note         = {Machine review of arXiv:2505.01530}
}

read the original abstract

Accurate extraction of key information from 2D engineering drawings is crucial for high-precision manufacturing. Manual extraction is slow and labor-intensive, while traditional Optical Character Recognition (OCR) techniques often struggle with complex layouts and overlapping symbols, resulting in unstructured outputs. To address these challenges, this paper proposes a novel hybrid deep learning framework for structured information extraction by integrating an Oriented Bounding Box (OBB) detection model with a transformer-based document parsing model (Donut). An in-house annotated dataset is used to train YOLOv11 for detecting nine key categories: Geometric Dimensioning and Tolerancing (GD&T), General Tolerances, Measures, Materials, Notes, Radii, Surface Roughness, Threads, and Title Blocks. Detected OBBs are cropped into images and labeled to fine-tune Donut for structured JSON output. Fine-tuning strategies include a single model trained across all categories and category-specific models. Results show that the single model consistently outperforms category-specific ones across all evaluation metrics, achieving higher precision (94.77% for GD&T), recall (100% for most categories), and F1 score (97.3%), while reducing hallucinations (5.23%). The proposed framework improves accuracy, reduces manual effort, and supports scalable deployment in precision-driven industries.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

11 extracted references · 9 canonical work pages

[1]

Extraction/conversion of geometric dimensions and tolerances for machining features,

J. Gao, D. T. Zheng, N. Gindy, and D. Clark, “Extraction/conversion of geometric dimensions and tolerances for machining features,” Int J Adv Manuf Technol, vol. 26, no. 4, pp. 405 –414, Aug. 2005, doi: 10.1007/s00170-004-2195-3

work page doi:10.1007/s00170-004-2195-3 2005
[2]

A datum -based model for practicing geometric dimensioning and tolerancing,

W. Sun and Y. Gao, “A datum -based model for practicing geometric dimensioning and tolerancing,” Journal of Engineering Technology, vol. 35, pp. 38–47, Sep. 2018

work page 2018
[3]

Tolerance Information Extraction for Mechanical Engineering Drawings – A Digital Image Processing and Deep Learning-based Model,

Y. Xu et al. , “Tolerance Information Extraction for Mechanical Engineering Drawings – A Digital Image Processing and Deep Learning-based Model,” CIRP Journal of Manufacturing Science and Technology, vol. 50, pp. 55– 64, Jun. 2024, doi: 10.1016/j.cirpj.2024.01.013

work page doi:10.1016/j.cirpj.2024.01.013 2024
[4]

Integration of Deep Learning for Automatic Recognition of 2D Engineering Drawings,

Y.-H. Lin, Y.-H. Ting, Y.-C. Huang, K.-L. Cheng, and W.- R. Jong, “Integration of Deep Learning for Automatic Recognition of 2D Engineering Drawings,” Machines, vol. 11, no. 8, Art. no. 8, Aug. 2023, doi: 10.3390/machines11080802

work page doi:10.3390/machines11080802 2023
[5]

Fine -Tuning Vision -Language Model for Automated Engineering Drawing Information Extraction,

M. T. Khan, L. Chen, Y. H. Ng, W. Feng, N. Y. J. Tan, and S. K. Moon, “Fine -Tuning Vision -Language Model for Automated Engineering Drawing Information Extraction,” Nov. 06, 2024, arXiv: arXiv:2411.03707. doi: 10.48550/arXiv.2411.03707

arXiv doi:10.48550/arxiv.2411.03707 2024
[6]

AutoCAD Mechanical 2022 Help | About Balloons (AutoCAD Mechanical Toolset) | Autodesk

“AutoCAD Mechanical 2022 Help | About Balloons (AutoCAD Mechanical Toolset) | Autodesk.” Accessed: Sep. 27, 2024. [Online]. Available: https://help.autodesk.com/view/AMECH_PP/2022/ENU/?g uid=GUID-F12F0EA0-0810-42EE-A3FE- 327041AFAEEE

work page 2022
[7]

Data Management and SPC Software,

“Data Management and SPC Software,” MeasurLink. Accessed: Sep. 27, 2024. [Online]. Available: https://measurlink.com/

work page 2024
[8]

You Only Look Once: Unified, Real -Time Object Detection,

J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You Only Look Once: Unified, Real -Time Object Detection,” May 09, 2016, arXiv: arXiv:1506.02640. doi: 10.48550/arXiv.1506.02640

arXiv doi:10.48550/arxiv.1506.02640 2016

Show all 11 references

[9]

tesseract-ocr/tesseract. (Sep. 26, 2024). C++. tesseract-ocr. Accessed: Sep. 27, 2024. [Online]. Available: https://github.com/tesseract-ocr/tesseract

2024
[10]

Leading Image & Video Data Annotation Platform | CVAT

“Leading Image & Video Data Annotation Platform | CVAT.” Accessed: Mar. 22, 2025. [Online]. Available: https://www.cvat.ai

2025
[11]

torchvision.transforms — Torchvision master documentation

“torchvision.transforms — Torchvision master documentation.” Accessed: Mar. 22, 2025. [Online]. Available: https://pytorch.org/vision/0.9/transforms.html

2025

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

Extraction/conversion of geometric dimensions and tolerances for machining features,

J. Gao, D. T. Zheng, N. Gindy, and D. Clark, “Extraction/conversion of geometric dimensions and tolerances for machining features,” Int J Adv Manuf Technol, vol. 26, no. 4, pp. 405 –414, Aug. 2005, doi: 10.1007/s00170-004-2195-3

work page doi:10.1007/s00170-004-2195-3 2005

[2] [2]

A datum -based model for practicing geometric dimensioning and tolerancing,

W. Sun and Y. Gao, “A datum -based model for practicing geometric dimensioning and tolerancing,” Journal of Engineering Technology, vol. 35, pp. 38–47, Sep. 2018

work page 2018

[3] [3]

Tolerance Information Extraction for Mechanical Engineering Drawings – A Digital Image Processing and Deep Learning-based Model,

Y. Xu et al. , “Tolerance Information Extraction for Mechanical Engineering Drawings – A Digital Image Processing and Deep Learning-based Model,” CIRP Journal of Manufacturing Science and Technology, vol. 50, pp. 55– 64, Jun. 2024, doi: 10.1016/j.cirpj.2024.01.013

work page doi:10.1016/j.cirpj.2024.01.013 2024

[4] [4]

Integration of Deep Learning for Automatic Recognition of 2D Engineering Drawings,

Y.-H. Lin, Y.-H. Ting, Y.-C. Huang, K.-L. Cheng, and W.- R. Jong, “Integration of Deep Learning for Automatic Recognition of 2D Engineering Drawings,” Machines, vol. 11, no. 8, Art. no. 8, Aug. 2023, doi: 10.3390/machines11080802

work page doi:10.3390/machines11080802 2023

[5] [5]

Fine -Tuning Vision -Language Model for Automated Engineering Drawing Information Extraction,

M. T. Khan, L. Chen, Y. H. Ng, W. Feng, N. Y. J. Tan, and S. K. Moon, “Fine -Tuning Vision -Language Model for Automated Engineering Drawing Information Extraction,” Nov. 06, 2024, arXiv: arXiv:2411.03707. doi: 10.48550/arXiv.2411.03707

arXiv doi:10.48550/arxiv.2411.03707 2024

[6] [6]

AutoCAD Mechanical 2022 Help | About Balloons (AutoCAD Mechanical Toolset) | Autodesk

“AutoCAD Mechanical 2022 Help | About Balloons (AutoCAD Mechanical Toolset) | Autodesk.” Accessed: Sep. 27, 2024. [Online]. Available: https://help.autodesk.com/view/AMECH_PP/2022/ENU/?g uid=GUID-F12F0EA0-0810-42EE-A3FE- 327041AFAEEE

work page 2022

[7] [7]

Data Management and SPC Software,

“Data Management and SPC Software,” MeasurLink. Accessed: Sep. 27, 2024. [Online]. Available: https://measurlink.com/

work page 2024

[8] [8]

You Only Look Once: Unified, Real -Time Object Detection,

J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You Only Look Once: Unified, Real -Time Object Detection,” May 09, 2016, arXiv: arXiv:1506.02640. doi: 10.48550/arXiv.1506.02640

arXiv doi:10.48550/arxiv.1506.02640 2016