{"id":"55d800cb-8e7c-499c-90f3-1c3a6a0dc712","arxiv_id":"2505.01530","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A YOLOv11-OBB detector plus a fine-tuned Donut transformer extracts structured JSON from 2D engineering drawings, with a single combined model beating per-category models in precision and hallucination rate.","lead":"This paper builds a two-stage pipeline that detects annotation regions in engineering drawings and parses them into structured JSON. It compares one shared model against nine category-specific models and reports that the shared model is more precise and hallucinates less.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported 97.3% F1 may be training-set performance: no held-out evaluation split is described, so the single-model superiority claim is not yet supported.","rationale":"The reader's weakest assumption is the absence of a train/validation/test split, and that is exactly the load-bearing concern here. All reported performance figures could be training-set accuracies, which would invalidate the central claim of superior generalization. The missing split is sufficient: without it, the metrics are uninterpretable. I also note the undefined JSON matching criterion and the lack of an end-to-end metric, but these are secondary to the split issue. Since the reader already flagged this and gave a CONDITIONAL verdict, my stress-test does not move the verdict; it reinforces the condition. The proposed concrete test would settle whether the concern lands by recomputing metrics on a genuinely held-out set.","tokens_in":5390,"tokens_out":4493,"duration_ms":46157,"concrete_test":"Ask the authors to provide the exact train/validation/test partition (e.g., patches cropped from drawings held out from YOLO training) and the JSON matching criterion, then recompute Fig. 4 on the held-out test set without augmentation. If the single-model F1 falls below the reported 97.3% or its ordering versus category-specific models changes, the central claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the single fine-tuned Donut model achieves 97.3% F1 and outperforms category-specific models across metrics. The paper never states which images were used for evaluation. Section II says 1,000 patches are augmented to 6,000 image-label pairs and used for fine-tuning; Section III reports per-category precision, recall, F1, and hallucination rate without defining a test set. If those numbers are computed on the training set or the original 1,000 patches used to generate it, they measure fit to the training distribution, not generalization to unseen drawings. No error bars or confidence intervals are given, and the matching criterion for JSON predictions is undefined, so even the raw numbers cannot be interpreted. The entire empirical comparison rests on these metrics, making the missing evaluation protocol the most load-bearing weakness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-stage pipeline for extracting structured information from 2D engineering drawings: YOLOv11 with oriented bounding boxes detects nine annotation categories (GD&T, General Tolerances, Measures, Materials, Notes, Radii, Surface Roughness, Threads, Title Blocks), and a fine-tuned Donut transformer parses the cropped patches into JSON. The authors compare a single Donut model trained on all categories with nine category-specific models, reporting that the single model gives higher precision, recall, F1, and lower hallucination rates. The claimed best result is 97.3% F1 with a 5.23% hallucination rate for GD&T. The paper is an empirical study on an in-house annotated dataset of 1,367 drawings, with 1,000 manually labeled patches augmented to 6,000 for Donut fine-tuning.","tokens_in":5497,"tokens_out":2159,"duration_ms":22379,"significance":"If the reported results are reliable, the contribution is practically useful: a modular, OCR-free framework that outputs structured JSON directly usable in CAD/CAM and quality-assurance workflows, with a semi-automated labeling procedure that reduces annotation burden. The comparison of single versus category-specific fine-tuning is a relevant design question for document-understanding transformers. The paper also provides concrete training details (30 epochs, Adam, batch size 1, RTX 4090) and a reproducible augmentation list. However, the significance is currently limited by the absence of a clear evaluation protocol: no held-out split, no end-to-end assessment including YOLO detection errors, and no defined matching criterion for JSON predictions. These omissions prevent the reader from verifying whether the reported numbers reflect generalization to unseen drawings.","major_comments":[{"comment":"The manuscript never states which images were used to compute the precision, recall, F1, and hallucination values in Fig. 4. Section II says 1,000 patches are augmented to 6,000 image-label pairs 'used for fine-tuning,' and Section III reports per-category metrics without defining a test set. If these numbers come from the training data or the original 1,000 patches, they measure fit to the training distribution, not generalization to unseen drawings. Please specify the exact train/validation/test split, the source of the reported numbers, and confirm that no augmented copies of test patches were present in training.","section":"Section II and Section III (Fig. 4)"},{"comment":"The sentence 'A slight reduction is observed for Title Blocks in the category-specific model, where recall drops from 72.9% to 56.3%' is internally inconsistent with the previous paragraph, which reports a Title Blocks precision of 72.9% for the single model and 55.1% for the category-specific model. The sentence appears to mix precision and recall values, and the claimed recall drop from 72.9% to 56.3% is not supported by any earlier statement. Please clarify the correct per-category precision and recall values for Title Blocks and audit the rest of the results for similar transcription errors.","section":"Section III, Title Blocks recall sentence"},{"comment":"The evaluation in Fig. 4 appears to measure only the Donut parsing stage on cropped patches, not the full hybrid pipeline. Since the framework includes YOLOv11-obb detection, detection misses, false positives, and inaccurate bounding boxes will propagate into the final structured output, but no end-to-end precision, recall, or F1 is reported. Please either provide end-to-end metrics on complete drawings or explicitly state and justify that the reported patch-level metrics are the intended evaluation scope.","section":"Section II and Section III"},{"comment":"The definitions of precision, recall, F1, and hallucination rate do not specify the matching criterion between predicted JSON and ground-truth JSON. Without knowing whether a prediction is considered correct only on exact string match, on semantic equivalence of field values, or on some partial-credit scheme, the reported numbers cannot be reproduced or interpreted. Additionally, no error bars, confidence intervals, or repeated-run statistics are given. Please define the matching criterion precisely and report variance or per-run results.","section":"Section III, metrics definitions"}],"minor_comments":[{"comment":"The abstract claims 'higher precision (94.77% for GD&T), recall (100% for most categories), and F1 score (97.3%)' but does not state whether these refer to the single model or the category-specific models; the results section should make this explicit.","section":"Abstract"},{"comment":"The augmentation list does not mention whether rotations are applied only to axis-aligned patches and how the JSON ground truth is transformed accordingly. Please specify how the ground-truth labels are adjusted under rotation and inversion.","section":"Section II, data augmentation"},{"comment":"The training setup states batch size 1 and 30 epochs but omits learning rate, warmup schedule, and tokenizer settings for Donut. Please add these details for reproducibility.","section":"Section II, hyperparameters"},{"comment":"The discussion attributes low F1 for Materials and General Tolerances to 'limited training samples,' but Fig. 2 shows the category distribution before and after augmentation. Please state the exact per-category training and test counts, since the augmentation appears to multiply all categories equally and would not resolve imbalance.","section":"Section III, Materials and General Tolerances"},{"comment":"The manuscript contains several grammatical and typographical errors (e.g., 'process es' in Section II, 'requ ire s' in Section II, 'outperform s' in Section IV). A careful proofreading pass is needed.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable applied contribution, but its central empirical claim currently rests on an underspecified evaluation protocol. The missing held-out split and the internal inconsistency in the Title Blocks numbers are the main blockers. If the authors can provide a clear evaluation protocol and corrected numbers, the paper could become acceptable. I would also suggest the editor ask for an end-to-end evaluation, or at least a clear statement of why patch-level metrics suffice."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take on arXiv:2505.01530. The paper builds a two-stage pipeline: YOLOv11-obb detects nine annotation categories on 2D engineering drawings, then Donut fine-tuned on cropped patches emits JSON. That combination is sensible and practical for manufacturing document understanding. The comparison of one single Donut model versus nine category-specific models is a legitimate experimental question, and the reported results favor the single model on precision, recall, and hallucination rate.\n\nWhat's genuinely new: the in-house annotated dataset (1,367 drawings, 11,469 patches, 1,000 expert-labeled and augmented to 6,000) and the fine-tuning comparison. The writing is clear and the method is easy to follow. Reporting hallucination rate is a nice touch, rare in this literature. They also honestly note the weak categories (Materials, General Tolerances) where both strategies struggle.\n\nThe soft spot is the evaluation protocol, and it is load-bearing. Nowhere does the paper state which images produced the numbers in Section III. Section II describes augmenting 1,000 patches to 6,000 training pairs; Section III reports per-category precision, recall, F1, and hallucination without defining a held-out test set. If those numbers come from training or augmented training data, the 97.3% F1 and the single-model superiority claim measure fit, not generalization. No error bars, no confidence intervals, and no defined matching criterion for JSON predictions. There is also an internal inconsistency: the Title Block recall sentence says recall drops from 72.9% to 56.3%, but 72.9% was earlier reported as the single model's precision for Title Blocks. And Donut is never cited, which is surprising for the model at the core of the paper.\n\nNone of this kills the idea, but it means the central empirical claim is not independently checkable. The authors need to specify the split, report test-set metrics with variance, add an end-to-end evaluation that includes YOLO detection errors, and compare against at least their own prior OCR-based method as a baseline. Without those, the paper is a recipe, not a result.\n\nWho is this for? Practitioners working on engineering drawing extraction who want a working recipe. For the broader document-understanding literature it is incremental. Still, the dataset and the comparison are worth referee time if the authors can fix the evaluation. I would send it to review with a clear request for the missing details. I would not cite it in its current form, but I'd keep an eye on a revised version.\n\nMaybe bring it to reading group as a case study in how easy it is to omit a test split.","headline":"Useful engineering pipeline, but the headline F1 is not verifiable—no train/test split is described, so treat the single-model superiority claim as unconfirmed.","tokens_in":6032,"tokens_out":2412,"would_cite":false,"duration_ms":23524,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A hybrid YOLOv11-obb and fine-tuned Donut pipeline extracts nine types of annotations from 2D engineering drawings as structured JSON, with a single multi-category model beating category-specific ones on precision, recall, and F1.","keywords":["2D engineering drawings","structured information extraction","document understanding transformer","Donut","YOLOv11","oriented bounding box","fine-tuning","hallucination rate"],"falsifier":"Take a held-out set of 2D engineering drawings never used for training or augmentation, run the full YOLOv11-obb plus Donut pipeline, and compute per-category precision, recall, F1, and hallucination rate on that set. If the F1 drops substantially below 97.3% or the single model stops beating the category-specific models, the central generalization claim is refuted; Materials and General Tolerances are the most likely categories to show this.","tokens_in":1357,"feed_emoji":"📐","tokens_out":1497,"duration_ms":62819,"temperature":0.7,"pith_summary":"The paper claims that pairing oriented object detection with an OCR-free document-parsing transformer can turn cluttered 2D engineering drawings into structured JSON annotations with near-perfect recall and low hallucination. Specifically, a single Donut model trained across all nine annotation categories (GD&T, general tolerances, measures, materials, notes, radii, surface roughness, threads, title blocks) outperforms nine separate category-specific models on precision, F1, and hallucination rate, reaching 97.3% F1 and a 5.23% hallucination rate for GD&T. The design matters because the detection stage first isolates oriented, overlapping symbols that generic OCR handles poorly, and the JSON output is directly usable by downstream manufacturing software. The paper's recommended configuration is the single model for broad, high-recall extraction.","feed_headline":"One model parses engineering drawings into JSON at 97.3% F1","feed_subtitle":"YOLOv11-obb crops annotations, Donut reads them, and the single model beats nine specialist models.","key_machinery":"The load-bearing mechanism is the two-stage pipeline. YOLOv11-obb first predicts oriented bounding boxes for nine annotation categories; each detected box is cropped into a small patch, converting a densely annotated drawing into simple single-annotation images. A fine-tuned Donut model—a transformer that maps document images directly to structured token sequences without OCR—then turns each patch into JSON. Data augmentation (sharpness, contrast, rotation, grayscale, inversion) expands 1,000 expert-labeled patches into 6,000 image-label pairs, and the single-model versus category-specific comparison is the paper's central experimental variable.","core_discovery":"The central claim, stated on the paper's own terms, is that a single fine-tuned Donut model trained across all nine annotation categories outperforms nine separate category-specific models on structured extraction from 2D engineering drawings. Across precision, recall, F1, and hallucination rate, the single model is equal or better on every category, with reported values of 94.77% precision for GD&T, 100% recall on most categories, and 97.3% overall F1, while cutting hallucination rates (5.23% for GD&T). The mechanism offered is exposure to the full annotation space, which gives the model broader contextual patterns and fewer false positives.","pith_inferences":["If the reported metrics come from the same data used for training or augmentation—the paper does not state a train/validation/test split—the 97.3% F1 is likely optimistic, and a held-out evaluation would be the decisive test.","The same 'crop oriented regions, then parse with an OCR-free transformer' template should transfer to other technical documents, such as electrical schematics or maps, where annotations are rotated and overlapping.","As more categories are added, the single-model advantage may erode unless class imbalance is addressed, since the categories that fail here are precisely the low-sample ones.","A human-in-the-loop verification step focused on high-hallucination categories would be a cheap way to make the pipeline production-safe."],"forward_implications":["A single multi-category model is the recommended deployment choice, since it beats category-specific models on every reported metric while keeping recall near 100% on most categories.","The pipeline's JSON output can feed directly into CAD/CAM, quality-assurance, and process-planning tools without additional post-processing.","The semi-automated labeling workflow—detect, crop, label a subset, augment—substantially reduces annotation effort and can be reused to add new drawing categories.","High recall on most categories means few annotations are missed, which is critical in precision manufacturing where a missed tolerance can scrap a part.","Materials and general tolerances remain weak spots, with hallucination rates far above the other categories, so those outputs need separate attention before industrial deployment."],"supporting_citations":[{"why":"Prior deep-learning recognition of 2D engineering drawings that the proposed pipeline builds on.","marker":"[4]"},{"why":"The fine-tuned OCR-based method on the same task that the hybrid pipeline is designed to outperform.","marker":"[5]"},{"why":"The YOLO detection family from which YOLOv11-obb is taken.","marker":"[8]"},{"why":"Generic OCR tools cited as struggling on complex layouts, forming the contrast class for the OCR-free Donut approach.","marker":"[9]"},{"why":"The annotation tool used to create the nine-category dataset.","marker":"[10]"},{"why":"The augmentation transforms that expand 1,000 labeled patches to 6,000 training pairs.","marker":"[11]"}],"fun_headline_variants":["Single model outdoes nine specialists on drawing parsing","One model to read all drawings: 97.3% F1","No more per-category models: single Donut wins","Drawing parsing: one model, fewer hallucinations","97.3% F1: one model beats category-specific ones"],"cache_read_input_tokens":8320,"weakest_assumption_plain":"The central claim assumes that the reported metrics measure performance on drawings the models have not been trained on or augmented from, but the paper gives no train/validation/test split.","fun_headline_variants_meta":{"raw":{"variants":["Single model outdoes nine specialists on drawing parsing","One model to read all drawings: 97.3% F1","No more per-category models: single Donut wins","Drawing parsing: one model, fewer hallucinations","97.3% F1: one model beats category-specific ones"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000252,"raw_usage":{"total_tokens":1548,"prompt_tokens":922,"completion_tokens":626,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":545}},"tokens_in":538,"tokens_out":626,"duration_ms":6339,"temperature":1.0,"reasoning_tokens":545,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:16:25.768053+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set of 2D engineering drawings never used for training or augmentation, run the full YOLOv11-obb plus Donut pipeline, and compute per-category precision, recall, F1, and hallucination rate on that set. If the F1 drops substantially below 97.3% or the single model stops beating the category-specific models, the central generalization claim is refuted; Materials and General Tolerances are the most likely categories to show this.","supporting_citations":[{"cited_title":"Integration of Deep Learning for Automatic Recognition of 2D Engineering Drawings,","cited_arxiv_id":null,"evidence_quote":"Prior deep-learning recognition of 2D engineering drawings that the proposed pipeline builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Generic OCR tools cited as struggling on complex layouts, forming the contrast class for the OCR-free Donut approach."},{"cited_title":"Leading Image & Video Data Annotation Platform | CVAT","cited_arxiv_id":null,"evidence_quote":"The annotation tool used to create the nine-category dataset."},{"cited_title":"torchvision.transforms — Torchvision master documentation","cited_arxiv_id":null,"evidence_quote":"The augmentation transforms that expand 1,000 labeled patches to 6,000 training pairs."}],"review_version":1}