{"id":"2968efc3-9eba-4742-85bc-10e8a525c270","arxiv_id":"2501.11836","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"On an augmented 400-image concrete damage dataset, YOLO-v7 instance segmentation reached 96.1% mAP@0.5 at 40 FPS, outperforming Mask R-CNN at 92.1% mAP@0.5 and 18 FPS.","lead":"This paper compares two off-the-shelf deep learning models, YOLO-v7 and Mask R-CNN, for detecting cracks and spalls in concrete from images. The authors report that YOLO-v7 is both more accurate and faster, and recommend it for real-time monitoring, but the evaluation protocol has several weaknesses that make the exact numbers unreliable.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 96.1% vs 92.1% mAP comparison is not interpretable unless the split occurred before augmentation; the paper does not say it did, and with 400 source images near-duplicate leakage is likely.","rationale":"The paper's central contribution is an empirical model comparison for concrete crack/spall instance segmentation. I read the strongest claim as the abstract's numerical comparison, which is the basis for recommending YOLO-v7 for real-time monitoring. For that claim to hold, the evaluation must be a fair test of generalization: the test images must not be augmented copies of training images. The paper reports augmentation to 10,995 images before presenting the 90/10 split (Table 1), and it never states that the split was done on the 400 pre-augmentation images. Given only 400 unique originals and 1,000 test images, train/test near-duplicate overlap is not a remote possibility but a structural risk. This would make both mAP numbers optimistic and could change the ranking between the two models, since the more flexible model can memorize augmented variants more easily. I checked for alternate weaknesses: the FPS values are reported without hardware details, and Table 1's arithmetic is internally inconsistent (9,000+1,000=10,000 vs 10,995), but these affect reproducibility more than the truth of the comparison. The reader's conditional verdict already identifies the same core assumption and asks for a split-before-augmentation rerun. My stress-test confirms that condition; it does not move the verdict.","tokens_in":8901,"tokens_out":5842,"duration_ms":60196,"concrete_test":"Obtain the actual split metadata/code from the authors or the CConCrack Kaggle dataset; compute perceptual hashes (e.g., pHash/dHash) for all 9,000 training and 1,000 test images and flag any test image whose hash is within a small Hamming distance of a training image. If any such near-duplicate pair exists, the leakage concern is confirmed, and the mAP comparison must be redone on a split of the 400 original images before augmentation. If no near-duplicate pairs exist and the authors confirm the split was pre-augmentation, the accuracy comparison is not invalidated by this concern; a speed benchmark on identical hardware would then be the remaining check.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the quantitative comparison in §3.3–3.4 (YOLO-v7 mAP@0.5 96.1% at 40 FPS vs Mask R-CNN 92.1% at 18 FPS). For this comparison to be meaningful, the test set must measure generalization to concrete damage not seen in training. Section 2.1 states that 400 images were augmented to 10,995 images, and Table 1 gives a 9,000/1,000 split; the paper never states that the split was applied to the original 400 images before augmentation. Because the augmentation pipeline in §2.2 includes flips, ±90° rotations, zoom crops, and mosaic (for YOLO-v7), every original image generates many near-duplicate variants. With only 400 source images and a 1,000-image test partition, it is highly plausible that augmented siblings of the same original appear in both training and test sets. If so, the reported mAP is inflated by near-duplicate retrieval rather than true detection. This directly undermines the accuracy side of the claimed YOLO-v7 advantage. The FPS comparison is also not reproducible as stated (no GPU model, batch size, or measurement protocol), and Table 1's 9,000+1,000=10,000 contradicts the 10,995 total, but the split-before-vs-after-augmentation question is the load-bearing uncertainty.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper benchmarks two instance segmentation models, Mask R-CNN and YOLO-v7, on a combined dataset of 400 concrete crack/spall images augmented to 10,995 images. The models are trained with transfer learning and evaluated with precision, recall, mAP@0.5, and FPS. The central claim is that YOLO-v7 achieves higher mAP@0.5 (96.1%) and higher throughput (40 FPS) than Mask R-CNN (92.1%, 18 FPS), making it preferable for real-time structural health monitoring. The paper also discusses limitations and future directions.","tokens_in":9227,"tokens_out":4373,"duration_ms":41684,"significance":"If the quantitative comparison is valid, the result is a useful practical benchmark for infrastructure inspection: it aligns with the general observation that one-stage detectors such as YOLO-v7 offer better speed-accuracy trade-offs than two-stage Mask R-CNN on similar segmentation tasks. The manuscript's strengths are the use of three publicly available datasets, the comparison of two standard architectures, and the explicit acknowledgment of limitations (Section 3.11). However, the validity cannot currently be assessed due to the unclear data-split point, inconsistent dataset counts, contradictory metric tables, and unreported statistical variability; these issues must be resolved before the comparison can be used to support the recommendation.","major_comments":[{"comment":"The manuscript does not state whether the 90/10 split was performed on the 400 original images before augmentation or on the augmented set. Because the augmentation pipeline generates many near-duplicate variants of each source image, a post-augmentation split can place sibling images of the same crack in both training and test partitions; with 400 source images and a 1,000-image test partition, this is a concrete risk that would inflate mAP by near-duplicate retrieval. Additionally, the reported counts are inconsistent: the text states 10,995 augmented images, while Table 1 sums to 10,000 (9,000 + 1,000). Please specify the split point and reconcile the counts; if the split was post-augmentation, re-run the comparison with a split performed before augmentation.","section":"§2.1–2.2, Table 1"},{"comment":"The mAP comparison for Mask R-CNN is internally inconsistent. Table 9 reports Mask R-CNN mAP@0.5 = 92.1%, but Table 7 gives class-specific AP values of 37.3% (crack) and 79.3% (spall), whose average cannot equal 92.1%; Table 6 also reports a 92.1% 'classification' AP50 alongside a 95.5% bounding-box AP50, without defining how these relate to the mAP used in Table 9. Please clarify which metric is reported and recompute the class-averaged mAP, since the central accuracy comparison rests on this number.","section":"§3.2–3.3, Tables 6, 7 and 9"},{"comment":"All accuracy and speed metrics appear to be single-run measurements with no error bars or number of repeated trials. The 4-point mAP gap between YOLO-v7 and Mask R-CNN may be within run-to-run variance for models trained with stochastic gradient descent; at minimum, report metrics across multiple seeds with means and standard deviations, and state the number of runs.","section":"§3.3 and §3.7, Tables 8 and 11"},{"comment":"The FPS comparison is not reproducible as reported because the manuscript does not state the GPU model, batch size at inference, input resolution after preprocessing (the text says all images resized to 640×640 but does not state whether this was used at test time for both models), or the measurement protocol for FPS. Please provide this information so the speed advantage can be independently verified.","section":"§3.7, Tables 9 and 11"}],"minor_comments":[{"comment":"The title contains a typographical error: 'DATA-D RIVEN' should be 'DATA-DRIVEN'.","section":"Title"},{"comment":"The statement 'Mask R-CNN: Superior precision for crack classification' is not supported by the tables: Table 10 shows lower overall precision for Mask R-CNN (92.0% vs 94.9%), and no crack-specific precision for Mask R-CNN is provided in Table 7.","section":"§3.4, second bullet"},{"comment":"The phrase 'Validation, Test' is ambiguous; it is unclear whether the 10% is a separate test set or a combined validation/test partition, and the paper does not state how the reported test metrics were computed.","section":"§2.1, Table 1"},{"comment":"The word 'Cutting' in Table 2 should be 'Cropping', and it is unclear whether Mask R-CNN received any model-specific augmentation or only the Roboflow augmentation; please clarify for a fair comparison.","section":"§2.2, Tables 2 and 3"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the core comparison is plausible but the evaluation protocol must be fixed. The inconsistent dataset counts and the contradiction between the Mask R-CNN class AP values and the reported mAP are the most serious issues. The paper is an arXiv preprint with a journal-style presentation; the reference list and figure quality would also need attention before formal publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a routine YOLO-v7 vs Mask R-CNN comparison on a small augmented concrete-crack dataset, and the headline numbers are not trustworthy because the train/test split appears to be made after augmentation. With 400 source images expanded to 10,995, the 1,000-image test partition almost certainly contains near-duplicates of training images, so the reported mAP could reflect retrieval of augmented siblings rather than generalization to unseen damage.\n\nCredit where due: the authors assembled a combined dataset from three public sources, labeled it at instance level, and compared two off-the-shelf segmentation models with practical deployment in mind. The qualitative conclusion that YOLO-v7 is faster and likely no worse than Mask R-CNN is plausible and consistent with prior benchmarks. The limitations section is honest about fine-crack resolution and dataset size.\n\nSoft spots, in order of severity. First, the split after augmentation: Table 1 explicitly provides the distribution 'after augmentation,' and the 9,000/1,000 counts do not add to the stated 10,995 total—995 images are unaccounted for. Second, the Mask R-CNN metrics do not cohere: Table 7 reports crack AP of 37.3% and spall AP of 79.3%, whose average is 58.3%, yet the paper claims 92.1% mAP50 for that model. That is a load-bearing inconsistency in the central comparison. Third, FPS numbers are given without hardware, batch size, or a measurement protocol, so the speed comparison is not reproducible. Fourth, single-run metrics without error bars mean the 96.1% vs 92.1% gap could be noise. Code and data are not released; the text mentions a Kaggle location without a link.\n\nThe leakage issue is fixable. If the authors can show the split was applied to the original 400 images before augmentation, the near-duplicate problem goes away, though the count mismatch and metric inconsistency must still be corrected. As submitted, the exact numbers should not be quoted in any secondary source.\n\nRecommendation: this deserves a serious referee, but not because it is a breakthrough. It is a straightforward engineering comparison that could be useful to infrastructure inspection practitioners if the protocol is fixed. I would send it out with a request for major revision: split before augmentation, multiple seeds, consistent per-class and aggregate metrics, hardware details, and released code/data. If the leakage is confirmed, it should be rejected.","headline":"Routine benchmark with a likely split-after-augmentation leak; the specific mAP numbers are not reliable until the protocol and inconsistent metrics are fixed.","tokens_in":9741,"tokens_out":5527,"would_cite":false,"duration_ms":53407,"reading_group":"maybe","serious_thinker":"unclear","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that YOLO-v7 instance segmentation detects cracks and spalls in concrete more accurately and faster than Mask R-CNN, reporting 96.1% mAP@0.5 at 40 FPS versus 92.1% at 18 FPS, and recommends YOLO-v7 for real-time…","keywords":["concrete structures","damage detection","deep learning","instance segmentation","YOLO-v7","Mask R-CNN","crack detection","structural health monitoring"],"falsifier":"A reader could rerun the same training protocol with the 400-image set split into training and test before any augmentation, then augment only the training side. If YOLO-v7's mAP@0.5 falls well below 96.1% or the gap with Mask R-CNN shrinks, the reported superiority is an artifact of data leakage rather than a property of the model.","tokens_in":8711,"feed_emoji":"🏗️","tokens_out":7743,"duration_ms":68245,"temperature":0.7,"pith_summary":"The paper aims to show that a modern one-stage instance-segmentation network can automate the detection of cracks and spalls in concrete surfaces well enough for real-time structural health monitoring. Using 400 annotated images expanded by augmentation to 10,995, it trained two models and reports that YOLO-v7 instance segmentation reaches 96.1% mean average precision at 40 frames per second, while Mask R-CNN reaches 92.1% at 18 frames per second. The authors conclude that YOLO-v7 is the better choice for live, high-throughput inspection and that Mask R-CNN remains an option for offline analysis. If the comparison holds, the practical payoff is automated, low-cost, continuous monitoring of bridges, tunnels, and walls.","feed_headline":"YOLO-v7 beats Mask R-CNN at concrete damage detection","feed_subtitle":"On 10,995 augmented images, it scores 96.1% mAP at 40 FPS while Mask R-CNN scores 92.1% at 18 FPS.","key_machinery":"The carrying object is instance segmentation, a vision task that labels each individual defect with its own pixel mask rather than only classifying whole images; this is what lets the system localize separate cracks and spalls. The comparison is carried by a fixed experimental pipeline: combining three public datasets into 400 images, expanding them to 10,995 via geometric and color augmentations plus model-specific transforms, fine-tuning both models from pre-trained weights, and scoring with mAP@0.5 and FPS. The mAP numbers are the bridge between model choice and deployment recommendation.","core_discovery":"The authors' central discovery is that, on their curated concrete-damage dataset, YOLO-v7 instance segmentation outperforms Mask R-CNN on every headline metric: mAP@0.5 of 96.1% versus 92.1%, precision of 94.9% versus 92.0%, recall of 94.3% versus 62.8%, and inference speed of 40 FPS versus 18 FPS. They interpret this as evidence that one-stage instance segmentation is the right architecture for real-time structural health monitoring, especially for catching spalls (mAP 99.5%) and, to a lesser degree, cracks (mAP 92.7%). The paper also reports qualitative generalization tests on random internet photos and videos, which the authors take as further evidence of robustness.","pith_inferences":["The reported accuracy is likely to be an upper bound on real-world generalization: the 90/10 split was applied after augmentation, so augmented copies of the same original image can appear in both training and test sets; a split-before-augmentation retest would reveal how much of the score comes from near-duplicate recognition.","If the speed-accuracy gap persists on a held-out, pre-augmentation test set, the practical consequence extends beyond concrete inspection: any instance-segmentation task with small annotated datasets and real-time requirements should prefer carefully tuned one-stage detectors over two-stage ones.","A natural testable extension is to add fine-crack classes at higher imaging resolution, because the paper's own limitation discussion predicts YOLO-v7 will lose accuracy on extremely fine cracks as resolution is constrained.","The qualitative internet-photo and video tests suggest some out-of-distribution robustness, but a quantitative evaluation with established crack benchmarks would be needed to claim the model generalizes across cameras and lighting."],"forward_implications":["YOLO-v7 instance segmentation can process concrete inspection video at 40 frames per second, so it could run on drone or wall-climbing camera feeds without waiting for offline batch analysis.","Mask R-CNN's recall of 62.8% on this dataset implies that roughly a third of defects would be missed in a live setting, making it a poorer fit for safety-critical screening despite its 92.1% mAP.","Both models detect spalls more reliably than cracks, so crack recall (88.6% for YOLO-v7) is the bottleneck that future datasets and architectures should target.","Transfer learning from a large pre-trained vision model allows a domain dataset of only 400 source images to reach near-96% detection performance after augmentation.","Deploying YOLO-v7 on edge devices with IoT cameras would enable on-site, real-time damage assessment without sending images to a central server, a direct corollary of the reported speed."],"supporting_citations":[{"why":"Provides 100 of the 400 source images used to build the combined training corpus.","marker":"[1]"},{"why":"Contributes 200 crack-and-spall images, the largest share of the combined dataset.","marker":"[2]"},{"why":"Contributes 100 lower-resolution crack images, broadening the image conditions represented in the dataset.","marker":"[3]"}],"fun_headline_variants":["YOLO-v7 wins speed and accuracy over Mask R-CNN","Deep learning cracks concrete damage detection at 40 FPS","Real-time concrete damage detection: YOLO-v7 leads at 96% mAP","Faster and sharper: YOLO-v7 outclasses Mask R-CNN on concrete","YOLO-v7 trumps Mask R-CNN for structural health monitoring"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 90/10 training/test split of the augmented dataset keeps each original concrete image on one side only; if augmented copies of the same crack appear in both training and test sets, the reported mAP mostly measures recognition of near-duplicates rather than true generalization.","fun_headline_variants_meta":{"raw":{"variants":["YOLO-v7 wins speed and accuracy over Mask R-CNN","Deep learning cracks concrete damage detection at 40 FPS","Real-time concrete damage detection: YOLO-v7 leads at 96% mAP","Faster and sharper: YOLO-v7 outclasses Mask R-CNN on concrete","YOLO-v7 trumps Mask R-CNN for structural health monitoring"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000342,"raw_usage":{"total_tokens":1906,"prompt_tokens":992,"completion_tokens":914,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":608,"completion_tokens_details":{"reasoning_tokens":812}},"tokens_in":608,"tokens_out":914,"duration_ms":8336,"temperature":1.0,"reasoning_tokens":812,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:48:22.350835+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could rerun the same training protocol with the 400-image set split into training and test before any augmentation, then augment only the training side. If YOLO-v7's mAP@0.5 falls well below 96.1% or the gap with Mask R-CNN shrinks, the reported superiority is an artifact of data leakage rather than a property of the model.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides 100 of the 400 source images used to build the combined training corpus."},{"cited_title":"Zhang, C","cited_arxiv_id":null,"evidence_quote":"Contributes 200 crack-and-spall images, the largest share of the combined dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes 100 lower-resolution crack images, broadening the image conditions represented in the dataset."}],"review_version":1}