REVIEW 2 major objections 6 minor 40 references
This paper proposes ScratchSim, a procedural synthetic data pipeline for surface scratch detection, and claims that fine-tuning from synthetic weights consistently beats training on real data alone, with mixed training recovering performanc
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 10:07 UTC pith:IXISJGZJ
load-bearing objection A solid, honestly-reported synthetic-to-real scratch detection benchmark with a genuinely useful procedural pipeline; the headline claim is plausible but rests on small real test sets that need caveats or expansion before publication. the 2 major comments →
ScratchSim: A Procedural Synthetic Data Pipeline for Surface Scratch Detection
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
On its own evidence, the paper's central discovery is that synthetic scratch data is not a replacement for real data but a powerful initialization and regularizer: fine-tuning a detector from synthetic weights yields the best or tied-best results in every configuration tested, exceeding the real-only baseline even when all real data is available. Mixed synthetic-real training lifts scarce-data performance dramatically — e.g., from mAP50 near 0.06 to over 0.5 at 10% real data for one detector — and at 50% real data already surpasses the 100% real-only baseline. These results hold for two material regimes (glossy and matte) and for both CNN and transformer-based detectors, suggesting the recip
What carries the argument
The mechanism is a procedural rendering pipeline (ScratchSim) that synthesizes scratch masks as random cubic Bézier curves on a high-resolution canvas, applies them to physically based materials tuned per object (roughness, specularity, normal and color modulation), restricts rendering to viewing angles where scratches are actually visible, randomizes scene, lighting, and camera pose, and automatically produces COCO-format bounding-box annotations via an AOV output node. This pipeline generates 10,000-image synthetic datasets per configuration; the paper then uses those datasets in four training strategies, with the decisive step being pretraining on synthetic images and fine-tuning on the a
Load-bearing premise
The strongest results are measured on a 16-image real test set for the glossy object and a single static-background test set for the matte object; if those tiny, narrow test sets don't represent the real deployment distribution, the claimed consistent advantage of synthetic fine-tuning may not generalize.
What would settle it
Take the same fine-tuned and real-only models and evaluate them on a large, diverse set of real scratched surfaces (hundreds of images spanning multiple backgrounds, lighting conditions, camera angles, and scratch severities); if real-only training matches or beats fine-tuning on that set, the paper's central claim is refuted.
If this is right
- Manufacturers can train edge-deployable scratch detectors with 10–50% of the real labeled data previously needed, by fine-tuning a synthetically pretrained model.
- The fine-tuning advantage persists across object materials (glossy automotive paint vs. matte powder coating) and across convolutional and transformer detectors, so it is a recipe-level effect, not an artifact of one model.
- Mixing synthetic and real data at the full 100% real-data setting still improves over real-only training, meaning synthetic data acts as a regularizer even when real data is plentiful.
- The automatic-annotation design removes the manual labeling bottleneck for synthetic scratch data, making large-scale defect-dataset creation feasible in industrial settings.
- The viewing-angle suppression prevents false-positive annotations at grazing angles, which is likely necessary for the realism of the synthetic-to-real transfer.
Where Pith is reading between the lines
- If the fine-tuning-over-real-only advantage holds broadly, the practical default for industrial defect detection could shift to 'synthetic pretraining first' — even for defects like dents or corrosion, which share the same low-annotation-burden logic.
- The paper's own ablation suggests that matching background alone doesn't help without a real-data anchor; a testable extension would be to vary synthetic background diversity while holding real data fixed, to find the point of diminishing returns.
- Because the headline Ferrari numbers rest on a 16-image test set, an obvious next check is to re-run the same fine-tuning comparison on a larger, multi-background real test set to confirm the advantage isn't an artifact of a narrow evaluation.
- The procedural scratch model — random Bézier curves with Gaussian blur and material-dependent modulation — could be stress-tested against real scratch statistics (length, curvature, depth) to see whether realism, not just diversity, is what drives transfer.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes ScratchSim, a procedural synthetic-data pipeline built on BlenderProc for generating annotated scratch images for industrial defect detection. The pipeline models two object types (a glossy toy Ferrari and a matte industrial grip), with configurable materials, camera modes, environment randomization, and automatic COCO annotations via an AOV mask. The authors evaluate four training strategies (synthetic-only, real-only, mixed, and fine-tuning from synthetic weights) on three lightweight detectors (YOLOX-S, YOLO26-n, LW-DETR-Tiny). On the Ferrari datasets, YOLO26 and LW-DETR are trained with three seeds and reported as mean±std; on the grip datasets, YOLOX-S is trained once per configuration. The main empirical findings are that synthetic-only training underperforms real-only training, but mixing synthetic with real data improves over real-only under scarce-data conditions, and fine-tuning from synthetic weights outperforms the real-only baseline on both objects and all detectors.
Significance. If the empirical claims hold, the pipeline is a practical recipe for reducing real annotated data for edge-deployable defect detectors, with public dataset release and a detailed, reproducible procedure. The paper's strengths include the explicit procedural modeling of materials and scratches, automatic annotation via AOV, multiple synthetic configuration variants, comparisons across CNN and transformer architectures, and three-run repeats with standard deviations for the Ferrari experiments. The cross-architecture and cross-object consistency of the fine-tuning advantage is a plus. However, the evaluation's external validity is limited by small, single-background real test sets, so the significance of the headline claims is currently conditional on those narrow test distributions.
major comments (2)
- [§5, §7.1, Table 1] The Ferrari real test set contains only 16 images, all captured in front of a static background (Section 5). Every headline number in Table 1, including the fine-tuning vs. real-only comparisons, is computed on this split. The reported standard deviations are training-seed variability; they do not capture test-set sampling variability, which on 16 images is substantial. The abstract's 'consistently outperforms' claim is therefore established only for a single, narrow test distribution. Please either (a) augment the real test set with more backgrounds/lighting/instances and report confidence intervals, or (b) soften the abstract and conclusion claims to the tested distribution and add an explicit generalizability caveat.
- [§7.2, Table 2] The grip experiments report a single AP/AR per configuration with no error bars or run count, unlike the three-run Ferrari experiments. The fine-tuning advantage over the real baseline is modest (e.g., AP 0.319 vs 0.234 for randomcolour_randomcam; 0.282 vs 0.234 for randomcolour_tripod). Without variance estimates, the reader cannot assess significance. In addition, the grip test set is 120 images against a static background under controlled lighting, a second single-background test distribution. Report multiple runs (or justify a single run) and state the deployment-generalization limitation.
minor comments (6)
- [Abstract] Typos: 'surface scratched' should be 'surface scratches'; 'Our evaluation show' should be 'Our evaluation shows'; the keywords line is missing a space after 'Keywords'.
- [§6] 'GPU's' should be 'GPUs'.
- [Table 1 footnote] Specify the confidence threshold used for the reported Ultralytics Precision/Recall values.
- [§7.1, 'Effect of random colored background'] The statement 'Once real images are present the effect reverses' is not true for the 10% real infusion (Synth (WB) + Real(10%) is 0.4319 vs Synth + Real(10%) 0.5065 for YOLO26); the reversal appears only at 25% and above. Rephrase to reflect this.
- [§7.2, Figure 6] The text reports precise percentages (90–95% vs 60%) derived from the ablation curves, but those curves appear to be based on single runs without error bars. Please state this limitation or add error bars.
- [Conclusion] Calling fine-tuning 'the most robust strategy' is somewhat stronger than the data show: in Table 1, fine-tuning and 100% infused are close for LW-DETR on mAP50-95 (e.g., 0.460 vs 0.459). A softer characterization, such as 'best or tied-best on these test sets,' would be more appropriate.
Circularity Check
No significant circularity: the central claim is an empirical benchmark evaluated on held-out real test data.
full rationale
This is an empirical systems/benchmark paper, not a derivation, so there is no equation-level reduction to inspect. The central claim—that fine-tuning from synthetic weights outperforms real-only training and that mixed training recovers scarce-real-data performance—is established by training detectors under different regimes and measuring detection metrics on real, manually annotated images. The paper states: 'All models are evaluated on a separate test set of real images that the models have not seen during training,' which grounds the headline numbers in external real data rather than in the synthetic pipeline's own annotations. The synthetic annotations are generated automatically from AOV masks, but the evaluation is not performed on those masks; it is performed on real COCO-annotated images. The self-citations ([22], [23], [24]) appear only in related-work and motivation contexts, e.g., 'Recent work by Kühn et al. [22] explores an end-to-end diffusion-based pipeline,' and are not used to define, justify, or force the paper's results. Empirical hyperparameters, such as the visibility threshold 'determined empirically through visual inspection' and epoch counts 'determined from preliminary experiments showing AP stagnation,' are model-selection choices, not predictions that are then relabeled as findings. The small real test sets and single-background grip evaluation are generalization/external-validity concerns, not circularity. Consequently, no circular step can be exhibited, and the honest finding is a score of 0.
Axiom & Free-Parameter Ledger
free parameters (5)
- visibility threshold t =
0.4 (grip), 0.55 (car)
- training epoch counts =
50–300 depending on regime
- batch sizes =
2–64
- scratch count range =
N ∈ [17,25]
- scratch bounding-box scale range =
sbox ∈ [canvas/10, canvas/4]
axioms (4)
- domain assumption Procedural Bézier-curve scratches applied via normal/roughness/color modulation are a sufficient proxy for real surface scratches for detector training.
- domain assumption The AOV-rendered scratch mask gives pixel-accurate ground truth aligned with visible scratches in the rendered image.
- domain assumption Real image annotations (manual COCO boxes) are correct and complete.
- domain assumption BlenderProc Cycles rendering with PBR materials produces scene statistics close enough to the webcam capture domain for transfer.
Cite this review
Pith. "Pith review of ScratchSim: A Procedural Synthetic Data Pipeline for Surface Scratch Detection." pith.science (2026). https://pith.science/paper/IXISJGZJ
@misc{pith2026260727065,
author = {Pith},
title = {Pith review of: ScratchSim: A Procedural Synthetic Data Pipeline for Surface Scratch Detection},
year = {2026},
howpublished = {\url{https://pith.science/paper/IXISJGZJ}},
note = {Machine review of arXiv:2607.27065}
}
read the original abstract
While automated defect detection such as the detection of surface scratched is an important aspect in industrial quality control, the scarcity of annotated defect data make this task challenging. This paper presents a procedural rendering pipeline that generates large-scale annotated synthetic training data using BlenderProc, with configurable material appearance, camera modes, and domain randomization, producing automatic COCO-format annotations. To show the potential of our approach, we evaluate four training strategies, namely synthetic-only, real-only, mixed, and fine-tuning from synthetic weights, across two objects with different material properties and three lightweight edge-deployable detectors, YOLOX, YOLO26, and LW-DETR. Our evaluation show that fine-tuning from synthetic weights consistently outperforms real-only training, and that mixed training effectively recovers performance under scarce real-data conditions, with findings validated across both convolutional and transformer-based architectures. The proposed approach enables scalable defect detection without the burden of large real annotated datasets, making it practical for on-device industrial inspection. The pipeline scripts, 3D model, and both synthetic and real annotated scratch datasets for a glossy toy Ferrari car will be made available through the project website upon acceptance.
Figures
Reference graph
Works this paper leans on
-
[1]
A review of metal surface defect detection technologies in industrial applications.IEEE Access, 13:48380–48400, 2025
Qi Qiao, Huiying Hu, Azlin Ahmad, and Ke Wang. A review of metal surface defect detection technologies in industrial applications.IEEE Access, 13:48380–48400, 2025
2025
-
[2]
Newman and Anil K
Timothy S. Newman and Anil K. Jain. A survey of automated visual inspection.Computer Vision and Image Understanding, 61(2):231–262, 1995
1995
-
[3]
An automatic surface defect inspection system for automobiles using machine vision methods.Sensors, 19(3), 2019
Qinbang Zhou, Renwen Chen, Bin Huang, Chuan Liu, Jie Yu, and Xiaoqing Yu. An automatic surface defect inspection system for automobiles using machine vision methods.Sensors, 19(3), 2019
2019
-
[4]
Surface defect detection methods for industrial products: A review.Applied Sciences, 11(16), 2021
Yajun Chen, Yuanyuan Ding, Fan Zhao, Erhu Zhang, Zhangnan Wu, and Linhao Shao. Surface defect detection methods for industrial products: A review.Applied Sciences, 11(16), 2021
2021
-
[5]
Blenderproc, 2019
Maximilian Denninger, Martin Sundermeyer, Dominik Winkelbauer, Youssef Zidan, Dmitry Olefir, Mohamad Elbadrawy, Ahsan Lodhi, and Harinandan Katam. Blenderproc, 2019
2019
-
[6]
Strobl, Matthias Humt, and Rudolph Triebel
Maximilian Denninger, Dominik Winkelbauer, Martin Sundermeyer, Wout Boerdijk, Markus Knauer, Klaus H. Strobl, Matthias Humt, and Rudolph Triebel. Blenderproc2: A procedural pipeline for photorealistic rendering. Journal of Open Source Software, 8(82):4901, 2023
2023
-
[7]
Nvidia isaac sim: Enabling scalable, gpu- accelerated simulation for robotics, 2026
Sicong Gao, Maurice Pagnucco, Tomasz Bednarz, and Yang Song. Nvidia isaac sim: Enabling scalable, gpu- accelerated simulation for robotics, 2026
2026
-
[8]
A survey of synthetic data augmentation methods in machine vision.Machine Intelligence Research, 21(5):831–869, 2024
Alhassan Mumuni, Fuseini Mumuni, and Nana Kobina Gerrar. A survey of synthetic data augmentation methods in machine vision.Machine Intelligence Research, 21(5):831–869, 2024
2024
-
[9]
Synthetic data augmentation for surface defect detection and classification using deep learning.Journal of Intelligent Manufacturing, 33(4):1007– 1020, 2022
Saksham Jain, Gautam Seth, Arpit Paruthi, Umang Soni, and Girish Kumar. Synthetic data augmentation for surface defect detection and classification using deep learning.Journal of Intelligent Manufacturing, 33(4):1007– 1020, 2022
2022
-
[10]
Lawrence Zitnick
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. InComputer Vision – ECCV 2014, 2014
2014
-
[11]
Yolox: Exceeding yolo series in 2021, 2021
Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun. Yolox: Exceeding yolo series in 2021, 2021
2021
-
[12]
Ultralytics yolo26: Unified real-time end-to-end vision models, 2026
Glenn Jocher, Jing Qiu, Mengyu Liu, Shuai Lyu, Fatih Cagatay Akyon, and Muhammet Esat Kalfaoglu. Ultralytics yolo26: Unified real-time end-to-end vision models, 2026
2026
-
[13]
Qiang Chen, Xiangbo Su, Xinyu Zhang, Jian Wang, Jiahui Chen, Yunpeng Shen, Chuchu Han, Ziliang Chen, Weixiang Xu, Fanrong Li, et al. Lw-detr: A transformer replacement to yolo for real-time detection.arXiv preprint arXiv:2406.03459, 2024
Pith/arXiv arXiv 2024
-
[14]
Review of surface-defect detection methods for industrial products based on machine vision.IEEE Access, 13:90668–90697, 2025
Quan Wang, Mengnan Wang, Jiadong Sun, Deji Chen, and Pei Shi. Review of surface-defect detection methods for industrial products based on machine vision.IEEE Access, 13:90668–90697, 2025
2025
-
[15]
Defect detection methods for industrial products using deep learning techniques: A review.Algorithms, 16(2), 2023
Alireza Saberironaghi, Jing Ren, and Moustafa El-Gindy. Defect detection methods for industrial products using deep learning techniques: A review.Algorithms, 16(2), 2023
2023
-
[16]
Real-time human pose recognition in parts from single depth images
Jamie Shotton, Andrew Fitzgibbon, Mat Cook, Toby Sharp, Mark Finocchio, Richard Moore, Alex Kipman, and Andrew Blake. Real-time human pose recognition in parts from single depth images. InCVPR 2011, pages 1297–1304, 2011
2011
-
[17]
Synthetic dataset generation methods for computer vision application
Matej Arlovic, Davor Damjanovic, Franko Hrzic, and Josip Balen. Synthetic dataset generation methods for computer vision application. In2024 International Conference on Smart Systems and Technologies (SST), pages 69–74, 2024
2024
-
[18]
de Melo, Antonio Torralba, Leonidas Guibas, James DiCarlo, Rama Chellappa, and Jessica Hodgins
Celso M. de Melo, Antonio Torralba, Leonidas Guibas, James DiCarlo, Rama Chellappa, and Jessica Hodgins. Next-generation deep learning based on simulators and synthetic data.Trends in Cognitive Sciences, 26(2):174– 187, 2022
2022
-
[19]
Investigating the generation of synthetic data for surface defect detection: A comparative analysis.Procedia CIRP, 130:767–773, 2024
Josefine Monnet, Oliver Petrovic, and Werner Herfs. Investigating the generation of synthetic data for surface defect detection: A comparative analysis.Procedia CIRP, 130:767–773, 2024. 57th CIRP Conference on Manufacturing Systems 2024 (CMS 2024)
2024
-
[20]
High-resolution image synthesis with latent diffusion models
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022
2022
-
[21]
Improving image generation with better captions
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, Wesam Manassra, Prafulla Dhariwal, Casey Chu, and Aditya Ramesh. Improving image generation with better captions. Technical report, OpenAI, 2023. 14 ScratchSim P .J Kühn, S.N. Sinha et al
2023
-
[22]
Synsur: An end-to-end generative pipeline for synthetic industrial surface defect generation and detection, 2026
Paul Julius Kühn, Mika Pommeranz, Arjan Kuijper, and Saptarshi Neil Sinha. Synsur: An end-to-end generative pipeline for synthetic industrial surface defect generation and detection, 2026
2026
-
[23]
Material classification based on training data synthesized using a btf database
Michael Weinmann, Juergen Gall, and Reinhard Klein. Material classification based on training data synthesized using a btf database. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors,Computer Vision – ECCV 2014, pages 156–171, Cham, 2014. Springer International Publishing
2014
-
[24]
S. N. Sinha, P. J. Kühn, M. S. Goschke, and M. Weinmann. 6d strawberry pose estimation: Real-time and edge ai solutions using purely synthetic training data.ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XI-2-2026:751–758, 2026
2026
-
[25]
Domain randomiza- tion for transferring deep neural networks from simulation to the real world
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomiza- tion for transferring deep neural networks from simulation to the real world. In2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 23–30, 2017
2017
-
[26]
Training deep networks with synthetic data: Bridging the reality gap by domain randomization
Jonathan Tremblay, Aayush Prakash, David Acuna, Mark Brophy, Varun Jampani, Cem Anil, Thang To, Eric Cameracci, Shaad Boochoon, and Stan Birchfield. Training deep networks with synthetic data: Bridging the reality gap by domain randomization. In2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1082–10828, 2018
2018
-
[27]
CARLA: An open urban driving simulator
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. InProceedings of the 1st Annual Conference on Robot Learning, pages 1–16, 2017
2017
-
[28]
Virtual kitti 2, 2020
Yohann Cabon, Naila Murray, and Martin Humenberger. Virtual kitti 2, 2020
2020
-
[29]
Mixing real and synthetic data to enhance neural network training – a review of current approaches, 2020
Viktor Seib, Benjamin Lange, and Stefan Wirtz. Mixing real and synthetic data to enhance neural network training – a review of current approaches, 2020
2020
-
[30]
Hybrid dnn training using both synthetic and real construction images to overcome training data shortage.Automation in Construction, 149:104771, 2023
Jinwoo Kim, Daeho Kim, SangHyun Lee, and Seokho Chi. Hybrid dnn training using both synthetic and real construction images to overcome training data shortage.Automation in Construction, 149:104771, 2023
2023
-
[31]
You only look once: Unified, real-time object detection
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 779–788, 2016
2016
-
[32]
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C. Berg. Ssd: Single shot multibox detector. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 21–37, Cham, 2016. Springer International Publishing
2016
-
[33]
Yolo-v1 to yolo-v8, the rise of yolo and its complementary nature toward digital manufac- turing and industrial defect detection.Machines, 11(7), 2023
Muhammad Hussain. Yolo-v1 to yolo-v8, the rise of yolo and its complementary nature toward digital manufac- turing and industrial defect detection.Machines, 11(7), 2023
2023
-
[34]
Rich feature hierarchies for accurate object detection and semantic segmentation
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 580–587, 2014
2014
-
[35]
Fast r-cnn
Ross Girshick. Fast r-cnn. In2015 IEEE International Conference on Computer Vision (ICCV), pages 1440–1448, 2015
2015
-
[36]
Faster r-cnn: Towards real-time object detection with region proposal networks.IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6):1137–1149, 2017
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks.IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6):1137–1149, 2017
2017
-
[37]
2022 ferrari daytona sp3
Ddiaz Design. 2022 ferrari daytona sp3. Sketchfab, 2022. Creative Commons Attribution 4.0 (CC BY 4.0)
2022
-
[38]
Ferrari daytona sp3 2022
vecarz. Ferrari daytona sp3 2022. Sketchfab, 2022. Creative Commons Attribution 4.0 (CC BY 4.0)
2022
-
[39]
Bop challenge 2020 on 6d object localization
Tomáš Hodaˇn, Martin Sundermeyer, Bertram Drost, Yann Labbé, Eric Brachmann, Frank Michel, Carsten Rother, and Jiˇrí Matas. Bop challenge 2020 on 6d object localization. In Adrien Bartoli and Andrea Fusiello, editors, Computer Vision – ECCV 2020 Workshops, pages 577–594, Cham, 2020. Springer International Publishing
2020
-
[40]
ambientcg - free textures, hdris and models, 2025
Lennart Demes. ambientcg - free textures, hdris and models, 2025. 15
2025
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.