ScratchSim: A Procedural Synthetic Data Pipeline for Surface Scratch Detection

Arjan Kuijper; Michael Weinmann; Paul Julius K\"uhn; Richard Hoffmann; Saptarshi Neil Sinha; Tiago Kleist

ScratchSim: A Procedural Synthetic Data Pipeline for Surface Scratch Detection

T0 review · 2 major / 6 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper proposes ScratchSim, a procedural synthetic data pipeline for surface scratch detection, and claims that fine-tuning from synthetic weights consistently beats training on real data alone, with mixed training recovering performanc

desk verdict A solid, honestly-reported synthetic-to-real scratch detection benchmark with a genuinely useful procedural pipeline; the headline claim is plausible but rests on small real test sets that need caveats or expansion before publication. read the letter →

arxiv 2607.27065 v3 pith:IXISJGZJ submitted 2026-07-29 cs.CV cs.AI

Paul Julius Kühn , Saptarshi Neil Sinha , Tiago Kleist , Richard Hoffmann , Arjan kuijper , Michael Weinmann This is my paper · ORCID

classification cs.CVcs.AI

keywords syntheticdatagenerationsurfacedefectdetectionproceduralrenderingscratchdomainadaptationfine-tuningedgedeploymentrandomization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to show that a fully synthetic, procedurally generated dataset of surface scratches — complete with automatic annotations — can substitute for most real labeled images when training defect detectors for industrial inspection. The key claim is that fine-tuning a detector pretrained on synthetic data consistently outperforms training on real data only, across both glossy and matte objects and across convolutional and transformer-based detector architectures. A second claim is that mixing synthetic and real images restores performance when real labels are scarce (10–50% of the full real set). If correct, this gives manufacturers a practical recipe for training on-device scratch detectors without collecting and annotating large real datasets.

What carries the argument

The mechanism is a procedural rendering pipeline (ScratchSim) that synthesizes scratch masks as random cubic Bézier curves on a high-resolution canvas, applies them to physically based materials tuned per object (roughness, specularity, normal and color modulation), restricts rendering to viewing angles where scratches are actually visible, randomizes scene, lighting, and camera pose, and automatically produces COCO-format bounding-box annotations via an AOV output node. This pipeline generates 10,000-image synthetic datasets per configuration; the paper then uses those datasets in four training strategies, with the decisive step being pretraining on synthetic images and fine-tuning on the a

What would settle it

Take the same fine-tuned and real-only models and evaluate them on a large, diverse set of real scratched surfaces (hundreds of images spanning multiple backgrounds, lighting conditions, camera angles, and scratch severities); if real-only training matches or beats fine-tuning on that set, the paper's central claim is refuted.

Watch

Extended reading notes

Core claim

On its own evidence, the paper's central discovery is that synthetic scratch data is not a replacement for real data but a powerful initialization and regularizer: fine-tuning a detector from synthetic weights yields the best or tied-best results in every configuration tested, exceeding the real-only baseline even when all real data is available. Mixed synthetic-real training lifts scarce-data performance dramatically — e.g., from mAP50 near 0.06 to over 0.5 at 10% real data for one detector — and at 50% real data already surpasses the 100% real-only baseline. These results hold for two material regimes (glossy and matte) and for both CNN and transformer-based detectors, suggesting the recip

Load-bearing premise

The strongest results are measured on a 16-image real test set for the glossy object and a single static-background test set for the matte object; if those tiny, narrow test sets don't represent the real deployment distribution, the claimed consistent advantage of synthetic fine-tuning may not generalize.

Editorial extensions

If this is right

Manufacturers can train edge-deployable scratch detectors with 10–50% of the real labeled data previously needed, by fine-tuning a synthetically pretrained model.
The fine-tuning advantage persists across object materials (glossy automotive paint vs. matte powder coating) and across convolutional and transformer detectors, so it is a recipe-level effect, not an artifact of one model.
Mixing synthetic and real data at the full 100% real-data setting still improves over real-only training, meaning synthetic data acts as a regularizer even when real data is plentiful.
The automatic-annotation design removes the manual labeling bottleneck for synthetic scratch data, making large-scale defect-dataset creation feasible in industrial settings.
The viewing-angle suppression prevents false-positive annotations at grazing angles, which is likely necessary for the realism of the synthetic-to-real transfer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the fine-tuning-over-real-only advantage holds broadly, the practical default for industrial defect detection could shift to 'synthetic pretraining first' — even for defects like dents or corrosion, which share the same low-annotation-burden logic.
The paper's own ablation suggests that matching background alone doesn't help without a real-data anchor; a testable extension would be to vary synthetic background diversity while holding real data fixed, to find the point of diminishing returns.
Because the headline Ferrari numbers rest on a 16-image test set, an obvious next check is to re-run the same fine-tuning comparison on a larger, multi-background real test set to confirm the advantage isn't an artifact of a narrow evaluation.
The procedural scratch model — random Bézier curves with Gaussian blur and material-dependent modulation — could be stress-tested against real scratch statistics (length, curvature, depth) to see whether realism, not just diversity, is what drives transfer.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know two things about this paper. First, the procedural scratch pipeline is the real contribution: a configurable BlenderProc-based generator with material-specific scratch application, viewing-angle suppression via |v·n|, and automatic AOV-based COCO annotation. That is genuinely new and well described. Second, the empirical claim—synthetic pre-training plus fine-tuning beats real-only training—is supported by internally consistent numbers across three detectors and many training regimes, but the evaluation has a clear weak spot in the size and diversity of the real test sets.

What the paper does well: the methodology is transparent, the ablations are thoughtful (background matching, real-data fractions, color/camera modes), and the Ferrari results come with three-run means and standard deviations. The pipeline details are specific enough to reproduce, and the decision to release the Ferrari data (if honored) is a plus. The related work is honest, and self-citations are used for motivation, not to define the result.

The soft spots are real but not fatal. The 16-image Ferrari test set and the single-background grip test set are small and narrow; the stress-test note is right that the "consistently outperforms" claim is stronger than what 16 images can confirm for deployment generalization. That said, the pattern holds across many conditions—multiple real-data fractions, two architectures, fine-tuning and mixed training—so it is not one lucky comparison. The grip results have no error bars, which is a legitimate criticism given the large differences between some configurations. Also, the grip data will not be released, and the Ferrari data only upon acceptance, so verification currently relies on the authors' word.

The central argument holds up: synthetic data helps, fine-tuning helps most, and the pipeline is a useful tool. This paper deserves a serious referee. I would send it out with a request for the authors to either add error bars to the grip experiments, expand the real test sets, or at minimum temper the generalization claim to match the evaluation scope. Who should read this: anyone working on synthetic data for industrial defect detection, and to a lesser extent sim-to-real transfer generally. I would cite it if I were working in this subfield, and I would bring it to a reading group focused on practical domain adaptation.

Referee Report

2 major / 6 minor

Summary. The paper proposes ScratchSim, a procedural synthetic-data pipeline built on BlenderProc for generating annotated scratch images for industrial defect detection. The pipeline models two object types (a glossy toy Ferrari and a matte industrial grip), with configurable materials, camera modes, environment randomization, and automatic COCO annotations via an AOV mask. The authors evaluate four training strategies (synthetic-only, real-only, mixed, and fine-tuning from synthetic weights) on three lightweight detectors (YOLOX-S, YOLO26-n, LW-DETR-Tiny). On the Ferrari datasets, YOLO26 and LW-DETR are trained with three seeds and reported as mean±std; on the grip datasets, YOLOX-S is trained once per configuration. The main empirical findings are that synthetic-only training underperforms real-only training, but mixing synthetic with real data improves over real-only under scarce-data conditions, and fine-tuning from synthetic weights outperforms the real-only baseline on both objects and all detectors.

Significance. If the empirical claims hold, the pipeline is a practical recipe for reducing real annotated data for edge-deployable defect detectors, with public dataset release and a detailed, reproducible procedure. The paper's strengths include the explicit procedural modeling of materials and scratches, automatic annotation via AOV, multiple synthetic configuration variants, comparisons across CNN and transformer architectures, and three-run repeats with standard deviations for the Ferrari experiments. The cross-architecture and cross-object consistency of the fine-tuning advantage is a plus. However, the evaluation's external validity is limited by small, single-background real test sets, so the significance of the headline claims is currently conditional on those narrow test distributions.

major comments (2)

[§5, §7.1, Table 1] The Ferrari real test set contains only 16 images, all captured in front of a static background (Section 5). Every headline number in Table 1, including the fine-tuning vs. real-only comparisons, is computed on this split. The reported standard deviations are training-seed variability; they do not capture test-set sampling variability, which on 16 images is substantial. The abstract's 'consistently outperforms' claim is therefore established only for a single, narrow test distribution. Please either (a) augment the real test set with more backgrounds/lighting/instances and report confidence intervals, or (b) soften the abstract and conclusion claims to the tested distribution and add an explicit generalizability caveat.
[§7.2, Table 2] The grip experiments report a single AP/AR per configuration with no error bars or run count, unlike the three-run Ferrari experiments. The fine-tuning advantage over the real baseline is modest (e.g., AP 0.319 vs 0.234 for randomcolour_randomcam; 0.282 vs 0.234 for randomcolour_tripod). Without variance estimates, the reader cannot assess significance. In addition, the grip test set is 120 images against a static background under controlled lighting, a second single-background test distribution. Report multiple runs (or justify a single run) and state the deployment-generalization limitation.

minor comments (6)

[Abstract] Typos: 'surface scratched' should be 'surface scratches'; 'Our evaluation show' should be 'Our evaluation shows'; the keywords line is missing a space after 'Keywords'.
[§6] 'GPU's' should be 'GPUs'.
[Table 1 footnote] Specify the confidence threshold used for the reported Ultralytics Precision/Recall values.
[§7.1, 'Effect of random colored background'] The statement 'Once real images are present the effect reverses' is not true for the 10% real infusion (Synth (WB) + Real(10%) is 0.4319 vs Synth + Real(10%) 0.5065 for YOLO26); the reversal appears only at 25% and above. Rephrase to reflect this.
[§7.2, Figure 6] The text reports precise percentages (90–95% vs 60%) derived from the ablation curves, but those curves appear to be based on single runs without error bars. Please state this limitation or add error bars.
[Conclusion] Calling fine-tuning 'the most robust strategy' is somewhat stronger than the data show: in Table 1, fine-tuning and 100% infused are close for LW-DETR on mAP50-95 (e.g., 0.460 vs 0.459). A softer characterization, such as 'best or tied-best on these test sets,' would be more appropriate.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claim is an empirical benchmark evaluated on held-out real test data.

full rationale

This is an empirical systems/benchmark paper, not a derivation, so there is no equation-level reduction to inspect. The central claim—that fine-tuning from synthetic weights outperforms real-only training and that mixed training recovers scarce-real-data performance—is established by training detectors under different regimes and measuring detection metrics on real, manually annotated images. The paper states: 'All models are evaluated on a separate test set of real images that the models have not seen during training,' which grounds the headline numbers in external real data rather than in the synthetic pipeline's own annotations. The synthetic annotations are generated automatically from AOV masks, but the evaluation is not performed on those masks; it is performed on real COCO-annotated images. The self-citations ([22], [23], [24]) appear only in related-work and motivation contexts, e.g., 'Recent work by Kühn et al. [22] explores an end-to-end diffusion-based pipeline,' and are not used to define, justify, or force the paper's results. Empirical hyperparameters, such as the visibility threshold 'determined empirically through visual inspection' and epoch counts 'determined from preliminary experiments showing AP stagnation,' are model-selection choices, not predictions that are then relabeled as findings. The small real test sets and single-background grip evaluation are generalization/external-validity concerns, not circularity. Consequently, no circular step can be exhibited, and the honest finding is a score of 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim is empirical, not derived, so the ledger is mostly about domain assumptions in the sim-to-real setup. The free parameters are hand-chosen pipeline and training settings; none are fitted to the real test set, but several (t, epoch counts) were tuned on validation and could inflate results if applied selectively.

free parameters (5)

visibility threshold t = 0.4 (grip), 0.55 (car)
Chosen empirically through visual inspection (Section 6). Controls which scratch annotations are suppressed, directly affecting synthetic label validity.
training epoch counts = 50–300 depending on regime
Determined from preliminary experiments showing AP stagnation on validation (Section 6, Appendix B), then applied across all compared regimes.
batch sizes = 2–64
Scaled proportionally to dataset size to maintain stable gradients (Section 6); confounds scarce-data comparisons.
scratch count range = N ∈ [17,25]
Draws uniformly at random; range chosen by hand to match observed real scratch density (Section 3.1).
scratch bounding-box scale range = sbox ∈ [canvas/10, canvas/4]
Chosen manually to reflect typical physical scratch size observed on the objects (Section 3.1).

assumptions (4)

domain assumption Procedural Bézier-curve scratches applied via normal/roughness/color modulation are a sufficient proxy for real surface scratches for detector training.
The entire transfer claim depends on this; no physical or perceptual validation of scratch geometry is provided (Section 3.1).
domain assumption The AOV-rendered scratch mask gives pixel-accurate ground truth aligned with visible scratches in the rendered image.
Appendix A assumes the mask exactly encodes which pixels contain scratches after thresholding and connected-component labeling.
domain assumption Real image annotations (manual COCO boxes) are correct and complete.
Used as evaluation ground truth (Section 5); no inter-annotator agreement or review protocol is reported.
domain assumption BlenderProc Cycles rendering with PBR materials produces scene statistics close enough to the webcam capture domain for transfer.
Core sim-to-real premise (Section 3.1); the paper shows evidence it holds for these two objects but does not prove it for new geometries.

how reviews work

0 comments

Cite this review

Pith. "Pith review of ScratchSim: A Procedural Synthetic Data Pipeline for Surface Scratch Detection." pith.science (2026). https://pith.science/paper/IXISJGZJ

@misc{pith2026260727065,
  author       = {Pith},
  title        = {Pith review of: ScratchSim: A Procedural Synthetic Data Pipeline for Surface Scratch Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IXISJGZJ}},
  note         = {Machine review of arXiv:2607.27065}
}

read the original abstract

While automated defect detection such as the detection of surface scratched is an important aspect in industrial quality control, the scarcity of annotated defect data make this task challenging. This paper presents a procedural rendering pipeline that generates large-scale annotated synthetic training data using BlenderProc, with configurable material appearance, camera modes, and domain randomization, producing automatic COCO-format annotations. To show the potential of our approach, we evaluate four training strategies, namely synthetic-only, real-only, mixed, and fine-tuning from synthetic weights, across two objects with different material properties and three lightweight edge-deployable detectors, YOLOX, YOLO26, and LW-DETR. Our evaluation show that fine-tuning from synthetic weights consistently outperforms real-only training, and that mixed training effectively recovers performance under scarce real-data conditions, with findings validated across both convolutional and transformer-based architectures. The proposed approach enables scalable defect detection without the burden of large real annotated datasets, making it practical for on-device industrial inspection. The pipeline scripts for generating synthetic scratches, 3D model, and both the synthetic and real annotated scratch datasets for a glossy toy Ferrari car are publicly available at https://github.com/saptarshineil/ScratchSim.