Pith. sign in

REVIEW 3 major objections 5 minor 26 references

SteelBlastQC: Shot-blasted Steel Surface Dataset with Interpretable Detection of Surface Defects

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A new expert-labeled dataset lets off-the-shelf computer-vision models classify shot-blasted steel surfaces as paint-ready or not with about 95% accuracy.

desk verdict Useful new dataset for shot-blasted steel QC, but the crop-level train/test split likely leaks surface identity, so the 95% accuracy claim is not yet credible. read the letter →

arxiv 2504.20510 v1 pith:O3LRKIX3 submitted 2025-04-29 cs.CV cs.NE

Irina Ruzavina , Lisa Sophie Theis , Jesse Lemeer , Rutger de Groen , Leo Ebeling , Andrej Hulak , Jouaria Ali , Guangzhi Tang
show 1 more author
Rico Mockel
This is my paper · ORCID
classification cs.CVcs.NE
keywords qualitycontrolshotblastingsteelsurfacedatasetdefectdetectioncomputervisioninterpretablemachinelearningconvolutionaltransformerautoencoder
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SteelBlastQC is a new, openly released dataset of 1,654 expert-labeled RGB images of shot-blasted steel surfaces, captured in an industrial setting and divided into 'ready for paint' and 'needs shot-blasting' classes. The paper's central claim is that this dataset is realistic and usable enough that standard supervised classifiers can judge paint-readiness at about 95% test accuracy, with a compact convolutional transformer and a support vector machine on ResNet-50 features performing almost identically. It also claims that the same models can be made interpretable through heatmaps, so a quality-control operator can see which surface regions drove the decision. This matters because shot-blasting inspection is currently manual, hazardous work, and a dependable automated check would improve safety, speed, and consistency in manufacturing.

What carries the argument

The central object is the SteelBlastQC dataset itself: 1,654 RGB images of 512×512 pixels, produced by cutting each centered 2048×2048 capture into 16 samples, with 888 images labeled 'ready for paint' and 766 labeled 'needs shot-blasting' by industrial experts against the SA 2.5 surface-cleanliness standard. The machine-learning machinery has three arms: a Compact Convolutional Transformer, whose convolutional tokenizer and sequence pooling combine local texture with global context; a linear SVM trained on 2048-dimensional global-average-pooled features from ResNet-50; and a convolutional autoencoder whose reconstruction error is compared with a statistical threshold to flag defects. Interpretability is carried by heatmaps: gradient-based attention from the transformer's final convolutional layer, feature-weight heatmaps for the SVM, and reconstruction-error maps for the autoencoder.

What would settle it

Run the identical training protocols on a split where every 512×512 patch from one 2048×2048 source image is assigned to the same side; if CCT or SVM accuracy drops by several points, the original 95% was partly due to patch-level leakage.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a purpose-built industrial surface dataset makes automated paint-readiness assessment straightforward: the medium Compact Convolutional Transformer reaches 95.0% test accuracy with F1-score 0.95, and an SVM trained on ResNet-50 features reaches 94.5% accuracy with the same F1-score, while a fully unsupervised convolutional autoencoder reaches 67.6% accuracy with 76.8% recall. The authors present this as validation that the dataset, rather than any specially engineered architecture, is the main contribution, and that interpretability does not cost accuracy: gradient-based and feature-weight heatmaps highlight welding lines, scratches, corrosion, and discoloration in line with expert judgment. The CAE baseline is described as a viable label-free option that is easily confused by shadows and by textures resembling the target surface.

Load-bearing premise

The 95 percent accuracy figure assumes the train/test split keeps all 16 image patches from the same original photograph in one side; if patches from the same surface appear on both sides, the score is inflated.

Editorial extensions

If this is right

  • Automated inspection at roughly 95% accuracy is within reach using standard supervised models, so shot-blasting quality control can shift from continuous human visual assessment to a vision-system check with human review of flagged regions.
  • The compact CCT configuration, about 1 million parameters, matches a 73-million-parameter model, so the practical deployment cost can be low enough for in-line or edge use.
  • Interpretability heatmaps align with known defect types such as welding lines, scratches, corrosion, and discoloration, giving operators a visual basis to trust or override a machine decision.
  • The unsupervised autoencoder path, at 67.6% accuracy with high recall, can serve as a label-free screening stage that catches most defective surfaces while generating many false alarms.
  • Releasing the dataset makes paint-readiness classification a reproducible benchmark for future defect-detection and interpretability methods on real industrial surfaces.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A stricter surface-level evaluation, where all 16 crops from one source image stay in the same split, would show how much of the reported 95% reflects generalization to new surface areas rather than similarity among crops from the same area.
  • The heatmaps could be used as weak supervision to train a pixel-level defect segmenter without costly manual annotation, since the paper shows activations localized to welding lines, scratches, corrosion, and discoloration.
  • The binary 'ready for paint' label could be extended to a continuous roughness estimate tied to the SA 2.5 standard, making the dataset useful for process control rather than only pass/fail inspection.
  • A self-supervised pretraining stage on the unlabeled steel-surface images might lift the autoencoder baseline toward the supervised methods, because the current CAE uses only half the data and no labels.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper introduces SteelBlastQC, a publicly released dataset of 1654 RGB images (512×512) of shot-blasted steel surfaces, labeled by domain experts as either 'ready for paint' or 'needs shot-blasting'. The authors evaluate three baselines: a Compact Convolutional Transformer (CCT), an SVM using ResNet-50 features, and a Convolutional Autoencoder (CAE) with an SPC threshold. They report test accuracies of 95% for CCT, 94.5% for SVM, and 67.6% for CAE, and provide interpretability heatmaps for each method. The paper claims that the supervised methods achieve the stated accuracy on a held-out test set, and that the dataset and code are openly available.

Significance. The dataset addresses a real industrial quality-control task with a new, expert-labeled image collection, and the open release of data and code is a welcome contribution. The comparative benchmarking and interpretability visualizations provide a useful starting point for future work. However, the central empirical claim of 95% accuracy is not yet credible because the evaluation protocol appears to leak information from the same source surfaces into both training and test sets, and because model configurations are selected using test accuracy. If the authors can validate the performance under a properly isolated split, the paper would be a solid resource for the community.

major comments (3)
  1. [III-C (Dataset Preprocessing)] The data split described in Section III-C is performed at the crop level: each 2048×2048 source image is cut into 16 samples of 512×512 and then an 80/20 random split is applied to the resulting 1654 crops, with no grouping by source image. Because all crops from a single source image are physically adjacent views of the same surface under the same controlled lighting, a random crop-level split almost certainly places highly correlated images in both training and test sets. The reported 95% accuracy is therefore not a valid estimate of performance on unseen steel surfaces. The authors must re-run the evaluation with a grouped split, holding out all 16 crops of a source image together, and report the resulting accuracy and metrics.
  2. [V-B2 and V-C2 (Model selection)] The model configuration and training duration are chosen on the basis of test-set accuracy. In Section V-B2, the three CCT configurations are compared and the medium model is selected because it 'performed the best'; in Section V-C2, the CAE is evaluated after different training epochs and the 20-epoch checkpoint is chosen because it 'yielded the best result.' This is test-set model selection, which inflates the reported accuracies. The authors should use a separate validation set for such choices and report test performance for the final selected configuration only, or at minimum explicitly acknowledge that the reported numbers include selection bias.
  3. [V (Table I)] The results in Table I are single point estimates without error bars, repeated runs, or cross-validation. Because the dataset consists of about 103 source images, a single split has high variance, and this is compounded by the leakage and selection issues above. Reporting confidence intervals, grouped cross-validation folds, or per-source performance would allow the reader to assess whether the 95% figure is stable.
minor comments (5)
  1. [Abstract and Table I] The abstract and Section V-D state that CCT and SVM both achieve 95% accuracy, while Table I reports 0.950 for CCT and 0.945 for SVM; the text should be consistent about whether SVM's accuracy rounds to 95%.
  2. [III-B (Dataset Collection)] The dataset is collected from a single steel yacht hull. The authors should state this as a limitation of the dataset's generalizability and temper the claim in the Introduction that the dataset 'enhances generalizability.'
  3. [III-C (Dataset Preprocessing)] The phrase 'fair performance assessment on unseen data out-of-the-box' is misleading given the crop-level split; this should be revised to describe the actual split procedure.
  4. [V-C2 (CAE)] The quantities µd and σd in Eq. (1) are not defined precisely; the authors should specify over which set of latent-space distances these statistics are computed (e.g., the training set of 'ready for paint' images).
  5. [IV-B (CCT)] The implementation of Grad-CAM focusing only on the last convolutional layer of the model's tokenizer is nonstandard; a brief justification or reference for this choice would help.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an empirical dataset benchmark with independent supervised accuracy measurements, not a derivation that reduces to its own inputs.

full rationale

SteelBlastQC is an empirical contribution: it introduces a labeled dataset and reports benchmark accuracies for CCT, SVM with ResNet-50 features, and a CAE. The central claims are external measurements on an 80/20 test split, not quantities derived from their own definitions. Equation (1) defines the CAE decision threshold from latent-space statistics with a hand-set constant C=0.01, but the resulting 67% accuracy, and the 95% accuracies of CCT and SVM, are then evaluated on held-out data rather than being equal to any fitted parameter or training target. No load-bearing step is justified only by a self-citation; all references are to standard external methods and datasets. The CAE epoch count and threshold constant are tuned choices, and the crop-level split could raise generalization concerns, but these are empirical-validity issues rather than instances of circular reasoning. No equation in the paper reduces by construction to its inputs, and no fitted parameter is renamed as a prediction. The paper's results therefore stand as independent empirical findings, and the appropriate circularity score is 0.

Assumptions & free parameters 2 free parameters · 2 assumptions · 0 invented entities

The central claims rest on two domain assumptions: the expert/manufacturer labels are correct, and the crop-level split creates independent train and test samples. The latter is not established and could materially inflate the reported accuracy. One fitted constant (C in the CAE threshold) and one selected hyperparameter (epoch count) shape the CAE results.

free parameters (2)
  • CAE decision threshold constant C = 0.01
    Introduced in Eq. (1) as T_d = mu_d + C*sigma_d. The text states C is set to a low value (0.01) in order to prioritize recall, so the reported CAE recall (0.768) depends directly on this hand-picked value.
  • CAE training epoch count = 20 epochs
    Weights were saved every step and the best epoch was selected afterwards based on results; the reported 67.6% accuracy and 0.68 F1 are for the epoch chosen on the data, making the metric a product of selection rather than a fixed protocol.
assumptions (2)
  • domain assumption Ground-truth labels ('ready for paint' vs 'needs shot-blasting') from the collaborating manufacturer and experts are correct and consistent with the ISO SA 2.5 cleanliness standard.
    Section III-A declares the ideal surface as SA 2.5; labels are provided by industrial shot-blasting experts, but no inter-annotator agreement or independent verification is reported.
  • domain assumption The 2048x2048 source images cropped to 512x512 samples are independent between train and test after an 80/20 random split.
    Section III-C describes splitting the resulting samples, not grouping by parent image; if crops from the same surface area appear in both splits, the accuracy estimate is inflated. The paper gives no evidence of surface-level independence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SteelBlastQC: Shot-blasted Steel Surface Dataset with Interpretable Detection of Surface Defects." pith.science (2026). https://pith.science/paper/O3LRKIX3

@misc{pith2026250420510,
  author       = {Pith},
  title        = {Pith review of: SteelBlastQC: Shot-blasted Steel Surface Dataset with Interpretable Detection of Surface Defects},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O3LRKIX3}},
  note         = {Machine review of arXiv:2504.20510}
}
read the original abstract

Automating the quality control of shot-blasted steel surfaces is crucial for improving manufacturing efficiency and consistency. This study presents a dataset of 1654 labeled RGB images (512x512) of steel surfaces, classified as either "ready for paint" or "needs shot-blasting." The dataset captures real-world surface defects, including discoloration, welding lines, scratches and corrosion, making it well-suited for training computer vision models. Additionally, three classification approaches were evaluated: Compact Convolutional Transformers (CCT), Support Vector Machines (SVM) with ResNet-50 feature extraction, and a Convolutional Autoencoder (CAE). The supervised methods (CCT and SVM) achieve 95% classification accuracy on the test set, with CCT leveraging transformer-based attention mechanisms and SVM offering a computationally efficient alternative. The CAE approach, while less effective, establishes a baseline for unsupervised quality control. We present interpretable decision-making by all three neural networks, allowing industry users to visually pinpoint problematic regions and understand the model's rationale. By releasing the dataset and baseline codes, this work aims to support further research in defect detection, advance the development of interpretable computer vision models for quality control, and encourage the adoption of automated inspection systems in industrial applications.

Figures

Figures reproduced from arXiv: 2504.20510 by the authors.

Figure 1
Figure 1. Example of data collected: passing quality control (left) and failing [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Our proposed interpretable surface defect detection pipeline using supervised and unsupervised deep learning approaches [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Interpretation heatmap examples of truly and falsely classified images [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Comparing interpretation heatmap of SVM using RGB (left) and [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Interpretation heatmap examples of truly and falsely classified images [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: CCT model heatmap on a defective surface [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: Original image (top), reconstructed image (middle), error heatmap [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Interpretation heatmap examples of truly and falsely classified images [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]
Figure 9
Figure 9. Figure 9: Heatmaps of defective surfaces for the CCT small model. [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: Heatmaps of defective surfaces for the SVM with ResNet50 Feature Extraction method. [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 17 canonical work pages

  1. [1]

    Laser modification of metal surfaces,

    D. A. Scott, “Laser modification of metal surfaces,” Optics and Lasers in Engineering , vol. 18, no. 1, pp. 1–13, 1993. [Online]. Available: https://www.sciencedirect.com/science/article/pii/0143816693900023

    work page arXiv 1993

  2. [2]

    Abrasive blasting systems,

    D. Hansel, “Abrasive blasting systems,” Metal Finishing , vol. 98, no. 7, pp. 23–37, 2000. [Online]. Available: https://www.shotpeener. com/library/pdf/1997002.pdf

    work page arXiv 2000

  3. [3]

    Optimal model-based control for automated robotized abrasive blasting system,

    V . B. Nguyen, “Optimal model-based control for automated robotized abrasive blasting system,” Journal of Manufacturing Processes , vol. 109, pp. 1–15, 2024. [Online]. Available: https://www.sciencedirect. com/science/article/pii/S1526612523010848

    work page 2024

  4. [4]

    Shot blasting goes high tech: Case studies document how new systems reduced manufacturing costs for end users,

    E. Holzknecht, “Shot blasting goes high tech: Case studies document how new systems reduced manufacturing costs for end users,” Metal Finishing, vol. 103, no. 7-8, pp. 25–27, 2005. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S002605760580460X

    work page 2005

  5. [5]

    Deep learning for automatic vision-based recognition of industrial surface defects: a survey,

    M. Prunella, R. M. Scardigno, D. Buongiorno, A. Brunetti, N. Longo, R. Carli, M. Dotoli, and V . Bevilacqua, “Deep learning for automatic vision-based recognition of industrial surface defects: a survey,” IEEE Access, vol. 11, pp. 43 370–43 423, 2023

    work page 2023

  6. [6]

    Interpretable deep learning: Interpretation, interpretability, trustworthi- ness, and beyond,

    X. Li, H. Xiong, X. Li, X. Wu, X. Zhang, J. Liu, J. Bian, and D. Dou, “Interpretable deep learning: Interpretation, interpretability, trustworthi- ness, and beyond,” Knowledge and Information Systems , vol. 64, no. 12, pp. 3197–3234, 2022

    work page 2022

  7. [7]

    Escaping the big data paradigm with compact transformers,

    A. Hassani, S. Walton, N. Shah, A. Abuduweili, J. Li, and H. Shi, “Escaping the big data paradigm with compact transformers,” 2022, arXiv preprint arXiv:2104.05704. [Online]. Available: https: //arxiv.org/abs/2104.05704

    arXiv 2022

  8. [8]

    Vapnik, The Nature of Statistical Learning Theory

    V . Vapnik, The Nature of Statistical Learning Theory . Springer, 1995

    work page 1995

Show all 26 references
  1. [9]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2016, pp. 770–778. [Online]. Available: https://arxiv.org/abs/1512.03385

    2016 arXiv

  2. [10]

    A better autoencoder for image: Convolutional autoencoder,

    Y . Zhang, “A better autoencoder for image: Convolutional autoencoder,”

  3. [11]

    A survey of vision-based methods for surface defects’ detection and classification in steel products,

    A. A. M. S. Ibrahim and J.-R. Tapamo, “A survey of vision-based methods for surface defects’ detection and classification in steel products,” Informatics, vol. 11, no. 2, p. Art. no. 25, 2024. [Online]. Available: https://www.mdpi.com/2227-9709/11/2/25

    2024

  4. [12]

    Neu metal surface defects database,

    J. Wang, Y . Wu, Z. Luo, and D. Chen, “Neu metal surface defects database,” 2016. [Online]. Available: https://github.com/neu-multi-robot/metal-surface-defects

    2016

  5. [13]

    Faster metallic surface defect detection using deep learning with channel shuffling,

    M. Y . Siddiqui and H. Ahn, “Faster metallic surface defect detection using deep learning with channel shuffling,” 2024, arXiv preprint arXiv:2406.14582. [Online]. Available: https://arxiv.org/abs/2406.14582

    2024 arXiv

  6. [14]

    Texture-ad: An anomaly detection dataset and benchmark for real algorithm development,

    T. Lei, B. Wang, S. Chen, S. Cao, and N. Zou, “Texture-ad: An anomaly detection dataset and benchmark for real algorithm development,” 2024, arXiv preprint arXiv:2409.06367. [Online]. Available: https://arxiv.org/abs/2409.06367

    2024 arXiv

  7. [15]

    Classification of surface defects on steel strip images using a modified alexnet model combined with support vector machine,

    A. Benabdelkader, A. Boukharouba, F. Bouchelaghem, and H. Bouchelaghem, “Classification of surface defects on steel strip images using a modified alexnet model combined with support vector machine,” Journal of Failure Analysis and Prevention , vol. 22, no. 2, pp. 478–490, 2022

    2022

  8. [16]

    Preparation of steel substrates before application of paints and related products — visual assessment of surface cleanliness,

    “Preparation of steel substrates before application of paints and related products — visual assessment of surface cleanliness,” ISO 8501-1:2007, 2007, international Organization for Standardization. [Online]. Available: https://www.iso.org/standard/11971.html

    2007

  9. [17]

    Grad-cam: Visual explanations from deep networks via gradient-based localization,

    R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” in International Conference on Computer Vision (ICCV) , 2017, pp. 618–626. [Online]. Available: https://arxiv.org/abs...

    2017 arXiv

  10. [18]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” 2023, arXiv preprint arXiv:1706.03762. [Online]. Available: https://arxiv.org/abs/1706.03762

    2023 arXiv

  11. [19]

    Grad-cam: Visual explanations from deep networks via gradient-based localization,

    R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” International Journal of Computer Vision , vol. 128, no. 2, pp. 336–359, 2019

    2019

  12. [20]

    Autoencoder-based anomaly detection for surface defect inspection,

    D.-M. Tsai and P.-H. Jen, “Autoencoder-based anomaly detection for surface defect inspection,” Advanced Engineering Informatics , vol. 48, p. 101272, 2021. [Online]. Available: https://www.sciencedirect.com/ science/article/pii/S1474034621000276

    2021

  13. [21]

    Vision transformer - pytorch,

    lucidrains, “Vision transformer - pytorch,” 2023. [Online]. Available: https://github.com/lucidrains/vit-pytorch/

    2023

  14. [22]

    Swin transformer: Hierarchical vision transformer using shifted windows,

    Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE/CVF international conference on computer vision , 2021, pp. 10 012–10 022

    2021

  15. [23]

    Attentional feature pyramid network for small object detection,

    K. Min, G.-H. Lee, and S.-W. Lee, “Attentional feature pyramid network for small object detection,” Neural Networks , vol. 155, pp. 439–450, 2022

    2022

  16. [24]

    A simple framework for contrastive learning of visual representations,

    T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in International conference on machine learning . PMLR, 2020, pp. 1597–1607

    2020

  17. [25]

    Momentum contrast for unsupervised visual representation learning,

    K. He, H. Fan, Y . Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2020, pp. 9729–9738. APPENDIX A HEATMAPS (FIGURE 9 AND FIGURE 10) We prov...

    2020

  18. [2018]

    Available: https://api.semanticscholar.org/CorpusID: 209442203

    [Online]. Available: https://api.semanticscholar.org/CorpusID: 209442203

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.