{"id":"b51ddc1a-25c3-4cdf-9609-29b1c6e2c381","arxiv_id":"2504.20510","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A new open dataset for shot-blasted steel surface readiness is presented, with supervised baselines reaching about 95% accuracy and an unsupervised baseline at 67%.","lead":"This paper releases a new dataset of 1,654 labeled images of shot-blasted steel surfaces and benchmarks three computer vision classifiers on it. The two supervised methods reach about 95% test accuracy, while the unsupervised autoencoder reaches 67%, and heatmaps are provided to show what the models look at.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Crop-level 80/20 split without grouping by source image likely inflates the reported 95% accuracy; a grouped split could materially change the headline claim.","rationale":"The reader's weakest assumption identifies exactly the load-bearing risk: the train/test split is performed on 512x512 crops without grouping by the 2048x2048 source image. This is the most serious threat to the paper's central claim because the 95% accuracy is the primary quantitative evidence that the dataset supports automated shot-blasting quality control. If crops from the same source appear in both sets, the test accuracy measures near-duplicate recognition rather than generalization to new surfaces. The paper text in Section III-C provides no indication that the split respects source-image boundaries, and no per-surface evaluation is reported. My independent reading confirms the reader's assessment, and the proposed grouped-split experiment is a direct, low-cost way to determine whether the concern lands. I do not see a separate concern that would change the verdict: the dataset release, the comparison of three baselines, and the interpretability illustrations are all valuable, but the evaluation protocol must be fixed before the accuracy claims are accepted. Since the reader already recommended a conditional verdict, my recommendation is unchanged.","tokens_in":9277,"tokens_out":1960,"duration_ms":21368,"concrete_test":"Re-run the medium CCT and SVM/ResNet-50 pipelines with a group-based split: all 16 crops originating from the same 2048x2048 source image must be assigned to the same fold, using the same 80/20 ratio. Report test accuracy, precision, recall, and F1 on this grouped split, and also report per-source-image majority-vote accuracy. If accuracy drops materially (e.g., more than 3-5 percentage points), the 95% headline claim is inflated by crop-level leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that CCT and SVM with ResNet-50 features reach about 95% test accuracy on SteelBlastQC. This claim rests on the evaluation protocol in Section III-C: each original 2048x2048 image is cut into 16 crops of 512x512, and then 'the data were split into a training and a testing set (ratio of 80/20)'. No grouping by source image is described. With roughly 103 source images (1654 crops / 16), a random crop-level split places crops from the same physical surface, captured under the same lighting and camera geometry, into both training and test sets. Adjacent crops overlap in content and share illumination conditions, so the model can memorize surface-specific cues rather than learn general 'ready for paint' versus 'needs shot-blasting' discrimination. The absence of any statement that the split is surface-independent, combined with the lack of error bars or per-source evaluation, makes the 95% figure unverifiable as a claim about generalization to unseen steel surfaces. A second, compounding issue is that the CCT configuration and the CAE threshold/epoch count appear to be selected using the test set, so the reported numbers are optimistic even under a clean split. The dataset itself is a useful contribution, but the headline accuracy cannot be trusted until the leakage risk is resolved.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces SteelBlastQC, a publicly released dataset of 1654 RGB images (512×512) of shot-blasted steel surfaces, labeled by domain experts as either 'ready for paint' or 'needs shot-blasting'. The authors evaluate three baselines: a Compact Convolutional Transformer (CCT), an SVM using ResNet-50 features, and a Convolutional Autoencoder (CAE) with an SPC threshold. They report test accuracies of 95% for CCT, 94.5% for SVM, and 67.6% for CAE, and provide interpretability heatmaps for each method. The paper claims that the supervised methods achieve the stated accuracy on a held-out test set, and that the dataset and code are openly available.","tokens_in":9465,"tokens_out":5248,"duration_ms":48894,"significance":"The dataset addresses a real industrial quality-control task with a new, expert-labeled image collection, and the open release of data and code is a welcome contribution. The comparative benchmarking and interpretability visualizations provide a useful starting point for future work. However, the central empirical claim of 95% accuracy is not yet credible because the evaluation protocol appears to leak information from the same source surfaces into both training and test sets, and because model configurations are selected using test accuracy. If the authors can validate the performance under a properly isolated split, the paper would be a solid resource for the community.","major_comments":[{"comment":"The data split described in Section III-C is performed at the crop level: each 2048×2048 source image is cut into 16 samples of 512×512 and then an 80/20 random split is applied to the resulting 1654 crops, with no grouping by source image. Because all crops from a single source image are physically adjacent views of the same surface under the same controlled lighting, a random crop-level split almost certainly places highly correlated images in both training and test sets. The reported 95% accuracy is therefore not a valid estimate of performance on unseen steel surfaces. The authors must re-run the evaluation with a grouped split, holding out all 16 crops of a source image together, and report the resulting accuracy and metrics.","section":"III-C (Dataset Preprocessing)"},{"comment":"The model configuration and training duration are chosen on the basis of test-set accuracy. In Section V-B2, the three CCT configurations are compared and the medium model is selected because it 'performed the best'; in Section V-C2, the CAE is evaluated after different training epochs and the 20-epoch checkpoint is chosen because it 'yielded the best result.' This is test-set model selection, which inflates the reported accuracies. The authors should use a separate validation set for such choices and report test performance for the final selected configuration only, or at minimum explicitly acknowledge that the reported numbers include selection bias.","section":"V-B2 and V-C2 (Model selection)"},{"comment":"The results in Table I are single point estimates without error bars, repeated runs, or cross-validation. Because the dataset consists of about 103 source images, a single split has high variance, and this is compounded by the leakage and selection issues above. Reporting confidence intervals, grouped cross-validation folds, or per-source performance would allow the reader to assess whether the 95% figure is stable.","section":"V (Table I)"}],"minor_comments":[{"comment":"The abstract and Section V-D state that CCT and SVM both achieve 95% accuracy, while Table I reports 0.950 for CCT and 0.945 for SVM; the text should be consistent about whether SVM's accuracy rounds to 95%.","section":"Abstract and Table I"},{"comment":"The dataset is collected from a single steel yacht hull. The authors should state this as a limitation of the dataset's generalizability and temper the claim in the Introduction that the dataset 'enhances generalizability.'","section":"III-B (Dataset Collection)"},{"comment":"The phrase 'fair performance assessment on unseen data out-of-the-box' is misleading given the crop-level split; this should be revised to describe the actual split procedure.","section":"III-C (Dataset Preprocessing)"},{"comment":"The quantities µd and σd in Eq. (1) are not defined precisely; the authors should specify over which set of latent-space distances these statistics are computed (e.g., the training set of 'ready for paint' images).","section":"V-C2 (CAE)"},{"comment":"The implementation of Grad-CAM focusing only on the last convolutional layer of the model's tokenizer is nonstandard; a brief justification or reference for this choice would help.","section":"IV-B (CCT)"}],"recommendation":"major_revision","confidential_remarks":"The reader's concern about crop-level leakage is well-founded: the manuscript never states that the split is grouped by source image, and the preprocessing description strongly implies it is not. I recommend requesting a grouped split and a revision of the selection protocol before this paper is published. The dataset itself is a useful resource, but the headline accuracy needs to be re-established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"SteelBlastQC is a genuine resource: a public, expert-labeled dataset of shot-blasted steel surface images for a binary \"ready for paint\" vs \"needs shot-blasting\" decision, a niche where no public data existed. The dataset comes from a real facility, includes realistic defect types (welding lines, scratches, corrosion, discoloration), and the authors release both data and baseline code. That is the real contribution, and it is enough to make the paper worth engaging with.\n\nThe benchmarks, though, are standard—CCT, SVM on ResNet-50 features, CAE—and the headline 95% accuracy should not be taken at face value. The evaluation protocol has two load-bearing problems. First, Section III-C splits the 16 crops from each 2048x2048 source image into train/test at the crop level, with no grouping by source image. With roughly 103 source images, crops from the same physical surface—same lighting, same camera geometry, often overlapping content—almost certainly land in both train and test. That lets models memorize surface-specific cues, so the 95% figure is not a credible estimate of generalization to new steel surfaces. Second, model selection uses the test set: the CCT configuration is chosen after comparing test accuracies (Section V-B2), and the CAE epoch count is selected by monitoring test performance (Section V-C2). That makes the reported numbers optimistic even under a clean split. No error bars or per-surface evaluation are provided, which compounds the problem.\n\nThese are fixable flaws. A grouped split by source image, plus a proper validation hold-out for hyperparameters, would give the benchmark credibility. The qualitative heatmaps are illustrative but not quantitatively evaluated; that is a minor concern compared to the split issue. On the positive side, the related work is honest about what NEU and Texture-AD do and do not cover, and the dataset itself fills a real gap.\n\nWho is this for? Practitioners in industrial surface inspection and researchers benchmarking anomaly detection on realistic, small-scale texture data. The dataset deserves to be published and cited; the accuracy claims need re-analysis before they can be trusted. I'd send this to a serious referee, with the expectation that the evaluation be redone with a grouped split. If the numbers drop, the paper still stands on the dataset contribution.","headline":"Useful new dataset for shot-blasted steel QC, but the crop-level train/test split likely leaks surface identity, so the 95% accuracy claim is not yet credible.","tokens_in":10068,"tokens_out":2743,"would_cite":true,"duration_ms":24815,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new expert-labeled dataset lets off-the-shelf computer-vision models classify shot-blasted steel surfaces as paint-ready or not with about 95% accuracy.","keywords":["quality control","shot blasting","steel surface dataset","surface defect detection","computer vision","interpretable machine learning","convolutional transformer","convolutional autoencoder"],"falsifier":"Run the identical training protocols on a split where every 512×512 patch from one 2048×2048 source image is assigned to the same side; if CCT or SVM accuracy drops by several points, the original 95% was partly due to patch-level leakage.","tokens_in":9033,"feed_emoji":"🔩","tokens_out":7043,"duration_ms":64939,"temperature":0.7,"pith_summary":"SteelBlastQC is a new, openly released dataset of 1,654 expert-labeled RGB images of shot-blasted steel surfaces, captured in an industrial setting and divided into 'ready for paint' and 'needs shot-blasting' classes. The paper's central claim is that this dataset is realistic and usable enough that standard supervised classifiers can judge paint-readiness at about 95% test accuracy, with a compact convolutional transformer and a support vector machine on ResNet-50 features performing almost identically. It also claims that the same models can be made interpretable through heatmaps, so a quality-control operator can see which surface regions drove the decision. This matters because shot-blasting inspection is currently manual, hazardous work, and a dependable automated check would improve safety, speed, and consistency in manufacturing.","feed_headline":"Shot-blasted steel QC hits 95 percent accuracy with new dataset","feed_subtitle":"Labeled images of real hull surfaces let standard vision models flag paint-ready steel as accurately as manual checks.","key_machinery":"The central object is the SteelBlastQC dataset itself: 1,654 RGB images of 512×512 pixels, produced by cutting each centered 2048×2048 capture into 16 samples, with 888 images labeled 'ready for paint' and 766 labeled 'needs shot-blasting' by industrial experts against the SA 2.5 surface-cleanliness standard. The machine-learning machinery has three arms: a Compact Convolutional Transformer, whose convolutional tokenizer and sequence pooling combine local texture with global context; a linear SVM trained on 2048-dimensional global-average-pooled features from ResNet-50; and a convolutional autoencoder whose reconstruction error is compared with a statistical threshold to flag defects. Interpretability is carried by heatmaps: gradient-based attention from the transformer's final convolutional layer, feature-weight heatmaps for the SVM, and reconstruction-error maps for the autoencoder.","core_discovery":"On the paper's own terms, the discovery is that a purpose-built industrial surface dataset makes automated paint-readiness assessment straightforward: the medium Compact Convolutional Transformer reaches 95.0% test accuracy with F1-score 0.95, and an SVM trained on ResNet-50 features reaches 94.5% accuracy with the same F1-score, while a fully unsupervised convolutional autoencoder reaches 67.6% accuracy with 76.8% recall. The authors present this as validation that the dataset, rather than any specially engineered architecture, is the main contribution, and that interpretability does not cost accuracy: gradient-based and feature-weight heatmaps highlight welding lines, scratches, corrosion, and discoloration in line with expert judgment. The CAE baseline is described as a viable label-free option that is easily confused by shadows and by textures resembling the target surface.","pith_inferences":["A stricter surface-level evaluation, where all 16 crops from one source image stay in the same split, would show how much of the reported 95% reflects generalization to new surface areas rather than similarity among crops from the same area.","The heatmaps could be used as weak supervision to train a pixel-level defect segmenter without costly manual annotation, since the paper shows activations localized to welding lines, scratches, corrosion, and discoloration.","The binary 'ready for paint' label could be extended to a continuous roughness estimate tied to the SA 2.5 standard, making the dataset useful for process control rather than only pass/fail inspection.","A self-supervised pretraining stage on the unlabeled steel-surface images might lift the autoencoder baseline toward the supervised methods, because the current CAE uses only half the data and no labels."],"forward_implications":["Automated inspection at roughly 95% accuracy is within reach using standard supervised models, so shot-blasting quality control can shift from continuous human visual assessment to a vision-system check with human review of flagged regions.","The compact CCT configuration, about 1 million parameters, matches a 73-million-parameter model, so the practical deployment cost can be low enough for in-line or edge use.","Interpretability heatmaps align with known defect types such as welding lines, scratches, corrosion, and discoloration, giving operators a visual basis to trust or override a machine decision.","The unsupervised autoencoder path, at 67.6% accuracy with high recall, can serve as a label-free screening stage that catches most defective surfaces while generating many false alarms.","Releasing the dataset makes paint-readiness classification a reproducible benchmark for future defect-detection and interpretability methods on real industrial surfaces."],"supporting_citations":[{"why":"Supplies the Compact Convolutional Transformer architecture used as the end-to-end supervised classifier.","marker":"[7]"},{"why":"Provides the statistical learning foundation for the linear SVM classifier used on extracted features.","marker":"[8]"},{"why":"Provides the pre-trained ResNet-50 feature extractor whose global-average-pooled vectors feed the SVM.","marker":"[9]"},{"why":"Defines the convolutional autoencoder architecture that the unsupervised baseline is built on.","marker":"[10]"},{"why":"Supplies the widely used NEU steel-defect dataset that the paper contrasts with its own surface-texture task.","marker":"[12]"},{"why":"Offers a non-free industrial texture anomaly dataset that motivates the need for an openly available shot-blasting-specific dataset.","marker":"[14]"},{"why":"Defines the SA 2.5 surface-cleanliness target that the expert labels are based on.","marker":"[16]"},{"why":"Provides the Grad-CAM method used to generate the transformer's interpretation heatmaps.","marker":"[19]"},{"why":"Supplies the autoencoder-plus-statistical-threshold anomaly detection scheme used for the CAE baseline.","marker":"[20]"}],"fun_headline_variants":["Steel-surface dataset hits 95% accuracy in paint-readiness checks","Shot-blasted steel QC: dataset drives 95% accuracy, interpretable heatmaps","Dataset for shot-blasted steel: 95% accurate paint-readiness classification","Interpretable steel-defect dataset: 95% accuracy on paint-ready vs reblast"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 95 percent accuracy figure assumes the train/test split keeps all 16 image patches from the same original photograph in one side; if patches from the same surface appear on both sides, the score is inflated.","fun_headline_variants_meta":{"raw":{"variants":["Steel-surface dataset hits 95% accuracy in paint-readiness checks","Shot-blasted steel QC: dataset drives 95% accuracy, interpretable heatmaps","Dataset for shot-blasted steel: 95% accurate paint-readiness classification","Interpretable steel-defect dataset: 95% accuracy on paint-ready vs reblast"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000714,"raw_usage":{"total_tokens":3215,"prompt_tokens":951,"completion_tokens":2264,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":2176}},"tokens_in":567,"tokens_out":2264,"duration_ms":16302,"temperature":1.0,"reasoning_tokens":2176,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:26:12.457763+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical training protocols on a split where every 512×512 patch from one 2048×2048 source image is assigned to the same side; if CCT or SVM accuracy drops by several points, the original 95% was partly due to patch-level leakage.","supporting_citations":[{"cited_title":"Vapnik, The Nature of Statistical Learning Theory","cited_arxiv_id":null,"evidence_quote":"Provides the statistical learning foundation for the linear SVM classifier used on extracted features."},{"cited_title":"A better autoencoder for image: Convolutional autoencoder,","cited_arxiv_id":null,"evidence_quote":"Defines the convolutional autoencoder architecture that the unsupervised baseline is built on."},{"cited_title":"Neu metal surface defects database,","cited_arxiv_id":null,"evidence_quote":"Supplies the widely used NEU steel-defect dataset that the paper contrasts with its own surface-texture task."},{"cited_title":"Preparation of steel substrates before application of paints and related products — visual assessment of surface cleanliness,","cited_arxiv_id":null,"evidence_quote":"Defines the SA 2.5 surface-cleanliness target that the expert labels are based on."},{"cited_title":"Autoencoder-based anomaly detection for surface defect inspection,","cited_arxiv_id":null,"evidence_quote":"Supplies the autoencoder-plus-statistical-threshold anomaly detection scheme used for the CAE baseline."}],"review_version":1}