Automated Corrosion Detection Using Crowd Sourced Training for Deep Learning

C.J. Powell; N. Birbilis; T. Drummond; W.T. Nash

REVIEW 4 major objections 5 minor 21 references

Automated Corrosion Detection Using Crowd Sourced Training for Deep Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A crowd-sourced labelling website trained a deep-learning corrosion detector whose validation accuracy rose from 66% to 93% in 37 days.

desk verdict A likeable, honest field deployment for crowd-sourced corrosion labelling, but the headline 66%→93% claim is confounded by an epoch-count change and an under-specified validation split. read the letter →

arxiv 1908.02548 v1 pith:UABZZSO6 submitted 2019-08-04 cs.HC eess.IV

W.T. Nash , C.J. Powell , T. Drummond , N. Birbilis This is my paper · ORCID

classification cs.HCeess.IV

keywords corrosiondetectioncrowd-sourcedlabellingdeeplearningimageclassificationconvolutionalneuralnetworkhuman-in-the-looptrainingrustdatasetgrowth

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports a proof-of-concept in which a public website collected crowd-sourced labels for whether photographs show corrosion (rust), and those labels were used to retrain a deep-learning classifier. Over 37 days, 2,539 votes from visitors raised the model's accuracy on a fixed validation set from 66% to 93%. The authors argue this demonstrates a practical route to building domain-specific visual detectors without a dedicated expert labelling team, and they extrapolate that roughly 30,000 votes would reach 99.9% accuracy. The website also lets users upload images for instant AI assessment, making the evolving model publicly usable.

What carries the argument

The central mechanism is a closed loop: the website shows four random images per page; users check boxes next to those they believe show corrosion; after five votes an image's majority label enters the training set; the model is retrained on the growing set; and the updated model then scores each newly uploaded image, with user uploads later entering the same voting loop. The classifier is a five-layer convolutional network modeled on the encoding side of the U-Net architecture, trained with cross-entropy loss and an adaptive-moment optimizer.

What would settle it

Retrain the original 600-image baseline model for 25 epochs on the same initial data, with no crowd-sourced additions, and measure validation accuracy; if it approaches 93%, the reported improvement is an artefact of longer training rather than crowd-sourced data.

Watch

Extended reading notes

Core claim

The core claim is that crowd-sourced voting on a simple binary question—does this image show corrosion or not?—can serve as the training signal for an automated corrosion detector, with the model improving from 66% to 93% validation accuracy over 37 days as 2,539 votes were accumulated. The claim includes the extrapolation that about 30,000 votes would be needed to reach 99.9% accuracy. The authors also document that many images receive deeply split votes, showing that non-expert labels carry noise and that 'corrosion' is a context-dependent judgment.

Load-bearing premise

The accuracy gain is attributed to the growing crowd-labelled dataset, but the baseline ran for 1 epoch while later models ran for 25 epochs; if longer training alone explains most of the gain, the crowd-sourcing claim collapses.

Editorial extensions

If this is right

A domain-specific detector can be bootstrapped from non-expert public labels, reaching practically useful accuracy (93%) within about five weeks of launch.
Extrapolating the observed learning curve, roughly 30,000 votes would be needed to reach 99.9% accuracy, which is useful for planning data-collection effort in remote inspection applications.
The same loop can be reused: user-uploaded images are immediately scored by the current model and, after five votes, enter the training set, so the system improves while being used.
Label ambiguity is intrinsic to visual corrosion: on context-dependent images voters disagree 40–60% of the time, which caps the accuracy any model trained on majority labels can reach.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the crowd-data effect is real, the same five-vote majority loop could generalize to other visual defects with clearer ground truth, such as cracks or spalls, where voter disagreement is lower, likely accelerating convergence.
The paper implicitly treats accuracy as average correctness over a fixed validation set; a per-class breakdown would reveal whether the gain comes from better corrosion recall, better no-corrosion precision, or both.
The self-selection of voters (people visiting a corrosion website) may mean the result does not transfer to a generic online crowd; a controlled comparison with general crowd workers would test transferability.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a modest proof-of-concept with some genuinely useful observational data. The authors built a website, collected 2,539 crowd labels and 310 user uploads over 37 days, retrained a small CNN, and report accuracy rising from 66% to 93% on a fixed validation set. The most valuable part is the voting-disagreement discussion: they show images where non-experts split 50/50 (weathering steel, rebar before concrete pour, an offshore platform) and reason honestly about context and expertise. That material is worth reading.

What is new is the deployment itself: the 37-day accuracy curve, the real vote counts, and the disagreement analysis. None of the components are new — crowd-sourced labelling goes back to ImageNet, and the authors already have corrosion-detection work — but measuring this particular pipeline in the wild is a small, legitimate contribution.

Soft spots, in order of severity. First, the central causal claim is not supported. The baseline model was trained for 1 epoch; every later model was trained for 25 epochs. The paper never separates dataset growth from training duration, so the 66%→93% improvement could largely reflect longer training. Without an epoch-matched control, 'crowd-sourced voting improved accuracy' is an overstatement. Second, the validation set is under-specified. The text says validation images do not form part of model training, but it never states that the 444 validation images are disjoint from the 859-image quiz pool. The arithmetic 600 + 444 > 859 means a straightforward disjoint split is impossible unless validation images came from outside that pool. If any validation images were shown in the quiz and later absorbed into training after five votes, the accuracy curve would be inflated. I do not think the stress-test claim is proven — an external expert-labelled validation set is plausible — but the authors need to state the provenance explicitly. Third, there are no error bars or repeated runs, and the ~30,000-vote extrapolation to 99.9% accuracy is a line drawn through a short curve with no uncertainty attached.

Recommendation: this deserves a serious referee, not a desk rejection. The flaws are fixable: add an epoch-matched baseline, describe the validation set precisely, and ideally release code and data. As it stands, the headline claim is unproven, but the paper is honest, clearly written, and reports real field observations.

Referee Report

4 major / 5 minor

Summary. The paper describes the deployment of a crowd-sourced website, corrosiondetector.com, that collects binary labels for the presence of corrosion in images. These labels are used to train a five-layer convolutional network. The paper reports that over 37 days, 2,539 votes and 310 user-uploaded images were collected, and that the model's accuracy on a fixed validation set improved from 66% at website launch to 93% at day 37. The authors also analyze voting ambiguity in several images, argue that corrosion labeling requires expert situational knowledge, and extrapolate that approximately 30,000 votes would be needed to reach 99.9% accuracy.

Significance. If the quantitative claim were supported, this would be a useful proof-of-concept for combining public engagement with model training for materials-degradation monitoring. The paper's discussion of the ambiguity of corrosion labels and the divergence between expert and non-expert judgments is thoughtful and adds value to the community. The paper also provides architectural details that aid reproducibility. However, the central empirical claim of crowd-sourced voting driving a 66% to 93% accuracy improvement is not currently established because of two confounds: possible overlap between the validation and training sets, and the increase from 1 to 25 training epochs. The extrapolation to 30,000 votes is also not justified.

major comments (4)

[Methodology]
[Methodology]
[Results and Discussion]
[Results and Discussion]

minor comments (5)

[Introduction]
[Methodology]
[Results and Discussion]
[References]
[Results and Discussion]

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity in the main accuracy claim; minor self-citations and a transparent trend extrapolation do not make the derivation circular.

full rationale

The central claim—that 2,539 crowd-sourced votes improved corrosion-detection accuracy from 66% to 93%—is not circular by construction. The accuracy is reported 'relative to the validation images (which do not form part of the model training and expertly labelled for the purposes of identifying the model error)' (Results and Discussion, Model Accuracy), so the performance metric is an external benchmark rather than a refit of the training labels. The improvement is an observed outcome of retraining on growing data, not a parameter fitted to the validation set. The baseline was trained for 1 epoch while later models used 25 epochs (Methodology); this is a confound for attributing the gain to crowd-sourced data, but it is an experimental-design issue, not a circular derivation. The 30,000-vote extrapolation is 'based on the empirical evidence from the present work' (Results and Discussion) and is transparently a trend extrapolation from the same accuracy curve, not an independent prediction; it is peripheral to the measured accuracy claim. The >65,000-image requirement is attributed to the authors' own prior work [11] (Introduction) and serves only as motivation, not as a premise that forces the measured 66% to 93% result. A validation-overlap risk is suggested by the arithmetic (600 training + 444 validation > 859 initial voting images), which would be fatal if both sets were drawn from the 859-image pool and validation images were allowed into training via the five-vote rule; however, the manuscript does not state that the 600 and 444 are subsets of the 859, and it explicitly asserts the validation set is outside training. That is a missing methodological detail and a leakage risk, but not a demonstrated circular reduction. Overall, the derivation chain for the main accuracy improvement is self-contained with respect to an external validation benchmark.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on crowd label quality, the validity of the fixed validation set, the comparability of training regimes, and an extrapolated trend. No new physical entities are introduced; the main free parameters are the vote threshold, epoch counts, learning rate, and the extrapolated 30,000-vote target.

free parameters (4)

Majority vote threshold = 5 votes
Images are labelled corrosion/no-corrosion by majority of at least 5 user votes; this threshold defines the training labels and is chosen by hand rather than optimized.
Training epoch counts = 1 (baseline) and 25 (subsequent)
The baseline model was trained for 1 epoch and later models for 25 epochs; this difference confounds the reported accuracy improvement.
Learning rate = 1e-4
ADAM optimizer learning rate used for training; a standard hyperparameter choice that affects convergence.
Extrapolated vote count for 99.9% accuracy = ~30,000 votes
Stated in Results as required to reach 99.9% accuracy; derived from an unspecified empirical trend in Figure 3b, not measured directly.

assumptions (5)

domain assumption Crowd-sourced majority labels provide a sufficiently accurate training signal for deep learning corrosion detection.
Core to the method; the paper acknowledges 40-60% voter disagreement and non-expert users, so this is an unverified premise. Introduced in Methodology and discussed in Results.
domain assumption The fixed 444-image validation set is expertly labelled, representative, and independent of training data.
All accuracy numbers in Figure 3b depend on this; the paper never details the expert labelling protocol or demonstrates validation independence from training.
ad hoc to paper The accuracy improvement is attributable to the growing crowd-sourced dataset, not the change from 1 to 25 training epochs.
The paper changes both dataset size and epoch count between baseline and final evaluation, but presents the gain as evidence for crowd-sourced training.
ad hoc to paper Model accuracy will continue to improve with more votes, allowing extrapolation to 30,000 votes for 99.9% accuracy.
The extrapolation is based on 'the empirical evidence from the present work' without a fitted functional form or uncertainty, assuming the trend continues.
standard math Standard deep learning training (Kaiming init, ADAM, cross-entropy) behaves as expected for this task.
Background tooling assumption; not directly proven but widely established in the literature.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Automated Corrosion Detection Using Crowd Sourced Training for Deep Learning." pith.science (2026). https://pith.science/paper/UABZZSO6

@misc{pith2026190802548,
  author       = {Pith},
  title        = {Pith review of: Automated Corrosion Detection Using Crowd Sourced Training for Deep Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UABZZSO6}},
  note         = {Machine review of arXiv:1908.02548}
}

read the original abstract

The automated detection of corrosion from images (i.e., photographs) or video (i.e., drone footage) presents significant advantages in terms of corrosion monitoring. Such advantages include access to remote locations, mitigation of risk to inspectors, cost savings and monitoring speed. The automated detection of corrosion requires deep learning to approach human level artificial intelligence (A.I.). The training of a deep learning model requires intensive image labelling, and in order to generate a large database of labelled images, crowd sourced labelling via a dedicated website was sought. The website (corrosiondetector.com) permits any user to label images, with such labelling then contributing to the training of a cloud based A.I. model - with such a cloud-based model then capable of assessing any fresh (or uploaded) image for the presence of corrosion. In other words, the website includes both the crowd sourced training process, but also the end use of the evolving model. Herein, the results and findings from the website (corrosiondetector.com) over the period of approximately one month, are reported.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 11 canonical work pages

[2]

LeCun, Y

Y. LeCun, Y. Bengio, G. Hinton, Deep learning, Nature. 521 (2015) 436–444. doi:10.1038/nature14539

doi:10.1038/nature14539 2015
[3]

W. Nash, T. Drummond, N. Birbilis, A review of deep learning in the study of materials degradation, Npj Mater. Degrad. 2 (2018) 1–12. doi:10.1038/s41529-018-0058-x

work page doi:10.1038/s41529-018-0058-x 2018
[4]

Rumelhart, G.E

D.E. Rumelhart, G.E. Hinton, R.J. Williams, Learning representations by back-propogation errors, Nature. 323 (1986) 533–536

work page 1986
[5]

Z. Cui, G. Gong, The effect of machine learning regression algorithms and sample size on individualized behavioral prediction with functional connectivity features, Neuroimage. 178 (2018) 622–637. doi:10.1016/j.neuroimage.2018.06.001

work page doi:10.1016/j.neuroimage.2018.06.001 2018
[6]

Al-Haddad, C.W

L. Al-Haddad, C.W. Morris, L. Boddy, Training radial basis function neural networks: Effects of training set size and imbalanced training sets, J. Microbiol. Methods. 43 (2000) 33–44. doi:10.1016/S0167-7012(00)00202-5

work page doi:10.1016/s0167-7012(00)00202-5 2000
[7]

Gonzalez-Diaz, E

R. Gonzalez-Diaz, E. Paluzo-Hidalgo, M.A. Gutiérrez-Naranjo, Representative datasets for neural networks, Electron. Notes Discret. Math. 68 (2018) 89–94. doi:10.1016/j.endm.2018.06.016

work page doi:10.1016/j.endm.2018.06.016 2018
[8]

Foody, M.B

G.M. Foody, M.B. McCulloch, W.B. Yates, The effect of training set size and composition on artificial neural network classification, Int. J. Remote Sens. 16 (1995) 1707–1723. doi:10.1080/01431169508954507

work page doi:10.1080/01431169508954507 1995
[9]

Fawzi, S.-M

A. Fawzi, S.-M. Moosavi-Dezfooli, P. Frossard, Robustness of classifiers: from adversarial to random noise, (2016). http://arxiv.org/abs/1608.08967

arXiv 2016

Show all 21 references

[10]

Rolnick, A

D. Rolnick, A. Veit, S. Belongie, N. Shavit, Deep Learning is Robust to Massive Label Noise, (2017) 1–10. http://arxiv.org/abs/1705.10694

2017 arXiv
[11]

W. Nash, T. Drummond, N. Birbilis, Deep Learning AI for Corrosion Detection, in: NACE International (Ed.), Corros. 2019, Nashville, TN,

2019
[12]

http://arxiv.org/abs/1506.06579

arXiv
[13]

Socher, Li-Jia Li, Kai Li, Li Fei-Fei, ImageNet: A large-scale hierarchical image database, 2009 IEEE Conf

Jia Deng, Wei Dong, R. Socher, Li-Jia Li, Kai Li, Li Fei-Fei, ImageNet: A large-scale hierarchical image database, 2009 IEEE Conf. Comput. Vis. Pattern Recognit. (2009) 248–255. doi:10.1109/CVPRW.2009.5206848

2009
[14]

T.Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, C.L. Zitnick, Microsoft COCO: Common objects in context, Lect. Notes Comput. Sci. (Including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics). 8693 LNCS (2014) 740–755. doi:10.1007/978-3-...

2014 doi
[15]

B. Zhou, H. Zhao, X. Puig, T. Xiao, S. Fidler, A. Barriuso, A. Torralba, Semantic Understanding of Scenes through the ADE20K Dataset, (2016). doi:10.1152/ajpendo.00144.2015

2016
[16]

Russakovsky, J

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A.C. Berg, L. Fei-Fei, ImageNet Large Scale Visual Recognition Challenge, Int. J. Comput. Vis. 115 (2015) 211–252. doi:10.1007/s11263-015-0816-y

2015 doi
[17]

Figueroa, Q

R.L. Figueroa, Q. Zeng-Treitler, S. Kandula, L.H. Ngo, Predicting sample size required for classification performance, BMC Med. Inform. Decis. Mak. 12 (2012). doi:10.1186/1472-6947-12-8

2012 doi
[18]

Mukherjee, P

S. Mukherjee, P. Tamayo, S. Rogers, R. Rifkin, A. Engle, C. Campbell, T.R. Golub, J.P. Mesirov, Estimating Dataset Size Requirements for Classifying DNA Microarray Data, J. Comput. Biol. 10 (2003) 119–142. doi:10.1089/106652703321825928

2003 doi
[19]

Ronneberger, P

O. Ronneberger, P. Fischer, T. Brox, U-Net: Convolutional Networks for Biomedical Image Segmentation, (2015) 1–8. doi:10.1007/978-3-319-24574-4_28

2015 doi
[20]

K. He, X. Zhang, S. Ren, J. Sun, Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification, Biochem. Biophys. Res. Commun. 498 (2015) 254–261. doi:10.1016/j.bbrc.2018.01.076

2015 doi
[2003]

doi:10.1201/9781482265491

doi
[2017]

doi:10.1109/CVPR.2017.544

2017–Janua (2017) 5122–5130. doi:10.1109/CVPR.2017.544

2017 doi

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [2]

LeCun, Y

Y. LeCun, Y. Bengio, G. Hinton, Deep learning, Nature. 521 (2015) 436–444. doi:10.1038/nature14539

doi:10.1038/nature14539 2015

[2] [3]

W. Nash, T. Drummond, N. Birbilis, A review of deep learning in the study of materials degradation, Npj Mater. Degrad. 2 (2018) 1–12. doi:10.1038/s41529-018-0058-x

work page doi:10.1038/s41529-018-0058-x 2018

[3] [4]

Rumelhart, G.E

D.E. Rumelhart, G.E. Hinton, R.J. Williams, Learning representations by back-propogation errors, Nature. 323 (1986) 533–536

work page 1986

[4] [5]

Z. Cui, G. Gong, The effect of machine learning regression algorithms and sample size on individualized behavioral prediction with functional connectivity features, Neuroimage. 178 (2018) 622–637. doi:10.1016/j.neuroimage.2018.06.001

work page doi:10.1016/j.neuroimage.2018.06.001 2018

[5] [6]

Al-Haddad, C.W

L. Al-Haddad, C.W. Morris, L. Boddy, Training radial basis function neural networks: Effects of training set size and imbalanced training sets, J. Microbiol. Methods. 43 (2000) 33–44. doi:10.1016/S0167-7012(00)00202-5

work page doi:10.1016/s0167-7012(00)00202-5 2000

[6] [7]

Gonzalez-Diaz, E

R. Gonzalez-Diaz, E. Paluzo-Hidalgo, M.A. Gutiérrez-Naranjo, Representative datasets for neural networks, Electron. Notes Discret. Math. 68 (2018) 89–94. doi:10.1016/j.endm.2018.06.016

work page doi:10.1016/j.endm.2018.06.016 2018

[7] [8]

Foody, M.B

G.M. Foody, M.B. McCulloch, W.B. Yates, The effect of training set size and composition on artificial neural network classification, Int. J. Remote Sens. 16 (1995) 1707–1723. doi:10.1080/01431169508954507

work page doi:10.1080/01431169508954507 1995

[8] [9]

Fawzi, S.-M

A. Fawzi, S.-M. Moosavi-Dezfooli, P. Frossard, Robustness of classifiers: from adversarial to random noise, (2016). http://arxiv.org/abs/1608.08967

arXiv 2016