{"id":"62fcdc8f-1c3f-48f8-9b9a-efe105f7749f","arxiv_id":"1908.02548","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A crowd-sourced website collected 2,539 image labels and improved a deep learning corrosion detector from 66% to 93% accuracy over 37 days.","lead":"This paper describes a website where visitors label photos for corrosion, feeding labels into a deep learning model that also checks new uploads. Over 37 days the model's accuracy on a fixed validation set rose from 66% to 93%, suggesting crowd-sourced training can work for corrosion detection.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Validation set may overlap the training set, making the 66%→93% accuracy curve invalid as evidence for crowd-sourced improvement.","rationale":"Good-faith reading: the paper is a proof-of-concept of crowd-sourced labelling; the central claim is that crowd votes improved accuracy from 66% to 93%. The reported validation accuracy is the only support for that claim. The most load-bearing unstated precondition is that validation images were never used in training. The arithmetic inconsistency (859 initial images versus 600 training + 444 validation) makes this precondition doubtful, and the website's mechanics (random quiz presentation, five-vote eligibility for training) make accidental leakage plausible. This is a stronger threat than the epoch-count confound identified by the reader because it invalidates the outcome metric itself; controlling for training epochs would not fix a leaked validation set. The authors may well have maintained a disjoint validation set, but the manuscript as written does not establish it. A conditional verdict remains appropriate pending this information; I would not reject outright because the concern is checkable and the authors can supply the exact split. The reader's weakest assumption was the epoch-count confound, though their rationale also mentions validation disjointness, so my agreement is partial.","tokens_in":7124,"tokens_out":4224,"duration_ms":44431,"concrete_test":"Request from the authors the exact image identifiers for the initial 600-image training set, the fixed 444-image validation set, and all images that entered training via five-vote eligibility; compute the intersection of validation IDs with each training snapshot. If any validation image appears in any training set, rerun the model evaluation on a disjoint held-out set and report the accuracy curve. If the intersection is empty (i.e., validation truly external), the concern is resolved, but the authors should still document this exclusion explicitly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on the fixed 444-image validation set being disjoint from training. The manuscript never states this, and the reported numbers imply it cannot be: the initial image pool was 859 images, while the initial training set was 600 images and the validation set was 444 images; 600 + 444 > 859, so a disjoint split is arithmetically impossible unless validation images were drawn from outside the 859-image pool, which is not described. In addition, the quiz presents images 'at random' and any image receiving five votes becomes eligible for training; nothing in the methodology excludes the 444 validation images from the quiz or from later training runs. If validation images were voted on and absorbed into the training set, the ascending accuracy curve reflects memorization or leakage rather than generalization, and the 30,000-vote extrapolation has no basis. This is a correctness risk independent of the epoch-count confound: even a clean epoch control would not rescue a leaked validation metric.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes the deployment of a crowd-sourced website, corrosiondetector.com, that collects binary labels for the presence of corrosion in images. These labels are used to train a five-layer convolutional network. The paper reports that over 37 days, 2,539 votes and 310 user-uploaded images were collected, and that the model's accuracy on a fixed validation set improved from 66% at website launch to 93% at day 37. The authors also analyze voting ambiguity in several images, argue that corrosion labeling requires expert situational knowledge, and extrapolate that approximately 30,000 votes would be needed to reach 99.9% accuracy.","tokens_in":7353,"tokens_out":5790,"duration_ms":59167,"significance":"If the quantitative claim were supported, this would be a useful proof-of-concept for combining public engagement with model training for materials-degradation monitoring. The paper's discussion of the ambiguity of corrosion labels and the divergence between expert and non-expert judgments is thoughtful and adds value to the community. The paper also provides architectural details that aid reproducibility. However, the central empirical claim of crowd-sourced voting driving a 66% to 93% accuracy improvement is not currently established because of two confounds: possible overlap between the validation and training sets, and the increase from 1 to 25 training epochs. The extrapolation to 30,000 votes is also not justified.","major_comments":[{"comment":"","section":"Methodology"},{"comment":"","section":"Methodology"},{"comment":"","section":"Results and Discussion"},{"comment":"","section":"Results and Discussion"}],"minor_comments":[{"comment":"","section":"Introduction"},{"comment":"","section":"Methodology"},{"comment":"","section":"Results and Discussion"},{"comment":"","section":"References"},{"comment":"","section":"Results and Discussion"}],"recommendation":"major_revision","confidential_remarks":"The arithmetic in the Methodology section (600 + 444 > 859) makes the assertion that the validation images 'do not form part of the model training' very hard to credit. Please ask the authors explicitly to provide a complete account of the provenance of the 444 validation images and the mechanism by which they were excluded from the quiz and training pipeline. If this cannot be demonstrated convincingly, the central quantitative conclusion of the paper is unsupported and the paper should not be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a modest proof-of-concept with some genuinely useful observational data. The authors built a website, collected 2,539 crowd labels and 310 user uploads over 37 days, retrained a small CNN, and report accuracy rising from 66% to 93% on a fixed validation set. The most valuable part is the voting-disagreement discussion: they show images where non-experts split 50/50 (weathering steel, rebar before concrete pour, an offshore platform) and reason honestly about context and expertise. That material is worth reading.\n\nWhat is new is the deployment itself: the 37-day accuracy curve, the real vote counts, and the disagreement analysis. None of the components are new — crowd-sourced labelling goes back to ImageNet, and the authors already have corrosion-detection work — but measuring this particular pipeline in the wild is a small, legitimate contribution.\n\nSoft spots, in order of severity. First, the central causal claim is not supported. The baseline model was trained for 1 epoch; every later model was trained for 25 epochs. The paper never separates dataset growth from training duration, so the 66%→93% improvement could largely reflect longer training. Without an epoch-matched control, 'crowd-sourced voting improved accuracy' is an overstatement. Second, the validation set is under-specified. The text says validation images do not form part of model training, but it never states that the 444 validation images are disjoint from the 859-image quiz pool. The arithmetic 600 + 444 > 859 means a straightforward disjoint split is impossible unless validation images came from outside that pool. If any validation images were shown in the quiz and later absorbed into training after five votes, the accuracy curve would be inflated. I do not think the stress-test claim is proven — an external expert-labelled validation set is plausible — but the authors need to state the provenance explicitly. Third, there are no error bars or repeated runs, and the ~30,000-vote extrapolation to 99.9% accuracy is a line drawn through a short curve with no uncertainty attached.\n\nRecommendation: this deserves a serious referee, not a desk rejection. The flaws are fixable: add an epoch-matched baseline, describe the validation set precisely, and ideally release code and data. As it stands, the headline claim is unproven, but the paper is honest, clearly written, and reports real field observations.","headline":"A likeable, honest field deployment for crowd-sourced corrosion labelling, but the headline 66%→93% claim is confounded by an epoch-count change and an under-specified validation split.","tokens_in":7807,"tokens_out":2563,"would_cite":false,"duration_ms":27448,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A crowd-sourced labelling website trained a deep-learning corrosion detector whose validation accuracy rose from 66% to 93% in 37 days.","keywords":["corrosion detection","crowd-sourced labelling","deep learning","image classification","convolutional neural network","human-in-the-loop training","rust detection","dataset growth"],"falsifier":"Retrain the original 600-image baseline model for 25 epochs on the same initial data, with no crowd-sourced additions, and measure validation accuracy; if it approaches 93%, the reported improvement is an artefact of longer training rather than crowd-sourced data.","tokens_in":6930,"feed_emoji":"⚙️","tokens_out":4913,"duration_ms":43896,"temperature":0.7,"pith_summary":"This paper reports a proof-of-concept in which a public website collected crowd-sourced labels for whether photographs show corrosion (rust), and those labels were used to retrain a deep-learning classifier. Over 37 days, 2,539 votes from visitors raised the model's accuracy on a fixed validation set from 66% to 93%. The authors argue this demonstrates a practical route to building domain-specific visual detectors without a dedicated expert labelling team, and they extrapolate that roughly 30,000 votes would reach 99.9% accuracy. The website also lets users upload images for instant AI assessment, making the evolving model publicly usable.","feed_headline":"Crowd votes lift rust-detection AI from 66% to 93%","feed_subtitle":"A public labelling loop turned 2,539 votes into a corrosion detector in 37 days.","key_machinery":"The central mechanism is a closed loop: the website shows four random images per page; users check boxes next to those they believe show corrosion; after five votes an image's majority label enters the training set; the model is retrained on the growing set; and the updated model then scores each newly uploaded image, with user uploads later entering the same voting loop. The classifier is a five-layer convolutional network modeled on the encoding side of the U-Net architecture, trained with cross-entropy loss and an adaptive-moment optimizer.","core_discovery":"The core claim is that crowd-sourced voting on a simple binary question—does this image show corrosion or not?—can serve as the training signal for an automated corrosion detector, with the model improving from 66% to 93% validation accuracy over 37 days as 2,539 votes were accumulated. The claim includes the extrapolation that about 30,000 votes would be needed to reach 99.9% accuracy. The authors also document that many images receive deeply split votes, showing that non-expert labels carry noise and that 'corrosion' is a context-dependent judgment.","pith_inferences":["If the crowd-data effect is real, the same five-vote majority loop could generalize to other visual defects with clearer ground truth, such as cracks or spalls, where voter disagreement is lower, likely accelerating convergence.","The paper implicitly treats accuracy as average correctness over a fixed validation set; a per-class breakdown would reveal whether the gain comes from better corrosion recall, better no-corrosion precision, or both.","The self-selection of voters (people visiting a corrosion website) may mean the result does not transfer to a generic online crowd; a controlled comparison with general crowd workers would test transferability."],"forward_implications":["A domain-specific detector can be bootstrapped from non-expert public labels, reaching practically useful accuracy (93%) within about five weeks of launch.","Extrapolating the observed learning curve, roughly 30,000 votes would be needed to reach 99.9% accuracy, which is useful for planning data-collection effort in remote inspection applications.","The same loop can be reused: user-uploaded images are immediately scored by the current model and, after five votes, enter the training set, so the system improves while being used.","Label ambiguity is intrinsic to visual corrosion: on context-dependent images voters disagree 40–60% of the time, which caps the accuracy any model trained on majority labels can reach."],"supporting_citations":[{"why":"Prior estimate that over 65,000 labelled images are needed for human-level corrosion detection; defines the data bottleneck this work targets.","marker":"[11]"},{"why":"Shows deep networks tolerate large label noise, supporting the viability of non-expert crowd labels.","marker":"[10]"},{"why":"Supplies the U-Net encoding architecture on which the five-layer classifier is modeled.","marker":"[19]"},{"why":"Supplies the weight-initialization scheme used to start training stably.","marker":"[20]"},{"why":"ImageNet is the canonical example of crowd-voted labelling at scale, the pattern this paper adapts.","marker":"[12]"}],"fun_headline_variants":["2,539 crowd votes train rust detector to 93% accuracy","Crowd-sourced labels boost corrosion AI from 66% to 93%","Public votes teach deep learning to spot rust in 37 days","37-day crowd effort lifts corrosion detector to 93%","Everyday voters help deep learning detect corrosion at 93%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The accuracy gain is attributed to the growing crowd-labelled dataset, but the baseline ran for 1 epoch while later models ran for 25 epochs; if longer training alone explains most of the gain, the crowd-sourcing claim collapses.","fun_headline_variants_meta":{"raw":{"variants":["2,539 crowd votes train rust detector to 93% accuracy","Crowd-sourced labels boost corrosion AI from 66% to 93%","Public votes teach deep learning to spot rust in 37 days","37-day crowd effort lifts corrosion detector to 93%","Everyday voters help deep learning detect corrosion at 93%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000504,"raw_usage":{"total_tokens":2411,"prompt_tokens":843,"completion_tokens":1568,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":459,"completion_tokens_details":{"reasoning_tokens":1477}},"tokens_in":459,"tokens_out":1568,"duration_ms":13151,"temperature":1.0,"reasoning_tokens":1477,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T15:18:37.270533+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the original 600-image baseline model for 25 epochs on the same initial data, with no crowd-sourced additions, and measure validation accuracy; if it approaches 93%, the reported improvement is an artefact of longer training rather than crowd-sourced data.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior estimate that over 65,000 labelled images are needed for human-level corrosion detection; defines the data bottleneck this work targets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the weight-initialization scheme used to start training stably."}],"review_version":1}