{"id":"d4e3ced6-4233-4f4c-9419-33dc133f74df","arxiv_id":"2411.18007","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A YOLOv8 membrane-extraction plus CNN classifier pipeline reads COVID-19 rapid test kit photos with 99.1% reported accuracy, but the claimed improvement over existing interpretation methods is not directly demonstrated.","lead":"The authors built a smartphone app that uses YOLOv8 to crop the test strip area from a photo of a rapid diagnostic test kit, then a CNN classifies the result as positive, negative, or invalid. The app reports over 99% accuracy on their test images, but the paper does not compare against human readers or prior automated readers.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim that YOLO membrane extraction improves accuracy is unsupported: no ablation or independent test-set guarantee, so the 99.1% result cannot be causally attributed to the cropping step.","rationale":"The reader's REJECT verdict is well supported. The most load-bearing unsupported assertion is the causal attribution of improved accuracy to YOLO membrane extraction. That claim requires at least an ablation removing the YOLO step, a human or prior-system baseline, and a test set whose images are independent of the training set. None of these is provided. The dataset description leaves open a concrete leakage mechanism: near-duplicate photos of the same physical kit or same capture session could straddle the batch split and inflate the 99.1% accuracy. The absence of confidence intervals and external validation compounds the problem. Because the central claim fails on the evidence actually presented, and because no independent support such as released code, external validation, or a user study with visually impaired participants is supplied, the rejection stands. My wording emphasizes the missing ablation as the primary logical gap, while the reader's weakest assumption also correctly flags the independence of the test set; the two concerns are complementary and both point to the same unsupported causal claim.","tokens_in":14587,"tokens_out":4123,"duration_ms":39764,"concrete_test":"Use the exact dataset from Sections 2.1-2.3 and run two controlled comparisons: (A) train the same CNN on full uncropped images, with identical grayscale and resizing, and compare accuracy, sensitivity, specificity, and Wilson confidence intervals against the YOLO-cropped pipeline on the same test set; (B) redo the split grouping images by physical test kit or capture session so that no kit or session spans train and test, then rerun the cropped pipeline. If (A) shows no statistically significant difference, the causal attribution to YOLO fails. If (B) lowers accuracy substantially, the reported 99.1% is partially due to leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is causal: the abstract states that the improvement in sensitivity and specificity 'can be attributed to the extraction of the membrane zones from the test kit images using the state-of-the-art YOLO algorithm.' The only evidence offered, however, is the end-to-end confusion matrix in Section 3.2, which reports 99.1% accuracy on 456 test images. No comparison is made against the same CNN trained on the full, uncropped smartphone images, nor against a human reader or a prior automated reader. Section 2.1 mixes a published dataset [24] with newly captured images, and Section 2.3 describes splitting data into 54/12/12 batches without stating whether batches are formed at the level of physical test kits, capture sessions, or individual images. If the same kit or the same photo session appears in both training and test partitions, the reported accuracy can be inflated by near-duplicate memorization, independent of whether YOLO cropping helps. The SHAP analysis in Section 3.3 illustrates a handful of correct and incorrect predictions but does not quantify the marginal contribution of the cropping stage. Thus the load-bearing assertion that YOLO extraction improves sensitivity and specificity is not supported by any controlled experiment, and the reported accuracy may reflect dataset leakage rather than the claimed mechanism.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a smartphone-based pipeline for reading COVID-19 rapid diagnostic tests: a YOLOv8 model detects and crops the membrane region, and a CNN classifies the cropped image as positive, negative, or invalid. The authors report roughly 99.1% test accuracy on a mixed dataset of previously published and newly captured images, and use SHAP to qualitatively inspect model decisions. The main claimed contribution is that YOLO-based membrane extraction improves classification sensitivity and specificity, enabling use by visually impaired users who cannot center the test kit in the frame. The paper also describes a deployed Android app with a Flask backend that returns results via a cloud pipeline.","tokens_in":14848,"tokens_out":2760,"duration_ms":25381,"significance":"If the causal claim were established, this would be a useful practical contribution to point-of-care diagnostics and accessibility: an off-the-shelf pipeline that lets users photograph an RDT without careful alignment, then automatically crops and classifies the result. The reported accuracy is high, and the inclusion of a working mobile app and SHAP analysis are strengths. However, the central causal claim is not supported by the experiments as presented: there is no ablation or comparison to the same classifier on uncropped images, no analysis of how YOLO cropping errors affect final classification, and no error bars or external validation. The paper's significance therefore hinges on an untested assumption, and the current evaluation is insufficient to justify the title's accessibility claim beyond a proof-of-concept.","major_comments":[{"comment":"The abstract states that improved sensitivity and specificity 'can be attributed to the extraction of the membrane zones' using YOLO, but §3.2 only reports the end-to-end confusion matrix on cropped images. There is no comparison with the same CNN trained or tested on full, uncropped smartphone images, no ablation of the YOLO step, and no analysis of the effect of YOLO cropping errors on test predictions. As written, the 99.1% accuracy could be due to the classifier alone or to dataset-specific cues, not to the cropping mechanism. Please add a controlled experiment, e.g., the same CNN on full images versus cropped images, and report the difference.","section":"Abstract and §3.2"},{"comment":"The data split is described as '80 batches' with 54/12/12 for train/validation/test, but it is not stated whether a 'batch' corresponds to a physical test kit, a capture session, or individual images. If multiple near-duplicate images from the same kit or the same photo session appear in both training and test partitions, the reported accuracy would be inflated by memorization rather than generalization. Please clarify the grouping unit and, ideally, split by physical test kit or capture session to ensure test-set independence.","section":"§2.3"},{"comment":"The confusion matrix and the 'approximately 99.1%' accuracy are presented without confidence intervals or any measure of uncertainty. With 456 test images and only 4 misclassifications, the binomial 95% confidence interval spans roughly 97.5% to 99.8%, so the reported figure is not distinguishable from other high-performing classifiers. Please provide confidence intervals and, if possible, bootstrap or repeated-seed results. A comparison with human readers or with previously published automated readers (e.g., the ALFA system or Mendels et al. [24]) would also help calibrate the practical significance.","section":"§3.2 and Figure 11"},{"comment":"The SHAP analysis is used to infer that the model relies on test/control lines and that misclassifications are due to smudges or faint lines, but the evidence is qualitative: a handful of heatmaps and the four incorrect test-set examples. It does not quantify the contribution of the YOLO cropping step, and the attribution of misclassifications to specific image marks is post-hoc and not statistically tested. The section also confusingly refers to 'pregnancy test' results in two places, which is inconsistent with the COVID-19 RDT context. Please correct the text and, if the SHAP analysis is intended to support the cropping claim, add a more systematic evaluation, e.g., comparing SHAP maps on correctly and incorrectly cropped images.","section":"§3.3"}],"minor_comments":[{"comment":"There is an inconsistency in input size: §2.3 states images are resized to 256×256 pixels, while Table 2 and Figure 5 state 128×128. Please reconcile these numbers.","section":"§2.3 and Table 2/Figure 5"},{"comment":"Several in-text citations appear to be misattributed. For example, the introduction credits 'Krizhevsky et al. [20]' with developing the xRCovid app, but xRCovid is described in the literature as the work of Dortet et al. [13]; reference [4] (van Grinsven et al.) is cited as an LFA study but the actual title is about hemorrhage detection in fundus images; and reference [7] and [24] both involve Mendels et al. but describe different works. Please carefully verify each citation against the reference list and correct the attributions.","section":"References"},{"comment":"The text in the SHAP section refers to 'classifying pregnancy test results' and 'pregnancy test membrane' when the paper is about COVID-19 RDTs. Please replace these with the correct terminology.","section":"§3.3"},{"comment":"The sentence 'In Figure 12(a), the sample is incorrectly classified as Positive instead of Negative' refers to misclassified samples, but Figure 12 is captioned as 'correctly classified samples.' The incorrect samples are shown in Figure 13. Please correct the figure reference.","section":"§3.3, Figures 12–13"},{"comment":"Ground-truth labeling is described as 'reviewed at least twice by human analysts, with a third review if needed,' but no inter-reviewer agreement or quality-control metric is reported. A brief note on labeling reliability would strengthen the dataset description.","section":"§2.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a plausible engineering demonstration, but the central claim of causality is not backed by the current experimental design. The missing ablation and unclear split grouping are fixable within a revision, and the reference/citation issues, while numerous, are also fixable. I would not reject outright, but I would require the authors to add a proper ablation and clarify the dataset split before reconsideration."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Honest take: this is a workmanlike system paper with a headline claim it doesn't support. The authors put together YOLOv8 membrane cropping, a three-class CNN, a mobile app, and SHAP analysis, and they report 99.1% test accuracy. That pipeline description is clear. But the abstract says the improvement 'can be attributed to the extraction of the membrane zones'—and there is no experiment that tests that attribution. No baseline of the CNN on full, uncropped images, no comparison to human readers or prior automated readers, no ablation, no error bars or confidence intervals. The 99.1% is just the end-to-end result on one split.\n\nThe split itself is underspecified: 54/12/12 batches, but batches of what? If multiple images from the same test kit or the same photo session end up in both training and test, the accuracy can be inflated by near-duplicate memorization. The paper doesn't rule that out.\n\nThere are also messy lapses. Section 2.3 says images are resized to 256×256, while Figure 5 and the architecture table say 128×128. The SHAP section talks about 'pregnancy test results' even though the paper is about COVID-19 RDTs. The introduction attributes the xRCovid app to Krizhevsky et al., which is incorrect. None of these are fatal on their own, but together they suggest the manuscript was not carefully checked.\n\nCredit where it's due: the dataset construction is real work, the system is end-to-end and appears to work on its own test set, and the SHAP examples give useful intuition. The accessibility goal is worthwhile, but there's no usability evaluation with visually impaired users, so it's a proposal rather than a validated feature.\n\nBottom line: the paper is a reasonable engineering report, but its central causal claim is unsupported, and the current evaluation doesn't meet the bar for a diagnostic paper. It could be a useful case study for why ablations matter, but I wouldn't rely on or cite the results. I would not send it to a serious referee in this form; it needs a comparison experiment and a clarified split first.","headline":"A competent system paper whose central causal claim—YOLO cropping improves accuracy—rests on no comparison, plus several sloppy inconsistencies.","tokens_in":15415,"tokens_out":3840,"would_cite":false,"duration_ms":32593,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that cropping the membrane region with YOLOv8 before CNN classification yields roughly 99.1% accuracy on rapid-test images, and that the crop is the cause of the improvement.","keywords":["rapid diagnostic tests","smartphone diagnostics","YOLOv8","convolutional neural networks","point-of-care diagnostics","SHAP analysis","lateral flow assays","visual impairment accessibility"],"falsifier":"Feed the same test-set images to the CNN once with the YOLO membrane crop and once with the full uncropped kit photo; if accuracy without cropping does not fall clearly below 99.1%, the central attribution of the gain to YOLO extraction is not supported. A second check would remove near-duplicate images between training and test sets and re-measure accuracy to see whether the reported figure reflects generalization or memorization.","tokens_in":14416,"feed_emoji":"📱","tokens_out":9741,"duration_ms":79277,"temperature":0.7,"pith_summary":"The paper proposes a smartphone-based system that reads rapid diagnostic tests (RDTs) automatically from a photo. A YOLOv8 detector first finds and crops the membrane region of the test kit, and a convolutional neural network then classifies the crop as positive, negative, or invalid and reports a confidence level. The authors report about 99.1% test accuracy and argue that the cropping step is what improves sensitivity and specificity, because it lets the classifier concentrate on the test and control lines rather than on background noise. They motivate the design as an accessibility aid: because the detector tolerates off-center or edge-of-frame test kits, a visually impaired user does not need perfect camera alignment. SHAP analysis is used to show that the network relies on the visible test and control lines and that the few errors come from external marks, smudges, or faded lines.","feed_headline":"AI crop step lifts rapid-test reading to 99.1% accuracy","feed_subtitle":"Cropping first lets off-center phone photos of rapid tests be read automatically, even by visually impaired users.","key_machinery":"The load-bearing mechanism is the two-stage extraction-then-classification pipeline. YOLOv8, a \"You Only Look Once\" object detector, is trained to place a bounding box around the membrane region, and that crop becomes the sole input to the CNN classifier. The paper's causal story is that cropping removes background clutter, brand text, and lighting artifacts, so the CNN can base its decision on the presence, absence, and intensity of the test and control lines. The second mechanism is SHAP (SHapley Additive exPlanations), which attributes each prediction to input pixels and is used to check that the model keys on the biological lines rather than on incidental image features.","core_discovery":"The central claim is that automatically extracting the membrane zone before classification materially improves rapid-test interpretation. The authors built a two-stage pipeline: YOLOv8 is trained on manually annotated kit images to detect the membrane, crop it, convert it to grayscale, and resize it; a five-convolutional-block CNN then classifies the crop. On a test set drawn from a previously published image collection plus images captured by the authors, the CNN reached roughly 99.1% accuracy, with per-class F1-scores of 0.985, 0.993, and 0.992 for invalid, negative, and positive, and macro precision and recall near 0.985 and 0.995. The authors attribute the high performance specifically to the YOLO membrane extraction, which they say separates genuine test lines from background noise. They further report that SHAP heatmaps place decision weight on the test and control lines, and that the four observed test-set misclassifications trace to external marks, smudges, or faint lines. A companion Android app uploads the photo to a server and returns a classification in roughly 11 seconds.","pith_inferences":["We infer that the claimed causal role of YOLO cropping has not actually been isolated because the paper reports no direct comparison between the CNN on cropped membranes and the same CNN on full uncropped kit photos, so the size of the cropping benefit remains untested.","We infer that the 99.1% figure is dataset-specific: the test set mixes a previously published image collection with the authors' own photos, and accuracy on unseen brands, lighting conditions, or user-captured images is not established.","We infer that the accessibility benefit for visually impaired users is a design inference rather than a measured outcome, because no user study with visually impaired participants is reported."],"forward_implications":["A user can photograph a rapid test without centering the kit and still receive an automated positive, negative, or invalid reading with a confidence score.","Interpreting faint or ambiguous test lines no longer depends solely on human visual judgment, so results should be more consistent across users.","The same crop-then-classify design can be retrained for other lateral-flow rapid tests, including veterinary and environmental assays, as the authors suggest.","Because the app returns a confidence level, a user or clinician can decide when to repeat or confirm a result.","SHAP heatmaps give developers a way to trace misclassifications to physical causes such as smudges or faded lines."],"supporting_citations":[{"why":"Supplies the previously published rapid-test image dataset, with positive and negative classes, that is reused for training and for test images in the current study.","marker":"[24]"},{"why":"Provides the YOLOv8 object-detection software used to train the membrane detector and to extract the crop regions.","marker":"[35]"},{"why":"Supplies the SHAP method used to attribute classifier decisions to individual pixels in the analysis of correct and incorrect predictions.","marker":"[36]"}],"fun_headline_variants":["AI crop lifts rapid-test accuracy to 99.1%","Crop-first AI reads off-center rapid tests, aids blind","YOLO crop then CNN hits 99.1% on rapid-test kits","Smartphone AI with YOLO crop reads tests for visually impaired"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported 99.1% test accuracy holds up on genuinely new smartphone photos, especially ones taken by visually impaired users, with no overlapping or repeated test-kit images across training and testing and no reliance on dataset-specific cues such as brand, background, or lighting.","fun_headline_variants_meta":{"raw":{"variants":["AI crop lifts rapid-test accuracy to 99.1%","Crop-first AI reads off-center rapid tests, aids blind","YOLO crop then CNN hits 99.1% on rapid-test kits","Smartphone AI with YOLO crop reads tests for visually impaired"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000515,"raw_usage":{"total_tokens":2559,"prompt_tokens":1060,"completion_tokens":1499,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":676,"completion_tokens_details":{"reasoning_tokens":1423}},"tokens_in":676,"tokens_out":1499,"duration_ms":10911,"temperature":1.0,"reasoning_tokens":1423,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:36:22.937382+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Feed the same test-set images to the CNN once with the YOLO membrane crop and once with the full uncropped kit photo; if accuracy without cropping does not fall clearly below 99.1%, the central attribution of the gain to YOLO extraction is not supported. A second check would remove near-duplicate images between training and test sets and re-measure accuracy to see whether the reported figure reflects generalization or memorization.","supporting_citations":[{"cited_title":"Using artificial intelligence to improve COVID -19 rapid diagnostic test result interpretation,","cited_arxiv_id":null,"evidence_quote":"Supplies the previously published rapid-test image dataset, with positive and negative classes, that is reused for training and for test images in the current study."},{"cited_title":"Jocher, A","cited_arxiv_id":null,"evidence_quote":"Provides the YOLOv8 object-detection software used to train the membrane detector and to extract the crop regions."},{"cited_title":"A Unified Approach to Interpreting Model Predictions,","cited_arxiv_id":null,"evidence_quote":"Supplies the SHAP method used to attribute classifier decisions to individual pixels in the analysis of correct and incorrect predictions."}],"review_version":1}