{"id":"b0dea563-c778-4331-9a1a-d3f9827270d7","arxiv_id":"2512.21335","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Monte Carlo dropout uncertainty filtering on a computational vertical-flow Lyme assay improved blinded sensitivity from 88.2% to 95.7% by excluding uncertain samples.","lead":"This paper adds an uncertainty filter to a paper-based Lyme disease test that uses a neural network to read results. The filter flags samples the network is unsure about, and in a blinded test, removing those samples raised reported sensitivity from 88.2% to 95.7%.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline sensitivity gain is per-cartridge, not per-patient, on only 29 patients; patient-level analysis may eliminate the 7.5-point improvement.","rationale":"The reader identified the validity and calibration of the figure of merit F as the weakest assumption. I agree that F is ad hoc and lacks calibration, but that concern is secondary: even a perfectly calibrated uncertainty metric would not support the headline claim if the only evidence is per-cartridge data from 29 patients with no confidence intervals or patient-level analysis. Conversely, if a patient-level reanalysis confirms the improvement, the F-calibration concern could be addressed by adding comparisons to MC variance or other rejection rules. Thus the most load-bearing issue is the unit of analysis and statistical robustness of the 88.2% to 95.7% improvement. The paper's blind-test design is a genuine strength, but the reported gain is fragile because the effective sample size is tiny and clustered. I keep the reader's CONDITIONAL verdict: the method is plausible and worth testing further, but the central claim is not yet established at the level of patient-level diagnostic sensitivity.","tokens_in":12867,"tokens_out":7502,"duration_ms":82799,"concrete_test":"Re-analyze the blind-test dataset at patient level: for each of the 29 patients, derive a single diagnosis from the three cartridges under both a majority-vote and an any-positive rule, then compute sensitivity before and after applying Fth = 8.5 with exact Clopper-Pearson 95% CIs and a cluster bootstrap that resamples patients. Also report the number of distinct patients whose cartridges were excluded and the per-patient FN/TP counts. If the sensitivity gain is not reproducible at patient level or the CIs overlap substantially, the headline claim should be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that MCDO filtering raised blind-test sensitivity from 88.2% to 95.7%. But these percentages are computed on 87 xVFA cartridges from only 29 patients (3 replicates per patient), not on independent patients. The 88.2% is consistent with 45/51 positive cartridges and the 95.7% with 44/46 after excluding 1 true-positive and 4 false-negative cartridges; the denominator changes and the excluded samples come from only a handful of patients. No confidence intervals, no patient-level contingency table, and no cluster-aware analysis are reported, so the 7.5-point gain could be within sampling noise or driven by one or two patients. The concern is aggravated by the Methods statement that the 'validation' set used to choose Fth = 8.5 is the same dataset used to train L0 ('We utilized the same datasets for both training and validation'), making the cutoff label-informed. If patient-level sensitivity (majority vote or any-positive rule per patient) shows no significant improvement, the claim that the method improves diagnostic sensitivity for patients is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents an MCDO-based uncertainty quantification pipeline for a paper-based computational vertical flow assay (xVFA) for Lyme disease diagnosis. A baseline neural-network classifier (L0) is trained on 93 xVFA cartridges from 31 patients; the pipeline then runs N=1000 dropout-perturbed forward passes (10% dropout) and defines a figure of merit F = 1/|B0 - <MC>|, where B0 is the baseline score and <MC> is the average MCDO score. Samples with F below a threshold Fth=8.5 are labeled 'Do not use' and excluded from the reported diagnostic performance. Fth is optimized on the same 93-sample dataset used for training L0. On an independent blind cohort of 87 cartridges from 29 patients, excluding 5 samples (4 false negatives and 1 true positive) increases cartridge-level sensitivity from 88.2% to 95.7%. The authors also show that N=50 yields similar performance and that low dropout rates (<=20%) are preferable. The central claim is that this framework autonomously flags unreliable neural-network predictions without ground-truth labels, improving diagnostic sensitivity.","tokens_in":13206,"tokens_out":5815,"duration_ms":60060,"significance":"If the reported improvement is robust, the work offers a practical, low-overhead mechanism for adding a quality-assurance step to computational POC sensors—an important step toward clinical deployment. The blind-testing design is a genuine strength, as is the demonstration that the inference-time overhead is small (0.66 s for N=50). However, because the threshold Fth is selected using labels from the same samples that trained L0, and because the outcome is reported at the cartridge level for only 29 patients without confidence intervals or a patient-level analysis, the magnitude and clinical meaning of the sensitivity gain are not yet established. The concept is plausible and worth pursuing, but the evidence as presented is insufficient for publication in its current form.","major_comments":[{"comment":"The Methods state: 'We utilized the same datasets for both training and validation due to the limited size of the training dataset.' This means that Fth=8.5 was tuned on the same 93 samples whose labels were used to train L0. The resulting validation-set improvement (81.5% to 89.8%) is therefore label-informed and likely optimistic. The blind test is independent, but the choice of Fth may still be overfit to idiosyncrasies of the training cohort. Please report the blind-test sensitivity for a range of Fth values (e.g., 5–12) to demonstrate that the improvement is not overly sensitive to the exact threshold, and ideally use a nested cross-validation or a separate validation cohort to select Fth.","section":"Methods, 'Baseline Lyme model architecture and training'"},{"comment":"The headline sensitivity values (88.2% and 95.7%) are computed per cartridge, not per patient: 87 cartridges from only 29 patients. Replicates from the same patient are correlated, so treating cartridges as independent inflates the effective sample size. The paper does not report a patient-level contingency table, cluster-aware confidence intervals, or the number of patients contributing the excluded false-negative cartridges. Without this analysis, the 7.5-point improvement could be within sampling noise or driven by a single patient. Please provide patient-level sensitivity (e.g., majority vote or any-positive rule) and cluster-bootstrapped 95% CIs before and after filtering, and state how many patients had at least one cartridge flagged 'Do not use'.","section":"Results, blind testing (Figure 5)"},{"comment":"The uncertainty figure of merit is F = 1/|B0 - <MC>|, and the authors note that 'the gap between these two sets of scores increases as the L0 output approaches the decision threshold of 0.5.' This raises the concern that F is largely a nonlinear transformation of the distance from the decision threshold. If so, a simpler abstention rule based on |B0 - 0.5| (a 'grey zone') might reproduce the same sensitivity gain. Please compare the proposed F-based filtering against such a baseline on the blind dataset—for example, exclude the same number of samples using a threshold on |B0 - 0.5| and report the resulting sensitivity. Without this comparison, the added value of the MCDO computation over the raw model score is not established.","section":"Results, Eq. (1) and related text"},{"comment":"The paper asserts that F is a valid, thresholdable proxy for prediction error, but it provides no calibration analysis and no comparison to alternative uncertainty metrics such as MCDO variance, predictive entropy, or MC interval width. The threshold Fth and the 10% dropout rate are free hyperparameters tuned on the validation set, and the SI shows that performance varies with dropout rate. Please add a calibration-style analysis (e.g., error rate versus F threshold) and a comparison with at least one alternative uncertainty metric, using a threshold selected only from the validation set and evaluated on the blind set. This would justify the specific definition of F and support the generalization of Fth across cohorts.","section":"Results/ Discussion, uncertainty calibration"}],"minor_comments":[{"comment":"Typo: 'disposal rapid diagnostic test' should be 'disposable rapid diagnostic test'.","section":"Introduction, first paragraph"},{"comment":"The claim that N=50 gives 'the same level of performance improvement' is based on a single threshold value and a single sensitivity point. Please report confidence intervals or a small bootstrap analysis for the sensitivity at N=50 to support the claim of comparability.","section":"Results, N=50 comparison"},{"comment":"The reported post-filter sensitivity is conditional on samples passing the reliability check. The paper states that 'Do not use' samples can be sent for follow-up testing, but it does not report the proportion of samples flagged 'Do not use' (the inconclusive rate) or an intention-to-test analysis. Please report the inconclusive rate and discuss how the overall diagnostic sensitivity would change if follow-up testing resolves the excluded samples.","section":"Results/Discussion, 'Do not use' handling"},{"comment":"The supporting information figures S1–S5 are cited but not described in detail in the main text; ensure each supplementary figure is referenced in order and that their key results (especially dropout-rate sensitivity) are integrated into the main text's robustness discussion.","section":"General"},{"comment":"The manuscript does not include a data and code availability statement. Given the role of the FOM definition and the importance of the threshold choice, providing the code and de-identified score distributions would aid reproducibility.","section":"Data and code availability"}],"recommendation":"major_revision","confidential_remarks":"The central idea—using MCDO to flag unreliable predictions in computational POC sensors—is timely and plausible, and the blind-testing element is a step in the right direction. However, the manuscript currently overstates the strength of the evidence: the threshold Fth is tuned on the same data that trained the model, the sensitivity gain is measured per cartridge rather than per patient with no confidence intervals, and the FOM is not compared to a simple distance-to-threshold abstention rule. These are load-bearing issues, not merely presentational. I recommend major revision rather than reject, because the needed analyses (patient-level statistics, threshold-sensitivity analysis, and comparison to a raw-score baseline) are feasible and would directly address the validity of the claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a legitimate first use of MCDO for computational point-of-care sensing, and the blind test is a real step, but the headline gain is computed on cartridges from only 29 patients and could easily be noise. Worth refereeing, but it needs patient-level statistics and more careful validation.\n\nWhat's new and good: The paper applies a known UQ method (MC dropout) to a practical POC platform (xVFA for Lyme) and defines a simple uncertainty figure of merit F = 1/|B0 - <MC>|. That's a first application as far as the references show. The reliability action (\"Trust\"/\"Do not use\") is a useful framing for clinical workflows, and the blind test on independent patient samples is more than many papers in this space do. The authors also test different dropout rates and N values, which gives the reader a sense of robustness.\n\nWhere it's soft: The central empirical claim—sensitivity from 88.2% to 95.7%—is made on 87 cartridges from 29 patients, not on patients. The denominators are 51 and 46, so the improvement is 4 false negatives removed out of 6. If you collapse to patient-level majority vote, the gain may vanish. The paper doesn't report that. Also, the threshold Fth=8.5 was chosen on the same dataset used to train L0 (Methods literally says 'We utilized the same datasets for both training and validation'), so it's partly label-informed; the blind test is independent, which saves the core claim, but there are no confidence intervals or cluster-aware tests anywhere. The FOM itself is a heuristic: reciprocal of the distance between baseline and MC mean. There's no calibration analysis or comparison to simply thresholding on the baseline score or MC variance. So it's not established that F adds value over cheaper alternatives.\n\nThe stress-test note about per-cartridge vs per-patient is fair and should be addressed. The authors should report patient-level sensitivity with appropriate denominators, add confidence intervals or at least a permutation test, and compare F against simpler rejection rules. None of this kills the paper; it's a proof-of-principle, and the blind test is real evidence. But the abstract overstates the result as diagnostic sensitivity when it's per-cartridge.\n\nWho this is for: people working on computational POC diagnostics or UQ for clinical ML. A serious referee should engage with this, mainly to push for honest statistics. I would accept it for peer review despite my reservations.","headline":"First MCDO application to a computational POC sensor, with a real blind test, but the headline sensitivity gain is per-cartridge on 29 patients and needs patient-level verification.","tokens_in":13612,"tokens_out":3263,"would_cite":true,"duration_ms":32732,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A Monte Carlo dropout uncertainty check lets a point-of-care Lyme test flag its own unreliable predictions, lifting blinded sensitivity from 88.2% to 95.7%.","keywords":["uncertainty quantification","Monte Carlo dropout","point-of-care diagnostics","vertical flow assay","Lyme disease","neural networks","diagnostic sensitivity","computational sensing"],"falsifier":"On a new blinded cohort, compute F for every sample and also compute the baseline score alone; if excluding low-F samples removes no more errors than excluding samples with baseline scores nearest 0.5 (or than random exclusion with the same exclusion rate), the claim that F carries independent uncertainty information is falsified. Alternatively, if the optimal Fth on a second validation cohort differs widely from 8.5, the single-threshold generalization fails.","tokens_in":12798,"feed_emoji":"🩺","tokens_out":3857,"duration_ms":32018,"temperature":0.7,"pith_summary":"The paper argues that neural-network-driven point-of-care sensors can police their own diagnostic outputs: by running each patient sample through many dropout-perturbed copies of the trained network and measuring how far the baseline prediction sits from the averaged perturbed predictions, the sensor can label its own prediction as trustworthy or not, with no need for ground-truth labels. On a paper-based vertical flow assay for Lyme disease, this autonomous reliability check improved diagnostic sensitivity on blinded patient samples from 88.2% to 95.7% by excluding high-uncertainty predictions and directing those samples to retesting. The central quantity is the figure of merit F = 1/|B0 - <MC>|, the reciprocal of the gap between the baseline score and the mean dropout score; low F flags unreliable predictions. If the approach generalizes, it gives computational POC sensors a built-in quality-assurance step that can reduce false negatives without changing the assay chemistry.","feed_headline":"Uncertainty check lifts Lyme test sensitivity to 96%","feed_subtitle":"Monte Carlo dropout lets a point-of-care sensor flag unreliable predictions on its own, no ground-truth labels needed.","key_machinery":"The central object is the uncertainty figure of merit F = 1/|B0 − <MC>|, where B0 is the baseline model's sigmoid output for a sample and <MC> is the mean output of N Monte Carlo dropout models (same architecture, random dropout masks at 10% during inference). The reciprocal gap encodes predictive uncertainty: a small gap means the baseline prediction is stable under dropout perturbations and is trusted; a large gap (low F) means the prediction is unstable and is flagged 'Do not use'. The threshold Fth is a single scalar tuned on the validation set; the method requires only the trained network and its dropout layers, no ground-truth labels at inference.","core_discovery":"On the paper's own terms, the discovery is that the divergence between a baseline neural network's prediction score and the average prediction score of Monte Carlo dropout-perturbed versions of the same network is a usable, label-free proxy for diagnostic error in a computational point-of-care sensor. Defined as F = 1/|B0 - <MC>|, this figure of merit, with a single threshold (Fth = 8.5) tuned on a validation cohort, flagged a subset of predictions as unreliable. Excluding those flagged samples improved the sensitivity of the Lyme xVFA platform from 81.5% to 89.8% on validation data and, on an independent blinded cohort, from 88.2% to 95.7%, while also raising overall accuracy. The authors p","pith_inferences":["A direct test of the FOM's validity would compare F against simpler alternatives (e.g., MC variance, entropy, or the baseline score's distance to the 0.5 threshold) on the same cohorts; the paper does not report such a comparison.","Because Fth is tuned on one validation set and applied to one blinded set, its portability to other populations, biobanks, or assay batches is an open question; the authors note that multi-center training data would be needed to set a unified threshold.","The sensitivity gain is achieved by excluding samples, not by reclassifying them; in a deployment, the excluded samples must be retested or referred, so the realized benefit depends on the follow-up pathway's own sensitivity and coverage.","The paper states that the same datasets were used for both training and validation due to limited sample size, which is a limitation worth monitoring; the FOM's calibration on the validation set could be optimistic for the blind test."],"forward_implications":["Samples flagged as 'Do not use' can be routed to repeat testing or gold-standard laboratory tests, reducing the clinical impact of false negatives.","The same MCDO-based quality-assurance step could be attached to other computational POC sensors (lateral flow, fluorescence, electrochemical) that use neural network inference.","Using as few as N=50 dropout models (0.66 s for 87 samples) matches the performance of N=1000, making the step practical for small, decentralized testing runs.","Low dropout rates (≤20%) during inference are preferable; larger rates destabilize the score distributions and degrade the filtering.","The approach adds a transparency layer for clinicians and regulators: each result carries an explicit reliability action, supporting trust in black-box diagnostic models."],"fun_headline_variants":["Autonomous doubt detector lifts Lyme test to 96% sensitivity","Label-free uncertainty boosts POC sensor sensitivity by 7.5 points","Monte Carlo dropout pinpoints bad predictions, raises reliability","Self-checking neural net improves point-of-care diagnostics","Point-of-care sensor learns to flag its own unreliable reads"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the reciprocal gap between the baseline score and the average dropout-perturbed score, F = 1/|B0 - <MC>|, is a better indicator of a wrong prediction than the baseline score itself, and that a single threshold on F generalizes across patient cohorts.","fun_headline_variants_meta":{"raw":{"variants":["Autonomous doubt detector lifts Lyme test to 96% sensitivity","Label-free uncertainty boosts POC sensor sensitivity by 7.5 points","Monte Carlo dropout pinpoints bad predictions, raises reliability","Self-checking neural net improves point-of-care diagnostics","Point-of-care sensor learns to flag its own unreliable reads"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00112,"raw_usage":{"total_tokens":4536,"prompt_tokens":819,"completion_tokens":3717,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":3633}},"tokens_in":563,"tokens_out":3717,"duration_ms":27441,"temperature":1.0,"reasoning_tokens":3633,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T14:07:20.132666+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a new blinded cohort, compute F for every sample and also compute the baseline score alone; if excluding low-F samples removes no more errors than excluding samples with baseline scores nearest 0.5 (or than random exclusion with the same exclusion rate), the claim that F carries independent uncertainty information is falsified. Alternatively, if the optimal Fth on a second validation cohort differs widely from 8.5, the single-threshold generalization fails.","supporting_citations":[],"review_version":1}