{"id":"013c046a-e86f-4c9b-95d2-997c964de01c","arxiv_id":"2501.01117","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A DNDF pipeline with feature selection, Bayesian tuning, SMOTE, and threshold optimization reports near-perfect AUCs on individual cough datasets but transfers poorly across datasets.","lead":"This paper tests deep neural decision trees and forests for detecting COVID-19 from cough sounds across five public datasets, reporting AUCs up to 0.99. Its cross-dataset experiments show much weaker transfer, which limits the robustness claim.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The evaluation protocol fits features, hyperparameters, and thresholds on the full dataset before 10-fold CV, so the reported AUCs are optimistically biased; the paper's own cross-dataset results are near chance except when Virufy samples leak across train/test.","rationale":"The reader's weakest_assumption identifies the same load-bearing flaw: the evaluation treats 10-fold CV as unbiased even though feature selection, hyperparameter tuning, and threshold selection all use the full dataset. This is indeed the most decisive issue because it directly undermines the only evidence for the headline claim. The intradataset AUCs are the claimed support for 'consistently outperforming state-of-the-art methods,' yet they are produced by a protocol that leaks test-fold information into model selection. The cross-dataset results, which avoid within-dataset selection bias but introduce dataset-shift confounds, are mostly near chance, and the few high values are explained by Virufy samples being present on both sides of the train/test split. The paper's own Section 5.4 acknowledges poor transfer and cites prior work on dataset bias, which further undercuts the claim of robustness. The negative cross-dataset finding is a useful empirical contribution, and the paper deserves credit for reporting it, but it contradicts rather than supports the stated central claim. A redesigned evaluation with nested CV, participant-level splits, and a fixed external test set would be needed before the headline could be accepted. Since the reader already reached REJECT and our analysis confirms that judgment, no verdict adjustment is needed.","tokens_in":33555,"tokens_out":4494,"duration_ms":41129,"concrete_test":"Re-run strategy 5 for each dataset with all model-selection steps nested inside the training folds: perform RFECV, Bayesian optimization, and threshold selection on the training portion of each fold only, and use participant-level grouping so samples from the same Virufy patient never span train and test (for Virufy and Virufy+NoCoCoDa). Compare the held-out AUC with Table 7. If the AUCs drop below roughly 0.8 (or near chance) for datasets that reported 0.97-0.99, the headline claim fails; if they remain high, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of consistent superiority rests on the intradataset AUCs in Tables 6-7 (0.97, 0.98, 0.92, 0.93, 0.99, 0.99). But the protocol in Sections 4.3-4.7 applies RFECV feature selection, Bayesian hyperparameter optimization, and ROC-AUC threshold moving once to the entire dataset before forming the 10 folds. This is selection on the full data: the test fold's labels influence which features, hyperparameters, and thresholds are chosen, so the cross-validated AUC is not an unbiased estimate of generalization. No nested cross-validation or independent hold-out set is described. The paper's own external evidence confirms the bias: Section 5.4 cross-dataset AUCs are near chance (0.51-0.73) for every non-overlapping pair, and the only high values (0.85, 0.87) occur when Virufy samples appear in both training and test sets (Virufy vs. Virufy+NoCoCoDa), which is leakage rather than transfer. The paper acknowledges the poor cross-dataset performance but still claims 'robust' and 'consistently outperforming.' Thus the high intradataset AUCs likely reflect selection bias and dataset correlations, not a reusable cough-COVID signature.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a cough-sound-based COVID-19 detection pipeline built on deep neural decision trees (DNDT) and deep neural decision forests (DNDF), combined with RFECV feature selection, Bayesian hyperparameter optimization, SMOTE, and ROC-AUC-based threshold moving. The authors evaluate the method on Cambridge asymptomatic/symptomatic, Coswara, COUGHVID, Virufy, and Virufy merged with NoCoCoDa, report intradataset AUCs of 0.92–0.99, perform a cross-dataset transfer study, and report an AUC of 0.97 on a combined dataset. The central claim is that the proposed pipeline consistently outperforms state-of-the-art methods and is robust across diverse cough datasets.","tokens_in":33798,"tokens_out":5374,"duration_ms":54368,"significance":"If the reported results were valid, the paper would provide a useful datapoint for cough-based COVID-19 screening and for comparing tree-based learned decision models with deep classifiers on audio features. The manuscript has several commendable features: it covers five widely used cough datasets, reports a full cross-dataset matrix, evaluates five incremental training strategies, and includes a broad comparative table of prior work. However, the empirical contribution is not machine-checkable (no code or data release is described), and the central evaluation protocol is invalid as reported. The claimed consistent superiority over prior work and the claimed robustness are not supported by the evidence actually presented in Tables 8 and 9. The cross-dataset results, which the paper itself acknowledges to be poor, are more consistent with dataset-specific shortcuts than with a reusable cough-COVID signature.","major_comments":[{"comment":"The evaluation protocol does not support the reported generalization performance. Section 4.3 applies RFECV feature selection to the full dataset to select feature subsets (71, 182, 33, 172, 46, and 188 features for the six datasets), Section 4.4 uses Bayesian optimization on the full extracted features and labels, and Section 4.6 selects the decision threshold by maximizing ROC-AUC over the full dataset. All of these steps are performed before the 10-fold stratified cross-validation described in Section 4.7. Consequently, the test folds contribute to feature choice, hyperparameter choice, and threshold choice, so the resulting AUCs are optimistically biased and are not unbiased estimates of performance on new data. The paper needs either a nested cross-validation scheme in which all model selection is performed inside each training fold, or a single independent test set withheld before any selection or tuning. The absence of any error bars or variance estimates further weakens the headline claims in Tables 6 and 7.","section":"§4.3–§4.7, Tables 6–7"},{"comment":"The cross-dataset results contradict the paper's claim of robustness. Most non-overlapping training/testing pairs in Table 9 yield AUCs between 0.51 and 0.73; for example, Cambridge (Asymptomatic) trained on COUGHVID gives 0.51, Coswara trained on Cambridge (Symptomatic) gives 0.52, and COUGHVID trained on Cambridge (Asymptomatic) gives 0.54. The only high cross-dataset values are Virufy → Virufy+NoCoCoDa (0.85) and Virufy+NoCoCoDa → Virufy (0.87), but Table 2 shows that Virufy+NoCoCoDa is constructed by adding NoCoCoDa samples to the Virufy samples, so these two datasets share their Virufy instances and the high values reflect sample overlap rather than transfer. The paper's own text in Section 5.4 acknowledges the poor cross-dataset performance, and the combined-dataset result in Section 5.5 does not remedy this because it is evaluated with the same flawed protocol and mixes all datasets into one training pool.","section":"§5.4, Table 9, Table 2"},{"comment":"The claim that the proposed method is 'consistently outperforming state-of-the-art methods' is not supported by Table 8. On Coswara, the proposed DNDT achieves AUC 0.84, below Zhang et al.'s 0.86. On COUGHVID, the proposed DNDT achieves 0.81, below Hamdi et al.'s 0.91 and Skander et al.'s 0.91; the proposed DNDF's 0.93 does not exceed those values. On Cambridge Asymptomatic, DNDF's 0.97 ties Aytekin et al., and DNDT's 0.95 is lower; on Cambridge Symptomatic, DNDF's 0.98 ties Aytekin et al.; on Virufy+NoCoCoDa, both proposed methods tie Melek's AUC of 0.99. Thus the comparison table shows at best parity on several datasets and outright inferiority on others, so the abstract's and conclusion's claims of consistent superiority are overstated.","section":"§5.3, Table 8"},{"comment":"The use of sample-level 10-fold stratified cross-validation is not justified for datasets with multiple recordings from the same subject. Section 4.1.4 states that Virufy contains 121 cough samples from 16 patients, and Section 4.1.5 states that NoCoCoDa contains 73 cough sounds from 10 participants. When recordings from the same patient can appear in both training and test folds, the model can exploit patient-specific recording characteristics rather than a general COVID-19 cough signature. This is especially relevant for the near-perfect AUCs of 0.99 reported for Virufy and Virufy+NoCoCoDa in Table 6. The authors should either perform subject-level splitting, where all samples from one participant are kept in the same fold, or explicitly justify why sample-level independence is appropriate for these data.","section":"§4.1.4–§4.1.5, §5.2"}],"minor_comments":[{"comment":"There are repeated typographical errors, including 'di fferences' in the abstract and 'V olume' in the references, which should be corrected.","section":"Abstract and body text"},{"comment":"The stated sampling rate is inconsistent: Section 4.1 says resampling at 22.5 kHz, while Section 4.2 says the acoustic signal is captured at 22 kHz. The authors should state which value was actually used.","section":"§4.1 and §4.2"},{"comment":"The caption and text for Figure 5(d) describe a percentage as '92.65' without the percent sign; this should be fixed.","section":"Figure 5"},{"comment":"The Virufy+NoCoCoDa row counts (121 COVID-19, 73 non-COVID-19) make the overlap with Virufy explicit, but the table would be clearer if it noted that the 121 COVID-19 samples are exactly the union of Virufy's 48 and NoCoCoDa's 73 positives.","section":"Table 2"},{"comment":"The sentence 'We examine the DNDF classifier in this context' is vague; the paper should state explicitly why only DNDF, and not DNDT, is used for the cross-dataset study.","section":"§5.4"}],"recommendation":"reject","confidential_remarks":"The methodological flaw is pervasive: feature selection, hyperparameter tuning, and threshold selection are all performed on the full evaluation data, so the headline AUCs cannot be interpreted as estimates of generalization. The paper's own cross-dataset results, which are mostly near chance, further undermine the central claim, and the comparison table does not show consistent superiority over prior work. In my view these are load-bearing issues that invalidate the main empirical claims as presented. If the authors were to completely re-run the evaluation with nested cross-validation, subject-level splits, and no threshold optimization on test data, the corrected results could form the basis of a substantially revised submission, but the current manuscript should not be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely useful thing here is Table 9: a systematic cross-dataset evaluation across five cough datasets showing that non-overlapping training and testing sets give near-chance AUCs (0.51–0.73). That negative result aligns with the known critiques of COVID-19 audio classifiers (Coppock et al., Akman et al.) and is a real empirical contribution. The paper also deserves credit for stating that poor transfer is due to recording conditions and demographic differences, and for citing the bias literature rather than ignoring it.\n\nThe problem is the intradataset evaluation. RFECV feature selection, Bayesian hyperparameter optimization, and ROC-AUC threshold moving are all applied once to the entire dataset before forming the 10 folds. That means the test fold's labels influence which features, hyperparameters, and thresholds are chosen, so the reported AUCs of 0.92–0.99 are not unbiased estimates of generalization. No nested CV, no held-out set, no error bars. The paper's own cross-dataset results confirm the risk: every non-overlapping pair is near chance, and the only high transfer numbers (0.85, 0.87) come when Virufy samples appear in both training and test sets. The text explicitly acknowledges the Virufy overlap, so this is leakage, not evidence of transfer.\n\nA few smaller issues: the comparison with prior work is limited to three studies, which narrows the claimed 'consistent outperformance'; the abstract and Table 10 disagree slightly on the combined-dataset specificity (not stated in abstract, but the AUC matches); and the precision values in the abstract (1, 1, 0.72, 0.93, 1, 1) are perfectly explained by the confusion matrices but would still benefit from confidence intervals, especially on the small Virufy dataset (121 samples from 16 patients).\n\nIn short: the negative cross-dataset finding is worth publishing, but the headline claim of robust, consistent superiority is not supported by the current protocol. The authors would need to redo the evaluation with nested CV, participant-level splits, and no overlapping subsets before those AUCs can be taken seriously. That is a heavy revision, but the negative result itself is a useful caution for the field.\n\nI would send this to peer review: a good referee can separate the flawed intradataset protocol from the valid cross-dataset message. I would not desk-reject it, but I would insist on the redesign. The paper is clearly written and the authors are honest about the transfer failures, so it is a serious piece of work despite the methodological flaw.","headline":"The cross-dataset matrix is a useful negative result, but the headline AUCs are inflated by fitting features, hyperparameters, and thresholds on the full data before CV.","tokens_in":34402,"tokens_out":1514,"would_cite":false,"duration_ms":16681,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims a deep neural decision forest with selected spectral features and tuned thresholds separates COVID-19 coughs at AUC 0.97-0.99 per dataset and 0.97 pooled, while cross-dataset transfer drops sharply.","keywords":["COVID-19 detection","cough sound analysis","deep neural decision forest","deep neural decision tree","cross-datasets evaluation","RFECV feature selection","Bayesian optimization","audio classification"],"falsifier":"Run a patient-level grouped cross-validation on the Virufy dataset, holding out entire patients rather than individual cough samples, and move feature selection and threshold selection inside the training folds; if the reported 0.99 AUC drops substantially, the claimed generalization depends on sample-level leakage.","tokens_in":33258,"feed_emoji":"🔊","tokens_out":8080,"duration_ms":70699,"temperature":0.7,"pith_summary":"The paper tries to establish that a cough recording alone can flag COVID-19 with high accuracy when classified by a differentiable decision forest, provided the audio is summarized by five spectral feature families and the model is tuned with the right feature subset, hyperparameters, class rebalancing, and decision threshold. It reports AUCs of 0.97, 0.98, 0.92, 0.93, 0.99, and 0.99 on Cambridge asymptomatic, Cambridge symptomatic, Coswara, COUGHVID, Virufy, and Virufy combined with NoCoCoDa, and 0.97 when all datasets are pooled. It also shows that models trained on one dataset and tested on another perform much worse, which it reads as demographic and geographic variation in cough acoustics and as evidence that pooled training helps generalization. If true, the work supports low-cost, app-based cough screening for COVID-19, not a standalone diagnosis.","feed_headline":"Cough-sound AI hits 0.97 AUC on combined COVID-19 data","feed_subtitle":"A five-stage pipeline claims cough-only COVID-19 screening at AUC up to 0.99 per dataset.","key_machinery":"The central object is the deep neural decision forest (DNDF), an ensemble of differentiable decision trees whose split decisions are sigmoid functions of learned features; the paper averages each tree's class distribution for the final prediction. Around this sits a five-stage pipeline: extraction of 193 spectral features (40 MFCCs, 128 mel-scaled spectrogram values, 6 tonal centroid values, 12 chromagram values, and 7 spectral contrast values), RFECV with an Extra-Trees estimator to select a dataset-specific feature subset, Bayesian optimization for hyperparameters, SMOTE to rebalance positive and negative coughs, and threshold moving over 0.1 to 1.0 to maximize ROC-AUC. The paper's evidence that this machinery matters is its strategy ladder, where the same classifiers climb from near-chance AUC with raw defaults to the reported highs as each component is added.","core_discovery":"The paper's central claim is that a specific training recipe, DNDF with RFECV feature selection, Bayesian optimization, SMOTE, and threshold moving, beats published cough-based COVID-19 classifiers on each of six dataset splits. The headline numbers are AUCs of 0.97 on Cambridge asymptomatic, 0.98 on Cambridge symptomatic, 0.92 on Coswara, 0.93 on COUGHVID, 0.99 on Virufy, and 0.99 on Virufy merged with NoCoCoDa, with precision scores of 1, 1, 0.72, 0.93, 1, and 1, respectively. On the pooled dataset of 3,398 coughs, the forest variant reaches accuracy 0.97, AUC 0.97, precision 0.95, recall 0.96, F1-score 0.96, and specificity 0.97. The paper also reports a cross-dataset study in which the same method is trained on one dataset and tested on the others; those numbers are mostly far lower, which the authors interpret as demographic and geographic variation in cough acoustics and as evidence that dataset integration improves generalizability. The claim is about the achieved metrics under the stated protocol, not about deployment in a clinic.","pith_inferences":["Beyond the paper: the near-perfect scores on Virufy and Virufy plus NoCoCoDa may be inflated by patient-level leakage, since Virufy holds 121 samples from only 16 patients and the same Virufy recordings also appear in the merged set.","Beyond the paper: the paper's own cross-dataset table is evidence that much of what the model learns is cohort-specific, so a deployment study should measure calibration on the target population before trusting the reported AUC.","Beyond the paper: a perturbation test, such as resampling, adding noise, or changing microphone type, would show whether a combined AUC of 0.97 reflects an acoustic disease signature or a dataset fingerprint."],"forward_implications":["Each element of strategy 5 contributes: threshold moving alone lifts AUC above the raw classifier, and adding RFECV, Bayesian optimization, and SMOTE improves it further on nearly every dataset.","The forest version beats the single tree on every dataset, which the authors attribute to the ensemble reducing prediction variance.","Pooling all five datasets into one training set yields the best overall balance (accuracy, AUC, precision, recall, F1, and specificity all around 0.95-0.97), supporting dataset integration.","The cross-dataset results, with several AUCs near 0.5, indicate that a model trained on one cohort cannot be assumed to transfer to another cohort; matched demographics or recording conditions matter.","Taken at face value, the results support cough-based triage or screening in settings where the test population resembles the training population, not a standalone diagnostic."],"supporting_citations":[{"why":"defines the deep neural decision tree and forest classifiers at the core of the pipeline.","marker":"[122]"},{"why":"supplies the Cambridge asymptomatic and symptomatic cough datasets used for training and comparison.","marker":"[9]"},{"why":"supplies the Coswara cough dataset used in the evaluation.","marker":"[13]"},{"why":"supplies the COUGHVID crowdsourced cough dataset used in the evaluation.","marker":"[11]"},{"why":"supplies the Virufy cough dataset used in the evaluation and in the merged set.","marker":"[10]"},{"why":"supplies the NoCoCoDa cough dataset, merged with Virufy for one evaluation split.","marker":"[14]"},{"why":"provides the SMOTE oversampling technique used in strategy 5 to handle class imbalance.","marker":"[12]"},{"why":"provides the recursive feature elimination with cross-validation rationale for feature selection.","marker":"[115]"},{"why":"provides the rationale that Bayesian optimization finds hyperparameters more efficiently than grid or random search.","marker":"[117]"},{"why":"serves as the closest prior multi-dataset baseline that the proposed method claims to outperform.","marker":"[109]"}],"fun_headline_variants":["Cough AI hits 0.97 AUC on combined COVID data","Deep forest cough test scores 0.97 AUC","Cough-based COVID detection: AUC 0.97 on pooled data","Deep neural forest beats cough-based COVID classifiers","Cough clips spot COVID with 0.97 AUC"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that 10-fold stratified cross-validation, with feature selection, hyperparameter tuning, and threshold selection performed on the same folds and with cough samples from the same patient allowed in different folds, gives an unbiased estimate of how the model would perform on new people.","fun_headline_variants_meta":{"raw":{"variants":["Cough AI hits 0.97 AUC on combined COVID data","Deep forest cough test scores 0.97 AUC","Cough-based COVID detection: AUC 0.97 on pooled data","Deep neural forest beats cough-based COVID classifiers","Cough clips spot COVID with 0.97 AUC"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00024,"raw_usage":{"total_tokens":1608,"prompt_tokens":1124,"completion_tokens":484,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":740,"completion_tokens_details":{"reasoning_tokens":402}},"tokens_in":740,"tokens_out":484,"duration_ms":4805,"temperature":1.0,"reasoning_tokens":402,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:34:44.406342+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a patient-level grouped cross-validation on the Virufy dataset, holding out entire patients rather than individual cough samples, and move feature selection and threshold selection inside the training folds; if the reported 0.99 AUC drops substantially, the claimed generalization depends on sample-level leakage.","supporting_citations":[{"cited_title":"Kontschieder, M","cited_arxiv_id":null,"evidence_quote":"defines the deep neural decision tree and forest classifiers at the core of the pipeline."},{"cited_title":"Misra, A","cited_arxiv_id":null,"evidence_quote":"provides the recursive feature elimination with cross-validation rationale for feature selection."},{"cited_title":"Eggensperger, M","cited_arxiv_id":null,"evidence_quote":"provides the rationale that Bayesian optimization finds hyperparameters more efficiently than grid or random search."},{"cited_title":"Chowdhury, A","cited_arxiv_id":null,"evidence_quote":"serves as the closest prior multi-dataset baseline that the proposed method claims to outperform."}],"review_version":1}