{"id":"ac6f3826-7349-43a7-b203-501fb065cf66","arxiv_id":"2505.08242","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"The paper benchmarks CNNs and vision transformers for congenital heart disease classification on the ZCHSound and DICOM datasets, achieving 73.9% and 80.72% accuracy respectively.","lead":"This paper tests whether standard deep learning models can detect congenital heart disease from heart-sound spectrograms and chest X-rays. It reports 73.9% accuracy on the audio dataset and 80.7% on the X-ray set, though the results are below prior published numbers.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DICOM headline 80.72% is a validation accuracy with early stopping and model selection on the same split; no independent test set or repeated runs is reported, so the central accuracy claim is not a reliable generalization estimate.","rationale":"The reader's weakest assumption concerned split randomness, stratification, and leakage. I agree partially, but the sharper issue is that the DICOM result is explicitly a validation accuracy, not a test accuracy. Even a perfectly random 80/20 split does not support the headline number when early stopping and model selection are performed on that same validation split and then reported as the result. This directly undermines the strongest claim: 80.72% is not demonstrated as a generalization estimate, and the small gaps between models could easily be selection noise. I considered other concerns—the novelty contradiction with reference [11], the uncontrolled 6-7% preprocessing gain claim, and the absence of code—but those primarily affect framing and contribution rather than the validity of the accuracy claim. The validation-as-test issue is the single most load-bearing problem because, if it lands, the central quantitative claim is not established. My recommended verdict remains CONDITIONAL, matching the reader, because the issue is addressable with a proper held-out evaluation; I see no evidence of misrepresentation beyond the paper's own inconsistent labels.","tokens_in":6974,"tokens_out":5798,"duration_ms":59630,"concrete_test":"Hold out a stratified 20% test split from DICOM, use the remaining 80% with an internal validation split for early stopping and model selection, train ResNet-18 with the same hyperparameters, and report accuracy/F1 on the held-out test split. If the held-out accuracy is materially below 80.72% (or within noise of Swin), the headline number is an artifact of validation-based selection. For ZCHSound, repeat with the requested split description and patient-exclusive grouping.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV.B states that DICOM models were trained with an '80/20 train/validation split' and that early stopping 'monitored the validation accuracy to halt training.' Table Ia labels the reported column 'Val acc.' for each model, not test accuracy. Figure 6, however, presents the same numbers as 'Test Accuracy Results.' There is no description of a held-out test set anywhere in the paper. On this single 20% validation split (about 166 images from 828), the authors compared four architectures, selected ResNet-18 as best, and used early stopping on validation accuracy. Reporting the validation accuracy after model selection and early stopping on that same data is an optimistic, selection-biased estimate; the gap between ResNet-18 (80.72%) and Swin-Tiny (80.12%) is within the range that could be entirely due to this bias on a small imbalanced dataset. For ZCHSound, no train/validation/test split is specified at all; Section V.A refers to 'test accuracy' but does not say how the test set was formed or whether patients were separated. Because no error bars or repeated runs are reported, the abstract's 73.9% and 80.72% figures do not currently establish the claimed classification performance.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript investigates deep learning and transformer models for congenital heart disease (CHD) classification using two public datasets: the ZCHSound heart-sound recordings and the DICOM/CHD-CXR chest X-ray images. For audio, the authors compare STFT, Mel-spectrogram, and Gramian Angular Field representations with ResNet-50V2, EfficientNetB0, and InceptionV3, and then combine base-model predictions via three late-fusion strategies (validation-accuracy weights, class-F1 weights, and a logistic-regression meta-ensemble). For X-rays, they compare ResNet-18, Inception-v3, ViT-Tiny, and Swin-Tiny after Gaussian blurring, contrast enhancement, and augmentation. The paper reports 73.9% accuracy on ZCHSound and 80.72% accuracy on DICOM, and claims that applying state-of-the-art vision transformers to the X-ray task is a novelty.","tokens_in":7178,"tokens_out":7369,"duration_ms":66456,"significance":"If the reported numbers were reliable, this would be a modest empirical contribution: it provides a head-to-head comparison of standard CNNs and transformers on two CHD datasets, shows that late fusion can provide small gains over the best audio base model, and highlights that a lightweight ResNet-18 outperforms heavier transformer models on the small X-ray dataset. The paper's emphasis on model complexity and deployability is a useful framing, and the authors are transparent about several limitations (e.g., choosing preprocessing parameters by assumption rather than tuning). However, the lack of a held-out test set for the X-ray experiments, the absence of any split description for the audio experiments, and the inconsistent reporting of validation vs. test accuracy mean that the central accuracy claims are not currently supported as generalization estimates. The paper also provides no code or reproducibility statement, which further limits the value of the empirical comparisons.","major_comments":[{"comment":"The DICOM accuracy headline (80.72%) is a validation accuracy, not a test accuracy. Section IV.B states that models were trained with an '80/20 train/validation split' and that early stopping 'monitored the validation accuracy to halt training,' and Table Ia labels the reported column 'Val acc.' Yet Figure 6 presents the same numbers as 'Test Accuracy Results.' Because the same 20% validation split is used for early stopping and for selecting ResNet-18 as the best model, the reported figure is an optimistically biased estimate with no independent test set. The authors must either introduce a proper three-way split (or nested cross-validation) and report test accuracy with confidence intervals, or explicitly re-label all figures/tables as validation accuracy and refrain from generalizing.","section":"§IV.B, Table Ia, Figure 6"},{"comment":"No train/validation/test split is described for the ZCHSound audio dataset. The text repeatedly refers to 'testing accuracy' and Figure 5 is titled 'Test Accuracy Results,' but Section IV.A never specifies how the 941 recordings were partitioned, whether stratification by class was performed, or whether any patient-level separation was enforced. Without this information, the 73.9% base-model accuracy and the 75.5%/76.9% late-fusion accuracies cannot be reproduced or assessed for leakage/selection bias. The authors should specify the split (ideally with repeated stratified k-fold or bootstrapped confidence intervals) and report per-class performance.","section":"§V.A, Table Ib, Figure 5"},{"comment":"The abstract reports 73.9% accuracy on ZCHSound, which corresponds to the STFT-ResNet base model in Table Ib, while the paper's own best late-fusion result (meta-ensemble, Figure 5) is 76.9%. This inconsistency in the headline number is confusing: if the abstract is meant to report the best achieved result, it should cite the meta-ensemble accuracy; if it intentionally reports the base model, the choice should be justified. As written, the abstract understates the paper's main audio result and does not reflect the fusion contribution described in the conclusions.","section":"Abstract vs. §V.A"},{"comment":"The claimed 6-7% accuracy gain from Gaussian blurring and contrast enhancement is not substantiated by any controlled comparison. Section IV.B states that the authors 'gained around 2-3% increase in accuracy' from GBF and that combined preprocessing 'was 6-7%,' but no ablation table or figure is provided showing model performance without these steps. Because this quantitative claim is reported as a finding in the conclusion, it needs a supporting experiment (e.g., training the same architectures with and without each preprocessing step).","section":"§IV.B, §V.B"}],"minor_comments":[{"comment":"The heading 'Classifiication models for DICOM CHD-CXR images dataset' contains a typo ('Classifiication') and should read 'Classification models for the DICOM CHD-CXR image dataset.'","section":"§IV.B heading"},{"comment":"The bar charts in Figures 5 and 6 do not show numerical value labels on the bars, making it difficult to read exact accuracies from the figures; adding value labels would improve clarity. The 'Original Paper' bar in Figure 5 is also not identified in the caption.","section":"Figures 5 and 6"},{"comment":"The composite Table I places the caption 'TABLE I: Model results' after Table Ib; the (a) and (b) sub-tables are referenced in the text but the caption does not explain the split. Please use separate captions or add a parenthetical explanation.","section":"Table I"},{"comment":"Reference [1] is cited for the claim that CHD accounts for 33% of major birth defects, but [1] is a study on long-term outcomes after myocardial infarction in CHD patients, which does not support that statistic; please replace it with an appropriate epidemiology reference.","section":"Introduction, Reference [1]"},{"comment":"The sentence 'ASD is has only 194 examples' contains a grammatical error, and the overfitting statement 'there was a little of overfit (around 20% difference)' is not quantified with train/validation accuracy values; please provide specific numbers or a learning curve.","section":"§V.B"},{"comment":"The claim that 'there is no papers that applied SOTA ViT models on this problem' is contradicted by the manuscript's own citation of Jiang et al. [11] (FlashViT), a vision transformer for CHD detection; this novelty claim should be removed or qualified to apply only to the specific architectures/dataset combination used here.","section":"§V.B novelty claim"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads like a well-documented course project rather than a complete research paper. The novelty claim is overstated given the authors' own citation of FlashViT [11], and the evaluation methodology needs substantial revision before the numbers can be taken seriously. If the authors can provide a proper three-way split with confidence intervals and correct the reporting, the paper might be suitable for a workshop or a short conference paper, but in its current form it does not meet the standard for a journal publication. The abstract's inconsistency (73.9% vs. 76.9%) should be fixed during revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: the 80.72% DICOM number is a validation accuracy, not a test accuracy. The paper says 80/20 train/validation split, early stopping on validation accuracy, and Table Ia labels the column 'Val acc.' Yet Figure 6 relabels the same numbers as 'Test Accuracy Results.' No held-out test set is described anywhere. After comparing four architectures and picking ResNet-18 on that same small validation split (~166 images from 828), the number carries selection bias. The gap between ResNet-18 (80.72%) and Swin-Tiny (80.12%) is within what that bias can explain.\n\nWhat the paper does well is modest but real: it runs standard CNNs and transformers on two public CHD datasets, reports parameter counts, and honestly says its audio results (up to 76.9% with meta-ensemble) do not beat the original paper's 93.4%. The comparison of STFT vs Mel vs GAF is a useful sanity check. The authors also flag their own constraints—time, no histogram analysis—which is more than many papers do.\n\nThe soft spots beyond the missing test set: (1) the ZCHSound split is not described at all—no train/validation/test breakdown, no patient-level separation, nothing. (2) The 6–7% preprocessing gain from Gaussian blur and contrast enhancement is asserted without a controlled with/without experiment. (3) No error bars or repeated runs; every number is a single split. (4) The claimed novelty—applying ViTs to this X-ray problem—is contradicted by their own reference [11] (FlashViT). (5) The abstract's 73.9% doesn't match the paper's best fusion figure of 76.9%. The inconsistency may just be reporting the base model, but as written it looks off.\n\nThere's no sign of dishonesty here, and no fabricated entities. But the central accuracy claims are not reliable generalization estimates as reported. The fix is straightforward: hold out a real test set, run multiple seeds, report means and standard deviations, clarify the split for audio, and correct the novelty claim. With those changes, the paper would be a legitimate incremental benchmark.\n\nWho's this for? Someone starting in CHD screening who wants a quick orientation on what off-the-shelf models do on these datasets. It doesn't deserve a prize, but it deserves a serious referee if the authors revise. My recommendation: send it to peer review, not desk reject—the flaws are methodological, not existential, and the clinical domain matters.","headline":"The headline accuracies are validation numbers with a missing test set; the benchmark is honest but not yet reliable.","tokens_in":7771,"tokens_out":2376,"would_cite":false,"duration_ms":24035,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Deep learning can recognise congenital heart disease from heart-sound spectrograms and chest X-rays, with 73.9% audio and 80.72% X-ray accuracy reported here.","keywords":["congenital heart disease","deep learning","vision transformer","heart sound classification","chest X-ray classification","STFT spectrogram","late fusion","medical image preprocessing"],"falsifier":"Re-run both pipelines with repeated five-fold cross-validation in which all recordings or X-rays from one patient stay in the same fold; if the average accuracy falls substantially below the reported 73.9% audio and 80.72% X-ray figures, or if per-class sensitivity for the smaller classes drops far below the overall accuracy, the claim of usable CHD recognition would be falsified.","tokens_in":6713,"feed_emoji":"🫀","tokens_out":8463,"duration_ms":80039,"temperature":0.7,"pith_summary":"This paper tries to establish that deep learning can recognise congenital heart disease from two non-invasive inputs: heart-sound recordings converted into images and chest X-rays. On the ZCHSound heart-sound data it reports 73.9% accuracy with a ResNet trained on STFT spectrograms, and shows that late fusion by class F1 weights or a logistic-regression meta-ensemble improves that to 75.5% and 76.9%, respectively. On the DICOM chest X-ray data it reports 80.72% accuracy with ResNet-18 after Gaussian blurring, contrast enhancement, and augmentation, and it treats the application of modern vision transformers to this X-ray task as the novel contribution. The paper also reports that its preprocessing pipeline adds 6–7 percentage points of accuracy on the X-ray set, and that the audio results remain below the 93.4% reported by the original ZCHSound study.","feed_headline":"Chest X-ray model hits 80.7% accuracy on congenital heart disease","feed_subtitle":"Heart-sound model reaches 73.9%; preprocessing and late fusion add up to 6.7 points.","key_machinery":"The mechanism on the audio side is the conversion of raw heart sounds into fixed-size 2D images, chiefly the short-time Fourier transform (STFT) spectrogram, which lets pretrained CNNs treat heartbeats as visual patterns; the paper tests mel-spectrogram and Gramian angular field alternatives and finds them weaker. On the image side, the mechanism is a fixed preprocessing pipeline, Gaussian blur with standard deviation 0.8 followed by histogram-equalisation contrast enhancement at factor 1.8, followed by small rotations, flips, brightness and contrast augmentation; the paper attributes a 6–7% accuracy gain to this pipeline. Finally, the late-fusion stage combines the audio model predictions by validation-accuracy weights, class-F1 weights, or a logistic-regression meta-ensemble, and the last two improve the base results.","core_discovery":"The paper's central claim is that two complementary deep-learning pipelines can separate four congenital heart disease types, ASD, VSD, PDA and PFO, from normal controls, with 73.9% accuracy on heart sounds and 80.72% accuracy on chest X-rays. On the audio side, the claim is that the STFT representation is the strongest of the three tested 2D transforms, that ResNet-50v2 is the best single model for it, and that late fusion using class-F1 weights or a meta-ensemble beats the individual models. On the image side, the claim is that a preprocessing sequence of Gaussian blur, histogram-equalisation contrast enhancement, and light augmentation yields a 6–7 percentage point gain, and that ResNet-18, with 11.7M parameters, reaches the best accuracy and F1 while Swin-Tiny comes close and ViT-Tiny falls behind. A further claim is that applying state-of-the-art vision transformers to CHD chest X-rays is new here and that their competitive performance, despite higher parameter counts, makes model complexity an explicit consideration for deployment.","pith_inferences":["A natural next step the paper leaves implicit is early fusion: instead of combining audio and X-ray predictions only at the score level, a single transformer could take the STFT spectrogram and a downsampled X-ray as two input tokens, which may capture cross-modal signs the late-fusion stage misses.","Because ZCHSound is class-imbalanced, macro-F1 and per-class sensitivity for ASD, PDA, and PFO are more informative than accuracy; the paper's reported late-fusion gains should be re-read against those metrics.","The paper itself notes it chose the blur and contrast settings without analysing the noise histograms, so the reported 6–7% gain is tied to those fixed settings rather than a general claim about preprocessing.","The audio claim that Mel and GAF underperform STFT could be tested cheaply by replacing the CNN backbones with a single vision transformer and seeing whether the representation ranking changes."],"forward_implications":["A simple 11.7M-parameter ResNet-18 with the described preprocessing is enough to reach 80.72% on the CHD-CXR X-ray data, so larger models are not automatically better for this four-class screening task.","Class-F1-weighted late fusion and a logistic-regression meta-ensemble each improve single-model audio accuracy, by 1.6 and 2.7 percentage points, suggesting fusion is worth keeping in the audio pipeline.","Among the tested audio representations, STFT clearly carries the classification: it reaches 73.9% with ResNet, while Mel and GAF stay between 51% and 62% depending on architecture, so future audio work should start from STFT or introduce new representations.","Swin-Tiny reaches 80.12% on X-rays, within 0.6 points of the best CNN, showing that hierarchical vision transformers are competitive for CHD detection even though ViT-Tiny is not.","If the reported gain of 6–7 points from preprocessing is real, then the same blur-and-contrast pipeline may transfer to other paediatric chest X-ray classification tasks where training data are limited."],"supporting_citations":[{"why":"Supplies the DICOM chest X-ray dataset and the ResNet-18 baseline whose 80.72% accuracy this paper reproduces and compares against.","marker":"[10]"},{"why":"Supplies the ZCHSound paediatric heart-sound database used for every audio experiment.","marker":"[5]"},{"why":"Provides the Transformer-based heart-sound classification context and the performance target that the audio late-fusion results are measured against.","marker":"[6]"},{"why":"Supplies the prior lightweight FlashViT approach to CHD detection that motivates applying transformers to X-ray images.","marker":"[11]"},{"why":"Frames machine-learning CHD diagnosis and is cited for the ZCHSound collection setting.","marker":"[3]"},{"why":"Formally grounds the Gaussian filter used in the denoising step of the image preprocessing pipeline.","marker":"[13]"},{"why":"Supports the Gaussian-blur preprocessing choice with prior pneumonia-classification results.","marker":"[15]"},{"why":"Supports the contrast-enhancement choice with prior results on chest X-ray classification.","marker":"[16]"}],"fun_headline_variants":["Deep learning hits 80.7% accuracy on congenital heart disease X-rays","Chest X-ray AI detects congenital heart defect types at 80.7% accuracy","Heart sound and X-ray models fuse to boost congenital heart disease diagnosis","Late fusion adds 6.7 points to congenital heart disease deep learning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The accuracy numbers stand or fall on the assumption that the train/validation/test data are split randomly, are stratified across the classes, and contain no overlapping patients between training and testing; the paper states only an 80/20 split for the X-ray data and gives no split details, stratification, or leakage check for the audio data.","fun_headline_variants_meta":{"raw":{"variants":["Deep learning hits 80.7% accuracy on congenital heart disease X-rays","Chest X-ray AI detects congenital heart defect types at 80.7% accuracy","Heart sound and X-ray models fuse to boost congenital heart disease diagnosis","Late fusion adds 6.7 points to congenital heart disease deep learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000529,"raw_usage":{"total_tokens":2502,"prompt_tokens":849,"completion_tokens":1653,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":465,"completion_tokens_details":{"reasoning_tokens":1571}},"tokens_in":465,"tokens_out":1653,"duration_ms":16214,"temperature":1.0,"reasoning_tokens":1571,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:59:22.698928+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run both pipelines with repeated five-fold cross-validation in which all recordings or X-rays from one patient stay in the same fold; if the average accuracy falls substantially below the reported 73.9% audio and 80.72% X-ray figures, or if per-class sensitivity for the smaller classes drops far below the overall accuracy, the claim of usable CHD recognition would be falsified.","supporting_citations":[{"cited_title":"Heart sound classification based on scaled spectrogram and partial least squares regression,","cited_arxiv_id":null,"evidence_quote":"Supplies the DICOM chest X-ray dataset and the ResNet-18 baseline whose 80.72% accuracy this paper reproduces and compares against."},{"cited_title":"The Role of Machine Learning in Congenital Heart Disease Diagnosis: Datasets, Algorithms, and Insights","cited_arxiv_id":"2501.04493","evidence_quote":"Supplies the ZCHSound paediatric heart-sound database used for every audio experiment."},{"cited_title":"Detection and diagnosis of congenital heart disease from chest x-rays with deep learning models,","cited_arxiv_id":null,"evidence_quote":"Provides the Transformer-based heart-sound classification context and the performance target that the audio late-fusion results are measured against."},{"cited_title":"Classification of heart sound signal using curve fitting and fractal dimension,","cited_arxiv_id":null,"evidence_quote":"Supplies the prior lightweight FlashViT approach to CHD detection that motivates applying transformers to X-ray images."},{"cited_title":"Long-term outcomes after myocardial infarction in middle-aged and older patients with congenital heart disease—a nationwide study,","cited_arxiv_id":null,"evidence_quote":"Frames machine-learning CHD diagnosis and is cited for the ZCHSound collection setting."},{"cited_title":"Jiang, J","cited_arxiv_id":null,"evidence_quote":"Formally grounds the Gaussian filter used in the denoising step of the image preprocessing pipeline."},{"cited_title":"Theory of edge detection,","cited_arxiv_id":null,"evidence_quote":"Supports the Gaussian-blur preprocessing choice with prior pneumonia-classification results."},{"cited_title":"Gaussian blurring technique for detecting and classifying acute lymphoblastic leukemia cancer cells from microscopic biopsy images,","cited_arxiv_id":null,"evidence_quote":"Supports the contrast-enhancement choice with prior results on chest X-ray classification."}],"review_version":1}