{"id":"a76cdb31-d061-46f1-b5e1-439b549cd94e","arxiv_id":"2505.03781","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"ALFRED combines ECG feature extraction, a hand-crafted rule module, and retrieval-augmented GPT-4o-mini to classify PTB-XL superclasses, claiming improved accuracy and explainability.","lead":"The paper proposes ALFRED, a zero-shot ECG diagnosis system that feeds extracted signal features and rule-based hints to GPT-4o-mini, augmented with expert-curated text from a retrieval database. It reports improved classification on one fold of the PTB-XL dataset, but without error bars, a rule-only baseline, or a complete method description.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported gains come from the rule module, not the LLM/RAG; without a rule-module-only control, 'expert knowledge enhances performance' is confounded and the central LLM/RAG claim is unsupported.","rationale":"The central empirical claim is that expert knowledge, delivered through RAG and an LLM, improves ECG diagnosis. The ablation table shows the biggest improvement when the rule module's outputs are added, but the rule module itself is never evaluated as a standalone classifier. This is a direct confound for attribution: the rule module is itself expert knowledge, so if it alone achieves the reported accuracy, the LLM and RAG are not necessary for the performance gain. The paper's own discussion credits rule results as the main source of improvement, making this control essential. I considered whether the use of only one fold or the lack of baselines is more load-bearing, but the rule-module confound is more fundamental because it questions whether the proposed mechanism (LLM+RAG) does any causal work at all. The missing control is also easy to supply, so the concern is concrete and testable. The reader's weakest assumption captures exactly this issue, and the reader's REJECT verdict remains appropriate: the manuscript, as written, does not support its headline claim. No change to the verdict is needed, but the finding should be communicated as a missing control rather than purely as missing statistical rigor.","tokens_in":4147,"tokens_out":6510,"duration_ms":64464,"concrete_test":"Run the rule module alone on PTB-XL fold 10: map its 40 category flags to the 5 PTB-XL superclasses using exactly the same mapping as in the paper, then compute PPV, NPV, Sens, and Spec on the same samples used for Table 1. If rule-only performance is at or above Ablation1/Proposed (e.g., PPV ≥ 0.414, Sens ≥ 0.458), the LLM/RAG contribution is unproven; if it is clearly lower (e.g., PPV ≤ 0.30), the confound is resolved and the framework claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Table 1, the largest performance jump is Base → Ablation1 (PPV 0.326→0.414, Sens 0.356→0.458), i.e., exactly when the rule module's True/False flags over 40 categories are injected into the prompt. The later additions of documents/knowledge change PPV by only +0.002 (Ablation2) or +0.037 (Ablation3), and Proposed is slightly worse than Ablation3 (PPV 0.443 vs 0.451). The paper never reports the rule module's own classification performance on the same fold, yet the rule module is a deterministic clinical algorithm that directly encodes disease criteria. If that module alone already yields the Ablation1 metrics, then the LLM is merely reformatting rule output, and the abstract's claim that expert-curated knowledge enhances LLM-based diagnosis is not established. The single-fold evaluation with no error bars or statistical tests further prevents any claim that the 0.037 PPV gain is 'significant.' The missing rule-module-only control is the load-bearing gap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ALFRED, a zero-shot ECG diagnosis framework that combines a UNet-based feature extractor, a rule module that encodes disease criteria into True/False flags over 40 categories, and a retrieval-augmented generation (RAG) prompt for GPT-4o-Mini. The prompt is enriched with (a) extracted lead-specific and global ECG features, (b) rule results, (c) retrieved definitions from a curated database of documents and expert-authored knowledge, and (d) instructions. The authors evaluate on PTB-XL fold 10, report classification metrics for the five superclasses across four ablations, and provide one qualitative example of LLM-generated explanations. The central claim is that expert-curated knowledge and RAG significantly enhance the accuracy and explainability of LLM-based ECG diagnosis.","tokens_in":4398,"tokens_out":4513,"duration_ms":48388,"significance":"If the central claim were fully supported, ALFRED would be a practically useful architecture for evidence-based LLM interpretation of ECG, with the public release of the application and the attempt to combine feature-based rules with RAG as notable strengths. The ablation structure is a sensible way of isolating component contributions, and the inclusion of rule results and knowledge in the prompt is an interesting design. However, the paper currently lacks the key control needed to attribute the reported gains to the LLM/RAG system rather than to the deterministic rule module, and the single-fold evaluation with no statistical analysis does not support the word 'significant.' The main contribution at this stage is the proposal and implementation; the empirical evidence for the claimed effectiveness is incomplete.","major_comments":[{"comment":"The single largest performance jump in Table 1 is from Base to Ablation1 (PPV 0.326→0.414, Sens 0.356→0.458), which occurs when the rule module's True/False flags are inserted into the prompt. The rule module is itself a deterministic diagnostic classifier that encodes disease criteria, yet its standalone classification performance on the same fold is never reported. Without this control, the paper's central claim—that expert-curated knowledge and RAG enhance LLM-based diagnosis—is confounded: the LLM may simply be reformatting the rule module's output, and the retrieval-augmented knowledge may contribute little beyond what the rule module already provides. Please report the rule-module-only metrics on the same PTB-XL fold, and, if possible, the performance of the rule module combined with the prompt but without retrieved knowledge.","section":"§3.2, Table 1"},{"comment":"The evaluation uses only fold 10, which the authors describe as having 'particularly high label quality,' but no justification is given for this selection, and no uncertainty quantification is provided. GPT-4o-Mini is a stochastic model, and the reported differences in the knowledge ablations are small (e.g., Ablation2 vs Ablation3 PPV 0.416 vs 0.451; Proposed vs Ablation3 PPV 0.443 vs 0.451). The abstract's phrase 'significantly enhances performance' and §3.2's 'significant improvement' are therefore not supported by the evidence. Please report results across all PTB-XL folds (or a justified subsample with error bars) and use bootstrap confidence intervals or repeated sampling of the LLM to assess whether the observed differences are beyond noise.","section":"§3.1, §3.2"},{"comment":"The rule module's diagnostic criteria and feature thresholds are not specified, and the UNet's hyperparameters are described only as 'carefully tuning' without listing depth or layer counts. This makes the paper non-reproducible and prevents the reader from analyzing which aspects of the rule module drive the improvements in Table 1. In particular, because the rule module outputs True/False diagnoses and the prompt directs the LLM to use those results as hints, the lack of detail about the rule module's logic is load-bearing for interpreting the ablation. Please provide the full rule definitions (or a supplemental file) and the UNet architecture details, along with the rule module's own performance on the same evaluation set.","section":"§2.1, §3.1"}],"minor_comments":[{"comment":"The interpretability assessment rests on a single qualitative example and author-affiliated experts' subjective judgment; a structured evaluation protocol (e.g., clinician rating with predefined criteria and blinding) or a larger set of examples would be needed to substantiate the 'highly effective and well-constructed' claim.","section":"§3.2, Example 1"},{"comment":"The phrase 'expert knowledge' is used to refer both to the rule module's diagnostic logic and to the curated database content; this conflation obscures which component is being credited in the abstract and in §3.2. Please use distinct terms for these two sources of expertise.","section":"Throughout"},{"comment":"There is a typographical error in 'for thePR interval' (missing space), and the list of the 30 lead-specific and 12 global features is not provided; including the feature list would improve reproducibility and readability.","section":"§2.2"},{"comment":"Several listed references (e.g., Abelson et al. 1985, Lixto, KL-ONE, hypertree decompositions) appear unrelated to ECG, LLMs, or the present content, suggesting a template/bibliography error that should be corrected.","section":"References"},{"comment":"The table's header layout is confusing: 'Framework Rule Results Augmented Information' with subcolumns 'ECG Features' and 'Rule Results' does not unambiguously show which options are active in each row; a per-row checkmark format or explicit boolean columns would be clearer.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's reference list contains multiple works unrelated to the topic, which indicates a serious editorial oversight in the submitted version. More importantly, the missing rule-module-only control is a correctable but central gap; if the rule module alone already achieves the Ablation1 metrics, the paper's claim about LLM/RAG enhancement would collapse. The single-fold evaluation and lack of statistical testing further limit the paper's current suitability for a journal, though the framework itself may be of interest to an applied venue after the control and statistical analysis are added."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nHere's the one-line take: ALFRED is a reasonable engineering pipeline with a missing control that guts its main claim. If you're looking for a clean demonstration that RAG plus expert knowledge improves LLM ECG diagnosis, this paper doesn't provide it.\n\nWhat is new: the combination of segmentation-derived features, a hand-written rule module, RAG over curated documents plus authored 'knowledge', and GPT-4o-mini in a zero-shot setting. That specific synthesis isn't in the prior work I know. They also released a public app, which is real and useful for exploring the idea.\n\nThe paper does a few things well. The ablations are structured clearly – they toggle rule results, document augmentation, and knowledge augmentation. The prompt design is described in enough detail to replicate (though not fully, see below). And the qualitative example shows the LLM can produce readable, feature-grounded explanations.\n\nNow the soft spots, in proportion. The load-bearing problem: the rule module is never evaluated alone. Table 1 shows the biggest jump is from Base (no rule results) to Ablation1 (rule results added): PPV 0.326 to 0.414, sensitivity 0.356 to 0.458. That's exactly the step where the deterministic rule module's True/False flags are inserted into the prompt. Later additions of documents or knowledge move PPV by 0.002 or 0.037. Without a rule-module-only baseline, we can't tell whether the LLM is doing anything more than formatting the rule module's output. In fact, Ablation3 out-performs the Proposed configuration (PPV 0.451 vs 0.443), which suggests adding knowledge to the feature augmentation actually hurts – hard to interpret without error bars.\n\nThe evaluation uses a single hand-picked fold (fold 10, 'particularly high label quality'), no error bars, no statistical tests. The claimed gains could be noise. Also, the reference list is full of unrelated classic AI papers (SICP, KL-ONE, hypertree decompositions) and none of the ECG-LLM prior work appears, so the citation pattern is a red flag. There are no citations to PTB-XL's original paper either (shown as [?] in the text).\n\nMissing implementation details: the UNet hyperparameters and rule thresholds are not given, so the feature extraction and rule module are not reproducible as described.\n\nThe abstract says 'expert knowledge significantly enhances performance.' Based on the evidence, that's unsupported. The strongest honest statement is: adding rule results to the prompt improved performance over a no-rule baseline on one fold, and we don't know how much the rule module alone would score.\n\nWho is this for? Researchers working on LLM-based ECG interpretation will find the pipeline idea worth a look, and the app is a useful artifact. But as a scientific paper, it needs a rule-module-only baseline, multi-fold evaluation with confidence intervals, and a proper citation pass.\n\nMy recommendation: send it to peer review with a strong request for revision. The core idea is testable and the missing control is fixable. If the rule module alone reproduces Ablation1's numbers, the paper's claim collapses; if it doesn't, the result becomes interesting. A referee should ask for exactly that experiment.\n\nCheers.","headline":"The pipeline is plausible but the evaluation never isolates the rule module, so the paper's central claim that expert-curated knowledge improves LLM diagnosis is not actually demonstrated.","tokens_in":4940,"tokens_out":2968,"would_cite":false,"duration_ms":28269,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A zero-shot ECG diagnosis framework that injects expert-curated knowledge into a retrieval-augmented LLM prompt improves classification and explanation quality on PTB-XL.","keywords":["ECG diagnosis","large language models","retrieval-augmented generation","expert-curated knowledge","zero-shot classification","PTB-XL","explainability","rule module"],"falsifier":"Re-run the ALFRED pipeline on PTB-XL fold 10 but replace the LLM's final decision with the rule module's own True/False labels for the five superclasses, or prompt the LLM with rule results but no retrieved text; if either variant matches the reported 0.477 sensitivity and 0.797 specificity, the claim that expert knowledge enhanced the diagnosis would be unsupported.","tokens_in":3963,"feed_emoji":"🫀","tokens_out":6934,"duration_ms":69178,"temperature":0.7,"pith_summary":"ALFRED is a zero-shot ECG diagnosis framework that asks a large language model to read an electrocardiogram as text: structured waveform features, deterministic rule-based verdicts, and retrieved definitions from a curated cardiology knowledge base. The paper's central claim is that retrieval-augmented generation alone is not enough, and that expert-curated knowledge added to the prompt improves both the accuracy and the explainability of automated ECG interpretation. On the tenth fold of the PTB-XL dataset, the full pipeline raises sensitivity from 0.356 to 0.477 and specificity from 0.754 to 0.797 relative to a retrieval-only baseline. A sympathetic reader would take this as evidence that structured domain expertise, not just a larger language model, is a practical lever for making LLM-based medical diagnosis more reliable.","feed_headline":"Expert knowledge boosts LLM ECG diagnosis in PTB-XL test","feed_subtitle":"Adding curated cardiology facts and rule-based hints to a retrieval prompt raises zero-shot sensitivity and specificity.","key_machinery":"The load-bearing object is the five-part retrieval-augmented prompt that turns a raw ECG recording into a textual evidence set: a neural segmentation model yields 30 lead-specific and 12 global features, a deterministic rule module marks 40 conditions true or false, a curated vector database supplies retrieved definitions for both the feature names and the diagnosis names, and an instruction block asks the LLM to respond in a structured format. That prompt forces the LLM to ground its reasoning in measured values and explicit clinical criteria rather than in general medical text alone, and it is the component the ablations toggle on and off.","core_discovery":"The paper's central claim is that a zero-shot LLM can diagnose 12-lead ECGs more reliably when its prompt is built from five parts: lead-specific and global waveform features, rule-module outputs for 40 conditions, retrieved textbook and web definitions for the feature names, retrieved definitions plus expert-authored knowledge for the disease names, and explicit instructions with a fixed response format. The authors report that adding the rule results produces the largest single improvement over the retrieval-only baseline, and that further adding expert knowledge to the disease-side retrieval improves specificity and positive predictive value over documents alone. They also claim that the resulting explanations were judged by their in-house medical experts to be effective, well-constructed, and understandable to non-experts.","pith_inferences":["A direct test of the attribution claim would run the rule module's own outputs directly to the five superclass labels, without any LLM or retrieval, and compare against the reported 0.477 sensitivity and 0.797 specificity; the paper never reports that standalone rule-module performance.","If the pattern transfers beyond PTB-XL, the same recipe—deterministic rules plus retrieved definitions plus a zero-shot LLM—could become a low-cost template for other structured medical signals where labelled data are scarce.","The explanation quality is currently assessed only qualitatively by the authors' in-house experts; a quantitative next step would ask cardiologists to rate or correct the explanations, or check whether the feature values cited by the LLM are the same ones that drive the rule module's verdict."],"forward_implications":["Adding rule-module outputs to the prompt produces the largest single jump in sensitivity in the reported ablations, from 0.356 to 0.458.","Switching the diagnosis-side augmented content from documents only to documents plus expert knowledge raises specificity from 0.761 to 0.798 and positive predictive value from 0.414 to 0.451 in the rule-enabled configuration.","Including both feature-side and diagnosis-side augmented information slightly reduces positive predictive value and specificity relative to diagnosis-side knowledge alone, which the authors attribute to longer prompts.","The system outputs a Result and an Explanation for each superclass, so the model's reasoning is inspectable and could be audited by clinicians."],"supporting_citations":[],"fun_headline_variants":["Expert knowledge ups zero-shot LLM ECG accuracy","RAG prompts plus expert rules improve LLM ECG diagnoses","LLM ECG diagnosis gains from curated cardiology","Zero-shot ECG AI benefits from expert-curated knowledge","Adding expert facts boosts LLM ECG interpretation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reported gains come from the LLM's use of expert-curated knowledge and retrieval, not from the deterministic rule module alone, whose standalone performance is never reported.","fun_headline_variants_meta":{"raw":{"variants":["Expert knowledge ups zero-shot LLM ECG accuracy","RAG prompts plus expert rules improve LLM ECG diagnoses","LLM ECG diagnosis gains from curated cardiology","Zero-shot ECG AI benefits from expert-curated knowledge","Adding expert facts boosts LLM ECG interpretation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000162,"raw_usage":{"total_tokens":1167,"prompt_tokens":799,"completion_tokens":368,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":415,"completion_tokens_details":{"reasoning_tokens":295}},"tokens_in":415,"tokens_out":368,"duration_ms":3917,"temperature":1.0,"reasoning_tokens":295,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:58:28.625022+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the ALFRED pipeline on PTB-XL fold 10 but replace the LLM's final decision with the rule module's own True/False labels for the five superclasses, or prompt the LLM with rule results but no retrieved text; if either variant matches the reported 0.477 sensitivity and 0.797 specificity, the claim that expert knowledge enhanced the diagnosis would be unsupported.","supporting_citations":[],"review_version":1}