{"id":"e7c4c27e-fd70-4ff6-b661-08b8f601018f","arxiv_id":"2505.01199","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"CaReAQA, an audio-language model tuned on GPT-4o-generated QA pairs from public stethoscope recordings, beats general-purpose audio models on its own benchmark, but the open-ended evaluation is partly self-referential and the abstract overstates the headline number.","lead":"Researchers built CaReAQA, an AI that listens to heart and lung sounds and answers open-ended diagnostic questions, and released a new benchmark of over 34,000 question-answer pairs. The paper's headline '86.2% accuracy' is actually a similarity score, not percent correct, which overstates the result.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The open-ended claim depends on GPT-4o judging semantic equivalence against GPT-4o-generated references, with no human calibration; the 70.6% 'accuracy' and 86.2 BERTScore are not yet established as diagnostic accuracy.","rationale":"The paper has real contributions: CaReSound is a new benchmark, and the closed-ended experiments on TR, Coswara, BMD, and FluSense use human labels and show CaReAQA ahead of baselines (Table 4), suggesting the audio encoder and fine-tuning are not vacuous. The open-ended claim, however, must stand on its own, and the evaluation has a specific structural weakness. In Sec. 3.3.1 the authors define Accuracy via GPT-4o binary judgments, and Appendix B shows the prompt: 'evaluate if the ground-truth and prediction are same/similar,' with no rubric, no example judgments, and no confidence threshold. The same model family generated the training references (Sec. 3.2, Appendix C), so GPT-4o's 'Yes' may be measuring style alignment with its own output distribution rather than clinical equivalence. Because CaReAQA is explicitly trained to mimic GPT-4o-generated QA pairs, it has an advantage before acoustic content is considered, while baselines are judged against a reference style they never saw. METEOR and BERTScore do not fix this; they are lexical and semantic similarities against the same synthetic references, not diagnostic truth. The abstract's '86.2% accuracy' is also an error: 86.2 is the combined BERTScore, and the combined Accuracy in Table 3 is 70.6. These are fixable reporting and validation problems rather than fundamental refutations, so the appropriate disposition is unchanged from the reader's CONDITIONAL: correct the abstract, add a human-calibrated judge, and report uncertainty and reconciled dataset statistics.","tokens_in":14674,"tokens_out":4006,"duration_ms":38773,"concrete_test":"Sample 300 open-ended test items stratified across the five CaReSound datasets. Have two clinicians, blind to model identity, independently judge whether each prediction is semantically equivalent to the reference, with a third adjudicating disagreements. Compute Cohen's kappa between GPT-4o's Yes/No judgments and the human consensus, then recompute the combined Accuracy for CaReAQA and every baseline using the human labels. If kappa is below 0.6, or if CaReAQA's human-based accuracy drops by more than 5 points relative to its GPT-4o-judged accuracy while baseline deltas differ, the open-ended result is an artifact of the uncalibrated judge and the abstract must be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that CaReAQA 'achieves 86.2% accuracy on open-ended diagnostic reasoning tasks' rests on Table 3, but the Accuracy column is not a direct correctness measure. It is the rate at which GPT-4o answered 'Yes' when asked whether the model's free-form prediction is 'same/similar' to a reference (Sec. 3.3.1, Appendix B). The references were themselves generated by GPT-4o from metadata (Sec. 3.2, Appendix C), and CaReAQA was fine-tuned on those GPT-4o-generated QA pairs. This creates a distributional alignment: CaReAQA outputs are in-distribution for the judge, while baseline models (LTU, Pengi, GAMA, Qwen2-Audio) produce different phrasing against a reference style they never saw. No human evaluation, inter-annotator agreement, or error bars calibrate these Yes/No judgments, so the 70.6% combined Accuracy and the 86.2 BERTScore (computed with bert-base-uncased, not a clinical measure) could largely reflect paraphrase and style similarity rather than diagnostic correctness. The closed-ended generalization results (Table 4) use real labels and give some evidence that the model maps audio to clinically meaningful categories, but they do not validate the open-ended reasoning claim. The abstract's '86.2% accuracy' is also a direct mislabel: 86.2 is the combined BERTScore, while the combined Accuracy in Table 3 is 70.6.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CaReAQA, an audio-language model for cardiac and respiratory audio question answering, and CaReSound, a benchmark dataset assembled from five public medical audio collections. QA pairs for training and evaluation are generated by GPT-4o from textual metadata, and the model combines a fine-tuned OPERA audio encoder, a transformer mapper, and LLaMA-3.2-3B with LoRA. The authors report open-ended QA results using BERTScore, METEOR, and a GPT-4o-based semantic 'Accuracy' metric, plus closed-ended classification accuracy on four unseen datasets. They claim CaReAQA outperforms existing audio-language baselines on open-ended diagnostic reasoning and generalizes to closed-ended classification.","tokens_in":14974,"tokens_out":5196,"duration_ms":48543,"significance":"If substantiated, the paper would provide a useful open resource: the CaReSound dataset and model are released on Hugging Face, and the ablation studies (LoRA rank, PEFT methods, mapping types, audio encoders) are systematic and informative. The closed-ended generalization results use real labels and provide credible evidence that the model captures clinically relevant audio categories. The explicit failure-case analysis in Appendix F and the statement in the conclusion that the model is not intended for direct clinical deployment are appropriate. The main significance hinges on the open-ended diagnostic reasoning claim, and that is exactly where the current evaluation is not yet convincing.","major_comments":[{"comment":"The abstract's claim of '86.2% accuracy on open-ended diagnostic reasoning tasks' is not supported by Table 3: the All* Accuracy column is 70.6, while 86.2 is the All* BERTScore. This is a load-bearing mislabel of the headline result and must be corrected in the abstract, conclusion, and any summary of results; the text in Section 4.2 already correctly refers to 'an average BERTScore of 86.2', so the discrepancy appears to be a reporting error rather than a substantive result, but it must be fixed.","section":"Abstract and Section 4.2, Table 3"},{"comment":"The open-ended Accuracy metric is computed by prompting GPT-4o to judge whether a prediction is 'same/similar' to a reference (Appendix B), but the references themselves were generated by GPT-4o from metadata during dataset construction (Section 3.2, Appendix C), and CaReAQA was fine-tuned on those same GPT-4o-generated answers. This creates a closed loop: CaReAQA's predictions are in-distribution for the judge, while baseline models produce out-of-distribution phrasings. The paper reports no human evaluation, no inter-annotator agreement, and no confidence intervals. As a result, the reported 70.6% accuracy and 86.2 BERTScore do not yet establish diagnostic correctness for open-ended reasoning. Please add independent human or clinical validation on a sample, report judge agreement statistics, and temper the conclusions accordingly.","section":"Section 3.3.1, Appendix B, Appendix C"},{"comment":"The dataset size is inconsistent. Section 3.5 states that the dataset contains 12,673 samples and is split into 10,138 training and 2,535 test samples, but Table 1 reports 16,273 samples in the All column, which matches the sum of the per-dataset rows (6,899 + 337 + 5,282 + 1,259 + 2,496). Please clarify which total is correct and re-state the split consistently. In addition, Table 2 reports BMD as having 5 classes while Section 3.3.2 describes the BMD-HS dataset as having six categories; these must be reconciled.","section":"Section 3.5 vs Table 1"},{"comment":"The comparison with general-purpose audio-language models is not fully controlled. CaReAQA is fine-tuned on CaReSound, whereas LTU, Pengi, Qwen2-Audio, and GAMA appear to be evaluated without any fine-tuning on the training distribution; the LLM-without-audio and Cascaded baselines are not trained under the same protocol either. The claim that CaReAQA 'outperforms baseline models' should be qualified as 'outperforms zero-shot baselines' unless the baselines are fine-tuned under an equivalent protocol, or an explicit statement is added confirming the absence of fine-tuning for those baselines.","section":"Section 4.2, Table 3"}],"minor_comments":[{"comment":"The citation for AugLy appears as an unresolved '(?)' in the text; please add the full reference.","section":"Section 4.1"},{"comment":"The heading 'Effect of LoRA Rank.' is duplicated in the text immediately before Table 7; please remove the duplicate.","section":"Section 4.4"},{"comment":"The dataset name is written inconsistently as both 'CirCor' and 'CIRCOR', and the model name appears as both 'Gama' and 'GAMA'; please use consistent spellings.","section":"Throughout"},{"comment":"The metric called 'Accuracy' is a GPT-4o binary judgment of semantic equivalence, not correctness in the usual classification sense; please rename it (e.g., 'semantic acceptance rate') or clearly define it in every occurrence to avoid confusion with the classification accuracy used in Table 4.","section":"Section 3.3.1"},{"comment":"The caption has a spacing typo: 'Figure 1:Spectrograms' should be 'Figure 1: Spectrograms'.","section":"Figure 1 caption"}],"recommendation":"major_revision","confidential_remarks":"The manuscript fits the conference scope and the dataset/model release is a positive contribution. The central open-ended reasoning claim, however, depends on an internally closed GPT-4o evaluation loop and is currently mislabeled in the abstract. I would support acceptance if the authors correct the headline metric and provide independent validation or clearly reframe the claim as semantic-similarity performance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Thanks for the pointer. Quick take: CaReSound is a genuinely useful new resource for the medical-audio QA subfield, and the model is a competent application of standard audio-LLM recipes. But the abstract's headline number is wrong — 86.2 is BERTScore, not accuracy — and the open-ended evaluation leans on GPT-4o as both answer generator and judge, so the open-ended accuracy numbers are not established as diagnostic accuracy. That doesn't sink the paper; the closed-ended generalization results rely on real labels and suggest real learning. It needs fixing before I'd trust the reasoning claims.\n\nWhat's actually new: CaReSound, assembled from five public datasets (ICBHI, KAUH, CirCor, SPRSound, ZCHSound) with metadata and 34,792 GPT-4o-generated QA pairs. The data and model are released, which is a concrete contribution. The modeling itself — OPERA encoder, transformer mapper, LoRA-tuned LLaMA-3.2-3B — is not architecturally novel, but the medical-audio application is new and the ablation study (PEFT methods, mapping types, encoder choices) is more thorough than many papers in this space. The closed-ended results on four unseen datasets (56.9% average accuracy, beating baselines) are the most convincing part; those use human labels and suggest the audio encoder is capturing clinically meaningful structure.\n\nWhere it gets soft. The abstract says “86.2% accuracy on open-ended diagnostic reasoning tasks.” Table 3 shows the combined Accuracy is 70.6; 86.2 is the BERTScore. That is a direct mislabel, not a rounding difference. The accuracy metric itself is GPT-4o answering Yes/No on whether a prediction matches a ground-truth answer that GPT-4o also generated from metadata. LLM-as-judge is common, but without a human-calibration study — agreement on a sample, error bars, or at least a qualitative error analysis — the open-ended numbers are hard to interpret, especially for medical claims. Dataset counts are also inconsistent: Section 3.5 says 12,673 samples while Table 1 sums to 16,273. Minor but sloppy. The failure-case appendix is a point in the paper's favor, but those failures highlight exactly what should be checked against human judgment: is “holosystolic high-pitched harsh” meaningfully different from “holosystolic low-pitched blowing”, and would a clinician or even the judge call it correct?\n\nBottom line: this is a solid benchmark paper with a reasonable model and concrete, fixable evaluation problems. The benchmark alone may be worth the paper, and the closed-ended results support the model's basic competence. The headline needs correction and the open-ended evaluation needs human grounding. I would send it to peer review — the benchmark deserves scrutiny and the community would benefit from a tightened evaluation. I'd bring it to reading group mainly to discuss evaluation design.","headline":"CaReSound is a useful new benchmark and the closed-ended results are credible, but the abstract's 86.2% accuracy claim mislabels BERTScore and the open-ended evaluation needs human calibration.","tokens_in":15562,"tokens_out":2728,"would_cite":true,"duration_ms":25882,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CaReAQA is an audio-language model that takes heart or lung sounds and a natural-language question and generates free-form diagnostic answers; the paper claims it outperforms general-purpose audio-language models on open-ended and…","keywords":["medical audio question answering","diagnostic reasoning","heart and lung sounds","audio-language model","CaReSound benchmark","open-ended question answering","auscultation","self-supervised audio encoder"],"falsifier":"Sample 200 open-ended test responses, have two clinicians independently mark each as semantically matching the reference or not, and compare their marks with GPT-4o's Yes/No verdicts: if clinician agreement with GPT-4o is low, or if GPT-4o says 'Yes' disproportionately to answers phrased like its own output, the reported 70.6% accuracy does not measure diagnostic agreement.","tokens_in":14433,"feed_emoji":"🩺","tokens_out":7782,"duration_ms":70003,"temperature":0.7,"pith_summary":"CaReAQA aims to make audio-based diagnostics open-ended: instead of classifying a recording into a fixed disease label, the model takes a heart or lung sound plus a clinician-style question and generates a free-form diagnostic answer. To train and test it, the paper assembles CaReSound, a benchmark of 16,273 recordings drawn from five public cardiac and respiratory datasets, with 34,792 question-answer pairs that an LLM generated from each recording's metadata. On the open-ended test split, the paper reports a combined BERTScore of 86.2 and a semantic accuracy of 70.6% measured by GPT-4o judgments, ahead of general-purpose audio-language baselines; the abstract's '86.2% accuracy' is the BERTScore, not the accuracy column. The same model reaches 56.9% average accuracy on four previously unseen closed-ended classification datasets. The broader claim is that aligning a self-supervised audio encoder with an LLM through synthetic QA data is a viable route to diagnostic reasoning from auscultation sounds.","feed_headline":"Heart-lung audio model answers open-ended diagnostic questions","feed_subtitle":"CaReAQA pairs a foundation audio encoder with an LLM and outperforms general-purpose baselines on a 16,273-clip benchmark.","key_machinery":"The mechanism is a three-part alignment stack. An OPERA contrastive audio encoder converts a 5-second log-mel spectrogram into embeddings; a transformer-based mapper projects those embeddings into the input space of LLaMA-3.2-3B; and the text question and audio embeddings are concatenated, positionally encoded, and fed through the LLM, which is fine-tuned with LoRA (rank 8) to maximize $p_\\theta(A_i \\mid X_q, X_a, A_{<i})$. The other load-bearing piece is CaReSound itself: GPT-4o writes three or more QA pairs per recording from metadata alone, so the dataset is large (34,792 pairs) but entirely synthetic.","core_discovery":"The paper's central claim is that open-ended diagnostic reasoning over cardiac and respiratory audio can be learned end-to-end by a model that maps audio embeddings into an LLM's token space and fine-tunes the whole stack on synthetic question-answer pairs. The evidence is the CaReSound benchmark and the performance comparison: CaReAQA scores 86.2 BERTScore and 70.6% GPT-4o-judged accuracy on combined open-ended questions, versus at best 56.8 BERTScore and 24.5% accuracy among the compared general-purpose baselines; on unseen closed-ended tasks it averages 56.9% accuracy. The paper also claims the design choices matter: LoRA fine-tuning of the LLM, a transformer mapper, and fine-tuning the OPERA audio encoder each outperform their alternatives in ablations.","pith_inferences":["Because the reference answers and the accuracy judgments both come from GPT-4o, the open-ended numbers measure agreement with one LLM's text style as much as diagnostic correctness; a clinician-labeled audit set would separate these.","The synthetic QA generation from metadata only means the model might answer from textual priors such as chest location or demographics rather than acoustic evidence; questions whose answers are deliberately contradicted by the audio would test this.","The recipe is portable: any medical or industrial sound domain with rich metadata could get a QA benchmark the same way, and the per-dataset variance suggests such benchmarks need per-domain reporting, not just aggregates."],"forward_implications":["Open-ended auscultation question answering is trainable without manually written clinical QA pairs: an LLM can generate the training data from structured metadata.","Domain-specific audio-language alignment transfers beyond the training label set: the same model answers questions in-domain and classifies four unseen datasets at 56.9% average accuracy.","On this benchmark, fine-tuning the audio encoder and the LLM with LoRA matters more than the choice of mapper or a larger parametric adapter capacity.","Failure cases mean the model is not yet a diagnostic instrument: overlapping auscultation patterns, rare diseases, and noisy recordings still produce wrong answers."],"supporting_citations":[{"why":"Supplies the ICBHI respiratory sound recordings with annotated respiratory cycles for CaReSound.","marker":"Sun, 2023"},{"why":"Supplies the KAUH respiratory recordings with diagnosis, sound type, and chest zone metadata.","marker":"Fraiwan et al., 2021"},{"why":"Supplies the CirCor pediatric heart sound recordings and murmur annotations.","marker":"Oliveira et al., 2021"},{"why":"Supplies the SPRSound pediatric respiratory recordings with record- and event-level labels.","marker":"Zhang et al., 2022"},{"why":"Supplies the ZCHSound congenital heart disease recordings.","marker":"Jia et al., 2024"},{"why":"GPT-4o generates the QA pairs from metadata and provides the binary semantic-match judgments used for open-ended accuracy.","marker":"OpenAI, 2024"},{"why":"The OPERA audio encoder provides the self-supervised representations that CaReAQA fine-tunes.","marker":"Zhang et al., 2024a"},{"why":"LLaMA-3.2-3B is the large language model backbone that CaReAQA fine-tunes with LoRA.","marker":"Touvron et al., 2023"},{"why":"BERTScore is the semantic similarity metric used to evaluate open-ended answers.","marker":"Zhang et al., 2019"},{"why":"GAMA serves as a general-purpose audio-language baseline that CaReAQA is claimed to outperform.","marker":"Ghosh et al., 2024"}],"fun_headline_variants":["AI answers open-ended diagnostic questions on heart and lung sounds","CaReAQA: audio-language model for diagnostic reasoning","Heart-lung audio model reasons like a clinician for QA","Open-ended audio queries get diagnostic answers from AI","CaReAQA: turning heart and lung sounds into answers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire open-ended evaluation depends on trusting GPT-4o to have written the reference answers correctly and to judge correctly whether the model's free-form responses mean the same thing as those references.","fun_headline_variants_meta":{"raw":{"variants":["AI answers open-ended diagnostic questions on heart and lung sounds","CaReAQA: audio-language model for diagnostic reasoning","Heart-lung audio model reasons like a clinician for QA","Open-ended audio queries get diagnostic answers from AI","CaReAQA: turning heart and lung sounds into answers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000502,"raw_usage":{"total_tokens":2430,"prompt_tokens":901,"completion_tokens":1529,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":1449}},"tokens_in":517,"tokens_out":1529,"duration_ms":10107,"temperature":1.0,"reasoning_tokens":1449,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:23:35.136545+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Sample 200 open-ended test responses, have two clinicians independently mark each as semantically matching the reference or not, and compare their marks with GPT-4o's Yes/No verdicts: if clinician agreement with GPT-4o is low, or if GPT-4o says 'Yes' disproportionately to answers phrased like its own output, the reported 70.6% accuracy does not measure diagnostic agreement.","supporting_citations":[{"cited_title":"Ibnian, and Mohanad Alkhodari","cited_arxiv_id":null,"evidence_quote":"Supplies the KAUH respiratory recordings with diagnosis, sound type, and chest zone metadata."},{"cited_title":"The circor digiscope dataset: from murmur detection to murmur classification","cited_arxiv_id":null,"evidence_quote":"Supplies the CirCor pediatric heart sound recordings and murmur annotations."},{"cited_title":"Sprsound: Open-source sjtu paediatric respiratory sound database","cited_arxiv_id":null,"evidence_quote":"Supplies the SPRSound pediatric respiratory recordings with record- and event-level labels."}],"review_version":1}