{"id":"fd95dcd7-e00e-4fa2-b72b-4ab42991b82c","arxiv_id":"2602.18452","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"RA-QA converts 11 public respiratory-audio datasets into 9M template-generated QA pairs and shows current audio-language models score near zero on clinical task accuracy.","lead":"This paper introduces RA-QA, a benchmark that turns 11 public respiratory-audio datasets into 9 million question-answer pairs and tests AI models on them. General audio-language models mostly fail these clinical-style questions, while simpler models only partly succeed.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RA-QA aggregates unanswerable metadata questions with audio-grounded ones, so failure scores may not demonstrate audio QA limits","rationale":"The reader's weakest assumption centers on label noise and whether template questions resemble real queries. Our concern is related but distinct: even with perfectly clean labels, many RA-QA questions are unanswerable from the given inputs (audio + question), so task-level correctness on those items does not measure audio-language understanding. This is more load-bearing because it affects the benchmark's construct validity, not just the reliability of the ground truth. A label-balanced subset guarantees that these unanswerable attributes are included, so the aggregate failure scores are not interpretable without a per-attribute breakdown. The paper does not provide such a breakdown, and its two case studies are insufficient to establish that the observed failures are due to heterogeneity rather than task impossibility. The requested test — re-evaluating on the subset of attributes where the audio-only oracle shows genuine acoustic signal — would resolve whether the central claim is supported. This is a concrete, low-cost check using already-existing baselines and data. Pending that check, the benchmark remains promising but the current conclusions about model robustness are premature, so a conditional recommendation is appropriate.","tokens_in":8808,"tokens_out":9368,"duration_ms":94530,"concrete_test":"Re-run the Table 2 evaluation restricted to acoustically grounded attributes, defined as those where the paper's own audio-only SVM baseline (Sec. 3.1) exceeds a pre-specified threshold above chance (e.g., MacroF1 > 0.3). Recompute all baseline scores (Pengi, multimodal classifier, CaReAQA-style) on this subset. If Pengi still scores near zero and the trained models still fall below majority on open-ended/multiple-choice, the heterogeneity claim survives. If models improve markedly or rankings change, the aggregate conclusions are confounded by unanswerable metadata questions.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim — that RA-QA's heterogeneity exposes limited robustness of current approaches — is supported only by aggregate MacroF1/MAE scores in Table 2. Those aggregates pool QA pairs from all four attribute families in Sec. 2.1: acoustic features, consultation context, demographics/health profile, and recording context. For the last three, the answer is drawn from source-dataset metadata (e.g., age, sex, symptom/testing status, recording device, SNR). The model's only inputs are the audio clip and the question text; for these pairs the answer is not present in either input. A model can only guess from training-set priors. The paper's own audio-only oracle baseline reaches only 0.49/0.57 MacroF1 on open-ended/multiple-choice, and for metadata attributes it is near chance by design. The label-balanced subset (Sec. 3.3) samples within each attribute, so unanswerable attributes are guaranteed to be represented. Pooling them with acoustically grounded items means the reported 'clinical correctness' conflates audio understanding with prior memorization. Consequently, the conclusion that Pengi and CaReAQA-style models 'fail under heterogeneity' may be an artifact of task design rather than evidence about respiratory audio QA. No per-attribute or per-dataset breakdown is provided; the two case studies do not fill this gap.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RA-QA, a benchmark for respiratory audio question answering. It curates 11 public respiratory-audio datasets, standardizes their metadata, and programmatically generates roughly 9 million QA pairs using template questions in three formats (open-ended, multiple-choice, single-verify) across four attribute families (acoustic features, consultation context, demographics/health profile, recording context). The authors benchmark several baselines: random and majority classifiers, an audio-only SVM, a late-fusion multimodal classifier, zero-shot Pengi, and a CaReAQA-style generative audio-to-LLM model. Results are reported as MacroF1/MAE for task-level correctness and BERTScore for semantic fidelity. The main claims are that RA-QA is a large, heterogeneous, reproducible benchmarking resource, and that it exposes the limited robustness of current audio-language models for respiratory audio QA.","tokens_in":9207,"tokens_out":4648,"duration_ms":52925,"significance":"If the resource is built as described, RA-QA is a substantial and potentially valuable contribution: it provides a standardized generation pipeline, a large public QA collection, and reproducible baseline evaluations with released code and QA pairs. The observation that high semantic similarity can coexist with low task-level correctness is worth reporting and will be of interest to the audio-language and health-AI communities. However, the headline conclusions about model failure and clinical correctness rest on aggregate scores that pool fundamentally different question types; these conclusions require per-attribute and per-dataset evidence before they can be accepted. The paper's strengths are its scale, reproducibility, and the inclusion of multiple baseline families.","major_comments":[{"comment":"The global benchmark scores pool all four attribute families, but only 'acoustic features' are answerable from the recording itself. For consultation context, demographics, and recording context, the answer is drawn from source metadata and is largely not recoverable from the audio and question alone; the audio-only baseline is near chance on those attributes by design. The label-balanced subset (Sec. 3.3) ensures these metadata-only attributes are present in the test set. Consequently, the aggregate MacroF1/MAE conflates audio understanding with prior memorization, and the conclusion that Pengi and CaReAQA-style models 'fail under heterogeneity' may be an artifact of task design. The two case studies in Table 3 do not fill this gap. I request per-attribute and per-dataset breakdowns, or a separate reporting of audio-answerable versus metadata-only questions.","section":"Sec. 2.1 and Table 2"},{"comment":"The 9M-pair scale is dominated by a single source: UK COVID-19 contributes 8,089,056 of 8,996,137 QA pairs (~90%), and also accounts for 232,129 of the 234,064 unique answers. This does not invalidate the resource, but it weakens the claim of broad heterogeneity unless results are shown to be consistent across sources. At present, the headline conclusions are driven almost entirely by one self-reported dataset, and no per-source analysis is provided.","section":"Table 1"},{"comment":"The benchmark treats source metadata labels, including self-reported symptoms and statuses, as ground truth for scoring 'clinical correctness.' The manuscript acknowledges that some labels are clinically validated and others are self-reported, but the evaluation protocol makes no distinction. This is an unvalidated proxy, and calling task-level accuracy 'clinical correctness' overstates the medical meaning of the numbers. A sensitivity analysis restricted to clinically validated subsets (e.g., ICBHI) or a separate reporting of self-reported vs. clinically validated labels is needed to support the clinical framing.","section":"Sec. 2.1 and Sec. 3.2"},{"comment":"The main trained baselines — audio-only SVM, multimodal classifier, and CaReAQA-style model — are reported without error bars or multiple-seed variation. Only the random baseline is described as repeated over 5 seeds. Given that the subset is sampled and the train/validation/test split is partly stratified, the observed differences, such as 0.59 vs. 0.51 on single-verify or 0.11 vs. 0.16 on multiple-choice, may be within variance. Confidence intervals or repeated runs with a fixed seed scheme are necessary to support the comparative claims.","section":"Sec. 3.1 and Table 2"}],"minor_comments":[{"comment":"The regression row header reads 'Regressive Open-ended'; 'Regressive' should be 'Regression.' Also, BERTScore is not reported for the multimodal classifier and Pengi is N/A for regression; the caption should explain these omissions so readers can compare across methods.","section":"Table 2"},{"comment":"The split rule is described as '70/15/15' but also 'Assigned splits followed the original dataset protocols when available.' Please clarify which datasets used original protocols and which used 70/15/15, and whether any dataset's original test set was reused in RA-QA's evaluation.","section":"Sec. 2.2"},{"comment":"The semantic-level metric (BERTScore) is very high for random and majority baselines (0.92–0.99), because template-generated answers are short and predictable. The paper should discuss what BERTScore adds beyond template overlap and why such high semantic fidelity is not itself evidence of clinical usefulness.","section":"Sec. 3.2"},{"comment":"The abstract says 'classical ML baselines' but the only unimodal ML baseline is an SVM. Consider naming it explicitly or broadening the description to 'shallow baselines.'","section":"Abstract / Sec. 3.1"},{"comment":"Reference [18] contains 'J. Ch' which appears to be an incomplete author name; please verify and complete. Also ensure the anonymous repository link will be replaced with a persistent DOI or archival link at publication.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable benchmark contribution with good reproducibility practices, but the central claims are currently broader than the evidence. The most important fix is to report results separately for audio-answerable versus metadata-only attributes, and preferably per dataset. If the authors add these analyses and temper the 'clinical correctness' wording, I would be willing to reconsider; as is, the aggregate numbers do not yet support the conclusion that current audio-language models are specifically limited in respiratory audio understanding."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one-sentence version: RA-QA is a real resource — a 9M-pair QA collection over 11 respiratory audio datasets with three question formats and four attribute families, plus a standardized pipeline and released code. It fills a gap that needed filling. But the central claim that 'current approaches fail under heterogeneity' is only partly supported, because the benchmark pools audio-grounded questions with questions whose answers live entirely in metadata, so the aggregate task scores conflate audio understanding with guessing priors from training distribution.\n\nWhat's new and good: applying QA generation from metadata to respiratory audio at this scale is new; CaReAQA is smaller and doesn't cover the same scope. The pipeline is straightforward but sound: standardize metadata, template questions, instantiate per patient. The splits are patient-wise and supposedly leakage-aware. Including an audio-only SVM as an oracle-style upper bound is the right instinct, and the Pengi zero-shot failure is credible — it does behave like a captioning model.\n\nThe soft spots, in order. First, the stress-test concern holds up on reading. The four attribute families in Sec. 2.1 include consultation context, demographics/health profile, and recording context; for those, the answer is drawn from the source dataset's metadata and is not in the audio (or even the question). A model can only guess from priors. The audio-only oracle scoring 0.49/0.57 MacroF1 on open-ended/multiple-choice is actually consistent with many questions not being acoustically grounded. The label-balanced subset samples within each attribute, so these unanswerable pairs are guaranteed to be present. The paper's conclusion that models fail under heterogeneity would be much more convincing with a per-attribute breakdown, separating acoustic features from the other three families. Without it, the headline result may be partly a task-design artifact. Second, the trained baselines score below majority on open-ended and multiple-choice (0.11 vs 0.19, 0.12 vs 0.22) and the paper doesn't discuss this. That's a red flag for the training setup, not just the models. Third, no error bars for the trained baselines; only the random baseline is seeded. Minor but easy to fix.\n\nPatient-level metadata includes self-reported labels; the paper acknowledges this in Sec. 2.1 but then treats the answers as 'clinical correctness' in Sec. 3.2. That's not fatal for a benchmark, but the framing oversells it.\n\nBottom line: the resource is worth having and the evaluation is not fatally flawed, but the paper needs revision before it's a canonical reference. It deserves a serious referee — send it to peer review with the expectation of major revision: per-attribute/per-dataset results, separate audio-grounded from metadata-only pairs, error bars, and a real discussion of the below-majority trained baselines.\n\nFor you: worth a reading-group slot if your group cares about audio-language benchmarks, and worth citing as a resource once the evaluation is tightened.","headline":"Useful 9M-pair respiratory-audio QA benchmark, but the headline claim about model failure is muddied by pooling audio-unanswerable metadata questions with audio-grounded ones.","tokens_in":9591,"tokens_out":2935,"would_cite":true,"duration_ms":31582,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RA-QA turns 11 respiratory audio datasets into 9 million QA pairs and shows current audio-language models rarely answer clinically correctly.","keywords":["respiratory audio","question answering","benchmark","audio-language models","multimodal","healthcare","heterogeneity","audio QA"],"falsifier":"A clinician-annotation study: have respiratory clinicians listen to a sample of RA-QA recordings and answer the template questions, then compare their answers with the metadata-derived ground truth. If agreement is low on self-reported attributes (e.g., symptoms or diagnosis in crowdsourced datasets), the benchmark's 'clinical correctness' scores and the conclusion that models fail would need to be reinterpreted; alternatively, if a model fine-tuned on RA-QA achieves high task-level accuracy on real-world patient questions collected from a telemedicine service, the template-based benchmark wou","tokens_in":8759,"feed_emoji":"🫁","tokens_out":3552,"duration_ms":36770,"temperature":0.7,"pith_summary":"This paper tries to establish that respiratory-audio question answering can be systematically benchmarked by converting existing public datasets into a large, format-diverse collection of question-answer pairs, and that doing so exposes a gap between what general audio-language models appear to understand and what they get clinically right. The authors build RA-QA: a pipeline that harmonizes 11 public respiratory audio datasets into 9 million QA pairs covering diagnostic and contextual attributes across cough, breath, speech, and auscultation recordings, with open-ended, multiple-choice, and yes/no question formats. They then evaluate classical and multimodal baselines under one protocol, showing that a general audio-language model transfers poorly (near-zero task-level accuracy on many question types) even when semantic similarity scores look moderate, and that question-conditioned trained models improve but remain far from clinically reliable. If correct, RA-QA provides the field with a reproducible stress test and a shared reference point for building respiratory-specific, question-conditioned models.","feed_headline":"9M respiratory-audio QA pairs expose AI's clinical blind spot","feed_subtitle":"New benchmark finds general audio-language models get answers fluent but clinically wrong; task-level scores near zero.","key_machinery":"The load-bearing mechanism is the RA-QA generation pipeline, which standardizes heterogeneous metadata, maps labels to descriptive text, instantiates JSON templates into open-ended, multiple-choice, and single-verify questions, and programmatically pairs each patient's metadata with their respiratory recordings to produce personalized QA pairs. The evaluation protocol's two metrics — BERTScore for semantic fidelity and MacroF1/MAE for task-level clinical correctness — are the instrument that reveals the semantic-versus-clinical gap.","core_discovery":"The central discovery is that RA-QA's heterogeneity — multiple question formats over the same recording, diverse devices and modalities, both categorical and numeric targets across four attribute families — breaks the assumption that semantic fidelity implies clinical correctness. The paper reports that a representative general audio-language model, prompted with audio and question, produces answers with near-zero task-level MacroF1 on open-ended and multiple-choice discriminative questions despite moderate BERTScore, and that even trained multimodal baselines show a wide gap between linguistic quality (up to 0.96 BERTScore) and clinical accuracy (0.11–0.59 MacroF1). The authors interpret th","pith_inferences":["Because RA-QA answers are generated from templates and metadata, the benchmark likely underestimates the difficulty of free-form patient questions; a natural extension is to collect real user queries and re-ask them against the same recordings.","The observed disconnect between BERTScore and MacroF1 suggests that other medical audio QA systems evaluated only on text-level metrics may be similarly overrated; re-scoring existing systems with label-extraction metrics is a direct test.","RA-QA's 9M pairs could serve as pretraining data for respiratory-specific audio-language models; the authors' own CaReAQA-style baseline hints that such training helps but is not sufficient, so scaling with more diverse formats may be a promising next step.","The self-reported nature of some source labels, if compared against clinically confirmed diagnoses, could be used to quantify label noise and its effect on benchmark scores."],"forward_implications":["General audio-language models that perform well on audio captioning cannot be assumed to transfer to respiratory QA; zero-shot performance can be near chance at the task level.","Reporting only semantic similarity can overstate clinical reliability; benchmark evaluations should report task-level correctness alongside text-level fidelity.","Question conditioning helps: a late-fusion multimodal classifier improves MacroF1 on single-verify questions over both audio-only and majority baselines, indicating that the question text carries disambiguating information.","Domain-trained audio-to-LLM alignment produces fluent answers (BERTScore up to 0.96) but still limited task correctness, so further work is needed to close the semantic-clinical gap.","The benchmark's leakage-aware splits and unified protocol allow reproducible comparison across future models."],"fun_headline_variants":["AI fluent but clinically wrong on respiratory audio QA","9M respiratory QA pairs: high fluency, near-zero accuracy","Benchmark reveals AI's clinical blind spot in respiratory audio","Heterogeneous audio QA: BERTScore high, MacroF1 near zero"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The benchmark treats the source datasets' metadata labels — some self-reported, some clinically validated — as ground truth for clinical correctness, and assumes that template-generated questions resemble the way real clinicians and patients will ask about respiratory audio.","fun_headline_variants_meta":{"raw":{"variants":["AI fluent but clinically wrong on respiratory audio QA","9M respiratory QA pairs: high fluency, near-zero accuracy","Benchmark reveals AI's clinical blind spot in respiratory audio","Heterogeneous audio QA: BERTScore high, MacroF1 near zero"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00058,"raw_usage":{"total_tokens":2525,"prompt_tokens":658,"completion_tokens":1867,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":402,"completion_tokens_details":{"reasoning_tokens":1796}},"tokens_in":402,"tokens_out":1867,"duration_ms":14354,"temperature":1.0,"reasoning_tokens":1796,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T04:33:43.309051+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A clinician-annotation study: have respiratory clinicians listen to a sample of RA-QA recordings and answer the template questions, then compare their answers with the metadata-derived ground truth. If agreement is low on self-reported attributes (e.g., symptoms or diagnosis in crowdsourced datasets), the benchmark's 'clinical correctness' scores and the conclusion that models fail would need to be reinterpreted; alternatively, if a model fine-tuned on RA-QA achieves high task-level accuracy on real-world patient questions collected from a telemedicine service, the template-based benchmark wou","supporting_citations":[],"review_version":1}